Lecture Week 9 Joint / Multivariate Probability Dists

Multivariate poss and ass probs

27/11/2012 ST2004 Wk 9 2012 1

Joint Probability Distributions

Random Variable Y a name

Two lists: (i) poss values y (ii) ass. probs

Tabulated, or Defined by a Formula

Discrete or Continuous

Expected Value of Y (or a function g(●) of Y)

Weighted Avg of poss values y (or g(y))

Summation ∑ or Integral ∫

27/11/2012 ST2004 Wk 9 2012 2

Wk8 Y scalar. Wk9 Y multivar. Mini-league EXCEL

In this course we only consider discrete, tabulated, multivariate prob dists. The most important multivar dist is the multivariate Normal; see Tijms 3rd Ed Ch 12

Multivariate Prob Dist • Marginal, Joint, Conditional Prob Dists

– Discrete, pmf

– Illustration Mini League

• Probs of Composite Events – Probability rules

• (Conditional) Exp Val and Vars – Covariance, Correlation

• Application – Optimal Prediction

27/11/2012 ST2004 Wk 9 2012 3

Winners

(i)(ii)(iii)

ABA 0.060 0.056 210 0.060 0.056

ABC 0.230 0.224 201 0.061 0.084

ACA 0.061 0.084 120 0.027 0.024

ACC 0.367 0.336 111 0.257 0.260

BBA 0.027 0.024 102 0.367 0.336

BBC 0.091 0.096 021 0.091 0.096

BCA 0.027 0.036 012 0.137 0.144

BCC 0.137 0.144

1 1 1 1

Prob dist of games won

A 0.121 0.140 by 0 1 2

B 0.118 0.120 A 0.240 0.620 0.140 1

C 0.504 0.480 B 0.420 0.460 0.120 1

#N/A 0.257 0.260 C 0.080 0.440 0.480 1

1 1

Outright

Winner

Rel

Freq Prob

Rel

Freq ProbPoints

for ABC

Rel

Freq Prob

Mini-League Multivariate Dists 3 teams play each other once

Simulation; 1000 reps

(i) AB 0.7 0.3

(ii) BC 0.4 0.6

(iii) AC 0.2 0.8

Game Prob of Winning

Pr(A winner)

Pr(B winner)

Pr(A winner)

Exp Val Var

A 0.9 0.37

B 0.7 0.45

C 1.4 0.4

27/11/2012 ST2004 Wk 9 2012 4

Using eg

Pr Winners of ( ),( ),( ) are A,B,A, resp

=Pr A wins ( ) B wins ( ) A wins ( )

Pr A wins ( ) Pr B wins ( ) Pr A wins ( )

0.7 0.4 0.2

i ii iii

i AND ii AND iii

i ii iii

Simulation Theory

Mini-League Multivariate Dists 3 teams play each other once

Simulation; 1000 reps

(i) AB 0.7 0.3

(ii) BC 0.4 0.6

(iii) AC 0.2 0.8

Game Prob of Winning

Pr(A winner)

Pr(B winner)

Pr(A winner)

Exp Val Var

A 0.9 0.37

B 0.7 0.45

C 1.4 0.4

Winners

(i)(ii)(iii)

ABA 0.060 0.056 210 0.060 0.056

ABC 0.230 0.224 201 0.061 0.084

ACA 0.061 0.084 120 0.027 0.024

ACC 0.367 0.336 111 0.257 0.260

BBA 0.027 0.024 102 0.367 0.336

BBC 0.091 0.096 021 0.091 0.096

BCA 0.027 0.036 012 0.137 0.144

BCC 0.137 0.144

1 1 1 1

Prob dist of games won

A 0.121 0.140 by 0 1 2

B 0.118 0.120 A 0.240 0.620 0.140 1

C 0.504 0.480 B 0.420 0.460 0.120 1

#N/A 0.257 0.260 C 0.080 0.440 0.480 1

1 1

Outright

Winner

Rel

Freq Prob

Rel

Freq ProbPoints

for ABC

Rel

Freq Prob

Challenge; obtain directly. Eg; what must the true iff NA=1?

27/11/2012 ST2004 Wk 9 2012 5

Answers

Probabilities and Events Events

Elementary A wins in match (i) T/F

Composite A wins league T/F

NANBNC = 021 T/F

Probabilities Value in (0,1) for status (T/F) of event

Represents uncertainty given information Pr(A wins league, given probs for each match, and indep)

Pr(A wins league, given that somebody wins, and probs, and indep)

Pr(A wins league, given no info, other than probs, and indep)

27/11/2012 ST2004 Wk 9 2012 6

Shorthand for ( (A wins 0) AND (B wins 2) AND (C wins 1) )

Composite Events

• Defining Events

X = (NA=0) where NA= games won by A

• Decomposing Composite Events

X = (NA=0) ≡ (A loses (i)) AND (A loses (iii))

X = (Team A tops the table)

Recursion for K teams

Score from k dice

Can’t simulate? Events not defined

Event Identity. Boolean operator

27/11/2012 ST2004 Wk 9 2012 7

Dice Recursion - explanation Event Identity

(Sk=r) ≡ (Sk-1=r-1)AND(Diek=1)OR…

OR(Sk-1=r-6)AND(Diek=6)

Pr(Sk=r) = Pr( (Sk-1=r-1)AND(Diek=1) )+ …

+Pr( (Sk-1=r-6)AND(Diek=6) )

=

=

= ( Pr(Sk-1=r-1) + … Pr(Sk-1=r-6) )/6

27/11/2012 ST2004 Wk 9 2012 8

27/11/2012 ST2004 Wk 9 2012

1 1 2

, generic / events 0 Pr( ),Pr( ) 1

Pr Pr Pr Pr

Pr Pr | Pr Pr | Pr

? Pr Pr

Pr 0 ,

; Pr 1

,

Special cases

given Y True

k k

X Y T F X Y

X Y X Y X AND Y

X Y X Y Y Y X X

better X Y

X AND Y mut exclusive disjoint

If X X exhaustive ie X OR X OR X

and mut excl th

O

n

D

R

AN

e

1 2Pr Pr Pr 1

Pr | Pr

kX X X

X Y X independent

Probability Rules; more than one event

9

Mini-League Event Identities

Events (A OR B tops table) leads to Pr (A OR B tops table) (A OR B wins league) leads to….. (C does not top table) Events for (Marg) Prob dist NA) (Joint) Prob dist of NA,NC

27/11/2012 ST2004 Wk 9 2012 10

Mini-League Prob Rules using Event Identities

Probs from explicit Event Identities Pr(A OR B tops table) Pr(A OR B wins league) Pr(C does not top table) (Marg) Prob dist NA

(Joint) Prob dist of NA,NC

Prob dist of games won

by 0 1 2

A 0.240 0.620 0.140 1

B 0.420 0.460 0.120 1

C 0.080 0.440 0.480 1

210 0.056

201 0.084

120 0.024

111 0.260

102 0.336

021 0.096

012 0.144

Points

for ABC Prob

27/11/2012 ST2004 Wk 9 2012 11

Probs from explicit Event Identities for more elementary events Pr(A OR B tops table)=Pr(210)+Pr(201)+Pr(111)+Pr(120)+Pr(021) =Pr(A tops) + Pr(B tops) – Pr(both top) Pr(A OR B wins league outright)=Pr(210)+Pr(201)+Pr(120)+Pr(021) Pr(C does not top table)=Pr(A OR B tops table)=1-Pr(C tops table) (Marg) Prob dist NA Pr(NA =0) Pr(NA =1) Pr(NA =2)

=Pr(021)+Pr(012) =0.240 0.620 0.140 (Joint) Prob dist of NA,NC eg Pr(NA=1,NC =1) =Pr(111)=0.260

Prob dist of games won

by 0 1 2

A 0.240 0.620 0.140 1

B 0.420 0.460 0.120 1

C 0.080 0.440 0.480 1

210 0.056

201 0.084

120 0.024

111 0.260

102 0.336

021 0.096

012 0.144

Points

for ABC Prob

27/11/2012 ST2004 Wk 9 2012 12

Answers In all cases several alternative constructions poss Mini-League

Prob Rules using Event Identities

Mini-League Joint Probabilites

Knowing that , conditional on, C wins 1 game (Marg) Prob dist of NA

(Joint) Prob dist of NA,NC

27/11/2012 ST2004 Wk 9 2012 13

‘jitter’ added as away of showing that the 9 red dots do not carry equal probabilities

Cov= -0.63

0 1 2

0 0.000 0.144 0.096 0.24

A 1 0.336 0.260 0.024 0.62

2 0.084 0.056 0.000 0.14

0.42 0.46 0.12 1

BDist AB

points

Answers

0 1 2

0 0.000 0.144 0.096

Dist AB

points

B

Knowing that , conditional on, C wins 0 game (Marg) Prob dist of NA

(Joint) Prob dist of NA,NB

0 0 1 2

0 0 0 0 0

A 1 0 0 27 27

2 0 60 0 60

0 60 27 87

B

Dist AB pts

given C=counts

27/11/2012 ST2004 Wk 9 2012 14

Frequencies Note in sample of 1000 sims, 87 have C winning 0 Of these 27 have NA= 1, NB= 2 Rel Freq of (NA=1,NB=2,NC=0)=27/1000 Of cases where NC=0 Rel Freq of (NA=1,NB=2)=27/87

Mini-League Conditional Probability Dists

Knowing that , conditional on, C wins at least 1 game (Marg) Prob dist of NA

(Joint) Prob dist of NA,NB

Winners

(i)(ii)(iii)

ABA 0.060 0.056

ABC 0.230 0.224

ACA 0.061 0.084

ACC 0.367 0.336

BBA 0.027 0.024

BBC 0.091 0.096

BCA 0.027 0.036

BCC 0.137 0.144

Rel

Freq Prob

27/11/2012 ST2004 Wk 9 2012 16

Mini-League Conditional Probability Dists

27/11/2012

#N/A

0 1 2

0 0.00 0.16 0.10 0.26 E[NA|C>0] 0.83

A 1 0.37 0.28 0.00 0.65 Var[NA|C>0] 0.32

2 0.09 0.00 0.00 0.09

0.46 0.44 0.10 1 E[NANB] 0.28

E[NB|C>0] 0.65 Cov[NA,NB|NC>0] -0.26

Var[NB|C>0] 0.44

Dist AB pts

given C>0

Pr(C>0)=

B

Knowing only elementary probs Prob dist of NA

Prob dist of NA,NB

Knowing additionally that C wins at least one game Prob dist of NA

Prob dist of NA,NB

0.9

0.37

, 0.21

| 0 0.83

| 0 0.32

, | 0 0.26

A

A

A B

A C

A C

A B C

E N

Var N

Cov N N

E N N

Var N N

Cov N N N

ST2004 Wk 9 2012 18

Cov= -0.21

0 1 2

0 0.000 0.144 0.096 0.24

A 1 0.336 0.260 0.024 0.62

2 0.084 0.056 0.000 0.14

0.42 0.46 0.12 1

Dist AB

points

B

Mini-League Expected Values Variance, Covariance

Exp Val, Variance for Sums

27/11/2012 ST2004 Wk 9 2012 20

,( ) ( )

2 ,

all x y

A B

A B

E X Y PossVals x y Pr X Y x y

E X E Y

Var X Y Var X Var Y Cov X Y

E N N

Var N N

2 2 2

More general

... ...

...

2 , 2 , ...

E aX bY cZ aE X bE Y cE Y

Var aX bY cZ a Var X b Var Y c Var Z

abCov X Y acCov X Z

Theory Tijms, Ch 9

Expected Values Directly via Sums

0.9

0.3

A

A

E N

Var N

27/11/2012 ST2004 Wk 9 2012 21

#N/A

0 1 2

0 0.00 0.16 0.10 0.26 E[NA|C>0] 0.83

A 1 0.37 0.28 0.00 0.65 Var[NA|C>0] 0.32

2 0.09 0.00 0.00 0.09

0.46 0.44 0.10 1 E[NA,NB] 0.28

E[NB|C>0] 0.65 Cov[NA,NB|NC>0] -0.26

Var[NB|C>0] 0.44

Dist AB pts

given C>0

Pr(C>0)=

B

#wins against B + # wins again

Alternative Simpler Calculatio

st C

0 / 1

0 1

0.3 0.7

0.7; 0.3 0.7 0.21

0.2

n

;

A

A plays B A plays C

A plays B

A plays B A plays B

A plays C A play

N

I I Binary vars

Dist I

Poss

Prob

E I Var I

E I Var I

0.2 0.8 0.16

0.7 0.2

0.21 0.16

s B

A A plays B A plays C

A A plays B A plays C

E N E I I

Var N Var I I

0.3 0.4

0.3 0.7 0.4 0.6

0.6 0.8

0.6 0.4 0.8 0.2

B

B

C

C

E N

Var N

E N

Var N

Theory follows

Knowing additionally that C wins at least one game Cond Prob dist of NA

Cond Prob dist of NA,NB

| 0 0.83

| 0 0.32

, | 0 0.26

A C

A C

A B C

E N N

Var N N

Cov N N N

27/11/2012 ST2004 Wk 9 2012 23

Mini-League Conditional Expectations

Answers

Joint Bivariate Conditional Prob Dists

0.920

0 1 2

0 0.000 0.157 0.104 0.261 E[NA|C>0] 0.83

A 1 0.365 0.283 0.000 0.648 Var[NA|C>0] 0.32

2 0.091 0.000 0.000 0.091

0.457 0.439 0.104 1

E[NB|C>0] 0.65 Cov[NA,NB|NC>0] -0.26

Var[NB|C>0] 0.44

Dist AB pts

given

Pr(C0)=

Pr(C0)=

B

Conditionally Decomposing Expectation

2

Roll Die

Roll another die = score

2,4 Roll 2 dice max

6 Toss coin : 2; : 6

1 1 1 1 13.5 (4.47) 2 6

2 3 6 2 2

3.907

Odd Y

Y

Head Y Tail Y

E Y

E Y

Var Y

27/11/2012 ST2004 Wk 9 2012 24

Conditionally Decomposing Expectation Simulation

27/11/2012 ST2004 Wk 9 2012 25

choose by dice roll

1

score

one die

2

max

two die

3

(2,6)eq

prob Roll

(1,3,5)

(2,4) (6)

=>option

Hence

Y Option 1 2 3

1 1 3 6 6 3 6 probs 0.5 0.333 0.167

2 6 2 6 1 1 6 Avg 3.48 4.48 3.98

3 4 6 2 4 2 6 Avg Y Avg of Sq 15.01 22.01 19.87

4 3 2 2 4 2 2 3.89 3.90

5 3 2 2 2 2 2 Avg Y2

6 5 5 2 3 1 5 18.14 18.15

7 5 3 2 1 1 5 Var Y

8 5 5 6 2 2 5 3.00 Hence Var 2.97

9 3 5 6 1 1 3

10 4 2 2 6 3 2

9999 4 5 6 4 2 5

10000 1 6 6 6 3 6

options

Wted Avg Options

Wted Avg (Sq Options)

Conditionally Decomposing Expectation Theory

Pr( ) Pr( , )

Pr( | ) Pr( )

| Pr( )

x x y

y x

y

E X x X x x X x Y y

x X x Y y Y y

E X Y y Y y

27/11/2012 ST2004 Wk 9 2012 26

0 1 2 NA 0 1 2

0 0.000 0.144 0.096 0.24 0 0 0.6 0.4 1 1.40 0.336

NA 1 0.336 0.260 0.024 0.62 1 0.542 0.419 0.039 1 0.50 0.308

2 0.084 0.056 0.000 0.14 2 0.6 0.4 0 1 0.40 0.056

Marg Dist B 0.42 0.46 0.12 1

E[NB] 0.7 0.7

ExpVal

NB

given

NA

NB

Mult

by

Pr(NA)

Cond dists given

NA

Jt Dist AB

points

NB

Conditionally Decomposing Expectation

1 1 12 3 6

2 1 1 12 3 6

Roll Die

Roll another die = score

2,4 Roll 2 dice max

6 Toss coin : 2; : 6

3.5 (4.47) 4

3.907

? (?) ?

Odd Y

Y

Head Y Tail Y

E Y

E Y

Var Y

27/11/2012 ST2004 Wk 9 2012 27

13

13

13

Choose from A, B, C with equal prob

number of games won by random team

prob

prob

prob

[ ]

[ ]

A

B

c

N

N N

N N

N N

E N

Var N

Challenge: project

Answers

Prediction : Theory

27/11/2012 ST2004 Wk 9 2012 28

Know C wins 1 game; what to predict for NA?

Optimal prediction

Know C wins 1 game; what to predict for NA?

• With Conditional Dist for NA given NC =1

– Most (cond) probable NA cond mode

– Conditional Exp Val NA min pred var

• With Exp Vals, Vars, Cov

– Best Linear Predictor least sq regression

27/11/2012 ST2004 Wk 9 2012 29

Pred NA = -0.40 + 1.46(NC)

Optimal pred: Cond Dist of X|Y=y

Pred by Most Prob, given NC =1 , predict as 1.0

Pred by Cond Exp Val , given NC =1 predict as 0.97

Theory: (Cond) Exp Val is ‘Min Sq Error Predictor’

All available info about X captured by cond prob dist, ie given Y

‘Min Sq Error Predictor’ is pred value that

minimises E[ Sq Error of Prediction]

(i) AB 0.7

(ii) BC 0.4

(iii) AC 0.2

Game Prob of Winning

Pr(A winner)

Pr(B winner)

Pr(A winner)

0.440

0 1 2 Given C = 1

0 0.000 0.000 0.218 0.218 0 1 2

A 1 0.000 0.591 0.000 0.591 0.22 0.59 0.19 1

2 0.191 0.000 0.000 0.191

0.1909 0.5909 0.2182 1 Exp Val 0.97

Var 0.41

Dist

NA

BDist AB pts

given C=1

Pr(C=1)=

27/11/2012 ST2004 Wk 9 2012 30

Min Sq Error Predictor

2

2

22

2

Predict random variable by some (specific) to be chosen

E is Squared Error -

Seek that that minimises E -

E - E - - -

-

= is that value of wit

X X

X X X

X X X

X X X X

Var X X

X X

2

h min E - X X

Theory general Not limited to discrete dists

27/11/2012 ST2004 Wk 9 2012 31

Prob dist of [ ] best

Given [ | ] best

X E X

Info E X Info

Best Lin Pred from Corr( X,Y) given that Y = y

Often prediction based on corr

not on cond dist.

Best Linear Pred

linear fn of obs value y of Y

chosen to min E[ Sq Error of Pred]

Exp Val Var

A 0.9 0.37

B 0.7 0.45

C 1.4 0.4

Th Corr

A B C

A 1 -0.515 -0.416

B -0.515 1 -0.566

C -0.416 -0.566 1

Cov= -0.21

0 1 2

0 0.000 0.144 0.096 0.24

A 1 0.336 0.260 0.024 0.62

2 0.084 0.056 0.000 0.14

0.42 0.46 0.12 1

Dist AB

points

B

confirm

27/11/2012 ST2004 Wk 9 2012 32

Jt Prob Dist leads to Cond Dist & Corr

Corr lead to Codoes not nd Dist

Best Linear Predictor

2 2

2 2

2

2

[ , ]; ; ; ; [ , ]

Seek , (and thus ) that minimises E -

-

Predict by some =

related random variable; to be chosen

X X Y Y

X Y

X Y

Cov X YE X Var X E Y Var Y Corr X Y

a b X X X

E X X

X X b Y equiv X a bY

Y b

2

2 22

22 2 2

2

- - 2 - - -

- 2 , 1

is that value of with min -

=

X Y

X X Y Y

X Y X

X

Y

X XX Y X Y

Y Y

E X b Y

E X b X Y b Y

Var X bCov X Y b Var Y b

b b E X X

X Y

X

Y

Y

Cf generic least-squares regression

Theory general Not limited to discrete dists

27/11/2012 ST2004 Wk 9 2012 33

Best Lin Pred from Corr(NA, NC) given that NC = 1 Exp Val Var

A 0.9 0.37

B 0.7 0.45

C 1.4 0.4

Th Corr

A B C

A 1 -0.515 -0.416

B -0.515 1 -0.566

C -0.416 -0.566 1

Pred NA = a + b(1) b =((0.37)/(0.40))0.5(-0.416) =-0.40 a =0.9-(-0.47)(0.4) = 1.46 Pred NA = 1.06 (cf 1.00, 0.97)

Cov= -0.16

0 1 2

0 0.000 0.096 0.144 0.24

A 1 0.024 0.260 0.336 0.62

2 0.056 0.084 0.000 0.14

0.08 0.44 0.48 1

CDist AC

points

27/11/2012 ST2004 Wk 9 2012 34

Homework

(i) AB 0.7

(ii) BC 0.5

(iii) AC 0.7

Game Prob of Winning

Pr(A winner)

Pr(B winner)

Pr(A winner)

• Compute: Joint Prob Dist for NA, NC; Corr [NA, NC] Prob Dist (NA- NC ) Marginal Dists, Exp Val, Var E [NA-NC] Var [NA-NC]

• Law of Cond Expectations Team A prize = (NA)2 if NB > NC

= (NA) else What is E[prize]? • Prediction for NA when NC = 0,1,2

Min Sq Error Pred for NA each NC = 0,1,2 Best Lin Pred of NAC each NC = 0,1,2

Mini League

27/11/2012 ST2004 Wk 9 2012 35

From dist NA-NC & from formulae