C HAPTER 5 Pairs of Random Variables

The Graduate School of Information Technology and Telecommunications, INHA UniversityThe Graduate School of Information Technology and Telecommunications, INHA Universityhttp://multinet.inha.ac.krhttp://multinet.inha.ac.kr

Multimedia Network Lab.Multimedia Network Lab.

CHAPTER 5

Pairs of Random Variables

Prof. Sang-Jo [email protected]

http://multinet.inha.ac.kr

The Graduate School of Information Technology and Telecommunications, INHA UniversityThe Graduate School of Information Technology and Telecommunications, INHA University

http://multinet.inha.ac.kr Multimedia Network LabMultimedia Network Lab..

What we are going to study? Extend the concepts to two random variables

Joint pmf, cdf, and pdf to calculate the probabilities of events that involve the joint behavior two random variables.

Expected value to define joint moments Correlation when they are not independent Conditional probabilities involving a pair of random variables.

2



3

Multiple Random Variables Vector Random Variable

X is a function that assigns a vector of real numbers to each outcome in S (sample space of the random experiment).

Example The random experiment of selecting one student from a class,

define the following functions : H ( ) = height of student in inches

W ( ) = weight of student in pounds A ( ) = age of student in years. (H ( ), W ( ), A ( )) is vector random variable

A function that assigns a pair of real numbers to each out come in sample space S.

Example 5.1 Example 5.2

,X Y X



Two Random Variables

4

S

A

X

B

x

y

The event involving a pair of RV (X,Y) are specified by conditions that we are interested in A={X+Y 10}, B={min(X,Y) 5}, C={X2+Y2 100}

To determine the probability that the pair X=(X,Y) in some region B in the plane, find the equivalent event for B in the underlying sample space S:

1 : , in A B X Y B X

in : , in P B P A P X Y B X



5

Events and Probabilities For -dimensional random variable X= , we

are particularly interested in events that have the product form

in in in where is a one-dimensional event that involves only Probability of product-form events in in in in in Probability of non-product form events B is approximated by the union of disjoint product-form

events

n 1 2 3( , , )X X X

1{A X 1}A 2X 2}A { nX }nAkA kX

1[ ] [{P A P X1}A 2X 2}A { nX }]nA

1[P X 1, , nA X ]nA

[ ] [ ] [ ]k kk k

P B P B P B



6

Example: Product-Form Events

X

y

Example 1

1 2 2{ } { }x X x Y y

1 2( , )x y 2 2( , )x y

x

y

x1 x2

y2

y1

Example 2

1 2 1 2{ } { }x X x y Y y



Pairs of Discrete Random Variables(1)

Vector random variable assumes values from some countable set SX,Y = {(xj, yk), j = 1, 2, • • •, k = 1, 2, • • •}. The joint probability mass function of specifies the probability of the product-form

event {X = xj} ∩ {Y= yk}, j = 1, 2, • • • ,k = 1, 2, • • •. This can be interpreted as the long-term relative frequency of the joint event {X =

xj} ∩ {Y= yk} in a sequence of repetitions of the random experiment.

The probability of any event B is the sum of the pmf over the outcomes in B

Note 5-1 (Example 5.5) How to show pmf graphically: Figure 5.5

7

,X YX

X

,( , )

[ ] ,j k

X Y j kx y B

P in B p x y

X

, ( , ) [( ) ( )] [ , ]X Y j k j k j kp x y P X x Y y P X x Y y

,1 1

( , ) 1X Y j kj k

p x y



Different Pairs of Random Variables

8




Marginal probability mass functions The joint pmf of X=(X,Y) provides the information about the joint

behavior of X and Y. We are also interested in the probabilities of events involving

each of the RV in isolation.

Similarly,

9

,1

( ) [ ][ , ]

( , );

X j j

j

X Y j kk

p x P X xP X x Y anyting

p x y

,1

( ) ( , ).Y k X Y j kj

p y p x y




Example A urn contains 3 red, 4 white and 5 blue balls. Now, 3 balls are drawn. Let

X and Y be the number of red and white balls chosen, respectively, find the joint probability mass function of X and Y,

10

}].{}[{),(, kYjXPkjP YX

j k 0 1 2 3 PX(k)0 10/220 40/220 30/220 4/220 84/2201 30/220 60/220 18/220 0 108/22

02 15/220 12/220 0 0 27/2203 1/220 0 0 0 1/220

PY(k) 56/220 112/220

48/220 4/22022060)1,1(

312

151413,

CCCCP YX

22012)1,2(

312

1423,

CCCP YX




Example 5.9 The number of bytes N in a message has a geometric

distribution with parameter p and range SN = {0, 1, 2, • • •}. Messages are broken into packets of maximum length M bytes. Let Q be the number of full packet in a message, and R be the number of bytes left over.

Find the joint pmf and the marginal pmf’s of Q and R. Solution

Q is the quotient of division of N by M, and R is the remainig bytes in the above division. Q takes on values in {0, 1, • • •}; that is, all non – negative integers.

R takes on values in {0, 1, • • • , M – 1}. Interestingly, the joint pmf is relatively easier to compute

11

.)1(][],[ rqMpprqMNPrRqQP




Marginal pmf of Q is given by

Marginal pmf of R is found to be

12

.,2,1,0,))(1(1

1)1(

)1(

)}]1(,,1,{[][1

0

qppp

ppp

pp

MqMqMqMinNPqQP

qMMM

qM

M

k

kqM

0

[ ] [ { , , 2 , }]1(1 ) , 0,1, , 1.

1qM r r

Mq

P R r P N in r M r M rpp p p r M

p



Joint cdf of X and Y (1) Defined as the probability of the product-form event

Joint cumulative distribution function of X and Y

Properties (i) , this is because is a subset of .(ii) This is because and are impossible events.

(iii) Marginal cumulative distribution functions

13

}{}{ 11 yYxX

},{),( 1111, yYxXPyxF YX

),(),( 22,11, yxFyxF YXYX 2121 yyandxx }{}{ 11 yYxX }{}{ 22 yYxX

0),(),( 1,1, xFyF YXYX

1{ } { }X Y y 1{ } { }X x Y 1),(, YXF

][),()(

][],[),()(

,

,

yYPyFyF

xXPYxXPxFxF

YXY

YXX



Joint cdf of X and Y (2)(iv) Joint cdf is continuous from the ‘north’ and the ‘east’

This is a generalization of the right continuity property of the one-dimensional cdf

(v) The probability of the rectangle is given by

Example 5.12

Then

X and Y are exponentially distributed with respective parameter α and β

14

),(),(lim

),(),(lim

,,

,,

bxFyxF

yaFyxF

yXYXbx

yXYXax

,(1 )(1 ), 0, 0

( , )0

x y

X Ye e x y

F x yotherwise

,

,

( ) lim ( , ) 1 , 0

( ) lim ( , ) 1 , 0

xX X Yy

yy X Yx

F x F x y e x

F y F x y e y

1 2 1 2,x x x y y y

1 2 1 2 , 2 2 , 2 1 , 1 2 , 1 1, , , , ,X Y X Y X Y X YP x X x y Y y F x y F x y F x y F x y



Joint cdf of X and Y (3) The cdf can be used to find the probability of events that can

be expressed as the union and intersection of semi-infinite rectangles.

In particular,

15

),(),(),(

],[),(

21,11,12,

212122,

yxFyxFyxF

yYyxXxPyxF

YXYXYX

YX

,

, , , ,

3 5 3 2

5 2

(1 )(1 ), , 0Given ( , )

0

[1 3,2 5](3,5) (3,2) (1,5) (1,2)

(1 )(1 ) (1 )(1 )(1 )(1 ) (1 )(1 )

x y

X Y

X Y X Y X Y X Y

e e x yF x y

otherwise

P X YF F F F

e e e ee e e e



Joint PDF of Two Continuous RVs Backgrounds

Joint cdf allows us to compute the probability of events that correspond to “rectangular” shapes in the plane.

To compute the probability of events corresponding to regions other than rectangles, any shape can be approximated by rectangles Bj,k.

The probability of the event can be approximated by the sum of the probabilities of infinitesimal rectangles, and if the cdf is sufficiently smooth, the probability of each rectangle can be expressed in terms of a density function.

16

1A X Y

2 2 1B X Y

, ,,

,j k

j k X Y j kj k x y B

P B P B f x y x y



Joint pdf of two jointly continuous RVs

When the random variables X and Y are jointly continuous, the probability of an event involving (X,Y) can be expressed as an integral of a joint probability density function. For every event B, which is a subset of the plane, we have

17

,[ ] ( , )X YB non negative

P X in B f x y dx dy

Joint cdf:

The pdf can be obtained from the cdf by differentiation:

.),(),( ,, ydxdyxfyxFy x

YXYX

.),(

),( ,2

, yxyxF

yxf YXYX

2 1

2 11 1 2 2 ,

, ,

[ , ] ( , )

and [ , ] ( , ) ( , ) .

b bX Y

a a

y dy x dxX Y X Y

y x

P a X b a Y b f x y dx dy

P x X x dx y Y y dy f x y dx dy f x y dxdy



Joint pdf of two jointly continuous RVs

Since

18

,

[ , ] 1, we have

1 ( , ) .X Y

P X Y

f x y dx dy

The marginal pdf’s and are obtained by differentiating the marginal cdf’s.

Note 5-2 (Example 5.15, Example 5.16)

)(xf X )(yfY

, ,

, ,

,

( ) ( , ) ( ) ( , ),

( ) ( , ) ( , )

( ) ( , ) .

X X Y Y X Y

xX X Y X Y

Y X Y

From F x F x and F y F y wehavedf x f x y dy dx f x y dydx

f y f x y dy



Jointly Gaussian random variables X and Y are Gaussian random variables with zero mean and unit variance.

The last integral is recognized as the Gaussian pdf with mean ρx and variance 1- ρ2, so the value of integral is one

Hence, fx(x) is the one-dim Gaussian pdf with zero mean and unit variance.

19

.2)1(22

12

12)(

arg

,12

1),(

2/

2

)1(2/)(2/

)1(2/])[(

2

)1(2/

)1(2/)2(

2

)1(2/

,)1(2/)2(

2,

2222

222222

2222

222

xxyx

xxyx

xyyx

X

yxyxYX

edyee

dyee

dyeexf

byobtainedisXofpdfinalmThe

yxeyxf



Independence of two random variables

Two events are independent if the knowledge that one has occurred gives no clue to the likelihood that the other will occur.

Let X and Y be discrete random variables. Let A1 be the event that X=x, A2 be the event that Y=y. If X and Y are independent, then A1 and A2 are independent.

For continuous random variables: Example

X1= number of students attending the lecture on given day X2= number of tests within that week X3= number of students having a cold X4= number of students having hair cut

Which pair of random variables are independent?

20

1 2 1 2

,

[ ] [ ] [ ]( , ) [ ] [ ] [ ] ( ) ( )X Y X y

P A A P A P Aor p x y p X x and Y y P X x P Y y p x p y

, ( , ) ( ) ( )X Y X Yf x y f x f y



Independence Definition Let X and Y be random variables with joint density fXY and

marginal densities fX and fY, respectively. X and Y are independent if and only if

Remark By integrating the above equation, we have

So that

X and Y are independent if and only if their joint cdf is equal to the product of its marginal cdf’s.

Let X and Y be independent random variables, then the random variables defined by g(X ) and h(Y ) are also independent.

21

.,),()(),( yxallforyfxfyxf YXXY

y x

XY

y x

XY xdxfydyfydxdyxf )()(),(

.,)()(),( yxallforyFxFyxF YXXY



Example Consider the jointly distributed Gaussian random variables with

the joint pdf:

The product of the marginals equals the joint pdf if and only of ρ=0. Hence, X and Y are independent if and only if ρ=0.

What is the interpretation of ρ? It is related to a concept called correlation (to be discussed later).

22

2 2 2

2 2

2 2

( 2 )/2(1 ) ,,2

/2 /2

( )/2

1( , ) , .2 1

1 1( ) , ( )2 2

1( ) ( ) , , .2

x xy yX Y

x yX Y

x yX Y

f x y e x y

f x e f y e

f x f y e x y



Joint Moments and Expected Values of a Function of Two Random Variables

Remember: Expected value of X: indicates the center of mass of the

distribution of X Variance: expected value of (X-m)2, provides a measure of the

spread of the distribution.

We are interested in: How X and Y vary together? Whether the variation of X and Y are correlated?

If X increases does Y tend to increase or to decrease? The joint moments of X and Y, which are defined as expected

values of functions of X and Y, provide this information.

23



24

Expected value of functions of RVs Let Z=g(X,Y ), expected value of Z is given by

Sum of random variables (Example 5.24) Z=X+Y

The random variables do not have to be independent in order that the above formula holds.

( , ) ( , ) X,Y jointly continuous[ ] .

( , ) ( , ) X,Y dixcrete

XY

i j XY i ji j

g x y f x y dxdyE Z

g x y p x y

].[][][][ general,In

].[][')'('')'('

'')','()''(][

2121 nn

YX

XY

XEXEXEXXXE

YExEdyyfydxxfx

dydxyxfyxYXE



Product of functions of independent random variables: Example 5.25 Suppose X and Y are independent random variables, and g(X,Y )

is separable where g(X,Y )=g1(X )g2(Y ), then

In general, if are independent random variables, then

25

1 2 1 2

1 2

1 2

[ ( ) ( )] ( ') ( ') ( ') ( ') ' '

( ') ( ') ' ( ') ( ') '

[ ( )] [ ( )].

X Y

X Y

E g X g Y g x g y f x f y dx dy

g x f x dx g y f y dy

E g X E g Y

nXX ,,1

)].([)]([)]([)]()()([ 22112211 nnnn XgEXgEXgEXgXgXgE



Joint Moments, Correlation, and Covariance

The jkth joint moment of X and Y is defined by

Correlation of X and Y is defined by E[XY], specially in Electrical Engineering When E [XY ]=0, then X and Y are orthogonal.

The jkth central moment of X and Y is defined as

When j =2, k =0, it gives VAR(X); and when j=0, k=2, it gives VAR(Y). When j=k=1, it gives COV(X,Y)=E[X-E[X])(Y-E[Y])].

26

( , ) , X and Y jointly continuous[ ]

( , ) X,Y discrete

j k XYj k

j ki n XY i n

i n

x y f x y dxdyE X Y

x y p x y

].])[(])[[( kj YEYXEXE



Covariance

1. COV[X,Y]=E[XY] if either of the random variables has mean zero.2. When X and Y are independent, then E [XY]=E [X]E[Y] so that

Correlation coefficient of X and Y

Where σX and σy are standard deviations of X and Y, respectively.

27

].[][][][][][][2][

]][][][][[),(

YEXEXYEYEXEYEXEXYE

YEXEXYEYXEXYEYXCOV

0),( YXCOV

,][][][),(

yXyXXY

YEXEXYEYXCOV

If XY=0 then X and Y are said to be uncorrelated.If X and Y are independent, then COV(X,Y)=0 so uncorrelated.



Correlation Definitions E[XY]: maximize when X goes large Y also goes large. E[{X-E(X)}{Y-E(Y)}]: covariance, consider mean values of X and Y

Compute correlation at the equal conditions: conceptually zero mean If a positive (negative) value of (X-E(X)) tends to be accompanied by a

positive (negative) values of (Y-E(Y)); then COV will be positive. If they tend to have opposite signs, then COV(X,Y) will be negative. If they sometimes have the same sign and sometimes have opposite sign,

then COV(X,Y) will be close to zero. Correlation coefficient XY:

Multiplying either X or Y by a large number will increase the covariance, so need to normalize the covariance.

Properties of ρXY

and so -1 ≤ρXY ≤ 1.

28

),1(2121

])[(])[])([(2])[(

][][0

2

2

2

2

2

XYXY

YYXX

YX

YEYYEYXEXXEXE

YEYXEXE



29



Example 5.28 Suppose X and Y have the joint pdf:

The marginal pdf’s are found to be

Correlation coefficient,

30

.0

02),(

otherwisexyee

yxfyx

XY

0

0 0

0

2

0

0

.1)1(2

2][

.41)(

21][

;45][)1(2][

,23)1(2][

0,22)(

0),1(22)(

dxxeexe

dydxexyeXYE

YVARandYE

XedxeexXVAR

dxexeXE

xedxeeyf

xeedyeexf

xxx

x yx

xx

xx

yx

y

yxY

xxx yxX

.5

1

41

45

21

231

XY



Conditional Probability and Conditional Expectation

Many random variables in practical interest Are not independent Output Y of a communication channel must depend on the input

X Consecutive samples of a waveform that varies slowly are likely

to be close in value.

We are interested in Computing the probability of event concerning the random

variable Y given that we know X=x. The expected value of Y given X=x.

31



32

Conditional Probability Recall the formula:

Case 1: X is a Discrete Random Variable For X and Y discrete random variables,

If X and Y are independent

How to calculate joint pmf from conditional marginal pmf’s

How to calculate

.][

],[]|[xXP

xXAinYPxXAinYP

in

[ in | ] |j

k Y j ky A

P Y A X x p y x

,

| k jY j k j Y j

k

P X x Y yp y x P Y y p y

P X x

, ,X Y k jp x y

, , | |X Y k j Y j k X k X k j Y jp x y p y x p x p x y p y

in in

in

,all all

all all

in , |

| in |k j A k j A

k j A k

X Y k j Y j k X kx y x y

X k Y j k k X kx y x

P Y A p x y p y x p x

p x p y x P Y A X x p x

Example 5.29Example 5.30



Suppose X is a discrete RV and Y is a continuous RV Conditional cdf of Y given X=xk

Conditional pdf of Y given X=xk

If X and Y are independent

33

0][,][

],[)|(

k

k

kkY xXP

xXPxXyYPxyF

).|()|( kYkY xyFdydxyf

[ , ] [ ] [ ]so ( | ) ( ) and ( | ) ( )

k k

Y k Y Y k Y

P Y y X x P Y y P X xF y x F y f y x f y



Case 2: X is a Continuous Random Variable If X is a continuous random variable, then Suppose X and

Y are jointly continuous random variables with a joint pdf that is continuous and non-zero over some region, then the conditional cdf of Y given X=x is defined by

Taking h 0,

The conditional pdf of Y given X=x is given by

If X and Y are independent, then so and .

34

.0)( xXP

).|(lim)|(0

hxXxyFxyF YhY

( ', ') ' ' ( , ') '[ , ][ ] ( )( ') '

when h is verysmall.

y x h yXY XY

xx h

XXx

f x y dx dy h f x y dyP Y y x X x hP x X x h hf xf x dx

.)(

')',()|(

xf

dyyxfxyF

X

y

XYY

.)(

),()|()|(xf

yxfxyFdydxyf

X

XYYY

)()(),( yfxfyxf YXXY )()|( yfxyf YY )()|( yFxyF YY



Example 5.32 Suppose the joint pdf of X and Y is given by

find and .

Solution

35

,0

02),(

otherwisexyee

yxfyx

XY

( | )Xf x y ( | )Yf y x

'0

' 2

( ) ( , ') ' 2 ' 2 (1 ), 0

( ) ( ', ) ' 2 ' 2 , 0 ,

xx y x xX XY

x y yY XY

f x f x y dy e e dy e e x

f y f x y dx e e dx e y

ｙ

.01)1(2

2)(

),()|(

,02

2)(

),()|( )(2

xyfore

eee

eexf

yxfxyf

xyforee

eeyf

yxfyxf

x

y

xx

yx

X

XYY

yxy

yx

Y

XYX



Example X = input voltage, Y = output = input + noise voltage. Here, X=1 or -1 with equal probabilities, the noise is uniformly

distributed from -2 to 2 with equal probabilities. Whe X=1, Y becomes uniformly distributed in [-1,3] so

The conditional cdf of Y:

For example,

36

1/ 4 1 3( | 1) .

0 otherwiseYy

f y X

.

3,1

31,4

1'41

1,0

')1|'(]1|[)1|(

1

y

yydy

y

dyXyfXyYPXyF

y

y

YY

0

1

1[ 0 | 1] ( ' | 1) ' (0 | 1) ,4

3 2 1[1 2 | 1] (2 | 1) (1| 1) ,4 4 4

1 1 1[ 0, 1] [ 0 | 1] [ 1]4 2 8

Y Y

Y Y

P Y X f y X dy F X

P Y X F X F X

P Y X P Y X P X

11

134

330

)1|(

y

yyy

XyFY



Conditional Expectation (1) The conditional expectation of Y given X=x is given by

When X and Y are both discrete random variables

On the other hand, E[Y|x] can be viewed as a function of x:

Correspondingly, this gives rise to the random variable:

37

.)|(]|[ dyxyyfxYE Y

[ | ] ( | )j

k j Y j ky

E Y x y p y x

].|[)( xYExg

].|[)( XYEXg



Conditional Expectation(2) What is

Note that

Suppose X and Y are jointly continuous random variables

Generalization [in the above proof, change y to h(y)]; and in particular,

38

?]]|[[ XYEE

[ | ] ( ) , is continuous.[ [ | ]]

[ | ] ( ), is discrete.k

X

k X kx

E Y x f x dx XE E Y X

E Y x p x X

].[)(

),(

)()|(

)(]|[]]|[[

YEdyyyf

dxdyyxfy

dxxfdyxyyf

dxxfxYEXYEE

Y

XY

XY

X

]]|)(([)]([ XYhEEYhE ]].|[[][ XYEEYE kk



Example 5.37 Binary Communication System The input X to a communication channel assumes the values +1

or -1 with probability 1/3 and 2/3. The output Y of the channel is given by Y=X+N, where N is zero-mean, unit variance Gaussian RV.

Find the mean of the output Y.

Solution Since Y is Gaussian RV with mean +1 when X=+1, and -1 when X=-1,

the conditional expected value of Y given X are

Since ,

39

| 1 1 and | 1 1E Y X E Y X

|E Y E E Y X

0

|

= | 1 1/ 3 1 2 / 3 1/ 3

k

k X kx

k

E Y E Y x p x

E Y X k P X k



40

Multiple Random Variables Joint cdf of

Joint pmf of

Marginal pmf's

Joint cdf of

Marginal pdf's

1 2, , , nX X X

1 2, , , nX X X

1 2, , , 1 2 1 1( , , , ) [ , , ]nX X X n n nF x x x P X x X x

1 2, , , 1 2 1 1( , , , ) [ , , ]nX X X n n np x x x P X x X x

1 2

1 1 1

, , , 1 2( ) [ ] ( , , , )j n

j j n

X j j j X X X nx x x x

p x P X x p x x x

1 2, , , nX X X1

1 2 1 1

' ' ' ', , , 1 2 , 1( , , , ) ( , )

n

n n

xx

X X X n X X n nF x x x f x x dx dx

1 2 1 1

' ', , , , 1 2 1 , , 1 1( , , , ) ( , , , )

n nX X X n X X n n nf x x x f x x x dx

C HAPTER 5 Pairs of Random Variables

Documents

Transcript of C HAPTER 5 Pairs of Random Variables