Chapter 5: JOINT PROBABILITYDISTRIBUTIONS

Part 2: Covariance and CorrelationSection 5-2

Consider the joint probability distribution fXY (x, y).

Is there a relationship between X and Y ? If so,what kind?

If you’re given information on X , does it giveyou information on the distribution of Y ? (Thinkof a conditional distribution). Or are they inde-pendent?

1

Below is a different joint probability distribu-tion for X and Y .

Does there seem to be a relationship between Xand Y ? Are they independent?

If you’re given information on X , does it giveyou information on the distribution of Y ?

How would you describe the relationship?

Is it stronger than the relationship on the pre-vious page? Do you know MORE about Y fora given X?

2

Below is a joint probability distribution for anindependent X and Y .

↑This picture is the give-awaythat they’re independent.

Does there seem to be a relationship between Xand Y ?

If you’re given information on X , does it giveyou information on the distribution of Y ?

3

Covariance

When two random variables are being consid-ered simultaneously, it is useful to describe howthey relate to each other, or how they vary to-gether.

A common measure of the relationship betweentwo random variables is the covariance.

•CovarianceThe covariance between the random variablesX and Y , denoted as cov(X, Y ), or σXY , is

σXY = E[(X − E(X))(Y − E(Y ))]

= E[(X − µX)(Y − µY )]

= E(XY )− E(X)E(Y )

= E(XY )− µXµY4

To calculate covariance, we need to find the ex-pected value of a function of X and Y . Thisis done similarly to how it was done in the uni-variate case...

For X, Y discrete,E[h(x, y)] =

∑x∑y h(x, y)fXY (x, y)

For X, Y continuous,E[h(x, y)] =

∫ ∫h(x, y)fXY (x, y)dxdy

—————————————————————

Covariance (i.e. σXY ) is an expected value ofa function of X and Y over the (X, Y ) space,if X and Y are continuous we can write

σXY =

∫ ∞−∞

∫ ∞−∞

(x−µX)(y−µY )fXY (x, y) dx dy

To compute covariance, you’ll probably use...

σXY = E(XY )− E(X)E(Y )

5

When does the covariance have a positive value?

In the integration we’re conceptually putting‘weight’ on values of (x− µX)(y − µY ).

What regions of (X, Y ) space has...(x− µX)(y − µY ) > 0?

• Both X and Y are above their means.

• Both X and Y are below their means.

• ⇒ Values along a line of positive slope.

A distribution that puts high probability on theseregions will have a positive covariance.

6

When does the covariance have a negativevalue?

In the integration we’re conceptually putting‘weight’ on values of (x− µX)(y − µY ).

What regions of (X, Y ) space has...(x− µX)(y − µY ) < 0?

•X is above its mean, and Y is below its mean.

• Y is above its mean, andX is below its mean.

• ⇒ Values along a line of negative slope.

A distribution that puts high probability on theseregions will have a negative covariance.

7

Covariance is a measure of the linear relationshipbetween X and Y .

If there is a non-linear relationship between Xand Y (such as a quadratic relationship), thecovariance may not be sensitive to this.

—————————————————————

When does the covariance have a zero value?

This can happen in a number of situations, butthere’s one situation that is of large interest...when X and Y are independent...

When X and Y are independent, σXY = 0.

8

If X and Y are independent, then...

σXY =

∫ ∞−∞

∫ ∞−∞

(x− µX)(y − µY )fXY (x,y) dx dy

=

∫ ∞−∞

∫ ∞−∞

(x− µX)(y − µY )fX(x)fY (y) dx dy

=

(∫ ∞−∞

(x− µX)fX(x)dx

)·(∫ ∞−∞

(y − µY )fY (y)dy

)=

(∫ ∞−∞

xfX(x)dx− µX)·(∫ ∞−∞

yfY (y)dy − µY)

= (E(X)− µX) · (E(Y )− µY )

= (µX − µX) · (µY − µY )

= 0

This does NOT mean... If covariance=0, thenX and Y are independent.

We can find cases to the contrary of the abovestatement, like when there is a strong quadraticrelationship between X and Y (so they’re notindependent), but you can still get σXY = 0.

Remember that covariance specifically looks fora linear relationship.

9

When X and Y are independent, σXY = 0.

For this distribution showing independence,there is equal weight along the positive and neg-ative diagonals.

10

A couple comments...

• You can also define covariance for discrete Xand Y :

σXY = E[(X − µX)(Y − µY )]

=∑x∑y(x− µX)(y − µY )fXY (x, y)

• And recall that you can get the expectedvalue of any function of X and Y :

E[h(X, Y )] =∫∞−∞

∫∞−∞ h(x, y)fXY (x, y) dx dy

or

E[h(X, Y )] =∑x∑y h(x, y)fXY (x, y)

11

Correlation

Covariance is a measure of the linear relationshipbetween two variables, but perhaps a more com-mon and more easily interpretable measure iscorrelation.

•CorrelationThe correlation (or correlation coefficient) be-tween random variables X and Y , denotedas ρXY , is

ρXY =cov(X, Y )√V (X)V (Y )

=σXYσXσY

.

Notice that the numerator is the covariance,but it’s now been scaled according to thestandard deviation of X and Y (which areboth > 0), we’re just scaling the covariance.

NOTE: Covariance and correlation will havethe same sign (positive or negative).

12

Correlation lies in [−1, 1], in other words,

−1 ≤ ρXY ≤ +1

Correlation is a unitless (or dimensionless)quantity.

Correlation...

• −1 ≤ ρXY ≤ +1

• If X and Y have a strong positive linear re-lation ρXY is near +1.

• If X and Y have a strong negative linearrelation ρXY is near −1.

• If X and Y have a non-zero correlation, theyare said to be correlated.

• Correlation is a measure of linear relation-ship.

• If X and Y are independent, ρXY = 0.

13

• Example: Recall the particle movement model

An article describes a model for the move-ment of a particle. Assume that a particlemoves within the region A bounded by the xaxis, the line x = 1, and the line y = x. Let(X, Y ) denote the position of the particle ata given time. The joint density of X and Yis given by

fXY (x, y) = 8xy for (x, y) ∈ A

a) Find cov(X, Y )

ANS: Earlier, we found E(X) = 45 ...

14

15

• Example: Book problem 5-43 p. 179.

The joint probability distribution is

x -1 0 0 1y 0 -1 1 0fXY 0.25 0.25 0.25 0.25

Show that the correlation between X and Yis zero, but X and Y are not independent.

16

17