III. Multi-Dimensional Random Variables and Application in Vector Quantization.

III. Multi-Dimensional Random Variables and Application in Vector Quantization


Karhunen-Loeve Decomposition

v1

u2 X=[X1, X2]

X = [X1, X2] is a sample vector from a two-dimensional random variable

defined over v1, v2 (unit vectors) co-ordinate system

Let u1, u2 (unit vectors) be a new co-ordinate system such that u1, u2 are the

eigen vectors of the covariance matrix RX

u1

v2

u1, u2 are orthogonal to each otherT T T T

X 1 1 1 2 X 1 2 1 1

T TT T T T TX 2 2 2 X 2 2 2 2 X 2 2

T T TX X 2 X 1 2 2 1

T T2 1 1 2 2 1

R u λ u u R u u λ u

R u λ u R u λ u u R u λ

R R u R u u λ u

u λ u u λ u

T2 1u u 0

For distinct eigen vectors, values



v1

u2 X=[X1, X2]





u1

T1 1u X u

T2 2u X u

v2

Representation of X over u1, u2

T T1 1 2 2

2 2T T T T

i i i ii 1 i 1

X u X u u X u

X u X u X u Xu

Projection of X over u1

unit vector u1

Projection of X over u2

unit vector u2



v1

u2 X=[X1, X2]





u1

T1 1u X u

T2 2u X u

v2

Diagonalization of RX

2T T T

X X i ii 1

2T T T

X X i ii 1

T TX i i i

2T T T

X i i ii 1

R X R u Xu

R X R u Xu

R u λ u

R X λ u Xu



v1

u2 X=[X1, X2]





u1

T1 1u X u

T2 2u X u

v2

T Ti i

2T T T

X i i ii 1

2T T T

X i i ii 1

T TX 1 1 1 2 2 2

Xu u X

R X λ u u X

R X λ u u X

R λ u u λ u u




v1

u2 X=[X1, X2]





u1

T1 1u X u

T2 2u X u

v2

T TX 1 1 1 2 2 2

1 1T TX 1 2

2 2

TX

R u λ u u λ u

λ 0 uR u u

0 λ u

R U ΛU

1 11 12 2 21 22u u u , u u u

1 11 12

2 21 22

u u uU

u u u

1

2

λ 0Λ

0 λ



K-L Transformation of 2-D Random Variable

v1

u2X



Let Y = [Y1, Y2] be the transformation of the random variable X over the co-

ordinate system u1, u2

u1

v2

X1

X2 Y

Y1

Y2



v1

u2X





u1

v2

X1

X2 Y

Y1

Y2 1 2

'x 1 2 x xX X m X X m m

T1 1

T2 2

T T T1 2 1 2

Y Xu

Y Xu

Y Y Y X u u XU

1 2

'Y 1 2 Y YY Y m Y Y m m

Define

Define E[X1] E[X2]

E[Y1] E[Y2]



v1

u2X





u1

v2

X1

X2 Y

Y1

Y2

'T 'X

'' '1

X 1 2'2

' ' ' '1 1 1 2

X ' ' ' '2 1 2 2

' ' ' '1 1 1 2

X ' ' ' '2 1 2 2

R E X X

XR E X X

X

X X X XR E

X X X X

E X X E X XR

E X X E X X

Covariance Matrix RX



v1

u2X





u1

v2

X1

X2 Y

Y1

Y2

'T 'Y

Y 1 2

T TY 1 2

T TY 1 2

T TY X 1 2

TY X

R E Y Y

m E Y E Y

m E Xu E Xu

m E X u E X u

m m u u

m m U

Covariance Matrix RY



v1

u2X





u1

v2

X1

X2 Y

Y1

Y2

' T TY X

' ' T

Y Y m XU m U

Y X U

'

'

'T 'Y

T' T ' TY

T ' TY

T ' T TY X

R E Y Y

R E X U X U

R E UX X U

R UE X X U UR U




v1

u2X





u1

v2

X1

X2 Y

Y1

Y2

TY X

TX

T TY

T

1Y

2

R UR U

R U ΛU

R U U ΛU U

UU I

λ 0R Λ

0 λ

Y1 and Y2 are UNCORRELATED




Therefore using principle component analysis, it is possible to transform a random variable X with components X1 and X2 that are correlated into another random variable Y whose components Y1 and Y2 are uncorrelated

v1

u2X

u1

v2

X1

X2 Y

Y1

Y2

TY X

Y

m m U

R Λ

u1, u2 are eigen vectors of RX

λ1, λ2 are eigen values of RX

1

2

uU

u

1

2

λ 0Λ

0 λ


Two-Dimensional Gaussian Distribution

In the previous slides, we have talked about the transformed random variable Y whose components are uncorrelated and have mean mY

and covariance matrix RY. What would be the distribution of Y. Well this depends on what is the distribution of X

Generally X and Y do not follow a Gaussian distribution. However, if X is a two-dimensional Gaussian distribution then Y as well would be a two-dimensional Gaussian distribution



Suppose we have a two-dimensional random variable Y whose components Y1 and Y2 are independent and Gaussian

1 2 1 2Y Y ,Y 1 2 Y 1 Y 2f y f y , y f y f y

2

i yi2i

y m

2σYi i 2

i

1f y e

2πσ

-3 -2 -1 0 1 2 30

0.02

0.04

0.06

0.08

0.1

0.12

0.14

0.16

Y1

f Y1(y

1)

mY1

=0,12=1




1 2 1 2Y Y ,Y 1 2 Y 1 Y 2f y f y , y f y f y

2

i yi2i

y m

2σYi i 2

i

1f y e

2πσ

2

i yi2i

y m

2σ2Y i 1 2

i

1f y e

2πσ

2 2

1 y 2 y1 22 21 2

y m y m

2σ 2σY 2 2

1 2

1 1f y e e

2πσ 2πσ

-4-2

02

4

-4-2

02

40

0.05

0.1

0.15

0.2

Y1

mY1

=0, mY2

=0, 12=1,

22=1

Y2

f y(y)




1 2 1 2Y Y ,Y 1 2 Y 1 Y 2f y f y , y f y f y

2

i yi2i

y m

2σYi i 2

i

1f y e

2πσ

2

i yi2i

y m

2σ2Y i 1 2

i

1f y e

2πσ

2 2

1 y 2 y1 22 21 2

y m y m

2σ 2σY 2 2

1 2

1 1f y e e

2πσ 2πσ

mY1

=0, mY2

=0, 12=1,

22=1

Y1

Y2

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3




2 2

2 y 2 y2 22 22 2

y m y m1

2 σ σ

Y 2 21 2

1f y e

2π σ σ

This formula is valid for any multi-dimensional Gaussian random variable whether its components are correlated or not

2 1

2 21 2Y 1 2 Y2

2

σ 0R σ σ det R

0 σ

2 ' '2 '21' 1 'T ' ' 1 1 2

Y 1 2 2 2'1 22

22

10

σ Y Y YY R Y Y Y

1 σ σY0

σ

' 1 'TY

1Y R Y

2Y 1

2Y

1f y e

2π det R



-4-2

02

4

-4

-2

0

2

40

0.05

0.1

0.15

0.2

Y1

mY1

=0, mY2

=0, 12=1,

22=1,

Y2

f Y(y

)

-4-2

02

4

-4

-2

0

2

40

0.05

0.1

0.15

0.2

Y1

mY1

=0, mY2

=0, 12=1,

22=1,

Y2

f Y(y

)

-4-2

02

4

-4

-2

0

2

40

0.05

0.1

0.15

0.2

0.25

Y1

mY1

=0, mY2

=0, 12=1,

22=1,

Y2

f Y(y

)

-4-2

02

4

-4

-2

0

2

40

0.5

1

1.5

Y1

mY1

=0, mY2

=0, 12=1,

22=1,

Y2

f Y(y

)

Y

1 0.25R

0.25 1

Y

1 0.5R

0.5 1

Y

1 0.75R

0.75 1

Y

1 0.99R

0.99 1



Y1

Y2

mY1

=0, mY2

=0, 12=1,

22=1,

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Y1

Y2

mY1

=0, mY2

=0, 12=1,

22=1,

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Y1

Y2

mY1

=0, mY2

=0, 12=1,

22=1,

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Y1

Y2

mY1

=0, mY2

=0, 12=1,

22=1,

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Y

1 0.25R

0.25 1

Y

1 0.5R

0.5 1

Y

1 0.75R

0.75 1

Y

1 0.99R

0.99 1



-4-2

02

4

-4

-2

0

2

40

0.05

0.1

0.15

0.2

Y1

mY1

=0, mY2

=0, 12=1,

22=1,

Y2

f Y(y

)

-4-2

02

4

-4

-2

0

2

40

0.05

0.1

0.15

0.2

Y1

mY1

=0, mY2

=0, 12=1,

22=1,

Y2

f Y(y

)

-4-2

02

4

-4

-2

0

2

40

0.05

0.1

0.15

0.2

0.25

Y1

mY1

=0, mY2

=0, 12=1,

22=1,

Y2

f Y(y

)

-4-2

02

4

-4

-2

0

2

40

0.5

1

1.5

Y1

mY1

=0, mY2

=0, 12=1,

22=1,

Y2

f Y(y

)

Y

1 0.25R

0.25 1

Y

1 0.5R

0.5 1

Y

1 0.75R

0.75 1

Y

1 0.99R

0.99 1



Y1

Y2

mY1

=0, mY2

=0, 12=1,

22=1,

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Y1

Y2

mY1

=0, mY2

=0, 12=1,

22=1,

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Y1

Y2

mY1

=0, mY2

=0, 12=1,

22=1,

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Y1

Y2

mY1

=0, mY2

=0, 12=1,

22=1,

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Y

1 0.25R

0.25 1

Y

1 0.5R

0.5 1

Y

1 0.75R

0.75 1

Y

1 0.99R

0.99 1



Y1

Y2

mY1

=0, mY2

=0, 12=1.5,

22=0.5,

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Y

1.5 0R

0 0.5



Y1

Y2

mY1

=0, mY2

=0, 12=1.5,

22=0.5,

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Y

1.5 0.25R

0.25 0.5



Y1

Y2

mY1

=0, mY2

=0, 12=1.5,

22=0.5,

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Y

1.5 0.5R

0.5 0.5



Y1

Y2

mY1

=0, mY2

=0, 12=1.5,

22=0.5,

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Y

1.5 0.75R

0.75 0.5



Y1

Y2

mY1

=0, mY2

=0, 12=1.9,

22=0.1,

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Y

1.9 0R

0 0.1



Y1

Y2

mY1

=0, mY2

=0, 12=1.9,

22=0.1,

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Y

1.9 0.25R

0.25 0.1



Y

1 0.5R

0.5 1



Y1

Y2

mY1

=0, mY2

=0, 12=1,

22=1,

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3Y

1 0.5R

0.5 1

More Energy

More Energy

Less Energy

Less Energy

Y1 and Y2 Correlated



Y1

Y2

mY1

=0, mY2

=0, 12=1,

22=1,

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3Y

1 0.5R

0.5 1

Equal

Energy

Equal Energy

Equal Energy

Equal Energy

By rotating the axis, the resultant transformed random variables are uncorrelated

III. Multi-Dimensional Random Variables and Application in Vector Quantization.

Documents

Transcript of III. Multi-Dimensional Random Variables and Application in Vector Quantization.