Mmse Estimation

7/24/2019 Mmse Estimation

1/12

10 - 1 MMSE Estimation S. Lall, Stanford 2011.02.02.01

10 - MMSE Estimation

Estimation given a pdf

Minimizing the mean square error

The minimum mean square error (MMSE) estimator

The MMSE and the mean-variance decomposition

Example: uniform pdf on the triangle

Example: uniform pdf on an L-shaped region

Example: Gaussian

Posterior covariance

Bias

Estimating a linear function of the unknown MMSE and MAP estimation


Estimation given a PDF

Suppose x: Rn is a random variable with pdfpx.

One can predictorestimatethe outcome as follows

Givencost function c: Rn Rn R

pickestimatexto minimize E c(x,x)

We will look at the cost function

c(x,x) = kx xk2

Then themean square error(MSE) is

Ekx xk2

=

Z kx xk2px(x) dx


2/12


Minimizing the MSE

Lets find the minimum mean-square error (MMSE) estimate ofx; we need to solve

minx

Ekx xk2

We have

Ekx xk2

= E

(x x)T(x x)

= E

xTx 2xTx + xTx

= Ekxk2 2xTE x+ xTx

Differentiating with respect to xgives the optimal estimate

xmmse= Ex


The MMSE estimate

The minimum mean-square error estimate ofxis

xmmse= Ex

Its mean square error is

E

kx xmmsek

2

= trace cov(x)

since Ekx xmmsek

2

= Ekx Exk2


3/12


The mean-variance decomposition

We can interpret this via the MVD. For any random variable z, we have

Ekzk2

= E

kz E zk2

+ kE zk2

Applying this to z=x xgives

Ekx xk2

= E

kx E xk2

+ kx Exk2

The first term is thevarianceofx; it is the error wecannot remove

The second term is the biasofx; the best we can do is make this zero.

!2 !1 0 1 2!3

!2

!1

0

1

2

3


The estimation problem

Suppose x, y are random variables, with joint pdfp(x, y).

We measure y=ymeas.

We would like to find the MMSE estimate ofxgiveny = ymeas.

Theestimatoris a function : Rm Rn.

We measure y = ymeas, and estimate xby xest= (ymeas)

We would like to find the function which minimizes the cost function

J= Ek(y) xk2


4/12


Notation

Well use the following notation.

py is themarginalor inducedpdf ofy

py(y) = Z p(x, y) dx

p|y is the pdfconditionedon y

p|y(x, y) =p(x, y)

py(y)


The MMSE estimator

Themean-square-error conditioned on y is econd(y), given by

econd(y) =

Z k(y) xk2p|y(x, y) dx

Then the mean square error J is given by

J= E

econd(y)

because

J=

Z Z k(y) xk2p(x, y) dxdy

=

Z py(y) econd(y) dy


5/12


The MMSE estimator

We can write the MSE conditioned on y as

econd(ymeas) = Ek(y) xk2 | y=ymeas

For eachymeas, we can pick a value for (ymeas)

So we have an MMSE prediction problem for each ymeas


The MMSE estimator

Apply the MVD to z=(y) xconditioned on y=w to give

econd(w) = Ek(y) xk2 | y=w

= E

kx h(w)k2 | y= w

+ k(w) h(w)k2

whereh(w) is the mean ofxconditioned ony = w

h(w) = E(x | y=w)

To minimizeecond(w)we therefore pick

(w) =h(w)


6/12


The error of the MMSE estimator

With this choice of estimator

econd(w) = Ekx h(w)k2 | y = w

= trace cov(x | y = w)


Summary: the MMSE estimator

The MMSE estimator is

mmse(ymeas) = E(x | y = ymeas)

econd(ymeas) = trace cov(x | y = ymeas)

We often write

xmmse= mmse(ymeas) econd(ymeas) = Ekx xmmsek

2 y=ymeas

The estimate only depends on the conditional pdfofx | y = ymeas

The means and covariance are those of theconditional pdf

The above formulae give the MMSE estimate for any pdfonxand y


7/12

0 0.5 1

0

0.5

1


Example: MMSE estimation

(x, y)are uniformly distributed on the triangle A.i.e., the pdf is

p(x, y) =

(2 if(x, y) A

0 otherwise

the conditional distribution ofx | y = ymeasis uniform on[ymeas, 1]

the MMSE estimator ofx given y = ymeas is

xmmse= E(x | y = ymeas) =1 + ymeas

2

the conditional mean square error of this estimate is

Ekx xmmsek

2 y = ymeas= 1

12(ymeas 1)

2

0 0.5 1

0

0.5

1



The mean square error is

Ekmmse(y) xk

2

= E

econd(y)

=Z 1

x=0

Z xy=0p(x, y) econd(y) dy dx

=

Z 1

x=0

Z xy=0

1

6(y 1)2 dy dx

= 1

24


8/12

0 0.5 1

0

0.5

1



(x, y)are uniformly distributed on the L-shapedregionA, i.e., the pdf is

p(x, y) =

(4

3 if(x, y) A

0 otherwise

the MMSE estimator ofx given y = ymeas is

xmmse= E(x | y = ymeas) =

(1

2 if0 ymeas

1

2

3

4 if 1

2< ymeas 1

the mean square error of this estimate is

Ekx xmmsek

2 y=ymeas=

( 1

12 if0 ymeas

1

2

1

48 if 1

2< ymeas 1

0 0.5 1

0

0.5

1



The mean square error is

Ekmmse(y) xk

2

= E

econd(y)

=Z

Ap(x, y) econd(y) dy dx

=

Z 1

x=0

Z 12

y=0

1

12p(x, y) dy dx+

Z 1

x=12

Z 12

y=12

1

48p(x, y) dy dx

= 1

16


9/12

!3 !2 !1 0 1 2 30

0.05

0.1

0.15

0.2

0.25

0.3

0.35

!3 !2 !1 0 1 2 3!3

!2

!1

0

1

2

3


MMSE estimation for Gaussians

Suppose

xy

N(,)where

= xy = x xy

Txy y

We know that the conditional density ofx | y = ymeas isN(1, 1)where

1= x+ xy1y (ymeas y)

1= x xy1y

Txy


MMSE estimation for Gaussians

The MMSE estimator when x, y are jointly Gaussian is

mmse(ymeas) =x+ xy1y (ymeas y)

econd(ymeas) = tracex xy

1y

Txy

The conditional MSE econd(y)is independent ofy; a special property of Gaussians

Hence the optimum MSE achieved is

Ekmmse(y) xk

2

= tracex xy

1y

Txy

The estimate xmmse is an affine function ofymeas


10/12


Posterior covariance

Lets look at the error z= (y) x. We have

cov(z | y = ymeas) = cov

x| y= ymeas

We use this to find a confidence region for the estimate

cov(x| y=ymeas

is called theposterior covariance

!3 !2 !1 0 1 2 3

!3

!2

!1

0

1

2

3


Example: MMSE for Gaussians

Here

xy

N(0,)with =

2.8 2.42.4 3

We measureymeas= 2

and the MMSE estimator is

mmse(ymeas) = 12122

ymeas

= 0.8ymeas= 1.6

The posterior covariance is Q = x xy1y yx = 0.88

Let 2 =QF12

(0.9), then 1.54and the confidence interval is

Prob

|x 1.6|

y = ymeas

= 0.9


11/12


Bias

The MMSE estimator is unbiased; that is the mean erroris zero.

Emmse(y) x

= 0


Estimating a linear function of the unknown

Suppose

p(x, y) is a pdf on x, y

q= C xis a random variable

we measure y and would like to estimate q

The optimal estimator is

qmmse= CE

x| y=ymeas

Because E

q| y = ymeas

= CE

x| y = ymeas

The optimal estimate ofCx is Cmultiplied by the optimal estimate ofx

Only works forlinear functionsofx


12/12

!3 !2 !1 0 1 2 3!3

!2

!1

0

1

2

3


MMSE and MAP estimation

Suppose xand y are random variables, with joint pdff(x, y)

Themaximum a posterior (MAP) estimate is the xthat maximizes

h(x, ymeas) = conditional pdf ofx | (y=ymeas)

The MAP estimate also maximizes the joint pdf

xmap= arg maxx

f(x, ymeas)

Whenx, y are jointly Gaussian, then the peak of

the conditional pdf is the conditional mean.

i.e., for Gaussians, the MMSE estimate is equalto the MAP estimate.

Mmse Estimation

Documents

Transcript of Mmse Estimation