Topics in MMSE Estimation for Sparse Approximation

Topics in MMSE Estimation for Sparse Approximation

Michael Elad The Computer Science Department The Technion – Israel Institute of technology Haifa 32000, Israel

*

Joint work with

Irad Yavneh Matan Protter Javier Turek

The CS Department, The Technion

Topics in MMSE Estimation For Sparse ApproximationBy: Michael Elad

2

Part I - Motivation Denoising

By Averaging Several Sparse Representations

3

Sparse Representation Denoising

K

N

DA fixed Dictionary

α

x

22

200 y.t.smin

D

Sparse representation modeling:

Assume that we get a noisy measurement vector

Our goal – recovery of x (or α).

The common practice – Approximate the solution of

y x v v

where is AWGN

D


4

Orthogonal Matching Pursuit

Ki1forrdzmin)i(ECompute 1ni

z

0

00

0

Sand

yyr

0,0n

D

2

nr

1nn

Initialization

Main Iteration

1.

2.

3.

4.

5.

)i(E)i(E,Ki1.t.siChoose 00

nn Spsup.t.symin:LS

Dnn yr:sidualReUpdate D

}i{SS:SUpdate 01nnn

StopYesNo

OMP finds one atom at a time for approximating the solution of

22

200 y.t.smin

D


5

Using several Representations

Consider the denoising problem

and suppose that we can find a group of J candidate solutions

such that

22

200 y.t.smin

D

J1jj

22

2j

0

0j

y

Nj

D

Basic Questions: What could we do with such

a set of competing solutions in order to better denoise y?

Why should this help?

How shall we practically find such a set of solutions?

Relevant work: [Leung & Barron (’06)] [Larsson & Selen (’07)] [Schintter et. al. (`08)] [Elad and Yavneh (’08)] [Giraud (‘08)] [Protter et. al. (‘10)] …


6

Generating Many Representations

Our Answer: Randomizing the OMP

Ki1forrdzmin)i(ECompute 1ni

z

0

00

0

Sand

yyr

0,0n

D

2

nr

1nn

Initialization

Main Iteration

1.

2.

3.

4.

5.

)i(E)i(E,Ki1.t.siChoose 00

nn Spsup.t.symin:LS

Dnn yr:sidualReUpdate D

}i{SS:SUpdate 01nnn

Stop

)i(EcexpyprobabilitwithiChoose 0

YesNo

* Larsson and Schnitter propose a more complicated and deterministic tree pruning method

*

For now, lets set the parameter c manually for best performance. Later we shall

define a way to set it automatically


7

Lets Try

0

0 010

y

Proposed Experiment :

Form a random D.

Multiply by a sparse vector α0 ( ).

Add Gaussian iid noise (σ=1) and obtain .

Solve the problem

using OMP, and obtain .

Use RandOMP and obtain .

Lets look at the obtained representations …

100

200

D

1000RandOMPj

j 1

0

20

0 2min s.t. y 100

D

+=

v y

OMP


8

Some Observations

0 10 20 30 400

50

100

150

Candinality

His

togra

m

Random-OMP cardinalitiesOMP cardinality

85 90 95 100 1050

50

100

150

200

250

300

350

Representation ErrorH

isto

gra

m

Random-OMP errorOMP error

0 0.1 0.2 0.3 0.40

50

100

150

200

250

300

Noise Attenuation

His

togr

am

Random-OMP denoisingOMP denoising

0 5 10 15 200.05

0.1

0.15

0.2

0.25

0.3

0.35

Cardinality

No

ise

Att

enu

ation

Random-OMP denoisingOMP denoising

0

0

2

0 22

0 2

ˆ

y

D D

D

2

2y D

We see that

•The OMP gives the sparsest solution

•Nevertheless, it is not the most effective for denoising.

•The cardinality of a representation does not reveal its efficiency.


9

The Surprise … (to some of us)

0 50 100 150 200-3

-2

-1

0

1

2

3

index

valu

e

Averaged Rep.Original Rep.OMP Rep.

1000RandOMPj

j 1

1ˆ

1000

Lets propose the average

as our representation

This representation IS NOT SPARSE AT ALL but its noise attenuation is: 0.06 (OMP gives 0.16)


10

0 0.1 0.2 0.3 0.4 0.50

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

0.5

OMP Denoising Factor

Ra

ndO

MP

Den

oisi

ng F

acto

r

OMP versus RandOMP resultsMean Point

Cases of zero solution, where

Repeat this Experiment …•Dictionary (random) of

size N=100, K=200

•True support of α is 10

• σx=1 and ε=10

•We run OMP for denoising.

•We run RandOMP J=1000 times and average

•Denoising is assessed by

J

1j

RandOMPjJ

1ˆ

2

20

220

y

ˆ

D

DD


2

2y 100

11

Part II - Explanation It is

Time to be More Precise


12

Our Signal Model

K

N

DA fixed Dictionary

α

x D is fixed and known.

Assume that α is built by: Choosing the support s with

probability P(s) from all the 2K possibilities Ω.

Lets assume that P(iS)=Pi

are drawn independently.

Choosing the αs coefficients using iid Gaussian entries .

The ideal signal is x=Dα=Dsαs.

The p.d.f. P(α) and P(x) are clear and known


2xN 0,

13

Adding Noise

K

N

DA fixed Dictionary

α

x

yv

+Noise Assumed:

The noise v is additive white Gaussian vector with probability Pv(v)

The conditional p.d.f.’s P(y|), P(|y), and even P(y|s), P(s|y), are all clear and well-

defined (although they may appear nasty).

2

2

2

yxexpCxyP


14

The Key – The Posterior P(|y)

P | yWe have access to

MAP MMSE

MAP ArgMax P( | y)ˆ MMSE E | y

The estimation of x is done by estimating α and multiplication by D.

These two estimators are impossible to compute, as we show next.

Oracle known

support s

oracle


*

* Actually, there is a delicate problem with this definition, due to the unavoidable mixture of continuous and discrete PDF’s. The solution is to estimate the MAP’s support S.

15

Lets Start with The Oracle

yP

P|yPy|Ps,y|P ss

s

2x

2s

s2

expP

2

2ss

s2

yexp|yP

D

2x

2s

2

2ss

s22

yexpy|P

D

s1

sTs2

1

2x

sTs2s hy

111ˆ

QDIDD

* When s is known

*

Comments:

• This estimate is both the MAP and MMSE.

• The oracle estimate of x is obtained by multiplication by Ds.


s

s s s

T 1|s| s ss s

x

P(y| s) P(y| s, )P( )d

....

h h log(det( ))exp

2 2

Q Q

Based on our prior for generating the support

i ji s j s

P s P 1 P

16

The MAP Estimation


MAP

s s

P(y| s)P(s)s ArgMax P(s| y) ArgMax

P(y)

T 1

MAP s ss s

s

ij

i s j sx

h h log(det( ))s ArgMax exp

2 2P

1 P

Q Q

17

The MAP Estimation

Implications:

The MAP estimator requires to test all the possible supports for the maximization. For the found support, the oracle formula is used.

In typical problems, this process is impossible as there is a combinatorial set of possibilities.

This is why we rarely use exact MAP, and we typically replace it with approximation algorithms (e.g., OMP).


ij

T 1s ss s

MAP

s

i s j sx

h h log(det( ))2 2

s ArgMax

log lP

o 1 Pg

Q Q

19

The MMSE Estimation

s

MMSE s,y|E)y|s(Py|Eˆ

s

sMMSE ˆ)y|s(Pˆ

Implications:

The best estimator (in terms of L2 error) is a weighted average of many sparse representations!!!

As in the MAP case, in typical problems one cannot compute this expression, as the summation is over a combinatorial set of possibilities. We should propose approximations here as well.


20

This is our c in

the Random-

OMP

The Case of |s|=1 and Pi=P

T 1s ss s

2T 2x

i2 2 2

ij

i j sx

x

s

h h log(det( ))P(s| y) exp

2 2P

1

1exp (y d)

2

P

Q Q

The i-th atom in D Based on this we can propose a greedy

algorithm for both MAP and MMSE:

MAP – choose the atom with the largest inner product (out of K), and do so one at a time, while freezing the previous ones (almost OMP).

MMSE – draw at random an atom in a greedy algorithm, based on the above probability set, getting close to P(s|y) in the overall draw (almost RandOMP).


21

Comparative Results

The following results correspond to a small dictionary (10×16), where the combinatorial formulas can be evaluated as well.Parameters:

• N,K: 10×16

• P=0.1 (varying cardinality)

• σx=1

• J=50 (RandOMP)

• Averaged over 1000 experiments


0.60.2 0.4 0.8 10

0.2

0.4

0.6

0.8

1

Rel

ativ

e R

epre

sent

atio

n M

ean-

Squ

ared

-Err

or

OracleMMSEMAPOMPRand-OMP

22

Part III – Diving In A Closer

Look At the Unitary Case IDDDD TT


23

Few Basic Observations

2 2T x

s s s2 2 2 2x x

Ts s2 2 s

2oracle 1 2x

ss s 2 2 s sx

1 1

1 1h y

h cˆ

Q D D I I

D

Q

yTDLet us denote

(The Oracle)


s

2 2x

log(det( )) 1S log

2 1 c

Q

T 1 2 2s ss

2 s 2

h h c2

Q

24

Back to the MAP Estimation


T 1s ss s i

ji s j sx

h h log(det( ) P1

)P S| y exp

2 2P

Q Q

ii s i si

222 ii2

Pq

11 cc

PP S| y exp

25

The MAP Estimator


ii s

qP S| y

222 i

i i2i

P 1 ccq exp

1 P

is obtained by maximizing the

expression

MAPS

Thus, every i such that qi>1 should be in the support, which leads to 22 2

i i 2MAP 2i i

i2c log

cˆ P 1 c

0 Otherwise

1 P

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

iMA

P

P=0.1=0.3x=1

ig

26

The MMSE Estimation


Some algebra

MMSE 2ii i

i

qcˆ

1 q

222 i

i i2i

P 1 ccq exp

1 P

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

iMM

SE

P=0.1=0.3x=1

and we get that

This result leads to a dense representation vector. The curve is a

smoothed version of the MAP one.

27

What About the Error ?


n2oracle 1 2 2

s i2 i 1

E trace ... c gˆ

Q

2 2MMSE MMSE oracle1ss

2 2s

n n2 2 4 2 2

i i i ii 1 i 1

E P s| y traceˆ ˆ ˆ

... c g c g g

Q

2 2MAP MAP MMSE MMSE

2 2

n n2 2 4 2 MAP

i i i i ii 1 i 1

E Eˆ ˆ ˆ ˆ

... c g c g I (1 2g)

ii

i

qg

1 q

28Topics in MMSE Estimation For Sparse ApproximationBy: Michael Elad

A Synthetic Experiment

0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 20

0.05

0.1

0.15

0.2

0.25

The following results correspond to a dictionary of size (100×100)Parameters:

• n,K: 100×100

• P=0.1

• σx=1

• Averaged over 1000 experiments

The average errors are shown relative to n2

Empirical OracleTheoretical OracleEmpirical MMSETheoretical MMSEEmpirical MAPTheoretical MAP

Relative Mean-Squared-Error

29

Part IV - Theory Estimation Errors


30

Useful Lemma


Then .

Let (ak,bk) k=1,2, … ,n be pairs of positive real numbers. Let m be the index of a pair such that

k m

k m

a ak .

b b

n

kk 1 mn

mk

k 1

aabb

Equality is obtained only if all the ratios ak/bk are equal.

n n2MMSE 2 2 4 2 2

i i i i2 i 1 i 1

E c g c g gˆ

n n2MAP 2 2 4 2 MAP

i i i i i2 i 1 i 1

E c g c g I (1 2g)ˆ

n2oracle 2 2i

2 i 1

E c gˆ

We are interested in this result because :

This leads to …

31

Theorem 1 – MMSE Error


2k

kk

P 1 cG

1 P

22MMSE

m2 m

2oracle

2m

1 e1 2log GE ˆ 4G 4

2E ˆ 1 OtherwiseG e

Define . Choose m such that .

m kk, G G

this error ratio bound becomes

kP P 1K

2MMSE

2

2oracle

2

E ˆConst logK

E ˆ

32

Theorem 2 – MAP Error


2k

kk

P 1 cG

1 P

2 1MAPm

2 m

2oracle

2m

11 2log G eE ˆ G

2E ˆ 1 OtherwiseG e

Define . Choose m such that .

m kk, G G

this error ratio bound becomes

kP P 1K

2MMSE

2

2oracle

2

E ˆConst logK

E ˆ

33

The Bounds’ Factors vs. P


10-4

10-3

10-2

10-1

100

0

5

10

15

20

25

P

Bo

und

Fa

cto

r

1m

m

m

11 2log G e

G

21 Otherwise

G e

2

mm

m

1 e1 2log G

4G 4

21 Otherwise

G e

Parameters:

• P=[0,1]

• σx=1

=0.3

Notice that the tendency of

the two estimators to align for P0

is not reflected in these bounds.

34

Part V – We Are Done Summary and Conclusions


35

Today We Have Seen that …

By finding the sparsest

representation and using it to recover the clean signal

How ?

Sparsity and Redundancy are

used for denoising of

signals/images

Can we do better?

Yes! Averaging

several rep’s lead to better denoising, as

it approximates

the MMSEMore on these (including the slides and the relevant papers) can be found in http://www.cs.technion.ac.il/~elad

Unitary case?

MAP and MMSE enjoy a closed-form, exact and cheap formulae.

Their error is bounded and tightly

related to the oracle’s error


Topics in MMSE Estimation for Sparse Approximation

Documents

Transcript of Topics in MMSE Estimation for Sparse Approximation