Going off the gridBenjamin Recht

Department of Computer SciencesUniversity of Wisconsin-Madison

Joint work withBadri BhaskarParikshit ShahGonnguo Tang

We live in a continuous world...

But we work with digital computers...

What is the price of living on the grid?

DOA Estimation

GPS Radar

Sampling

Ultrasound

!"# $%%% &'()*+, '* %-%).$*. +*/ 0%,%12%/ 2'3$10 $* 1$)1($20 +*/ 0402%-05 6',7 "5 *'7 85 0%32%-9%) "#:"

;7 *(0 $1 ?@ABC

DOA Estimation

GPS Radar

Sampling

Ultrasound

!"# $%%% &'()*+, '* %-%).$*. +*/ 0%,%12%/ 2'3$10 $* 1$)1($20 +*/ 0402%-05 6',7 "5 *'7 85 0%32%-9%) "#:"

;7 *(0 $1 ?@ABC

DOA Estimation

GPS Radar

Sampling

Ultrasound

!"# $%%% &'()*+, '* %-%).$*. +*/ 0%,%12%/ 2'3$10 $* 1$)1($20 +*/ 0402%-05 6',7 "5 *'7 85 0%32%-9%) "#:"

;7 *(0 $1 ?@ABC

DOA Estimation

GPS Radar

Sampling

Ultrasound

!"# $%%% &'()*+, '* %-%).$*. +*/ 0%,%12%/ 2'3$10 $* 1$)1($20 +*/ 0402%-05 6',7 "5 *'7 85 0%32%-9%) "#:"

;7 *(0 $1 ?@ABC

DOA Estimation

GPS Radar

Sampling

Ultrasound

!"# $%%% &'()*+, '* %-%).$*. +*/ 0%,%12%/ 2'3$10 $* 1$)1($20 +*/ 0402%-05 6',7 "5 *'7 85 0%32%-9%) "#:"

;7 *(0 $1 ?@ABC

for some

Observe a sparse combination of sinusoids

Spectrum Estimation: find a combination of sinusoids agreeing with time series data• Classic techniques require sensitive frequency localization or

knowledge of model order.

• Can we leverage convex geometry for new algorithms for and analysis of spectrum estimation?

xm =sX

k=1

ckei2⇡muk uk 2 [0, 1)

Sparse combinations from the set

A =

8>>>>><

>>>>>:

2

666664

ei✓

ei2⇡�+i✓

ei4⇡�+i✓

...ei2⇡�n+i✓

3

777775:

✓ 2 [0, 2⇡) ,� 2 [0, 1)

9>>>>>=

>>>>>;

Linear Inverse Problems• Find me a solution of

• Φ m x n, m

• 1-sparse vectors of Euclidean norm 1

• Convex hull is the unit ball of the l1 norm

1

1

-1

-1

Sparsity

minimize kxk1subject to �x = y

x1

x2

Φx=y

• Search for best linear combination of fewest atoms• “rank” = fewest atoms needed to describe the model

Parsimonious Models

atomsmodel weights

rank

Atomic Norm Minimization

• Generalizes existing, powerful methods• Rigorous formula for developing new analysis

algorithms• Tightest bounds on number of measurements

needed for model recovery in all common models• One algorithm prototype for many data-mining

applications

minimize kzkAsubject to �z = yIDEA:

Chandrasekaran, Recht, Parrilo, and Willsky 2010

Spectrum Estimation

u?k 2 [0, 1)for some

Observe a sparse combination of sinusoids

A =

8>>>><

>>>>:

2

66664

ei✓

ei2⇡�+i✓

ei4⇡�+i✓

· · ·ei2⇡�n+i✓

3

77775:

✓ 2 [0, 2⇡) ,� 2 [0, 1)

9>>>>=

>>>>;

Atomic Set

Observe: (signal plus noise)

Classical techniques (Prony, Matrix Pencil, MUSIC, ESPRIT, Cadzow), use the fact that noiseless moment matrices are low-rank:

rX

k=1

�k

2

664

1e�ki

e2�ki

e3�ki

3

775

2

664

1e�ki

e2�ki

e3�ki

3

775

⇤

=

2

664

µ0 µ1 µ2 µ3µ̄1 µ0 µ1 µ2µ̄2 µ̄1 µ0 µ1µ̄3 µ̄2 µ̄1 µ0

3

775 ⌫ 0

x

?m =

sX

k=1

c

?ke

i2⇡mu?k

y = x? + !

Atomic Norm for Spectrum Estimation

• How do we solve the optimization problem?• Can we approximate the true signal from partial and

noisy measurements?• Can we estimate the frequencies from partial and

noisy measurements?

IDEA:minimize kzkAsubject to k�z � yk �

Atomic Set:

A =

8>>>>><

>>>>>:

2

666664

ei✓

ei2⇡�+i✓

ei4⇡�+i✓

...ei2⇡�n+i✓

3

777775:

✓ 2 [0, 2⇡) ,� 2 [0, 1)

9>>>>>=

>>>>>;

• Convex hull is characterized by linear matrix inequalities (Toeplitz positive semidefinite)

• Moment Curve of [t,t2,t3,t4,...],

Which atomic norm for sinusoids?for somexm =

sX

k=1

ckei2⇡muk uk 2 [0, 1)

Assume are positive for simplicityck

rX

k=1

ck

2

664

1e

i2⇡uk

e

i4⇡uk

e

i6⇡uk

3

775

2

664

1e

i2⇡uk

e

i4⇡uk

e

i6⇡uk

3

775

⇤

=

2

664

x0 x̄1 x̄2 x̄3

x1 x0 x̄1 x̄2

x2 x1 x0 x̄1

x3 x2 x1 x0

3

775 ⌫ 0

t 2 S1

kxkA = inf⇢

12 t+

12w0 :

t x

⇤

x toep(w)

�⌫ 0

�

A A [ {�A}

conv(A [ {�A})conv(A)

cos(2⇡u1k)/2� cos(2⇡u2k)/2cos(2⇡u1k)/2 + cos(2⇡u2k)/2

u2 = 0.1411u1 = 0.1413

Nearly optimal rates

u?k 2 [0, 1)for some

Observe a sparse combination of sinusoids

Observe: (signal plus noise)

minp 6=q

d(up, uq) �4

nAssume frequencies are far apart:

x

?m =

sX

k=1

c

?ke

i2⇡mu?k

1

n

kx̂� x?k22 C�

2s log(n)

n

Error Rate:

1

n

kx̂� x?k22 �C

0�

2s

n

No algorithm can do better than

even if we knew all of the frequencies (uk*)

1

n

kx̂� x?k22 �C

0�

2s

n

No algorithm can do better than

even if we knew all of the frequencies (uk*)

No algorithm can do better than

even if the frequencies are well-separated

E1

n

kx̂� x?k22�� C

0�

2s log(n/s)

n

y = x? + !

x̂ = argminx

12kx� yk

22 + µkxkASolve:

• Frequencies generated at random with 1/n separation. Random phases, fading amplitudes.

• Cadzow and MUSIC provided model order. AST and LASSO estimate noise power.

P(�

)

�• Performance profile over all

parameter values and settings• For algorithm s:Ps(�) =

# {p 2 P : MSEs(p) �mins MSEs(p)}#(P)

Lower is better Higher is better

Mean Square Error Performance Profile

Extracting the decomposition

• How do you extract the frequencies? Look at the dual norm:

• Dual norm is the maximum modulus of a polynomial

kvk⇤A = maxa2A

ha, vi = maxu2[0,1)

�����

nX

k=1

vke2⇡iku

�����

At optimality, maximum modulus is attained at the support of the signal

Localization Guarantees

Spurious Amplitudes

Weighted Frequency Deviation

Near region approximation

10 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

1

0

Frequency

Dua

l Pol

ynom

ial M

agni

tude

True FrequencyEstimated Frequency

0.16/n Spurious Frequency

�

l:f̂l�F

|ĉl| � C1��

k2 (n)

n

������cj �

�

l:f̂l�Nj

ĉl

������� C3�

�k2 (n)

n

�

l:f̂l�Nj

|ĉl|�

fj�Td(fj , f̂l)

�2� C2�

�k2 (n)

n

Frequency Localization

20 40 60 80 1000

0.2

0.4

0.6

0.8

1

β

Ps(β

)

AST

MUSIC

Cadzow

100 200 300 400 5000

0.2

0.4

0.6

0.8

1

β

Ps(β

)

AST

MUSIC

Cadzow

5 10 15 200

0.2

0.4

0.6

0.8

1

β

Ps(β

)

AST

MUSIC

Cadzow

−10 −5 0 5 10 15 200

20

40

60

80

100

k = 32

SNR (dB)

m1

AST

MUSIC

Cadzow

−10 −5 0 5 10 15 200

0.01

0.02

0.03

0.04

k = 32

SNR (dB)

m2

AST

MUSIC

Cadzow

−10 −5 0 5 10 15 200

1

2

3

4

5

k = 32

SNR (dB)

m3

AST

MUSIC

Cadzow

Spurious Amplitudes Weighted Frequency Deviation Near region approximation

�

l:f̂l�F

|ĉl| � C1��

k2 (n)

n

������cj �

�

l:f̂l�Nj

ĉl

������� C3�

�k2 (n)

n

�

l:f̂l�Nj

|ĉl|�

fj�Td(fj , f̂l)

�2� C2�

�k2 (n)

n

Incomplete Data/Random Sampling

• Observe a random subset of samples

• On the grid, Candes, Romberg & Tao (2004)

• Off the grid, new, Compressed Sensing extended to the wide continuous domain

• Recover the missing part by solving

• Extract frequencies from the dual optimal solution

T ⇢ {0, 1, . . . , n� 1}

Full observation

Random sampling

minimize

zkzkA

subject to zT = xT .

minimize

zkzkA

subject to zT = xT .

Exact Recovery (off the grid)Theorem: (Candès and Fernandez-Granda 2012) A line spectrum with minimum frequency separation " > 4/s can be recovered from the first 2s Fourier coefficients via atomic norm minimization.

Theorem: (Tang, Bhaskar, Shah, and R. 2012) A line spectrum with minimum frequency separation " > 4/n can be recovered from most subsets of the first n Fourier coefficients of size at least O(s log(s) log(n)).

• s random samples are better than s equispaced samples.• On a grid, this is just compressed sensing• Off the grid, this is new

• No balancing of coherence as n grows.

{

"

WANT {uk, ck}

xm =

sX

k=1

ck exp(2⇡imuk)

−0.50

0.51

−0.5

0

0.5

0

0.5

1

SDP

−0.50

0.51

−0.5

0

0.5

0

0.5

1

BP:4x

−0.50

0.51

−0.5

0

0.5

0

0.5

1

BP:64x

100

105

0

0.2

0.4

0.6

0.8

1

β

P(β

)

Performance Profile − Solution Accuracy

SDP

BP 4

BP 16

BP 64

noiseless

GPS UltrasoundRadar

astronomyimaging seismology

x(t) =kX

j=1

cjg(t� ⌧j)ei2⇡fjtx(t) =kX

j=1

cjg(t� ⌧j⌧j⌧ )ei2⇡fjfjf t

x̂(!) = dPSF(!)kX

j=1

cje�i2⇡!sj

Summary and Future Work• Unified approach for continuous problems in

estimation.• Obviates incoherence and basis mismatch• New insight into bounds on estimation error and exact

recovery through convex duality and algebraic geometry

• State-of-the-art results in classic fields of spectrum estimation and system identification (ask me later!)

`

• Recovery of general sparse signal trains

• Scaling atomic norm algorithms

• More atomization in signal processing... (moments, PDEs, deep atoms)

AcknowledgementsWork developed with Venkat Chandrasekaran, Babak Hassibi (Caltech), Weiyu Xu (Iowa), Pablo A. Parrilo, Alan Willsky (MIT), Maryam Fazel (Washington) Badri Bhaskar, Rob Nowak, Nikhil Rao, Gongguo Tang (Wisconsin).

http://pages.cs.wisc.edu/~brecht/publications.htmlFor all references, see:

http://pages.cs.wisc.edu/~brechthttp://pages.cs.wisc.edu/~brecht

References

• Atomic norm denoising with applications to line spectral estimation. Badri Narayan Bhaskar, Gongguo Tang, and Benjamin Recht. Submitted to IEEE Transactions on Signal Processing. 2012.

• Compressed sensing off the grid. Gongguo Tang, Badri Bhaskar, Parikshit Shah, and Benjamin Recht. Submitted to IEEE Transactions on Information Theory. 2012.

• Near Minimax Line Spectral Estimation. Gongguo Tang, Badri Narayan Bhaskar, and Benjamin Recht. Submitted to IEEE Transactions on Information Theory, 2013.

• The convex geometry of inverse problems. Venkat Chandrasekaran, Benjamin Recht, Pablo Parrilo, and Alan Willsky. To Appear in Foundations on Computational Mathematics. 2012.

• All references can be found at

http://pages.cs.wisc.edu/~brecht/publications.html

http://pages.cs.wisc.edu/~brechthttp://pages.cs.wisc.edu/~brecht