On Fourier and Wavelets: Representation, Approximation … · ISIT04 1 Cargese, Corsica, July 2004...

ISIT04 1

Cargese, Corsica, July 2004

On Fourier and Wavelets: Representation, Approximation and Compression

Martin Vetterl i EPFL & UC Berkeley

1. Introduction through History

2. Fourier and Wavelet Representations

3. Wavelets and Approximation Theory

4. Wavelets and Compression

5. Going to Two Dimensions: Non-Separable Constructions

6. Beyond Shift Invariant Subspaces: Finite Rate of Innovation

7. Conclusions and Outlook

Report Documentation Page Form ApprovedOMB No. 0704-0188

Public reporting burden for the collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, gathering andmaintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information,including suggestions for reducing this burden, to Washington Headquarters Services, Directorate for Information Operations and Reports, 1215 Jefferson Davis Highway, Suite 1204, ArlingtonVA 22202-4302. Respondents should be aware that notwithstanding any other provision of law, no person shall be subject to a penalty for failing to comply with a collection of information if itdoes not display a currently valid OMB control number.

1. REPORT DATE 07 JAN 2005

2. REPORT TYPE N/A

3. DATES COVERED -

4. TITLE AND SUBTITLE On Fourier and Wavelets: Representation, Approximation and Compression

5a. CONTRACT NUMBER

5b. GRANT NUMBER

5c. PROGRAM ELEMENT NUMBER

6. AUTHOR(S) 5d. PROJECT NUMBER

5e. TASK NUMBER

5f. WORK UNIT NUMBER

7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) EPFL; UC Berkeley

8. PERFORMING ORGANIZATIONREPORT NUMBER

9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. SPONSOR/MONITOR’S ACRONYM(S)

11. SPONSOR/MONITOR’S REPORT NUMBER(S)

12. DISTRIBUTION/AVAILABILITY STATEMENT Approved for public release, distribution unlimited

13. SUPPLEMENTARY NOTES See also ADM001750, Wavelets and Multifractal Analysis (WAMA) Workshop held on 19-31 July 2004.,The original document contains color images.

14. ABSTRACT

15. SUBJECT TERMS

16. SECURITY CLASSIFICATION OF: 17. LIMITATION OF ABSTRACT

UU

18. NUMBEROF PAGES

84

19a. NAME OFRESPONSIBLE PERSON

a. REPORT unclassified

b. ABSTRACT unclassified

c. THIS PAGE unclassified

Standard Form 298 (Rev. 8-98) Prescribed by ANSI Std Z39-18

ISIT04 2

Acknowledgements

Collaborations: Sponsors: NSF Switzerland

• T.Blu, EPFL• M.Do, UIUC• P.L.Dragott i , Imper ia l Col lege• P.Marzi l iano, NIT Singapore• I .Maravic, EPFL• R.Shukla, EPFL• C.Weidmann, TRC Vienna

Discussions and Interactions:

• A.Cohen, Par is VI• I . Daubechies, Pr inceton• R.DeVore, Carol ina• D. Donoho, Stanford• M.Gastpar, Berkeley• V.Goyal , MIT• J. Kovacevic, CMU• S. Mal lat , Polytech. & NYU• M.Unser, EPFL

ISIT04 3

Outl ine


• From Rainbows to Spectras• Signal Representat ions• Approximat ions• Compression





6. Beyond Shift Invariant Subspaces


ISIT04 4

From Rainbows to Spectras

Von Freiberg, 1304: Pr imary and secondary rainbow

Newton and Goethe

ISIT04 5

Signal Representations

1807: Fourier upsets the French Academy.. . .

Fourier Series: Harmonic series, frequency changes, f

0

, 2f

0

, 3f

0

, . . .

But. . . 1898: Gibbs’ paper 1899: Gibbs’ correction

Orthogonality, convergence, complexity

++=

t

f(t)

ISIT04 6

1910: Alfred Haar discovers the Haar wavelet

“dual” to the Fourier construction

Haar series:

• Scale changes S

0

, 2S

0

, 4S

0

, 8S

0

. . .• or thogonal i ty

+++=

time

f(t)

t

t

t

m = 0

m = 1

m = -1

ISIT04 7

Theorem 1 (Shannon-48, Whittaker-35, Nyquist-28, Gabor-46)

I f a funct ion f ( t ) contains no frequencies higher than W cps, i t is com-pletely determined by giv ing i ts ordinates at a ser ies of points spaced 1/(2W) seconds apart .[ i f approx. T long, W wide, 2TW numbers speci fy the funct ion]

I t is a representation theorem:

• {s inc(t-n)}

n in Z

, is an orthogonal basis for BL[-

π ,π

]• f ( t ) in BL[-

π ,π

] can be wri t ten as

Note:

• Shannon-BW, BL suff ic ient , not necessary.• many var iat ions, non-uni form etc• Kotelnikov-33!

f t )( ) f n( ) c t n–( )sin⋅n∑=

ĭ20 ĭ15 ĭ10 ĭ5 0 5 10 15 20ĭ0.4

ĭ0.2

0

0.2

0.4

0.6

0.8

1

0 100 200 300 400 500 600 700 800 900 1000ĭ8

ĭ6

ĭ4

ĭ2

0

2

4

6

8

10

... slow...!

ISIT04 8

Representations, Bases and Frames

Ingredients:

• as set of vectors, or ‘ ’atoms’’ ,

• an inner product, e.g. • a ser ies expansion

Many possibil i t ies:

• or thonormal bases (e.g. Four ier ser ies, wavelet ser ies)• b ior thogonal bases• overcomplete systems or f rames

Note: no t ransforms, uncountable

ϕn{ }

ϕn f,⟨ ⟩ ϕn f⋅( )∫=

f t( ) ϕn f,⟨ ⟩ ϕn t( )⋅n∑=

ϕ1

ϕ0

e0

e1e1 ϕ1=

ϕ1

e0 ϕ0=

ϕ1 e1

e0 ϕ0=

ϕ2ϕ0~

UTFOB BOB

ISIT04 9

Approximations, aproximation.. .

The l inear approximation method

Given an orthonormal basis

{g

n

}

for a space

S

and a s ignal

,

the best l inear approximat ion is given by the project ion onto a f ixed sub-space of s ize

M

( independent of f ! )

The error (MSE) is thus

Ex: Truncated Four ier ser ies project onto f i rst M vectors corresponding to largest expected inner products, typical ly LP

f f gn,⟨ ⟩ gn⋅n∑=

f̂M f gn,⟨ ⟩n JM∈∑ gn⋅=

ε̂M f f̂–2

f gn,⟨ ⟩2

n JM∉∑= =

ISIT04 10

The Karhunen-Loeve Transform: The Linear View

Best Linear Approximation in an MSE sense:

Vector processes., i . i .d.:

Consider l inear approximation in a basis

Then:

Karhunen-Loeve transform (KLT):For 0<M<N, the expected squared error is minimized for the basis {gn} where gm are the eigenvectors of RX ordered in order of decreasing eigenvalues.

Proof: eigenvector argument induct ively.

Note: Karhunen-47, Loeve-48, Hotel l ing-33, PCA, KramerM-56, TC

X X0 X1 … XN 1–, , ,[ ]T

= E Xi[ ] 0= E X XT

⋅[ ] RX=

XM̂ X gn,⟨ ⟩n 0=

M 1–

∑ gn⋅= M N<

E ε̂M[ ] RXgn gn,⟨ ⟩n M=

N 1–

∑=

ISIT04 11

Geometric intuit ion: Principal axes of distribution:

Shapes: el l ipsoids

To first approximation, keep al l coefficients above a threshold:

This can be used in many sett ings, classification, denoising,and compression ( inverse waterfi l l ing thm)

N

λλλλi

ISIT04 12

Compression: How many bits for Mona Lisa?

<=> {0,1}

ISIT04 13

A few numbers.. .

D.Gabor, September 1959 (Editorial IRE)" . . . the 20 bi ts per second which, the psychologists assure us, the human eye is capable of taking in, . . . "

Index al l pictures ever taken in the history of mankind• b i ts

Huffman code Mona Lisa index• a few bi ts (Lena Y/N?, Mona Lisa. . . ) , what about R(D) . . . .

Search the Web!• ht tp: / /www.google.com, 5-50 bi l l ion images onl ine, or 33-36 bi ts

JPEG• 186K.. . There is plenty of room at the bottom!• JPEG2000 takes a few less, thanks to wavelets. . .

Note: 2(256x256x8) possible images (D.Field)

Homework in Cover-Thomas, Kolmogorov, MDL, Occam, DNA, etc

( f rom a contemporary: 0 bi ts, I don’ t care for th is modern stuff )

100 years 1010

⋅ 44∼

ISIT04 14

Source Coding: some background

Exchanging description complexity for distortion:• rate-distort ion theory [Shannon:58, Berger:71]• known in few cases.. . l ike i . i .d. Gaussians (but t ight : no better way!)

• typical ly: d i ff icul t , s imple models, h igh complexi ty (e.g. VQ)• high rate resul ts, low rate of ten unknown

distortion

D(R)

N 0 σ2,( ) D R( ) σ2

22R–

⋅=

complexity

or -6dB/bit

ISIT04 15

Limitations of the Standard Models

“Splendeurs et misères de la fonction débit-distortion” (after Balsac)

Precise results• beaut i fu l (maybe too much for i ts own good)• upper and lower bounds• construct ive

Problems• complexi ty: exponent ia l in code length • code construct ion: f inding good codes is hard

Paradox:• Best codes used in pract ice are subopt imal (Effros)• t ransform codes dominate the scene of ‘ ’ real ’ ’ compression

Audio/Image/Video: distortion measures?

So: unlike in lossless compression, lossy compression uses IT in a l imited way;)

ISIT04 16

New image coding standard . . . JPEG 2000

Old versus new JPEG: D(R) on log scale

Main points:• improvement by a few dB’s• lot more funct ional i t ies (e.g. progressive download on internet)• at h igh rate ~ -6db per bi t : KLT behavior• low rate behavior: much steeper: NL approximat ion effect? • is th is the l imi t?

0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2−50

−45

−40

−35

−30

−25

−20

-6 dB/bit

old JPEG (DCT)

new JPEG (wavelets)

ISIT04 17

The Swiss Army Knife Formula of Transform Coding [Goyal00]

Model: i id vector process of size N, µµµµ=0, Rx, MSE, high rate• vector quant izer, entropy code

• t ransform and scalar quant izers, entropy code

Trace min: ortho; diff . entropy min: independence• Gaussian case: coincide! but in general not . . .

γα γ1– β x̂x

x̂x

α1

α2

αN

T γ

y1

y2

yN

i1

i2

iN

β1

β2

βN

i1

i2

iN

y1

y2

yN

T1–γ

1–bits

D R( ) 112N---------- tr T

1–T

1–( )

T( ) 2

2N----Σh yi( )

22R–

⋅ ⋅ ⋅=

ISIT04 18

Representation, Approximation and Compression:Why does it matter anyway?

Parsimonious or sparse representation of visual information is key in• storage and transmission• indexing, searching, c lassi f icat ion, watermarking• denois ing, enhancing, resolut ion change

But: i t is also a fundamental question in • informat ion theory• s ignal / image processing• approximat ion theory• v is ion research

Successes of wavelets in image processing:• compression (JPEG2000)• denois ing• enhancement• c lassi f icat ion

Thesis: Wavelet models play an important role

Antithesis: Wavelets are just another fad!

ISIT04 19

Interaction of topics

• AT: determinist ic set t ing, large classes of fcts• HA: funct ion c lasses, existence, embeddings• IT: boundings, converses, stochast ic set t ing• SP: bases, algor i thms, complexi ty

The interaction is the fun!

approximationtheory

signalprocessing

informationtheory

harmonicanalysis

ISIT04 20

Outline


2. Fourier and Wavelet Representations• Four ier and Local Four ier Transforms• Wavelet Transforms• Piecewise Smooth Signal Representat ions






ISIT04 21

2. Fourier and Wavelet Representations: Spaces

Norms:

Hilbert spaces:

Inner product:

Orthogonal i ty:

Banach spaces:

x p x n[ ]p

n∑ 1 p⁄

= f p f t( )p

td

∞–

∞

∫ 1 p⁄

=

l2 Z( ) x: x 2 ∞<( ){ }= L2 R( ) f: f 2 ∞<( ){ }=

x y,⟨ ⟩ x*

n[ ]y n[ ]n∑= f g,⟨ ⟩= f

*t( )g t( ) td∫

x⊥y x y,⟨ ⟩ 0=⇔

x f s.t. x p, f p ∞<, p general

ĭ1

ĭ0.5

0

0.5

1

1.5

p ∞=

p 2=

p 1=

0 p 1< <

x 0[ ]

x 1[ ]

p-norm = 1

ISIT04 22

A Tale of Two Representations: Fourier versus Wavelets

Orthonormal Series Expansion

Time-Frequency Analysis and Uncertainty Principle

Then

f αnϕnn Z∈∑= αn ϕn f,⟨ ⟩= <ϕn ϕm>, δn m–= f 2 α 2=

f t( ) F ω( )↔ ∆2t t

2f t( ) td∫= ∆

2ω ω

2F ω( ) ωd∫=

∆2t ∆

2ω⋅ π

2---≥

ω

t

not arbi t rar i ly sharp in t ime and frequency

ISIT04 23

Local Fourier Basis?

The Gabor or Short- t ime Four ier Transform

Time-frequency atoms local ized at

When “smal l enough”

where

Example: Spectrogram

ϕm n, t( ) w t nT–( )ejmω0 t nT–( )–

=

nT mω0,( )

t ime

frequency

T ω0,

f t( ) c Fm n, ϕm n, t( )⋅≈ Fm n, <ϕm n, f>,=

ISIT04 24

The Bad News.. .

Balian-Low Theorem

is a short- t ime Four ier f rame with cr i t ical sampl ing ( )

then ei ther

or

or: there is no good local or thogonal Four ier basis!

Example of a basis: block based Fourier series

Note: consequence of BL Thm on OFDM, RIAA

ϕm n, Tω0 2π=

∆2t ∞= ∆

2ω ∞=

ωsinω

------------∼

t2TT0T–

ISIT04 25

The Good News!

There exist good local cosine bases.

Replace complex modulat ion ( ) by appropr iate cosine modulat ion

where is a power complementary window

Result : MP3!

Many general isat ions. . .

ejmω0t

ϕm n, t( ) w t nT–( ) π2--- m 1

2---+

t nT– T2---+

cos=

w t( )

w t nT–( )n∑ 1=

w t( )2

TT

ISIT04 26

Another Good News!

Replace (shi f t , modulat ion)

by (shi f t , scale)

or

then there exist “good” local ized orthonormal bases, or wavelet bases

Ψm n, t( ) 2m 2⁄–

Ψ t 2m

n–

2m-----------------

= n m Z∈,

t ime

frequency

ISIT04 27

Examples of bases

Haar Daubechies, D2

0 5 10 15 20 25 30 35ĭ1

0

1h(1)

0 5 10 15 20 25 30 35 40ĭ1

0

1h(2)

0 5 10 15 20 25 30 35 40 45ĭ1

0

1h(3)

0 5 10 15 20 25 30 35 40 45ĭ1

0

1g(3)

ISIT04 28

Wavelets and representation of piecewise smooth functions

Goal: eff icient representation of signals l ike:

where:• Wavelet act as s ingular i ty detectors• Scal ing funct ions catch smooth parts• “Noise” is c i rcular ly symmetr ic

Note: Fourier gets al l Gibbs-ed up!

Wavelets

Noise

time

f(t) Scaling functions

ISIT04 29

Key characteristics of wavelets and scaling functions

Wavelets derived from fi l ter banks, ortho-LP with N zeroes at π , [Daubechies-88],

Scaling function:

Orthonormal wavelet family:

Scaling function and approximations• Strang-Fix theorem: i f has N zeros at mult ip les of 2π (but the or-

ig in) , then spans polynomials up to degree N-1

• Two scale equat ion:

• smoothness: fo l lows from N, α = 0,203 N

G z( ) 1 z1–

+( )N

R z( )⋅=

φ ω( ) G ej ω 2

i( )⁄( )

i 1=

∞

∏=

ψm n,t( ) 2

m/2–ψ 2

m–t n–( )=

φ ω( )ϕ t n–( ){ }n Z∈

cn ϕ t n–( )⋅n∑ t

k= k 0 1 …N 1–, ,=

ϕ t( ) 1

2------- gn ϕ 2t n–( )⋅

n∑⋅=

ISIT04 30

Lowpass f i l ters and scaling functions reproduce polynomials• I terate of Daubechies L=4 lowpass f i l ter reproduces l inear ramp

Scaling functions catch ‘ ’ trends’’ in signals

0 50 100 150 200 250−0.05

0

0.05

0.1

0.15

0.2

0 50 100 150 200 250 300 350 400 450−200

−100

0

100

200

300

400

500

scalingfunction

linearramp

ISIT04 31

Wavelet approximations• wavelet ψ has N zero moments, k i l ls polynomials up to deg. N-1• wavelet of length L= 2N-1, or 2N-1 coeffs inf luenced by s ingular i ty at

each scale, wavelet are s ingular i ty detectors,• wavelet coeff ic ients of smooth funct ions decays fast ,

e.g. f in cp,m << 0

Note: al l this is in 1 dimension only, 2D is another story.. .

ψm n, f,⟨ ⟩ c2m p 1

2---–

=

0 0.5 1 1.5 2 2.5 3−1.5

−1

−0.5

0

0.5

1

1.5

2

1 5

−1

−0.5

0

0.5

1

1.5

2

ISIT04 32

How about singularit ies?

If we have a singularity of order n at the origin (0 : Dirac, 1: Heavis ide, . . . ) , the CWT transform behaves as

In the orthogonal wavelet series: same behavior, but only L=2N-1 co-eff icients influenced at each scale!

• e.g. Dirac/Heavis ide: behavior as and , m<<0

Wavelets catch and characterize singularit ies!

X a 0,( ) cn an 1

2---–

⋅=

Time

Scale

Time

1 2 3 4

Time

0

1

2

3

4

5

6

Amplitude

small

2m– 2⁄

2m 2⁄

large

ISIT04 33

Thus: a piecewise smooth signal expands as:

• lowpass catches trends, polynomials• a s ingular i ty inf luences only L wavelets at each scale• wavelet coeff ic ients decay fast

ĭ2

0

2x

ĭ0.5

0

0.5X

h(1)

ĭ0.5

0

0.5

Xh(2)

ĭ0.2

0

0.2

Xh(3)

0 100 200 300 400 500 600 700 800 900 1000

ĭ1

0

1X

g(3)

ISIT04 34

Outline



3. Wavelets and Approximation Theory• Sobolev and Besov spaces• Non- l inear approximat ion• Four ier versus wavelet , LA versus NLA





ISIT04 35

More Spaces

Cp spaces: p-t imes di ff . wi th bounded der ivat ives -> Taylor expansions

Holder/Lipschitz αααα : local ly α smooth (non- integer)

Sobolev Spaces

I f then is - t imes cont inuously di fferent iable

Equivalent ly decays at

Besov Spaces wi th respect to a basis ( typical ly wavelets)

or wavelet expansion has f in i te norm

Ws

R( )

f l2

R( )∈ ω2s

F ω( )2ωd

∞–

∞

∫ ∞<

s n 12---+> f n

F ω( ) 1

1 ω+( )s 1 2⁄ ε+ +-------------------------------------------

Bp I( )

f l2

I( )∈

f B p, <Ψm n, f>,p

n∑

m∑ 1 p⁄

∞<=

lp

FT

f t( )

t

F ω( )

ISIT04 36

From linear to non-l inear approximation theory

The non-l inear approximation method

Given an orthonormal basis {gn} for a space S and a s ignal

,

the best nonl inear approximat ion is given by the project ion onto an adapted subspace of s ize M (dependent on f ! )

The error (MSE) is thus

and .

Difference: take the f irst M coeffs ( l inear) or take the largest M coeffs (non-l inear)

f f gn,⟨ ⟩ gn⋅n∑=

f̃M f gn,⟨ ⟩n IM∈∑ gn⋅=

IM: f gn,⟨ ⟩n IM∈

f gm,⟨ ⟩m IM∉

≥ set of M largest , ⟨ ⟩

ε̃M f f̃–2

f gn,⟨ ⟩2

n IM∉∑= =

ε̃M ε̂M≤

ISIT04 37

Nonlinear approximation• This is a s imple but nonl inear scheme• Clear ly, i f AM(.) is the NL approximat ion scheme:

This could be called ‘‘adaptive subspace f i t t ing’’

From a compression point of view, you ‘‘pay’’ for the adaptivity• in general , th is wi l l cost

b i ts

which cannot be spent on coeff ic ient representat ion anymore

LA: p ick a subspace a pr ior i NLA pick af ter seeing the data

AM x( ) AM y( )+ AM x y+( )≠

log Nk

ISIT04 38

Non-Linear Approximation Example

Nonlinear approximation power depends on basis

Example:

Two different bases for [0 ,1] :• Four ier ser ies • Wavelet ser ies: Haar wavelets

Linear approximation in Fourier or wavelet bases

Nonlinear approximation in a Fourier basis

Nonlinear approximation in a wavelet basis

11/sqrt2t

f( t )

cs t

ej2πkt

{ }k ℑ∈

ε̂M 1 M⁄∼

ε̃M 1 M⁄∼

ε̃M 1 2M

⁄∼

ISIT04 39

Fourier versus Wavelet bases, LA versus NLA

N= 1024, M=64

Fourier ( left): LA versus NLA does not matter

Wavelets (r ight): NLA does orders of magnitude better!

0 100 200 300 400 500 600 700 800 900 1000

0

0.5

1

0 100 200 300 400 500 600 700 800 900 1000

0

0.5

1

0 100 200 300 400 500 600 700 800 900 1000

0

0.5

1

0 100 200 300 400 500 600 700 800 900 1000

0

0.5

1

0 100 200 300 400 500 600 700 800 900 1000

0

0.5

1

0 100 200 300 400 500 600 700 800 900 1000

0

0.5

1

ISIT04 40

Nonlinear approximation theory and wavelets

Approximation results for piecewise smooth fcts• between discont inui t ies,

behavior by Sobolev or Besov regular i ty• k der ivat ives coeffs when • Besov spaces can be def ined with wavelet bases. I f

then [DeVoreJL92]:

f ( t )

t

⇒ 2m k 1/2–( )

∼ m 0«

f G p, f gn,⟨ ⟩p

∑( )1 p⁄

= ∞< 0 p 2< <

ε̃M o M1 2 p⁄–

( )=

ISIT04 41

Approximation in Sobolev and Besov Spaces

Linear Approximation, Ws[0,N]• Sobolev-s: uni formly smooth• Four ier : • Wavelets:

Non-Linear Approximation • Besov-s: smooth between a f in i te # of

d iscont inui t ies• Four ier : does not work, • Wavelets: approximat ion power given by the smoothness!• Key: effect of d iscont inui t ies l imi ted, because wavelets are

concentrated around discont inut ies• f ( t ) in Ws(0,N) between f in i te # of d iscont inui t ies,

then f( t ) in Bp(0,N) (wavelet of compact support)• Then:

• resul t can be ref ined to get

ε̂M M2s– δ–

= δ 0>ε̂M M

2s– δ–= δ 0>

ε̃M M1–

=

ε̃M M1 2

p---–

= 1p--- s<

ε̃M M2s– δ–

= δ 0>

102

103

104

10ĭ10

10ĭ9

10ĭ8

10ĭ7

10ĭ6

10ĭ5

10ĭ4

10ĭ3

Number of retained coefficients (N)

MS

E

Nonlinear Approximation Error for Piecewise Smooth Functions

FourierWavelet

ISIT04 42

Outline




4. Wavelets and Compression • A smal l but instruct ive example• piecewise polynomials and D(R)• piecewise smooth and D(R)• improved wavelet schemes




ISIT04 43


Compression is just one bit tr ickier than approximation.. .

A small but instructive example:

Assume• x[n] = α δ [n-k] , s ignal is of length N, k is U[0,N-1] and α is N(0,1). • This is a Gaussian RV at locat ion k

• Note: Rx = I !

Linear approximation:

Non-l inear approximation, M > 0:

Nk

~N(0,1)

ε̂M1M-----=

ε̃M 0=

ISIT04 44

Given budget R for block of size N :

1. Linear approximation and KLT: equal distribution of R/N bits

This is the optimal l inear approximation and compression!

2. Rate-distor tion analysis [Weidmann:99]

High rate case:• Obvious scheme: pointer + quant izer

• This is the R(D) behavior for R >> Log N• Much better than l inear approximat ion

Low rate case:• Hamming case solved, inc. mult ip le spikes:

- there is a l inear decay at low rates• L2 case: upper bounds that beat l inear approx.

D R( ) c σ2

22 R N⁄( )–

⋅ ⋅=

D R( ) c σ2

22 R Nlog–( )–

⋅ ⋅=

ISIT04 45

Example 1: Binary, Hamming, 1 and k spikes

Example 2: Bernoull i -Gaussian

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

R/Rmax

Ham

min

g D

isto

rtio

n D

/Dm

ax

K = 16, 12, 8, 4, 2 Spikes in N = 32 Positions

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

R/log(N)

Ham

min

g D

isto

rtio

n

Single Spike in N = 2, 3, 4, 6, 10, 20 Positions

N=20

N=2

K=4

K=16

N=32

0 0.2 0.4 0.6 0.8 1 1.2 1.4ĭ50

ĭ45

ĭ40

ĭ35

ĭ30

ĭ25

ĭ20

ĭ15

ĭ10

ĭ5

0

Rate [bits]

Dis

tort

ion

[MS

E d

B]

Bernoulliĭ0.11 x Gaussian Spike

BlahutĭArimotoUpper BoundAsymptote

p=0.11

ISIT04 46

Piecewise smooth functions: pieces are Lipschitz-αααα

The fol lowing D(R) behavior is reachable [CohenDGO:02]:

There are 2 modes:• corresponding to the Lipschi tz-α p ieces

• corresponding to the discont inui t ies

Wavelets

Noise

time

f(t) Scaling functions

D R( ) c1 R2α–

⋅ c3 R 2c4– R⋅

⋅ ⋅+=

R2α–

R 2c– R⋅

⋅

ISIT04 47

Lipschitz-αααα pieces: Linear Approximation

The wavelet transform at scale j decays as ( j << 0)

Keep coeff icients up to scale J, or choose a stepsize ∆ for a quantizer

Therefore, M ~ 2J coeff icientsSquared error:

Rate:• number of coeff ic ients

Thus

Just as good as Fourier (~R−2α) , but local!

wj 2j α 1 2⁄+( )

≈

∆ 2J α 1 2⁄+( )

≈

2j–

22j α 1 2⁄+( )

⋅j ∞–=

J–

∑ 22Jα–

M2α–

∼ ∼

c M⋅

D R( ) c R2α–

⋅∼

ISIT04 48

Rate-distortion bounds for piecewise polynomial functions

D(R) behavior of nonlinear approximation with wavelets

Consider the simplest case: Haar!Recal l that

and consider descr ib ing the

s igni f icantcoeff ic ients

Choose a stepsize ∆ for a quantizer. Therefore• number of scales J before coefs set to zero • number of b i ts per coeff ic ient

, thus

Distortion: number of scales t imes

Thus

1

t

f ( t ) uniformly distributedrandom constant

ε̃M 2M–

≅̃ cj 2j 2⁄

≅

J

∆

2 j /2 ~ c j

j

1/∆( )log∼

1/∆( )log∼ R J2

∼

∆2

J 2J–

⋅∼⋅

Dw R( ) C3 R 2c2– R⋅

⋅ ⋅=

ISIT04 49

Rate-distortion behavior using an oracle

An oracle decides to optimally code a piecewise polynomialby al locating bits ‘ ’where needed’’:

Consider the s implest case

Two approximation errors• ∆ t: quant izat ion of step locat ion• ∆a: quant izat ion of ampl i tude

Rate al location: Rt versus Ra

Result :

f ( t ) , f^( t )

t1

f

f^

Dp R( ) C1 2R 2⁄–

⋅=

ISIT04 50

Piecewise polynomial, with max degree N

A. Nonlinear approximation with wavelets having N+1 zero moments

B. Oracle-based method

Thus • wavelets are a gener ic but subopt imal scheme• oracle method asymptot ical ly super ior but dependent on the model

Conclusion on compression of piecewise smooth functions:

D(R) behavior has two modes:

• 1/polynomial decay: cannot be (substant ia l ly) improved• exponent ia l mode: can be improved, important at low rates

Dw R( ) C′w 1 α CwR+( ) 2CwR–

⋅ ⋅=

Dp R( ) C′p 2Cp R⋅( )–

⋅=

D R( ) c1 R2α–

⋅ c3 R 2c4– R⋅

⋅ ⋅+=

ISIT04 51

Can we improve wavelet compression? Footprints!

Key: Remove depencies accross scales:• dynamic programming: Vi terbi- l ike algor i thm• t ree based algor i thms: pruning and jo in ing• wavelet footpr ints: wavelet vector quant izat ion

Theorem [DragottiV:03]:

Consider a piecewise smooth s ignal f ( t ) , where pieces are Lipschi tz-α . There exists a piecewise polynomial p( t ) wi th pieces of maximum degree such that the residual is uni formly Lipschi tz-α .

This is a generic split into piecewise polynomial and smooth residual

α rα t( ) f t( ) p t( )–=

+=

function f(t) piecewise polynomial Lipschitz-α

ISIT04 52

Footprint Basis and Frames

Suboptimality of wavelets for piecewise polynomials is dueto independent coding of dependent wavelet coeff icients

Compression with wavelet footprints

Theorem: [DragottiV:03]Given a bounded piecewise polynomial of deg D with K discont inui-t ies. Then,a footpr int based coder achieves

This is a computat ional effect ive method to get oracle performance

What is more, the gener ic spl i t “p iecewise smooth” into“uni formly smooth + piecewise polynomial” a l lows to f ix wavelet scenar ios, to obtain

This can be used for denoising and superresolution

Dw R( ) C R 2R–

⋅ ⋅∼

D R( ) c1 2c2 R⋅( )–

⋅=

D R( ) c1 R2α–

⋅ c2 2c3– R⋅

⋅+=

ISIT04 53

Denoising (use coherence across scale)

50 100 150 200 250 300 350 400 450 500−3

−2

−1

0

1

2

3

4

5

6

7

50 100 150 200 250 300 350 400 450 500−3

−2

−1

0

1

2

3

4

5

6

7

50 100 150 200 250 300 350 400 450 500−3

−2

−1

0

1

2

3

4

5

6

7

50 100 150 200 250 300 350 400 450 500−3

−2

−1

0

1

2

3

4

5

6

7

50 100 150 200 250 300 350 400 450 500−3

−2

−1

0

1

2

3

4

5

6

7

Original signal Noisy Signal (SNR=15.62dB)

Hard-Thresholding (SNR=21.3dB) Cycle-Spinning (SNR=25.4dB)

Denoising with Footprints (SNR=27.2dB)

This is a vector thresholdingmethod adapted to wavelet singularities

ISIT04 54

Outline





5. Going to Two Dimensions: Non-Separable Constructions• the need for t ru ly two-dimensional construct ions• t ree based methods• non-separable bases and frames



ISIT04 55


Going to two dimensions requires non-separable bases

Objects in two dimensions we are interested in

• textures: per pixel• smooth surfaces: per object !

smooth surface, polynomial

smooth boundary c2

texture

D R( ) C0 22R–

⋅=D R( ) C1 2

2R–⋅=

ISIT04 56

Models of the world:

Many proposed models:• mathematical d i ff icul t ies• one size f i ts a l l . . .• Lena is not PC, but is she BV?

But: Fourier, DCT, wavelets use a separable approach ( l ine/column.. . )

=> geometry based image processing

Gauss-Markov Piecewise polynomial the usual suspect

ISIT04 57

Recent work on geometric image processing

Long history: compression, vision, f i l ter banks

Current affairs:

Signal adapted schemes• Bandelets [LePennec & Mal lat ] : wavelet expansions centered at

at d iscont inui ty as wel l as along smooth edges• Non- l inear t i l ings [Cohen, Mattei ] : adapt ive segmentat ion• Tree structured approaches [Shukla et a l , Baraniuk et a l ]

Bases and Frames• Wedgelets [Donoho]: Basic element is a wedge• Ridgelets [Candes, Donoho]: Basic element is a r idge• Curvelets [Candes, Donoho]

Scal ing law: width ~length2

L(R2) set up• Mult id i rect ional pyramids and contour lets [Do et a l ]

Discrete-space set-up, l (Z2) Tight f rame with smal l redundancyComputat ional f ramework

This is where the action is!

ISIT04 58

Nonseparable schemes and approximation

Approximation properties:• wavelets good for point s ingular i t ies• r idgelets good for r idges• curvelets good for curves

Consider c2 boundary between two csts

Rate of approximation, M-term NLA in bases, c2 boundary• Four ier : O(M-1/2)• Wavelets: O(M-1)• Curvelets: O(M-2)

Xl tW l t

# curvelet coeffs O(2j/2)# wavelet coeffs O(2j)

ISIT04 59

Compression of non-separable objects

Objects we know how to compress.. . .

Approximation• Wavelets• Ridgelets

Rate/distortion• Oracle • Wavelets. . . .poor• Ridgelets. . . .subopt imal• adapt ive schemes: c lose to oracle• f ixed basis: under invest igat ion

Basis element

EM M1–

∼EM 2

M–∼

D R( ) C 22R–

⋅=

ISIT04 60

Tree Based Geometric Compression [ShuklaDDV:03]

Idea• t ree and quadtree algor i thms popular, many pruning algor i thms

opt imal i ty proofs for wedgelets [Donoho:99]• new pruning and jo in ing algor i thm

Intuit ion: ful l tree dyadic tree pruned & joined tree

Results: Rate-distortion optimal for piecewise polynomials

that is, l ike an oracle method (up to constants)

(1,0)

(4,8) (4,9)

(1,1)

(2,3)

(3,5)(3,4)

(2,2)

(0,0)

(2,3)

(4,8)(4,9)

(3,4) (3,5)

(0,0)

(2,2)

(1,0) (1,1)

Joint Encoding Joint Encoding

D R( ) 2c2R–

∼D R( ) R 2c1– R⋅

⋅∼D R( ) R1–

∼NJ 2J

∼ NJ J∼ NJ J0

∼

D R( ) c1 2c2 R⋅( )–

⋅=

ISIT04 61

Extension to Quadtree:

• Example

Results:• consider a piecewise polynomial 2D signal , wi th polynomial

boundar ies, the fo l lowing rate-distort ion behavior is achieved

• th is is l ike an oracle method, and >> than prune algor i thmswhich have a penal ty

• complexi ty: polynomial

D R( ) c3 2c4 R⋅( )–

⋅=

R

ISIT04 62

The prune-join quadtree algorithm

• polynomial f i t to surface and to boundary on a quadtree• rate-distort ion opt imal t ree pruning and jo in ing

Note: careful R(D) optimization!

quadtree with R(D) pruning R(D) Joining of ‘’similar’’ leaves

ISIT04 63

Geometric Compression versus JPEG2000 at 0.11 bits/pixel, PSNR:

28.95 27.75

30.01 29.22

pruned-joined quadtree JPEG2000

ISIT04 64

Behavior of tree algorithms on piecewise smooth fcts

ppf: piecewise polynomial functions

psf: piecewise smooth functions, a-smooth

at most log penalty with polynomial complexity(and a bit more work gets r id of logs.. . )

Interesting scaling laws, good behavior in practice!

Signal Class

Oracle Coder

Wavelet Coder

Prune tree Coder

Prune-join tree Coder

1-D PPF

2-D PPF

1-D PSF

2-D PSF

2 cR– 2c3R–

2c1 R–

2 dR– 2c5R–

2c2 R–

2c4 R–

R 2a– R 2a–

R a– RlogR

------------ a

RlogR

------------ 2a Rlog

R------------ 2a

RlogR

------------ aRlog

R------------

RlogR

------------

ISIT04 65

Directional bases and contourlets [M.Do]

Goal: f ind a discrete-space construction that has good approximation properties for smooth functions with smooth boundaries

• d i rect ional analysis as in a Radon transform• mult i resolut ion as in wavelets and pyramids• computat ional ly easy• bases or low redundancy frames

Background:• curvelets [Candes-Donoho] indicate that ‘ ’good’’ f ixed bases

do exist for approximat ion of p iecewise smooth 2D funct ions• a f requency-direct ion relat ionship indicates a scal ing law

Idea: • d i rect ional analysis: d i rect ions are key• mult i resolut ion analysis

Result:• one-more- let : contour lets!

d j1 2⁄

∼

ISIT04 66

Directional Fi l ter Banks [BambergerS:92, DoV:02]• d iv ide 2-D spectrum into s l ices wi th i terated tree-structured f-banks

6

3210

5

6

7

3 2

5

74

1

4

w1

(-pi,-pi)

(pi,pi)w2

0

Q

+ x^

Q

x

Qy0

Qy1

ISIT04 67

Example of directional basis functions • 64 channels, e lementary f i l ters are Haar f i l ters• orthonormal direct ional basis• 64 equivalent f i l ters, the 32 ‘ ’most ly hor izontal ’ ’ ones are shown

This ressembles a ‘ ’ local Radon transform’’ , or radonlets!• changes of s ign ( for or thonormal i ty)• approximate l ines (discret izat ions)

ISIT04 68

Mult iresolution directional pyramid

Result: • “ t ight” pyramid and orthogonal d i rect ional channels => t ight f rame• low redundancy < 4/3, computat ional ly eff ic ient

2,2

imagebandpassdirectionalchannels

bandpassdirectionalchannels

ISIT04 69

A directional multiresolution analysis

Theorem [Do:01]: For a f in i te number of d i rect ions, th is generates a t ight f rame for L2(R2) wi th f rame bound equal 1

Method: Define embedded lowpass direct ional spaces Vj,k and direct ional bandpass spaces Wj,k

This defines contourlets: how do they compare to wavelets?

Approximation: M-term NLA sat isf ies [CandesD:00]

possible wi th s inc f i l ters, . . . open problem i f compact support contour lets exist . . . .

��

��

��

��

V_k

V_2k

V_2k+1

w2

w1

f fcontourlet–2 1

M2-------∼

ISIT04 70

Basis functions: wavelets versus contourlets

50 100 150 200 250

50

100

150

200

250

50 100 150 200 250

50

100

150

200

250

ISIT04 71

Expansion Example

Pepper image and its expansion

Compression, denoising, inverse problems:if i t is sparse, i t is a good start!

ISIT04 72

Example: denoising with contourlets

original noisy

wavelet contourlets 13.8 dB 15.4 dB

ISIT04 73

Outline






6. Beyond Shift Invariant Subspaces: Finite Rate of Innovation• Shi f t - Invar iance and Mult i resolut ion Analysis• A Var iat ion on a Theme by Shannon• A Representat ion Theorem


ISIT04 74

Shift- Invariance and Multiresolution Analysis

Most sampling results require shift- invariant subspaces•

Wavelet constructions rely in addit ion on scale-invariance•

Multiresolution analysis (Mallat, Meyer) gives a powerful framework. Yet i t requires a subspace structure.. .

Example: uniform or B-splines

Question: can sampling be generalized beyond subspaces?

Note: Shannon BW suff ic ient , not necessary!

f t( ) V∈ f t nT–( ) V∈⇔ n Z∈

f t( ) V0∈ f 2m

t( ) V m–∈⇔ m Z∈

f(t)

t

φ(t)

t

ISIT04 75

A Variation on a Theme by Shannon

Shannon, BL case: or 1/T degrees of f reedom per uni t of t ime

But: a single discont inui ty, and no more sampl ing theorem.. .

Q: Are there other s ignals wi th f in i te number of degrees of f reedom per uni t of t ime that al low exact sampl ing resul ts?=> Finite rate of innovation

Usual setup:

: signal, : sampling kernel, :fi l tering of and :samples

f t( ) f nT( ) c t T⁄ n–( )sinn Z∈∑=

ω

F (ω)f(t)

t

F

x t( ) h t( ) y t( ) x t( ) yn

ISIT04 76

A Toy Example

K Diracs on the interval: 2K degrees of freedom. Periodic case:

Key: The Four ier ser ies is a weighted sum of K exponent ia ls

Result: Taking 2k+1 samples f rom a lowpass version of BW-(2K+1) a l lows to perfect ly recover x( t )

Method: Yule-Walker system, annihi lat ing f i l ter, Vandermonde system

Note: Relat ion to spectral est imat ion and ECC (Ber lekamp-Massey)

x t( ) ck1τ--- e

j2πm t tk–( )τ-------------------------------

m Z∈∑

k 0=

K 1–

∑=ckδ t tk– nτ–( )k 0=

K 1–

∑n Z∈∑=

X m[ ] 1τ--- ck e

j– 2πmtkτ---------------------

k 0=

K 1–

∑=

ISIT04 77

A Representation Theorem [VMB:02]

For the class of periodic FRI signals which includes• sequences of Diracs• non-uni form or f ree knot spl ines• piecewise polynomials

there exist sampling schemes with a sampling rate of the order of the rate of innovation with perfect reconstruction at polynomial cost.

Variations: • f in i te length s ignals, local kernels• Two-dimensions

−128 0 128 256 384 512 640 768 896 1024 1152−0.2

−0.15

−0.1

−0.05

0

0.05

0.1

0.15

−600 −400 −200 0 200 400 600−5

0

5

10

15

20x 10

−3

−128 0 128 256 384 512 640 768 896 1024 1152−0.15

−0.1

−0.05

0

0.05

0.1

x t( ) h t( ) y t( ) yn,

ISIT04 78

and the noisy case.. . .

Use subspace methods ( I .Maravic)

Application example: UWB ( low rate of innovat ion. . .but lots of noise!)

0 100 200 300 400 500 600 700 800 900 1000ĭ3

ĭ2

ĭ1

0

1

2

3

4

5Noisy nonuniform spline and reconstructed signal

original noisy signalreconstructed signal

0 100 200 300 400 500 600 700 800 900 1000ĭ5

ĭ4

ĭ3

ĭ2

ĭ1

0

1

2

3

4

5Noisy piecewise linear signal and reconstructed signal

original noisy signalreconstructed signal

0 2000 4000 6000 8000 10000 12000−0.02

−0.015

−0.01

−0.005

0

0.005

0.01

0.015

0.02

0.025

t

Transmitted sequence of UWB pulses & Received signal

ISIT04 79

A local algorithm for FRI sampling [DVB:04]

The return of Strang-Fix!

local, polynomial complexity reconstruction, for diracs and piecewise polynomials

a0 (tĭt

0)

a1 (tĭt

1)

t0 t0

a0 (tĭt

0)

a1 (tĭt

1)

∑n yn = a0 + a1

∑n nyn = a0t0 + a1t1

0 t

a0 (tĭt

0)

a1 (tĭt

1)

a0 (tĭt

0)

a1 (tĭt

1)

0 t

∑n n2yn = a0t

20 + a1t

21

∑n n3yn = a0t

30 + a1t

31

ISIT04 80

Conclusions

Wavelets and the French revolution

• too ear ly to say?• f rom smooth to piecewise smooth funct ions

Sparsity and the Art of Motorcycle Maintenance

• sparsi ty as a key feature wi th many appl icat ions• denois ing, inverse problems, compression

LA versus NLA: • approximat ion rates can be vast ly di fferent!

To first order, operational, high rate, D(R)• improvements st i l l possible• low rate analysis di ff icul t

Two-dimensions: • real ly harder! and none used in JPEG2000.. .• approximat ion starts to be understood, compression most ly open• contour let leads to eff ic ient a lgor i thms

Beyond subspaces: • FRI resul ts on sampl ing, many open quest ions!

ISIT04 81

Outlook

Do we understand image representation/compression better?• h igh rate, h igh resolut ion: there is promise• low rate: room at the bottom?

New images• p lenopt ic funct ions (set of a l l possible images)

• non BL images (FRI?)• manifolds, structure of natural images

Distributed images• interact ive approximat ion/compression• SW, WZ, DKLT.. .

ISIT04 82

Why Image Representation Remains a Fascinating Topic. . .

A lone student standingin front of four tanks.

ISIT04 83

Publications

For overviews:• D.Donoho, M.Vetter l i , R.DeVore and I .Daubechies, Data Compres-

sion and Harmonic Analysis, IEEE Tr. on IT, Oct.1998. • M. Vetter l i , Wavelets, approximat ion and compression, IEEE Signal

Processing Magazine, Sept. 2001

Coming up:• M.Vetter l i , J .Kovacevic and V.Goyal , Four ier and Wavelets:

Theory, Algor i thms and Appl icat ions, Prent ice-Hal l , 200X ;)

For more detai ls, Theses• C.Weidmann, Ol igoquant izat ion in low-rate lossy source coding, PhD

Thesis, EPFL, 2000.• M. N. Do, Direct ional Mult i resolut ion Image Representat ions , Ph.D.

Thesis, EPFL, 2001.• P. L. Dragott i , Wavelet Footpr ints and Frames for Signal Processing

and Communicat ions, PhD Thesis, EPFL, 2002.• R.Shukla, Rate-distort ion opt imized geometr ical image processing,

PhD Thesis, EPFL, 2004.

ISIT04 84

Papers:• A.Cohen, I .Daubechies, O.Gul ieruz and M.Orchard, On the impor-

tance of combining wavelet-based non- l inear approximat ion wi th cod-ing strategies, IEEE Tr. on IT, 2002

• P. L. Dragott i , M. Vetter l i . Wavelets footpr ints: theory, a lgor i thms and appl icat ions, IEEE Transact ions on Signal Processing, May 2003.

• R. Shukla, P. L. Dragott i , M. N. Do and M. Vetter l i , Rate-distort ion opt imized tree structured compression algor i thms for piecewise smooth images, IEEE Transact ions Image Processing, 2004.

• M. N. Do and M. Vetter l i , Framing pyramids. IEEE Transact ions on Signal Processing, Sept. 2003.

• M. N. Do and M. Vetter l i , Contour lets. in Beyond Wavelets, J. Sto-eckler and G. V. Wel land eds., Academic Press, 2003.

• M.N.Do and M.Vetter l i , Contour lets: A computat ional f ramework for d i rect ional mult i resolut ion image representat ion, submit ted, 2003.

• C.Weidmann, M.Vetter l i , Rate-distort ion behavior of sparse sources, IEEE Tr. on IT, under revis ion.

• M. Vetter l i , P. Marzi l iano and T. Blu, Sampl ing s ignals wi th f in i te rate of innovat ion, IEEE Transact ions on SP, June 2002.

• I . Maravic and M. Vetter l i , Sampl ing and Reconstruct ion of Signals wi th Fini te Rate of Innovat ion in the Presence of Noise, IEEE Trans-act ions on SP, 2004, submit ted.

On Fourier and Wavelets: Representation, Approximation … · ISIT04 1 Cargese, Corsica, July 2004...

Documents

Transcript of On Fourier and Wavelets: Representation, Approximation … · ISIT04 1 Cargese, Corsica, July 2004...