Wavelets, Approximation, and Compressionee225b/sp10/handouts/...Wavelets, Approximation, and...

Wavelets, Approximation,and Compression

Martin Vetterli

Over the last decade or so, wavelets have had agrowing impact on signal processing theoryand practice, both because of their unifyingrole and their successes in applications (see

also [42] and [38] in this issue). Filter banks, which lie atthe heart of wavelet-based algorithms, have become stan-dard signal processing operators, used routinely in appli-cations ranging from compression to modems. Thecontributions of wavelets have often been in the subtle in-terplay between discrete-time and continuous-time signalprocessing. The purpose of this article is to look at recentwavelet advances from a signal processing perspective. Inparticular, approximation results are reviewed, and theimplication on compression algorithms is discussed. Newconstructions and open problems are also addressed.

Bases, Approximation, andCompressionFinding a good basis to solve a problemdates at least back to Fourier and his in-vestigation of the heat equation [18].The series proposed by Fourier has sev-eral distinguishing features:� The series is able to represent any fi-nite energy function on an interval.� The basis functions are eigenfunctionsof linear shift invariant systems or, inother words, Fourier series diagonalizelinear, shift invariant operators.

Similarly, the sinc basis used in thesampling theorem is able to representany bandlimited function, and process-ing can be done on samples instead ofthe function itself. In short, a basis ischosen both for its ability to represent an object of interest(for example, good approximation with few coefficients)and for its operational value (for example, diagonalizationof certain operators).

To be more formal, assume we have a space of func-tions S and we wish to represent an element f S∈ . Thespace Scan be, for example, integrable functions on the in-terval [ , ]0 1 with finite square integral

f t dt( ) 2

0

1

∫ < ∞, (1)

which we denote L2 0 1( , ). The first question is then to find abasis forS, that is a set of basis functions{ }ϕ i i I∈ inS such thatany element f S∈ can be written as a linear combination

fi I

i i=∈∑ α ϕ . (2)

The example closest to the heart of signal processingpeople is certainly the expansion of bandlimited functionsin terms of the sinc function. Assume the space of realfunctions bandlimited to [ , ]−π π in Fourier domain andhaving a finite square integral. We denote this space byBL2 ( , )−π π . Then the Shannon sampling theorem saysthat any function in that space can be written as [36], [28]

f t t nn

n( ) ( )= −=−∞

∞

∑ α sinc ,(3)

where α n f n= ( )are the samples of f t( )at the integers, and

sinc( )sin( )

tt

t=

ππ (4)

and its integer shifts are the basis func-tions.

A question of immediate concern isin what sense (3) is actually true. Con-sider � ( )f tN as the approximation usingN terms, with N an odd integer

� ( ) ( )( ) /

( ) /

f t t nNn N

N

n= −=− −

−

∑1 2

1 2

α sinc

and a norm to measure the approxima-tion error. In this paper, we are solely concerned with theL2 norm of the error

f t f t f t f t dtN N( ) � ( ) ( ) � ( )/

− = −

−∞

∞

∫2

2 1 2

.(5)

Then we say that (3) is true in the sense that

SEPTEMBER 2001 IEEE SIGNAL PROCESSING MAGAZINE 591053-5888/01/$10.00©2001IEEE

©20

01IM

AG

ES

TA

TE

Authorized licensed use limited to: Univ of Calif Berkeley. Downloaded on March 29,2010 at 16:22:04 EDT from IEEE Xplore. Restrictions apply.

lim ( ) � ( )N Nf t f t

→ ∞− =

20.

(6)

If we have a set of functions { }ϕ i i I∈ such that all func-tions in S can be approximated arbitrarily precisely as in(6), then we say that the set is complete in S. If further-more the elements are linearly independent, then we saythat { }ϕ i i I∈ is a basis for S. Among all possible bases, or-thogonal bases are particularly desirable. In that case, thebasis functions or basis vectors are all mutually orthogo-nal. Normalizing their norm to 1, or ϕ i i I

21= ∈, , an

orthonormal set of vectors satisfies

ϕ ϕ δi j ij, = , (7)

where the inner product .,. is defined as

ϕ ϕ ϕ ϕi j i jt t dt, ( ) ( )*=−∞

∞

∫ (8)

in continuous time, and as

ϕ ϕ ϕ ϕi j in

jn n, [ ] [ ]*=∈∑

�

(9)

in discrete time. (For discrete-time signals, we will con-sider the space of square summable sequences denoted byl Z2 ( ).) For orthonormal bases, the expansion formula (2)becomes

f fii I

i=∈∑ ϕ ϕ, . (10)

This is a representation formula for f in the orthonormalbasis { }ϕ i i I∈ .(More general formulas, for example inbiorthogonal bases or in frames, are also possible, but arenot needed for our discussion.) Given the representationof f in an orthonormal basis as in (10), its orthogonalprojection into a fixed subspace of dimension N spannedby { }ϕ n n N= −0 1… is denoted � ( )f tN

� ,f fN nn

N

n==

−

∑ ϕ ϕ0

1

.(11)

This is the linear approximation of f by a projectiononto a fixed subspace of dimension N. It is linear be-cause the approximation of α βf g+ is equal to theweighted sum of approximations, α β� �f g+ , as can bereadily verified. But this is only one of many possibleN-term approximations of f . Thus, we are led to the ap-proximation problem: Given objects of interest andspaces in which they are embedded, we wish to know howfast an N-term approximation converges

f t f t NN( ) ( ) ~ ( )−2

fct , (12)

where f tN ( ) stands for an approximation of f t( ) whichinvolves N elements, to be chosen appropriately.

This immediately raises a number of questions. Differ-ent bases can give very different rates of approximation.Then, there are various ways to choose the N terms used inthe approximation. A fixed subset (e.g., the first N terms)

leads to a linear, subspace approximation as in (11).Adaptive schemes, to be discussed later, are nonlinear.Will different choices of the subset lead to different ratesof approximation? Such questions are at the heart of ap-proximation theory and are relevant when choosing a ba-sis and an approximation method for a given signalprocessing problem. For example, denoising in waveletbases has led to interesting results for piecewise smoothsignals precisely because of the superior approximationproperties of wavelets for such signals.

We are now ready to address the last problem we shallconsider, namely the compression problem. This involvesnot only approximation quality, but also descriptioncomplexity. There is a cost associated with describing f N ,and this cost depends on the approximation method.Typically, the coefficient values and their locations needto be described, which involves quantization of the coeffi-cients and indexing their locations.

Calling R the number of bits used in the description,we can define a function � ( )D R as

� ( )D R f f R= −2, (13)

where f R explicitely indicates that R bits are used to ap-proximate f . (At this point, we should introduce an ex-pectation over the class of functions to approximate.Formally, approximating a single function has zero com-plexity in the information theoretic sense. For simplicity,we leave this implicit for the moment.) Of course, f R de-pends on the basis chosen to represent f , and thus � ( )D Rcan have very different behaviors, depending on the basis.While � ( )D R resembles the distortion rate function D R( )defined in information theory, the two functions are notidentical by any means. D R( ) is defined as the infimum ofthe distortion of any coding scheme using R bits [7] (us-ing arbitrary long blocks of samples in discrete time, forexample) while � ( )D R is geared specifically at approxima-tion in bases. In certain cases, � ( )D R can be of the same or-der as D R( ) while in others, the theoretical minimumgiven by D R( ) can be much better. Yet, � ( )D R is usuallyachievable in practice with reasonable complexity, whileD R( )often remains an unachievable limit. (It should alsobe noted that the true distortion-rate function is knownonly in very few cases.)

To wrap up this section, we reviewed three relatedproblems around bases in spaces. The first was the exis-tence of a basis, the second was the approximation powerof a basis, and the third was the compression power. In the

60 IEEE SIGNAL PROCESSING MAGAZINE SEPTEMBER 2001

If a function is piecewisesmooth, with isolateddiscontinuities, then Fourierapproximation is poor becauseof the discontinuities.


sequel, we shall explore the relationship of these problems,in particular for wavelet bases. The review paper [15] alsodiscusses the relationship between harmonic analysis,computational methods, and signal compression.

Approximation Propertiesof Filter Banks and WaveletsAlthough orthogonal filter banks as used in subband cod-ing have been discussed by Usevitch [42], we reviewthem here with an emphasis on their projection and ap-proximation properties.

Orthogonal Filter BanksWhen thinking of filtering, one usually thinks about fre-quency selectivity. For example, an ideal discrete-timelowpass filter with cut-off frequency ω πc < takes any in-put signal and projects it onto the subspace of signalsbandlimited to [ , ]−ω ωc c . Orthogonal discrete-time filterbanks perform a similar projection which we now review.Assume a discrete-time filter with finite impulse responseg n g g g L0 0 0 00 1 1[ ] { [ ], [ ], , [ ]}= −… , L even, and theproperty

g n g n k k0 0 2[ ], [ ]− = δ , (14)

that is, the impulse response is orthogonal to its evenshifts, and g0 2

1= . Denote by G z0 ( ) the z-transform ofthe impulse response g n0 [ ]

G z g n zn

Ln

00

1

0( ) [ ]==

−−∑ ,

(15)

with an associated region of convergence covering thez-plane except the origin. Assume further that g n0 [ ] is alowpass filter, that is, its discrete-time Fourier transformhas most of its energy in the region [ / , / ]−π π2 2 . Thendefine a high-pass filter g n1 [ ] with z-transform G z1 ( ) asfollows:

G z z G zL1

10

1( ) ( )= −− + − . (16)

Three operations have been applied:� z z→ − corresponds to modulation by ( )−1 n , or trans-forming the lowpass into a highpass.� − → − −z z 1 applies time-reversal to the impulse re-sponse.� Multiplication by z L− +1 makes the time-reversed im-pulse response causal.

For example, if g n a b c d0 [ ] { , , , }= , theng n d c b a1 [ ] { , , , }= − − . This special way of obtaining ahighpass from a lowpass, introduced as conjugate quadra-ture filters (QCF) [39], [ 27], has the following properties:

g n g n k k1 1 2[ ], [ ]− = δ , (17)

that is, g n1 [ ] is also orthogonal to its even shifts, and

g n g n k0 1 2 0[ ], [ ]− = (18)

or { [ ], [ ]}g n g n0 1 and their even shifts are mutually or-thogonal. The last step is to show that the orthonormal set

{ [ ], [ ]} ,g n k g n l k l0 12 2− − ∈�(19)

is an orthonormal basis for l2 ( )� , the space of squaresummable sequences. This requires to check complete-ness, which can be done for example by verifyingParseval’s relation [44]. Thus, any sequence from l2 ( )�

can be written as

x n g n k g n lk

k ll

[ ] [ ] [ ]= ⋅ − + ⋅ −∈ ∈∑ ∑

� �

α β0 12 2 , (20)

where

α k g n k x n k= − ∈0 2[ ], [ ] , � (21)

β l g n l x n l= − ∈1 2[ ], [ ] , �. (22)

Now, how do we implement the expansion formula (20)?Simply using a multirate filter bank as depicted in Fig. 1.The analysis filters H z0 ( ) and H z1 ( ) are time-reversedversions of G z0 ( ) and G z1 ( ), or

H z G z G z ii i i( ) ~ ( ) ( ), , .= = =−1 0 1 (23)

It is not hard to verify that the analysis filter bank (theleft part of Fig. 1) computes indeed the coefficients α kand β l (remember that convolution includes time rever-sal) and that the synthesis filter bank (the right part of Fig.1) realizes the reconstruction as in (20).

We shall have a second look at the lowpass analy-sis/synthesis branch, as shown in Fig. 2. Using standardmultirate signal processing analysis, the output of such afiltering-downsampling-upsampling-filtering gives anoutput y n0 [ ] with z-transform

Y z G z H z X z H z X z0 0 0 012

( ) ( ) [ ( ) ( ) ( ) ( )]= ⋅ ⋅ + − − ,(24)

that is, both a filtered version of X z( )and a filtered aliasedversion involving X z( )− . It can be verified that the opera-tor from x n[ ] to y n0 [ ] is linear, self-adjoint, andidempotent, that is, it is an orthogonal projection of theinput space l2 ( )Z onto a subspace given by the span of{ [ ]}g n k k0 2− ∈Z . We call this subspace V1 , for which{ [ ]}g n k k0 2− ∈Z forms an orthonormal basis.

Considering similarly the highpass branch, we see thatit produces a projection of l2 ( )Z onto a subspaceW1 givenby the span of { [ ]}g n l l1 2− ∈Z . The spaces V1 and W1 areorthogonal (see (18)) and their direct sum forms l2 ( )Z , or

l V W2 1 1( )Z = + . (25)

Thus, an orthogonal filter bank splits the input space intoa lowpass approximation space V1 and its (highpass) or-

SEPTEMBER 2001 IEEE SIGNAL PROCESSING MAGAZINE 61


thogonal complementW1 . The spaceV1 corresponds to acoarse approximation, while W1 contains additional de-tails. This is the first step in the multiresolution analysisthat is obtained when iterating the highpass/lowpass divi-sion on the lowpass branch (see Fig. 3).

Discrete-Time Polynomials and Filter BanksSignal processing specialists intuitively think of prob-lems in terms of sinusoidal bases. Approximation theoryspecialists think often in terms of other series, like theTaylor series, and thus, of polynomials as basic buildingblocks. We now look at how polynomials are processedby filter banks.

A discrete-time polynomial signal of degree M is com-posed of a linear combination of monomial signals

p n n m Mm m( ) [ ]= ≤ ≤0 . (26)

We shall now see that such monomial (and thereforepolynomial) signals are eigensignals of certain multirateoperators. We need to consider lowpass filters G z0 ( ) thathave a certain number N >0 of zeroes at z = −1, or ω π=on the unit circle. That is, the filter factors as

G z z R zN0

101( ) ( ) ( )= + − . (27)

Clearly, because of (16), the highpass G z1 ( ) has N zerosat z =1(or ω =0), while H z0 ( )and H z1 ( )have N zeros atz = −1 and 1, respectively because of (23).

Then, monomials up to degree N −1 are eigensignalsof the lowpass branch shown in Fig. 2, that is, filtering byH z G z0 0

1( ) ( )= − , downsampling by 2, upsampling by 2,and finally filtering by G z0 ( ). This result is a consequenceof the Strang-Fix theorem (see for example [41]), and wegive here an intuitive reasoning. Because the highpassbranch has an analysis filter

H z z G z z z R zL L1

10

1 101( ) ( ) ( ) ( )= − = − ⋅ −− − − , (28)

with N zeros at z =1, all polynomials up to degree N −1get zeroed out in this branch. This is a consequence of themoment theorem of the Fourier transform [44], [28].Since the filter bank has the perfect reconstruction prop-erty, these polynomials are therefore perfectly replicatedby the lowpass branch only, showing that they areeigensignals. It is worth pointing out that sinusoids arenot eigensignals of such multirate operators. A conse-quence of this eigensignal property is that we can write

n g n k m Nm

kk

m= ⋅ − ≤=−∞

∞

∑ α( ) [ ]0 2 ,(29)

where α km( ) are appropriate coefficients. This shows that

monomials up to degree N −1 are in the span of{ [ ]}g n k k0 2− ∈Z . (Note that polynomials are not in l2 ( )Z ,but any restriction to an interval is. So (29) is true forwindowed polynomials up to boundary effects.) Con-sider now an iterated filter bank as shown in Fig. 3. AfterJ stages of decomposition and synthesis, the equivalentsynthesis lowpass filter g nJ

0( ) [ ] has z-transform

G z G zJ

j

Jj

0 00

12( ) ( ) ( )=

=

−

∏ ,(30)

as can be verified using standard multirate identities [43],[44].

Consider now what happens if a monomial signal ofdegree smaller than N as given in (27) enters the analysisfilter bank. It gets cancelled in all highpass filters and getsthus reproduced by the lowpass branch alone. In otherwords, linear combinations of g nJ

0( ) [ ]and its shifts by2 J

can reproduce polynomials up to degree N −1, or simi-larly to (29), we can write

n g n lm

lJ lm J J= ⋅ −∑ α ,

( ) ( ) [ ]0 2 . (31)

Therefore, we have seen that discrete-time polynomialslive in the “coarse” approximation space of discrete-timefilter banks, and this up to degree N −1when the lowpassfilter has N zeros at z = −1.

Continuous-Time Polynomials and WaveletsAs is well known, a strong link exists between iterated fil-ter banks and wavelets. For example, filter banks can beused to generate wavelet bases [9], and filter banks can beused to calculate wavelet series [23]. It comes thus as nosurprise that the properties seen in discrete time regard-ing polynomial representation carry over to continuoustime. While these properties are directly related to mo-ment properties of wavelets and thus hold in general, wereview them in the context of wavelets generated from or-thogonal finite impulse response (FIR) filter banks. As-sume again that the lowpass filter has N zeros at ω π= ,and thus, the highpass has N zeros atω =0. From the two


nk x

H1

H0

2

2

y1

y0

2

2

G1

G0

x1

x0

0

nk

x∧

Analysis Synthesis

� 1. Multirate filter bank. Two channel analysis followed by syn-thesis.

nk g n( )∼2 2 g n( ) nk

� 2. Lowpass branch of the two channel filter bank in Fig. 1. Thediscrete-time polynomial nk is an eigensignal of this operator.


scale relation of scaling function and wavelet, we get thatthe Fourier transform of the wavelet can be factored as

( )Ψ Φ( ) /ω ωω= ⋅

1

2 212G e j ,

(32)

where Φ( )ω is the Fourier transform of the scaling func-tion. SinceΦ( )0 1= , we see from (32) that Ψ( )ω has exactlyN zeros at ω =0, or N zero moments following the mo-ment theorem of the Fourier transform. Therefore, theinner product between any wavelet (arbitrary scale andshift) with polynomials up to degree N −1 will be zero.Thus, polynomials can be represented as linear combina-tions of scaling functions alone, or similarly to (31), thereare coefficients γ l

m( ) such that

t t lm

ll

m= −∑ γ ϕ( ) ( ). (33)

An example, using the Daubechies scaling functionbased on a four-tap filter [9], is shown in Fig. 4 over a fi-nite interval. The linear function is perfectly represented,up to boundary effects.

Discontinuities in Filter Bankand Wavelet RepresentationsWhat happens if a signal is discontinuous at some pointt 0 ? We know that Fourier series do not like discontinu-ities, since they destroy uniform convergence. Waveletshave two desirable properties as far as discontinuities areconcerned. First, they focus locally on the discontinuity aswe go to finer and finer scales. That is because of the scal-ing relation of wavelets, where the function set ψ m n t, ( ) isdefined as

ψ ψm nm mt t n m n,

/( ) ( ) ,= − ∈− −2 22 Z, (34)

where m→ −∞ corresponds to fine details. Thus, as mgrows negative, the wavelet becomes “sharper.” If the dis-continuity is isolated, and the surroundings are smooth,all wavelet inner products except the ones at the disconti-nuity will be zero, and around the discontinuity, L −1in-ner products are different from zero when the wavelet hassupport length L.

Second, the magnitude evolution across scales of thenonzero wavelet inner products characterizes the discon-tinuity. This is a well-known characteristic of the continu-ous wavelet transform [25], [10] and holds as well for the

orthonormal wavelet series. For example, for a Dirac,f t t( ) ( ),= δ the wavelet coefficients, given by

c fm n m n, , ,= ψ ,

will evolve as

| |~,/c m n

m2 2− , (35)

for n around zero, as can be seen from (34). For aHeaviside (step) function the behavior is

| |~,/c m n

m2 2 , (36)

and more generally, for a singularity of order k (a singu-larity of order kappears when there are kbounded deriva-tives, and the ( )k +1 th derivative is unbounded. TheHeaviside function has k =0, a continuous function with anondifferentiable point has k =1), the behavior is

c m nk m

,( )~2

12

+ , (37)

for the L −1coefficients around the discontinuity at scalem. Consider now a signal that is mostly smooth, except atisolated points where it has various types of singularities.Clearly, the energy of the wavelet coefficients will be con-centrated around the points of singularity, with a behav-ior indicated by (37). If the signal is polynomial betweenthe singularities, then all other wavelet coefficients will bezero, and the polynomial pieces will be “caught” by thescaling functions. This is shown in Fig. 5.

The above discussion focused on continuous-time sig-nals, so what about discrete-time one? Technically, thereis no concept of continuity for sequences. However, qual-itatively, the same happens as in continuous time. In addi-tion, the sequence can be piecewise polynomial, in whichcase we have a piecewise equivalent to the above discus-sion. In that case, the lowpass branch catches the coarsetrend, while the highpass and bandpass channels are zeroexcept around singular points. At these points, there willbe L −1nonzero coefficients at each scale, and their mag-nitude will be governed as in (37).

To conclude, we have discussed how piecewise poly-nomials or piecewise smooth functions have wavelet ex-pansion coefficients concentrated mostly around singular


H1

H0 H1

H0

H1

H0 H1

H0

2

2

2

22

2

Stage J

Analysis

x

Stage 1

Stage 2

� 3. Iterated filter bank. The lowpass branch gets split repeatedlyto get a discrete-time wavelet transform.

Wavelets are well suited toapproximate one-dimensionalpiecewise smooth signals whennonlinear approximation isallowed.


points, while scaling functions or lowpass filters describethe polynomial or smooth trend.

Linear and NonLinear ApproximationsSo far, we have reviewed the construction of bases, espe-cially of wavelet bases. In that case, we were concerned withan exact representation of any signal or function from agiven space. We turn now to approximate representationsand contrast linear with nonlinear methods. For an in-depthdiscussion of such methods, in particular in the wavelet con-text, we refer to the excellent exposition in [24].

Linear ApproximationAssume a spaceV and an orthonormal basis{ }g n n N∈ forV .Thus, a function f V∈ canbewrittenas a linear combination

f g f gnn N

n=∈∑ , . (38)

The best (in the squared error sense) linear approxima-tion of f in the subspace V , denoted as �f , is given by theorthogonal projection of f onto a fixed subspace ofV (see

(11)). Assume it is a subspace W of dimension M, andspanned by the first M vectors of the basis, orW span g g g g M= −{ , , , , }0 1 2 1… . Then �f is given by

� ,f g f gM nn

M

n==

−

∑0

1

,(39)

and the squared error of the approximation is

� � ,ε = − ==

∞

∑f f g fn M

n2

2 2.

Because the subspace W is fixed, independent of f ,the approximation is linear. This follows from the factthat the approximation of f g+ equals the sum of theirapproximations. An obvious question is to find the bestbasis for linear approximation (in the squared errorsense) given a class of objects. To be specific, we reviewthe classic result on linear approximation of random vec-tor processes. Consider a vector process X X X= [ , ,0 1X X N

T2 1, , ]… − where X is an independent, identically

distributed (iid) process with zero mean ( [ ] )E X i =0 andcovariance

E X X RTX[ ]⋅ = , (40)

where R X is known a priori. Because we are interested inleast squares approximation, we will not need any higherorder statistical characterization. Consider now the linearapproximation in an orthonormal basis{ , , , },g g g M NM0 1 1… − < , that is

� ,X g X gM nn

M

n==

−

∑0

1

.(41)

What is the expected squared error, or �ε M =E X X M−

� 2? Rewrit ing X X M− � as ( � ),X X M−

( � )X X M− , one finds after some manipulation that [20]

X X g XX gm M

N

mT T

m− ==

−

∑� 2 1

,(42)

which leads to the expected squared error

� [ ]ε Mm M

N

mT T

mm M

N

mT

X mg E XX g g R g= = ⋅ ⋅=

−

=

−

∑ ∑1 1

.(43)

The question is now: over all possible orthonormalbases for � N , which will minimize (43) for allM N= −1 1, ,… ? Given that R X is positive semidefinite,the answer is that the best basis is given by the set ofeigenvectors of R X ordered with decreasing eigenvalues.The approximation error is then

�ε λM mm M

N

==

−

∑1

,(44)


3.5

3

2.5

2

1.5

1

0.5

0

−0.5

−10 10 15 20 25 305

� 4. Reproduction of a linear function as a combination of scal-ing functions.

� 5. A piecewise smooth function and its wavelet expansion. Thewavelet coefficients are different from zero only in the vicinityof the discontinuity, while the coarse behavior is representedby the scaling coefficients only.


where λ m is the mth largest eigenvalue of R X . The geo-metric intuition behind this result is that the eigenvectorsare the principal axes of the N-dimensional distributionand the best M-dimensional subspace gathering most“energy” (or expected l2 norm) corresponds to the princi-pal axes with largest eigenvalues (see Fig. 6).

The orthogonal transform given by the eigenvectors ofR X (ordered by decreasing eigenvalues) is of course thewell-known Karhunen-Loève transform (KLT) as ex-plained by Goyal [20], but it is useful to recapitulate itsessential property: it is the best basis for linear subspaceapproximation of a vector process specified by second or-der statistics. Again, the second-order statistics are as-sumed to be known a priori, so the subspace is chosen apriori as well and is fixed.(If the correlation matrix has tobe estimated on the fly, then we have a different, adaptivesituation, which is not a linear approximation. For an ex-ample of such an adaptive KLT, see [22].)

In the case of a jointly Gaussian vector process, astronger result holds, namely, it is also the best “com-pression” transform (when transform coefficients arequantized and entropy coded). See [19]-[21] for a re-view of this result.

Nonlinear Approximation in Orthonormal BasesConsider the same set up as above, but with a different ap-proximation rule. Instead of (38), where the first M coef-ficients in the orthonormal expansion are used, we keepthe M largest coefficients instead. That is, we define anindex set I M of the M largest inner products, or

g f g f m I n Im n m m, ,≥ ∈ ∉for every and . (45)

Then, we define, the best nonlinear approximation as:

~ ,f g f gMn I

n nM

= < >∈∑ ,

(46)

which leads to an approximation error

~ ~ ,ε M nn I

f f g fM

= − = < >∉∑

2

2 2 .

Clearly

~ �ε εM M≤ . (47)

(We could call this adaptive linear approximation oradaptive subspace approximation. However, the com-monly used term is nonlinear approximation. More gen-eral nonlinear schemes could also be considered, but arebeyond the scope of this article.)

Let us show that the simple modification of pickingthe largest rather than the first M inner products createsindeed a nonlinear scheme. Consider two functions f andh. In general, the M largest coefficients will not corre-spond to the same set for f and h, and thus, approximat-

ing f h+ is certainly not the sum of the approximations off and h, which has typically more than M terms.

Linear versus Nonlinear ApproximationNow, the question is if (45) leads to large differences inapproximation quality, and if there are bases where thedifference is larger than in others. Consider the exampleof a piecewise constant function with a discontinuity.While simple, it exposes fundamental differences be-tween linear and nonlinear approximation, as well as be-tween Fourier and wavelet bases.

To be specific, take the interval[ , ]0 1 and use Fourier se-ries or wavelet series to represent a function which is con-stant except for a jump at a random point t 0 distributeduniformly over [ , ]0 1 .

Consider Fourier series first. Because of the disconti-nuity, the Fourier series coefficients decay slowly

α βn n n, ~ /1 , (48)

whereα n andβ n are the cosine and sine coefficients at fre-quency 2πn. Linear approximation by the first M terms(or first M / 2 sines and cosines) leads to an error of theorder

� ~ / ~( )ε MF

n M

nM=

∞

∑ 1 12 .(49)

Picking the M largest coefficients will not change the ap-proximation order for the following reason. While someof theα βn n, coefficients will be small in the early terms ofexpansion (the sine and cosine in the numerator of theFourier series coefficient can sometimes be small), mostof the M largest coefficients are still gathered in the firstkM coefficients, where k is a small integer. So, to first ap-proximation, the set of M largest coefficients has a largeoverlap with the set of the first M coefficients, and thusthe nonlinear approximation error satisfies

~ ~( )ε MF

M1 .

(50)


� 6. Two-dimensional distribution indicated by a scatter plot. Theprincipal axes are indicated, and the best one-dimensional ap-proximation is the axis corresponding to the biggest spread.


Consider now the wavelet expansion using a Haarwavelet, where

Ψ( )/

/tt

t=≤ <

− ≤ <

1 0 1 21 1 2 1

0 else (51)

and the wavelets are defined as in (34) withm = − −0 1 2, , ,…While such a wavelet seems unfairly closeto the function to be represented, let us point out thatthere are still an infinite number of nonzero wavelet coef-ficients unless t 0 is a dyadic rational. In addition, the be-havior we are about to show holds much more generally.

First, recall that there will be a single nonzero waveletcoefficient at each scale m around the discontinuity, andthis coefficient has size

| | ~,/c m n

m2 2 , (52)

where m→ −∞. When doing linear approximation, onetakes coarser scales first (where the larger coefficients lie),and goes to finer scales next. While all coefficients but oneare zero, we are not allowed to adapt. If M J=2 , one cantake J −1 scales (from 0 to − +J 2), at which point theonly nonzero coefficient is of order 2 2− J / . The residualquadratic error is then of order 2 − J and thus

� ~( )ε MW

M1 ,

(53)

just as in the Fourier case. However, if we go to the non-linear approximation scheme, things change dramati-cally. Picking the M largest coefficients allows us to goacross M scales (since there is only one nonzero coeffi-cient per scale) at which point the coefficient is of order2 2− M / . The quadratic error is then

~ ~( )eMW M2 − . (54)

We have gone from 1/ M decay in Fourier (linear andnonlinear) or wavelet linear approximation, to exponen-tial decay with nonlinear approximation in wavelet bases.

What we have just seen holds much more generally. If afunction is piecewise smooth, with isolated discontinuities,then Fourier approximation is poor because of the disconti-nuities. In the wavelet case with a wavelet that has enoughzero moments so that the inner products in the smooth re-gion are small, few wavelet coefficients are sufficient to cap-ture the discontinuities, and nonlinear approximationoutperforms linear schemes by orders of magnitude. Fig. 7shows an example, where it can be seen that nonlinear ap-proximation is vastly superior in the wavelet case, while itmakes little difference in the Fourier case. Also, note thatwith linear approximation, there is little difference betweenFourier and wavelets. For a review of some strong approxi-mation results for wavelet expansions of smooth andpiecewise smooth functions, see [12], [ 24].

Nonlinear Approximationand CompressionSo far, we have considered keeping M elements from abasis, either a fixed set (the first M typically) or an adap-tive set (corresponding to the largest projection). In com-pression, we have first to describe the coefficient set,which has zero cost when this set is fixed (linear approxi-mation) but has a nontrivial cost when it is adaptive (non-linear approximation). In that case, there are( )N

Mpossible

subsets, and the rate to describe the subset is equal to theentropy of the distribution of the subsets, or at most


1

0.5

0

0 100 200 300 400 500 600 700 800 900 1000

1

0.5

0

100 200 300 400 500 600 700 800 900 1000

1

0.5

0100 200 300 400 500 600 700 800 900 1000

1

0.5

0

100 200 300 400 500 600 700 800 900 1000

1

0.5

0

100 200 300 400 500 600 700 800 900 1000

1

0.5

0

100 200 300 400 500 600 700 800 900 1000

0

0

0

0

0

(a)

(b)

� 7. Fourier versus wavelet bases and linear versus nonlinearapproximation. The signal is discrete time of length 1024, andM = 64 coefficients are retained. (a) Fourier case. Original ontop, linear approximation in the middle, and nonlinear approx-imation at the bottom. The MSE is 2.7 and 2.4 respectively. (b)Wavelet case, with six levels of decomposition and a wavelethaving three zero moments. Original on top, linear approxima-tion in the middle, and nonlinear approximation at the bottom.The MSE is 3.5 and 0.01, respectively.


log 2

NM

.

(55)

In addition, it is necessary to quantize and entropycode the various inner products. For complexity reasons,one usually considers only scalar quantization and en-tropy coding of individual coefficients, rather than vectorquantization and vector entropy coding. Using usualhigh rate quantization results [19], [20], spending R bitson a particular coefficient leads to a distortion-ratetrade-off of the form

D R CdR( )= −2 2 , (56)

where Cd depends on the distribution of the coefficient(e.g., σ 2 for a uniform distribution of variance σ 2 or(π σe / )6 2 for a Gaussian random variable of varianceσ 2 ).Using the M-term approximation rates (as for example in(49), (50) or (53), (54)) together with the set descriptioncomplexity (55) and distortion-rate for coefficients (56),one can derive achievable upper bounds for the distor-tion-rate tradeoff of linear and nonlinear approximationschemes.

An Example of Linear versus NonlinearApproximation and CompressionAn example that highlights the fundamental differencebetween linear and nonlinear approximation in compres-sion is as follows. Consider a random vector process ofsize N, where each realization has only one nonzero valueat a random location k, the others being zero, or

X n n k n N[ ] [ ] { , , }= ⋅ − ∈ −α δ 0 1… , (57)

where α is Gaussian N( , )0 2σ and k is an integer uniformlydistributed over{ }0 1…N − . In other words, for each realiza-tion, we pick uniformly a location between 0 and N −1, andat that location, place a Gaussian random variable of meanzero and covariance σ 2 , as shown schematically in Fig. 8.

Consider first the linear approximation problem. Theautocovariance matrix is equal to

RN

IX = ⋅σ 2

,(58)

and therefore, using the KLT approach, the standard ba-sis is already the best basis (not unique) for linearsubspace approximation. Thus, R bits are evenly distrib-uted across N positions, and the overall distortion-ratebehavior is

D R CLR N( ) /= ⋅ ⋅ −σ 2 22 . (59)

Note that this is what the linear, KLT approach wouldtell us to do, even though it is clearly suboptimal. A betterapproach, at least if R is large enough, is to spend log 2 N bits to point to the location that is active in agiven realization, and spend the remaining R N− log 2

R N− log 2 bits on describing the Gaussian random vari-able. This leads to

D R CNLR N( ) ( log )= ⋅ ⋅ − −σ 2 22 2 , (60)

which for R N>> log 2 is clearly superior to (59). For lowrates (R N~ log 2 and below), the situation is more diffi-cult; refer to Weidmann [45] for a thorough analysis.

The lesson from the above example is that the lineartheory is misleading when processes are far from jointlyGaussian. The KLT leads to decorrelation, which meansindependence only in the Gaussian case. And it turns outthat many signals of interest for compression are notjointly Gaussian.

Real Coders Do Nonlinear ApproximationLet us see what “real” coders do. Actually, long beforewavelets and nonlinear approximation theory, compres-sion engineers developed methods to adapt the linear,KLT-based theory to real world signals. Two main ingre-dients are used, namely locality and adaptivity. First, sig-nals are divided into localized subsignals (for exampleusing blocking as in block-based transform coding), andthese subsignals are transformed individually. Then,these “local” transforms are coded adaptively, dependingon their characteristics. This is reminiscent of our previ-ous example: if only a few large coefficients appear but inunknown locations, then it pays to localize them. With-out getting into too much details, Fig. 9 shows a typicalset of discrete cosine transform (DCT) coefficients asused in JPEG and the subsequent quantization, runlength, and entropy coding. With this method, whenthere are few large coefficients, they can be representedwith few bits, and we have indeed a nonlinear approxima-tion scheme in a local Fourier-like basis.

It is worthwhile to point out that any adaptive schemecreates nonlinearity. For example, adaptive bit allocationbetween transform blocks (more bits to high activityblocks, less to low activity ones) is already a nonlinear ap-proach. Also, adaptive best bases [6], [34], dynamic pro-gramming based allocations [29], [34], adaptivesegmentation [31], [32], and matching pursuit [26] areall examples of nonlinear schemes used in compression.

It is worth pointing out that the investigation of therate-distortion behavior of “real” coders is a difficulttopic, both because of the nonlinearity of approximation


∼N(0,1)

Nk

� 8. Pointing process, with uniform choice of a single locationwhere a Gaussian random variable is placed.


and quantization and because the interplay of approxima-tion, quantization and entropy coding is nontrivial. Inparticular, the low rate behavior is tricky. Fig. 10 showsthe logarithm of the squared error of JPEG and of a wave-let-based coder (SPIHT [35]). While the high rate behav-ior is as expected (namely, −6 dB per bit on log scale), thelow rate behavior is different, since the quadratic errorfalls off much more rapidly at very low rates. An intuitiveexplanation is that at very low rates, the classic rate alloca-tion does not hold, since many transform coefficients donot get any rate at all. The few remaining coefficients arethus getting more precision, leading to a faster decay ofthe error. Using a different analysis, Mallat et al. [24]have shown a 1/ R behavior at low rates for transformcodes.

Compression of Piecewise Polynomial SignalsLet us return to one-dimensional piecewise smooth sig-nals. Wavelets are well suited to approximate such signalswhen nonlinear approximation is allowed. To study com-pression behavior, consider the simpler case of piecewisepolynomials, with discontinuities. To make matters easy,let us look again at the signal we used earlier to study non-linear approximation, but this time include quantizationand bit allocation. A simple analysis of the approximaterate distortion behavior of a step function goes as follows.Coefficients decay as 2 2m / , so the number of scales J in-volved, if a quantizer of size � is used, is of the order oflog ( / )2 1 � . The number of bits per coefficient is also ofthe order of log ( / )2 1 � , so the rate R is of the order

R J~ log ( / ) ~22 21 � . (61)

The distortion or squared error is proportional to�2 (foreach coefficient), times the number of scales or, using� = −2 J

D J J~ ⋅ −2 . (62)

Using (61), we get

D R C RWC R( )= ⋅ ⋅ −

1 2 2

(63)

for the distortion-rate behavior of a wavelet scheme. Notethat we ignored the cost of indexing the location. Thiscost turns out to be quite small (order J), because the co-efficients are all gathered around the discontinuity. Formore general signals, Prandoni [31] has shown that non-linear approximation of piecewise polynomials with max-imal degree N, using a wavelet compression where thewavelet has N +1 zero moments, leads to

( )D R C C RW W WC RW( )= ′′ + ′ ⋅ −1 2 (64)

where the constants depend on N and the number of dis-continuities. The disturbing news is the R term in theexponent, which is very far from the expected high rate


−15

−20

−25

−30

−35

−40

−45

−500 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2

� 10. Squared error (on log scale) versus bit rate for JPEG (uppercurve) and SPIHT [35] (lower curve). At high rates, the typical−6 dB/bit is apparent, while at very low rates, a steeper decayis typical.

DC AC(0,1) AC(0,7)

AC(7,0) AC(7,7)

(a)

(b)

� 9. Example of 64 DCT coefficients from an 8 × 8 block andzig-zag scanning. (a) Zig-zag scanning of the 8 × 8 block ofDCT coefficients to obtain a one-dimensional array of coeffi-cients. (b) Example of an array of 64 coefficients (magnitudeonly) with a threshold T (deadzone quantizer) andquantization of the coefficients above T . Subsequently, runlength coding is used to index the large coefficients, and en-tropy coding to represent their value.


behavior. Such behavior has been shown more generallyfor piecewise smooth functions by Cohen et al. [5]. (Notethat this is a much broader class than piecewise polynomi-als.) A direct approach to compression of piecewise poly-nomials, based on an oracle tell ing us wherediscontinuities are, leads to

D R Cp pC Rp( )= ′ ⋅ − ⋅2 , (65)

and such behavior is achievable using dynamic program-ming [31]. Clearly (65) is much superior to (64) at highrates. Now, why is the wavelet compression performingsuboptimally? The intuitive reason is simple: by com-pressing the coefficients across scales independently, one

does not take advantage of the interscale dependencies.Several attempts have been made to use such dependen-cies. For example, zero trees [37] predict lack of energyacross scales, and hidden Markov models [8] indicate de-pendencies across scales. In the case of piecewise polyno-mials with discontinuities, the behavior across scales isactually deterministic, and this can be modeled usingwavelet footprints [16]. Using footprints on a waveletdecomposition of a piecewise polynomial allows one toachieve the best possible behavior given in (65), while us-ing a wavelet transform. Examples of using footprints fordenoising can be seen in Fig. 11.

The Two-Dimensional CaseGiven the good performance of wavelets for piecewisesmooth functions in one dimension, one would hope forgood results in two dimensions as well.(Here, we con-sider separable wavelets in two dimensions, which arecommonly used in practice.) Unfortunately, such is notthe case. In essence, wavelets are good at catchingzero-dimensional singularities, but two-dimensional


0.4

0.2

0

−0.2

−0.4200 400 600 800 1000

0.2

0

−0.2

−0.4200 400 600 800 1000

0.4

0.4

0.2

0

−0.2

−0.4200 400 600 800 1000

0.2

0

−0.2

−0.4200 400 600 800 1000

0.4

(a) (b)

(c) (d)

� 11. Denoising in wavelet domain using thresholding and footprints. (a) Original. (b) Noisy version (SNR = 11.1 dB). (c) Denoising us-ing footprints (SNR = 31.4 dB). (d) Denoising using standard wavelet thresholding (SNR = 20.1 dB).

While a powerful N-termapproximation is desirable, itmust be followed by appropriatecompression.


piecewise smooth signals resembling images have one-di-mensional singularities. That is, smooth regions are sepa-rated by edges but edges themselves are typically smoothcurves. Intuitively, wavelets will be good at isolating thediscontinuity orthogonal to the edge, but will not see thesmoothness along the edge. This can be verified by look-ing at the approximation power of a two-dimensionalwavelet basis. Such a basis is obtained by a tensor-productof one dimensional wavelets. An analysis similar to that inthe “Linear versus Noninear Approximation” Sectionshows that the M-term nonlinear approximation of asimple piecewise constant function with a linear disconti-nuity leads to a quadratic error of the order

~ ~ /ε M M1 . (66)

This disappointing behavior indicates that more pow-erful bases are needed in higher dimensions.

True Two-Dimensional BasesAs the wavelet example shows, separable bases are notsuited for “true” two-dimensional objects. What isneeded are transforms and bases that include some formof “geometry” and that are truly two dimensional. (Thenotion of geometry is not easy to formalize in our con-text, but the intuition is that the dimensions are not inde-pendent, and certain shapes are more likely than others.)

Besides the two-dimensional Fourier and wavelettransforms, which are both separable, the Radon trans-form plays a key role. This transform, studied early in the20th century [33], was rediscovered several times in fieldsranging from astronomy to medical imaging (see [11] foran excellent overview). The Radon transform maps afunction f x y( , ) into RA tf ( , )θ by taking line integrals atangle θ and location t

RA t f x y x y t dxdyf ( , ) ( , ) ( cos sin )θ δ θ θ= + −∫ ∫ . (67)

A key insight to construct directional bases from the Ra-don transform was provided by Candès and Donoho [2],[3] with the ridgelet transform. The idea is to map aone-dimensional singularity, like a straight edge, into apoint singularity using the Radon transform. Then, thewavelet transform can be used to handle the point singu-lar i ty. To develop the intuit ion, consider atwo-dimensional Heaviside-like function which is −1 onthe left of a line δ θ θ( cos sin )x y t0 0 0+ − , and 1 on theright. This is an “infinite” edge function of angle θ0 andlocation t 0 .

Because of the infinite extent, we cannot take the inte-gral in (67) without additional constraints (like applyinga smooth window), but intuitively, for any angle θ θ≠ 0 ,the projection RA tf ( , )θ is smooth in t, while for θ0 , theresult is a one-dimensional Heaviside function in t with asingularity in t 0 . It is thus natural to take a wavelet trans-form along t, leading to the definition of the continuousridgelet transform as [3]

RI a b t RA t dtf a b f( , , ) ( ) ( , ),θ ψ θ=∫ , (68)

where ψ ψa b t a t b a,/( ) ( / )= −−1 2 and ψ( )t is a wavelet

with at least one zero moment. The “atoms” of analysisare infinite ridges at an angle θ, location b and scale a,where the profile of the ridge is given by the wavelet. Onesuch ridgelet is shown in Fig. 12.

The transform in (68) is continuous in all parameters.Appropriate discretization and localization leads to setsof localized directional ridges that can efficiently repre-sent two-dimensional functions with edge-like disconti-nuities (more precisely, one can show that a framerepresentation is obtained [3]).

It is also possible to develop discrete schemes that workdirectly on finite size, sampled data, while emulating theprinciples of the ridgelet transform. Such schemes typicallyimplement a discrete Radon transform in space or Fourierdomain (in the latter case, using the projection slice theo-rem) followed by a discrete wavelet transform. One exam-


0.2

0

−0.2

−0.4

0.4

11

0.6

0.80.6

0.40.2

0 00.2

0.40.6

0.8

x2 x1

� 12. Typical ridgelet, at angle θ with a profile given by ψa b t, ( ).

w1

w2

1

2

3

456

7

8

1

2

3

4 5

6

7

8

� 13. Frequency division of a directional filter bank with 8 bands.


ple is given in [40], where ridgelet frames are constructedfor image denoising. Another example is the constructionof an orthonormal ridgelet basis from the finite Radontransform and the discrete wavelet transform [13].

The important point with ridgelets is that unlikewavelets, they achieve fast N-term approximation of ob-jects with straight edges [3]. For the example of aHeaviside function, one gets an M-term nonlinear ap-proximation with

~ ~ε MM2 − . (69)

Therefore, they can be used as building blocks in morecomplex schemes to approximate objects with smoothedges. One such scheme uses localized ridgelets of appro-priate size along edges (e.g., with fixed or adaptiveblocksizes [40]).

Ridgelets can be combined with multiresolutionschemes to get multiscale ridgelets. Combined withbandpass filtering, this gives rise to curvelets [4].

Directional Filter BanksTo get directional analysis, one can alternatively use di-rectional filter banks [1]. In such a case, the basis func-tions are given by the filter impulse responses and theirtranslates with respect to the subsampling grid. Such fil-ter banks can be designed directly, or through iteration ofelementary filter banks. They lead to bases if criticallysubsampled, or frames if oversampled.

Fig. 13 shows the frequency division achieved by anideal directional filter bank with eight channels. Whenthis is combined with a multiresolution, pyramidal de-composition, one obtains a curvelet-like decomposition[4], [14]. Specifically, a pyramidal decomposition intobandpass channels (see Fig. 14(a)) is followed by a direc-tional analysis of the bandpass channels. The number ofdirections is increased as frequency increases, and the re-sulting frequency split is shown in Fig. 14(b). This systemis called a pyramidal directional filter bank (PDFB). Atest image and its decomposition is shown in Fig. 15,showing how different directions are separated, and thisin a multiresolution manner.

As an example application, a simple denoising algo-rithm (based on thresholding the small magnitude coeffi-cients in the decomposition) is applied on both a waveletdecomposition (this is a standard denoising procedure)and a pyramidal directional filter bank decomposition. Ascan be seen in Fig. 16, the PDFB catches directionalitymore efficiently, producing more pleasing visual resultsand better SNR performance. This indicates the potentialof such nonseparable, directional multiresolution schemes.

Two-Dimensional Bases and CompressionAs we had seen in one dimension, a good N-term approx-imation is not yet a guarantee for good compression.While a powerful N-term approximation is desirable, itmust be followed by appropriate compression. Thus, the

topic of compression of two-dimensional piecewisesmooth functions is still quite open. Several promisingapproaches are currently under investigation, includingcompression in ridgelet and curvelet domain, compres-sion along curves using “bandelets” [30] and generaliza-tion of footprints in two dimensions or edgeprints [17].


(2,2)

Multiscale Dec. Directional Dec.(a)

(b)

� 14. Pyramidal directional filter bank. (a) A standard multiscaledecomposition into octave bands, where the lowpass channelis subsampled while the highpass is not. (b) Resulting fre-quency division, where the number of directions is increasedwith frequency.

(a) (b)

� 15. Pyramidal directional filter bank decomposition. (a) Simpletest image. (b) Decomposition into directional bandpass im-ages (8, 4 and 4) and a lowpass image.


ConclusionThe interplay of representation, approximation, andcompression of signals was reviewed. For piecewisesmooth signals, we showed the power of wavelet-basedmethods, in particular for the one-dimensional case. Fortwo-dimensional signals, where wavelets do not providethe answer for piecewise smooth signals with curve singu-larities, new approaches and open problems were indi-cated. Such approaches rely on new bases with potentiallyhigh impact on image processing, for such problems asdenoising, compression and classification.

AcknowledgmentsThe comments of the reviewers are gratefully acknowl-edged. The author would like to thank P.L. Dragotti andM. Do, both of EPFL, for insightful comments, and forproviding figures on footprints and directional bases, re-spectively. Discussions with many people involved in thiscrossdisciplinary topic have been very helpful, in particu-lar with I. Daubechies, R. DeVore, D. Donoho, S.Mallat, and M. Unser.

Martin Vetterli received his Engineering degree fromEidgenoessische Technische Hochschule Zuerich(ETHZ), his M.S. from Stanford University, and his Doc-torate from Ecole Polytechnique Federale de Lausanne(EPFL). He was with Columbia University, New York,and the University of California at Berkeley before joiningthe Communication Systems Department of the SwissFederal Institute of Technology in Lausanne. His researchinterests include signal processing and communications, inparticular, wavelet theory and applications, image andvideo compression, joint source-channel coding, andself-organized communication systems. He has has wonbest paper awards from EURASIP in 1984 and the IEEESignal Processing Society in 1991 and 1996. He is theco-author, with J. Kova�cevic, of the textbook Wavelets andSubband Coding (Prentice-Hall, 1995).

References[1] R.H. Bamberger and M.J.T. Smith, “A filter bank for the directional de-

composition of images: Theory and design,” IEEE Trans. Signal Processing,vol. 40., pp. 882-893, Apr. 1992.

[2] E. Candès, “Ridgelets: Theory and applications,” Ph.D. dissertation, Dept.Statistics, Stanford University, Stanford, CA, 1998.

[3] E. Candès and D.L. Donoho, “Ridgelets: A key to higher-dimensionalintermittency?,” Phil. Trans. R. Soc. London A., pp. 2495-2509, 1999.

[4] E.J. Candès and D.L. Donoho, “Curvelets—A surprisingly effectivenonadaptive representation for objects with edges,” in Curve and SurfaceFitting, A. Cohen, C. Rabutand, and L.L. Schumaker, Eds. Saint-Malo:Vanderbilt University Press, 1999.

[5] A. Cohen, W. Dahmen, and I. Daubechies, “Tree approximation and opti-mal encoding,” Appl. Computational Harmonic Anal., to be published.

[6] R.R. Coifman and M.V. Wickerhauser, “Entropy-based algorithms for bestbasis selection,” IEEE Trans. Inform. Theory (Special Issue on Wavelet Trans-forms and Multiresolution Signal Analysis), vol. 38, pp. 713-718, Mar. 1992.

[7] T.M. Cover and J.A. Thomas, Elements of Information Theory. New York:Wiley, 1991.

[8] M. Crouse, R.D. Nowak, and R.G. Baraniuk, “Wavelet-based signal pro-cessing using hidden Markov models,” IEEE Trans. Signal Processing (SpecialIssue on Wavelets and Filterbanks), pp. 886-902, Apr. 1998.

[9] I. Daubechies, “Orthonormal bases of compactly supported wavelets,”Commun. Pure Appl. Math., vol. 41, pp. 909-996, Nov. 1988.

[10] I. Daubechies, Ten Lectures on Wavelets. Philadelphia, PA: SIAM, 1992.

[11] S.R. Deans, The Radon Transform and Some of its Applications. New York:Wiley, 1983.

[12] R.A. DeVore, B. Jawerth, and B. J. Lucier, “Image compression throughwavelet transform coding,” IEEE Trans. Inform. Theory (Special Issue onWavelet Transforms and Multiresolution Signal Analysis), vol. 38, pp.719-746, Mar. 1992.

[13] M. Do and M. Vetterli, “Orthonormal finite ridgelet transform for imagecompression,” in Proc. IEEE Int. Conf. Image Processing, ICIP 2000, Van-couver, Canada, Sept. 2000, pp. 367-370.

[14] M. Do and M. Vetterli, “Pyramidal directional filter banks and curvelets,”in Proc. IEEE Int. Conf. Image Processing, ICIP 2001, Patras, Greece, Oct.2001.

[15] D. Donoho, M. Vetterli, R. DeVore, and I Daubechies, “Data compres-sion and harmonic analysis,” IEEE Trans. Inform Theory (Special Issue, Infor-mation Theory: 1948-1998 Commemorative Issue), vol. 44, pp. 2435-2476,Oct. 1998.

[16] P.L. Dragotti and M. Vetterli, “Wavelet transform footprints: Catchingsingularities for compression and denoising,” in Proc. IEEE Int. Conf. ImageProcessing, ICIP 2000, Vancouver, Canada, Sept. 2000, pp. 363-366.

[17] P.L. Dragotti and M. Vetterli, “Footprints and edgeprints for imagedenoising and compression,” in Proc. IEEE Int. Conf. Image Processing, ICIP2001, Patras, Greece, Oct. 2001.

[18] J. Fourier, Théorie Analytique de la Chaleur. Paris, France:Gauthier-Villars, 1888.

[19] A. Gersho and R.M. Gray, Vector Quantization and Signal Compression.Norwell, MA: Kluwer, 1992.

[20] V.K. Goyal, “Theoretical foundations of transform coding,” IEEE SignalProcessing Mag., vol. 18, pp. 9-21, Sept. 2001

[21] V.K. Goyal, Transform Coding. SIAM, to be published.


(a) (b)

� 16. Comparison of threshold-based denoising methods usingwavelets (a) and a pyramidal directional filter bank (b). TheSNR is 13.82 dB and 15.42 dB, respectively.


[22] V.K. Goyal, J. Zhuang, and M. Vetterli, “Transform coding with back-ward adaptive updates,” IEEE Trans. Inform. Theory, to be published.

[23] S. Mallat, “A theory for multiresolution signal decomposition: The wave-let representation,” IEEE Trans. Pattern Recognition Machine Intell., vol. 11,pp. 674-693, July 1989.

[24] S. Mallat, A Wavelet Tour of Signal Processing. San Diego, CA: Academic,1998.

[25] S. Mallat and W.L. Hwang, “Singularity detection and processing withwavelets,” IEEE Trans. Inform. Theory (Special Issue on Wavelet Transformsand Multiresolution Signal Analysis), vol. 38, pp. 617-643, Mar. 1992.

[26] S.G. Mallat and Z. Zhang, “Matching pursuits with time-frequency dictio-naries,” IEEE Trans. Signal Processing (Special Issue on Wavelets and SignalProcessing), vol. 41, pp. 3397-3415, Dec. 1993.

[27] F. Mintzer, “Filters for distortion-free two-band multirate filter banks,”IEEE Trans. Acoust. Speech Signal Processing, vol. 33, pp. 626-630, June1985.

[28] A.V. Oppenheim and R.W. Schafer, Discrete-Time Signal Processing.Englewood Cliffs, NJ: Prentice-Hall, 1989.

[29] A. Ortega, K. Ramchandran, and M. Vetterli, “Optimal trellis-based buf-fered compression and fast approximations,” IEEE Trans. Image Processing,vol. 3, pp. 26-40, Jan. 1994.

[30] E. Le Pennec and S. Mallat, “Image compression with geometric wave-lets,” in Proc. IEEE Int. Conf. Image Processing, ICIP 2000, Vancouver, Can-ada, Sept. 2000, pp. 661-664.

[31] P. Prandoni, “Optimal segmentation techniques for piecewise stationarysignals,” Ph.D. dissertation, EPFL, Communications Systems, June 1999.

[32] P. Prandoni and M. Vetterli, “Approximation and compression ofpiecewise-smooth functions,” Phil. Trans. Roy. Soc. London, vol. 357, p.1760, Sept. 1999.

[33] J. Radon, “Ueber die bestimmung von funktionen durch ihreintegralwerte längst gewisser mannigfaltigkeiten,” Berichte Sächsische

Akademie der Wissenschaften, Leipzig, pp. KI. 69, 262-267, 1917.

[34] K. Ramchandran and M. Vetterli, “Best wavelet packet bases in a rate-dis-tortion sense,” IEEE Trans. Image Processing, vol. 2, pp. 160-175, Apr.1993.

[35] A. Said and W.A. Pearlman, “A new, fast, and efficient image codec basedon set partitioning in hierarchical trees,” IEEE Trans. Circuits Syst. Video,vol. 6, pp. 243-249, June 1996.

[36] C.B. Shannon, “A mathematical theory of communication,” Bell Syst. Tech.J., vol. 27, 1948.

[37] J.M. Shapiro, “Embedded image coding using zerotrees of wavelet coeffi-cients,” IEEE Trans. Signal Processing (Special Issue, Wavelets and Signal Pro-cessing), vol. 41, pp. 3445-3462, Dec. 1993.

[38] T. Skodras, C. Christopoulos, and T. Ebrahimi, “The JPEG 2000 still im-age compression standard,” IEEE Signal Processing Mag., vol. 18, pp. 36-58,Sept. 2001.

[39] M.J.T. Smith and T.P. Barnwell III, “Exact reconstruction for tree-struc-tured subband coders,” IEEE Trans. Acoust., Speech, and Signal Processing,vol. 34, pp. 431-441, June 1986.

[40] J.L. Starck, E. Candès, and D. Donoho, “The curvelet transform for im-age denoising,” IEEE Trans. Image Processing, submitted for publication.

[41] G. Strang and T. Nguyen, Wavelets and Filter Banks. Cambridge, MA:Wellesley-Cambridge, 1996.

[42] B. Usevitch, “Wavelet-based image compression,” IEEE Signal ProcessingMag., vol. 18, pp. 22-35, Sept. 2001.

[43] P.P. Vaidyanathan, Multirate Systems and Filter Banks. Englewood Cliffs,NJ: Prentice-Hall, 1993.

[44] M. Vetterli and J. Kova�cevic, Wavelets and Subband Coding. EnglewoodCliffs, NJ: Prentice-Hall, 1995.

[45] Claudio Weidmann, “Oligoquantization in low-rate lossy source coding,”Ph.D. dissertation, EPFL, Communication Systems, July 2000.



Wavelets, Approximation, and Compressionee225b/sp10/handouts/...Wavelets, Approximation, and...

Documents

Transcript of Wavelets, Approximation, and Compressionee225b/sp10/handouts/...Wavelets, Approximation, and...