Applied Harmonic Analysis meets Compressed Sensing · PDF fileGoalforToday Challenges for...
Transcript of Applied Harmonic Analysis meets Compressed Sensing · PDF fileGoalforToday Challenges for...
Applied Harmonic Analysis meets Compressed Sensing
Gitta Kutyniok
(Technische Universitat Berlin)
ICERM Program “Network Science and Graph Algorithms”February 4, 2014
Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 1 / 50
Outline
1 Imaging SciencesInspiring Empirical ResultsGoal for Today
2 Review of Compressed SensingSeparation via Compressed SensingInpainting via Compressed Sensing
3 Algorithmic AspectsShearlet SystemsNumerical Results
4 Theoretical AnalysisGeometrically Clustered SparsityAnalysis of Separation (joint with D. L. Donoho)Analysis of Inpainting (joint with E. King and X. Zhuang)
Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 2 / 50
Ill-posed Inverse Problems in Data Analysis
Two Challenges:
Modern Data in general is often composed of two or moremorphologically distinct constituents. Task: Separation of components given the composed data.
Applications often cause loss of information or necessary informationcan not be collected. Task: Recovery of missing data given the observed data.
Novel Approach:
Applied Harmonic Analysis
Compressed Sensing
First empirical results by J. L. Starck, M. Elad, and D. L. Donoho.
Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 3 / 50
Separating Artifacts in Images, I
+
(Source: Starck, Elad, and Donoho; 2006)
Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 4 / 50
Separating Artifacts in Images, II
Neurobiological Imaging:
Detection of characteristics of Alzheimer.
Separation of spines and dendrites.
+
(Source: Brandt, K, Lim, and Sundermann; 2010)
Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 5 / 50
Inpainting, I
(Source: Hennenfent and Herrmann; 2008)
Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 6 / 50
Inpainting, II
(Source: King, K, Lim, Zhuang; 2012)
Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 7 / 50
Applied Harmonic Analysis Approach
Methodology:Exploit a carefully designed representation system (ψλ)λ ⊆ H:
H ⊇ C ∋ f −→ (〈f , ψλ〉)λ −→∑
λ
〈f , ψλ〉ψλ = f
Two Main Goals:
(1) Decomposition
(2) Efficient representations
Main Desiderata:
Multiscale representation system.
Partition of Fourier domain.
Fast decomposition and reconstruction algorithm.
Optimally sparse approximation of the considered class.
Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 8 / 50
Goal for Today
Challenges for Today:
Methodology to derive the empirical results!◮ Applied Harmonic Analysis.◮ Compressed Sensing.
Improvement of the methodology!◮ Shearlets as sparsifying system.
Analysis of the methodology!◮ Continuum model.◮ Geometrically clustered sparsity.
Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 9 / 50
Goal for Today
Challenges for Today:
Methodology to derive the empirical results!◮ Applied Harmonic Analysis.◮ Compressed Sensing.
Improvement of the methodology!◮ Shearlets as sparsifying system.
Analysis of the methodology!◮ Continuum model.◮ Geometrically clustered sparsity.
Another Path to Imaging Science are Variational Approaches:
Contributors: Bertozzi, Burger, Chan, Esedoglu, Kang, Osher, Sapiro,Setzer, Shen, Steidl, Vese, Weikert, ...
Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 9 / 50
How does Compressed Sensing come into play?
Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 10 / 50
Underdetermined Situations
Separation:
Observe a signal x composed of two subsignals x1 and x2:
x = x1 + x2.
Extract the two subsignals x1 and x2 from x , if only x is known.
Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 11 / 50
Underdetermined Situations
Separation:
Observe a signal x composed of two subsignals x1 and x2:
x = x1 + x2.
Extract the two subsignals x1 and x2 from x , if only x is known.
The two components are geometrically different.
Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 11 / 50
Underdetermined Situations
Separation:
Observe a signal x composed of two subsignals x1 and x2:
x = x1 + x2.
Extract the two subsignals x1 and x2 from x , if only x is known.
The two components are geometrically different.
Inpainting:
Given a signalx = xK + xM ∈ HK ⊕HM .
Recover x , if only xK is known.
Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 11 / 50
Underdetermined Situations
Separation:
Observe a signal x composed of two subsignals x1 and x2:
x = x1 + x2.
Extract the two subsignals x1 and x2 from x , if only x is known.
The two components are geometrically different.
Inpainting:
Given a signalx = xK + xM ∈ HK ⊕HM .
Recover x , if only xK is known.
The original signal is sparse within a frame.
Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 11 / 50
Birth of Separation via Compressed Sensing
Composition of Sinusoids and Spikes sampled at n points:
x = x01 + x02 = Φ1c01 +Φ2c
02 = [ Φ1 | Φ2 ]
[
c01c02
]
,
where
x , c01 , and c02 are n × 1.
Φ1 is the n × n-Fourier matrix ((Φ1)t,k = e2πitk/n).
Φ2 is the n × n-Identity matrix.
0 50 100 150 200 250-1
-0.5
0
0.5
1
Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 12 / 50
Sparsity and ℓ1
Assumption: Letting A be an n × N-matrix, n << N, the seeked solutionc0 of x = Ac0 satisfies:
‖c0‖0 = #{i : c0i 6= 0} is ‘small’, i.e., c0 is sparse.
Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 13 / 50
Sparsity and ℓ1
Assumption: Letting A be an n × N-matrix, n << N, the seeked solutionc0 of x = Ac0 satisfies:
‖c0‖0 = #{i : c0i 6= 0} is ‘small’, i.e., c0 is sparse.
Ideal: Solve...(P0) min
c‖c‖0 subject to x = Ac
Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 13 / 50
Sparsity and ℓ1
Assumption: Letting A be an n × N-matrix, n << N, the seeked solutionc0 of x = Ac0 satisfies:
‖c0‖0 = #{i : c0i 6= 0} is ‘small’, i.e., c0 is sparse.
Ideal: Solve...(P0) min
c‖c‖0 subject to x = Ac
Basis Pursuit (Chen, Donoho, Saunders; 1998)
(P1) minc
‖c‖1 subject to x = Ac
Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 13 / 50
Sparsity and ℓ1
Assumption: Letting A be an n × N-matrix, n << N, the seeked solutionc0 of x = Ac0 satisfies:
‖c0‖0 = #{i : c0i 6= 0} is ‘small’, i.e., c0 is sparse.
Ideal: Solve...(P0) min
c‖c‖0 subject to x = Ac
Basis Pursuit (Chen, Donoho, Saunders; 1998)
(P1) minc
‖c‖1 subject to x = Ac
Meta-Result: If the solution c0 is sufficiently sparse, and A is sufficientlyincoherent, then c0 can be recovered from x via (P1).
Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 13 / 50
First Results of Compressed Sensing
Composition of Sinusoids and Spikes sampled at n points:
x = x01 + x02 = Φ1c01 +Φ2c
02 = [ Φ1 | Φ2 ]
[
c01c02
]
.
Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 14 / 50
First Results of Compressed Sensing
Composition of Sinusoids and Spikes sampled at n points:
x = x01 + x02 = Φ1c01 +Φ2c
02 = [ Φ1 | Φ2 ]
[
c01c02
]
.
Theorem (Donoho, Huo; 2001)If #(Sinusoids) + #(Spikes) = ‖(c01 )‖0 + ‖(c02 )‖0 < (1 +
√n)/2, then
(c01 , c02 ) = argmin(‖c1‖1 + ‖c2‖1) subject to x = Φ1c1 +Φ2c2.
Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 14 / 50
First Results of Compressed Sensing
Composition of Sinusoids and Spikes sampled at n points:
x = x01 + x02 = Φ1c01 +Φ2c
02 = [ Φ1 | Φ2 ]
[
c01c02
]
.
Theorem (Donoho, Huo; 2001)If #(Sinusoids) + #(Spikes) = ‖(c01 )‖0 + ‖(c02 )‖0 < (1 +
√n)/2, then
(c01 , c02 ) = argmin(‖c1‖1 + ‖c2‖1) subject to x = Φ1c1 +Φ2c2.
Theorem (Bruckstein, Elad; 2002)(Donoho, Elad; 2003)Let A = (ai)
Ni=1 be an n× N-matrix with normalized columns, n << N,
and let c0 satisfy
‖c0‖0 <1
2
(
1 + µ(A)−1)
,
with coherence µ(A) = maxi 6=j |〈ai , aj〉|. Thenc0 = argmin‖c‖1 subject to x = Ac .
Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 14 / 50
Birth of Inpainting via Compressed Sensing
Main Idea:Let
x0 ∈ H be a signal.
Φ be an ONB (x0 = Φc0).
H = HM ⊕HK with orthogonal projections PM and PK .
ℓ1 Minimization Problem (Elad, Starck, Querre, Donoho; 2005):
c = argmin‖c‖1 subject to PKx0 = PKΦc x = Φc
Theorem (Donoho, Elad; 2003)
=⇒ If ‖c0‖0 < 12(1 + µ(PKΦ)
−1), then x0 = x .
Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 15 / 50
Two Paths
Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 16 / 50
Avalanche of Recent Work
Problem: Solve x = Ac0 with A an n × N-matrix (n < N).
Deterministic World:
Mutual coherence of A = (ak)k .
Bound ‖c0‖0 dependent on µ(A).
Efficiently solve the problem x = Ac0.
Contributors: Bruckstein, Cohen, Dahmen, DeVore, Donoho, Elad,Eldar, Fuchs, Gribonval, Huo, K, Rauhut, Temlyakov, Tropp, ...
Random World:
Restricted isometry constants of a random A = (ak)k .
Bound ‖c0‖0 by n/(2 log(N/n))(1 + o(1)).
Efficiently solve the problem x = Ac0 with high probability.
Contributors: Candes, Cohen, Dahmen, DeVore, Donoho, K,Krahmer, Rauhut, Romberg, Tanner, Tao, Tropp, Ward, ...
Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 17 / 50
Novel Direction for Sparsity
Geometric Sparsity (Donoho, K; 2009):
y = Ax0 with A an n × N-matrix (n < N).
Nonzeros of x0 often◮ arise not in arbitrary patterns,◮ but are rather highly structured.
Interactions between columns of A inill-posed problems
◮ is not arbitrary,◮ but rather geometrically driven.
Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 18 / 50
Novel Direction for Sparsity
Geometric Sparsity (Donoho, K; 2009):
y = Ax0 with A an n × N-matrix (n < N).
Nonzeros of x0 often◮ arise not in arbitrary patterns,◮ but are rather highly structured.
Interactions between columns of A inill-posed problems
◮ is not arbitrary,◮ but rather geometrically driven.
Other results on “structured sparsity”:
Joint sparsity (Fornasier, Rauhut; 2008)
Block sparsity (Eldar, Kuppinger, Bolcskei; 2010)
Fusion frame sparsity (Boufonous, K, Rauhut; 2011)
...
Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 18 / 50
Which Sparsifying System to choose...?
Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 19 / 50
First: Separation
Main Idea:
Two morphologically distinct components:◮ Points◮ Curves
Choose suitable representation systems which provide optimallysparse representations of
◮ pointlike structures −→ Wavelets◮ curvelike structures −→ ???
Minimize the ℓ1 norm of the coefficients.
This forces◮ the pointlike objects into the wavelets part of the expansion◮ the curvelike objects into the ??? part.
Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 20 / 50
Review of 2-D Wavelets
Definition (1D): Let φ ∈ L2(R) be a scaling function and ψ ∈ L2(R) be awavelet. Then the associated wavelet system is defined by
{φ(x −m) : m ∈ Z} ∪ {2j/2 ψ(2jx −m) : j ≥ 0,m ∈ Z}.
Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 21 / 50
Review of 2-D Wavelets
Definition (1D): Let φ ∈ L2(R) be a scaling function and ψ ∈ L2(R) be awavelet. Then the associated wavelet system is defined by
{φ(x −m) : m ∈ Z} ∪ {2j/2 ψ(2jx −m) : j ≥ 0,m ∈ Z}.
Definition (2D): A wavelet system is defined by
{φ(1)(x −m) : m ∈ Z2} ∪ {2jψ(i)(2jx −m) : j ≥ 0,m ∈ Z
2, i = 1, 2, 3},
where ψ(1)(x) = φ(x1)ψ(x2),
φ(1)(x) = φ(x1)φ(x2) and ψ(2)(x) = ψ(x1)φ(x2),
ψ(3)(x) = ψ(x1)ψ(x2).
Theorem: Discrete wavelets provide optimally sparse approximations forfunctions f ∈ L2(R2), which are C 2 apart from point singularities:
‖f − fN‖22 ≍ N−1, N → ∞.
Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 21 / 50
First: Separation
Main Idea:
Two morphologically distinct components:◮ Points◮ Curves
Choose suitable representation systems which provide optimallysparse representations of
◮ pointlike structures −→ Wavelets◮ curvelike structures −→ ???
Minimize the ℓ1 norm of the coefficients.
This forces◮ the pointlike objects into the wavelets part of the expansion◮ the curvelike objects into the ??? part.
Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 22 / 50
Second: Inpainting
Main Idea:
Choose suitable representation system which provide optimallysparse representations of the original image.
Minimize the ℓ1 norm of the coefficients.
This fills in the missing part automatically.
Question: What is a good model for an image?
Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 23 / 50
Second: Inpainting
Main Idea:
Choose suitable representation system which provide optimallysparse representations of the original image.
Minimize the ℓ1 norm of the coefficients.
This fills in the missing part automatically.
Question: What is a good model for an image?
Field et al., 1993
Require: System to sparsify curvelike structures −→ ???
Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 23 / 50
Fitting Model for Anisotropic Structures
Definition (Donoho; 2001):The set of cartoon-like images E2(R2) is defined by
E2(R2) = {f ∈ L2(R2) : f = f0 + f1 · χB},
where B ⊂ [0, 1]2 with ∂B a closed C 2-curve, f0, f1 ∈ C 20 ([0, 1]
2).
Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 24 / 50
Fitting Model for Anisotropic Structures
Definition (Donoho; 2001):The set of cartoon-like images E2(R2) is defined by
E2(R2) = {f ∈ L2(R2) : f = f0 + f1 · χB},
where B ⊂ [0, 1]2 with ∂B a closed C 2-curve, f0, f1 ∈ C 20 ([0, 1]
2).
Theorem (Donoho; 2001):Let (ψλ)λ ⊆ L2(R2). Allowing only polynomial depth search, the optimalasymptotic approximation error of f ∈ E2(R2) is
‖f − fN‖22 ≍ N−2, N → ∞, where fN =∑
λ∈IN
cλψλ.
Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 24 / 50
Beyond Wavelets...
Observation:
Wavelets only achieve ‖f − fN‖22 ≍ N−1, N → ∞.
Wavelets can not approximate curvilinear singularities optimallysparse.
Reason: Isotropic structure of wavelets:
2jψ(
(
2j 00 2j
)
x −m)
Intuitive explanation:
Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 25 / 50
Main Goal in Applied Harmonic Analysis
Design a representation system which...
...is generated by one ‘mother function’,
...provides optimally sparse approximation of cartoons,
...allows for compactly supported analyzing elements,
...is associated with fast decomposition algorithms,
...treats the continuum and digital ‘world’ uniformly.
Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 26 / 50
Main Goal in Applied Harmonic Analysis
Design a representation system which...
...is generated by one ‘mother function’,
...provides optimally sparse approximation of cartoons,
...allows for compactly supported analyzing elements,
...is associated with fast decomposition algorithms,
...treats the continuum and digital ‘world’ uniformly.
Non-exhaustive list of approaches:
Ridgelets (Candes and Donoho; 1999)
Curvelets (Candes and Donoho; 2002)
Contourlets (Do and Vetterli; 2002)
Bandlets (LePennec and Mallat; 2003)
Shearlets (K and Labate; 2006)
Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 26 / 50
Scaling and Orientation
Parabolic scaling:
Aj =
(
2j 0
0 2j/2
)
, j ∈ Z.
Historical remark:
1970’s: Fefferman und Seeger/Sogge/Stein.
Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 27 / 50
Scaling and Orientation
Parabolic scaling:
Aj =
(
2j 0
0 2j/2
)
, j ∈ Z.
Historical remark:
1970’s: Fefferman und Seeger/Sogge/Stein.
Orientation via shearing:
Sk =
(
1 k0 1
)
, k ∈ Z.
Advantage:
Shearing leaves the digital grid Z2 invariant.
Uniform theory for the continuum and digital situation.
Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 27 / 50
(Cone-adapted) Discrete Shearlet Systems
Definition (K, Labate; 2006):The (cone-adapted) discrete shearlet system SH(φ,ψ, ψ) generated byφ ∈ L2(R2) and ψ, ψ ∈ L2(R2) is the set
{φ(· −m) : m ∈ Z2}
∪{23j/4ψ(SkAj · −m) : j ≥ 0, |k | ≤ ⌈2j/2⌉,m ∈ Z2}
∪{23j/4ψ(Sk Aj · −m) : j ≥ 0, |k | ≤ ⌈2j/2⌉,m ∈ Z2}.
Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 28 / 50
(Cone-adapted) Discrete Shearlet Systems
Definition (K, Labate; 2006):The (cone-adapted) discrete shearlet system SH(φ,ψ, ψ) generated byφ ∈ L2(R2) and ψ, ψ ∈ L2(R2) is the set
{φ(· −m) : m ∈ Z2}
∪{23j/4ψ(SkAj · −m) : j ≥ 0, |k | ≤ ⌈2j/2⌉,m ∈ Z2}
∪{23j/4ψ(Sk Aj · −m) : j ≥ 0, |k | ≤ ⌈2j/2⌉,m ∈ Z2}.
General Framework:
Parabolic Molecules (Grohs, K; 2013)
α-Molecules (Grohs, Keiper, K, Schafer; 2014)
Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 28 / 50
Compactly Supported Shearlets
Theorem (Kittipoom, K, Lim; 2012):
Let φ,ψ, ψ ∈ L2(R2) be compactly supported, and let ψ, ˆψ satisfy certaindecay condition. Then SH(φ,ψ, ψ) forms a shearlet frame withcontrollable frame bounds.
Remark: Exemplary class with B/A ≈ 4.
Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 29 / 50
Compactly Supported Shearlets
Theorem (Kittipoom, K, Lim; 2012):
Let φ,ψ, ψ ∈ L2(R2) be compactly supported, and let ψ, ˆψ satisfy certaindecay condition. Then SH(φ,ψ, ψ) forms a shearlet frame withcontrollable frame bounds.
Remark: Exemplary class with B/A ≈ 4.
Theorem (K, Lim; 2011):
Let φ,ψ, ψ ∈ L2(R2) be compactly supported, and let ψ, ˆψ satisfy certaindecay condition. Then SH(φ,ψ, ψ) provides an optimally sparseapproximation of f ∈ E2(R2), i.e.,
‖f − fN‖22 ≤ C · N−2 · (logN)3, N → ∞.
Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 29 / 50
Recent Approaches to Fast Shearlet Transforms
www.ShearLab.org:
Separable Shearlet Transform (Lim; 2009)
Digital Shearlet Transform (K, Shahram, Zhuang; 2011)
2D&3D (parallelized) Shearlet Transform (K, Lim, Reisenhofer; 2013)
Additional Code:
Filter-based implementation (Easley, Labate, Lim; 2009)
Theoretical Approaches:
Adaptive Directional Subdivision Schemes (K, Sauer; 2009)
Shearlet Unitary Extension Principle (Han, K, Shen; 2011)
Gabor Shearlets (Bodmann, K, Zhuang; 2013)
Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 30 / 50
Image Separation: Points + Curves
MCALab 120 (52.74 sec) ShearLab (33.75 sec)
(Source: K, Lim; 2011)
Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 31 / 50
Inpainting
Original Noisy Version (80% missing)
Curvelets (29.95dB, 182.15sec) Shearlets (31.04dB, 85.18sec)
(Source: Lim; 2012)
Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 32 / 50
What is the Fundamental Mathematical Concept
behind the Empirical Success?
Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 33 / 50
General Analysis of Separation
Signal Model:x = x01 + x02 ∈ H
Remarks:
Given two tight frames Φ1, Φ2 (Φi(ΦTi x) = x for all x).
Too many decompositions x = Φ1c1 +Φ2c2.
Use x = Φ1(ΦT1 x1) + Φ2(Φ
T2 x2), where x = x1 + x2.
Norm is placed on analysis rather than synthesis side.
Decomposition Technique:
(x⋆1 , x⋆2 ) = argminx1,x2‖Φ
T1 x1‖1 + ‖ΦT
2 x2‖1 subject to x = x1 + x2
Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 34 / 50
Relative Sparsity and Cluster Coherence
Let Φ1 = (ϕ1,i )i∈I1 and Φ2 = (ϕ2,i )i∈I2.
Definition:
For each i = 1, 2, x0i is relatively sparse in Φi w.r.t. Λi , if
‖1Λc1ΦT1 x
01‖1 + ‖1Λc
2ΦT2 x
02‖1 ≤ δ.
We call Λ1 and Λ2 sets of significant coefficients.
We define cluster coherence for Λ1 by
µc(Λ1) = maxj∈I2
∑
i∈Λ1
|〈ϕ1,i , ϕ2,j〉|.
Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 35 / 50
Central Estimate
Theorem (Donoho, K; 2011):Suppose x01 and x02 are relatively sparse with Λ1 and Λ2 sets of significantcoefficients. Then
‖x⋆1 − x01‖2 + ‖x⋆2 − x02‖2 ≤2δ
1− 2µc,
whereµc = max(µc(Λ1), µc (Λ2)).
δ: Relative sparsity measure.
µc : Cluster coherence.
Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 36 / 50
Continuum Model
Neurobiological Geometric Mixture in 2D:
Point Singularity:
P(x) =P∑
i=1
|x − xi |−3/2
Curvilinear Singularity:
C =
∫
δτ(t)dt, τ a closed C 2-curve.
Observed Signal:f = P + C
Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 37 / 50
Scale-Dependent Decomposition
Observed Object:f = P + C.
Subband Decomposition:Wavelets and shearlets use the same scaling subbands!
fj = Pj + Cj , Pj = P ⋆ Fj and Cj = C ⋆ Fj .
ℓ1-Minimization:
(Wj ,Sj) = argmin‖(〈Wj , ψλ〉)λ‖1 + ‖(〈Sj , ση〉)η‖1 s.t. fj = Wj + Sj
Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 38 / 50
Application of Previous Result
x : Filtered signal fj (= Pj + Cj).Φ1: Wavelets filtered with Fj .
Φ2: Shearlets filtered with Fj .
Λ1: Significant wavelet coefficients of 〈ψλ,Pj 〉.Λ2: Significant shearlet coefficients of 〈ση, Cj 〉.δ: Degree of approximation by significant coefficients.
µc(Λ1), µc(Λ2): Cluster coherence of wavelets-shearlets.
Estimate of error: 2δ1−2µc
.
Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 39 / 50
Application of Previous Result
x : Filtered signal fj (= Pj + Cj).Φ1: Wavelets filtered with Fj .
Φ2: Shearlets filtered with Fj .
Λ1: Significant wavelet coefficients of 〈ψλ,Pj 〉?Λ2: Significant shearlet coefficients of 〈ση, Cj 〉?δ: Degree of approximation by significant coefficients.
µc(Λ1), µc(Λ2): Cluster coherence of wavelets-shearlets.
Estimate of error: 2δ1−2µc
= o(‖Pj‖2 + ‖Cj‖2) as j → ∞.
Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 40 / 50
Microlocal Analysis Heuristics
Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 41 / 50
Microlocal Analysis Heuristics
Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 41 / 50
Asymptotic Separation
Theorem (Donoho, K; 2011)
‖Wj − Pj‖2 + ‖Sj − Cj‖2‖Pj‖2 + ‖Cj‖2
→ 0, j → ∞.
At all sufficiently fine scales, nearly-perfect separation is achieved!
Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 42 / 50
Asymptotic Separation
Theorem (Donoho, K; 2011)
‖Wj − Pj‖2 + ‖Sj − Cj‖2‖Pj‖2 + ‖Cj‖2
→ 0, j → ∞.
At all sufficiently fine scales, nearly-perfect separation is achieved!
Theorem (K; 2013)Using thresholding as separation strategy, we can even prove that
WF (∑
j
Fj ⋆Wj) = WF (P) and WF (∑
j
Fj ⋆ Sj) = WF (C).
Exact separation of the wavefront sets!
Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 42 / 50
General Analysis of Inpainting
Signal Model:x0 = PKx
0 + PMx0 ∈ HK ⊕HM ,
where PK (PM) is the orthogonal projection onto HK (HM).
Remarks:
Given a tight frame Φ.
Redundancy Too many decompositions x = Φc .
Use x = Φ(ΦT x).
Norm is placed on analysis rather than synthesis side.
Inpainting Technique:
x⋆ = argminx‖ΦT x‖1 subject to PKx = PKx0.
Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 43 / 50
Central Estimate
Theorem (King, K, Zhuang; 2013)Suppose x0 is δ-relatively sparse in Φ with Λ a set of significantcoefficients, and HM is the masked subspace. Then
‖x⋆ − x0‖2 ≤2δ
1− 2µc.
δ = δ(x0,Φ,Λ): Relative sparsity measure.
µc = µc(Φ,Λ,HM): Cluster coherence.
Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 44 / 50
Continuum Model
Curvilinear Singularity:
C =
∫ ρ
−ρw(t)δτ(t)dt,
τ : [−1, 1] → R2 a C 2-curve, ρ < 1, and w : [−ρ, ρ] → R
+0 ‘bump’.
Mask:Mh = {(x1, x2) ∈ R
2 : |x1| ≤ h}, h > 0.
Observed Signal:f = 1R2\Mh
· C.Subband Decomposition:
C 7→ Cj = C ⋆ Fj .
Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 45 / 50
Scale-Dependent Decomposition
Asymptotic analysis:
Consider h = hj .
Set fj = 1R2\Mhj· Cj .
Algorithm:
Sj = argmin‖(〈Sj , ση〉)η‖1 s.t. fj = 1R2\Mhj· Sj
Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 46 / 50
Scale-Dependent Decomposition
Asymptotic analysis:
Consider h = hj .
Set fj = 1R2\Mhj· Cj .
Algorithm:
Sj = argmin‖(〈Sj , ση〉)η‖1 s.t. fj = 1R2\Mhj· Sj
Microlocal Analysis Heuristics:
Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 46 / 50
Asymptotic Inpainting
Theorem (King, K, Zhuang; 2013)For hj = o(2−j/2) as j → ∞, shearlet inpainting satisfies
‖Sj − Cj‖2‖Cj‖2
→ 0, j → ∞.
Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 47 / 50
Asymptotic Inpainting
Theorem (King, K, Zhuang; 2013)For hj = o(2−j/2) as j → ∞, shearlet inpainting satisfies
‖Sj − Cj‖2‖Cj‖2
→ 0, j → ∞.
Theorem (King, K, Zhuang; 2013)For hj = o(2−j ) as j → ∞, wavelet inpainting satisfies
‖Wj − Cj‖2‖Cj‖2
→ 0, j → ∞.
Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 47 / 50
Asymptotic Inpainting
Theorem (King, K, Zhuang; 2013)For hj = o(2−j/2) as j → ∞, shearlet inpainting satisfies
‖Sj − Cj‖2‖Cj‖2
→ 0, j → ∞.
Theorem (King, K, Zhuang; 2013)For hj = o(2−j ) as j → ∞, wavelet inpainting satisfies
‖Wj − Cj‖2‖Cj‖2
→ 0, j → ∞.
Theorem (King, K, Zhuang; 2013)“In case of thresholding, if hj = ω(2−j ), wavelets fail.”
Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 47 / 50
Let’s conclude...
Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 48 / 50
What to take Home...?
Compressed Sensing solves underdetermined linear systems ofequations exactly if the solution is sparse and the matrix is incoherent.
(Compactly supported) Shearlets provide optimally sparseapproximations of anisotropic features with a unified treatment of thecontinuum and digital world.
The Geometric Separation Problem and the Inpainting Problem canbe solved by these methodologies.
Asymptotically optimal performance can be theoretically proved.
Key features of our analysis:◮ Continuum model.◮ Geometrically clustered sparsity and cluster coherence.◮ Microlocal analysis viewpoint.
Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 49 / 50
Technische Universität BerlinApplied Functional Analysis Group
THANK YOU!
References available at:
www.math.tu-berlin.de/∼kutyniokCode available at:
www.ShearLab.org
Related Books:Y. Eldar and G. KutyniokCompressed Sensing: Theory and ApplicationsCambridge University Press, 2012.
G. Kutyniok and D. LabateShearlets: Multiscale Analysis for Multivariate DataBirkhauser-Springer, 2012.
Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 50 / 50