Enhanced Compressive Sensing using Iterative Support...
Transcript of Enhanced Compressive Sensing using Iterative Support...
![Page 1: Enhanced Compressive Sensing using Iterative Support …optimization/L1/ISD/Yilun_Defense_Slides.pdfAcknowledgement Introduction Theoretical Results of ISD Support Detection for Fast](https://reader033.fdocuments.in/reader033/viewer/2022050400/5f7e0fd134e29b4317169000/html5/thumbnails/1.jpg)
Acknowledgement Introduction Theoretical Results of ISD Support Detection for Fast Decaying Signals Numerical Experiments Conclusions
Enhanced Compressive Sensing usingIterative Support Detection
Yilun Wang
Department of Computational and Applied MathematicsRice University
06-22-2009
1 / 47
![Page 2: Enhanced Compressive Sensing using Iterative Support …optimization/L1/ISD/Yilun_Defense_Slides.pdfAcknowledgement Introduction Theoretical Results of ISD Support Detection for Fast](https://reader033.fdocuments.in/reader033/viewer/2022050400/5f7e0fd134e29b4317169000/html5/thumbnails/2.jpg)
Acknowledgement Introduction Theoretical Results of ISD Support Detection for Fast Decaying Signals Numerical Experiments Conclusions
Acknowledgement
Contributors:
Wotao Yin
Thesis Advisors:
Wotao Yin and Yin Zhang
Committee Members:
William W. Symes
Kevin F. Kelly (ECE)
2 / 47
![Page 3: Enhanced Compressive Sensing using Iterative Support …optimization/L1/ISD/Yilun_Defense_Slides.pdfAcknowledgement Introduction Theoretical Results of ISD Support Detection for Fast](https://reader033.fdocuments.in/reader033/viewer/2022050400/5f7e0fd134e29b4317169000/html5/thumbnails/3.jpg)
Acknowledgement Introduction Theoretical Results of ISD Support Detection for Fast Decaying Signals Numerical Experiments Conclusions
Outline
1 IntroductionReview: CS Reconstruction AlgorithmsAn Iterative Support Detection Algorithm (ISD)
2 Theoretical Results of ISDThe Truncated Null Space PropertyRecoverability ImprovementReconstruction Error BoundsAlgorithmic Convergence Behavior
3 Support Detection for Fast Decaying Signals
4 Numerical ExperimentsReview of Compared AlgorithmsExperiment SetupNumerical Results
5 Conclusions
3 / 47
![Page 4: Enhanced Compressive Sensing using Iterative Support …optimization/L1/ISD/Yilun_Defense_Slides.pdfAcknowledgement Introduction Theoretical Results of ISD Support Detection for Fast](https://reader033.fdocuments.in/reader033/viewer/2022050400/5f7e0fd134e29b4317169000/html5/thumbnails/4.jpg)
Acknowledgement Introduction Theoretical Results of ISD Support Detection for Fast Decaying Signals Numerical Experiments ConclusionsReview: CS Reconstruction Algorithms An Iterative Support Detection Algorithm (ISD)
Outline
1 IntroductionReview: CS Reconstruction AlgorithmsAn Iterative Support Detection Algorithm (ISD)
2 Theoretical Results of ISDThe Truncated Null Space PropertyRecoverability ImprovementReconstruction Error BoundsAlgorithmic Convergence Behavior
3 Support Detection for Fast Decaying Signals
4 Numerical ExperimentsReview of Compared AlgorithmsExperiment SetupNumerical Results
5 Conclusions
4 / 47
![Page 5: Enhanced Compressive Sensing using Iterative Support …optimization/L1/ISD/Yilun_Defense_Slides.pdfAcknowledgement Introduction Theoretical Results of ISD Support Detection for Fast](https://reader033.fdocuments.in/reader033/viewer/2022050400/5f7e0fd134e29b4317169000/html5/thumbnails/5.jpg)
Acknowledgement Introduction Theoretical Results of ISD Support Detection for Fast Decaying Signals Numerical Experiments ConclusionsReview: CS Reconstruction Algorithms An Iterative Support Detection Algorithm (ISD)
Compressive Sensing
Assumption: x̄ ∈ Rn: unknown sparse signal, has ≤ k nonzeros(k � n).
Objective: reconstruct x̄ from m linear measurements (m� n)
b = Ax̄ , where A ∈ Rm×n.
The ideal reconstruction model: `0 minimization.
min ‖x‖0 s.t . Ax = b
Advantage: requires fewest measurements: O(k).Disadvantage: computational cost is prohibitive.
5 / 47
![Page 6: Enhanced Compressive Sensing using Iterative Support …optimization/L1/ISD/Yilun_Defense_Slides.pdfAcknowledgement Introduction Theoretical Results of ISD Support Detection for Fast](https://reader033.fdocuments.in/reader033/viewer/2022050400/5f7e0fd134e29b4317169000/html5/thumbnails/6.jpg)
Acknowledgement Introduction Theoretical Results of ISD Support Detection for Fast Decaying Signals Numerical Experiments ConclusionsReview: CS Reconstruction Algorithms An Iterative Support Detection Algorithm (ISD)
Practical Alternatives
Reasonably fast, but requires at least O(k log(n)) measurements
Basis pursuit methods (BP)Greedy methods
Reasonably fast, requires slightly less than O(k log(n))(empirically) measurements
Smooth `0 algorithm (SL0)
Slower, but requires much less than O(k log(n)) measurementsIterative Reweighted `1 Minimization (IRL1)Iterative Reweighted Least Squares Method (IRLS) (smallest m)
The faster the better; the less measurements.
6 / 47
![Page 7: Enhanced Compressive Sensing using Iterative Support …optimization/L1/ISD/Yilun_Defense_Slides.pdfAcknowledgement Introduction Theoretical Results of ISD Support Detection for Fast](https://reader033.fdocuments.in/reader033/viewer/2022050400/5f7e0fd134e29b4317169000/html5/thumbnails/7.jpg)
Acknowledgement Introduction Theoretical Results of ISD Support Detection for Fast Decaying Signals Numerical Experiments ConclusionsReview: CS Reconstruction Algorithms An Iterative Support Detection Algorithm (ISD)
Advantages of the New Algorithm
BP: replace the `0 norm by the `1 norm.
(BP) min ‖x‖1 s.t . Ax = b
ISD, the new algorithm, is a variant of BP.As fast as BP.Requires as few measurements as IRLS.
7 / 47
![Page 8: Enhanced Compressive Sensing using Iterative Support …optimization/L1/ISD/Yilun_Defense_Slides.pdfAcknowledgement Introduction Theoretical Results of ISD Support Detection for Fast](https://reader033.fdocuments.in/reader033/viewer/2022050400/5f7e0fd134e29b4317169000/html5/thumbnails/8.jpg)
Acknowledgement Introduction Theoretical Results of ISD Support Detection for Fast Decaying Signals Numerical Experiments ConclusionsReview: CS Reconstruction Algorithms An Iterative Support Detection Algorithm (ISD)
Algorithmic Framework
ISD (iterative support detection algorithm)
Input: A and b; output: x∗
1 Set the iteration counter k ← 0 and initialize the set of detected entries I(0) ← ∅;
2 While the stopping condition is not met, do
1 T (k) ← (I(k))C = {1, 2, . . . , n} \ I(k);2 x (k) ← solve a truncated `1 minimization problem:
x (k) = arg min ‖xT (k)‖1 s.t . Ax = b.
3 I(k+1) ← detect partial support using x (k) as the reference; e.g.,
I(k+1) = {i : |(x (k))i | > ε(k)}
where ε(k) = ‖x (k)‖∞/5k+1, for example.4 k ← k + 1.
3 x∗ ← x (k).
Note that a BP problem is solved in the first iteration
8 / 47
![Page 9: Enhanced Compressive Sensing using Iterative Support …optimization/L1/ISD/Yilun_Defense_Slides.pdfAcknowledgement Introduction Theoretical Results of ISD Support Detection for Fast](https://reader033.fdocuments.in/reader033/viewer/2022050400/5f7e0fd134e29b4317169000/html5/thumbnails/9.jpg)
Acknowledgement Introduction Theoretical Results of ISD Support Detection for Fast Decaying Signals Numerical Experiments ConclusionsReview: CS Reconstruction Algorithms An Iterative Support Detection Algorithm (ISD)
A Demo (1)
Test data generation:x̄ : sparse Gaussian signal with length n = 200 and sparsity levelk = 25.Number of measurements m = 60.Gaussian matrix A ∈ Rm×n.
m is too small for BP to reconstruct x̄ successfully.
9 / 47
![Page 10: Enhanced Compressive Sensing using Iterative Support …optimization/L1/ISD/Yilun_Defense_Slides.pdfAcknowledgement Introduction Theoretical Results of ISD Support Detection for Fast](https://reader033.fdocuments.in/reader033/viewer/2022050400/5f7e0fd134e29b4317169000/html5/thumbnails/10.jpg)
Acknowledgement Introduction Theoretical Results of ISD Support Detection for Fast Decaying Signals Numerical Experiments ConclusionsReview: CS Reconstruction Algorithms An Iterative Support Detection Algorithm (ISD)
A Demo (2)
0 50 100 150 200−2
−1.5
−1
−0.5
0
0.5
1
1.5
20−th iter. (total,det,c−det,w−det)=(25,18,14,4), Err = 4.38e−001
true signaltrue nonzerofalse nonzero
Reconstruction by BP.Reconstruction error is large.14 true nonzeros are detected.
10 / 47
![Page 11: Enhanced Compressive Sensing using Iterative Support …optimization/L1/ISD/Yilun_Defense_Slides.pdfAcknowledgement Introduction Theoretical Results of ISD Support Detection for Fast](https://reader033.fdocuments.in/reader033/viewer/2022050400/5f7e0fd134e29b4317169000/html5/thumbnails/11.jpg)
Acknowledgement Introduction Theoretical Results of ISD Support Detection for Fast Decaying Signals Numerical Experiments ConclusionsReview: CS Reconstruction Algorithms An Iterative Support Detection Algorithm (ISD)
A Demo (3)
0 50 100 150 200−2
−1.5
−1
−0.5
0
0.5
1
1.5
21−st iter. (total,det,c−det,w−det)=(25,27,21,6), Err = 1.69e−001
true signaltrue nonzerofalse nonzero
Reconstruction by the truncated `1 minimization.Reconstruction error is smaller.21 true nonzeros are detected.
11 / 47
![Page 12: Enhanced Compressive Sensing using Iterative Support …optimization/L1/ISD/Yilun_Defense_Slides.pdfAcknowledgement Introduction Theoretical Results of ISD Support Detection for Fast](https://reader033.fdocuments.in/reader033/viewer/2022050400/5f7e0fd134e29b4317169000/html5/thumbnails/12.jpg)
Acknowledgement Introduction Theoretical Results of ISD Support Detection for Fast Decaying Signals Numerical Experiments ConclusionsReview: CS Reconstruction Algorithms An Iterative Support Detection Algorithm (ISD)
A Demo (4)
0 50 100 150 200−2
−1.5
−1
−0.5
0
0.5
1
1.5
22−nd iter. (total,det,c−det,w−det)=(25,25,25,0), Err = 1.83e−015
true signaltrue nonzerofalse nonzero
Reconstruction by the truncated `1 minimization.Reconstruction error is very small.25 true nonzeros are all detected.
12 / 47
![Page 13: Enhanced Compressive Sensing using Iterative Support …optimization/L1/ISD/Yilun_Defense_Slides.pdfAcknowledgement Introduction Theoretical Results of ISD Support Detection for Fast](https://reader033.fdocuments.in/reader033/viewer/2022050400/5f7e0fd134e29b4317169000/html5/thumbnails/13.jpg)
Acknowledgement Introduction Theoretical Results of ISD Support Detection for Fast Decaying Signals Numerical Experiments ConclusionsReview: CS Reconstruction Algorithms An Iterative Support Detection Algorithm (ISD)
A Demo (5)
0 50 100 150 200−2
−1.5
−1
−0.5
0
0.5
1
1.5
23−rd iter. (total,det,c−det,w−det)=(25,25,25,0), Err = 9.16e−016
true signaltrue nonzerofalse nonzero
Reconstruction by the truncated `1 minimization.Exact reconstruction.
13 / 47
![Page 14: Enhanced Compressive Sensing using Iterative Support …optimization/L1/ISD/Yilun_Defense_Slides.pdfAcknowledgement Introduction Theoretical Results of ISD Support Detection for Fast](https://reader033.fdocuments.in/reader033/viewer/2022050400/5f7e0fd134e29b4317169000/html5/thumbnails/14.jpg)
Acknowledgement Introduction Theoretical Results of ISD Support Detection for Fast Decaying Signals Numerical Experiments ConclusionsThe Truncated Null Space Property Recoverability Improvement Reconstruction Error Bounds Algorithmic Convergence Behavior
Outline
1 IntroductionReview: CS Reconstruction AlgorithmsAn Iterative Support Detection Algorithm (ISD)
2 Theoretical Results of ISDThe Truncated Null Space PropertyRecoverability ImprovementReconstruction Error BoundsAlgorithmic Convergence Behavior
3 Support Detection for Fast Decaying Signals
4 Numerical ExperimentsReview of Compared AlgorithmsExperiment SetupNumerical Results
5 Conclusions
14 / 47
![Page 15: Enhanced Compressive Sensing using Iterative Support …optimization/L1/ISD/Yilun_Defense_Slides.pdfAcknowledgement Introduction Theoretical Results of ISD Support Detection for Fast](https://reader033.fdocuments.in/reader033/viewer/2022050400/5f7e0fd134e29b4317169000/html5/thumbnails/15.jpg)
Acknowledgement Introduction Theoretical Results of ISD Support Detection for Fast Decaying Signals Numerical Experiments ConclusionsThe Truncated Null Space Property Recoverability Improvement Reconstruction Error Bounds Algorithmic Convergence Behavior
Definition of the Truncated Null Space Property
A ∈ Rm×n has the Truncated Null Space Property (t-NSP) withorder L, γ > 0 and 0 < t ≤ n, if
‖vS‖1 ≤ γ‖vT\S‖1,
for each T ⊂ {1, . . . , n} with |T | = t and each index set S ⊂ Twith |S| ≤ L and each v ∈ N (A).
t=n: reduces to Null Space Property.
15 / 47
![Page 16: Enhanced Compressive Sensing using Iterative Support …optimization/L1/ISD/Yilun_Defense_Slides.pdfAcknowledgement Introduction Theoretical Results of ISD Support Detection for Fast](https://reader033.fdocuments.in/reader033/viewer/2022050400/5f7e0fd134e29b4317169000/html5/thumbnails/16.jpg)
Acknowledgement Introduction Theoretical Results of ISD Support Detection for Fast Decaying Signals Numerical Experiments ConclusionsThe Truncated Null Space Property Recoverability Improvement Reconstruction Error Bounds Algorithmic Convergence Behavior
Example of the Truncated Null Space Property
Let t = 4, L = 2, v =(
0.1 0.5 0.2 1.5 2.4)T
,
T =[
2 3 4 5]
and S =[
2 4]⊂ T and
T \ S =[
3 5].
2 = ‖vS‖1 ≤ γ‖vT\S‖1 = 2.6γ
So γ ≥ 2/2.6.
t-NSP represented by (t , L, γ̄), where γ̄ is the minimum of allfeasible γ for all v ∈ N (A).
γ̄ is monotonic in L for any t fixed.
16 / 47
![Page 17: Enhanced Compressive Sensing using Iterative Support …optimization/L1/ISD/Yilun_Defense_Slides.pdfAcknowledgement Introduction Theoretical Results of ISD Support Detection for Fast](https://reader033.fdocuments.in/reader033/viewer/2022050400/5f7e0fd134e29b4317169000/html5/thumbnails/17.jpg)
Acknowledgement Introduction Theoretical Results of ISD Support Detection for Fast Decaying Signals Numerical Experiments ConclusionsThe Truncated Null Space Property Recoverability Improvement Reconstruction Error Bounds Algorithmic Convergence Behavior
Sufficient Condition For a Single Truncated `1
Minimization (1)
Theorem
For given T, suppose that A has t-NSP for (|T |, L, γ̄ < 1). For anygiven x̄ ∈ Rn, it is the unique minimizer of
min{‖xT‖1 : Ax = Ax̄}
if ‖x̄T‖0 ≤ L.
T = Zn: classic result for BP.
17 / 47
![Page 18: Enhanced Compressive Sensing using Iterative Support …optimization/L1/ISD/Yilun_Defense_Slides.pdfAcknowledgement Introduction Theoretical Results of ISD Support Detection for Fast](https://reader033.fdocuments.in/reader033/viewer/2022050400/5f7e0fd134e29b4317169000/html5/thumbnails/18.jpg)
Acknowledgement Introduction Theoretical Results of ISD Support Detection for Fast Decaying Signals Numerical Experiments ConclusionsThe Truncated Null Space Property Recoverability Improvement Reconstruction Error Bounds Algorithmic Convergence Behavior
Sufficient Condition For a Single Truncated `1
Minimization (2)
In particular, Gaussian measurement matrices have the t-NSP.
‖x̄‖0 < c m1+log n
m(BP)
‖x̄‖0 < dc + c m−d
1+log (n−d)(m−d)
(Truncated `1 Minimization)
d = dc + dw .
dc : number of correct detections.dw : number of wrong detections.
Objective:dc + c m−d
1+log (n−d)(m−d)
> c m1+log n
m
18 / 47
![Page 19: Enhanced Compressive Sensing using Iterative Support …optimization/L1/ISD/Yilun_Defense_Slides.pdfAcknowledgement Introduction Theoretical Results of ISD Support Detection for Fast](https://reader033.fdocuments.in/reader033/viewer/2022050400/5f7e0fd134e29b4317169000/html5/thumbnails/19.jpg)
Acknowledgement Introduction Theoretical Results of ISD Support Detection for Fast Decaying Signals Numerical Experiments ConclusionsThe Truncated Null Space Property Recoverability Improvement Reconstruction Error Bounds Algorithmic Convergence Behavior
The Truncated `1 Minimization Over BP
Define k(d) = c m−d1+log (n−d)
(m−d)
Objective:
dc + k(0) > k(d)
⇔ dc >
∫ d
0|k ′(d)| (1)
(1) is reasonably likely to reach:|k ′(d)| < 1, and often smaller than 2/5 in practice.
19 / 47
![Page 20: Enhanced Compressive Sensing using Iterative Support …optimization/L1/ISD/Yilun_Defense_Slides.pdfAcknowledgement Introduction Theoretical Results of ISD Support Detection for Fast](https://reader033.fdocuments.in/reader033/viewer/2022050400/5f7e0fd134e29b4317169000/html5/thumbnails/20.jpg)
Acknowledgement Introduction Theoretical Results of ISD Support Detection for Fast Decaying Signals Numerical Experiments ConclusionsThe Truncated Null Space Property Recoverability Improvement Reconstruction Error Bounds Algorithmic Convergence Behavior
Reconstruction Error Bounds of the Truncated `1
Minimization
Theorem
For given T, suppose that A has t-NSP for (|T |, L, γ̄ < 1). For anygiven x̄ ∈ Rn with ‖x̄T‖0 > L, x∗ is the minimizer of
min{‖xT‖1 : Ax = Ax̄}.
Then ‖(x∗ − x̄)T‖1 ≤ 2cσL(x̄T )1, where c = 1+γ̄1−γ̄ , σL(x̄T )1 is the `1
error of the best L-term approximation of x̄T .
20 / 47
![Page 21: Enhanced Compressive Sensing using Iterative Support …optimization/L1/ISD/Yilun_Defense_Slides.pdfAcknowledgement Introduction Theoretical Results of ISD Support Detection for Fast](https://reader033.fdocuments.in/reader033/viewer/2022050400/5f7e0fd134e29b4317169000/html5/thumbnails/21.jpg)
Acknowledgement Introduction Theoretical Results of ISD Support Detection for Fast Decaying Signals Numerical Experiments ConclusionsThe Truncated Null Space Property Recoverability Improvement Reconstruction Error Bounds Algorithmic Convergence Behavior
Convergence Behavior of ISD (1)
Assume A has t-NSP (t , L, γ̄)and consider the k -th iteration of ISD.
k -th iteration:
min ‖xT (k)‖1 s.t . Ax = b.
Let t (k) ← |T (k)|, L(k) ← ‖x̄T (k)‖0, how about corresponding γ̄(k)?
If γ̄(k) < 1, succeed in reconstruction.If γ̄(k) ≥ 1, go to next iteration.
Objective: γ̄(1) > γ̄(2) > . . . > γ̄(k) > γ̄(k+1) > . . ., until below 1.
Condition on support detection: ?
21 / 47
![Page 22: Enhanced Compressive Sensing using Iterative Support …optimization/L1/ISD/Yilun_Defense_Slides.pdfAcknowledgement Introduction Theoretical Results of ISD Support Detection for Fast](https://reader033.fdocuments.in/reader033/viewer/2022050400/5f7e0fd134e29b4317169000/html5/thumbnails/22.jpg)
Acknowledgement Introduction Theoretical Results of ISD Support Detection for Fast Decaying Signals Numerical Experiments ConclusionsThe Truncated Null Space Property Recoverability Improvement Reconstruction Error Bounds Algorithmic Convergence Behavior
Convergence Behavior of ISD (2)
Theorem
Suppose that A has t-NSP for (t1, L1, γ̄1) as well as (t2, L2, γ̄2) witht2 < t1 and L2 < L1. If (L1 − L2) > γ̄1(t1 − t2 − (L1 − L2)), thenγ̄2 < γ̄1.
Comments:(L1 − L2) represents the number of increased correct detectionsand (t1 − t2 − (L1 − L2)) represents the number of increasedwrong detections.
Number of increased correct detections should be larger thannumber of increased wrong detections by a certain amount.
22 / 47
![Page 23: Enhanced Compressive Sensing using Iterative Support …optimization/L1/ISD/Yilun_Defense_Slides.pdfAcknowledgement Introduction Theoretical Results of ISD Support Detection for Fast](https://reader033.fdocuments.in/reader033/viewer/2022050400/5f7e0fd134e29b4317169000/html5/thumbnails/23.jpg)
Acknowledgement Introduction Theoretical Results of ISD Support Detection for Fast Decaying Signals Numerical Experiments ConclusionsThe Truncated Null Space Property Recoverability Improvement Reconstruction Error Bounds Algorithmic Convergence Behavior
Summary
The theoretical properties of the truncated `1 minimization arestudied.
These theoretical results indicate an iterative support detectionalgorithm is likely to work, given effective support detection.
These theoretical results do not rely on the concreteimplementations of support detection.
23 / 47
![Page 24: Enhanced Compressive Sensing using Iterative Support …optimization/L1/ISD/Yilun_Defense_Slides.pdfAcknowledgement Introduction Theoretical Results of ISD Support Detection for Fast](https://reader033.fdocuments.in/reader033/viewer/2022050400/5f7e0fd134e29b4317169000/html5/thumbnails/24.jpg)
Acknowledgement Introduction Theoretical Results of ISD Support Detection for Fast Decaying Signals Numerical Experiments Conclusions
Outline
1 IntroductionReview: CS Reconstruction AlgorithmsAn Iterative Support Detection Algorithm (ISD)
2 Theoretical Results of ISDThe Truncated Null Space PropertyRecoverability ImprovementReconstruction Error BoundsAlgorithmic Convergence Behavior
3 Support Detection for Fast Decaying Signals
4 Numerical ExperimentsReview of Compared AlgorithmsExperiment SetupNumerical Results
5 Conclusions
24 / 47
![Page 25: Enhanced Compressive Sensing using Iterative Support …optimization/L1/ISD/Yilun_Defense_Slides.pdfAcknowledgement Introduction Theoretical Results of ISD Support Detection for Fast](https://reader033.fdocuments.in/reader033/viewer/2022050400/5f7e0fd134e29b4317169000/html5/thumbnails/25.jpg)
Acknowledgement Introduction Theoretical Results of ISD Support Detection for Fast Decaying Signals Numerical Experiments Conclusions
Practical Support Detection Methods
The implementations of practical support detection,
rely on features of different signals.vary for different kinds of signals.
25 / 47
![Page 26: Enhanced Compressive Sensing using Iterative Support …optimization/L1/ISD/Yilun_Defense_Slides.pdfAcknowledgement Introduction Theoretical Results of ISD Support Detection for Fast](https://reader033.fdocuments.in/reader033/viewer/2022050400/5f7e0fd134e29b4317169000/html5/thumbnails/26.jpg)
Acknowledgement Introduction Theoretical Results of ISD Support Detection for Fast Decaying Signals Numerical Experiments Conclusions
Threshold-Based ISD for Fast Decaying Signals
Examples:
True signals: fast decaying signals (e.g. sparse Gaussiansignals)
Threshold based support detection method:
I(k+1) = {i : |x (k)i | > ε(k)}, k = 0, 1, 2, . . . ,
Threshold-based support detection is numerically provedeffective for fast decaying signals.
Large (in magnitude) components of the reconstruction likely toindicated true large nonzeros of the true signal.
26 / 47
![Page 27: Enhanced Compressive Sensing using Iterative Support …optimization/L1/ISD/Yilun_Defense_Slides.pdfAcknowledgement Introduction Theoretical Results of ISD Support Detection for Fast](https://reader033.fdocuments.in/reader033/viewer/2022050400/5f7e0fd134e29b4317169000/html5/thumbnails/27.jpg)
Acknowledgement Introduction Theoretical Results of ISD Support Detection for Fast Decaying Signals Numerical Experiments Conclusions
How to Choose Threshold Value?
For sparse Gaussian signals: look for “first significant jump” in theincreasingly sorted reconstruction
0 50 100 150 200−1.5
−1
−0.5
0
0.5
1
1.5n=200, m=60, k=25, Err = 4.53e−001
true signaltrue nonzerofalse nonzero
(a) Failed reconstruction
0 50 100 150 2000
0.2
0.4
0.6
0.8
1
1.2
1.4n=200, m=60, k=25, Err = 4.53e−001
false nonzerotrue nonzeroadopted threshold vauereference threshold value
(b) Threshold value based on jump de-tection
|x (k)i+1| − |x
ki | > τ (k), where τ (k) =‖x (k)‖∞/m.
27 / 47
![Page 28: Enhanced Compressive Sensing using Iterative Support …optimization/L1/ISD/Yilun_Defense_Slides.pdfAcknowledgement Introduction Theoretical Results of ISD Support Detection for Fast](https://reader033.fdocuments.in/reader033/viewer/2022050400/5f7e0fd134e29b4317169000/html5/thumbnails/28.jpg)
Acknowledgement Introduction Theoretical Results of ISD Support Detection for Fast Decaying Signals Numerical Experiments ConclusionsReview of Compared Algorithms Experiment Setup Numerical Results
Outline
1 IntroductionReview: CS Reconstruction AlgorithmsAn Iterative Support Detection Algorithm (ISD)
2 Theoretical Results of ISDThe Truncated Null Space PropertyRecoverability ImprovementReconstruction Error BoundsAlgorithmic Convergence Behavior
3 Support Detection for Fast Decaying Signals
4 Numerical ExperimentsReview of Compared AlgorithmsExperiment SetupNumerical Results
5 Conclusions
28 / 47
![Page 29: Enhanced Compressive Sensing using Iterative Support …optimization/L1/ISD/Yilun_Defense_Slides.pdfAcknowledgement Introduction Theoretical Results of ISD Support Detection for Fast](https://reader033.fdocuments.in/reader033/viewer/2022050400/5f7e0fd134e29b4317169000/html5/thumbnails/29.jpg)
Acknowledgement Introduction Theoretical Results of ISD Support Detection for Fast Decaying Signals Numerical Experiments ConclusionsReview of Compared Algorithms Experiment Setup Numerical Results
Acceleration of ISD
Loose stopping tolerances of each truncated `1 minimizationduring support detection.
Warm-starting scheme: the output of the current truncated `1minimization as the initial point of the next truncated `1minimization.
29 / 47
![Page 30: Enhanced Compressive Sensing using Iterative Support …optimization/L1/ISD/Yilun_Defense_Slides.pdfAcknowledgement Introduction Theoretical Results of ISD Support Detection for Fast](https://reader033.fdocuments.in/reader033/viewer/2022050400/5f7e0fd134e29b4317169000/html5/thumbnails/30.jpg)
Acknowledgement Introduction Theoretical Results of ISD Support Detection for Fast Decaying Signals Numerical Experiments ConclusionsReview of Compared Algorithms Experiment Setup Numerical Results
Review of Compared Algorithms (1)
Noise-free measurements b = Ax̄
IRL1:
x (k) ← minx{
nX
i=1
w (k)i |xi | : Ax = b},
wherew (k)
i = (|x (k−1)i + η|)−1
,
IRLS:x (k) ← min
x{X
i
w̃ (k)i |xi |2 : Ax = b},
wherex (k) = Qk AT(AQk AT)−1b
Qk is the diagonal matrix with entries 1/w̃ (k)i and the weights are set as
w̃ (k)i = (|x (k−1)
i |2 + ζ)−1
30 / 47
![Page 31: Enhanced Compressive Sensing using Iterative Support …optimization/L1/ISD/Yilun_Defense_Slides.pdfAcknowledgement Introduction Theoretical Results of ISD Support Detection for Fast](https://reader033.fdocuments.in/reader033/viewer/2022050400/5f7e0fd134e29b4317169000/html5/thumbnails/31.jpg)
Acknowledgement Introduction Theoretical Results of ISD Support Detection for Fast Decaying Signals Numerical Experiments ConclusionsReview of Compared Algorithms Experiment Setup Numerical Results
Review of Compared Algorithms (2)
Noisy measurements b = Ax̄ + z, where z ∼ N(0, σ).
L1/L2:
minx‖x‖1 +
12ρ‖b − Ax‖2
2}
ISD:
x (k) ← minx‖x
T (k)‖1 +1
2ρ‖b − Ax‖2
2}
IRL1:
x (k) ← minx{
nX
i=1
w (k)i |xi | +
12ρ‖b − Ax‖2
2},
IRLS:x (k) ← min
x{X
i
w̃ (k)i |xi |2 : Ax = b},
wherex (k) = Qk AT(AQk AT)−1b
31 / 47
![Page 32: Enhanced Compressive Sensing using Iterative Support …optimization/L1/ISD/Yilun_Defense_Slides.pdfAcknowledgement Introduction Theoretical Results of ISD Support Detection for Fast](https://reader033.fdocuments.in/reader033/viewer/2022050400/5f7e0fd134e29b4317169000/html5/thumbnails/32.jpg)
Acknowledgement Introduction Theoretical Results of ISD Support Detection for Fast Decaying Signals Numerical Experiments ConclusionsReview of Compared Algorithms Experiment Setup Numerical Results
Comparison Settings
BP (L1/L2), IRL1, IRLS, and ISD.
BP (L1/L2), each reweighted `1 subproblem of IRL1, eachtruncated `1 subproblem of ISD, are all solved by YALL1package.
The same final stopping tolerance for these 4 algorithms, in orderfor fair comparison.
On the same computing platform: MATLAB on Linux.
32 / 47
![Page 33: Enhanced Compressive Sensing using Iterative Support …optimization/L1/ISD/Yilun_Defense_Slides.pdfAcknowledgement Introduction Theoretical Results of ISD Support Detection for Fast](https://reader033.fdocuments.in/reader033/viewer/2022050400/5f7e0fd134e29b4317169000/html5/thumbnails/33.jpg)
Acknowledgement Introduction Theoretical Results of ISD Support Detection for Fast Decaying Signals Numerical Experiments ConclusionsReview of Compared Algorithms Experiment Setup Numerical Results
Experimental Settings
b = Ax̄ + z, where z ∼ N(0, σ). A were Gaussian matrices generated by randn(m,n) andtransformed to have orthonormal rows.
# Nonzero Entries Noise σ Dimension n Sparsity k Range of m Tests for each m
1 Gaussian 0 600 40 [80:10:220] 100Gaussian 0 3000 100 [200:50:800] 100
2Gaussian 0.0001 2000 100 325 200Gaussian 0.001 2000 100 325 200Gaussian 0.01 2000 100 325 200
3 Bernoulli 0 600 40 [80:10:220] 1004 2D Phantom 0.002 65536 4765 9830 1
Table: Summary of test sets.
33 / 47
![Page 34: Enhanced Compressive Sensing using Iterative Support …optimization/L1/ISD/Yilun_Defense_Slides.pdfAcknowledgement Introduction Theoretical Results of ISD Support Detection for Fast](https://reader033.fdocuments.in/reader033/viewer/2022050400/5f7e0fd134e29b4317169000/html5/thumbnails/34.jpg)
Acknowledgement Introduction Theoretical Results of ISD Support Detection for Fast Decaying Signals Numerical Experiments ConclusionsReview of Compared Algorithms Experiment Setup Numerical Results
Test set 1
80 100 120 140 160 180 200 2200
0.5
1
1.5
2
2.5
m
Run
ning
tim
e (s
)
n=600, k=40, m=80:10:220
BPISDIRLSIRL1
(c) CPU time
200 300 400 500 600 700 8000
10
20
30
40
50
60
70
80
m
Run
ning
tim
e (s
)
n=3000, k=100, m=200:50:800
BPISDIRLSIRL1
(d) CPU time
Figure: Test set 1: Comparisons in CPU time.
34 / 47
![Page 35: Enhanced Compressive Sensing using Iterative Support …optimization/L1/ISD/Yilun_Defense_Slides.pdfAcknowledgement Introduction Theoretical Results of ISD Support Detection for Fast](https://reader033.fdocuments.in/reader033/viewer/2022050400/5f7e0fd134e29b4317169000/html5/thumbnails/35.jpg)
Acknowledgement Introduction Theoretical Results of ISD Support Detection for Fast Decaying Signals Numerical Experiments ConclusionsReview of Compared Algorithms Experiment Setup Numerical Results
Test set 1
80 100 120 140 160 180 200 2200
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
m
Sta
rdar
d V
aria
tion
of r
unni
ng ti
me
(s)
n=600, k=40, m=80:10:220
BPISDIRLSIRL1
(a) Standard variation in CPU time
200 300 400 500 600 700 8000
5
10
15
20
25
30
35
m
Sta
rdar
d va
riatio
n of
run
ning
tim
e (s
)
n=3000, k=100, m=200:50:800
BPISDIRLSIRL1
(b) Standard variation in CPU time
Figure: Test set 1: Comparisons in standard variation of CPU time.
35 / 47
![Page 36: Enhanced Compressive Sensing using Iterative Support …optimization/L1/ISD/Yilun_Defense_Slides.pdfAcknowledgement Introduction Theoretical Results of ISD Support Detection for Fast](https://reader033.fdocuments.in/reader033/viewer/2022050400/5f7e0fd134e29b4317169000/html5/thumbnails/36.jpg)
Acknowledgement Introduction Theoretical Results of ISD Support Detection for Fast Decaying Signals Numerical Experiments ConclusionsReview of Compared Algorithms Experiment Setup Numerical Results
Test set 1
80 100 120 140 160 180 200 2200
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
m
Exa
ct r
econ
stru
ctio
n fr
eque
ncy
n=600, k=40, m=80:10:220
BPISDIRLSIRL1
(a) Exact reconstruction frequency
200 300 400 500 600 700 8000
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
m
Exa
ct r
econ
stru
ctio
n fr
eque
ncy
n=3000, k=100, m=200:50:800
BPISDIRLSIRL1
(b) Exact reconstruction frequencies
Figure: Test set 1: Comparisons in recoverability.
36 / 47
![Page 37: Enhanced Compressive Sensing using Iterative Support …optimization/L1/ISD/Yilun_Defense_Slides.pdfAcknowledgement Introduction Theoretical Results of ISD Support Detection for Fast](https://reader033.fdocuments.in/reader033/viewer/2022050400/5f7e0fd134e29b4317169000/html5/thumbnails/37.jpg)
Acknowledgement Introduction Theoretical Results of ISD Support Detection for Fast Decaying Signals Numerical Experiments ConclusionsReview of Compared Algorithms Experiment Setup Numerical Results
Test set 2
0 50 100 150 2000
5
10
15
20
25
30
35
Trials
Run
ning
tim
e (s
)
n=2000, m=325, k=100
L1/L2ISDIRLSIRL1
(a) CPU time
0 50 100 150 20010
−4
10−3
10−2
10−1
100
Trials
Rel
ativ
e re
cons
truc
tion
erro
rs in
L2−
norm
n=2000, m=325, k=100
L1/L2ISDIRLSIRL1
(b) Reconstruction errors in`2 norm
0 50 100 150 20010
−4
10−3
10−2
10−1
100
101
Trials
Rel
ativ
e re
cons
truc
tion
erro
rs in
L1−
norm
n=2000, m=325, k=100
L1/L2ISDIRLSIRL1
(c) Reconstruction errors in`1 norm
Figure: Test set 2 with σ = 0.0001: Comparisons in CPU time andreconstruction errors
37 / 47
![Page 38: Enhanced Compressive Sensing using Iterative Support …optimization/L1/ISD/Yilun_Defense_Slides.pdfAcknowledgement Introduction Theoretical Results of ISD Support Detection for Fast](https://reader033.fdocuments.in/reader033/viewer/2022050400/5f7e0fd134e29b4317169000/html5/thumbnails/38.jpg)
Acknowledgement Introduction Theoretical Results of ISD Support Detection for Fast Decaying Signals Numerical Experiments ConclusionsReview of Compared Algorithms Experiment Setup Numerical Results
Test set 2
0 50 100 150 2000
5
10
15
20
25
30
Trials
Run
ning
tim
e (s
)
n=2000, m=325, k=100
L1/L2ISDIRLSIRL1
(a) CPU time
0 50 100 150 20010
−3
10−2
10−1
100
Trials
Rel
ativ
e re
cons
truc
tion
erro
rs in
L2−
norm
n=2000, m=325, k=100
L1/L2ISDIRLSIRL1
(b) Reconstruction errors in`2 norm
0 50 100 150 20010
−3
10−2
10−1
100
101
Trials
Rel
ativ
e re
cons
truc
tion
erro
rs in
L1−
norm
n=2000, m=325, k=100
L1/L2ISDIRLSIRL1
(c) Reconstruction errors in`1 norm
Figure: Test set 2 with σ = 0.001: Comparisons in CPU time andreconstruction errors
38 / 47
![Page 39: Enhanced Compressive Sensing using Iterative Support …optimization/L1/ISD/Yilun_Defense_Slides.pdfAcknowledgement Introduction Theoretical Results of ISD Support Detection for Fast](https://reader033.fdocuments.in/reader033/viewer/2022050400/5f7e0fd134e29b4317169000/html5/thumbnails/39.jpg)
Acknowledgement Introduction Theoretical Results of ISD Support Detection for Fast Decaying Signals Numerical Experiments ConclusionsReview of Compared Algorithms Experiment Setup Numerical Results
Test set 2
0 50 100 150 2000
5
10
15
20
25
30
Trials
Run
ning
tim
e (s
)
n=2000, m=325, k=100
L1/L2ISDIRLSIRL1
(a) CPU time
0 50 100 150 20010
−2
10−1
100
Trials
Rel
ativ
e re
cons
truc
tion
erro
rs in
L2−
norm
n=2000, m=325, k=100
L1/L2ISDIRLSIRL1
(b) Reconstruction errors in`2 norm
0 50 100 150 20010
−2
10−1
100
Trials
Rel
ativ
e re
cons
truc
tion
erro
rs in
L1−
norm
n=2000, m=325, k=100
L1/L2ISDIRLSIRL1
(c) Reconstruction errors in`1 norm
Figure: Test set 2 with σ = 0.01: Comparisons in CPU time andreconstruction errors
39 / 47
![Page 40: Enhanced Compressive Sensing using Iterative Support …optimization/L1/ISD/Yilun_Defense_Slides.pdfAcknowledgement Introduction Theoretical Results of ISD Support Detection for Fast](https://reader033.fdocuments.in/reader033/viewer/2022050400/5f7e0fd134e29b4317169000/html5/thumbnails/40.jpg)
Acknowledgement Introduction Theoretical Results of ISD Support Detection for Fast Decaying Signals Numerical Experiments ConclusionsReview of Compared Algorithms Experiment Setup Numerical Results
Test set 3
80 100 120 140 160 180 200 2200
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
m
Exa
ct r
econ
stru
ctio
n fr
eque
ncy
n=600, k=40, m=80:10:220
BPISDIRLSIRL1
Figure: Test set 3 with sparse Bernoulli signals: Comparisons inrecoverability.
Threshold-based support detection does not works for sparse Bernoullisignals.
40 / 47
![Page 41: Enhanced Compressive Sensing using Iterative Support …optimization/L1/ISD/Yilun_Defense_Slides.pdfAcknowledgement Introduction Theoretical Results of ISD Support Detection for Fast](https://reader033.fdocuments.in/reader033/viewer/2022050400/5f7e0fd134e29b4317169000/html5/thumbnails/41.jpg)
Acknowledgement Introduction Theoretical Results of ISD Support Detection for Fast Decaying Signals Numerical Experiments ConclusionsReview of Compared Algorithms Experiment Setup Numerical Results
Test set 4
Original
256 x 256
(a)
Figure: n=65536, m=9830, k=4765.
The true image can be sparsely represented by the 2D Haar wavelets.
41 / 47
![Page 42: Enhanced Compressive Sensing using Iterative Support …optimization/L1/ISD/Yilun_Defense_Slides.pdfAcknowledgement Introduction Theoretical Results of ISD Support Detection for Fast](https://reader033.fdocuments.in/reader033/viewer/2022050400/5f7e0fd134e29b4317169000/html5/thumbnails/42.jpg)
Acknowledgement Introduction Theoretical Results of ISD Support Detection for Fast Decaying Signals Numerical Experiments ConclusionsReview of Compared Algorithms Experiment Setup Numerical Results
Test set 4
BP: SNR=10.67dB, Err=2.54e−01, CPU time=44.02 s
Sample ratio: 15%
(a)
Figure: Reconstruction by L1/L2.
42 / 47
![Page 43: Enhanced Compressive Sensing using Iterative Support …optimization/L1/ISD/Yilun_Defense_Slides.pdfAcknowledgement Introduction Theoretical Results of ISD Support Detection for Fast](https://reader033.fdocuments.in/reader033/viewer/2022050400/5f7e0fd134e29b4317169000/html5/thumbnails/43.jpg)
Acknowledgement Introduction Theoretical Results of ISD Support Detection for Fast Decaying Signals Numerical Experiments ConclusionsReview of Compared Algorithms Experiment Setup Numerical Results
Test set 4
IRL1: SNR=19.08dB, Err=9.64e−02, CPU time=519.10 s
Sample ratio: 15%
(a)
ISD: SNR=37.47dB, Err=1.16e−02, CPU time=173.55 s
Sample ratio: 15%
(b)
Figure: The left plot is the reconstruction by IRL1; the right one is thereconstruction by ISD.
43 / 47
![Page 44: Enhanced Compressive Sensing using Iterative Support …optimization/L1/ISD/Yilun_Defense_Slides.pdfAcknowledgement Introduction Theoretical Results of ISD Support Detection for Fast](https://reader033.fdocuments.in/reader033/viewer/2022050400/5f7e0fd134e29b4317169000/html5/thumbnails/44.jpg)
Acknowledgement Introduction Theoretical Results of ISD Support Detection for Fast Decaying Signals Numerical Experiments Conclusions
Outline
1 IntroductionReview: CS Reconstruction AlgorithmsAn Iterative Support Detection Algorithm (ISD)
2 Theoretical Results of ISDThe Truncated Null Space PropertyRecoverability ImprovementReconstruction Error BoundsAlgorithmic Convergence Behavior
3 Support Detection for Fast Decaying Signals
4 Numerical ExperimentsReview of Compared AlgorithmsExperiment SetupNumerical Results
5 Conclusions
44 / 47
![Page 45: Enhanced Compressive Sensing using Iterative Support …optimization/L1/ISD/Yilun_Defense_Slides.pdfAcknowledgement Introduction Theoretical Results of ISD Support Detection for Fast](https://reader033.fdocuments.in/reader033/viewer/2022050400/5f7e0fd134e29b4317169000/html5/thumbnails/45.jpg)
Acknowledgement Introduction Theoretical Results of ISD Support Detection for Fast Decaying Signals Numerical Experiments Conclusions
Conclusions
Effective support detection improves CS reconstruction from boththeoretical points of view and practical performances.
In particular, iterative thresholding is effective on sparse signalswith fast decaying distribution of nonzero values.
On-going and future work:Other specific support detection methods for 2D images, video, etc.Other priors: TV(x).ISD applied to other reconstruction algorithms such as SL0 andgreedy methods.
45 / 47
![Page 46: Enhanced Compressive Sensing using Iterative Support …optimization/L1/ISD/Yilun_Defense_Slides.pdfAcknowledgement Introduction Theoretical Results of ISD Support Detection for Fast](https://reader033.fdocuments.in/reader033/viewer/2022050400/5f7e0fd134e29b4317169000/html5/thumbnails/46.jpg)
Acknowledgement Introduction Theoretical Results of ISD Support Detection for Fast Decaying Signals Numerical Experiments Conclusions
Figure: From http://blog.mozilla.com/sumo/2008/09/15/the-vision-for-sumo-5/
46 / 47
![Page 47: Enhanced Compressive Sensing using Iterative Support …optimization/L1/ISD/Yilun_Defense_Slides.pdfAcknowledgement Introduction Theoretical Results of ISD Support Detection for Fast](https://reader033.fdocuments.in/reader033/viewer/2022050400/5f7e0fd134e29b4317169000/html5/thumbnails/47.jpg)
Acknowledgement Introduction Theoretical Results of ISD Support Detection for Fast Decaying Signals Numerical Experiments Conclusions
80 100 120 140 160 180 200 2200
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
m
Exa
ct r
econ
stru
ctio
n fr
eque
ncy
n=600, k=40, m=80:10:220
BP2 truncated iterations4 truncated iterations6 truncated iterations8 truncated iterations
(a) Exact reconstruction frequencies
200 300 400 500 600 700 8000
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
m
Exa
ct r
econ
stru
ctio
n fr
eque
ncy
n=3000, k=100, m=200:50:800
BP2 truncated iterations4 truncated iterations6 truncated iterations8 truncated iterations
(b) Exact reconstruction frequencies
Figure: Test set 1: Comparisons in recoverability.
47 / 47