Compressive Sensing in Noise and the Role of Adaptivity
Transcript of Compressive Sensing in Noise and the Role of Adaptivity
![Page 1: Compressive Sensing in Noise and the Role of Adaptivity](https://reader031.fdocuments.in/reader031/viewer/2022022822/621c3d53c1a30d0cda4a0908/html5/thumbnails/1.jpg)
Compressive Sensing in Noise
and the Role of Adaptivity
Mark A. Davenport
Georgia Institute of Technology
School of Electrical and Computer Engineering
![Page 2: Compressive Sensing in Noise and the Role of Adaptivity](https://reader031.fdocuments.in/reader031/viewer/2022022822/621c3d53c1a30d0cda4a0908/html5/thumbnails/2.jpg)
Compressive Sensing in Noise
When (and how well) can we
estimate from the measurements ?
-sparse
![Page 3: Compressive Sensing in Noise and the Role of Adaptivity](https://reader031.fdocuments.in/reader031/viewer/2022022822/621c3d53c1a30d0cda4a0908/html5/thumbnails/3.jpg)
Nonadaptive
Compressive Sensing
![Page 4: Compressive Sensing in Noise and the Role of Adaptivity](https://reader031.fdocuments.in/reader031/viewer/2022022822/621c3d53c1a30d0cda4a0908/html5/thumbnails/4.jpg)
Stable Signal Recovery
Typical (worst-case) guarantee: If satisfies the RIP
Even if is provided by an oracle, the error can
still be as large as .
Given ,
find
![Page 5: Compressive Sensing in Noise and the Role of Adaptivity](https://reader031.fdocuments.in/reader031/viewer/2022022822/621c3d53c1a30d0cda4a0908/html5/thumbnails/5.jpg)
Stable Signal Recovery: Part II
Suppose now that satisfies
In this case our guarantee becomes
Unit-norm rows
![Page 6: Compressive Sensing in Noise and the Role of Adaptivity](https://reader031.fdocuments.in/reader031/viewer/2022022822/621c3d53c1a30d0cda4a0908/html5/thumbnails/6.jpg)
Expected Performance
• Worst-case bounds can be pessimistic
• What about the average error?
– assume is white noise with variance
– for oracle-assisted estimator
– if is Gaussian, then for -minimization
![Page 7: Compressive Sensing in Noise and the Role of Adaptivity](https://reader031.fdocuments.in/reader031/viewer/2022022822/621c3d53c1a30d0cda4a0908/html5/thumbnails/7.jpg)
White Signal Noise
Suppose has orthogonal rows with norm equal to .
If is white noise with variance , then is white noise
with variance .
What if our signal is contaminated with noise?
![Page 8: Compressive Sensing in Noise and the Role of Adaptivity](https://reader031.fdocuments.in/reader031/viewer/2022022822/621c3d53c1a30d0cda4a0908/html5/thumbnails/8.jpg)
White Signal Noise
Suppose has orthogonal rows with norm equal to .
If is white noise with variance , then is white noise
with variance .
What if our signal is contaminated with noise?
3dB loss per octave of subsampling
![Page 9: Compressive Sensing in Noise and the Role of Adaptivity](https://reader031.fdocuments.in/reader031/viewer/2022022822/621c3d53c1a30d0cda4a0908/html5/thumbnails/9.jpg)
Noise Folding
[Davenport, Laska, Treichler, Baraniuk - 2011]
![Page 10: Compressive Sensing in Noise and the Role of Adaptivity](https://reader031.fdocuments.in/reader031/viewer/2022022822/621c3d53c1a30d0cda4a0908/html5/thumbnails/10.jpg)
There exists matrices (with unit-norm rows) such that for
any (sparse) we have
• We are using most of our “sensing power” to sense entries
that aren’t even there!
• Tremendous loss in signal-to-noise ratio (SNR)
• It’s hard to imagine any way to avoid this…
Room For Improvement?
and are almost orthogonal
![Page 11: Compressive Sensing in Noise and the Role of Adaptivity](https://reader031.fdocuments.in/reader031/viewer/2022022822/621c3d53c1a30d0cda4a0908/html5/thumbnails/11.jpg)
Can We Do Better?
Via a better choice of ? Via a better recovery algorithm?
[Candès and Davenport - 2011]
If with , then there
exists an such that for any and any
If with , then there
exists an such that for any and any
![Page 12: Compressive Sensing in Noise and the Role of Adaptivity](https://reader031.fdocuments.in/reader031/viewer/2022022822/621c3d53c1a30d0cda4a0908/html5/thumbnails/12.jpg)
Intuition
Suppose that with and that
![Page 13: Compressive Sensing in Noise and the Role of Adaptivity](https://reader031.fdocuments.in/reader031/viewer/2022022822/621c3d53c1a30d0cda4a0908/html5/thumbnails/13.jpg)
Proof Recipe
Ingredients (Makes servings)
• Lemma 1: There exists a set of -sparse vectors such that
•
• for all
• for some
• Lemma 2: Define
Suppose is a set of -sparse vectors such that
for all .
Then .
Instructions
Combine ingredients and add a dash of linear algebra.
![Page 14: Compressive Sensing in Noise and the Role of Adaptivity](https://reader031.fdocuments.in/reader031/viewer/2022022822/621c3d53c1a30d0cda4a0908/html5/thumbnails/14.jpg)
The Details
![Page 15: Compressive Sensing in Noise and the Role of Adaptivity](https://reader031.fdocuments.in/reader031/viewer/2022022822/621c3d53c1a30d0cda4a0908/html5/thumbnails/15.jpg)
Lemma 1
Lemma 1: There exists a set of -sparse points such that
•
• for all
• for some
Strategy
Construct by sampling (with replacement) from
Repeat for iterations.
With probability , the remaining properties are satisfied.
Key: Matrix Bernstein Inequality [Ahlswede and Winter, 2002]
![Page 16: Compressive Sensing in Noise and the Role of Adaptivity](https://reader031.fdocuments.in/reader031/viewer/2022022822/621c3d53c1a30d0cda4a0908/html5/thumbnails/16.jpg)
Adaptive Sensing
![Page 17: Compressive Sensing in Noise and the Role of Adaptivity](https://reader031.fdocuments.in/reader031/viewer/2022022822/621c3d53c1a30d0cda4a0908/html5/thumbnails/17.jpg)
Think of sensing as a game of 20 questions
Simple strategy: Use measurements to find the support,
and the remainder to estimate the values.
Adaptive Sensing
![Page 18: Compressive Sensing in Noise and the Role of Adaptivity](https://reader031.fdocuments.in/reader031/viewer/2022022822/621c3d53c1a30d0cda4a0908/html5/thumbnails/18.jpg)
Thought Experiment
Suppose that after measurements we have perfectly
estimated the support.
![Page 19: Compressive Sensing in Noise and the Role of Adaptivity](https://reader031.fdocuments.in/reader031/viewer/2022022822/621c3d53c1a30d0cda4a0908/html5/thumbnails/19.jpg)
Does Adaptivity Really Help?
Sometimes…
• Noise-free measurements, but non-sparse signal
– adaptivity doesn’t help if you want a uniform guarantee
– probabilistic adaptive algorithms can reduce the required
number of measurements from to
[Indyk et al. – 2011]
• Noisy setting
– distilled sensing [Haupt et al. – 2007, 2010]
– adaptivity can reduce the estimation error to
Which is it?
![Page 20: Compressive Sensing in Noise and the Role of Adaptivity](https://reader031.fdocuments.in/reader031/viewer/2022022822/621c3d53c1a30d0cda4a0908/html5/thumbnails/20.jpg)
Which Is It?
Suppose we have a budget of measurements of the form
where and
The vector can have an arbitrary dependence on the
measurement history, i.e.,
[Arias-Castro, Candès, and Davenport - 2011]
Theorem
There exist with such that for any adaptive
measurement strategy and any recovery procedure ,
Thus, in general, adaptivity does not seem to help!
![Page 21: Compressive Sensing in Noise and the Role of Adaptivity](https://reader031.fdocuments.in/reader031/viewer/2022022822/621c3d53c1a30d0cda4a0908/html5/thumbnails/21.jpg)
Proof Strategy
Step 1: Consider a prior on sparse signals with nonzeros of
amplitude
Step 2: Show that if given a budget of measurements,
you cannot detect the support very well
Step 3: Immediately translate this into a lower bound on the
MSE
To make things simpler, we will consider a Bernoulli prior
instead of a uniform -sparse prior:
![Page 22: Compressive Sensing in Noise and the Role of Adaptivity](https://reader031.fdocuments.in/reader031/viewer/2022022822/621c3d53c1a30d0cda4a0908/html5/thumbnails/22.jpg)
Proof of Main Result
Let and set
For any estimator , define
Whenever or ,
![Page 23: Compressive Sensing in Noise and the Role of Adaptivity](https://reader031.fdocuments.in/reader031/viewer/2022022822/621c3d53c1a30d0cda4a0908/html5/thumbnails/23.jpg)
Proof of Main Result
Thus,
Plug in and this reduces to
Lemma
Under the Bernoulli prior, any estimate satisfies
![Page 24: Compressive Sensing in Noise and the Role of Adaptivity](https://reader031.fdocuments.in/reader031/viewer/2022022822/621c3d53c1a30d0cda4a0908/html5/thumbnails/24.jpg)
Key Ideas in Proof of Lemma
![Page 25: Compressive Sensing in Noise and the Role of Adaptivity](https://reader031.fdocuments.in/reader031/viewer/2022022822/621c3d53c1a30d0cda4a0908/html5/thumbnails/25.jpg)
Pinsker’s Inequality
Key Ideas in Proof of Lemma
![Page 26: Compressive Sensing in Noise and the Role of Adaptivity](https://reader031.fdocuments.in/reader031/viewer/2022022822/621c3d53c1a30d0cda4a0908/html5/thumbnails/26.jpg)
Adaptivity in Practice
![Page 27: Compressive Sensing in Noise and the Role of Adaptivity](https://reader031.fdocuments.in/reader031/viewer/2022022822/621c3d53c1a30d0cda4a0908/html5/thumbnails/27.jpg)
Adaptivity In Practice
Suppose that and that
Binary Search [Iwen and Tewfik – 2011, Davenport and Arias-Castro – 2012]
– split measurements into stages
– in each stage, use measurements to decide if the nonzero is
in the left or right half of the “active set”
– after subdividing times, return support
![Page 28: Compressive Sensing in Noise and the Role of Adaptivity](https://reader031.fdocuments.in/reader031/viewer/2022022822/621c3d53c1a30d0cda4a0908/html5/thumbnails/28.jpg)
Adaptivity In Practice
Suppose that and that
Binary Search [Iwen and Tewfik – 2011, Davenport and Arias-Castro – 2012]
– split measurements into stages
– in each stage, use measurements to decide if the nonzero is
in the left or right half of the “active set”
– after subdividing times, return support
![Page 29: Compressive Sensing in Noise and the Role of Adaptivity](https://reader031.fdocuments.in/reader031/viewer/2022022822/621c3d53c1a30d0cda4a0908/html5/thumbnails/29.jpg)
Adaptivity In Practice
Suppose that and that
Binary Search [Iwen and Tewfik – 2011, Davenport and Arias-Castro – 2012]
– split measurements into stages
– in each stage, use measurements to decide if the nonzero is
in the left or right half of the “active set”
– after subdividing times, return support
![Page 30: Compressive Sensing in Noise and the Role of Adaptivity](https://reader031.fdocuments.in/reader031/viewer/2022022822/621c3d53c1a30d0cda4a0908/html5/thumbnails/30.jpg)
Adaptivity In Practice
Suppose that and that
Binary Search [Iwen and Tewfik – 2011, Davenport and Arias-Castro – 2012]
– split measurements into stages
– in each stage, use measurements to decide if the nonzero is
in the left or right half of the “active set”
– after subdividing times, return support
![Page 31: Compressive Sensing in Noise and the Role of Adaptivity](https://reader031.fdocuments.in/reader031/viewer/2022022822/621c3d53c1a30d0cda4a0908/html5/thumbnails/31.jpg)
Experimental Results
[Arias-Castro, Candès, and Davenport - 2011]
nonadaptive adaptive
![Page 32: Compressive Sensing in Noise and the Role of Adaptivity](https://reader031.fdocuments.in/reader031/viewer/2022022822/621c3d53c1a30d0cda4a0908/html5/thumbnails/32.jpg)
Open Questions
• No method can succeed when , but the binary
search approach succeeds as long as
[Davenport and Arias-Castro; Malloy and Nowak - 2012]
• Practical algorithms that work well for all values of
• Optimal algorithms for
• New theory for restricted adaptive measurements
– single-pixel camera: 0/1 measurements
– magnetic resonance imaging (MRI): Fourier measurements
– analog-to-digital converters: linear filter measurements
• New sensors and architectures that can actually acquire
adaptive measurements