Fast algorithms for nonconvex compressive sensing
Transcript of Fast algorithms for nonconvex compressive sensing
Fast algorithms for nonconvexcompressive sensing
Rick Chartrand
Los Alamos National Laboratory
New Mexico Consortium
February 25, 2009
Slide 1 of 16
Operated by Los Alamos National Security, LLC for NNSA
Outline
Motivating example
Nonconvex compressive sensing
Algorithms
Summary
Slide 2 of 16
Operated by Los Alamos National Security, LLC for NNSA
Motivating example
Sparse tomographySuppose we want to reconstruct animage from samples of its Fouriertransform. How many samples dowe need?
Shepp-Logan phantom
Consider radial sampling,such as in MRI or (roughly)CT.
Ω
Slide 3 of 16
Operated by Los Alamos National Security, LLC for NNSA
Motivating example
Nonconvexity
Fewer measurements are needed with nonconvex minimization:
minu‖∇u‖pp, subject to (Fu)|Ω = (Fx)|Ω.
With p = 1, solution is u = x with 18 lines ( |Ω||x| = 6.9%).
With p = 1/2, 10 lines suffice ( |Ω||x| = 3.8%).
backprojection, 18 lines p = 1, 18 lines p = 12
, 10 lines p = 1, 10 lines
Slide 4 of 16
Operated by Los Alamos National Security, LLC for NNSA
Motivating example
Nonconvexity
Fewer measurements are needed with nonconvex minimization:
minu‖∇u‖pp, subject to (Fu)|Ω = (Fx)|Ω.
With p = 1, solution is u = x with 18 lines ( |Ω||x| = 6.9%).
With p = 1/2, 10 lines suffice ( |Ω||x| = 3.8%).
backprojection, 18 lines p = 1, 18 lines
p = 12
, 10 lines p = 1, 10 lines
Slide 4 of 16
Operated by Los Alamos National Security, LLC for NNSA
Motivating example
Nonconvexity
Fewer measurements are needed with nonconvex minimization:
minu‖∇u‖pp, subject to (Fu)|Ω = (Fx)|Ω.
With p = 1, solution is u = x with 18 lines ( |Ω||x| = 6.9%).
With p = 1/2, 10 lines suffice ( |Ω||x| = 3.8%).
backprojection, 18 lines p = 1, 18 lines p = 12
, 10 lines p = 1, 10 lines
Slide 4 of 16
Operated by Los Alamos National Security, LLC for NNSA
Motivating example
New results
These are old results (Mar. 2006); what’s new?
I Reconstruction (to 50 dB) in 13 seconds (versus literature-best1–3 minutes).
I Exact reconstruction from 9 lines (3.5% of Fourier transform).
10 lines fastest 10-line re-covery
9 lines recovery fromfewest samples
Slide 5 of 16
Operated by Los Alamos National Security, LLC for NNSA
Motivating example
New results
These are old results (Mar. 2006); what’s new?I Reconstruction (to 50 dB) in 13 seconds (versus literature-best
1–3 minutes).
I Exact reconstruction from 9 lines (3.5% of Fourier transform).
10 lines fastest 10-line re-covery
9 lines recovery fromfewest samples
Slide 5 of 16
Operated by Los Alamos National Security, LLC for NNSA
Motivating example
New results
These are old results (Mar. 2006); what’s new?I Reconstruction (to 50 dB) in 13 seconds (versus literature-best
1–3 minutes).I Exact reconstruction from 9 lines (3.5% of Fourier transform).
10 lines fastest 10-line re-covery
9 lines recovery fromfewest samples
Slide 5 of 16
Operated by Los Alamos National Security, LLC for NNSA
Nonconvex compressive sensing
Optimization for sparse recovery
I Let x ∈ RN be sparse: ‖Ψx‖0 = K, K N .I Suppose Φ is an M ×N matrix, M N , with Φ and Ψ
incoherent. For example, Φ = (ϕij), i.i.d. ϕij ∼ N(0, σ2).
minu‖Ψu‖0, s.t. Φu = Φx.
minu‖Ψu‖1, s.t. Φu = Φx.
minu‖Ψu‖pp, s.t. Φu = Φx,
Unique solution is u = x with opti-mally small M , but is NP-hard.
M ≥ 2K suffices w.h.p.
Can be solved efficiently; requiresmore measurements for reconstruc-tion. M ≥ CK log(N/K)
where 0 < p < 1. Solvable in prac-tice; requires fewer measurementsthan `1.M ≥ C1(p)K+pC2(p)K log(N/K)(with V. Staneva)
Slide 6 of 16
Operated by Los Alamos National Security, LLC for NNSA
Nonconvex compressive sensing
Optimization for sparse recovery
I Let x ∈ RN be sparse: ‖Ψx‖0 = K, K N .I Suppose Φ is an M ×N matrix, M N , with Φ and Ψ
incoherent. For example, Φ = (ϕij), i.i.d. ϕij ∼ N(0, σ2).
minu‖Ψu‖0, s.t. Φu = Φx.
minu‖Ψu‖1, s.t. Φu = Φx.
minu‖Ψu‖pp, s.t. Φu = Φx,
Unique solution is u = x with opti-mally small M , but is NP-hard.
M ≥ 2K suffices w.h.p.
Can be solved efficiently; requiresmore measurements for reconstruc-tion. M ≥ CK log(N/K)
where 0 < p < 1. Solvable in prac-tice; requires fewer measurementsthan `1.M ≥ C1(p)K+pC2(p)K log(N/K)(with V. Staneva)
Slide 6 of 16
Operated by Los Alamos National Security, LLC for NNSA
Nonconvex compressive sensing
The geometry of `p
minu ‖u‖pp, subject to Φu = Φxp = 2:
x
Φu = Φx
|u1|p + |u2|p + |u3|p = 0.1p
Slide 7 of 16
Operated by Los Alamos National Security, LLC for NNSA
Nonconvex compressive sensing
The geometry of `p
minu ‖u‖pp, subject to Φu = Φxp = 2:
x
Φu = Φx
|u1|p + |u2|p + |u3|p = 0.2p
Slide 7 of 16
Operated by Los Alamos National Security, LLC for NNSA
Nonconvex compressive sensing
The geometry of `p
minu ‖u‖pp, subject to Φu = Φxp = 2:
x
Φu = Φx
|u1|p + |u2|p + |u3|p = 0.3p
Slide 7 of 16
Operated by Los Alamos National Security, LLC for NNSA
Nonconvex compressive sensing
The geometry of `p
minu ‖u‖pp, subject to Φu = Φxp = 2:
x
Φu = Φx
|u1|p + |u2|p + |u3|p = 0.4p
Slide 7 of 16
Operated by Los Alamos National Security, LLC for NNSA
Nonconvex compressive sensing
The geometry of `p
minu ‖u‖pp, subject to Φu = Φxp = 2:
x
Φu = Φx
|u1|p + |u2|p + |u3|p = 0.5p
Slide 7 of 16
Operated by Los Alamos National Security, LLC for NNSA
Nonconvex compressive sensing
The geometry of `p
minu ‖u‖pp, subject to Φu = Φxp = 1:
x
Φu = Φx
|u1|p + |u2|p + |u3|p = 0.1p
Slide 8 of 16
Operated by Los Alamos National Security, LLC for NNSA
Nonconvex compressive sensing
The geometry of `p
minu ‖u‖pp, subject to Φu = Φxp = 1:
x
Φu = Φx
|u1|p + |u2|p + |u3|p = 0.2p
Slide 8 of 16
Operated by Los Alamos National Security, LLC for NNSA
Nonconvex compressive sensing
The geometry of `p
minu ‖u‖pp, subject to Φu = Φxp = 1:
x
Φu = Φx
|u1|p + |u2|p + |u3|p = 0.3p
Slide 8 of 16
Operated by Los Alamos National Security, LLC for NNSA
Nonconvex compressive sensing
The geometry of `p
minu ‖u‖pp, subject to Φu = Φxp = 1:
x
Φu = Φx
|u1|p + |u2|p + |u3|p = 0.4p
Slide 8 of 16
Operated by Los Alamos National Security, LLC for NNSA
Nonconvex compressive sensing
The geometry of `p
minu ‖u‖pp, subject to Φu = Φxp = 1:
x
Φu = Φx
|u1|p + |u2|p + |u3|p = 0.5p
Slide 8 of 16
Operated by Los Alamos National Security, LLC for NNSA
Nonconvex compressive sensing
The geometry of `p
minu ‖u‖pp, subject to Φu = Φxp = 1:
x
Φu = Φx
|u1|p + |u2|p + |u3|p = 0.6p
Slide 8 of 16
Operated by Los Alamos National Security, LLC for NNSA
Nonconvex compressive sensing
The geometry of `p
minu ‖u‖pp, subject to Φu = Φxp = 1:
x
Φu = Φx
|u1|p + |u2|p + |u3|p = 0.7p
Slide 8 of 16
Operated by Los Alamos National Security, LLC for NNSA
Nonconvex compressive sensing
The geometry of `p
minu ‖u‖pp, subject to Φu = Φxp = 1/2:
x
|u1|p + |u2|p + |u3|p = 0.1p
Φu = Φx
Slide 9 of 16
Operated by Los Alamos National Security, LLC for NNSA
Nonconvex compressive sensing
The geometry of `p
minu ‖u‖pp, subject to Φu = Φxp = 1/2:
x
|u1|p + |u2|p + |u3|p = 0.2p
Φu = Φx
Slide 9 of 16
Operated by Los Alamos National Security, LLC for NNSA
Nonconvex compressive sensing
The geometry of `p
minu ‖u‖pp, subject to Φu = Φxp = 1/2:
x
|u1|p + |u2|p + |u3|p = 0.3p
Φu = Φx
Slide 9 of 16
Operated by Los Alamos National Security, LLC for NNSA
Nonconvex compressive sensing
The geometry of `p
minu ‖u‖pp, subject to Φu = Φxp = 1/2:
x
|u1|p + |u2|p + |u3|p = 0.4p
Φu = Φx
Slide 9 of 16
Operated by Los Alamos National Security, LLC for NNSA
Nonconvex compressive sensing
The geometry of `p
minu ‖u‖pp, subject to Φu = Φxp = 1/2:
x
|u1|p + |u2|p + |u3|p = 0.5p
Φu = Φx
Slide 9 of 16
Operated by Los Alamos National Security, LLC for NNSA
Nonconvex compressive sensing
The geometry of `p
minu ‖u‖pp, subject to Φu = Φxp = 1/2:
x
|u1|p + |u2|p + |u3|p = 0.6p
Φu = Φx
Slide 9 of 16
Operated by Los Alamos National Security, LLC for NNSA
Nonconvex compressive sensing
The geometry of `p
minu ‖u‖pp, subject to Φu = Φxp = 1/2:
x
|u1|p + |u2|p + |u3|p = 0.7p
Φu = Φx
Slide 9 of 16
Operated by Los Alamos National Security, LLC for NNSA
Nonconvex compressive sensing
The geometry of `p
minu ‖u‖pp, subject to Φu = Φxp = 1/2:
x
|u1|p + |u2|p + |u3|p = 0.8p
Φu = Φx
Slide 9 of 16
Operated by Los Alamos National Security, LLC for NNSA
Nonconvex compressive sensing
The geometry of `p
minu ‖u‖pp, subject to Φu = Φxp = 1/2:
x
|u1|p + |u2|p + |u3|p = 0.9p
Φu = Φx
Slide 9 of 16
Operated by Los Alamos National Security, LLC for NNSA
Nonconvex compressive sensing
The geometry of `p
minu ‖u‖pp, subject to Φu = Φxp = 1/2:
x
|u1|p + |u2|p + |u3|p = 1p
Φu = Φx
Slide 9 of 16
Operated by Los Alamos National Security, LLC for NNSA
Nonconvex compressive sensing
Why might global minimization be possible?
Consider the ε-regularized, constraint-eliminated objective:
Fε(t) =N∑i=1
[xi + (V t)i
]2 + ε
p/2,
whereR(V ) = N (Φ). A moderate ε fills in the local minima.
Slide 10 of 16
Operated by Los Alamos National Security, LLC for NNSA
Nonconvex compressive sensing
Why might global minimization be possible?
Consider the ε-regularized, constraint-eliminated objective:
Fε(t) =N∑i=1
[xi + (V t)i
]2 + ε
p/2,
whereR(V ) = N (Φ). A moderate ε fills in the local minima.
ε = 0
Slide 10 of 16
Operated by Los Alamos National Security, LLC for NNSA
Nonconvex compressive sensing
Why might global minimization be possible?
Consider the ε-regularized, constraint-eliminated objective:
Fε(t) =N∑i=1
[xi + (V t)i
]2 + ε
p/2,
whereR(V ) = N (Φ). A moderate ε fills in the local minima.
ε = 1
Slide 10 of 16
Operated by Los Alamos National Security, LLC for NNSA
Nonconvex compressive sensing
Why might global minimization be possible?
Consider the ε-regularized, constraint-eliminated objective:
Fε(t) =N∑i=1
[xi + (V t)i
]2 + ε
p/2,
whereR(V ) = N (Φ). A moderate ε fills in the local minima.
ε = 0.1
Slide 10 of 16
Operated by Los Alamos National Security, LLC for NNSA
Nonconvex compressive sensing
Why might global minimization be possible?
Consider the ε-regularized, constraint-eliminated objective:
Fε(t) =N∑i=1
[xi + (V t)i
]2 + ε
p/2,
whereR(V ) = N (Φ). A moderate ε fills in the local minima.
ε = 0.01
Slide 10 of 16
Operated by Los Alamos National Security, LLC for NNSA
Nonconvex compressive sensing
Why might global minimization be possible?
Consider the ε-regularized, constraint-eliminated objective:
Fε(t) =N∑i=1
[xi + (V t)i
]2 + ε
p/2,
whereR(V ) = N (Φ). A moderate ε fills in the local minima.
ε = 0.001
Slide 10 of 16
Operated by Los Alamos National Security, LLC for NNSA
Algorithms
Semiconvex regularization
Now we generalize an approach of J. Yang, W. Yin, Y. Zhang, and Y.Wang. Consider a mollified `p objective on R2:
ϕ(t) =
γ|t|2 if |t| ≤ α|t|p/p− δ if |t| > α
The parameters are chosen tomake ϕ ∈ C1.
ϕ(t)
t1α
Now we seek ψ such that
ϕ(t) = minw
ψ(w) + (β/2)|t− w|22
This can be found by convex duality, as |t|22/2−ϕ(t)/β is convex ifβ = αp−2.
Slide 11 of 16
Operated by Los Alamos National Security, LLC for NNSA
Algorithms
Semiconvex regularization
Now we generalize an approach of J. Yang, W. Yin, Y. Zhang, and Y.Wang. Consider a mollified `p objective on R2:
ϕ(t) =
γ|t|2 if |t| ≤ α|t|p/p− δ if |t| > α
The parameters are chosen tomake ϕ ∈ C1.
ϕ(t)
t1α
Now we seek ψ such that
ϕ(t) = minw
ψ(w) + (β/2)|t− w|22
This can be found by convex duality, as |t|22/2−ϕ(t)/β is convex ifβ = αp−2.
Slide 11 of 16
Operated by Los Alamos National Security, LLC for NNSA
Algorithms
A splitting approach
Now we consider an unconstrained `p minimization problem, andreplace
minu
N∑i=1
ϕ((Du)i) + (µ/2)‖Φu− b‖22
with the split version
minu,w
N∑i=1
ψ(wi) + (β/2)‖Du− w‖22 + (µ/2)‖Φu− b‖22,
which we solve by alternate minimization.
Slide 12 of 16
Operated by Los Alamos National Security, LLC for NNSA
Algorithms
Easy iterations
Holding u fixed, the w-subproblem is separable, and its solutioncomes from the convex duality:
wi = max
0, |(Du)i| −
|(Du)i|p−1
β
(Du)i|(Du)i|
.
This generalizes shrinkage ( or soft thresholding ) to `p.
Holding w fixed, the u-problem is quadratic:
(βDTD + µΦTΦ)u = βDTw + µΦT b.
If Φ is a Fourier sampling operator, we can solve this in the Fourierdomain. This is very fast!
Slide 13 of 16
Operated by Los Alamos National Security, LLC for NNSA
Algorithms
Easy iterations
Holding u fixed, the w-subproblem is separable, and its solutioncomes from the convex duality:
wi = max
0, |(Du)i| −
|(Du)i|p−1
β
(Du)i|(Du)i|
.
This generalizes shrinkage ( or soft thresholding ) to `p.
Holding w fixed, the u-problem is quadratic:
(βDTD + µΦTΦ)u = βDTw + µΦT b.
If Φ is a Fourier sampling operator, we can solve this in the Fourierdomain. This is very fast!
Slide 13 of 16
Operated by Los Alamos National Security, LLC for NNSA
Algorithms
Easy iterations
Holding u fixed, the w-subproblem is separable, and its solutioncomes from the convex duality:
wi = max
0, |(Du)i| −
|(Du)i|p−1
β
(Du)i|(Du)i|
.
This generalizes shrinkage ( or soft thresholding ) to `p.
Holding w fixed, the u-problem is quadratic:
(βDTD + µΦTΦ)u = βDTw + µΦT b.
If Φ is a Fourier sampling operator, we can solve this in the Fourierdomain. This is very fast!
Slide 13 of 16
Operated by Los Alamos National Security, LLC for NNSA
Algorithms
Enforcing equality
minu,w
N∑i=1
ψ(wi) + (β/2)‖Du− w‖22 + (µ/2)‖Φu− b‖22
Typically one enforces w = Du and Φu = b by iteratively growingβ and µ larger (continuation).
We get better results from Bregman iteration (generalizing T.Goldstein, S. Osher):
minu,w
N∑i=1
ψ(wi)+(β/2)‖Du−w−Bw‖22 +(µ/2)‖Φu−b−Bu‖22,
and update Bn+1w = Bnw + w −Du (inner loop),
Bm+1u = Bmu + b− Φu (outer loop, if desired).
Slide 14 of 16
Operated by Los Alamos National Security, LLC for NNSA
Algorithms
Enforcing equality
minu,w
N∑i=1
ψ(wi) + (β/2)‖Du− w‖22 + (µ/2)‖Φu− b‖22
Typically one enforces w = Du and Φu = b by iteratively growingβ and µ larger (continuation).
We get better results from Bregman iteration (generalizing T.Goldstein, S. Osher):
minu,w
N∑i=1
ψ(wi)+(β/2)‖Du−w−Bw‖22 +(µ/2)‖Φu−b−Bu‖22,
and update Bn+1w = Bnw + w −Du (inner loop),
Bm+1u = Bmu + b− Φu (outer loop, if desired).
Slide 14 of 16
Operated by Los Alamos National Security, LLC for NNSA
Algorithms
A surprise
I An early coding mistake led to the use of p = −1/2. This iswhat made the 9-line phantom reconstruction possible.
I Results decay slowly as p is decreased below −1/2.I Negative p-values were used by Rao and Kreutz-Delgado in an
iteratively-reweighted least squares (IRLS) algorithm. Theirnegative p results were worse than their positive p results,which were hampered by getting stuck in local minima.
I A mollified IRLS approach (with Wotao Yin) apparently avoidslocal minima, but negative p results are not better than positivep results.
Slide 15 of 16
Operated by Los Alamos National Security, LLC for NNSA
Algorithms
A surprise
I An early coding mistake led to the use of p = −1/2. This iswhat made the 9-line phantom reconstruction possible.
I Results decay slowly as p is decreased below −1/2.
I Negative p-values were used by Rao and Kreutz-Delgado in aniteratively-reweighted least squares (IRLS) algorithm. Theirnegative p results were worse than their positive p results,which were hampered by getting stuck in local minima.
I A mollified IRLS approach (with Wotao Yin) apparently avoidslocal minima, but negative p results are not better than positivep results.
Slide 15 of 16
Operated by Los Alamos National Security, LLC for NNSA
Algorithms
A surprise
I An early coding mistake led to the use of p = −1/2. This iswhat made the 9-line phantom reconstruction possible.
I Results decay slowly as p is decreased below −1/2.I Negative p-values were used by Rao and Kreutz-Delgado in an
iteratively-reweighted least squares (IRLS) algorithm. Theirnegative p results were worse than their positive p results,which were hampered by getting stuck in local minima.
I A mollified IRLS approach (with Wotao Yin) apparently avoidslocal minima, but negative p results are not better than positivep results.
Slide 15 of 16
Operated by Los Alamos National Security, LLC for NNSA
Algorithms
A surprise
I An early coding mistake led to the use of p = −1/2. This iswhat made the 9-line phantom reconstruction possible.
I Results decay slowly as p is decreased below −1/2.I Negative p-values were used by Rao and Kreutz-Delgado in an
iteratively-reweighted least squares (IRLS) algorithm. Theirnegative p results were worse than their positive p results,which were hampered by getting stuck in local minima.
I A mollified IRLS approach (with Wotao Yin) apparently avoidslocal minima, but negative p results are not better than positivep results.
Slide 15 of 16
Operated by Los Alamos National Security, LLC for NNSA
Summary
Summary
I Nonconvex compressive sensing allows sparse signals to berecovered with even fewer measurements than “traditional”compressive sensing.
I Decreasing p also improves robustness to noise, and speedsup convergence.
I Regularizing the objective appears to keep algorithms fromconverging to nonglobal minima.
I For Fourier-sampling measurements, such as MRI, a very fastalgorithm is available.
math.lanl.gov/~rick
Slide 16 of 16
Operated by Los Alamos National Security, LLC for NNSA