Post on 31-Mar-2018
Seminar Talk at ICMSEC, Chinese Academy of Sciences December 2nd, 2008 1
A Class of Fast Algorithms for TV BasedImage Reconstruction
Yin ZhangDepartment of CAAM
Rice University
Joint work with: Junfeng Yang, Yilun Wang and Wotao Yin
Seminar Talk at ICMSEC, Chinese Academy of Sciences December 2nd, 2008 2
Outline
• Introduction– Image formation equation
– Maximum likelihood estimation
– Maximum a posteriori estimation
– Regularization
• A fast alternating algorithm– Motivation and algorithm
– Relation with half-quadratic technique
– Optimality and convergence results
• Numerical results and extensions
Seminar Talk at ICMSEC, Chinese Academy of Sciences December 2nd, 2008 3
Introduction
• Image formation equation.
f = Ku + ω
– u: original image
– K : convolution operator
– ω: random noise
– f : observation
Our purpose is to recover u from f (deconvolve and denoise) as well as
possible.
Seminar Talk at ICMSEC, Chinese Academy of Sciences December 2nd, 2008 4
• Deconvolution is severely ill-conditioned. Let F be the 2D Fourier transform
matrix. The equation is equivalent to
f = K ˆu + ω,
where f = Ff and K = FKF−1(diagonal). A tempting solution would be
udirect = F−1(K−1f) = F
−1(ˆu + K−1ω).
Does this work?
Seminar Talk at ICMSEC, Chinese Academy of Sciences December 2nd, 2008 5
Experiment 1. Blur: (’gaussian’,11,5); Noise: N (0, 10−8).
Original Blurry
Blurry&Noisy Direct inverse
Noise is amplified!
Seminar Talk at ICMSEC, Chinese Academy of Sciences December 2nd, 2008 6
Cut off high frequencies (Weiner Filter):
ui =
0, if |fi/Kii| > M ;
fi/Kii, otherwise.
Result of experiment 1 after cutting off some high frequencies:
Cut off high frequencies
Seminar Talk at ICMSEC, Chinese Academy of Sciences December 2nd, 2008 7
Statistics Interpretation:
• Maximum likelihood estimation. Given ω ∼ N (0, σ2), the MLE of u is
uMLE = arg maxu
Pr{f |u}
= arg minu
(− log(Pr{f |u}))
= arg minu‖Ku− f‖2.
Thus, MLE, LS and direct inverse are all equivalent. They do not work. When
noise is correlated, i.e., ω ∼ N (0,Σ), MLE becomes weighted LS.
Seminar Talk at ICMSEC, Chinese Academy of Sciences December 2nd, 2008 8
Another Statistics Viewpoint:
• Maximum a posteriori estimation. Given ω ∼ N (0, σ2), the MAP of u is
uMAP = arg maxu
Pr{u|f}
= arg maxu
Pr{u}Pr{f |u}
Pr{f}
= arg minu{− log(Pr{u})− log(Pr{f |u})}
= arg minu
Φprior(u) + ‖Ku− f‖2.
Thus, Φprior(u) enforces some prior constraints on u, which is called
regularization. Qusetion: what kind of prior do we need?
Seminar Talk at ICMSEC, Chinese Academy of Sciences December 2nd, 2008 9
• Regularization.
minu
Φreg(u) + µ‖Ku− f‖22
– Tikhonov-like regularization (notice 2-norm squared)
Φreg(u) = ΦTik(u) ,∑
j∈J
‖D(j)u‖22,
for some J ⊂ {0, 1, 2, . . .}, where
∗ D(0): identity matrix
∗ D(j), j = 1, 2: the 1st order finite difference matrices
∗ D(j), j = 3, 4, 5: the 2nd order . . . (used by MATLAB “deconvreg”).
The solution satisfies
∑
j∈J
(D(j))⊤D(j) + µK⊤K
u = µK⊤f.
Seminar Talk at ICMSEC, Chinese Academy of Sciences December 2nd, 2008 10
Experiment 2. Result of Tikhonov regularization. Blur: (’gaussian’,21,11);
Noise: N (0, 10−6).
Blurry&Noisy Recovered by "deconvreg"
Advantages: Not so sensitive to noise, easy to compute.
Seminar Talk at ICMSEC, Chinese Academy of Sciences December 2nd, 2008 11
Disadvantage of Square: Incapable of recovering image discontinuities.
minu∈R11
φ(u) =∑
i
|ui+1 − ui|2, s.t. u1 = 0, u11 = 255.
2 4 6 8 100
50
100
150
200
250
i
u i
Unique solution
Seminar Talk at ICMSEC, Chinese Academy of Sciences December 2nd, 2008 12
– Total variation regularization (Rudin, Osher and Fatemi, 1992).
Φreg(u) = TV(u) ,∑
i
‖Diu‖.
∗ ‖Diu‖: the variation of u at pixel i, where
Diu =
(D(1)u)i
(D(2)u)i
∈ R2.
∗∑
i is taken over all pixels.
∗ The sum represents a 1-norm.
∗ ‖ · ‖: the 2-norm (isotropic) or the 1-norm (anisotropic).
Seminar Talk at ICMSEC, Chinese Academy of Sciences December 2nd, 2008 13
Advantage of 1-norm: Permits sharp edges in images.
minu∈R11
TV(u) =∑
i
|ui+1 − ui|, s.t. u1 = 0, u11 = 255.
2 4 6 8 100
50
100
150
200
250
i
u iPossible solution
Seminar Talk at ICMSEC, Chinese Academy of Sciences December 2nd, 2008 14
Experiment 3. Compare Tikhonov with TV regularization. The same inputs
as in experiment 2.
Recovered by "deconvreg" Recovered by FTVd
Disadvantages: More expensive in computation, stair-casing effect.
Seminar Talk at ICMSEC, Chinese Academy of Sciences December 2nd, 2008 15
TV/L2 : minu
∑
i
‖Diu‖2 +µ
2‖Ku− f‖2.
It’s a convex program, large-scale, still ill-conditioned and requires “real-time”
processing.
• Some existing methods.
– Lagged diffusivity method (Vogel & Oman, 1995). Given uk, uk+1 is
determined by solving
∑
i
D⊤i
Diu
‖Diuk‖α+ µK⊤(Ku− f) = 0,
which is a linearization to the optimality condition of
minu
∑
i
‖Diu‖α +µ
2‖Ku− f‖2.
Here ‖ · ‖α ,√
‖ · ‖2 + α for some small α > 0.
Most earlier methods were based on solving (Euler-Langrange) PDE.
Seminar Talk at ICMSEC, Chinese Academy of Sciences December 2nd, 2008 16
– Iterative Shrinkage/Thresholding based methods (Daubechies, Defrise &
De Mol, 2004). Given uk, the original IST method iterates as
uk+1 = Ψµ
(
uk − λkK⊤(Kuk − f))
,
where λk > 0 and
Ψµ(ξ) , arg minu
TV(u) +µ
2‖u− ξ‖2.
There exist several variants of IST methods, e.g., TwIST (Bioucas-Dias &
Figueiredo, 2007).
– Second-order cone programming approach (Goldfarb & Yin, 2005).
– Iterative Denoising (Michael Ng et al 2007). Much faster, but ......
Seminar Talk at ICMSEC, Chinese Academy of Sciences December 2nd, 2008 17
A Fast Alternating Algorithm
• Motivation. The problem is
minu
∑
i
‖Diu‖+µ
2‖Ku− f‖2.
By introducing wi ∈ R2, TV/L2 is approximated by, for β ≫ 0,
minwi,u
∑
i
(
‖wi‖+β
2‖wi −Diu‖
2
)
+µ
2‖Ku− f‖2.
The approximation problem allows very fast alternating minimization.
Seminar Talk at ICMSEC, Chinese Academy of Sciences December 2nd, 2008 18
A simple and important lemma:
Lemma 1 Given a positive integer d. For any β > 0 and t ∈ Rd, it holds
max
{
‖t‖ −1
β, 0
}
t
‖t‖= arg min
s∈Rd
{
‖s‖+β
2‖s− t‖2
}
,
where we follow the convention 0 · (0/0) = 0.
An important Observation: Finite differences, D(1) and D(2) can be treated
as discrete convolution under suitable boundary conditions.
Consequently, D(1) and D(2) and K are circulant matrices under the
periodic boundary conditions for u, and all can be diagonalized by FFT.
Seminar Talk at ICMSEC, Chinese Academy of Sciences December 2nd, 2008 19
• Our Simple Algorithm:
– w-subproblem. Fixing u, minimizing w.r.t. w reduces to
minwi
‖wi‖+β
2‖wi −Diu‖
2, ∀i.
Separate and closed form solutions at all pixels i:
wi = max
{
‖Diu‖ −1
β, 0
}
Diu
‖Diu‖, ∀i.
Linear time complexity: O(n2).
Seminar Talk at ICMSEC, Chinese Academy of Sciences December 2nd, 2008 20
– u-subproblem. Fixing {wi}, minimizing w.r.t. u reduces to
minu
β
2
∑
i
‖wi −Diu‖2 +
µ
2‖Ku− f‖2.
Its normal equations are(
∑
i
D⊤i Di +
µ
βK⊤K
)
u =∑
i
D⊤i wi +
µ
βK⊤f
or equivalently
2∑
j=1
(D(j))⊤D(j) +µ
βK⊤K
u =
2∑
j=1
(D(j))⊤wj +µ
βK⊤f,
where wj = {wi(j) : i = 1, . . . , n2} for j = 1, 2.
This system can be solved by 2 FFTs at a cost of O(n2 log n).
Seminar Talk at ICMSEC, Chinese Academy of Sciences December 2nd, 2008 21
– Continuation/path-following. Initialize β small, and then increase it
gradually. The previous solution is used to warm-start the next problem.
0 100 200 300 400 5000
0.02
0.04
0.06
0.08
0.1
0.12
Iteration
||uk −
u* ||/||u
* ||
With continuationWithout continuation
0 100 200 300 400 5000
0.02
0.04
0.06
0.08
0.1
0.12
Iteration
||uk −
u* ||/||u
* ||
With continuationWithout continuation
0 100 200 300 400 5000
0.02
0.04
0.06
0.08
0.1
0.12
Iteration
||uk −
u* ||/||u
* ||
With continuationWithout continuation
0 100 200 300 400 5000
0.02
0.04
0.06
0.08
0.1
0.12
Iteration
||uk −
u* ||/||u
* ||
With continuationWithout continuation
0 100 200 300 400 5000
0.02
0.04
0.06
0.08
0.1
0.12
Iteration
||uk −
u* ||/||u
* ||
With continuationWithout continuation
0 100 200 300 400 5000
0.02
0.04
0.06
0.08
0.1
0.12
Iteration
||uk −
u* ||/||u
* ||
With continuationWithout continuation
0 100 200 300 400 5000
0.02
0.04
0.06
0.08
0.1
0.12
Iteration
||uk −
u* ||/||u
* ||
With continuationWithout continuation
0 100 200 300 400 5000
0.02
0.04
0.06
0.08
0.1
0.12
Iteration
||uk −
u* ||/||u
* ||
With continuationWithout continuation
0 100 200 300 400 50011
12
13
14
15
16
17
Iteration
SN
R (
dB)
With continuationWithout continuation
0 100 200 300 400 50011
12
13
14
15
16
17
Iteration
SN
R (
dB)
With continuationWithout continuation
0 100 200 300 400 50011
12
13
14
15
16
17
Iteration
SN
R (
dB)
With continuationWithout continuation
Test on continuation: β = 20, 21, . . . , 210.
Continuation not only accelerates the speed, but also, unexpectedly,
enhances solution robustness.
Seminar Talk at ICMSEC, Chinese Academy of Sciences December 2nd, 2008 22
Given β > 0, we solve the approximation problem by alternately
minimizing w.r.t. w and u.
– FTVd (Fast TV deconvolution). Input K, f , µ > 0, βmax ≫ 0 and
γ > 1; Initialize β = β0 > 0 and u = u0.
While β <= βmax, Do
1) Solve the approximation to certain accuracy for uβ .
2) Update u← uβ , β ← γ ∗ β.
End Do
Seminar Talk at ICMSEC, Chinese Academy of Sciences December 2nd, 2008 23
• Relation with half-quadratic technique. Given β > 0, FTVd solves
minu,w
∑
i
{
‖wi‖+β
2‖Diu− wi‖
2
}
+µ
2‖Ku− f‖2.
The above is equivalent to
minu
∑
i
φ (Diu) +µ
2‖Ku− f‖2,
where φ(t), t ∈ R2, is defined as
φ(t) =
β2 ‖t‖
2, if ‖t‖ ≤ 1/β;
‖t‖ − 12β
, otherwise.
This is an extension to the half-quadratic transform (German and Yang 1995).
Seminar Talk at ICMSEC, Chinese Academy of Sciences December 2nd, 2008 24
• Optimality. A pair (w, u) solves the approximation problem iff
wi/‖wi‖+ β(wi −Diu) = 0 i ∈ I1 , {i : wi 6= 0},
β‖Diu‖ ≤ 1 i ∈ I2 , {i : wi = 0},
βD⊤(Du− w) + µK⊤(Ku− f) = 0.
Eliminating w, the final equations become
∑
i∈I1
D⊤i
Diu
‖Diu‖+∑
i∈I2
D⊤i hi + µK⊤(Ku− f) = 0,
where hi = βDiu satisfies ‖hi‖ ≤ 1, which is an approximation to the
optimality condition of TV/L2.
Seminar Talk at ICMSEC, Chinese Academy of Sciences December 2nd, 2008 25
• Convergence results. Let D =(
D(1); D(2))
,
M = D⊤D + (µ/β) ·K⊤K and T = DM−1D⊤.
AssumingN (D) ∩N (K) = {0}, for fixed β we have
1. The sequence {(wk, uk)} generated by FTVd converges to a solution
(w∗, u∗) of the approximation problem.
2. Finite convergence. wkL ≡ w∗
L in finite number of iterations.
3. q-linear convergence. For k sufficiently large, there hold
(a) ‖D(uk+1 − u∗)‖ ≤√
‖(T 2)EE‖ · ‖D(uk − u∗)‖;
(b) ‖wk+1 − w∗‖ ≤√
‖(T 2)EE‖ · ‖wk − w∗‖;
(c) ‖uk+1 − u∗‖M ≤√
‖TEE‖ · ‖uk − u∗‖M .
Here L = {i, ‖Diu∗‖ < 1/β} and E = {1, 2, . . . , n2} \ L.
Seminar Talk at ICMSEC, Chinese Academy of Sciences December 2nd, 2008 26
Numerical results and extensions
• Restoration of grayscale images. Kernel: (’gaussian’,21,10); Noise: Gaussian
white with mean zero and std= 10−3.
Blurry&Noisy. SNR: 6.3dB Lag.D. SNR: 12.2dB, CPU: 512.1s FTVd. SNR: 12.6dB, Iter: 11, CPU: 1.9s
Seminar Talk at ICMSEC, Chinese Academy of Sciences December 2nd, 2008 27
Blurry&Noisy. SNR: 7.7dB Lag.D. SNR: 14.4dB, CPU: 1918.0s FTVd. SNR: 14.8dB, Iter: 9, CPU: 7.0s
Blurry&Noisy. SNR: 9.1dB Lag.D. SNR: 15.0dB, CPU: 7306.7s FTVd. SNR: 15.5dB, Iter: 10, CPU: 31.2s
Seminar Talk at ICMSEC, Chinese Academy of Sciences December 2nd, 2008 28
• Speed comparison with Lagged Diffusivity method. Noise: Gaussian, mean
zero and std= 10−3; Blur: (’gaussian’,hsize,10).
5 10 15 200
1000
2000
3000
4000
5000
6000
hsize
LD: C
PU
(s)
camera, 256×256lena, 512×512man, 1024×1024
5 10 15 200
10
20
30
40
50
60
hsize
FT
Vd:
CP
U(s
)
camera, 256×256lena, 512×512man, 1024×1024
5 10 15 200
50
100
150
200
250
300
hsize
Tim
e co
mpa
rison
: LD
/FT
Vd
camera, 256×256lena, 512×512man, 1024×1024
5 10 15 206
6.5
7
7.5
8
8.5
9
hsize
ISN
R(d
B)
Lena−FTVdLena−LDMan−FTVdMan−LD
Seminar Talk at ICMSEC, Chinese Academy of Sciences December 2nd, 2008 29
• Multichannel image deconvolution. Let u be a RGB image. The image
formulation equation f = Ku + ω becomes
fr
fg
f b
=
Krr Krg Krb
Kgr Kgg Kgb
Kbr Kbg Kbb
ur
ug
ub
+
ωr
ωg
ωb
.
TV is extended to
MTV(u) ,∑
i
‖(I3 ⊗Di)u‖,
where
(I3⊗Di)u =[
D(1)ur, D(2)ur, D(1)ug, D(2)ug, D(1)ub, D(2)ub]
i∈ R
6.
Seminar Talk at ICMSEC, Chinese Academy of Sciences December 2nd, 2008 30
Generally, let u ∈ Rmn2
be a m-channel image and K = [Kjk]mjk=1 be a
cross-channel blurring matrix. The TV/L2 model is extended as
minu
∑
i
αi‖Giu‖+µ
2‖Ku− f‖2,
where Gi = Im ⊗Di, andDi is a 1st and/or higher order local finite
difference operator. It is approximated by
minu,w
∑
i
(
αi‖wi‖+β
2‖wi −Giu‖
2
)
+µ
2‖Ku− f‖2.
– Fixing u, the minimizer function for w is given explicitly by:
wi = max
{
‖Giu‖ −αi
β, 0
}
Giu
‖Giu‖, ∀i.
Seminar Talk at ICMSEC, Chinese Academy of Sciences December 2nd, 2008 31
– The u-subproblem is equivalent to
∑
j
(G(j))⊤G(j) +µ
βK⊤K
u =∑
j
(G(j))⊤wj +µ
βK⊤f.
By pre- and post- multiplying Im ⊗ F and its inverse, respectively, the
coefficient matrix becomes
Λ11 Λ12 . . . Λ1m
Λ21 Λ22 . . . Λ2m
......
. . ....
Λm1 Λm2 . . . Λmm
,
with each Λij a diagonal matrix. Thus u-subproblem is easily solved by FFTs
and low complexity Gaussian elimination.
Seminar Talk at ICMSEC, Chinese Academy of Sciences December 2nd, 2008 32
Restoration from cross-channel blur and Gaussian noise:
Original Blurry&Noisy. SNR: 6.70dB FTVd: SNR: 18.49dB, t = 4.29s
Original Blurry&Noisy. SNR: 8.01dB FTVd: SNR: 19.54dB, t = 16.86s
Seminar Talk at ICMSEC, Chinese Academy of Sciences December 2nd, 2008 33
• Deconvolution in the presence of impulsive noise. Cameraman degraded by
convolution and 10% salt-and-pepper noise. Right: solution of TV/L2.
Seminar Talk at ICMSEC, Chinese Academy of Sciences December 2nd, 2008 34
For impulsive noise, the ℓ1-norm fidelity is more suitable. We recover u as
the solution of the TV/L1 model:
minu
∑
i
‖Diu‖+ µ‖Ku− f‖1.
The approximation problem is given by
minw,z,u
∑
i
(
‖wi‖+β
2‖wi −Diu‖
2
)
+ µ(
‖z‖1 +γ
2‖z − (Ku− f)‖2
)
.
Minimization w.r.t. w, z and u each is easy!
Seminar Talk at ICMSEC, Chinese Academy of Sciences December 2nd, 2008 35
Restoration from Gaussian blur and salt-and-pepper noise:
Blurry&Noisy: 30% Salt&Pepper 40% Salt&Pepper 50% Salt&Pepper 60% Salt&Pepper
FTVd. µ: 13, t: 15.1s, SNR: 14.16dB FTVd. µ: 10, t: 13.9s, SNR: 13.21dB FTVd. µ: 8, t: 13.5s, SNR: 12.35dB FTVd. µ: 4, t: 16.8s, SNR: 11.08dB
Seminar Talk at ICMSEC, Chinese Academy of Sciences December 2nd, 2008 36
Restoration from cross-channel blur and random-valued noise:
40% RV 50% RV 60% RV
µ: 8, t: 117s, SNR: 16.04dB µ: 4, t: 138s, SNR: 14.06dB µ: 2, t: 136s, SNR: 10.60dB
Seminar Talk at ICMSEC, Chinese Academy of Sciences December 2nd, 2008 37
• MRI reconstruction. In MR imaging system, MR scanner collects data:
fp = Fpu + ω ∈ CM , M ≪ N.
Without noise, under certain desirable conditions, it holds
u = arg minu{TV(u) : Fpu = fp} .
In the presence of noise, we recover u via
minu
TV(u) +µ
2‖Fpu− fp‖
2.
When u has sparse/compressible representation under certain wavelet basis,
we recover it via
minu
TV(u) + τ‖Ψ⊤u‖1 +µ
2‖Fpu− fp‖
2.
FTVd can be extended to solve the above TVL1-L2 problem.
Seminar Talk at ICMSEC, Chinese Academy of Sciences December 2nd, 2008 38
Sparse (under TV) image reconstruction. Left to right: Original, Fourier domain
samples (9.36%), reconstructed image (RelErr: 4.48%). Gaussian noise with
mean zero and std= .01.
Seminar Talk at ICMSEC, Chinese Academy of Sciences December 2nd, 2008 39
Compressible (under wavelet) image reconstruction. Sample ratio: 9.64%; Noise:
Gaussian, mean zero, std= .01; Left: original brain image; Right: reconstructed
(RelErr: 11.58%).
Seminar Talk at ICMSEC, Chinese Academy of Sciences December 2nd, 2008 40
• Summary.
– FTVd converges without the assumption of strictly convexity.
– Finite convergence of auxiliary variables is established.
– Linear convergence rate is established and the convergence factor
depends on a submatrix.
– FTVd is fast for TV based problem because it fully exploits problem
structure and utilizes FFT.
Seminar Talk at ICMSEC, Chinese Academy of Sciences December 2nd, 2008 41
References
[1] L. Rudin and S. Osher and E. Fatemi, Nonlinear total variation based noise
removal algorithms, Phys. D, vol.60, 259-268, 1992.
[2] C. R. Vogel and M. E. Oman, Fast total variation based image reconstruction,
Proc. ASME design engineering conferences, 3, 1009-1015, 1995.
[3] D. Goldfarb and W. Yin, Second-order cone programming methods for total
variation-based image restoration, SIAM J. Sci. Comput., vol.27, 2, 622-645,
2005.
[4] I. Daubechies, M. Defriese, and C. De Mol, An iterative thresholding
algorithm for linear inverse problems with a sparsity constraint, Commun.
Pure Appl. Math., vol. LVII, pp. 14131457, 2004.
[5] J. Bioucas-Dias, and M. Figueiredo, A new TwIST: Two-step iterative
thresholding algorithm for image restoration, IEEE Trans. Imag. Process.,
vol. 16, no. 12, pp. 2992–3004, 2007.
Seminar Talk at ICMSEC, Chinese Academy of Sciences December 2nd, 2008 42
[6] Y. Wang, J. Yang, W. Yin and Y. Zhang, A new alternating minimization
algorithm for total variation image reconstruction, SIAM J. Imag. Sci., vol. 1,
no. 3, 248–272, 2008.
[7] J. Yang, W. Yin, Y. Zhang, and Y. Wang, A fast algorithm for edge-preserving
variational multichannel image restoration, TR08-09, CAAM, Rice University,
Submitted to SIIMS.
[8] J. Yang, Y. Zhang, and W. Yin, An efficient TVL1 algorithm for deblurring of
multichannel images corrupted by impulsive noise, TR08-12, CAAM, Rice
University, Submitted to SISC.
[9] J. Yang, Y. Zhang, and W. Yin, A fast TVL1-L2 minimization algorithm for
signal reconstruction from partial Fourier data, TR08-27, CAAM, Rice
University, Submitted to IEEE JSTSP.
Seminar Talk at ICMSEC, Chinese Academy of Sciences December 2nd, 2008 43
Codes available at:
http://www.caam.rice.edu/∼optimization/L1/ftvd
Acknowledgments
• Junfeng Yang has been supported by the Chinese Scholarship Council during
his visit to Rice University.
• Wotao Yin has been supported in part by ONR Grant N00014-08-1-1101 and
NSF CAREER Award DMS-0748839.
• Yin Zhang has been supported in part by NSF Grant DMS-0811188 and ONR
Grant N00014-08-1-1101.