Learned D-AMP: Principled Neural Network basedCompressive Image Recovery
Christopher A. Metzler, Ali Mousavi, Richard G. Baraniuk
Rice University
NIPS 2017
Presenter: Guoyin Wang
January 4, 2019
Christopher A. Metzler, Ali Mousavi, Richard G. Baraniuk (Rice)LD-AMP January 4, 2019 1 / 22
Overview
1 Major Contribution
2 ModelProblem DescriptionAlgorithmsTraining Network
3 Experiments
Christopher A. Metzler, Ali Mousavi, Richard G. Baraniuk (Rice)LD-AMP January 4, 2019 2 / 22
Major Contribution
Unrolling: a process to turn an iterative algorithm (D-AMP) into adeep neural net (LD-AMP) which is interpretable and maintainconvergence guarantees.Efficiently train a deep neural network.Outperform the state-of-the-art BM3D-AMP and NLR-CS algorithmsin terms of both accuracy and run time on compressive imagingproblem.
Christopher A. Metzler, Ali Mousavi, Richard G. Baraniuk (Rice)LD-AMP January 4, 2019 3 / 22
Compressive Imaging Problem
Linear measurements y ∈ Rm of the signal of interest x ∈ Rn iscaptured via y = Ax + ε, where A ∈ Rm×n is a measurement matrixand ε ∈ Rm is noise.Image Inverse Problems: given the measurements y and themeasurement matrix A, a computational imaging system seeks torecover x .Compressive Sensing (CS): when m < n this problem is ill-posedinverse problem, and prior knowledge about x must be used torecovery the signal, such as fact that x ∈ C , where C is the set of allnatural images.
Christopher A. Metzler, Ali Mousavi, Richard G. Baraniuk (Rice)LD-AMP January 4, 2019 4 / 22
Compressive Imaging Problem
When no measurement noise ε presented, we want to find the imagexo at the intersection of the set C and the affine subspace{x |y = Ax}The optimization formulation:
argminx‖y − Ax‖22 subject to x ∈ C . (1)
Christopher A. Metzler, Ali Mousavi, Richard G. Baraniuk (Rice)LD-AMP January 4, 2019 5 / 22
IT and AMP
Low cost iterative algorithm are proposed to solve this problem:
Iterative Thresholding (IT) Algorithm
IT Algorithmz t = y − Ax t ,
x t+1 = η(x t + AHz t).η(·) = Thresholding non-linearity (2)
Christopher A. Metzler, Ali Mousavi, Richard G. Baraniuk (Rice)LD-AMP January 4, 2019 6 / 22
IT and AMP
Effective noise νt = x t + AHz t − xo should be i.i.d Gaussian, but ITAlgorithm fails to maintain this property.
Approximate Message Passing (AMP) Algorithm
AMP Algorithm
bt = nz t−1〈η′(x t−1 + AHz t−1)〉m ,
z t = y − Ax t + bt ,
x t+1 = η(x t + AHz t). (3)
where 〈·〉 denotes the average of a vector, η′ represents the derivative of ηand b is Onsager correction term which removes the bias.
Christopher A. Metzler, Ali Mousavi, Richard G. Baraniuk (Rice)LD-AMP January 4, 2019 7 / 22
D-IT and D-AMP
Nonlinear function η can be replaced by a powerful denoiser.Say xo + σz is a noisy observation of a natural image, with xo ∈ C ,z ∼ N(0, I) and σ as the standard deviation of the noise.Denoiser Dσ would simply find the point in the set C that is closestto the observation xo + σz
Dσ(xo + σz) = argminx‖xo + σz − x‖22 subject to x ∈ C . (4)
Such denoiser is a projection onto C , ideally should return estimate xcloser to xo than xo + σz .
Christopher A. Metzler, Ali Mousavi, Richard G. Baraniuk (Rice)LD-AMP January 4, 2019 8 / 22
D-IT and D-AMP
Denoising-based IT (D-IT) Algorithm
D-IT Algorithm z t = y − Ax t ,
x t+1 = Dσ̂t (x t + AHz t). (5)
where σ̂t is the estimated standard deviation of effective noise νt . Notethat
Dσ̂t (x t + AHz t) = Dσ̂t (xo + νt) (6)
Christopher A. Metzler, Ali Mousavi, Richard G. Baraniuk (Rice)LD-AMP January 4, 2019 9 / 22
D-IT and D-AMP
Denoising-based AMP (D-AMP) Algorithm
D-AMP Algorithmbt = z t−1divDσ̂t−1(x t−1 + AHz t−1)
m ,
z t = y − Ax t + bt ,
σ̂t = ‖z t‖2√m ,
x t+1 = Dσ̂t (x t + AHz t). (7)
‖zt‖2√m serves as a useful and accurate estimate of the standard deviation of
νt and typically, D-AMP algorithms use a Monte-Carlo approximation forthe divergence divD(·).
Christopher A. Metzler, Ali Mousavi, Richard G. Baraniuk (Rice)LD-AMP January 4, 2019 10 / 22
D-IT and D-AMP
(a) D-IT Iterations (b) D-AMP Iterations
Figure: Reconstruction behavior of D-IT (left) and D-AMP (right) with anidealized denoiser. Because D-IT allows bias to build up over iterations of thealgorithm, its denoiser becomes ineffective at projecting onto the set C of allnatural images. The Onsager correction term enables D-AMP to avoid this issue.
Christopher A. Metzler, Ali Mousavi, Richard G. Baraniuk (Rice)LD-AMP January 4, 2019 11 / 22
Unrolling Process
Example: IT Algorithm:
IT Algorithmz t = y − Ax t ,
x t+1 = η(x t + AHz t).η(·) = Thresholding non-linearity
Christopher A. Metzler, Ali Mousavi, Richard G. Baraniuk (Rice)LD-AMP January 4, 2019 12 / 22
Unrolling Process
z t = y − Ax t ,
x t+1 = η(x t + AHz t).z t+1 = y − Ax t+1,
x t+2 = η(x t+1 + AHz t+1).
Feed training data, i.e., (xo, yo) pairs as label-input pairs, feedsforward through the network, calculate errors and backpropagate.The free parameters of η are leaned during training.
Christopher A. Metzler, Ali Mousavi, Richard G. Baraniuk (Rice)LD-AMP January 4, 2019 13 / 22
Unroll D-AMP
Unroll D-AMP
bl =z l−1divDl
σ̂l−1(x l−1 + AHz l−1)m ,
z l = y − Ax l + bl ,
σ̂l = ‖z l‖2√m ,
x l+1 = Dlσ̂l (x l + AHz l). (8)
Christopher A. Metzler, Ali Mousavi, Richard G. Baraniuk (Rice)LD-AMP January 4, 2019 14 / 22
CNN-based Denoiser
A denoiser that easily propagates gradients is needed.Denoising Convolutional Neural Network (DnCNN) satisfies this andis more accurate and far faster than competing techniques like BM3D.
Consists of 16 to 20 CNN layers.First layer uses 64 3× 3× c filters (c is number of color channels).Next 14 to 18 layers use 64 3× 3× 64 filters + batch normalization.Final layer uses c 3× 64 filters to reconstruct.
Christopher A. Metzler, Ali Mousavi, Richard G. Baraniuk (Rice)LD-AMP January 4, 2019 15 / 22
Learned D-AMP
LDAMP Neural Network
bl =z l−1divDl
w l−1(σ̂l−1)(xl−1 + AHz l−1)
m ,
z l = y − Ax l + bl ,
σ̂l = ‖z l‖2√m ,
x l+1 = Dlw l (σ̂l )(x
l + AHz l). (9)
where Dlw l (σ̂l ) to indicate that layer l of the network uses denoiser Dl , that
this denoiser depends on its weights/biases w l , and that these weightsmay be a function of the estimated standard deviation of the noise σ̂l .
Christopher A. Metzler, Ali Mousavi, Richard G. Baraniuk (Rice)LD-AMP January 4, 2019 16 / 22
Leaned D-AMP
Figure: Two layers of the LDAMP neural network. When used with the DnCNNdenoiser, each denoiser block is a 16 to 20 convolutional-layer neural network.
LD-IT network is nearly identical but does not compute Onsagercorrection term.
Christopher A. Metzler, Ali Mousavi, Richard G. Baraniuk (Rice)LD-AMP January 4, 2019 17 / 22
Training LDIT and LDAMP
End-to-end training: Train all the weights of the networksimultaneously.Layer-by-layer training: Train a 1 AMP layer network to recover thesignal, fix these weights, add an AMP layer, and repeat until we havetrained a 10 layer network.Denoiser-by-denoiser training: Decouple the denoisers from therest of the network and train each on denoising problems at differentnoise levels.Later-by-layer training and denoiser-by-denoiser training for LDAMPare found minimum-mean-squared-error (MMSE) optimal.
Christopher A. Metzler, Ali Mousavi, Richard G. Baraniuk (Rice)LD-AMP January 4, 2019 18 / 22
Experiments
Datasets Berkeley’s BSD-500 dataset, 400 images for training, 50 forvalidation, and 50 for testing.Evaluation Metric PSNR = 10 log10( 2552
mean((x̂−xo)2)) when the pixelrange is 0 to 255.
Christopher A. Metzler, Ali Mousavi, Richard G. Baraniuk (Rice)LD-AMP January 4, 2019 19 / 22
Results
(a) Original Image (b) TVAL3 (26.4 dB, 6.85 sec) (c) BM3D-AMP (27.2 dB,75.04 sec)
(d) LDAMP (28.1 dB, 1.22sec)
Figure: Reconstructions of 512× 512 Boat test image sampled at a rate ofmn = 0.05 using coded diffraction pattern measurements and no measurementnoise. LDAMP’s reconstructions are noticeably cleaner and far faster than thecompeting methods.
Christopher A. Metzler, Ali Mousavi, Richard G. Baraniuk (Rice)LD-AMP January 4, 2019 20 / 22
Results
Table: PSNRs and run times (sec) of 128× 128 reconstructions withi.i.d. Gaussian measurements and no measurement noise at various sampling rates.
Methodmn = 0.10 m
n = 0.15 mn = 0.20 m
n = 0.25
PSNR Time PSNR Time PSNR Time PSNR Time
TVAL3 21.5 2.2 22.8 2.9 24.0 3.6 25.0 4.3BM3D-AMP 23.1 4.8 25.1 4.4 26.6 4.2 27.9 4.1LDIT 20.1 0.3 20.7 0.4 21.1 0.4 21.7 0.5LDAMP 23.7 0.4 25.7 0.5 27.2 0.5 28.5 0.6NLR-CS 23.2 85.9 25.2 104.0 26.8 124.4 28.2 146.3
Christopher A. Metzler, Ali Mousavi, Richard G. Baraniuk (Rice)LD-AMP January 4, 2019 21 / 22
Results
Table: PSNRs and run times (sec) of 128× 128 reconstructions with codeddiffraction measurements and no measurement noise at various sampling rates.
Methodmn = 0.10 m
n = 0.15 mn = 0.20 m
n = 0.25
PSNR Time PSNR Time PSNR Time PSNR Time
TVAL3 24.0 0.52 26.0 0.46 27.9 0.43 29.7 0.41BM3D-AMP 23.8 4.55 25.7 4.29 27.5 3.67 29.1 3.40LDIT 22.9 0.14 25.6 0.14 27.4 0.14 28.9 0.14LDAMP 25.3 0.26 27.4 0.26 28.9 0.27 30.5 0.26NLR-CS 21.6 87.82 22.8 87.43 25.1 87.18 26.4 86.87
Christopher A. Metzler, Ali Mousavi, Richard G. Baraniuk (Rice)LD-AMP January 4, 2019 22 / 22
Top Related