Multigrid methods for variational approaches in image ... · Multigrid methods for variational...
Transcript of Multigrid methods for variational approaches in image ... · Multigrid methods for variational...
Multigrid methods for variational approaches in imageprocessing
H. KostlerFriedrich-Alexander Universitat Erlangen-Nurnberg,Germany
January 15, 2007
H. Kostler, Universitat Erlangen-Nurnberg () MG for variational approaches January 15, 2007 1 / 31
Contents
1 Multigrid framework for variational approaches
2 A curvature based optical flow regularizerOptical flow modelExperimental results
3 A variational approach for video codingVideo CompressionVideo DecompressionExperimental results
4 Medical image segmentationModelExperimental results
5 Future work
H. Kostler, Universitat Erlangen-Nurnberg () MG for variational approaches January 15, 2007 2 / 31
Multigrid framework for variational approaches
Goals and applications
Many efficient implementations for different variational image processingproblems exist, we try to implement a software package to support thedevelopment and efficient implementation of new variational approaches.Applications:
Optical flow
Non-rigid image registration
Tomographic image reconstruction
Image Inpainting
Image segmentation
Video compression
Motion Blur
Image denoising
H. Kostler, Universitat Erlangen-Nurnberg () MG for variational approaches January 15, 2007 3 / 31
Multigrid framework for variational approaches
Problem formulation
General problem
We try to minimize the energy functional
E [u] := D[I,u]︸ ︷︷ ︸data term
+ α S[u]︸︷︷︸regularizer
(1)
with weighting parameter α ∈ R+. We regularize the problem to get aunique solution.
H. Kostler, Universitat Erlangen-Nurnberg () MG for variational approaches January 15, 2007 4 / 31
Multigrid framework for variational approaches
Multigrid framework I
Parallel C++ Code
Based on a robust and efficient multigrid solver
Figure: Schematic application flow.
H. Kostler, Universitat Erlangen-Nurnberg () MG for variational approaches January 15, 2007 5 / 31
Multigrid framework for variational approaches
Multigrid framework II: Features
Different smoothers
Matrix dependent transfer operators
Cell-centered, node-based and staggered grids
Parallelization in 2D and 3D
Iterant recombination
Galerkin or standard coarsening
Hardware optimized solver components for special cases
H. Kostler, Universitat Erlangen-Nurnberg () MG for variational approaches January 15, 2007 6 / 31
A curvature based optical flow regularizer Optical flow model
Optical flow problem
Figure: The optical flow at the pixel (x,y) is the 2D-velocity vector (anapproximation to the real motion) (u, v) = ( dx
dt , dydt ).
H. Kostler, Universitat Erlangen-Nurnberg () MG for variational approaches January 15, 2007 7 / 31
A curvature based optical flow regularizer Optical flow model
Variational approach I
Ek(u) :=
∫Ω(Ixu + Iyv + It)
2 + αSk(u)dx (2)
Data term is brightness constancy assumption. Regularizers are diffusion(Horn and Schunck)
S1(u) = ‖∇u‖2 + ‖∇v‖2
or curvatureS2(u) = (∆u)2 + (∆v)2
that is a special case of the div-curl based regularizer [GP96]
S2′(u) = α1‖∇divu‖2 + α2‖∇curlu‖2 ,
where α1 = α2 = 1.
H. Kostler, Universitat Erlangen-Nurnberg () MG for variational approaches January 15, 2007 8 / 31
A curvature based optical flow regularizer Optical flow model
Variational approach II
The resulting Euler-Lagrange equations constituting a necessary conditionfor a minimum of Ek(u), k ∈ 1, 2 are
−α∆ku + Ix(Ixu + Iyv + It) = 0 (3a)
−α∆kv + Iy (Ixu + Iyv + It) = 0 (3b)
with natural Neumann boundary conditions.
H. Kostler, Universitat Erlangen-Nurnberg () MG for variational approaches January 15, 2007 9 / 31
A curvature based optical flow regularizer Optical flow model
Curvature based regularizer: Why?
Many different regularizers exist for optical flow, why do we introduceanother one?
Could be useful for moving fluids, can be extended to div-curlregularizer
Necessary for proper mathematical treatment of landmarks in opticalflow models (e.g. for Motion Blur)
H. Kostler, Universitat Erlangen-Nurnberg () MG for variational approaches January 15, 2007 10 / 31
A curvature based optical flow regularizer Optical flow model
Curvature based regularizer: Implementation
For the curvature based regularizer we introduce a variable w = ∆u andafter discretization of the Euler-Lagrange equations one has to solve thelinear system
Lh(x)
uh(x)vh(x)w1
h (x)w2
h (x)
=
00
−Ix(x, t)It(x, t)−Iy (x, t)It(x, t)
with x ∈ Ωh ,
Lh(x) =
−(1− β)∆h − β 0 1 0
0 −(1− β)∆h − β 0 1I 2x (x, t) Ix(x, t)Iy (x, t) −α∆h 0
Ix(x, t)Iy (x, t) I 2y (x, t) 0 −α∆h
.
H. Kostler, Universitat Erlangen-Nurnberg () MG for variational approaches January 15, 2007 11 / 31
A curvature based optical flow regularizer Optical flow model
Curvature based regularizer: Extensions
Affine transformations are in the kernel of the curvature regularizer
Combine diffusion and curvature regularizer using β ∈ [0, . . . , 1]
S3(u) = βS1(u) + (1− β)S2(u) .
Isotropic curvature regularizer
S4(u) = g(|∇I |2)((∆u)2 + (∆v)2
)with
g(s2) =1
2√
s2 + ε2.
H. Kostler, Universitat Erlangen-Nurnberg () MG for variational approaches January 15, 2007 12 / 31
A curvature based optical flow regularizer Experimental results
Moving Rectangle Results
Figure: Rectangle moving to the right.
Resulting velocity field of diffusion regularizer (Horn and Schunck)Resulting velocity field of curvature regularizer
H. Kostler, Universitat Erlangen-Nurnberg () MG for variational approaches January 15, 2007 13 / 31
A curvature based optical flow regularizer Experimental results
Yosemite Results I
Figure: Frame 8 of Yosemite sequence with clouds (where the brightnessconstancy assumption is not fulfilled) and ground truth motion field (frames 8–9).
H. Kostler, Universitat Erlangen-Nurnberg () MG for variational approaches January 15, 2007 14 / 31
A curvature based optical flow regularizer Experimental results
Yosemite Results II
50
55
60
65
70
75
140 145 150 155 160 165 170 175 180 185
50
55
60
65
70
75
140 145 150 155 160 165 170 175 180 185
Figure: Optical flow for Frames 8 and 9 from Yosemite sequence using diffusion(left) and combined (β = 0.4) (right) regularizer.
H. Kostler, Universitat Erlangen-Nurnberg () MG for variational approaches January 15, 2007 15 / 31
A curvature based optical flow regularizer Experimental results
Yosemite Results III
50
55
60
65
70
75
140 145 150 155 160 165 170 175 180 185
50
55
60
65
70
75
140 145 150 155 160 165 170 175 180 185
Figure: Optical flow for Frames 8 and 9 from Yosemite sequence usinghomogeneous (left) and isotropic (right) curvature regularizer.
H. Kostler, Universitat Erlangen-Nurnberg () MG for variational approaches January 15, 2007 16 / 31
A curvature based optical flow regularizer Experimental results
Yosemite Results IV: Average Angular error
8.6
8.8
9
9.2
9.4
9.6
9.8
10
10.2
10.4
10.6
0 0.2 0.4 0.6 0.8 1
AA
E
beta
AAE for alpha=500AAE for alpha=1500AAE for alpha=5000
Figure: AAE plot of the calculated optical flow between pictures 8 and 9 from theYosemite sequence for α = 500, α = 1500 and α = 5000.
H. Kostler, Universitat Erlangen-Nurnberg () MG for variational approaches January 15, 2007 17 / 31
A variational approach for video coding Video Compression
Compression scheme
Idea for video compression [GWW+05]
Some points are selected from each frame, such that they are sufficient fora good reconstruction of the original video sequence. We call these pointslandmarks. The rest of the points is dropped.
B-Tree Triangular Coding: fast recursive subdivision scheme todetermine landmarks
Huffman coding to store values
We do NOT (so far):
Vector quantization of colorsUse temporal information
H. Kostler, Universitat Erlangen-Nurnberg () MG for variational approaches January 15, 2007 18 / 31
A variational approach for video coding Video Decompression
Decompression scheme I
Solve(1− c(x))Lu − c(x)(u − f ) = δtu (4)
c(x) =
1 x ∈ Ω1
0 else(5)
f (x) =
v(x) x ∈ Ω1
0 else(6)
From these equations it follows that for every point x ∈ Ω1: u = v = f .At every other point x ∈ Ω \ Ω1 the differential equation Lu = 0 has to besolved.
H. Kostler, Universitat Erlangen-Nurnberg () MG for variational approaches January 15, 2007 19 / 31
A variational approach for video coding Video Decompression
Decompression scheme II
For the regularizer L we use homogeneous diffusion (HD)
Lu := ∆u , (7)
nonlinear isotropic diffusion (NID)
Lu = div(g(|∇u|2)∇u) (8)
and edge enhancing diffusion (EED)
Lu = div(g(∇uσ∇Tuσ
)∇u). (9)
H. Kostler, Universitat Erlangen-Nurnberg () MG for variational approaches January 15, 2007 20 / 31
A variational approach for video coding Experimental results
Compression Results I
Figure: One Frame from the original video sequence with marked parts forzooming showing water dropped in a glass.
H. Kostler, Universitat Erlangen-Nurnberg () MG for variational approaches January 15, 2007 21 / 31
A variational approach for video coding Experimental results
Compression Results II
Figure: Improvement achieved by BTTC based selection. Original (UL), BTTC8% (UR), random 8% (LR), random 20 % (LL).
H. Kostler, Universitat Erlangen-Nurnberg () MG for variational approaches January 15, 2007 22 / 31
A variational approach for video coding Experimental results
Compression Results III
K 1
.1
I 0.3
5K
1.1
I 0
.50
K 1
.1
I 0.9
3K
1.7
I 0
.93
K 2
.1
I 1.2
3K
2.7
I 1
.49
K 3
.2
I 2.0
0m
peg
mjp
eg
0
500
1000
1500
File
size
in K
b
Figure: Filesizes of Mpeg, Mjpeg and Pdevc - K: Keyframe Bit per pixel, I: Bitper pixel in inner frames.
H. Kostler, Universitat Erlangen-Nurnberg () MG for variational approaches January 15, 2007 23 / 31
A variational approach for video coding Experimental results
Decompression Results
Figure: Comparison between homogeneous, nonlinear isotropic and nonlinearanisotropic (edge enhancing) diffusion model.
H. Kostler, Universitat Erlangen-Nurnberg () MG for variational approaches January 15, 2007 24 / 31
A variational approach for video coding Experimental results
Decompression Runtimes
Regularizer Gauss-Seidel Multigrid
HD 1.2 3.7NID 0.05 0.55EED 0.03 0.43
Table: Comparison of decompression times (in fps) on a Pentium M, 2 GHzLaptop for different regularizers using Gauss-Seidel or multigrid iteration for aresolution of 320× 240 pixels.
An optimistic estimate for the maximum speed of the HD regularizer is 50fps measured for a similar setting on a Core2 Duo, 2.4 GHz (Conroe) usinga hardware optimized multigrid solver.
H. Kostler, Universitat Erlangen-Nurnberg () MG for variational approaches January 15, 2007 25 / 31
A variational approach for video coding Experimental results
Video quality
(a) MPEG 0.245 bpp (b) PDEVC 2.2 bpp
Figure: Image quality for Mpeg and Pdevc (HD).
H. Kostler, Universitat Erlangen-Nurnberg () MG for variational approaches January 15, 2007 26 / 31
Medical image segmentation Model
Stochastic model I
Idea for image segmentation based on random walks [Gra06]
First landmarks are set manually for presegmentation. Then an isotropicdiffusion is used to determine the segmented regions. The model isstochastic, the solution we compute gives the probability for each point tobelong to a certain region.
H. Kostler, Universitat Erlangen-Nurnberg () MG for variational approaches January 15, 2007 27 / 31
Medical image segmentation Model
Stochastic model II
The energy functional (Dirichlet integral)
ED [u] =1
2
∫Ω|∇u|2dx (10)
has the combinatorical formulation
ED [u] =1
2xTLx =
∑eij∈E
wij(xi − xj)2 (11)
with wij = e−β(I (xi )−I (xj )). L is a combinatorical Laplacian matrix, eij areedges of a graph that corresponds to the discretization grid. Note thesimilarity to a discretized isotropic linear diffusion model.
H. Kostler, Universitat Erlangen-Nurnberg () MG for variational approaches January 15, 2007 28 / 31
Medical image segmentation Experimental results
Results on medical images
Figure: Half-automatic segmentation result using 5 V(2,2)-cycles (right) for anabdominal scan with marked region (left).
H. Kostler, Universitat Erlangen-Nurnberg () MG for variational approaches January 15, 2007 29 / 31
Future work
Future work
Further evaluate curvature based optical flow regularizer
Improve compression scheme for video coding, evaluate practical use
Hardware optimized multigrid implementation to achieve real timevideo decompression and image segmentation
Next project: Medical image denoising given several noisy images ofthe same object
H. Kostler, Universitat Erlangen-Nurnberg () MG for variational approaches January 15, 2007 30 / 31
Future work
Related Literature
S. Gupta and J. Prince.Stochastic models for div-curl optical flow methods.IEEE Signal Processing Letters, 3(2):32–34, 1996.
L. Grady.Random Walks for Image Segmentation.IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINEINTELLIGENCE, 28(11):1, 2006.
I. Galic, J. Weickert, M. Welk, A. Bruhn, A. Belyaev, and H.P. Seidel.Towards PDE-based image compression.Proceedings of Variational, geometric, and level set methods incomputer vision, Lecture notes in computer science, pages 37–48,2005.
H. Kostler, Universitat Erlangen-Nurnberg () MG for variational approaches January 15, 2007 31 / 31