Fast Global Stereo Matching Via Energy Pyramid Minimization
-
Upload
bruno-conejo-bconejogmailcom -
Category
Science
-
view
88 -
download
6
Transcript of Fast Global Stereo Matching Via Energy Pyramid Minimization
ISPRS – PCV 2014
Fast global matching via energy pyramid (disparity esAmaAon) Zurich, 9/5/2014 Bruno Conejo, Phd student ([email protected]) with S. Leprince, F. Ayoub & JP. Avouac (GPS, Caltech)
B.Conejo -‐ Fast global Matching via Energy Pyramid 2
Disparity is inversely propor?onal to depth!
Epipolar geometry stereo-‐imaging setup Introduc?on:
Reference Image Target Image
Disparity map
Given a stereo-‐pair of images (Ir ,It) how to retrieve the most probable disparity map d*?
Regulariza?on: priors on disparity
Matching: encourages similarity
In term of probability, we need to es?mate the Maximum A Posteriori (MAP) of:
B.Conejo -‐ Fast global Matching via Energy Pyramid 3
Modeling: a bayesian approach
Gibbs measure relates probability density func?on to energy:
Energy of configura?on x
Normalizing constant
Reference Image: Ir Target Image: It
From the Gibbs measure we relates probabili?es to the energies (EM , ER , E):
Matching: Similarity criteria (L1, L2, ZNCC, ...)
Regulariza?on: Piecewise constant prior:
Modulated by radiometric discon?nuity:
B.Conejo -‐ Fast global Matching via Energy Pyramid 4
Modeling: con?nuous Condi?onal Random Field (CRF)
First order Condi?onal Random Field (CRF):
p q
Associated graph
Reference Image
Set of nodes Set of edges
We need to globally op?mize a con?nuous CRF over all possible disparity maps (D):
However, this is a non-‐convex problem: varia?onal approaches can not work!
B.Conejo -‐ Fast global Matching via Energy Pyramid 5
Modeling: non-‐convexity
Many local minima!
Solu%on: Restrict d to take value in a finite discrete set, i.e., the “search space” encoded by a label space. This leads to globally op?mize a first order discrete CRF (s?ll NP-‐Hard) : -‐ Message passing (quadra?c w.r.t search space): Loopy BP, TRW-‐S, DD-‐MRF, … -‐ Making move (linear w.r.t search space) : α-‐exp, β-‐swap, Fast-‐PD, …
B.Conejo -‐ Fast global Matching via Energy Pyramid
Discrete op?miza?on
6
Pairwise term: Encodes prior (regulariza?on) Unary term: Encodes similarity (matching) Label space: Encodes for each node all poten?al disparity to evaluate
Mul?-‐scale approaches
We work with large images (30,000 by 30,000) and we have large disparity range (-‐300,300). A direct approach is inefficient (even impossible) and unnecessary! Locally the disparity range is “small”. We can use a mul?-‐scale approach: -‐ Coarsest scales: “large” dispari?es with low
spa?al frequencies (natural topography).
-‐ Finest scales: “small” dispari?es with high spa?al frequencies (man made objects).
Two mul?-‐scale schemes are possible: -‐ Image pyramid (classic, GM-‐IP algorithm). -‐ Energy pyramid (ours, GM-‐EP algorithm).
B.Conejo -‐ Fast global Matching via Energy Pyramid 7
Reference Image
Associated disparity
B.Conejo -‐ Fast global Matching via Energy Pyramid 8
Mul?-‐scale: GM-‐IP algorithm (Image Pyramid)
Build & Opt. CRF
Build & Opt. CRF
Ir It Algorithm 1) Build pyramid of image for each image by itera?ve downsampling 2) Compute and op?mize CRF at coarsest scale 3) Define new search space around current solu?on 4) Repeat (2-‐3) un?l finest scale
B.Conejo -‐ Fast global Matching via Energy Pyramid 9
Mul?-‐scale: GM-‐EP algorithm (Energy Pyramid)
Op?mize CRF
Op?mize CRF
Energy of CRF
◊©◊©◊
◊©◊©◊
Algorithm: 1) Compute CRF at finest scale 2) Build energy pyramid by
itera?ve downsampling 3) Op?mize CRF at coarsest
scale 4) Define new search space
around current solu?on 5) Repeat (3-‐4) un?l finest scale
B.Conejo -‐ Fast global Matching via Energy Pyramid 10
Mul?-‐scale: CRF sparsifica?on
Label before op?m. Label amer op?m. Label space to explore Removed label range
B.Conejo -‐ Fast global Matching via Energy Pyramid 11
Mul?-‐scale: GM-‐IP(Image Pyramid) vs GM-‐EP (Energy Pyramid)
The image pyramid yields a smoothed representa?on of the energy and destroys local minimums, especially at coarse scale:
Different minima! Energy pyramid Image pyramid
B.Conejo -‐ Fast global Matching via Energy Pyramid 12
Stereo-‐imaging in urban context: (Reference Image)
B.Conejo -‐ Fast global Matching via Energy Pyramid 13
Stereo-‐imaging in urban context: (Target Image)
B.Conejo -‐ Fast global Matching via Energy Pyramid 14
Stereo-‐imaging in urban context: (Reference Image)
B.Conejo -‐ Fast global Matching via Energy Pyramid 15
Stereo-‐imaging in urban context: Area of interest in reference image
Baseline = no pyramid (direct op@miza@on)
Quan?ta?ve comparison between algorithms:
Unary terms: ZNCC with 5x5 windows 4 scales, 1 itera?on per scale.
B.Conejo -‐ Fast global Matching via Energy Pyramid 16
Coarse scale: GM-‐EP (Energy Pyramid) vs GM-‐IP(Image Pyramid)
B.Conejo -‐ Fast global Matching via Energy Pyramid 17
Coarse scale: GM-‐EP (Energy Pyramid) vs GM-‐IP(Image Pyramid)
B.Conejo -‐ Fast global Matching via Energy Pyramid 18
Finest scale: GM-‐EP (Energy Pyramid) vs GM-‐IP(Image Pyramid)
B.Conejo -‐ Fast global Matching via Energy Pyramid 19
Finest scale: GM-‐EP (Energy Pyramid) vs GM-‐IP(Image Pyramid)
B.Conejo -‐ Fast global Matching via Energy Pyramid 20
GM-‐EP (Energy Pyramid) vs MicMac (Image Pyramid)
B.Conejo -‐ Fast global Matching via Energy Pyramid 21
GM-‐EP (Energy Pyramid) vs MicMac (Image Pyramid)
Key points: • A versa?le matching model efficiently op?mized with state of the art discrete
op?miza?on technique.
• Energy pyramid yields a beper representa?on of the energy. Future work: • Modeling:
• Impact of images’ noise, • Symmetry w.r.t. the images, • Occlusions, • CRF parameters (unary terms, weights of CRF, distance func?on).
• Op?miza?on • Auto defini?on of the search-‐space, • Mul?grid instead of mul?scale, • Paralleliza?on for shared and distributed memory architectures.
B.Conejo -‐ Fast global Matching via Energy Pyramid 22
Conclusion & Future work:
B.Conejo -‐ Fast global Matching via Energy Pyramid 23
B.Conejo -‐ Fast global Matching via Energy Pyramid 24
Stereo-‐imaging in urban context: (Reference Image)
B.Conejo -‐ Fast global Matching via Energy Pyramid 25
Stereo-‐imaging in urban context: (Target Image)
B.Conejo -‐ Fast global Matching via Energy Pyramid 26
Finest scale: GM-‐EP (Energy Pyramid) vs MicMac (Image Pyramid)
Micmac GM-‐EP (Energy Pyramid)
B.Conejo -‐ Fast global Matching via Energy Pyramid 27
Finest scale: GM-‐EP (Energy Pyramid) vs MicMac (Image Pyramid)
B.Conejo -‐ Fast global Matching via Energy Pyramid 28
Mul?-‐scale: GM-‐IP algorithm (Image Pyramid)
B.Conejo -‐ Fast global Matching via Energy Pyramid 29
Mul?-‐scale: GM-‐EP algorithm (Energy Pyramid)