A global approach Finding correspondence between a pair of epipolar lines for all pixels...
-
Upload
jonathan-daniel -
Category
Documents
-
view
215 -
download
0
description
Transcript of A global approach Finding correspondence between a pair of epipolar lines for all pixels...
![Page 1: A global approach Finding correspondence between a pair of epipolar lines for all pixels simultaneously Local method: no guarantee we will have one to.](https://reader035.fdocuments.in/reader035/viewer/2022062600/5a4d1b427f8b9ab0599a1918/html5/thumbnails/1.jpg)
A global approach• Finding correspondence between a pair of epipolar
lines for all pixels simultaneously
![Page 2: A global approach Finding correspondence between a pair of epipolar lines for all pixels simultaneously Local method: no guarantee we will have one to.](https://reader035.fdocuments.in/reader035/viewer/2022062600/5a4d1b427f8b9ab0599a1918/html5/thumbnails/2.jpg)
A global approach
left
right
left
right
left
right
Define an evaluation score for each configuration, choose the best matching configuration
![Page 3: A global approach Finding correspondence between a pair of epipolar lines for all pixels simultaneously Local method: no guarantee we will have one to.](https://reader035.fdocuments.in/reader035/viewer/2022062600/5a4d1b427f8b9ab0599a1918/html5/thumbnails/3.jpg)
A global approach• How to define the evaluation score?
• How about the sum of corresponding pixel difference?
![Page 4: A global approach Finding correspondence between a pair of epipolar lines for all pixels simultaneously Local method: no guarantee we will have one to.](https://reader035.fdocuments.in/reader035/viewer/2022062600/5a4d1b427f8b9ab0599a1918/html5/thumbnails/4.jpg)
Ordering constraint• Order of matching features usually the same
in both images• But not always: occlusion
![Page 5: A global approach Finding correspondence between a pair of epipolar lines for all pixels simultaneously Local method: no guarantee we will have one to.](https://reader035.fdocuments.in/reader035/viewer/2022062600/5a4d1b427f8b9ab0599a1918/html5/thumbnails/5.jpg)
Dynamic programming• Treat pixel correspondence as graph problem
Left imageLeft imagepixelspixels
Right image pixelsRight image pixels11 22 33 44
11
2233
44
![Page 6: A global approach Finding correspondence between a pair of epipolar lines for all pixels simultaneously Local method: no guarantee we will have one to.](https://reader035.fdocuments.in/reader035/viewer/2022062600/5a4d1b427f8b9ab0599a1918/html5/thumbnails/6.jpg)
Dynamic programming• Find min-cost path through graph
Left imageLeft imagepixelspixels
Right image pixelsRight image pixels11 22 33 44
11
2233
44
11
3344
11
22223344
![Page 7: A global approach Finding correspondence between a pair of epipolar lines for all pixels simultaneously Local method: no guarantee we will have one to.](https://reader035.fdocuments.in/reader035/viewer/2022062600/5a4d1b427f8b9ab0599a1918/html5/thumbnails/7.jpg)
Dynamic Programming Results
![Page 8: A global approach Finding correspondence between a pair of epipolar lines for all pixels simultaneously Local method: no guarantee we will have one to.](https://reader035.fdocuments.in/reader035/viewer/2022062600/5a4d1b427f8b9ab0599a1918/html5/thumbnails/8.jpg)
Energy minimization• Another global approach to improve quality of
correspondences• Assumption: disparities vary (mostly) smoothly• Minimize energy function:
Edata+Esmoothness
• Edata: how well does disparity match data
• Esmoothness: how well does disparity matchthat of neighbors – regularization
![Page 9: A global approach Finding correspondence between a pair of epipolar lines for all pixels simultaneously Local method: no guarantee we will have one to.](https://reader035.fdocuments.in/reader035/viewer/2022062600/5a4d1b427f8b9ab0599a1918/html5/thumbnails/9.jpg)
Stereo as energy minimization• Matching Cost Formulated as Energy
• “data” term penalizing bad matches
• “neighborhood term” encouraging spatial smoothness
),(),(),,( ydxyxdyxD JI
similar)something(or
d2 and d1 labels with pixelsadjacentofcost
21
21 ),(dd
ddV
)2,2(),1,1(
2,21,1),(
, ),(),,(})({yxyxneighbors
yxyxyx
yx ddVdyxDdE
![Page 10: A global approach Finding correspondence between a pair of epipolar lines for all pixels simultaneously Local method: no guarantee we will have one to.](https://reader035.fdocuments.in/reader035/viewer/2022062600/5a4d1b427f8b9ab0599a1918/html5/thumbnails/10.jpg)
Energy minimization
• Many local minimum• Why?• Gradient descent doesn’t work well
• In practice, disparities only piecewise smooth• Design smoothness function that doesn’t penalize
large jumps too much• Example: V()=min(||, K)
– Non-convex
)2,2(),1,1(
2,21,1),(
, ),(),,(})({yxyxneighbors
yxyxyx
yx ddVdyxDdE
![Page 11: A global approach Finding correspondence between a pair of epipolar lines for all pixels simultaneously Local method: no guarantee we will have one to.](https://reader035.fdocuments.in/reader035/viewer/2022062600/5a4d1b427f8b9ab0599a1918/html5/thumbnails/11.jpg)
Energy minimization• Hard to find global minima of non-smooth functions
• Many local minima• Provably NP-hard
• Practical algorithms look for approximate minima (e.g., simulated annealing)
![Page 12: A global approach Finding correspondence between a pair of epipolar lines for all pixels simultaneously Local method: no guarantee we will have one to.](https://reader035.fdocuments.in/reader035/viewer/2022062600/5a4d1b427f8b9ab0599a1918/html5/thumbnails/12.jpg)
Energy minimization via graph cuts
Labels (disparities)
d1
d2
d3
edge weight
edge weight
),,( 3dyxD
),( 11 ddV
![Page 13: A global approach Finding correspondence between a pair of epipolar lines for all pixels simultaneously Local method: no guarantee we will have one to.](https://reader035.fdocuments.in/reader035/viewer/2022062600/5a4d1b427f8b9ab0599a1918/html5/thumbnails/13.jpg)
• Graph Cost• Matching cost between images• Neighborhood matching term• Goal: figure out which labels are connected to which pixels
d1
d2
d3
Energy minimization via graph cuts
![Page 14: A global approach Finding correspondence between a pair of epipolar lines for all pixels simultaneously Local method: no guarantee we will have one to.](https://reader035.fdocuments.in/reader035/viewer/2022062600/5a4d1b427f8b9ab0599a1918/html5/thumbnails/14.jpg)
Energy minimization via graph cuts
d1
d2
d3
• Graph Cut• Delete enough edges so that
– each pixel is connected to exactly one label node • Cost of a cut: sum of deleted edge weights• Finding min cost cut equivalent to finding global minimum of
energy function
![Page 15: A global approach Finding correspondence between a pair of epipolar lines for all pixels simultaneously Local method: no guarantee we will have one to.](https://reader035.fdocuments.in/reader035/viewer/2022062600/5a4d1b427f8b9ab0599a1918/html5/thumbnails/15.jpg)
Computing a multiway cut
• With 2 labels: classical min-cut problem• Solvable by standard flow algorithms
– polynomial time in theory, nearly linear in practice• More than 2 terminals: NP-hard
[Dahlhaus et al., STOC ‘92]• Efficient approximation algorithms exist
• Yuri Boykov, Olga Veksler and Ramin Zabih, Fast Approximate Energy Minimization via Graph Cuts, International Conference on Computer Vision, September 1999.
• Within a factor of 2 of optimal• Computes local minimum in a strong sense
– even very large moves will not improve the energy
![Page 16: A global approach Finding correspondence between a pair of epipolar lines for all pixels simultaneously Local method: no guarantee we will have one to.](https://reader035.fdocuments.in/reader035/viewer/2022062600/5a4d1b427f8b9ab0599a1918/html5/thumbnails/16.jpg)
Move examples
Starting point
Red-blue swap move
Green expansion move
![Page 17: A global approach Finding correspondence between a pair of epipolar lines for all pixels simultaneously Local method: no guarantee we will have one to.](https://reader035.fdocuments.in/reader035/viewer/2022062600/5a4d1b427f8b9ab0599a1918/html5/thumbnails/17.jpg)
The swap move algorithm1. Start with an arbitrary labeling2. Cycle through every label pair (A,B) in some order
2.1 Find the lowest E labeling within a single AB-swap2.2 Go there if it’s lower E than the current labeling
3. If E did not decrease in the cycle, we’re done Otherwise, go to step 2
Original graph
A
B
AB subgraph(run min-cut on this graph)
B
A
![Page 18: A global approach Finding correspondence between a pair of epipolar lines for all pixels simultaneously Local method: no guarantee we will have one to.](https://reader035.fdocuments.in/reader035/viewer/2022062600/5a4d1b427f8b9ab0599a1918/html5/thumbnails/18.jpg)
The expansion move algorithm
1. Start with an arbitrary labeling2. Cycle through every label A in some order
2.1 Find the lowest E labeling within a single A-expansion
2.2 Go there if it’s lower E than the current labeling3. If E did not decrease in the cycle, we’re done Otherwise, go to
step 2
Multi-way cut A sequence of binary optimization problems
![Page 19: A global approach Finding correspondence between a pair of epipolar lines for all pixels simultaneously Local method: no guarantee we will have one to.](https://reader035.fdocuments.in/reader035/viewer/2022062600/5a4d1b427f8b9ab0599a1918/html5/thumbnails/19.jpg)
Stereo results
ground truthscene
• Data from University of Tsukuba
http://cat.middlebury.edu/stereo/
![Page 20: A global approach Finding correspondence between a pair of epipolar lines for all pixels simultaneously Local method: no guarantee we will have one to.](https://reader035.fdocuments.in/reader035/viewer/2022062600/5a4d1b427f8b9ab0599a1918/html5/thumbnails/20.jpg)
Results with window correlation
normalized correlation(best window size)
ground truth
![Page 21: A global approach Finding correspondence between a pair of epipolar lines for all pixels simultaneously Local method: no guarantee we will have one to.](https://reader035.fdocuments.in/reader035/viewer/2022062600/5a4d1b427f8b9ab0599a1918/html5/thumbnails/21.jpg)
Results with graph cuts
ground truthgraph cuts(Potts model E,expansion move algorithm)
![Page 22: A global approach Finding correspondence between a pair of epipolar lines for all pixels simultaneously Local method: no guarantee we will have one to.](https://reader035.fdocuments.in/reader035/viewer/2022062600/5a4d1b427f8b9ab0599a1918/html5/thumbnails/22.jpg)
Results with graph cuts
ground truthgraph cuts(Potts model E,expansion move algorithm)