Stereo Computation using Iterative Graph-Cuts Vision Course - Weizmann Institute.
-
date post
21-Dec-2015 -
Category
Documents
-
view
216 -
download
1
Transcript of Stereo Computation using Iterative Graph-Cuts Vision Course - Weizmann Institute.
![Page 1: Stereo Computation using Iterative Graph-Cuts Vision Course - Weizmann Institute.](https://reader036.fdocuments.in/reader036/viewer/2022062421/56649d605503460f94a41f82/html5/thumbnails/1.jpg)
Stereo Computationusing
Iterative Graph-Cuts
Vision Course - Weizmann Institute
![Page 2: Stereo Computation using Iterative Graph-Cuts Vision Course - Weizmann Institute.](https://reader036.fdocuments.in/reader036/viewer/2022062421/56649d605503460f94a41f82/html5/thumbnails/2.jpg)
The “Binocular” Stereo problem
Two views of the same scene from a slightly different point of view
“Left”
Camera
“Right”
Camera
![Page 3: Stereo Computation using Iterative Graph-Cuts Vision Course - Weizmann Institute.](https://reader036.fdocuments.in/reader036/viewer/2022062421/56649d605503460f94a41f82/html5/thumbnails/3.jpg)
The stereo problem
•Both images are very similar (like images that you see with your two eyes)
• Most of the pixels in the left image are present in the right image (except for few occlusions)
![Page 4: Stereo Computation using Iterative Graph-Cuts Vision Course - Weizmann Institute.](https://reader036.fdocuments.in/reader036/viewer/2022062421/56649d605503460f94a41f82/html5/thumbnails/4.jpg)
Rectification
• Image Reprojection– reproject image planes onto
common plane parallel to baseline
![Page 5: Stereo Computation using Iterative Graph-Cuts Vision Course - Weizmann Institute.](https://reader036.fdocuments.in/reader036/viewer/2022062421/56649d605503460f94a41f82/html5/thumbnails/5.jpg)
After rectification: all correspondences are along the same horizontal scan lines
The stereo problem
(pixels in one image simply shift horizontally in the other image)
![Page 6: Stereo Computation using Iterative Graph-Cuts Vision Course - Weizmann Institute.](https://reader036.fdocuments.in/reader036/viewer/2022062421/56649d605503460f94a41f82/html5/thumbnails/6.jpg)
The relation between depth and disparities
• Origin at midpoint between camera centers
• Axes parallel to those of the two (rectified) cameras
Depth and Disparity are inverse proportional
![Page 7: Stereo Computation using Iterative Graph-Cuts Vision Course - Weizmann Institute.](https://reader036.fdocuments.in/reader036/viewer/2022062421/56649d605503460f94a41f82/html5/thumbnails/7.jpg)
The stereo problem
•The horizontal shifts between the images are sometimes called: “disparities”
•The Disparities are related to depth: Closer objects have larger disparities
![Page 8: Stereo Computation using Iterative Graph-Cuts Vision Course - Weizmann Institute.](https://reader036.fdocuments.in/reader036/viewer/2022062421/56649d605503460f94a41f82/html5/thumbnails/8.jpg)
The stereo problem: compute the disparity map between two
images
![Page 9: Stereo Computation using Iterative Graph-Cuts Vision Course - Weizmann Institute.](https://reader036.fdocuments.in/reader036/viewer/2022062421/56649d605503460f94a41f82/html5/thumbnails/9.jpg)
Traditional Approaches• Matching small windows around each pixel• Each window is matched independently Modern approaches• Finding coherent correspondences for all pixels
- “Graph cuts”- “Belief Propagation”
![Page 10: Stereo Computation using Iterative Graph-Cuts Vision Course - Weizmann Institute.](https://reader036.fdocuments.in/reader036/viewer/2022062421/56649d605503460f94a41f82/html5/thumbnails/10.jpg)
Window-Based Approach
•Compute a cost for each location
•Location with the lowest cost wins
![Page 11: Stereo Computation using Iterative Graph-Cuts Vision Course - Weizmann Institute.](https://reader036.fdocuments.in/reader036/viewer/2022062421/56649d605503460f94a41f82/html5/thumbnails/11.jpg)
General Problem : Ambiguity
Left Right
scanline
![Page 12: Stereo Computation using Iterative Graph-Cuts Vision Course - Weizmann Institute.](https://reader036.fdocuments.in/reader036/viewer/2022062421/56649d605503460f94a41f82/html5/thumbnails/12.jpg)
Window-Based Approach
Small Window Large Window
noisy in low texture areas blurred boundaries
![Page 13: Stereo Computation using Iterative Graph-Cuts Vision Course - Weizmann Institute.](https://reader036.fdocuments.in/reader036/viewer/2022062421/56649d605503460f94a41f82/html5/thumbnails/13.jpg)
Results with best window size(still not good enough)
Window-based matching(best window size)
Ground truth
![Page 14: Stereo Computation using Iterative Graph-Cuts Vision Course - Weizmann Institute.](https://reader036.fdocuments.in/reader036/viewer/2022062421/56649d605503460f94a41f82/html5/thumbnails/14.jpg)
Graph Cuts
Ground truthGraph cuts
![Page 15: Stereo Computation using Iterative Graph-Cuts Vision Course - Weizmann Institute.](https://reader036.fdocuments.in/reader036/viewer/2022062421/56649d605503460f94a41f82/html5/thumbnails/15.jpg)
Maximum flow problem
• Max flow problem:– Each edge is a “pipe”– Find the largest flow F of
“water” that can be sent from the “source” to the “sink” along the pipes
– Edge weights give the pipe’s capacity
“source”
A graph with two terminals
S T
“sink”
flow F
![Page 16: Stereo Computation using Iterative Graph-Cuts Vision Course - Weizmann Institute.](https://reader036.fdocuments.in/reader036/viewer/2022062421/56649d605503460f94a41f82/html5/thumbnails/16.jpg)
Minimum cut problem
• Min cut problem:– Find the cheapest way
to cut the edges so that the “source” is completely separated from the “sink”
– Edge weights now represent cutting “costs”
a cut C
“source”
A graph with two terminals
S T
“sink”
![Page 17: Stereo Computation using Iterative Graph-Cuts Vision Course - Weizmann Institute.](https://reader036.fdocuments.in/reader036/viewer/2022062421/56649d605503460f94a41f82/html5/thumbnails/17.jpg)
Max flow/Min cut theorem
• Max Flow = Min Cut:– Maximum flow saturates the
edges along the minimum cut.
– Ford and Fulkerson, 1962– Problem reduction!
• Ford and Fulkerson gave first polynomial time algorithm for globally optimal solution
“source”
A graph with two terminals
S T
“sink”
![Page 18: Stereo Computation using Iterative Graph-Cuts Vision Course - Weizmann Institute.](https://reader036.fdocuments.in/reader036/viewer/2022062421/56649d605503460f94a41f82/html5/thumbnails/18.jpg)
Min-Cut: Important Rule
No subset of the cut can also be a cut
This is not a minimal cut
![Page 19: Stereo Computation using Iterative Graph-Cuts Vision Course - Weizmann Institute.](https://reader036.fdocuments.in/reader036/viewer/2022062421/56649d605503460f94a41f82/html5/thumbnails/19.jpg)
Energy Minimization Using Iterative Graph cuts
Fast Approximate Energy Minimization via Graph Cuts
Yuri Boykov, Olga Veksler and Ramin Zabih
Pami 2001More papers, code: http://www.cs.cornell.edu/~rdz/graphcuts.html
![Page 20: Stereo Computation using Iterative Graph-Cuts Vision Course - Weizmann Institute.](https://reader036.fdocuments.in/reader036/viewer/2022062421/56649d605503460f94a41f82/html5/thumbnails/20.jpg)
To do better we need a better model of images
• We can make reasonable assumptions about the surfaces in the world
• Usually assume that the surfaces are smooth• Can pose the problem of finding the
corresponding points as an energy (or cost) minimization:
how well the pixels match up for different
disparities
neighboring pixels have similar disparities
)()()( fEfEfE smoothdata f - assignment
![Page 21: Stereo Computation using Iterative Graph-Cuts Vision Course - Weizmann Institute.](https://reader036.fdocuments.in/reader036/viewer/2022062421/56649d605503460f94a41f82/html5/thumbnails/21.jpg)
),(
),( ),()()(qp
qpqp
ppp ffVfDfE
To do better we need a better model of images
• We can make reasonable assumptions about the surfaces in the world
• Usually assume that the surfaces are smooth• Can pose the problem of finding the
corresponding points as an energy (or cost) minimization:
Data term is calculated for
each pixels
Smoothness is calculated on
neighbor pixels
f - assignment
p,q - pixels
![Page 22: Stereo Computation using Iterative Graph-Cuts Vision Course - Weizmann Institute.](https://reader036.fdocuments.in/reader036/viewer/2022062421/56649d605503460f94a41f82/html5/thumbnails/22.jpg)
Example for Smoothness terms
• Quadratic
• L1
• Truncated L1
• Potts model )(),( qpqp ffKffv
),min( Kff qp
qp ff
2)( qp ff
![Page 23: Stereo Computation using Iterative Graph-Cuts Vision Course - Weizmann Institute.](https://reader036.fdocuments.in/reader036/viewer/2022062421/56649d605503460f94a41f82/html5/thumbnails/23.jpg)
Constructing a Graph to Solve the Stereo Problem
![Page 24: Stereo Computation using Iterative Graph-Cuts Vision Course - Weizmann Institute.](https://reader036.fdocuments.in/reader036/viewer/2022062421/56649d605503460f94a41f82/html5/thumbnails/24.jpg)
Constructing a Graph to Solve the Stereo Problem
![Page 25: Stereo Computation using Iterative Graph-Cuts Vision Course - Weizmann Institute.](https://reader036.fdocuments.in/reader036/viewer/2022062421/56649d605503460f94a41f82/html5/thumbnails/25.jpg)
Constructing a Graph to Solve the Stereo Problem
The labels of each pixel are the possible disparity values
![Page 26: Stereo Computation using Iterative Graph-Cuts Vision Course - Weizmann Institute.](https://reader036.fdocuments.in/reader036/viewer/2022062421/56649d605503460f94a41f82/html5/thumbnails/26.jpg)
The labels of each pixel are the possible disparity values
Constructing a Graph to Solve the Stereo Problem
![Page 27: Stereo Computation using Iterative Graph-Cuts Vision Course - Weizmann Institute.](https://reader036.fdocuments.in/reader036/viewer/2022062421/56649d605503460f94a41f82/html5/thumbnails/27.jpg)
Relation between the Energy and the Graph labeling
problem
),(
),( ),()()(qp
qpqp
ppp ffVfDfE
Data term Smoothness term
p q1
10
{fp=10}
{fq=2}
![Page 28: Stereo Computation using Iterative Graph-Cuts Vision Course - Weizmann Institute.](https://reader036.fdocuments.in/reader036/viewer/2022062421/56649d605503460f94a41f82/html5/thumbnails/28.jpg)
Relation between the Energy and the Graph labeling
problem
),(
),( ),()()(qp
qpqp
ppp ffVfDfE
Data term Smoothness term
p q1
10
Dp(10)
V(p,q)(1, 10)
![Page 29: Stereo Computation using Iterative Graph-Cuts Vision Course - Weizmann Institute.](https://reader036.fdocuments.in/reader036/viewer/2022062421/56649d605503460f94a41f82/html5/thumbnails/29.jpg)
Iterative graph-cuts
• Use an iterative scheme to find a “good” local optimum of the energy function.
• In each iteration: convert the original multi-label problem to a binary one, and solve it by finding a minimal graph-cut (max-flow).
• The most popular scheme is the expansion move.
• -expansion: set the label of each pixel to be either or the current label.
![Page 30: Stereo Computation using Iterative Graph-Cuts Vision Course - Weizmann Institute.](https://reader036.fdocuments.in/reader036/viewer/2022062421/56649d605503460f94a41f82/html5/thumbnails/30.jpg)
Types of Moves
Problem: A lot of local minima
A Single Pixel Move
![Page 31: Stereo Computation using Iterative Graph-Cuts Vision Course - Weizmann Institute.](https://reader036.fdocuments.in/reader036/viewer/2022062421/56649d605503460f94a41f82/html5/thumbnails/31.jpg)
Types of Moves
Any pixel can change its label to alpha
Expansion Move
![Page 32: Stereo Computation using Iterative Graph-Cuts Vision Course - Weizmann Institute.](https://reader036.fdocuments.in/reader036/viewer/2022062421/56649d605503460f94a41f82/html5/thumbnails/32.jpg)
Types of Moves
Claim (without proof): The difference between the optimal solution and the solution from the iterative expansion moves is bounded
Expansion Move
![Page 33: Stereo Computation using Iterative Graph-Cuts Vision Course - Weizmann Institute.](https://reader036.fdocuments.in/reader036/viewer/2022062421/56649d605503460f94a41f82/html5/thumbnails/33.jpg)
Energy Minimization Algorithm
1. Start with arbitrary labeling f
2. Set success = 0
3. For each label – Find– If set and success =1
4. If success =1 goto (2)
5. Return f
expansion- one within among )(minargˆ ffEf L
)()( fEfE ff :
![Page 34: Stereo Computation using Iterative Graph-Cuts Vision Course - Weizmann Institute.](https://reader036.fdocuments.in/reader036/viewer/2022062421/56649d605503460f94a41f82/html5/thumbnails/34.jpg)
Conditions on the Smoothness for using expansion moves:
In other words: V should be a metric
Note : The Quadratic smoothness is not a metric
![Page 35: Stereo Computation using Iterative Graph-Cuts Vision Course - Weizmann Institute.](https://reader036.fdocuments.in/reader036/viewer/2022062421/56649d605503460f94a41f82/html5/thumbnails/35.jpg)
For each pair of vertices such that we add a ‘dummy’ vertex (together with the respective edges as shown in the table).
Nqp },{Nqp },{
![Page 36: Stereo Computation using Iterative Graph-Cuts Vision Course - Weizmann Institute.](https://reader036.fdocuments.in/reader036/viewer/2022062421/56649d605503460f94a41f82/html5/thumbnails/36.jpg)
The Relation between the cut and
the Energy• Given a cut C, we define a labeling fc by:
• The cost of a cut C is |C| = E(fC) (plus a constant)
oldp
p ff
If the cut C separates p and If the cut C separates p and
![Page 37: Stereo Computation using Iterative Graph-Cuts Vision Course - Weizmann Institute.](https://reader036.fdocuments.in/reader036/viewer/2022062421/56649d605503460f94a41f82/html5/thumbnails/37.jpg)
The Relation between the cut and
the Energy
The caseqp ff
![Page 38: Stereo Computation using Iterative Graph-Cuts Vision Course - Weizmann Institute.](https://reader036.fdocuments.in/reader036/viewer/2022062421/56649d605503460f94a41f82/html5/thumbnails/38.jpg)
The Relation between the cut and
the Energy
The case qp ff
![Page 39: Stereo Computation using Iterative Graph-Cuts Vision Course - Weizmann Institute.](https://reader036.fdocuments.in/reader036/viewer/2022062421/56649d605503460f94a41f82/html5/thumbnails/39.jpg)
Conditions on the Smoothness for using expansion moves:
In other words: V should be a metric
![Page 40: Stereo Computation using Iterative Graph-Cuts Vision Course - Weizmann Institute.](https://reader036.fdocuments.in/reader036/viewer/2022062421/56649d605503460f94a41f82/html5/thumbnails/40.jpg)
),(
),( ),()()(qp
qpqp
ppp ffVfDfE
Data term Smoothness term
Again, we can describe the problem using a graph
{fp=5} means – p belongs to the segment ’5’.
V(p,q)(1, 10)– The penalty for assigning p and q to different segments (1 and 10 respectively).
D(p)(5)– The penalty for assigning p to segment 5.
![Page 41: Stereo Computation using Iterative Graph-Cuts Vision Course - Weizmann Institute.](https://reader036.fdocuments.in/reader036/viewer/2022062421/56649d605503460f94a41f82/html5/thumbnails/41.jpg)
The Data & Smoothness termsThe penalty for assigning p and q to different segments should be high if the colors of pixels p and q are similar.For example:
As a data term, we have only one constraint: That the user scribbles will be assigned with the correct label:
For pixels inside user
scribblesfor the rest of the pixels
![Page 42: Stereo Computation using Iterative Graph-Cuts Vision Course - Weizmann Institute.](https://reader036.fdocuments.in/reader036/viewer/2022062421/56649d605503460f94a41f82/html5/thumbnails/42.jpg)
Graph Cuts can be used for problems other than
Stereo ! (segmentation, noise removal, image
stitching, etc’).