Lecture 19: Solving the Correspondence Problem with Graph Cuts CAP 5415 Fall 2006.
-
Upload
junior-lester -
Category
Documents
-
view
221 -
download
2
Transcript of Lecture 19: Solving the Correspondence Problem with Graph Cuts CAP 5415 Fall 2006.
![Page 1: Lecture 19: Solving the Correspondence Problem with Graph Cuts CAP 5415 Fall 2006.](https://reader036.fdocuments.in/reader036/viewer/2022062308/56649f525503460f94c76ae3/html5/thumbnails/1.jpg)
Lecture 19: Solving the Correspondence Problem with Graph Cuts
CAP 5415Fall 2006
![Page 2: Lecture 19: Solving the Correspondence Problem with Graph Cuts CAP 5415 Fall 2006.](https://reader036.fdocuments.in/reader036/viewer/2022062308/56649f525503460f94c76ae3/html5/thumbnails/2.jpg)
Announcements
PS4 Only need to search over a limited number of
disparities
Schedule Changes Project Proposals due next Monday Class canceled on Wednesday Will have class on November 22
![Page 3: Lecture 19: Solving the Correspondence Problem with Graph Cuts CAP 5415 Fall 2006.](https://reader036.fdocuments.in/reader036/viewer/2022062308/56649f525503460f94c76ae3/html5/thumbnails/3.jpg)
Stereo Once we have rectified images, the hard problem is
find the corresponding points Then we can triangulate Known as the correspondence problem When people say “stereo algorithm”, they usually
mean “correspondence algorithm”
![Page 4: Lecture 19: Solving the Correspondence Problem with Graph Cuts CAP 5415 Fall 2006.](https://reader036.fdocuments.in/reader036/viewer/2022062308/56649f525503460f94c76ae3/html5/thumbnails/4.jpg)
A very simple algorithm
To find the point that matches this point
![Page 5: Lecture 19: Solving the Correspondence Problem with Graph Cuts CAP 5415 Fall 2006.](https://reader036.fdocuments.in/reader036/viewer/2022062308/56649f525503460f94c76ae3/html5/thumbnails/5.jpg)
A very simple algorithm
Look at a square window around that point
and compare it to windows in the other image
Could measure sum of squared differences (SSD) or correlation
![Page 6: Lecture 19: Solving the Correspondence Problem with Graph Cuts CAP 5415 Fall 2006.](https://reader036.fdocuments.in/reader036/viewer/2022062308/56649f525503460f94c76ae3/html5/thumbnails/6.jpg)
What's wrong with this algorithm?
This set of correspondences don't bother it
Areas of the image without texture will be a problem for this algorithm.
![Page 7: Lecture 19: Solving the Correspondence Problem with Graph Cuts CAP 5415 Fall 2006.](https://reader036.fdocuments.in/reader036/viewer/2022062308/56649f525503460f94c76ae3/html5/thumbnails/7.jpg)
To do better we need a better model of images
We can make reasonable assumptions about the surfaces in the world
Usually assume that the surfaces are smooth Can pose the problem of finding the
corresponding points as an energy (or cost) minimization
The data term measures how well the local windows match up for different disparities
![Page 8: Lecture 19: Solving the Correspondence Problem with Graph Cuts CAP 5415 Fall 2006.](https://reader036.fdocuments.in/reader036/viewer/2022062308/56649f525503460f94c76ae3/html5/thumbnails/8.jpg)
The smoothness cost
The smoothness is usually implemented by penalizing differences in the disparity
Different penalty functions lead to different assumptions about the surfaces log(1+x^2) x^2 First deriv, Second derivative
![Page 9: Lecture 19: Solving the Correspondence Problem with Graph Cuts CAP 5415 Fall 2006.](https://reader036.fdocuments.in/reader036/viewer/2022062308/56649f525503460f94c76ae3/html5/thumbnails/9.jpg)
Today
Assuming that there a discrete set of possible disparities
How do we optimize this?
![Page 10: Lecture 19: Solving the Correspondence Problem with Graph Cuts CAP 5415 Fall 2006.](https://reader036.fdocuments.in/reader036/viewer/2022062308/56649f525503460f94c76ae3/html5/thumbnails/10.jpg)
Today
I'll be talking about the Graph Cuts algorithm Very popular Used heavily in vision and graphics Paper: “Fast Approximate Energy Minimization Notation for the rest of the lecture
![Page 11: Lecture 19: Solving the Correspondence Problem with Graph Cuts CAP 5415 Fall 2006.](https://reader036.fdocuments.in/reader036/viewer/2022062308/56649f525503460f94c76ae3/html5/thumbnails/11.jpg)
Simple Way to Optimize - ICM
Choose a pixel Fix the disparities at the rest of the pixels Set the pixel to the optimal disparity Iterate
![Page 12: Lecture 19: Solving the Correspondence Problem with Graph Cuts CAP 5415 Fall 2006.](https://reader036.fdocuments.in/reader036/viewer/2022062308/56649f525503460f94c76ae3/html5/thumbnails/12.jpg)
ICM
Advantages Energy always guaranteed to decrease Easy to code
Disadvantages Convergence Can you think of the big on?
![Page 13: Lecture 19: Solving the Correspondence Problem with Graph Cuts CAP 5415 Fall 2006.](https://reader036.fdocuments.in/reader036/viewer/2022062308/56649f525503460f94c76ae3/html5/thumbnails/13.jpg)
Making Bigger Moves
The problem with ICM is that you can only modify one pixel at a time
Get stuck in local minima easily We need a way of moving multiple pixels at
once Boykov, Veksler and Zabih introduced two
types of moves: alpha-beta swap alpha-expansion
![Page 14: Lecture 19: Solving the Correspondence Problem with Graph Cuts CAP 5415 Fall 2006.](https://reader036.fdocuments.in/reader036/viewer/2022062308/56649f525503460f94c76ae3/html5/thumbnails/14.jpg)
Types of Moves
ICM – One pixel moves
(From BKZ-PAMI 01)
![Page 15: Lecture 19: Solving the Correspondence Problem with Graph Cuts CAP 5415 Fall 2006.](https://reader036.fdocuments.in/reader036/viewer/2022062308/56649f525503460f94c76ae3/html5/thumbnails/15.jpg)
alpha-beta swap
Fix all nodes that aren't labeled alpha or beta With remaining nodes, find optimal swap
Some alpha nodes change to beta Vice Versa Some stay the same (From BKZ-PAMI 01)
![Page 16: Lecture 19: Solving the Correspondence Problem with Graph Cuts CAP 5415 Fall 2006.](https://reader036.fdocuments.in/reader036/viewer/2022062308/56649f525503460f94c76ae3/html5/thumbnails/16.jpg)
alpha-expansion
Any node can change to alpha
(From BKZ-PAMI 01)
![Page 17: Lecture 19: Solving the Correspondence Problem with Graph Cuts CAP 5415 Fall 2006.](https://reader036.fdocuments.in/reader036/viewer/2022062308/56649f525503460f94c76ae3/html5/thumbnails/17.jpg)
Basic Algorithm
ICM: Compute one-pixel moves until convergence
Graph Cuts: Compute swaps (or expansions) until convergence
Since the optimal swap is being computed each time, energy always decreases
![Page 18: Lecture 19: Solving the Correspondence Problem with Graph Cuts CAP 5415 Fall 2006.](https://reader036.fdocuments.in/reader036/viewer/2022062308/56649f525503460f94c76ae3/html5/thumbnails/18.jpg)
Important Question
How do we find the swaps? Use min-cut from graphs Slightly different from the min-cut problem that
we discussed in the context of normalized cuts
Terminal NodeTerminal Node
![Page 19: Lecture 19: Solving the Correspondence Problem with Graph Cuts CAP 5415 Fall 2006.](https://reader036.fdocuments.in/reader036/viewer/2022062308/56649f525503460f94c76ae3/html5/thumbnails/19.jpg)
Minimum Cut A cut is a set of edges that we will remove so
that the terminal nodes are separated If a cost is assigned to each edge, the cost of
the cut is the weight assigned to each edge Minimum Cut can be found in polynomial time
(Ford and Fulkerson)
Terminal NodeTerminal Node
![Page 20: Lecture 19: Solving the Correspondence Problem with Graph Cuts CAP 5415 Fall 2006.](https://reader036.fdocuments.in/reader036/viewer/2022062308/56649f525503460f94c76ae3/html5/thumbnails/20.jpg)
Important Rule No proper subset of the cut can also be a cut This is not a minimum cut
Terminal NodeTerminal Node
![Page 21: Lecture 19: Solving the Correspondence Problem with Graph Cuts CAP 5415 Fall 2006.](https://reader036.fdocuments.in/reader036/viewer/2022062308/56649f525503460f94c76ae3/html5/thumbnails/21.jpg)
Using the minimum-cut to find a swap
Consider a 1-D image
Terminal Node
Terminal Node
(From BKZ-PAMI 01)
![Page 22: Lecture 19: Solving the Correspondence Problem with Graph Cuts CAP 5415 Fall 2006.](https://reader036.fdocuments.in/reader036/viewer/2022062308/56649f525503460f94c76ae3/html5/thumbnails/22.jpg)
Can guarantee that every node will have at least one t-link
(From BKZ-PAMI 01)
![Page 23: Lecture 19: Solving the Correspondence Problem with Graph Cuts CAP 5415 Fall 2006.](https://reader036.fdocuments.in/reader036/viewer/2022062308/56649f525503460f94c76ae3/html5/thumbnails/23.jpg)
How to see that
(From BKZ-PAMI 01)
![Page 24: Lecture 19: Solving the Correspondence Problem with Graph Cuts CAP 5415 Fall 2006.](https://reader036.fdocuments.in/reader036/viewer/2022062308/56649f525503460f94c76ae3/html5/thumbnails/24.jpg)
Actually Computing the Min-Cut Could use code to solve general min-cut
problems Boykov and Kolmogorov have released an
algorithm optimized for vision problems
![Page 25: Lecture 19: Solving the Correspondence Problem with Graph Cuts CAP 5415 Fall 2006.](https://reader036.fdocuments.in/reader036/viewer/2022062308/56649f525503460f94c76ae3/html5/thumbnails/25.jpg)
Expansion Move
The difference between the optimal solution and the solution from the expansion move is bounded
![Page 26: Lecture 19: Solving the Correspondence Problem with Graph Cuts CAP 5415 Fall 2006.](https://reader036.fdocuments.in/reader036/viewer/2022062308/56649f525503460f94c76ae3/html5/thumbnails/26.jpg)
What functions can be minimized using graph cuts
V is called a metric if it obeys all 3 Can use Expansion Moves
V is called a semimetric if it obeys the last two Can only use Swap-Moves
Subject of recent research.
![Page 27: Lecture 19: Solving the Correspondence Problem with Graph Cuts CAP 5415 Fall 2006.](https://reader036.fdocuments.in/reader036/viewer/2022062308/56649f525503460f94c76ae3/html5/thumbnails/27.jpg)
Graph-Cuts has been shown to perform very well
![Page 28: Lecture 19: Solving the Correspondence Problem with Graph Cuts CAP 5415 Fall 2006.](https://reader036.fdocuments.in/reader036/viewer/2022062308/56649f525503460f94c76ae3/html5/thumbnails/28.jpg)
From “A Comparative Study of Energy Minimization Methods for
Markov Random Fields”
![Page 29: Lecture 19: Solving the Correspondence Problem with Graph Cuts CAP 5415 Fall 2006.](https://reader036.fdocuments.in/reader036/viewer/2022062308/56649f525503460f94c76ae3/html5/thumbnails/29.jpg)
From “A Comparative Study of Energy Minimization Methods for
Markov Random Fields”
![Page 30: Lecture 19: Solving the Correspondence Problem with Graph Cuts CAP 5415 Fall 2006.](https://reader036.fdocuments.in/reader036/viewer/2022062308/56649f525503460f94c76ae3/html5/thumbnails/30.jpg)
Used for much more than just stereo
![Page 31: Lecture 19: Solving the Correspondence Problem with Graph Cuts CAP 5415 Fall 2006.](https://reader036.fdocuments.in/reader036/viewer/2022062308/56649f525503460f94c76ae3/html5/thumbnails/31.jpg)
![Page 32: Lecture 19: Solving the Correspondence Problem with Graph Cuts CAP 5415 Fall 2006.](https://reader036.fdocuments.in/reader036/viewer/2022062308/56649f525503460f94c76ae3/html5/thumbnails/32.jpg)