Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost....
Transcript of Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost....
![Page 1: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/1.jpg)
Stereo
![Page 2: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/2.jpg)
Overview of today’s lecture• Revisiting triangulation.
• Disparity.
• Stereo rectification.
• Stereo matching.
• Improving stereo matching.
• Structured light.
Slide credits: Some of these slides were adapted directly from:
• Kris Kitani (16-385, Spring 2017).• Srinivasa Narasimhan (16-823, Spring 2017).
![Page 3: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/3.jpg)
![Page 4: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/4.jpg)
Revisiting triangulation
![Page 5: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/5.jpg)
How would you reconstruct 3D points?
Left image Right image
![Page 6: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/6.jpg)
How would you reconstruct 3D points?
Left image Right image1. Select point in one image (how?)
![Page 7: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/7.jpg)
How would you reconstruct 3D points?
Left image Right image1. Select point in one image (how?)2. Form epipolar line for that point in second image
(how?)
![Page 8: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/8.jpg)
How would you reconstruct 3D points?
Left image Right image
1. Select point in one image (how?)2. Form epipolar line for that point in second image
(how?)3. Find matching point along line (how?)
![Page 9: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/9.jpg)
How would you reconstruct 3D points?
Left image Right image1. Select point in one image (how?)2. Form epipolar line for that point in second image
(how?)3. Find matching point along line (how?)4. Perform triangulation (how?)
![Page 10: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/10.jpg)
Triangulation
right imageleft image
3D point
left camera with matrix
right camera with matrix
![Page 11: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/11.jpg)
How would you reconstruct 3D points?
Left image Right image
1. Select point in one image2. Form epipolar line for that point in second image3. Find matching point along line (how?)4. Perform triangulation
Simplify the search process
![Page 12: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/12.jpg)
Stereo rectification
![Page 13: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/13.jpg)
What’s different between these two images?
![Page 14: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/14.jpg)
![Page 15: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/15.jpg)
![Page 16: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/16.jpg)
Objects that are close move more or less?
![Page 17: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/17.jpg)
The amount of horizontal movement is inversely proportional to …
![Page 18: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/18.jpg)
The amount of horizontal movement is inversely proportional to …
… the distance from the camera.
More formally…
![Page 19: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/19.jpg)
image plane
camera center camera center
3D point
![Page 20: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/20.jpg)
image plane
![Page 21: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/21.jpg)
![Page 22: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/22.jpg)
(baseline)
![Page 23: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/23.jpg)
(baseline)
How is X related to x?
![Page 24: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/24.jpg)
(baseline)
![Page 25: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/25.jpg)
(baseline)
How is X related to x’?
![Page 26: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/26.jpg)
(baseline)
![Page 27: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/27.jpg)
(baseline)
Disparity(wrt to camera origin of image plane)
![Page 28: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/28.jpg)
(baseline)
Disparityinversely proportional to depth
![Page 29: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/29.jpg)
Nomad robot searches for meteorites in Antarticahttp://www.frc.ri.cmu.edu/projects/meteorobot/index.html
Real-time stereo sensing
![Page 30: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/30.jpg)
![Page 31: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/31.jpg)
SubaruEyesight system
Pre-collision braking
![Page 32: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/32.jpg)
![Page 33: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/33.jpg)
What other vision system uses disparity for depth sensing?
![Page 34: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/34.jpg)
![Page 35: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/35.jpg)
![Page 36: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/36.jpg)
![Page 37: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/37.jpg)
![Page 38: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/38.jpg)
![Page 39: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/39.jpg)
This is how 3D movies work
![Page 40: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/40.jpg)
Let’s think about how to compute depth from two images of the same object
![Page 41: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/41.jpg)
1. Rectify images (make epipolar lines horizontal)
2. For each pixela. Find epipolar lineb. Scan line for best matchc. Compute depth from disparity
![Page 42: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/42.jpg)
How can you make the epipolar lines horizontal?
![Page 43: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/43.jpg)
image plane
camera center camera center
3D point
What’s special about these two cameras?
![Page 44: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/44.jpg)
t
x
x’
R = I t = (T, 0, 0)
When this relationship holds:When are epipolar lines horizontal?
![Page 45: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/45.jpg)
t
x
x’
When are epipolar lines horizontal?
R = I t = (T, 0, 0)
Let’s try this out…
This always has to hold for rectified images
When this relationship holds:
![Page 46: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/46.jpg)
t
x
x’
R = I t = (T, 0, 0)
Write out the constraint
Let’s try this out…
When this relationship holds:
This always has to hold for rectified images
When are epipolar lines horizontal?
![Page 47: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/47.jpg)
t
x
x’
R = I t = (T, 0, 0)
y coordinate is always the same!
Write out the constraint
Let’s try this out…
This always has to hold
The image of a 3D point will always be on the same horizontal
line
When this relationship holds:When are epipolar lines horizontal?
![Page 48: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/48.jpg)
It’s hard to make the image planes exactly parallel
![Page 49: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/49.jpg)
How can you make the epipolar lines horizontal?
![Page 50: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/50.jpg)
Use stereo rectification?
![Page 51: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/51.jpg)
What is stereo rectification?
![Page 52: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/52.jpg)
What is stereo rectification?
Reproject image planes onto a common plane parallel to the line between camera centers
How can you do this?
![Page 53: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/53.jpg)
Stereo rectificatio
n
![Page 54: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/54.jpg)
Steps
1. Optical centers2. New camera coordinate frame3. Image warping function
![Page 55: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/55.jpg)
The optical center from a projection matrix
![Page 56: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/56.jpg)
The optical center from a projection matrix
• Where is camera center c?
![Page 57: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/57.jpg)
c1 c2c2-c1
New camera coordinate frame
![Page 58: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/58.jpg)
c1 c2c2-c1
New x-axis (r1): c2-c1
|c2-c1|New camera coordinate frame
![Page 59: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/59.jpg)
c1 c2c2-c1
R1(:, 3)
New camera coordinate frameNew x-axis (r1): c2-c1
|c2-c1|
New x-axis (r2): R1(:, 3) x r1
![Page 60: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/60.jpg)
c1 c2c2-c1
r2
New camera coordinate frame
r1
New x-axis (r1): c2-c1
|c2-c1|
New x-axis (r2): R1(:, 3) x r1
![Page 61: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/61.jpg)
c1 c2c2-c1
r2
New camera coordinate frame
r1
New x-axis (r1): c2-c1
|c2-c1|
New x-axis (r2): R1(:, 3) x r1
New x-axis (r3): r2 x r1
![Page 62: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/62.jpg)
c1 c2c2-c1
New x-axis (r1): c2-c1
|c2-c1|
r2
New camera coordinate frame
New x-axis (r2): R1(:, 3) x r1
r1
New x-axis (r3): r2 x r1
r3
![Page 63: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/63.jpg)
![Page 64: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/64.jpg)
![Page 65: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/65.jpg)
For assignments: A way to get R from E
SVD: Let
We get FOUR solutions:
two possible rotations two possible translations
![Page 66: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/66.jpg)
We get FOUR solutions:
Compute determinant of R, valid solution must be equal to 1(note: det(R) = -1 means rotation and reflection)
Compute 3D point using triangulation, valid solution has positive Z value(Note: negative Z means point is behind the camera )
Which one do we choose?
![Page 67: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/67.jpg)
Positive depth test
• E -> R1, T1
3D coordinate X
Perform triangulation, with P1, P2 and x1, x2
Check if positive
K1
K2
![Page 68: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/68.jpg)
camera center
image plane
optic
al a
xis
Camera Icon
Find the configuration where the points is in front of both cameras
Let’s visualize the four configurations…
![Page 69: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/69.jpg)
Find the configuration where the points is in front of both cameras
![Page 70: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/70.jpg)
Find the configuration where the points is in front of both cameras
![Page 71: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/71.jpg)
Unrectified
![Page 72: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/72.jpg)
Unrectified
Rectified
![Page 73: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/73.jpg)
Unrectified
Rectified
Unrectified
Rectified
![Page 74: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/74.jpg)
What can we do after rectification?
![Page 75: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/75.jpg)
Stereo matching
![Page 76: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/76.jpg)
Depth Estimation via Stereo Matching
![Page 77: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/77.jpg)
1. Rectify images (make epipolar lines horizontal)
2. For each pixela. Find epipolar lineb. Scan line for best matchc. Compute depth from disparity
How would you do this?
![Page 78: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/78.jpg)
Find this templateHow do we detect the template in he following image?
![Page 79: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/79.jpg)
Find this templateHow do we detect the template in the following image?
Solution 3: Use sum of squared differences (SSD).
filter
image
outputWhat will
the output
look like?
![Page 80: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/80.jpg)
Find this templateHow do we detect the template in the following image?
Solution 3: Use sum of squared differences (SSD).
filter
image
output True detection
1-output
thresholding
What could go wrong?
![Page 81: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/81.jpg)
Find this templateHow do we detect the template in the following image?
Solution 3: Use sum of squared differences (SSD).
filter
image
output
1-output
Not robust to local intensity changes
![Page 82: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/82.jpg)
Find this templateHow do we detect the template in the following image?
Observations so far:
• subtracting mean deals with brightness bias
• dividing by standard deviation removes contrast bias
Can we combine the two effects?
![Page 83: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/83.jpg)
Find this templateHow do we detect the template in the following image?
Solution 4: Normalized cross-correlation (NCC).
filter
image
output
template mean
local patch mean
What will the
output look like?
![Page 84: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/84.jpg)
Find this templateHow do we detect the template in the following image?
Solution 4: Normalized cross-correlation (NCC).
True detections
1-output
thresholding
![Page 85: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/85.jpg)
Find this templateHow do we detect the template in the following image?
Solution 4: Normalized cross-correlation (NCC).
True detections
1-output
thresholding
![Page 86: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/86.jpg)
What is the best method?It depends on whether you care about speed or invariance.
• Zero-mean: Fastest, very sensitive to local intensity.
• Sum of squared differences: Medium speed, sensitive to intensity offsets.
• Normalized cross-correlation: Slowest, invariant to contrast and brightness.
![Page 87: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/87.jpg)
Stereo Block Matching
Matching cost
disparity
Left Right
scanline
• Slide a window along the epipolar line and compare contents of that window with the reference window in the left image
• Matching cost: SSD or normalized correlation
![Page 88: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/88.jpg)
SSD
![Page 89: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/89.jpg)
Normalized cross-correlation
![Page 90: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/90.jpg)
Similarity Measure FormulaSum of Absolute Differences (SAD)
Sum of Squared Differences (SSD)
Zero-mean SAD
Locally scaled SAD
Normalized Cross Correlation (NCC)
SAD SSD NCC Ground truth
![Page 91: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/91.jpg)
Effect of window size
W = 3 W = 20
![Page 92: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/92.jpg)
Effect of window size
W = 3 W = 20
Smaller window+ More detail- More noise
Larger window+ Smoother disparity maps- Less detail- Fails near boundaries
![Page 93: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/93.jpg)
When will stereo block matching fail?
![Page 94: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/94.jpg)
When will stereo block matching fail?
textureless regions repeated patterns
specularities
![Page 95: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/95.jpg)
Improving stereo matching
![Page 96: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/96.jpg)
Block matching Ground truth
What are some problems with the result?
![Page 97: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/97.jpg)
How can we improve depth estimation?
![Page 98: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/98.jpg)
How can we improve depth estimation?Too many discontinuities.
We expect disparity values to change slowly.
Let’s make an assumption: depth should change smoothly
![Page 99: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/99.jpg)
Stereo matching as …
Energy Minimization
What defines a good stereo correspondence?1. Match quality
– Want each pixel to find a good match in the other image2. Smoothness
– If two pixels are adjacent, they should (usually) move about the same amount
![Page 100: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/100.jpg)
{ {
(block matching result) (smoothness function)
Want each pixel to find a good match in the other image
Adjacent pixels should (usually) move about the same amount
data term smoothness term
energy function(for one pixel)
![Page 101: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/101.jpg)
Stereo as energy minimization
I(x, y) J(x, y)
y = 141
C(x, y, d); the disparity space image (DSI)x
d
![Page 102: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/102.jpg)
Stereo as energy minimization
y = 141
x
d
Simple pixel / window matching: choose the minimum of each column in the DSI independently:
![Page 103: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/103.jpg)
Stereo as energy minimization
y = 141
x
d
![Page 104: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/104.jpg)
Stereo as energy minimization
y = 141
x
d
![Page 105: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/105.jpg)
Stereo as energy minimization
y = 141
x
d
![Page 106: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/106.jpg)
Stereo as energy minimizationBetter objective function
{ {match cost smoothness cost
Want each pixel to find a good match in the other image
Adjacent pixels should (usually) move about the same amount
![Page 107: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/107.jpg)
Stereo as energy minimization
match cost:
![Page 108: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/108.jpg)
Stereo as energy minimization
match cost:
smoothness cost:
Left and right
: set of neighboring pixels
![Page 109: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/109.jpg)
Smoothness cost
“Potts model”
L1 distance
How do we choose V?
![Page 110: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/110.jpg)
Stereo as energy minimization
y = 141
x
d
![Page 111: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/111.jpg)
Stereo as energy minimization
y = 141
x
d
![Page 112: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/112.jpg)
Stereo as energy minimization
y = 141
x
d
![Page 113: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/113.jpg)
Dynamic programming
Can minimize this independently per scanline using dynamic programming (DP)
: minimum cost of solution such that d(x,y) = d
![Page 114: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/114.jpg)
Dynamic programming
y = 141
x
d
![Page 115: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/115.jpg)
Dynamic programming
Cost of optimal path from the left
y = 141
x
d
![Page 116: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/116.jpg)
Dynamic programming
How to compute for this pixel?
y = 141
x
d
![Page 117: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/117.jpg)
Dynamic programming
How to compute for this pixel?Only 3 ways to get here...
y = 141
x
d
![Page 118: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/118.jpg)
Dynamic programming
This cost is ...
y = 141
x
d
![Page 119: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/119.jpg)
Dynamic programming
This cost is ...+ +
D Ed Es
y = 141
x
d
![Page 120: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/120.jpg)
Dynamic programming
y = 141
x
d
Fill D all the way to the right
![Page 121: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/121.jpg)
Dynamic programming
y = 141
x
d
Pick the best one from the right column
![Page 122: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/122.jpg)
Dynamic programming
y = 141
x
d
Back track....
![Page 123: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/123.jpg)
Dynamic programming
y = 141
x
d
Back track....
![Page 124: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/124.jpg)
Stereo as energy minimization
y = 141
x
d
![Page 125: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/125.jpg)
Dynamic Programming
![Page 126: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/126.jpg)
Dynamic programming is optimal…
179
![Page 127: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/127.jpg)
Coherent stereo on 2D grid• Scanline stereo generates streaking artifacts
• Can’t use dynamic programming to find spatially coherent disparities/ correspondences on a 2D grid
![Page 128: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/128.jpg)
Stereo as energy minimization
match cost:
smoothness cost:
Left and right
: set of neighboring pixels
![Page 129: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/129.jpg)
Stereo as energy minimization
match cost:
smoothness cost:
4-connected neighborhood
: set of neighboring pixels
![Page 130: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/130.jpg)
Pixelwise
![Page 131: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/131.jpg)
Pixelwise
![Page 132: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/132.jpg)
Pixelwise -> Scanline wise
y = 141
![Page 133: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/133.jpg)
Pixelwise -> Scanline wise
y = 141
![Page 134: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/134.jpg)
Pixelwise -> Scanline wise
y = 141
![Page 135: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/135.jpg)
Pixelwise
![Page 136: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/136.jpg)
Pixelwise
![Page 137: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/137.jpg)
Stereo as energy minimization
match cost:
smoothness cost:
4-connected neighborhood
8-connected neighborhood
: set of neighboring pixels
![Page 138: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/138.jpg)
Smoothness cost
L1 distance
How do we choose V?
![Page 139: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/139.jpg)
Stereo matching as energy minimization
I1 I2 D
• Graph-cuts can be used to minimize such energyY. Boykov, O. Veksler, and R. Zabih, Fast Approximate Energy Minimization via Graph Cuts, PAMI 2001
![Page 140: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/140.jpg)
Same formulation with more images• Change label from disparity to depth• Change Ed(d) by using more images
...
![Page 141: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/141.jpg)
Same formulation with more images• Change label from disparity to depth• Change Ed(d) by using more images
![Page 142: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/142.jpg)
Same formulation with more images• Change label from disparity to depth• Change Ed(d) by using more images
...
![Page 143: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/143.jpg)
Same formulation with more images• Change label from disparity to depth• Change Ed(d) by using more images
... ... ...
Cost of assigning this depthcomes from
![Page 144: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/144.jpg)
With more images, engineering and tricks
197
![Page 145: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/145.jpg)
Structured light
![Page 146: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/146.jpg)
Use controlled (“structured”) light to make correspondences easier
Disparity between laser points on the same scanline in the images determines the 3-D coordinates of the laser point on object
![Page 147: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/147.jpg)
Use controlled (“structured”) light to make correspondences easier
![Page 148: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/148.jpg)
Structured light and two cameras
I J
laser
![Page 149: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/149.jpg)
I J
Structured light and one camera
Projector acts like “reverse” camera
![Page 150: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/150.jpg)
Digital Michelangelo Projecthttp://graphics.stanford.edu/projects/mich/
Example: Laser scanner
![Page 151: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/151.jpg)
![Page 152: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/152.jpg)
![Page 153: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/153.jpg)
![Page 154: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/154.jpg)
![Page 155: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/155.jpg)
![Page 156: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/156.jpg)
![Page 157: Stereo - cs.sfu.cafurukawa/cmpt762/slides/11-stereo.pdf · Stereo Block Matching. Matching cost. disparity. Left. Right. scanline • Slide a window along the epipolar line and compare](https://reader035.fdocuments.in/reader035/viewer/2022062414/5e86220067f16178145e127d/html5/thumbnails/157.jpg)
Basic reading:• Szeliski textbook, Section 8.1 (not 8.1.1-8.1.3), Chapter 11, Section 12.2.• Hartley and Zisserman, Section 11.12.
References