Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective...
Transcript of Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective...
![Page 1: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/1.jpg)
Stereo matching
Reading: Chapter 11 In Szeliski’s book
![Page 2: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/2.jpg)
Schedule (tentative)
2
# date topic1 Sep.22 Introduction and geometry2 Sep.29 Invariant features 3 Oct.6 Camera models and calibration4 Oct.13 Multiple-view geometry5 Oct.20 Model fitting (RANSAC, EM, …)6 Oct.27 Stereo Matching7 Nov.3 2D Shape Matching 8 Nov.10 Segmentation9 Nov.17 Shape from Silhouettes10 Nov.24 Structure from motion11 Dec.1 Optical Flow 12 Dec.8 Tracking (Kalman, particle filter)13 Dec.15 Object category recognition 14 Dec.22 Specific object recognition
![Page 3: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/3.jpg)
3
An Application: Mobile Robot Navigation
The Stanford Cart,H. Moravec, 1979.
The INRIA Mobile Robot, 1990.
Courtesy O. Faugeras and H. Moravec.
![Page 4: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/4.jpg)
Stereo
NASA Mars Rover
![Page 5: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/5.jpg)
Standard stereo geometry
pure translation along X-axis
PointGrey Bumblebee
![Page 6: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/6.jpg)
Standard stereo geometry
![Page 7: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/7.jpg)
Stereo: basic idea
error
depth
![Page 8: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/8.jpg)
Stereo matching
• Search is limited to epipolar line (1D)• Look for most similar pixel
?
![Page 9: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/9.jpg)
9
Stereopsis
Figure from US Navy Manual of Basic Optics and Optical Instruments, prepared by Bureau of Naval Personnel. Reprinted by Dover Publications, Inc., 1969.
![Page 10: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/10.jpg)
10
Human Stereopsis: Reconstruction
Disparity: d = r-l = D-F.
d=0
d<0In 3D, the horopter.
![Page 11: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/11.jpg)
11
Human Stereopsis: experimental horopter…
![Page 12: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/12.jpg)
12
Iso-disparity curves: planar retinas
X0X1
C1
X∞
C2
Xi Xj
iii
i
=−−
−−
=11:
10110111
:
111011
0
the retina act as if it were flat!
![Page 13: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/13.jpg)
13
What if F is not known?
Human Stereopsis: Reconstruction
Helmoltz (1909):
• There is evidence showing the vergence anglescannot be measured precisely.
• Humans get fooled by bas-relief sculptures.
• There is an analytical explanation for this.
• Relative depth can be judged accurately.
![Page 14: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/14.jpg)
14
Human Stereopsis: Binocular Fusion
How are the correspondences established?
Julesz (1971): Is the mechanism for binocular fusiona monocular process or a binocular one??• There is anecdotal evidence for the latter (camouflage).
• Random dot stereograms provide an objective answerBP!
![Page 15: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/15.jpg)
15
A Cooperative Model (Marr and Poggio, 1976)
Excitory connections: continuity
Inhibitory connections: uniqueness
Iterate: C = Σ C - wΣ C + C .e i 0Reprinted from Vision: A Computational Investigation into the Human Representation and Processing of Visual Information by David Marr. 1982 by David Marr. Reprinted by permission of Henry Holt and Company, LLC.
![Page 16: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/16.jpg)
16
Correlation Methods (1970--)
Normalized Correlation: minimize θ instead.
Slide the window along the epipolar line until w.w’ is maximized.2Minimize |w-w’|.
![Page 17: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/17.jpg)
17
Correlation Methods: Foreshortening Problems
Solution: add a second pass using disparity estimates to warpthe correlation windows, e.g. Devernay and Faugeras (1994).
Reprinted from “Computing Differential Properties of 3D Shapes from Stereopsis without 3D Models,” by F. Devernay and O. Faugeras, Proc. IEEE Conf. on Computer Vision and Pattern Recognition (1994). 1994 IEEE.
![Page 18: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/18.jpg)
18
Multi-Scale Edge Matching (Marr, Poggio and Grimson, 1979-81)
• Edges are found by repeatedly smoothing the image and detectingthe zero crossings of the second derivative (Laplacian).• Matches at coarse scales are used to offset the search for matchesat fine scales (equivalent to eye movements).
![Page 19: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/19.jpg)
19
Multi-Scale Edge Matching (Marr, Poggio and Grimson, 1979-81)
One of the twoinput images
Image Laplacian
Zeros of the Laplacian
Reprinted from Vision: A Computational Investigation into the Human Representation and Processing of Visual Information by David Marr. 1982 by David Marr. Reprinted by permission of Henry Holt and Company, LLC.
![Page 20: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/20.jpg)
20
Multi-Scale Edge Matching (Marr, Poggio and Grimson, 1979-81)
Reprinted from Vision: A Computational Investigation into the Human Representation and Processing of Visual Information by David Marr. 1982 by David Marr. Reprinted by permission of Henry Holt and Company, LLC.
![Page 21: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/21.jpg)
21
Pixel Dissimilarity• Absolute difference of intensities
c=|I1(x,y)- I2(x-d,y)|• Interval matching [Birchfield 98]
– Considers sensor integration– Represents pixels as intervals
![Page 22: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/22.jpg)
22
Alternative Dissimilarity Measures
• Rank and Census transforms [Zabih ECCV94]
• Rank transform:– Define window containing R pixels around each
pixel– Count the number of pixels with lower
intensities than center pixel in the window– Replace intensity with rank (0..R-1)– Compute SAD on rank-transformed images
• Census transform: – Use bit string, defined by neighbors, instead of
scalar rank
• Robust against illumination changes
![Page 23: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/23.jpg)
23
Rank and Census Transform Results
• Noise free, random dot stereograms• Different gain and bias
![Page 24: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/24.jpg)
24
Systematic Errors of Area-based Stereo
• Ambiguous matches in textureless regions
• Surface over-extension [Okutomi IJCV02]
![Page 25: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/25.jpg)
25
Surface Over-extension• Expected value of E[(x-
y)2] for x in left and y in right image is:Case A: σF
2+ σB2+(μF- μB)2 for
w/2-λ pixels in each row
Case B: 2 σB2 for w/2+λ pixels
in each row
Right image
Left image
Disparity of back surface
![Page 26: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/26.jpg)
26
Surface Over-extension• Discontinuity
perpendicular to epipolar lines
• Discontinuity parallel to epipolar lines
Right image
Left image
Disparity of back surface
![Page 27: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/27.jpg)
27
Over-extension and shrinkage
• Turns out that:
for discontinuities perpendicular to epipolar lines
• And:
for discontinuities parallel to epipolar lines
26ww
≤≤− λ
22ww
≤≤− λ
![Page 28: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/28.jpg)
28
Random Dot Stereogram Experiments
![Page 29: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/29.jpg)
29
Random Dot Stereogram Experiments
![Page 30: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/30.jpg)
30
Offset Windows
Equivalent to using minnearby costResult: loss of depth accuracy
![Page 31: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/31.jpg)
31
Discontinuity Detection
• Use offset windows only where appropriate– Bi-modal distribution of SSD – Pixel of interest different than mode
within window
![Page 32: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/32.jpg)
32
Compact Windows
• [Veksler CVPR03]: Adapt windows size based on:– Average matching error per pixel– Variance of matching error– Window size (to bias towards larger
windows)
• Pick window that minimizes cost
![Page 33: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/33.jpg)
33
Integral Image
Sum of shaded part
Compute an integral image for pixel dissimilarity at each possible disparity
A C
DB
Shaded area = A+D-B-CIndependent of size
![Page 34: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/34.jpg)
34
Results using Compact Windows
![Page 35: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/35.jpg)
35
Rod-shaped filters
• Instead of square windows aggregate cost in rod-shaped shiftable windows [Kim CVPR05]
• Search for one that minimizes the cost (assume that it is an iso-disparity curve)
• Typically use 36 orientations
![Page 36: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/36.jpg)
36
Locally Adaptive Support
Apply weights to contributions of neighboringpixels according to similarity and proximity [Yoon CVPR05]
![Page 37: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/37.jpg)
37
Locally Adaptive Support
• Similarity in CIE Lab color space:
• Proximity: Euclidean distance
• Weights:
![Page 38: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/38.jpg)
38
Locally Adaptive Support: Results
Locally Adaptive Support
![Page 39: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/39.jpg)
39
Locally Adaptive Support: Results
![Page 40: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/40.jpg)
40
Cool ideas
• Space-time stereo (varying illumination, not shape)
![Page 41: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/41.jpg)
41
Challenges
• Ill-posed inverse problem– Recover 3-D structure from 2-D
information
• Difficulties– Uniform regions– Half-occluded pixels
![Page 42: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/42.jpg)
Occlusions
(Slide from Pascal Fua)
![Page 43: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/43.jpg)
Exploiting scene constraints
![Page 44: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/44.jpg)
44
The Ordering Constraint
But it is not always the case..
In general the pointsare in the same orderon both epipolar lines.
![Page 45: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/45.jpg)
Ordering constraint
11 22222 3 4,54,54,5 6 11 2,3 6
21 3 4,5 61
2,3
45
6
surface slice surface as a path
occlusion right
occlusion left
![Page 46: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/46.jpg)
Uniqueness constraint
• In an image pair each pixel has at most one corresponding pixel– In general one corresponding pixel– In case of occlusion there is none
![Page 47: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/47.jpg)
Disparity constraint
surface slice surface as a path
bounding box
use reconstructed features to determine bounding box
constantdisparitysurfaces
![Page 48: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/48.jpg)
48
Dynamic Programming (Baker and Binford, 1981)
Find the minimum-cost path going monotonicallydown and right from the top-left corner of thegraph to its bottom-right corner.
• Nodes = matched feature points (e.g., edge points).• Arcs = matched intervals along the epipolar lines.• Arc cost = discrepancy between intervals.
![Page 49: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/49.jpg)
49
Stereo matching
Optimal path(dynamic programming )
Similarity measure(SSD or NCC)
Constraints• epipolar• ordering• uniqueness• disparity limit• disparity gradient limit
Trade-off• Matching cost (data)• Discontinuities (prior)
(Cox et al. CVGIP’96; Koch’96; Falkenhagen´97;Van Meerbergen,Vergauwen,Pollefeys,VanGool IJCV‘02)
![Page 50: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/50.jpg)
50
Dynamic Programming (Ohta and Kanade, 1985)
Reprinted from “Stereo by Intra- and Intet-Scanline Search,” by Y. Ohta and T. Kanade, IEEE Trans. on Pattern Analysis and MachineIntelligence, 7(2):139-154 (1985). 1985 IEEE.
![Page 51: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/51.jpg)
51
Hierarchical stereo matchingD
owns
ampl
ing
(Gau
ssia
n py
ram
id)
Dis
pari
ty p
ropa
gati
on
Allows faster computation
Deals with large disparity ranges
(Falkenhagen´97;Van Meerbergen,Vergauwen,Pollefeys,VanGool IJCV‘02)
![Page 52: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/52.jpg)
52
Disparity map
image I(x,y) image I´(x´,y´)Disparity map D(x,y)
(x´,y´)=(x+D(x,y),y)
![Page 53: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/53.jpg)
53
Example: reconstruct image from neighboring images
![Page 54: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/54.jpg)
![Page 55: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/55.jpg)
55
Semi-global optimization
• Optimize: E=Edata+E(|Dp-Dq|=1)+E(|Dp-Dq|>1) [Hirshmüller CVPR05]– Use mutual information as cost
• NP-hard using graph cuts or belief propagation (2-D optimization)
• Instead do dynamic programming along many directions – Don’t use visibility or ordering constraints– Enforce uniqueness– Add costs
![Page 56: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/56.jpg)
56
Results of Semi-global optimization
![Page 57: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/57.jpg)
57
Results of Semi-global optimization
No. 1 overall in Middlebury evaluation(at 0.5 error threshold as of Sep. 2006)
![Page 58: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/58.jpg)
Energy minimization
(Slide from Pascal Fua)
![Page 59: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/59.jpg)
Graph Cut
(Slide from Pascal Fua)
(general formulation requires multi-way cut!)
![Page 60: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/60.jpg)
(Boykov et al ICCV‘99)
(Roy and Cox ICCV‘98)
Simplified graph cut
![Page 61: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/61.jpg)
Belief Propagation
per pixel +left +right +up +down
2 3 4 5 20...subsequent iterations
(adapted from J. Coughlan slides)
first iteration
Belief of one node about another gets propagated through messages (full pdf, not just most likely state)
![Page 62: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/62.jpg)
![Page 63: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/63.jpg)
63
Rectification
All epipolar lines are parallel in the rectified image plane.
![Page 64: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/64.jpg)
64
Image pair rectification
simplify stereo matching by warping the images
Apply projective transformation so that epipolar linescorrespond to horizontal scanlines
e
e
map epipole e to (1,0,0)
try to minimize image distortion
problem when epipole in (or close to) the image
He001
=
![Page 65: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/65.jpg)
65
Planar rectification
Bring two views to standard stereo setup(moves epipole to ∞)(not possible when in/close to image)
~ image size~ image size
(calibrated)
Distortion minimization(uncalibrated)
(standard approach)
![Page 66: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/66.jpg)
66
![Page 67: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/67.jpg)
67
![Page 68: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/68.jpg)
68
Polar re-parameterization around epipolesRequires only (oriented) epipolar geometryPreserve length of epipolar linesChoose ∆θ so that no pixels are compressed
original image rectified image
Polar rectification(Pollefeys et al. ICCV’99)
Works for all relative motionsGuarantees minimal image size
![Page 69: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/69.jpg)
69
polar rectification: example
![Page 70: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/70.jpg)
70
polar rectification: example
![Page 71: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/71.jpg)
71
Example: Béguinage of Leuven
Does not work with standard Homography-based approaches
![Page 72: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/72.jpg)
72
Example: Béguinage of Leuven
![Page 73: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/73.jpg)
General iso-disparity surfaces(Pollefeys and Sinha, ECCV’04)(Pollefeys and Sinha, ECCV’04)
Example: polar rectification preserves disp.
Application: Active vision
Also interesting relation to human horopter
![Page 74: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/74.jpg)
74
Reconstruction from Rectified Images
Disparity: d=u’-u. Depth: z = -B/d.
![Page 75: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/75.jpg)
75
Three Views
The third eye can be used for verification..
![Page 76: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/76.jpg)
width of a pixel
Choosing the stereo baseline
•What’s the optimal baseline?– Too small: large depth error– Too large: difficult search problem, more
occlusions
Large Baseline Small Baseline
all of thesepoints projectto the same pair of pixels
![Page 77: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/77.jpg)
The Effect of Baseline on Depth Estimation
(Okutami and Kanade PAMI’93)
![Page 78: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/78.jpg)
![Page 79: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/79.jpg)
80
I1 I2 I10
Reprinted from “A Multiple-Baseline Stereo System,” by M. Okutami and T. Kanade, IEEE Trans. on PatternAnalysis and Machine Intelligence, 15(4):353-363 (1993). \copyright 1993 IEEE.
![Page 80: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/80.jpg)
Some Solutions• Multi-view linking (Koch et al. ECCV98)• Best of left or right (Kang et al. CVPR01)• Best k out of n , ...
Problem: visibility
![Page 81: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/81.jpg)
• Multi-baseline, multi-resolution• At each depth, baseline and
resolution selected proportional to that depth
• Allows to keep depth accuracy constant!
Variable Baseline/Resolution Stereo(Gallup et al., CVPR08)
![Page 82: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/82.jpg)
Variable Baseline/Resolution Stereo: comparison
![Page 83: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/83.jpg)
84
Multi-view depth fusion
• Compute depth for every pixel of reference image– Triangulation– Use multiple views– Up- and down sequence– Use Kalman filter
(Koch, Pollefeys and Van Gool. ECCV‘98)
Allows to compute robust texture
![Page 84: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/84.jpg)
85
Real-time stereo on graphics hardware
• Computes Sum-of-Square-Differences• Hardware mip-map generation used to aggregate
results over support region• Trade-off between small and large support window
Yang and Pollefeys CVPR03
140M disparity hypothesis/sec on Radeon 9700proe.g. 512x512x20disparities at 30Hz
Shape of a kernel for summing up 6 levels
![Page 85: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/85.jpg)
87
(1x1)
(1x1+2x2)
(1x1+2x2+4x4+8x8)
(1x1+2x2+4x4+8x8+16x16)
Combine multiple aggregation windows using hardware mipmap and multiple texture units in single pass
![Page 86: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/86.jpg)
Plane-sweep multi-view matching
• Simple algorithm for multiple cameras• no rectification necessary• doesn’t deal with occlusions
Collins’96; Roy and Cox’98 (GC); Yang et al.’02/’03 (GPU)
![Page 87: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/87.jpg)
Fast GPU-based plane-sweeping stereo
Plane-sweep multi-view depth estimation(Yang & Pollefeys, CVPR’03)
![Page 88: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/88.jpg)
Plane Sweep Stereo
Plane Sweep Matching Score
Images Projected on Plane
90
![Page 89: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/89.jpg)
Plane Sweep Stereo Result
• Ideal for GPU processing• 512x384 depthmap, 50 planes, 5 images:
50 Hz
91
![Page 90: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/90.jpg)
Plane Sweep Stereo
Plane Sweep Matching Score (SAD)
Images Projected on Plane
92
![Page 91: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/91.jpg)
Window-Based Matching - Local Stereo
• Pixel-to-pixel matching is ambiguous for local stereo
• Use a matching window (SAD/SSD/NCC)
• Assumes all pixels in window are on the plane
93
![Page 92: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/92.jpg)
Multi-directional plane-sweeping stereo
• Select best-cost over depths AND orientation
3D model from 11 video frames (hand-held)
Automatic computations of dominant orientations
(Gallup, Frahm & Pollefeys, CVPR07)
![Page 93: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/93.jpg)
Plane Priors and Quicksweep
without plane prior with plane prior
)()|()|( iiiii PcPcP πππ ∝
Use SfM points as a prior for stereo
For real-time: test only the planes with high prior probability
![Page 94: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/94.jpg)
Selecting Best Depth / Surface Normal Combining multiple sweeps
Sweep 1 Sweep 2 Sweep 3
How to combine sweeps?
Global labeling
Graph Cut:1-2 seconds
Continuous Formulation: 45 ms
Local best cost
OR
(Zach, Gallup, Frahm,Niethammer VMV 2008)
![Page 95: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/95.jpg)
Results
1474 frame reconstruction, 11 images in stereo,512x384 grayscale, 48 plane hypotheses (quick sweep),
processing rate 20.0 Hz (Nvidia 8800 GTX)
98
![Page 96: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/96.jpg)
Fast visibility-based fusion of depth-maps(Merrell , Akbarzadeh, Wang, Mordohai,Frahm, Yang, Nister, Pollefeys ICCV07)
visibility constraintsstability-based fusion
Two fusion approachesconfidence-based fusion
O(n2) O(n)
Fast on GPU as key operation is rendering depth-maps into other views
O(n)
• Complements very fast multi-view stereo
![Page 97: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/97.jpg)
100
Fast video-based modeling of cities
• Fast video processing pipeline• - up to 26Hz on single CPU/GPU• - Most image processing on GPU• (x10-x100 faster)
• - Exploits urban structure
• - Generates textured 3D mesh
100
(Pollefeys et al. IJCV, 2008)
![Page 98: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/98.jpg)
101
building surveyed to 6mm
cm
cm
cm
cm
cm
cm
cm
error histogram
RMS error: 13.4cm mean error: 6.8cmmedian error: 3.0cm
3D-from-video evaluation: Firestone building
![Page 99: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/99.jpg)
102
3D-from-video evaluation: Middlebury Multi-View Stereo Evaluation
Benchmark
102
Ring datasets: 47 images
Results competitive but much, much faster(30 minutes → 30 seconds)
![Page 100: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/100.jpg)
1.3 million video frames(Chapel Hill, NC)
103
![Page 101: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/101.jpg)
• 1.3 million frames (2 cams per side)• 26 Hz reconstruction frame rate
Computation time:1PC (3Ghz CPU+ Nvidia 8800 GTX):
14hrs @ 26fps2 weeks @ 1fps
2.5 years @ 1fpm104
![Page 102: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/102.jpg)
105
• 1.3 million frames (2 cams per side)• 26 Hz reconstruction frame rate
Computation time:1PC (3Ghz CPU+ Nvidia 8800 GTX):
14hrs @ 26fps
![Page 103: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/103.jpg)
106
![Page 104: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/104.jpg)
Another Scene
108
view 1 view N. . . . . .
![Page 105: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/105.jpg)
Depthmap
109
view 1 referenceview
view N. . . . . .
![Page 106: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/106.jpg)
3D Model: Standard Stereo
110
![Page 107: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/107.jpg)
3D Model: Standard Stereo
111
![Page 108: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/108.jpg)
3D Model: Standard Stereo
112
![Page 109: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/109.jpg)
3D Model: Standard Stereo
113
![Page 110: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/110.jpg)
How Accurate Is Stereo?
b
z
f
d
dbfz −=
ddbfdzz ∆=∆≈∆ 2'
Δz
Δd
dbfzz ∆≈∆
2
b = baselinef = focal lengthd = disparityz = depth
114
![Page 111: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/111.jpg)
• Multi-baseline, multi-resolution• At each depth, baseline and resolution
selected proportional to that depth• Allows to keep depth accuracy
constant!
Variable Baseline/Resolution Stereo (Gallup et al., CVPR08
![Page 112: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/112.jpg)
•Analysis of matching scores at various baselines and scales
– Location of global minimum unchanged– Value of global minimum preserved
Verification of consistent results
![Page 113: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/113.jpg)
Variable Baseline/Resolution Stereo: comparison
![Page 114: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/114.jpg)
Results
138
![Page 115: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/115.jpg)
Depth Resolution Limits
139
![Page 116: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/116.jpg)
Piecewise Planar Stereo
•Fit planes to scene•But also handle non-planar regions•Use planar appearance•Ensure consistency over video sequences
140
![Page 117: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/117.jpg)
3D Reconstruction from Video
Street-Side Video Real-Time Stereo
141
![Page 118: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/118.jpg)
Method
Video Frame Depthmap withRANSAC planes
Planar ClassProbability Map
Graph-Cut Labeling
3D Model
Flowchart
142
(Gallup et al. CVPR 2010)
![Page 119: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/119.jpg)
149
![Page 120: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/120.jpg)
Heightmap Model
• Enforces vertical facades• One continuous surface, no holes• 2.5D: Fast to compute, easy to store
150
(Gallup et al. 3DPVT 2010)
![Page 121: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/121.jpg)
Occupancy Voting
d
)/exp( σdV −=
emptyV λ−=
depth
voxel
voxel
This is alog-oddslikelihood
camera 151
![Page 122: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/122.jpg)
n-layer heightmap fusion
• Generalize to n-layer heightmap• Each layer is a transition from full/empty or
empty/full
• Compute layer positions with dynamic programming
• Use model selection (BIC) to determine number of layers
152
(Gallup et al. DAGM’10)
![Page 123: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/123.jpg)
n-layer heightmap fusion
Richar
ICVSS 2010 153
![Page 124: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/124.jpg)
Reconstruction from Photo Collections• Millions of images download from Flickr• Camera poses computed using (Li et al.
ECCV’08)
• Depthmaps from GPU planesweep
154
![Page 125: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/125.jpg)
Results
![Page 126: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/126.jpg)
156
More on stereo …
The Middleburry Stereo Vision Research Pagehttp://cat.middlebury.edu/stereo/
Recommended reading
D. Scharstein and R. Szeliski. A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms.IJCV 47(1/2/3):7-42, April-June 2002. PDF file (1.15 MB) - includes current evaluation.Microsoft Research Technical Report MSR-TR-2001-81, November 2001. PDF file (1.27 MB).
![Page 127: Stereo matching - CVG · simplify stereo matching . by warping the images. Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole](https://reader030.fdocuments.in/reader030/viewer/2022040912/5e86256ce644a064341f5b18/html5/thumbnails/127.jpg)
256
Next week:2D Shape Matching