Post on 12-Oct-2020
CSE 152, Spring 2018 Introduction to Computer Vision
Stereo(Part 2)
Introduction to Computer VisionCSE 152Lecture 9
CSE 152, Spring 2018 Introduction to Computer Vision
Announcements• Homework 3 is due May 9, 11:59 PM• Reading:
– Chapter 7: Stereopsis
CSE 152, Spring 2018 Introduction to Computer Vision
Stereo Vision Outline• Offline: Calibrate cameras & determine
“epipolar geometry”• Online
1. Acquire stereo images2. Rectify images to convenient epipolar geometry3. Establish correspondence 4. Estimate depthA
B
C
D
CSE 152, Spring 2018 Introduction to Computer Vision
RectificationGiven a pair of images, transform both images so that epipolar lines are scan lines.
e1
e2
CSE 152, Spring 2018 Introduction to Computer Vision
RectificationUnder perspective projection, the mapping from a plane to a plane is given by a 2D projective transformation (homography)
(uL,vL)
(xL, yL)
xL
yL
wL
HL
uL
vL
1
CSE 152, Spring 2018 Introduction to Computer Vision
Rectification
(uL,vL)
(xL, yL)
Under perspective projection, the mapping from a plane to a plane is given by a 2D projective transformation (homography)
(uR,vR)
(xR, yR)
xR
yR
wR
HR
uR
vR
1
xL
yL
wL
HL
uL
vL
1
Two images – Two projective transformations
CSE 152, Spring 2018 Introduction to Computer Vision
Epipolar Rectification• Create pair of virtual cameras
– Virtual cameras have the same camera centers as real cameras
– Both virtual cameras have the same:
• Camera rotation matrix R• Camera calibration matrix K
• Rectification transformation matricesH KvirtualRvirtualRreal
T Kreal
SceneCameracenters
Realimages
CSE 152, Spring 2018 Introduction to Computer Vision
Image pair rectification
Simplify stereo matching by warping the images
Apply projective transformation so that epipolar linescorrespond to horizontal scanlines
e
H should map epipole e to (1,0,0), a point at infinity on the x-axis
H should minimize image distortion
Note that rectified images are usually not rectangularSee textbook for complete method
He001
(uL,vL) e(xL, yL)
H
CSE 152, Spring 2018 Introduction to Computer Vision
RectificationGiven a pair of images, transform both images so that epipolar lines are scan lines.
Input Images
CSE 152, Spring 2018 Introduction to Computer Vision
RectificationGiven a pair of images, transform both images so that epipolar lines are scan lines.
Rectified ImagesSee Section 7.2.1 for specific method
epipolar lines run parallel with the x-axis and are
aligned between two views (no y disparity)
CSE 152, Spring 2018 Introduction to Computer Vision
Rectification
Original
Rectified
CSE 152, Spring 2018 Introduction to Computer Vision
Rectification• Epipolar lines
CSE 152, Spring 2018 Introduction to Computer Vision
Rectification
Original
Rectified
CSE 152, Spring 2018 Introduction to Computer Vision
Polar Rectification
Homography-basedRectification
Polar Rectification
Alternative epipolarrectification method that
minimizes pixel distortion
CSE 152, Spring 2018 Introduction to Computer Vision
Polar Rectification
Epipoles are in images(white dot on ball)
Homography-based rectificationis not possible
CSE 152, Spring 2018 Introduction to Computer Vision
Features on same epipolar line
Truco Fig. 7.5
CSE 152, Spring 2018 Introduction to Computer Vision
Mobi: Stereo-based navigation
CSE 152, Spring 2018 Introduction to Computer Vision
Epipolar correspondence
CSE 152, Spring 2018 Introduction to Computer Vision
Symbolic Map
CSE 152, Spring 2018 Introduction to Computer Vision
Using epipolar & constant Brightness constraints for stereo matching
For each epipolar lineFor each pixel in the left image
• compare with every pixel on same epipolar line in right image
• pick pixel with minimum match cost• This will never work, so:
match windows
(Seitz)
CSE 152, Spring 2018 Introduction to Computer Vision
Finding Correspondences
W(pl) W(pr)
CSE 152, Spring 2018 Introduction to Computer Vision
Comparing Windows: =?
f g
Mostpopular
(Camps)
For each window, match to closest window on epipolar line in other image.
CSE 152, Spring 2018 Introduction to Computer Vision
Correspondence Search Algorithm
For i = 1:nrowsfor j=1:ncols
best(i,j) = -1for k = mindisparity:maxdisparity
c = Match_Metric(I1(i,j),I2(i,j+k),winsize)if (c > best(i,j))
best(i,j) = cdisparities(i,j) = k
endend
endend
O(nrows * ncols * disparities * winx * winy)
I1 I2
uv
d
I1 I2
uv
d
CSE 152, Spring 2018 Introduction to Computer Vision
Match Metric Summary
vu vu
vu
IvduIIvuI
IvduIIvuI
, ,
222
211
,2211
,,
,,
vu
vduIvuI,
221 ,,
vu
vuvuIvduI
IvduI
IvuI
IvuI,
2
,
222
22
,
211
11
,
,
,
,
vu
vduIvuI,
21 ,,
vu
vduIvuI,
'2
'1 ,,
nm
kkk vuInmIvuI,
' ,,,
vu
vduIvuIHAMMING,
'2
'1 ,,,
vuInmIBITSTRINGvuI kknmk ,,, ,'
MATCH METRIC DEFINITION
Normalized Cross-Correlation (NCC)
Sum of Squared Differences (SSD)
Normalized SSD
Sum of Absolute Differences (SAD)
Zero Mean SAD
Rank
Census
These two are actually
the same
vu
IvduIIvuI,
_
22
_
11 ),(),(
CSE 152, Spring 2018 Introduction to Computer Vision
Stereo results
Ground truthScene
– Data from University of Tsukuba
(Seitz)
CSE 152, Spring 2018 Introduction to Computer Vision
Results with window correlation
Window-based matching(best window size)
Ground truth
(Seitz)
CSE 152, Spring 2018 Introduction to Computer Vision
Results with better method
Using global optimizationBoykov et al., Fast Approximate Energy Minimization via Graph Cuts,
International Conference on Computer Vision, September 1999.
Ground truth
(Seitz)
CSE 152, Spring 2018 Introduction to Computer Vision
State of the Art Results
Using neural networks Ground truth
S. Drouyer, S. Beucher, M. Bilodeau, M. Moreaud, and L. Sorbier. Sparse stereo disparity map densification using hierarchical image
segmentation. 13th International Symposium on Mathematical Morphology.
CSE 152, Spring 2018 Introduction to Computer Vision
Some Issues• Epipolar ordering• Ambiguity• Window size• Window shape• Lighting• Half occluded regions
CSE 152, Spring 2018 Introduction to Computer Vision
A challenge: Multiple Interpretations
Each feature on left epipolar line match oneand only one feature on right epipolar line.
CSE 152, Spring 2018 Introduction to Computer Vision
Multiple Interpretations
Each feature on left epipolar line match oneand only one feature on right epipolar line.
CSE 152, Spring 2018 Introduction to Computer Vision
Multiple Interpretations
Each feature on left epipolar line match oneand only one feature on right epipolar line.
CSE 152, Spring 2018 Introduction to Computer Vision
Multiple Interpretations
Each feature on left epipolar line match oneand only one feature on right epipolar line.
CSE 152, Spring 2018 Introduction to Computer Vision
Some Issues• Epipolar ordering• Ambiguity• Window size• Window shape• Lighting• Half occluded regions
CSE 152, Spring 2018 Introduction to Computer Vision
Ambiguity
CSE 152, Spring 2018 Introduction to Computer Vision
Some Issues• Epipolar ordering• Ambiguity• Window size• Window shape• Lighting• Half occluded regions
CSE 152, Spring 2018 Introduction to Computer Vision
Window size
W = 3 W = 20
Better results with adaptive window• T. Kanade and M. Okutomi, A Stereo Matching
Algorithm with an Adaptive Window: Theory and Experiment,, Proc. International Conference on
Robotics and Automation, 1991. • D. Scharstein and R. Szeliski. Stereo matching with
nonlinear diffusion. International Journal of Computer Vision, 28(2):155-174, July 1998
• Effect of window size
(Seitz)
CSE 152, Spring 2018 Introduction to Computer Vision
Some Issues• Epipolar ordering• Ambiguity• Window size• Window shape• Lighting• Half occluded regions
CSE 152, Spring 2018 Introduction to Computer Vision
Window Shape and Forshortening
CSE 152, Spring 2018 Introduction to Computer Vision
Wl
Wr
Wp
U1U2
Window Shape: Fronto-parallel Configuration
CSE 152, Spring 2018 Introduction to Computer Vision
Some Issues• Epipolar ordering• Window size• Ambiguity• Window shape• Lighting• Half occluded regions
CSE 152, Spring 2018 Introduction to Computer Vision
Lighting Conditions (Photometric Variations)
W(Pl) W(Pr)
CSE 152, Spring 2018 Introduction to Computer Vision
Some Issues• Epipolar ordering• Ambiguity• Window size• Window shape• Lighting• Half occluded regions
CSE 152, Spring 2018 Introduction to Computer Vision
Half occluded regions
CSE 152, Spring 2018 Introduction to Computer Vision
Summary of Stereo ConstraintsCONSTRAINT BRIEF DESCRIPTION
1-D Epipolar Search Arbitrary images of the same scene may be rectified based onepipolar geometry such that stereo matches lie along one-dimensional scanlines. This reduces the computational complexityand also reduces the likelihood of false matches.
Monotonic Ordering Points along an epipolar scanline appear in the same order in bothstereo images, assuming that all objects in the scene areapproximately the same distance from the cameras.
Image Brightness Constancy
Assuming Lambertian surfaces, the brightness of correspondingpoints in stereo images are the same.
Match Uniqueness For every point in one stereo image, there is at most onecorresponding point in the other image.
Disparity Continuity Disparities vary smoothly (i.e. disparity gradient is small) over most ofthe image. This assumption is violated at object boundaries.
Disparity Limit The search space may be reduced significantly by limiting thedisparity range, reducing both computational complexity and thelikelihood of false matches.
Fronto-Parallel Surfaces
The implicit assumption made by area-based matching is that objectshave fronto-parallel surfaces (i.e. depth is constant within the regionof local support). This assumption is violated by sloping and creasedsurfaces.
Feature Similarity Corresponding features must be similar (e.g. edges must haveroughly the same length and orientation).
Structural Grouping Corresponding feature groupings and their connectivity must beconsistent.
(From G. Hager)
CSE 152, Spring 2018 Introduction to Computer Vision
Next Lecture• Early vision: multiple images
– Structure from motion• Reading:
– Chapter 8: Structure from Motion