Post on 31-May-2020
Lecture 6 -Silvio Savarese 5-Feb-14
Professor Silvio Savarese
Computational Vision and Geometry Lab
Lecture 6Stereo SystemsMulti-view geometry
Lecture 5 -Silvio Savarese 5-Feb-14
• Stereo systems• Rectification• Correspondence problem
• Multi-view geometry• The SFM problem• Affine SFM
Reading: [AZ] Chapter: 4 “Estimation – 2D perspective transformationsChapter: 9 “Epipolar Geometry and the Fundamental Matrix Transformation”Chapter: 11 “Computation of the Fundamental Matrix F”
[FP] Chapters: 10 “The geometry of multiple views”
Lecture 6Stereo SystemsMulti-view geometry
• Epipolar Plane • Epipoles e1, e2
• Epipolar Lines
• Baseline
Epipolar geometry
O1O2
x2
X
x1
e1 e2
= intersections of baseline with image planes
= projections of the other camera center
For details see CS131A
Lecture 9
OO’
pp’
P
Epipolar Constraint
0pEpT
E = Essential Matrix(Longuet-Higgins, 1981)
RTE
OO’
pp’
P
Epipolar Constraint
0pFpT
F = Fundamental Matrix(Faugeras and Luong, 1992)
1
KRTKFT
O1O2
X
e2
x1 x2
e2
Example: Parallel image planes
K K
x
y
z
• Epipolar lines are horizontal
• Epipoles go to infinity
• y-coordinates are equal
1
1
1 y
x
x
1
2
2 y
x
x
Rectification: making two images “parallel”
H
Courtesy figure S. Lazebnik
For details on how to rectify two views see CS131A Lecture 9
Why are parallel images useful?
• Makes the correspondence problem easier• Makes triangulation easy• Enables schemes for image interpolation
Application: view morphingS. M. Seitz and C. R. Dyer, Proc. SIGGRAPH 96, 1996, 21-30
Morphing without using geometry
Rectification
x
x
x
y
y
y
î1
î0
îS
I0
I1
Is
P
C1
Cs
C0
Why are parallel images useful?
• Makes the correspondence problem easier• Makes triangulation easy• Enables schemes for image interpolation
Point triangulation
O O’
P
e2
p p’e2
K Kx
y
z
u
v
u
v
f
u u’
Baseline B
z
O O’
P
f
z
fBuu
'
Note: Disparity is inversely proportional to depth
Computing depth
= disparity
Disparity maps
http://vision.middlebury.edu/stereo/
Stereo pair
Disparity map / depth map Disparity map with occlusions
u u’
z
fBuu
'
Why are parallel images useful?
• Makes the correspondence problem easier• Makes triangulation easy• Enables schemes for image interpolation
Correspondence problem
p’
P
p
O1O2
Given a point in 3d, discover corresponding observations
in left and right images [also called binocular fusion problem]
Correspondence problem
p’
P
p
O1O2
Given a point in 3d, discover corresponding observations
in left and right images [also called binocular fusion problem]
•A Cooperative Model (Marr and Poggio, 1976)
•Correlation Methods (1970--)
•Multi-Scale Edge Matching (Marr, Poggio and Grimson, 1979-81)
Correspondence problem
[FP] Chapters: 11
Correlation Methods (1970--)
210
p p’
195 150 210
Correlation Methods (1970--)
• Pick up a window around p(u,v)
p
Correlation Methods (1970--)
• Pick up a window around p(u,v)
• Build vector w
• Slide the window along v line in image 2 and compute w’
• Keep sliding until w∙w’ is maximized.
Correlation Methods (1970--)
Normalized Correlation; minimize:)ww)(ww(
)ww)(ww(
Left Right
scanline
Correlation methods
Norm. corrCredit slide S. Lazebnik
– Smaller window
- More detail
- More noise
– Larger window
- Smoother disparity maps
- Less prone to noise
Window size = 3 Window size = 20
Correlation methods
Credit slide S. Lazebnik
Issues
•Fore shortening effect
•Occlusions
OO’
O O’
O O’
Issues
• Small error in
measurements
implies large error
in estimating
depth
• It is desirable
to have small
B/z ratio!
•Homogeneous regions
Issues
Hard to match pixels in these regions
•Repetitive patterns
Issues
Correspondence problem is difficult!
- Occlusions
- Fore shortening
- Baseline trade-off
- Homogeneous regions
- Repetitive patterns
Apply non-local constraints to help enforce the correspondences
Results with window search
Data
Ground truth
Credit slide S. Lazebnik
Window-based matching
Lecture 5 -Silvio Savarese 5-Feb-14
• Stereo systems• Rectification• Correspondence problem
• Multi-view geometry• The SFM problem• Affine SFM
Lecture 6Stereo SystemsMulti-view geometry
Courtesy of Oxford Visual Geometry Group
Structure from motion problem
Structure from motion problem
x1j
x2j
xmj
Xj
M1
M2
Mm
Given m images of n fixed 3D points
•xij = Mi Xj , i = 1, … , m, j = 1, … , n
From the mxn correspondences xij, estimate:
•m projection matrices Mi
•n 3D points Xj
x1j
x2j
xmj
Xj
motion
structure
M1
M2
Mm
Structure from motion problem
Affine structure from motion(simpler problem)
ImageWorld
Image
From the mxn correspondences xij, estimate:
•m projection matrices Mi (affine cameras)
•n 3D points Xj
Finite cameras
p
q
r
R
Q
P
O
XTRKx
M
??10
0100
0010
0001
TR
Question:
Finite cameras
p
q
r
R
Q
P
O
XTRKx
10
TR
0100
0010
0001
KM 3x3
Canonical perspective projection matrix
Affine homography
(in 3D)
Affine
Homography
(in 2D)
100
y0
xs
K oy
ox
p
q
r
R
Q
O
P
Affine cameras
100
yα0
xsα
oy
ox
K
10
TR
1000
0010
0001
KM
Canonical affine projection matrix
(points at infinity are mapped as points at infinity)
R’
Q’
P’
What’s the main difference???
Transformation in 2D
Affinities:
1
y
x
H
1
y
x
10
tA
1
'y
'x
a
Affine cameras
XTRKx
100
00
00
y
x
K
10
TR
1000
0010
0001
KM
10
bA
1000
]affine44[
1000
0010
0001
]affine33[ 2232221
1131211
baaa
baaa
M
12
1
232221
131211 XbAXx EucM
b
b
Z
Y
X
aaa
aaa
y
x
[Homogeneous]
[non-homogeneous
image coordinates] bAMMEuc
;1
PEucM
Pp
p’
;1
PbAPp M
v
u bAM
Affine cameras
M = camera matrix
To recap: from now on we define M as the camera matrix for the affine case
The Affine Structure-from-Motion Problem
Given m images of n fixed points Pj (=Xi) we can write
Problem: estimate the m 24 matrices Mi and
the n positions Pj from the mn correspondences pij .
2m n equations in 8m+3n unknowns
Two approaches:- Algebraic approach (affine epipolar geometry; estimate F; cameras; points)
- Factorization method
How many equations and how many unknown?
N of cameras N of points
A factorization method –Tomasi & Kanade algorithm
C. Tomasi and T. Kanade. Shape and motion from image streams under orthography: A factorization
method. IJCV, 9(2):137-154, November 1992.
• Centering the data
• Factorization
Next lecture
Multiple view geometry