lecture6 stereo systems - Stanford...
Transcript of lecture6 stereo systems - Stanford...
![Page 1: lecture6 stereo systems - Stanford Universityweb.stanford.edu/class/cs231a/lectures/lecture6_stereo_systems.pdf · •Pick up a window Waround image 1 p=(u,v) •Build vector w Example:](https://reader034.fdocuments.in/reader034/viewer/2022051901/5ff0b91f0364f06605054ddc/html5/thumbnails/1.jpg)
Lecture 6 -Silvio Savarese 2-Feb-18
ProfessorSilvioSavareseComputationalVisionandGeometryLab
Lecture6StereoSystemsMulti-viewgeometry
![Page 2: lecture6 stereo systems - Stanford Universityweb.stanford.edu/class/cs231a/lectures/lecture6_stereo_systems.pdf · •Pick up a window Waround image 1 p=(u,v) •Build vector w Example:](https://reader034.fdocuments.in/reader034/viewer/2022051901/5ff0b91f0364f06605054ddc/html5/thumbnails/2.jpg)
Lecture 6 -Silvio Savarese 2-Feb-18
• Stereosystems• Rectification• Correspondenceproblem
• Multi-viewgeometry• TheSFMproblem• AffineSFM
Reading: [AZ] Chapter:9“Epip.Geom.andtheFundam.MatrixTransf.”[AZ] Chapter:18“Nviewcomputationalmethods”[FP] Chapters:7“Stereopsis”[FP] Chapters:8“StructurefromMotion”
Lecture6StereoSystemsMulti-viewgeometry
![Page 3: lecture6 stereo systems - Stanford Universityweb.stanford.edu/class/cs231a/lectures/lecture6_stereo_systems.pdf · •Pick up a window Waround image 1 p=(u,v) •Build vector w Example:](https://reader034.fdocuments.in/reader034/viewer/2022051901/5ff0b91f0364f06605054ddc/html5/thumbnails/3.jpg)
• Epipolar Plane • Epipoles e, e’
• Epipolar Lines• Baseline
Epipolar geometry
p’
P
p
e e’
= intersections of baseline with image planes = projections of the other camera center
O1 O2
![Page 4: lecture6 stereo systems - Stanford Universityweb.stanford.edu/class/cs231a/lectures/lecture6_stereo_systems.pdf · •Pick up a window Waround image 1 p=(u,v) •Build vector w Example:](https://reader034.fdocuments.in/reader034/viewer/2022051901/5ff0b91f0364f06605054ddc/html5/thumbnails/4.jpg)
pp’
P
Epipolar Constraint
0=¢pEpT
E = Essential Matrix(Longuet-Higgins, 1981)
[ ] RTE ×= ´
O1 O2
![Page 5: lecture6 stereo systems - Stanford Universityweb.stanford.edu/class/cs231a/lectures/lecture6_stereo_systems.pdf · •Pick up a window Waround image 1 p=(u,v) •Build vector w Example:](https://reader034.fdocuments.in/reader034/viewer/2022051901/5ff0b91f0364f06605054ddc/html5/thumbnails/5.jpg)
E =
0 −Tz TyTz 0 −Tx−Ty Tx 0
"
#
$$$$
%
&
''''
R
Essentialmatrix
[ ] RTE ×= ´
![Page 6: lecture6 stereo systems - Stanford Universityweb.stanford.edu/class/cs231a/lectures/lecture6_stereo_systems.pdf · •Pick up a window Waround image 1 p=(u,v) •Build vector w Example:](https://reader034.fdocuments.in/reader034/viewer/2022051901/5ff0b91f0364f06605054ddc/html5/thumbnails/6.jpg)
pp’
P
Epipolar Constraint
0pFpT =¢
F = Fundamental Matrix(Faugeras and Luong, 1992)
[ ] 1-´
- ¢××= KRTKF T
O1 O2
![Page 7: lecture6 stereo systems - Stanford Universityweb.stanford.edu/class/cs231a/lectures/lecture6_stereo_systems.pdf · •Pick up a window Waround image 1 p=(u,v) •Build vector w Example:](https://reader034.fdocuments.in/reader034/viewer/2022051901/5ff0b91f0364f06605054ddc/html5/thumbnails/7.jpg)
Parallel image planes
• Epipolar lines are horizontal• Epipoles go to infinity• v-coordinates are equal
p =pupv1
!
"
####
$
%
&&&&
p ' =p 'upv1
!
"
####
$
%
&&&&
O1 O2
P
e’p p’
e
x
y
z
u
v
![Page 8: lecture6 stereo systems - Stanford Universityweb.stanford.edu/class/cs231a/lectures/lecture6_stereo_systems.pdf · •Pick up a window Waround image 1 p=(u,v) •Build vector w Example:](https://reader034.fdocuments.in/reader034/viewer/2022051901/5ff0b91f0364f06605054ddc/html5/thumbnails/8.jpg)
O1 O2
Parallel image planes
E=?K1=K2 = knownHint :
x parallel to O1O2
R = I T = (T, 0, 0)
P
e’p p’
e
x
y
z
u
v
![Page 9: lecture6 stereo systems - Stanford Universityweb.stanford.edu/class/cs231a/lectures/lecture6_stereo_systems.pdf · •Pick up a window Waround image 1 p=(u,v) •Build vector w Example:](https://reader034.fdocuments.in/reader034/viewer/2022051901/5ff0b91f0364f06605054ddc/html5/thumbnails/9.jpg)
E =
0 −Tz TyTz 0 −Tx−Ty Tx 0
"
#
$$$$
%
&
''''
R =0 0 00 0 −T0 T 0
"
#
$$$
%
&
'''
Essentialmatrixforparallelimages
T = [ T 0 0 ]R = I
[ ] RTE ×= ´
![Page 10: lecture6 stereo systems - Stanford Universityweb.stanford.edu/class/cs231a/lectures/lecture6_stereo_systems.pdf · •Pick up a window Waround image 1 p=(u,v) •Build vector w Example:](https://reader034.fdocuments.in/reader034/viewer/2022051901/5ff0b91f0364f06605054ddc/html5/thumbnails/10.jpg)
O1
P
p p’
Parallel image planes
x
y
z
horizontal!What are the directions of epipolar lines?
l = E p '
l l’
u
v
e’e
u’
v’
=0 0 00 0 −T0 T 0
"
#
$$$
%
&
'''
u 'v '1
"
#
$$$
%
&
'''=
0−TT v '
"
#
$$$
%
&
'''
![Page 11: lecture6 stereo systems - Stanford Universityweb.stanford.edu/class/cs231a/lectures/lecture6_stereo_systems.pdf · •Pick up a window Waround image 1 p=(u,v) •Build vector w Example:](https://reader034.fdocuments.in/reader034/viewer/2022051901/5ff0b91f0364f06605054ddc/html5/thumbnails/11.jpg)
Parallel image planes
How are pand p’ related?
pT ⋅ E p ' = 0
P
e’p p’
e
y
z
xO1
u
v
u’
v’
![Page 12: lecture6 stereo systems - Stanford Universityweb.stanford.edu/class/cs231a/lectures/lecture6_stereo_systems.pdf · •Pick up a window Waround image 1 p=(u,v) •Build vector w Example:](https://reader034.fdocuments.in/reader034/viewer/2022051901/5ff0b91f0364f06605054ddc/html5/thumbnails/12.jpg)
Parallel image planes
( ) ( )T
0 0 0 0p E p 0 1 0 0 0 1 0
0 0 1
uu v T v u v T Tv Tv
T Tv
¢é ù æ ö æ öç ÷ ç ÷ê ú¢ ¢ ¢= Þ - = Þ - = Þ =ç ÷ ç ÷ê úç ÷ ç ÷¢ê úë û è ø è ø
How are p and p’ related?
P
e’p p’
e
y
z
O1x
⇒ v = "v
u
v
u’
v’
![Page 13: lecture6 stereo systems - Stanford Universityweb.stanford.edu/class/cs231a/lectures/lecture6_stereo_systems.pdf · •Pick up a window Waround image 1 p=(u,v) •Build vector w Example:](https://reader034.fdocuments.in/reader034/viewer/2022051901/5ff0b91f0364f06605054ddc/html5/thumbnails/13.jpg)
Parallel image planes
Rectification: making two images “parallel” Why it is useful? • Epipolar constraint ® v = v’
• New views can be synthesized by linear interpolation
P
e’p p’
e
y
z
O1x
u
v
u’
v’
![Page 14: lecture6 stereo systems - Stanford Universityweb.stanford.edu/class/cs231a/lectures/lecture6_stereo_systems.pdf · •Pick up a window Waround image 1 p=(u,v) •Build vector w Example:](https://reader034.fdocuments.in/reader034/viewer/2022051901/5ff0b91f0364f06605054ddc/html5/thumbnails/14.jpg)
Rectification:makingtwoimages“parallel”
H
Courtesy figure S. Lazebnik
![Page 15: lecture6 stereo systems - Stanford Universityweb.stanford.edu/class/cs231a/lectures/lecture6_stereo_systems.pdf · •Pick up a window Waround image 1 p=(u,v) •Build vector w Example:](https://reader034.fdocuments.in/reader034/viewer/2022051901/5ff0b91f0364f06605054ddc/html5/thumbnails/15.jpg)
Application: view morphingS. M. Seitz and C. R. Dyer, Proc. SIGGRAPH 96, 1996, 21-30
![Page 16: lecture6 stereo systems - Stanford Universityweb.stanford.edu/class/cs231a/lectures/lecture6_stereo_systems.pdf · •Pick up a window Waround image 1 p=(u,v) •Build vector w Example:](https://reader034.fdocuments.in/reader034/viewer/2022051901/5ff0b91f0364f06605054ddc/html5/thumbnails/16.jpg)
Rectification
u
u
u
v
v
v
î2
î1
îS
I1
I2
Is
P
C2
Cs
C1
H1
H2
![Page 17: lecture6 stereo systems - Stanford Universityweb.stanford.edu/class/cs231a/lectures/lecture6_stereo_systems.pdf · •Pick up a window Waround image 1 p=(u,v) •Build vector w Example:](https://reader034.fdocuments.in/reader034/viewer/2022051901/5ff0b91f0364f06605054ddc/html5/thumbnails/17.jpg)
![Page 18: lecture6 stereo systems - Stanford Universityweb.stanford.edu/class/cs231a/lectures/lecture6_stereo_systems.pdf · •Pick up a window Waround image 1 p=(u,v) •Build vector w Example:](https://reader034.fdocuments.in/reader034/viewer/2022051901/5ff0b91f0364f06605054ddc/html5/thumbnails/18.jpg)
![Page 19: lecture6 stereo systems - Stanford Universityweb.stanford.edu/class/cs231a/lectures/lecture6_stereo_systems.pdf · •Pick up a window Waround image 1 p=(u,v) •Build vector w Example:](https://reader034.fdocuments.in/reader034/viewer/2022051901/5ff0b91f0364f06605054ddc/html5/thumbnails/19.jpg)
![Page 20: lecture6 stereo systems - Stanford Universityweb.stanford.edu/class/cs231a/lectures/lecture6_stereo_systems.pdf · •Pick up a window Waround image 1 p=(u,v) •Build vector w Example:](https://reader034.fdocuments.in/reader034/viewer/2022051901/5ff0b91f0364f06605054ddc/html5/thumbnails/20.jpg)
From its reflection!
![Page 21: lecture6 stereo systems - Stanford Universityweb.stanford.edu/class/cs231a/lectures/lecture6_stereo_systems.pdf · •Pick up a window Waround image 1 p=(u,v) •Build vector w Example:](https://reader034.fdocuments.in/reader034/viewer/2022051901/5ff0b91f0364f06605054ddc/html5/thumbnails/21.jpg)
![Page 22: lecture6 stereo systems - Stanford Universityweb.stanford.edu/class/cs231a/lectures/lecture6_stereo_systems.pdf · •Pick up a window Waround image 1 p=(u,v) •Build vector w Example:](https://reader034.fdocuments.in/reader034/viewer/2022051901/5ff0b91f0364f06605054ddc/html5/thumbnails/22.jpg)
Deep view morphingD. Ji, J. Kwon, M. McFarland, S. Savarese, CVPR 2017
![Page 23: lecture6 stereo systems - Stanford Universityweb.stanford.edu/class/cs231a/lectures/lecture6_stereo_systems.pdf · •Pick up a window Waround image 1 p=(u,v) •Build vector w Example:](https://reader034.fdocuments.in/reader034/viewer/2022051901/5ff0b91f0364f06605054ddc/html5/thumbnails/23.jpg)
Deep view morphingD. Ji, J. Kwon, M. McFarland, S. Savarese, CVPR 2017
![Page 24: lecture6 stereo systems - Stanford Universityweb.stanford.edu/class/cs231a/lectures/lecture6_stereo_systems.pdf · •Pick up a window Waround image 1 p=(u,v) •Build vector w Example:](https://reader034.fdocuments.in/reader034/viewer/2022051901/5ff0b91f0364f06605054ddc/html5/thumbnails/24.jpg)
Deep view morphingD. Ji, J. Kwon, M. McFarland, S. Savarese, CVPR 2017
![Page 25: lecture6 stereo systems - Stanford Universityweb.stanford.edu/class/cs231a/lectures/lecture6_stereo_systems.pdf · •Pick up a window Waround image 1 p=(u,v) •Build vector w Example:](https://reader034.fdocuments.in/reader034/viewer/2022051901/5ff0b91f0364f06605054ddc/html5/thumbnails/25.jpg)
Whyareparallelimagesuseful?
• Makestriangulationeasy• Makesthecorrespondenceproblemeasier
![Page 26: lecture6 stereo systems - Stanford Universityweb.stanford.edu/class/cs231a/lectures/lecture6_stereo_systems.pdf · •Pick up a window Waround image 1 p=(u,v) •Build vector w Example:](https://reader034.fdocuments.in/reader034/viewer/2022051901/5ff0b91f0364f06605054ddc/html5/thumbnails/26.jpg)
Point triangulation
O O’
P
e’p p’e
x
y
z
u
v
u
v
p =pupv1
!
"
####
$
%
&&&&
p ' =p 'upv1
!
"
####
$
%
&&&&
pu − p 'udisparity = [Eq. 1]∝B ⋅ fz
B
z
Disparity is inversely proportional to depth z!
![Page 27: lecture6 stereo systems - Stanford Universityweb.stanford.edu/class/cs231a/lectures/lecture6_stereo_systems.pdf · •Pick up a window Waround image 1 p=(u,v) •Build vector w Example:](https://reader034.fdocuments.in/reader034/viewer/2022051901/5ff0b91f0364f06605054ddc/html5/thumbnails/27.jpg)
p'upuf
B
z
O O’
P
pu − p 'u
Disparity is inversely proportional to depth z!
Computing depth
disparity = [Eq. 1]
uu
(0, pv, 1)
∝B ⋅ fz
![Page 28: lecture6 stereo systems - Stanford Universityweb.stanford.edu/class/cs231a/lectures/lecture6_stereo_systems.pdf · •Pick up a window Waround image 1 p=(u,v) •Build vector w Example:](https://reader034.fdocuments.in/reader034/viewer/2022051901/5ff0b91f0364f06605054ddc/html5/thumbnails/28.jpg)
Disparity mapshttp://vision.middlebury.edu/stereo/
Stereo pair
Disparity map / depth map
pu p’u[Eq. 1]
pu − p 'u ∝B ⋅ fz
![Page 29: lecture6 stereo systems - Stanford Universityweb.stanford.edu/class/cs231a/lectures/lecture6_stereo_systems.pdf · •Pick up a window Waround image 1 p=(u,v) •Build vector w Example:](https://reader034.fdocuments.in/reader034/viewer/2022051901/5ff0b91f0364f06605054ddc/html5/thumbnails/29.jpg)
Whyareparallelimagesuseful?
• Makestriangulationeasy• Makesthecorrespondenceproblemeasier
![Page 30: lecture6 stereo systems - Stanford Universityweb.stanford.edu/class/cs231a/lectures/lecture6_stereo_systems.pdf · •Pick up a window Waround image 1 p=(u,v) •Build vector w Example:](https://reader034.fdocuments.in/reader034/viewer/2022051901/5ff0b91f0364f06605054ddc/html5/thumbnails/30.jpg)
Correspondence problem
p’
P
p
O O’
Given a point in 3D, discover corresponding observations in left and right images [also called binocular fusion problem]
p’ belongs to l’ = FT pp belongs to l = F p’
![Page 31: lecture6 stereo systems - Stanford Universityweb.stanford.edu/class/cs231a/lectures/lecture6_stereo_systems.pdf · •Pick up a window Waround image 1 p=(u,v) •Build vector w Example:](https://reader034.fdocuments.in/reader034/viewer/2022051901/5ff0b91f0364f06605054ddc/html5/thumbnails/31.jpg)
Correspondence problem
p’
P
p
O O’
When images are rectified, this problem is much easier!
![Page 32: lecture6 stereo systems - Stanford Universityweb.stanford.edu/class/cs231a/lectures/lecture6_stereo_systems.pdf · •Pick up a window Waround image 1 p=(u,v) •Build vector w Example:](https://reader034.fdocuments.in/reader034/viewer/2022051901/5ff0b91f0364f06605054ddc/html5/thumbnails/32.jpg)
•A Cooperative Model (Marr and Poggio, 1976)
•Correlation Methods (1970--)
•Multi-Scale Edge Matching (Marr, Poggio and Grimson, 1979-81)
Correspondence problem
[FP] Chapters: 7
![Page 33: lecture6 stereo systems - Stanford Universityweb.stanford.edu/class/cs231a/lectures/lecture6_stereo_systems.pdf · •Pick up a window Waround image 1 p=(u,v) •Build vector w Example:](https://reader034.fdocuments.in/reader034/viewer/2022051901/5ff0b91f0364f06605054ddc/html5/thumbnails/33.jpg)
p
Correlation Methods (1970--)
image 1
Where is is ?p '
Image 2
p =uv1
!
"
###
$
%
&&&
p ' =u 'v1
!
"
###
$
%
&&&
![Page 34: lecture6 stereo systems - Stanford Universityweb.stanford.edu/class/cs231a/lectures/lecture6_stereo_systems.pdf · •Pick up a window Waround image 1 p=(u,v) •Build vector w Example:](https://reader034.fdocuments.in/reader034/viewer/2022051901/5ff0b91f0364f06605054ddc/html5/thumbnails/34.jpg)
p
image 1
Where is is ?p '
Image 2
Correlation Methods (1970--)
100
uu
p =uv1
!
"
###
$
%
&&&
p ' =u 'v1
!
"
###
$
%
&&&
![Page 35: lecture6 stereo systems - Stanford Universityweb.stanford.edu/class/cs231a/lectures/lecture6_stereo_systems.pdf · •Pick up a window Waround image 1 p=(u,v) •Build vector w Example:](https://reader034.fdocuments.in/reader034/viewer/2022051901/5ff0b91f0364f06605054ddc/html5/thumbnails/35.jpg)
u
p
image 1
Where is is ?p '
Image 2
Correlation Methods (1970--)
10030 170 100
Image 2
130
What’s the problem with this?
u
![Page 36: lecture6 stereo systems - Stanford Universityweb.stanford.edu/class/cs231a/lectures/lecture6_stereo_systems.pdf · •Pick up a window Waround image 1 p=(u,v) •Build vector w Example:](https://reader034.fdocuments.in/reader034/viewer/2022051901/5ff0b91f0364f06605054ddc/html5/thumbnails/36.jpg)
• PickupawindowW around
image 1
p = (u,v )• Buildvectorw
image 2
W
u u
Window-based correlation
u
vp
![Page 37: lecture6 stereo systems - Stanford Universityweb.stanford.edu/class/cs231a/lectures/lecture6_stereo_systems.pdf · •Pick up a window Waround image 1 p=(u,v) •Build vector w Example:](https://reader034.fdocuments.in/reader034/viewer/2022051901/5ff0b91f0364f06605054ddc/html5/thumbnails/37.jpg)
• PickupawindowW around
image 1
p = (u,v )• Buildvectorw
Example:W isa3x3windowinredw isa9x1vectorw=[100,100,100,90,100,20,150,150,145]T
• SlidethewindowWalongv=inimage2andcomputew’(u)foreachu
image 2
v
100 100100
90 100 20
150 150145
• ComputethedotproductwT w’(u)foreachuandretainthemaxvalue
uu
Window-based correlation
W
u u ' = u + d = u −1
v
![Page 38: lecture6 stereo systems - Stanford Universityweb.stanford.edu/class/cs231a/lectures/lecture6_stereo_systems.pdf · •Pick up a window Waround image 1 p=(u,v) •Build vector w Example:](https://reader034.fdocuments.in/reader034/viewer/2022051901/5ff0b91f0364f06605054ddc/html5/thumbnails/38.jpg)
Window-based correlation
image 1
Example:W isa3x3windowinredw isa9x1vectorw=[100,100,100,90,100,20,150,150,145]T
image 2
uu
What’s the problem with this?
u ' = u + d = u −1
v100 100100
90 100 20
150 150145
W
u
![Page 39: lecture6 stereo systems - Stanford Universityweb.stanford.edu/class/cs231a/lectures/lecture6_stereo_systems.pdf · •Pick up a window Waround image 1 p=(u,v) •Build vector w Example:](https://reader034.fdocuments.in/reader034/viewer/2022051901/5ff0b91f0364f06605054ddc/html5/thumbnails/39.jpg)
Changes of brightness/exposure
Changes in the mean and the variance of intensity values in corresponding windows!
![Page 40: lecture6 stereo systems - Stanford Universityweb.stanford.edu/class/cs231a/lectures/lecture6_stereo_systems.pdf · •Pick up a window Waround image 1 p=(u,v) •Build vector w Example:](https://reader034.fdocuments.in/reader034/viewer/2022051901/5ff0b91f0364f06605054ddc/html5/thumbnails/40.jpg)
Normalized cross-correlation
Find u that maximizes:(w−w)T ( "w (u)− "w )(w−w) ( "w (u)− "w )
[Eq. 2]
w = w '(u) =mean value within Wlocated at ubar in image 1
mean value within W located at u in image 2
image 1 image 2
uu
u ' = u + d = u +1
v100 100100
90 100 20
150 150145
W
u
![Page 41: lecture6 stereo systems - Stanford Universityweb.stanford.edu/class/cs231a/lectures/lecture6_stereo_systems.pdf · •Pick up a window Waround image 1 p=(u,v) •Build vector w Example:](https://reader034.fdocuments.in/reader034/viewer/2022051901/5ff0b91f0364f06605054ddc/html5/thumbnails/41.jpg)
Image 1 Image 2
scanline
Example
NCC
Credit slide S. Lazebnik
u
v
![Page 42: lecture6 stereo systems - Stanford Universityweb.stanford.edu/class/cs231a/lectures/lecture6_stereo_systems.pdf · •Pick up a window Waround image 1 p=(u,v) •Build vector w Example:](https://reader034.fdocuments.in/reader034/viewer/2022051901/5ff0b91f0364f06605054ddc/html5/thumbnails/42.jpg)
– Smaller window- More detail- More noise
– Larger window- Smoother disparity maps- Less prone to noise
Window size = 3 Window size = 20
Effect of the window’s size
Credit slide S. Lazebnik
![Page 43: lecture6 stereo systems - Stanford Universityweb.stanford.edu/class/cs231a/lectures/lecture6_stereo_systems.pdf · •Pick up a window Waround image 1 p=(u,v) •Build vector w Example:](https://reader034.fdocuments.in/reader034/viewer/2022051901/5ff0b91f0364f06605054ddc/html5/thumbnails/43.jpg)
Issues
•Fore shortening effect
•Occlusions
O O’
O O’
![Page 44: lecture6 stereo systems - Stanford Universityweb.stanford.edu/class/cs231a/lectures/lecture6_stereo_systems.pdf · •Pick up a window Waround image 1 p=(u,v) •Build vector w Example:](https://reader034.fdocuments.in/reader034/viewer/2022051901/5ff0b91f0364f06605054ddc/html5/thumbnails/44.jpg)
O O’
Issues
• However, when B/z is small, small errors in measurements imply large error in estimating depth
• To reduce the effect of foreshortening and occlusions, it is desirable to have small B / z ratio!
O O’
u u’ ue’ u u’ ue’
Large B / z ratio
Small B / z ratio
![Page 45: lecture6 stereo systems - Stanford Universityweb.stanford.edu/class/cs231a/lectures/lecture6_stereo_systems.pdf · •Pick up a window Waround image 1 p=(u,v) •Build vector w Example:](https://reader034.fdocuments.in/reader034/viewer/2022051901/5ff0b91f0364f06605054ddc/html5/thumbnails/45.jpg)
• Homogeneous regions
Issues
mismatch
![Page 46: lecture6 stereo systems - Stanford Universityweb.stanford.edu/class/cs231a/lectures/lecture6_stereo_systems.pdf · •Pick up a window Waround image 1 p=(u,v) •Build vector w Example:](https://reader034.fdocuments.in/reader034/viewer/2022051901/5ff0b91f0364f06605054ddc/html5/thumbnails/46.jpg)
•Repetitive patterns
Issues
![Page 47: lecture6 stereo systems - Stanford Universityweb.stanford.edu/class/cs231a/lectures/lecture6_stereo_systems.pdf · •Pick up a window Waround image 1 p=(u,v) •Build vector w Example:](https://reader034.fdocuments.in/reader034/viewer/2022051901/5ff0b91f0364f06605054ddc/html5/thumbnails/47.jpg)
Correspondence problem is difficult!
- Occlusions- Fore shortening- Baseline trade-off- Homogeneous regions- Repetitive patterns
Apply non-local constraints to help enforce the correspondences
![Page 48: lecture6 stereo systems - Stanford Universityweb.stanford.edu/class/cs231a/lectures/lecture6_stereo_systems.pdf · •Pick up a window Waround image 1 p=(u,v) •Build vector w Example:](https://reader034.fdocuments.in/reader034/viewer/2022051901/5ff0b91f0364f06605054ddc/html5/thumbnails/48.jpg)
• Uniqueness – For any point in one image, there should be at most
one matching point in the other image
• Ordering– Corresponding points should be in the same order in
both views
• Smoothness– Disparity is typically a smooth function of x (except
in occluding boundaries)
Non-local constraints
![Page 49: lecture6 stereo systems - Stanford Universityweb.stanford.edu/class/cs231a/lectures/lecture6_stereo_systems.pdf · •Pick up a window Waround image 1 p=(u,v) •Build vector w Example:](https://reader034.fdocuments.in/reader034/viewer/2022051901/5ff0b91f0364f06605054ddc/html5/thumbnails/49.jpg)
Lecture 5 -Silvio Savarese 2-Feb-18
• Stereosystems• Rectification• Correspondenceproblem
• Multi-viewgeometry• TheSFMproblem• AffineSFM
Lecture6StereoSystemsMulti-viewgeometry
![Page 50: lecture6 stereo systems - Stanford Universityweb.stanford.edu/class/cs231a/lectures/lecture6_stereo_systems.pdf · •Pick up a window Waround image 1 p=(u,v) •Build vector w Example:](https://reader034.fdocuments.in/reader034/viewer/2022051901/5ff0b91f0364f06605054ddc/html5/thumbnails/50.jpg)
Courtesy of Oxford Visual Geometry Group
Structure from motion problem
![Page 51: lecture6 stereo systems - Stanford Universityweb.stanford.edu/class/cs231a/lectures/lecture6_stereo_systems.pdf · •Pick up a window Waround image 1 p=(u,v) •Build vector w Example:](https://reader034.fdocuments.in/reader034/viewer/2022051901/5ff0b91f0364f06605054ddc/html5/thumbnails/51.jpg)
Structure from motion problem
x1j
x2j
xmj
Xj
M1
M2
Mm
Given m images of n fixed 3D points
•xij = Mi Xj , i = 1, … , m, j = 1, … , n
![Page 52: lecture6 stereo systems - Stanford Universityweb.stanford.edu/class/cs231a/lectures/lecture6_stereo_systems.pdf · •Pick up a window Waround image 1 p=(u,v) •Build vector w Example:](https://reader034.fdocuments.in/reader034/viewer/2022051901/5ff0b91f0364f06605054ddc/html5/thumbnails/52.jpg)
From the mxn observations xij, estimate: •m projection matrices Mi
•n 3D points Xj
x1j
x2j
xmj
Xj
motionstructure
M1
M2
Mm
Structure from motion problem
![Page 53: lecture6 stereo systems - Stanford Universityweb.stanford.edu/class/cs231a/lectures/lecture6_stereo_systems.pdf · •Pick up a window Waround image 1 p=(u,v) •Build vector w Example:](https://reader034.fdocuments.in/reader034/viewer/2022051901/5ff0b91f0364f06605054ddc/html5/thumbnails/53.jpg)
Affine structure from motion(simpler problem)
Image 1
World point Xj
Image i
From the mxn observations xij, estimate: •m projection matrices Mi (affine cameras)•n 3D points Xj
xij
![Page 54: lecture6 stereo systems - Stanford Universityweb.stanford.edu/class/cs231a/lectures/lecture6_stereo_systems.pdf · •Pick up a window Waround image 1 p=(u,v) •Build vector w Example:](https://reader034.fdocuments.in/reader034/viewer/2022051901/5ff0b91f0364f06605054ddc/html5/thumbnails/54.jpg)
=
m1
m2
0 0 0 1
!
"
####
$
%
&&&&
=
m1
m2
m3
!
"
####
$
%
&&&&
X
xE = (m1X,m2X)T
x =M X=
A2 x3 b2 x101x3 1
!
"##
$
%&&
magnification
úúú
û
ù
êêê
ë
é=
3
2
1
mmm
x =M X M = A bv 1
!
"#
$
%&=
m1
m2
m3
!
"
####
$
%
&&&&
X =m1Xm2Xm3X
!
"
####
$
%
&&&&
xE = (m1Xm3X
, m2Xm3X
)T
Perspective
Affine
= A b!"
#$X = A b!
"#$
xyz1
!
"
%%%%
#
$
&&&&
=AXE+b
[Eq. 3]
M =
m1
m2
m3
!
"
####
$
%
&&&&=
m1Xm2X1
!
"
####
$
%
&&&&
XE =xyz
!
"
###
$
%
&&&
![Page 55: lecture6 stereo systems - Stanford Universityweb.stanford.edu/class/cs231a/lectures/lecture6_stereo_systems.pdf · •Pick up a window Waround image 1 p=(u,v) •Build vector w Example:](https://reader034.fdocuments.in/reader034/viewer/2022051901/5ff0b91f0364f06605054ddc/html5/thumbnails/55.jpg)
Xj
xij =AiX j +bi
Affine cameras
For the affine case (in Euclidean space)
xi j
x1j
Image i
Image 1
[Eq. 4]
2x1 2x3 2x13x1
![Page 56: lecture6 stereo systems - Stanford Universityweb.stanford.edu/class/cs231a/lectures/lecture6_stereo_systems.pdf · •Pick up a window Waround image 1 p=(u,v) •Build vector w Example:](https://reader034.fdocuments.in/reader034/viewer/2022051901/5ff0b91f0364f06605054ddc/html5/thumbnails/56.jpg)
The Affine Structure-from-Motion Problem
Given m images of n fixed points Xj we can write
Problem: estimate m matrices Ai, m matrices biand the n positions Xj from the m´n observations xij .
2m ´ n equations in 8m + 3n - 8 unknowns
How many equations and how many unknown?
N. of cameras N. of pointsxij =AiX j +bi for i = 1, …,m and j = 1, … ,n
![Page 57: lecture6 stereo systems - Stanford Universityweb.stanford.edu/class/cs231a/lectures/lecture6_stereo_systems.pdf · •Pick up a window Waround image 1 p=(u,v) •Build vector w Example:](https://reader034.fdocuments.in/reader034/viewer/2022051901/5ff0b91f0364f06605054ddc/html5/thumbnails/57.jpg)
The Affine Structure-from-Motion Problem
Two approaches:
- Algebraic approach (affine epipolar geometry; estimate F; cameras; points)
- Factorization method
![Page 58: lecture6 stereo systems - Stanford Universityweb.stanford.edu/class/cs231a/lectures/lecture6_stereo_systems.pdf · •Pick up a window Waround image 1 p=(u,v) •Build vector w Example:](https://reader034.fdocuments.in/reader034/viewer/2022051901/5ff0b91f0364f06605054ddc/html5/thumbnails/58.jpg)
Next lecture
Multiple view geometry:Affine and Perspective structure from Motion