Projective 3D geometry class 4 Multiple View Geometry Comp 290-089 Marc Pollefeys.
Multiple View Geometry in Computer Vision
description
Transcript of Multiple View Geometry in Computer Vision
![Page 1: Multiple View Geometry in Computer Vision](https://reader033.fdocuments.in/reader033/viewer/2022061500/56814f65550346895dbd1a96/html5/thumbnails/1.jpg)
Multiple View Geometryin Computer Vision
Marc Pollefeys
Comp 290-089
![Page 2: Multiple View Geometry in Computer Vision](https://reader033.fdocuments.in/reader033/viewer/2022061500/56814f65550346895dbd1a96/html5/thumbnails/2.jpg)
Multiple View Geometry
a
bc
A
(a,b) A
(a,b) c
f(a,b,c)=0
a
b
c
(a,b,c) (a,b,c)(reconstruction)
(calibration)
(transfer)
![Page 3: Multiple View Geometry in Computer Vision](https://reader033.fdocuments.in/reader033/viewer/2022061500/56814f65550346895dbd1a96/html5/thumbnails/3.jpg)
Course objectives
• To understand the geometric relations between multiple views of scenes.
• To understand the general principles of parameter estimation.
• To be able to compute scene and camera properties from real world images using state-of-the-art algorithms.
![Page 4: Multiple View Geometry in Computer Vision](https://reader033.fdocuments.in/reader033/viewer/2022061500/56814f65550346895dbd1a96/html5/thumbnails/4.jpg)
Relation to other vision/image courses
• Focuses on geometric aspects• No image processing
• Comp 254: Image Processing an AnalysisMostly orthogonal to this course,
complementary
• Comp 256: Computer Vision (fall 2003)Will be much broader, based on new book:“Computer Vision: a modern approach”David Forsyth and Jean Ponce
![Page 5: Multiple View Geometry in Computer Vision](https://reader033.fdocuments.in/reader033/viewer/2022061500/56814f65550346895dbd1a96/html5/thumbnails/5.jpg)
Material
Textbook:Multiple View Geometry in Computer Visionby Richard Hartley and Andrew ZissermanCambridge University Press
Alternative book:The Geometry from Multiple Imagesby Olivier Faugeras and Quan-Tuan LuongMIT Press
On-line tutorial:http://www.cs.unc.edu/~marc/tutorial.pdf
http://www.cs.unc.edu/~marc/tutorial/
![Page 6: Multiple View Geometry in Computer Vision](https://reader033.fdocuments.in/reader033/viewer/2022061500/56814f65550346895dbd1a96/html5/thumbnails/6.jpg)
Learning approach
• read the relevant chapters of the books and/or reading assignements before the course.
• In the course the material will then be covered in detail and motivated with real world examples and applications.
• Small hands-on assignements will be provided to give students a "feel" of the practical aspects.
• Students will also read and present some seminal papers to provide a complementary view on some of the covered topics.
• Finally, there will also be a project where students will implement an algorithm or approach using concepts covered by the course.
Grade distribution
• Class participation: 20% • Hands-on assignments: 10% • Paper presentation: 10% • Implementation assignment/project: 40% • Final: 20%
![Page 7: Multiple View Geometry in Computer Vision](https://reader033.fdocuments.in/reader033/viewer/2022061500/56814f65550346895dbd1a96/html5/thumbnails/7.jpg)
Applications
• MatchMovingCompute camera motion from video (to register real an virtual object
motion)
![Page 8: Multiple View Geometry in Computer Vision](https://reader033.fdocuments.in/reader033/viewer/2022061500/56814f65550346895dbd1a96/html5/thumbnails/8.jpg)
Applications
• 3D modeling
![Page 9: Multiple View Geometry in Computer Vision](https://reader033.fdocuments.in/reader033/viewer/2022061500/56814f65550346895dbd1a96/html5/thumbnails/9.jpg)
Content
• Background: Projective geometry (2D, 3D), Parameter estimation, Algorithm evaluation.
• Single View: Camera model, Calibration, Single View Geometry.
• Two Views: Epipolar Geometry, 3D reconstruction, Computing F, Computing structure, Plane and homographies.
• Three Views: Trifocal Tensor, Computing T.• More Views: N-Linearities, Multiple view
reconstruction, Bundle adjustment, auto-calibration, Dynamic SfM, Cheirality, Duality
![Page 10: Multiple View Geometry in Computer Vision](https://reader033.fdocuments.in/reader033/viewer/2022061500/56814f65550346895dbd1a96/html5/thumbnails/10.jpg)
Multiple View Geometry course schedule(tentative)
Jan. 7, 9 Intro & motivation Projective 2D Geometry
Jan. 14, 16
(no course) Projective 2D Geometry
Jan. 21, 23
Projective 3D Geometry Parameter Estimation
Jan. 28, 30
Parameter Estimation Algorithm Evaluation
Feb. 4, 6 Camera Models Camera Calibration
Feb. 11, 13
Single View Geometry Epipolar Geometry
Feb. 18, 20
3D reconstruction Fund. Matrix Comp.
Feb. 25, 27
Structure Comp. Planes & Homographies
Mar. 4, 6 Trifocal Tensor Three View Reconstruction
Mar. 18, 20
Multiple View Geometry MultipleView Reconstruction
Mar. 25, 27
Bundle adjustment Papers
Apr. 1, 3 Auto-Calibration Papers
Apr. 8, 10
Dynamic SfM Papers
Apr. 15, 17
Cheirality Papers
Apr. 22, 24
Duality Project Demos
![Page 11: Multiple View Geometry in Computer Vision](https://reader033.fdocuments.in/reader033/viewer/2022061500/56814f65550346895dbd1a96/html5/thumbnails/11.jpg)
Fast Forward!
• Quick overview of what is coming…
![Page 12: Multiple View Geometry in Computer Vision](https://reader033.fdocuments.in/reader033/viewer/2022061500/56814f65550346895dbd1a96/html5/thumbnails/12.jpg)
Background
La reproduction interdite (Reproduction Prohibited), 1937, René Magritte.
![Page 13: Multiple View Geometry in Computer Vision](https://reader033.fdocuments.in/reader033/viewer/2022061500/56814f65550346895dbd1a96/html5/thumbnails/13.jpg)
• Points, lines & conics• Transformations
• Cross-ratio and invariants
Projective 2D Geometry
![Page 14: Multiple View Geometry in Computer Vision](https://reader033.fdocuments.in/reader033/viewer/2022061500/56814f65550346895dbd1a96/html5/thumbnails/14.jpg)
Projective 3D Geometry
• Points, lines, planes and quadrics
• Transformations
• П∞, ω∞ and Ω ∞
![Page 15: Multiple View Geometry in Computer Vision](https://reader033.fdocuments.in/reader033/viewer/2022061500/56814f65550346895dbd1a96/html5/thumbnails/15.jpg)
Estimation
How to compute a geometric relation from correspondences, e.g. 2D trafo
• Linear (normalized), non-linear and Maximum Likelihood Estimation
• Robust (RANSAC)
![Page 16: Multiple View Geometry in Computer Vision](https://reader033.fdocuments.in/reader033/viewer/2022061500/56814f65550346895dbd1a96/html5/thumbnails/16.jpg)
Evaluation and error analysis
How good are the results we get• Bounds on performance
• Covariance propagation & Monte-Carlo estimation
residual
error
JΣJΣ XP1T
![Page 17: Multiple View Geometry in Computer Vision](https://reader033.fdocuments.in/reader033/viewer/2022061500/56814f65550346895dbd1a96/html5/thumbnails/17.jpg)
Single-View Geometry
The Cyclops, c. 1914, Odilon Redon
![Page 18: Multiple View Geometry in Computer Vision](https://reader033.fdocuments.in/reader033/viewer/2022061500/56814f65550346895dbd1a96/html5/thumbnails/18.jpg)
Camera Models
X.xλor
1
10
t
1
1
1
11
λ3
PR
Z
Y
X
pf
pf
y
x
y
x
T
Mostly pinhole camera model
but also affine cameras, pushbroom camera, …
![Page 19: Multiple View Geometry in Computer Vision](https://reader033.fdocuments.in/reader033/viewer/2022061500/56814f65550346895dbd1a96/html5/thumbnails/19.jpg)
Camera Calibration
• Compute P given (m,M)(normalized) linear, MLE,…
• Radial distortion
![Page 20: Multiple View Geometry in Computer Vision](https://reader033.fdocuments.in/reader033/viewer/2022061500/56814f65550346895dbd1a96/html5/thumbnails/20.jpg)
More Single-View Geometry
• Projective cameras and planes, lines, conics and quadrics.
• Camera center and camera rotation
• Camera calibration and vanishing points, calibrating conic and the IAC
** CPPQ T
coneQCPP T
![Page 21: Multiple View Geometry in Computer Vision](https://reader033.fdocuments.in/reader033/viewer/2022061500/56814f65550346895dbd1a96/html5/thumbnails/21.jpg)
Single View Metrology
Antonio Criminisi
![Page 22: Multiple View Geometry in Computer Vision](https://reader033.fdocuments.in/reader033/viewer/2022061500/56814f65550346895dbd1a96/html5/thumbnails/22.jpg)
Two-View Geometry
The Birth of Venus (detail), c. 1485, Sandro Botticelli
![Page 23: Multiple View Geometry in Computer Vision](https://reader033.fdocuments.in/reader033/viewer/2022061500/56814f65550346895dbd1a96/html5/thumbnails/23.jpg)
Epipolar Geometry
Fundamental matrix Essential matrix 0xx' FT 0x̂[t]'x̂ RT
'PP,F 'PP,E
![Page 24: Multiple View Geometry in Computer Vision](https://reader033.fdocuments.in/reader033/viewer/2022061500/56814f65550346895dbd1a96/html5/thumbnails/24.jpg)
Two-View Reconstruction
![Page 25: Multiple View Geometry in Computer Vision](https://reader033.fdocuments.in/reader033/viewer/2022061500/56814f65550346895dbd1a96/html5/thumbnails/25.jpg)
Epipolar Geometry Computation
(normalized) linear:
minimal:
MLE:
RANSAC… and automated two view matching
0xx' FT
0λdet 21 FF
0).1,,,',',',',','( fyxyyyxyxyxxx
0x̂'x̂for which
'x̂,x'x̂,x 2
i
2
ii
iiii dd
FT
![Page 26: Multiple View Geometry in Computer Vision](https://reader033.fdocuments.in/reader033/viewer/2022061500/56814f65550346895dbd1a96/html5/thumbnails/26.jpg)
RectificationWarp images to simplify epipolar geometry
![Page 27: Multiple View Geometry in Computer Vision](https://reader033.fdocuments.in/reader033/viewer/2022061500/56814f65550346895dbd1a96/html5/thumbnails/27.jpg)
Structure Computation
• Points: Linear, optimal, direct optimal
• Also lines and vanishing points
![Page 28: Multiple View Geometry in Computer Vision](https://reader033.fdocuments.in/reader033/viewer/2022061500/56814f65550346895dbd1a96/html5/thumbnails/28.jpg)
Planes and Homographies
TT )1,v(π Relation between plane and H given P and P’Relation between H and F, H from F, F from H
The infinity homography H∞
![Page 29: Multiple View Geometry in Computer Vision](https://reader033.fdocuments.in/reader033/viewer/2022061500/56814f65550346895dbd1a96/html5/thumbnails/29.jpg)
Three-View Geometry
The Birth of Venus (detail), c. 1485, Sandro Botticelli
![Page 30: Multiple View Geometry in Computer Vision](https://reader033.fdocuments.in/reader033/viewer/2022061500/56814f65550346895dbd1a96/html5/thumbnails/30.jpg)
Trifocal Tensor
![Page 31: Multiple View Geometry in Computer Vision](https://reader033.fdocuments.in/reader033/viewer/2022061500/56814f65550346895dbd1a96/html5/thumbnails/31.jpg)
Three View Reconstruction
• (normalized) linear• minimal (6 points)• MLE (Gold Standard)
![Page 32: Multiple View Geometry in Computer Vision](https://reader033.fdocuments.in/reader033/viewer/2022061500/56814f65550346895dbd1a96/html5/thumbnails/32.jpg)
Multiple-View Geometry
The Birth of Venus (detail), c. 1485, Sandro Botticelli
![Page 33: Multiple View Geometry in Computer Vision](https://reader033.fdocuments.in/reader033/viewer/2022061500/56814f65550346895dbd1a96/html5/thumbnails/33.jpg)
Multiple View Geometry
wxyzpqrs
lszkryjqxipwiiii Qxxxx 0
Quadrifocal tensor
wxyzpqrs
srqp Qllll 081 parameters, but only 29 DOF!
![Page 34: Multiple View Geometry in Computer Vision](https://reader033.fdocuments.in/reader033/viewer/2022061500/56814f65550346895dbd1a96/html5/thumbnails/34.jpg)
Multiple View Reconstruction
• Affine factorization• Projective factorization
n
mn
mm
n
n
XXX
P
P
P
xxx
xxx
xxx
21
m
2
1
21
222
21
112
11
![Page 35: Multiple View Geometry in Computer Vision](https://reader033.fdocuments.in/reader033/viewer/2022061500/56814f65550346895dbd1a96/html5/thumbnails/35.jpg)
Multiple View Reconstruction
• Sequential reconstruction
![Page 36: Multiple View Geometry in Computer Vision](https://reader033.fdocuments.in/reader033/viewer/2022061500/56814f65550346895dbd1a96/html5/thumbnails/36.jpg)
Bundle AdjustmentMaximum Likelyhood Estimation for complete structure and motion
U1
U2
U3
WT
W
V
P1 P2 P3 M
J JJN T
12xm 3xn(in general
much larger)
m
k
n
iikd
1 1
2
ki X̂P̂,x
![Page 37: Multiple View Geometry in Computer Vision](https://reader033.fdocuments.in/reader033/viewer/2022061500/56814f65550346895dbd1a96/html5/thumbnails/37.jpg)
Bundle AdjustmentMaximum Likelyhood Estimation for complete structure and motion
m
k
n
iikd
1 1
2
ki X̂P̂,x
WT V
U-WV-1WT
NI0
WVI 1
11xm 3xn
![Page 38: Multiple View Geometry in Computer Vision](https://reader033.fdocuments.in/reader033/viewer/2022061500/56814f65550346895dbd1a96/html5/thumbnails/38.jpg)
Bundle adjustment
No bundle adjustment
Bundle adjustment needed to avoid drift of virtual objectthroughout sequence
Bundle adjustment (including radial distortion)
![Page 39: Multiple View Geometry in Computer Vision](https://reader033.fdocuments.in/reader033/viewer/2022061500/56814f65550346895dbd1a96/html5/thumbnails/39.jpg)
*
*
projection
constraints
Tii
Tiii Ωω KKPP
Auto-calibration
Tijiijj
HH ωω
![Page 40: Multiple View Geometry in Computer Vision](https://reader033.fdocuments.in/reader033/viewer/2022061500/56814f65550346895dbd1a96/html5/thumbnails/40.jpg)
Dynamic Structure from Motion
3
2
1
321
223
222
221
113
112
111
21
222
21
112
11
S
S
S
PPP
PPP
PPP
xxx
xxx
xxx
mmmmmmmn
mm
n
n
lll
lll
lll
![Page 41: Multiple View Geometry in Computer Vision](https://reader033.fdocuments.in/reader033/viewer/2022061500/56814f65550346895dbd1a96/html5/thumbnails/41.jpg)
Cheirality
iiiiii
Xof hullconvex preserves
0λ allor 0λ allwith 'XλX
T
T
Oriented projective geometry
Allows to use fact that points are in front of camera• to recover quasi-affine reconstruction• to determine order for image warping• to determine orientation for rectification
with epipoles in images• etc.
![Page 42: Multiple View Geometry in Computer Vision](https://reader033.fdocuments.in/reader033/viewer/2022061500/56814f65550346895dbd1a96/html5/thumbnails/42.jpg)
Duality
Gives possibility to interchange role of P and X in algorithms
dc
db
da
P
d
c
b
a
WZ
WY
WX
W
Z
Y
X
dc
db
da
XP
TTTT
TTTT
)1,1,1(e,)1,0,0(e,)0,1,0(e,)0,0,1(e
)1,0,0,0(E,)0,1,0,0(E,)0,0,1,0(E,)0,0,0,1(E
4321
4321