Computing F and rectification class 14
description
Transcript of Computing F and rectification class 14
![Page 1: Computing F and rectification class 14](https://reader035.fdocuments.in/reader035/viewer/2022062801/5681436b550346895dafeaff/html5/thumbnails/1.jpg)
Computing F and rectification
class 14
Multiple View GeometryComp 290-089Marc Pollefeys
![Page 2: Computing F and rectification class 14](https://reader035.fdocuments.in/reader035/viewer/2022062801/5681436b550346895dafeaff/html5/thumbnails/2.jpg)
Multiple View Geometry course schedule(subject to change)
Jan. 7, 9 Intro & motivation Projective 2D Geometry
Jan. 14, 16
(no class) Projective 2D Geometry
Jan. 21, 23
Projective 3D Geometry (no class)
Jan. 28, 30
Parameter Estimation Parameter Estimation
Feb. 4, 6 Algorithm Evaluation Camera Models
Feb. 11, 13
Camera Calibration Single View Geometry
Feb. 18, 20
Epipolar Geometry 3D reconstruction
Feb. 25, 27
Fund. Matrix Comp. Structure Comp.
Mar. 4, 6 Planes & Homographies Trifocal Tensor
Mar. 18, 20
Three View Reconstruction
Multiple View Geometry
Mar. 25, 27
MultipleView Reconstruction
Bundle adjustment
Apr. 1, 3 Auto-Calibration Papers
Apr. 8, 10
Dynamic SfM Papers
Apr. 15, 17
Cheirality Papers
Apr. 22, 24
Duality Project Demos
![Page 3: Computing F and rectification class 14](https://reader035.fdocuments.in/reader035/viewer/2022062801/5681436b550346895dafeaff/html5/thumbnails/3.jpg)
Two-view geometry
Epipolar geometry
3D reconstruction
F-matrix comp.
Structure comp.
![Page 4: Computing F and rectification class 14](https://reader035.fdocuments.in/reader035/viewer/2022062801/5681436b550346895dafeaff/html5/thumbnails/4.jpg)
Epipolar geometry: direct computation
0Fxx'T
0f1''''''
1'''''' 111111111111
nnnnnnnnnnnn yxyyyxyxyxxx
yxyyyxyxyxxx
0)λIFFdet(Fdet)λFFdet( 1-12221
Basic equation
8-point algorithm (normalize!)
7-point algorithm (impose rank 2)
![Page 5: Computing F and rectification class 14](https://reader035.fdocuments.in/reader035/viewer/2022062801/5681436b550346895dafeaff/html5/thumbnails/5.jpg)
Epipolar geometry: iterative computation
Maximum Likelihood Estimation
i
iiii dd 22 'x̂,x'x̂,x
(= least-squares for Gaussian noise)
0x̂F'x̂ subject to T
22
21
2
2T2
1T
2T
FxFxFx'Fx'
Fxx'
2
221
2
2T2
1T
T
FxFx
1
Fx'Fx'
1Fxx'
i
iiii dFd2T2 x'F,xx,x'
Sampson error (first order approx. to MLE)
Symmetric epipolar error
![Page 6: Computing F and rectification class 14](https://reader035.fdocuments.in/reader035/viewer/2022062801/5681436b550346895dafeaff/html5/thumbnails/6.jpg)
Automatic computation of F
(i) Interest points(ii) Putative correspondences(iii) RANSAC (iv) Non-linear re-estimation of F(v) Guided matching(repeat (iv) and (v) until stable)
![Page 7: Computing F and rectification class 14](https://reader035.fdocuments.in/reader035/viewer/2022062801/5681436b550346895dafeaff/html5/thumbnails/7.jpg)
• Extract feature points to relate images
• Required properties:• Well-defined
(i.e. neigboring points should all be different)
• Stable across views(i.e. same 3D point should be extracted as feature for neighboring viewpoints)
Feature points
![Page 8: Computing F and rectification class 14](https://reader035.fdocuments.in/reader035/viewer/2022062801/5681436b550346895dafeaff/html5/thumbnails/8.jpg)
homogeneous
edge
corner
dxdyyxwyI
xI
W yIxI
,
M
M should have large eigenvalues
(e.g.Harris&Stephens´88; Shi&Tomasi´94)
MTSSD
Find points that differ as much as possible from all neighboring points
Feature = local maxima (subpixel) of F(1, 2)
Feature points
![Page 9: Computing F and rectification class 14](https://reader035.fdocuments.in/reader035/viewer/2022062801/5681436b550346895dafeaff/html5/thumbnails/9.jpg)
Select strongest features (e.g. 1000/image)
Feature points
![Page 10: Computing F and rectification class 14](https://reader035.fdocuments.in/reader035/viewer/2022062801/5681436b550346895dafeaff/html5/thumbnails/10.jpg)
Evaluate NCC for all features withsimilar coordinates
Keep mutual best matchesKeep mutual best matches
Still many wrong matches!Still many wrong matches!
10101010 ,,´´, e.g. hhww yyxxyx
?
Feature matching
![Page 11: Computing F and rectification class 14](https://reader035.fdocuments.in/reader035/viewer/2022062801/5681436b550346895dafeaff/html5/thumbnails/11.jpg)
0.96 -0.40 -0.16 -0.39 0.19
-0.05 0.75 -0.47 0.51 0.72
-0.18 -0.39 0.73 0.15 -0.75
-0.27 0.49 0.16 0.79 0.21
0.08 0.50 -0.45 0.28 0.99
1 5
24
3
1 5
24
3
Gives satisfying results for small image motions
Feature example
![Page 12: Computing F and rectification class 14](https://reader035.fdocuments.in/reader035/viewer/2022062801/5681436b550346895dafeaff/html5/thumbnails/12.jpg)
• Requirement to cope with larger variations between images• Translation, rotation, scaling• Foreshortening• Non-diffuse reflections• Illumination
geometric transformations
photometric changes
Wide-baseline matching…
![Page 13: Computing F and rectification class 14](https://reader035.fdocuments.in/reader035/viewer/2022062801/5681436b550346895dafeaff/html5/thumbnails/13.jpg)
Wide baseline matching for two different region types
(Tuytelaars and Van Gool BMVC 2000)
Wide-baseline matching…
![Page 14: Computing F and rectification class 14](https://reader035.fdocuments.in/reader035/viewer/2022062801/5681436b550346895dafeaff/html5/thumbnails/14.jpg)
Step 1. Extract featuresStep 2. Compute a set of potential matchesStep 3. do
Step 3.1 select minimal sample (i.e. 7 matches)
Step 3.2 compute solution(s) for F
Step 3.3 determine inliers
until (#inliers,#samples)<95%
samples#7)1(1
matches#inliers#
#inliers 90%
80%
70% 60%
50%
#samples
5 13 35 106 382
Step 4. Compute F based on all inliersStep 5. Look for additional matchesStep 6. Refine F based on all correct matches
(generate hypothesis)
(verify hypothesis)
RANSAC
![Page 15: Computing F and rectification class 14](https://reader035.fdocuments.in/reader035/viewer/2022062801/5681436b550346895dafeaff/html5/thumbnails/15.jpg)
restrict search range to neighborhood of epipolar line (1.5 pixels)
relax disparity restriction (along epipolar line)
Finding more matches
![Page 16: Computing F and rectification class 14](https://reader035.fdocuments.in/reader035/viewer/2022062801/5681436b550346895dafeaff/html5/thumbnails/16.jpg)
• Degenerate cases• Planar scene• Pure rotation
• No unique solution• Remaining DOF filled by noise• Use simpler model (e.g. homography)
• Model selection (Torr et al., ICCV´98, Kanatani, Akaike)
• Compare H and F according to expected residual error (compensate for model complexity)
Degenerate cases:
![Page 17: Computing F and rectification class 14](https://reader035.fdocuments.in/reader035/viewer/2022062801/5681436b550346895dafeaff/html5/thumbnails/17.jpg)
Model selection
n = number of measurements (inliers+outliers)r = dimension of datak = motion model parametersd = dimension of structure
structure
motion
MLE
Geometric Robust Information Criterion
![Page 18: Computing F and rectification class 14](https://reader035.fdocuments.in/reader035/viewer/2022062801/5681436b550346895dafeaff/html5/thumbnails/18.jpg)
Model selection
Video tracking
H
F
Dominant planes
![Page 19: Computing F and rectification class 14](https://reader035.fdocuments.in/reader035/viewer/2022062801/5681436b550346895dafeaff/html5/thumbnails/19.jpg)
• Absence of sufficient features (no texture)• Repeated structure ambiguity
(Schaffalitzky and Zisserman, BMVC‘98)
• Robust matcher also finds Robust matcher also finds support for wrong hypothesissupport for wrong hypothesis• solution: detect repetition solution: detect repetition
More problems:
![Page 20: Computing F and rectification class 14](https://reader035.fdocuments.in/reader035/viewer/2022062801/5681436b550346895dafeaff/html5/thumbnails/20.jpg)
geometric relations between two views is fully
described by recovered 3x3 matrix F
two-view geometry
![Page 21: Computing F and rectification class 14](https://reader035.fdocuments.in/reader035/viewer/2022062801/5681436b550346895dafeaff/html5/thumbnails/21.jpg)
Image pair rectification
simplify stereo matching by warping the images
Apply projective transformation so that epipolar linescorrespond to horizontal scanlines
e
e
map epipole e to (1,0,0)
try to minimize image distortion
problem when epipole in (or close to) the image
![Page 22: Computing F and rectification class 14](https://reader035.fdocuments.in/reader035/viewer/2022062801/5681436b550346895dafeaff/html5/thumbnails/22.jpg)
Planar rectification
Bring two views Bring two views to standard stereo setupto standard stereo setup
(moves epipole to )(not possible when in/close to image)
~ image size
(calibrated)(calibrated)
Distortion minimization(uncalibrated)
(standard approach)
![Page 23: Computing F and rectification class 14](https://reader035.fdocuments.in/reader035/viewer/2022062801/5681436b550346895dafeaff/html5/thumbnails/23.jpg)
![Page 24: Computing F and rectification class 14](https://reader035.fdocuments.in/reader035/viewer/2022062801/5681436b550346895dafeaff/html5/thumbnails/24.jpg)
![Page 25: Computing F and rectification class 14](https://reader035.fdocuments.in/reader035/viewer/2022062801/5681436b550346895dafeaff/html5/thumbnails/25.jpg)
Polar re-parameterization around epipolesRequires only (oriented) epipolar geometryPreserve length of epipolar linesChoose so that no pixels are compressed
original image rectified image
Polar rectification(Pollefeys et al. ICCV’99)
Works for all relative motionsGuarantees minimal image size
![Page 26: Computing F and rectification class 14](https://reader035.fdocuments.in/reader035/viewer/2022062801/5681436b550346895dafeaff/html5/thumbnails/26.jpg)
polar rectification: example
![Page 27: Computing F and rectification class 14](https://reader035.fdocuments.in/reader035/viewer/2022062801/5681436b550346895dafeaff/html5/thumbnails/27.jpg)
polar rectification: example
![Page 28: Computing F and rectification class 14](https://reader035.fdocuments.in/reader035/viewer/2022062801/5681436b550346895dafeaff/html5/thumbnails/28.jpg)
Example: Béguinage of Leuven
Does not work with standard Homography-based approaches
![Page 29: Computing F and rectification class 14](https://reader035.fdocuments.in/reader035/viewer/2022062801/5681436b550346895dafeaff/html5/thumbnails/29.jpg)
Example: Béguinage of Leuven
![Page 30: Computing F and rectification class 14](https://reader035.fdocuments.in/reader035/viewer/2022062801/5681436b550346895dafeaff/html5/thumbnails/30.jpg)
Exploiting motion and scene constraints
• Ordering constraint• Uniqueness constraint• Disparity limit• Disparity continuity constraint
• Epipolar constraint Epipolar constraint (through rectification)
![Page 31: Computing F and rectification class 14](https://reader035.fdocuments.in/reader035/viewer/2022062801/5681436b550346895dafeaff/html5/thumbnails/31.jpg)
Ordering constraint
11 22 33 4,54,5 66 11 2,32,3 44 55 66
2211 33 4,54,5 6611
2,32,3
44
55
66
surface slicesurface slice surface as a pathsurface as a path
occlusion right
occlusion left
![Page 32: Computing F and rectification class 14](https://reader035.fdocuments.in/reader035/viewer/2022062801/5681436b550346895dafeaff/html5/thumbnails/32.jpg)
Uniqueness constraint
• In an image pair each pixel has at most one corresponding pixel• In general one corresponding pixel• In case of occlusion there is none
![Page 33: Computing F and rectification class 14](https://reader035.fdocuments.in/reader035/viewer/2022062801/5681436b550346895dafeaff/html5/thumbnails/33.jpg)
Disparity constraint
surface slicesurface slice surface as a pathsurface as a path
bounding box
dispa
rity b
and
use reconsructed features to determine bounding box
constantdisparitysurfaces
![Page 34: Computing F and rectification class 14](https://reader035.fdocuments.in/reader035/viewer/2022062801/5681436b550346895dafeaff/html5/thumbnails/34.jpg)
Disparity continuity constraint
• Assume piecewise continuous surface
piecewise continuous disparity• In general disparity changes
continuously• discontinuities at occluding
boundaries
![Page 35: Computing F and rectification class 14](https://reader035.fdocuments.in/reader035/viewer/2022062801/5681436b550346895dafeaff/html5/thumbnails/35.jpg)
Stereo matching
Optimal path(dynamic programming )
Similarity measure(SSD or NCC)
Constraints• epipolar
• ordering
• uniqueness
• disparity limit
• disparity gradient limit
Trade-off
• Matching cost (data)
• Discontinuities (prior)
(Cox et al. CVGIP’96; Koch’96; Falkenhagen´97; Van Meerbergen,Vergauwen,Pollefeys,VanGool IJCV‘02)
![Page 36: Computing F and rectification class 14](https://reader035.fdocuments.in/reader035/viewer/2022062801/5681436b550346895dafeaff/html5/thumbnails/36.jpg)
Hierarchical stereo matching
Dow
nsam
plin
g
(Gau
ssia
n p
yra
mid
)
Dis
pari
ty p
rop
ag
ati
on
Allows faster computation
Deals with large disparity ranges
(Falkenhagen´97;Van Meerbergen,Vergauwen,Pollefeys,VanGool IJCV‘02)
![Page 37: Computing F and rectification class 14](https://reader035.fdocuments.in/reader035/viewer/2022062801/5681436b550346895dafeaff/html5/thumbnails/37.jpg)
Disparity map
image I(x,y) image I´(x´,y´)Disparity map D(x,y)
(x´,y´)=(x+D(x,y),y)
![Page 38: Computing F and rectification class 14](https://reader035.fdocuments.in/reader035/viewer/2022062801/5681436b550346895dafeaff/html5/thumbnails/38.jpg)
Example: reconstruct image from neighboring
images
![Page 39: Computing F and rectification class 14](https://reader035.fdocuments.in/reader035/viewer/2022062801/5681436b550346895dafeaff/html5/thumbnails/39.jpg)
Multi-view depth fusion
• Compute depth for every pixel of reference image• Triangulation• Use multiple views• Up- and down
sequence• Use Kalman filter
(Koch, Pollefeys and Van Gool. ECCV‘98)
Allows to compute robust texture
![Page 40: Computing F and rectification class 14](https://reader035.fdocuments.in/reader035/viewer/2022062801/5681436b550346895dafeaff/html5/thumbnails/40.jpg)
Assignment 2Compute F automatically from image
pair
(matches, 7-point, RANSAC, 8-point, iterative,more matches, epipolar lines, etc.)
(due by Wednesday 19/03/03)
![Page 41: Computing F and rectification class 14](https://reader035.fdocuments.in/reader035/viewer/2022062801/5681436b550346895dafeaff/html5/thumbnails/41.jpg)
Next class: reconstructing points and lines