Relations between image coordinates Given coordinates in one image, and the tranformation Between...
-
date post
19-Dec-2015 -
Category
Documents
-
view
218 -
download
0
Transcript of Relations between image coordinates Given coordinates in one image, and the tranformation Between...
![Page 1: Relations between image coordinates Given coordinates in one image, and the tranformation Between cameras, T = [R t], what are the image coordinates In.](https://reader030.fdocuments.in/reader030/viewer/2022032703/56649d2a5503460f949fe758/html5/thumbnails/1.jpg)
Relations between image coordinates
Given coordinates in one image, and the tranformationBetween cameras, T = [R t], what are the image coordinatesIn the other camera’s image.
![Page 2: Relations between image coordinates Given coordinates in one image, and the tranformation Between cameras, T = [R t], what are the image coordinates In.](https://reader030.fdocuments.in/reader030/viewer/2022032703/56649d2a5503460f949fe758/html5/thumbnails/2.jpg)
Definitions
![Page 3: Relations between image coordinates Given coordinates in one image, and the tranformation Between cameras, T = [R t], what are the image coordinates In.](https://reader030.fdocuments.in/reader030/viewer/2022032703/56649d2a5503460f949fe758/html5/thumbnails/3.jpg)
Essential Matrix: Relating between image coordinates
camera coordinate systems, related by a rotation R and a translation T:
€
O x→
O' x→
' OO'→
Are Coplanar, so:
€
x'=R t
000 1
⎡
⎣ ⎢
⎤
⎦ ⎥x€
O' x→
'• O x→
× OO'→ ⎛
⎝ ⎜
⎞ ⎠ ⎟= 0
![Page 4: Relations between image coordinates Given coordinates in one image, and the tranformation Between cameras, T = [R t], what are the image coordinates In.](https://reader030.fdocuments.in/reader030/viewer/2022032703/56649d2a5503460f949fe758/html5/thumbnails/4.jpg)
€
x'=R t
000 1
⎡
⎣ ⎢
⎤
⎦ ⎥x
€
x'•r t × Rx( ) = 0
€
O' x→
'• O x→
× OO'→ ⎛
⎝ ⎜
⎞ ⎠ ⎟= 0
€
x'• E x( ) = 0
x'E x = 0
E =
0 −tz ty
tz 0 −tx
−ty tx 0
⎡
⎣
⎢ ⎢ ⎢
⎤
⎦
⎥ ⎥ ⎥R
![Page 5: Relations between image coordinates Given coordinates in one image, and the tranformation Between cameras, T = [R t], what are the image coordinates In.](https://reader030.fdocuments.in/reader030/viewer/2022032703/56649d2a5503460f949fe758/html5/thumbnails/5.jpg)
What does the Essential matrix do?
It represents the normal to the epipolar line in the other image
€
n = E x
xon epipolar line' ⇒ n ⋅ x ' = 0
n1x1 + n2x2 + n31 = 0
y = mx + b( )⇒ b = −n3, m = −n1
n2
The normal defines a line in image 2:
![Page 6: Relations between image coordinates Given coordinates in one image, and the tranformation Between cameras, T = [R t], what are the image coordinates In.](https://reader030.fdocuments.in/reader030/viewer/2022032703/56649d2a5503460f949fe758/html5/thumbnails/6.jpg)
What if cameras are uncalibrated? Fundamental Matrix
Choose world coordinates as Camera 1. Then the extrinsic parameters for camera 2 are just R and tHowever, intrinsic parameters for both cameras are unknown.Let C1 and C2 denote the matrices of intrinsic parameters. Then the pixel coordinates measured are not appropriate for the Essential matrix. Correcting for this distortion creates a new matrix: the Fundamental Matrix.
C=
€
x'measured = C2x ' xmeasured = C1x
x '( )tE x = 0⇒ C2
−1x 'measured( )tE C1
−1xmeasured( ) = 0
x 'measured( )tF xmeasured = 0
F = C2−tEC1
−1
![Page 7: Relations between image coordinates Given coordinates in one image, and the tranformation Between cameras, T = [R t], what are the image coordinates In.](https://reader030.fdocuments.in/reader030/viewer/2022032703/56649d2a5503460f949fe758/html5/thumbnails/7.jpg)
Computing the fundamental Matrix
Computing : I Number of Correspondences Given perfect image points (no noise) in general position. Each point correspondence generates one constraint on the fundamental matrix
Constraint for one point
Each constraint can be rewritten as a dot product.Stacking several of these results in:
![Page 8: Relations between image coordinates Given coordinates in one image, and the tranformation Between cameras, T = [R t], what are the image coordinates In.](https://reader030.fdocuments.in/reader030/viewer/2022032703/56649d2a5503460f949fe758/html5/thumbnails/8.jpg)
Stereo Reconstruction
![Page 9: Relations between image coordinates Given coordinates in one image, and the tranformation Between cameras, T = [R t], what are the image coordinates In.](https://reader030.fdocuments.in/reader030/viewer/2022032703/56649d2a5503460f949fe758/html5/thumbnails/9.jpg)
> 8 Point matches
![Page 10: Relations between image coordinates Given coordinates in one image, and the tranformation Between cameras, T = [R t], what are the image coordinates In.](https://reader030.fdocuments.in/reader030/viewer/2022032703/56649d2a5503460f949fe758/html5/thumbnails/10.jpg)
C
![Page 11: Relations between image coordinates Given coordinates in one image, and the tranformation Between cameras, T = [R t], what are the image coordinates In.](https://reader030.fdocuments.in/reader030/viewer/2022032703/56649d2a5503460f949fe758/html5/thumbnails/11.jpg)
Reconstruction Steps
![Page 12: Relations between image coordinates Given coordinates in one image, and the tranformation Between cameras, T = [R t], what are the image coordinates In.](https://reader030.fdocuments.in/reader030/viewer/2022032703/56649d2a5503460f949fe758/html5/thumbnails/12.jpg)
Determining Extrinsic Camera Parameters
€
M1 = C1 Ir 0 [ ]
M2 = C2 Rr t [ ]
Why can we just use this as the external parameters for camera M1? Because we are only interested in the relative position of the two cameras.
• First we undo the Intrinsic camera distortions by defining newnormalized cameras
€
M1
norm = C1−1M1 and M2
norm = C2−1M2
![Page 13: Relations between image coordinates Given coordinates in one image, and the tranformation Between cameras, T = [R t], what are the image coordinates In.](https://reader030.fdocuments.in/reader030/viewer/2022032703/56649d2a5503460f949fe758/html5/thumbnails/13.jpg)
Determining Extrinsic Camera Parameters• The normalized cameras contain unknown parameters
€
M1
norm = C1−1M1 ⇒ M
1
norm = Ir 0 [ ]
M2norm = C2
−1M2 ⇒ M2norm = R
r t [ ]
• However, those parameters can be extracted from the Fundamental matrix
€
F =C2−tEC1
−1
E = C2t FC1
€
E =0 −tz ty
tz 0 −tx
−ty tx 0
⎡
⎣
⎢ ⎢ ⎢
⎤
⎦
⎥ ⎥ ⎥R =
r t ×R
![Page 14: Relations between image coordinates Given coordinates in one image, and the tranformation Between cameras, T = [R t], what are the image coordinates In.](https://reader030.fdocuments.in/reader030/viewer/2022032703/56649d2a5503460f949fe758/html5/thumbnails/14.jpg)
Extract t and R from the Essential Matrix
How do we recover t and R? Answer: SVD of E
€
E =USV t
S diagonal
U,V orthogonal and det() =1 (rotation)
€
R = UWV t or R = UW tV t
W =
0 −1 0
1 0 0
0 0 1
⎡
⎣
⎢ ⎢ ⎢
⎤
⎦
⎥ ⎥ ⎥
€
rt = u3 or
r t = −u3
![Page 15: Relations between image coordinates Given coordinates in one image, and the tranformation Between cameras, T = [R t], what are the image coordinates In.](https://reader030.fdocuments.in/reader030/viewer/2022032703/56649d2a5503460f949fe758/html5/thumbnails/15.jpg)
Reconstruction AmbiguitySo we have 4 possible combinations of translations and rotations giving 4 possibilities for M2
norm = [R | t]
1. M2norm = [UWtVt | t]
2. M2norm = [UWVt | t]
3. M2norm = [UWtVt | -t]
4. M2norm = [UWVt | -t]
![Page 16: Relations between image coordinates Given coordinates in one image, and the tranformation Between cameras, T = [R t], what are the image coordinates In.](https://reader030.fdocuments.in/reader030/viewer/2022032703/56649d2a5503460f949fe758/html5/thumbnails/16.jpg)
Which one is right?
![Page 17: Relations between image coordinates Given coordinates in one image, and the tranformation Between cameras, T = [R t], what are the image coordinates In.](https://reader030.fdocuments.in/reader030/viewer/2022032703/56649d2a5503460f949fe758/html5/thumbnails/17.jpg)
Both Cameras must be facing the same direction
![Page 18: Relations between image coordinates Given coordinates in one image, and the tranformation Between cameras, T = [R t], what are the image coordinates In.](https://reader030.fdocuments.in/reader030/viewer/2022032703/56649d2a5503460f949fe758/html5/thumbnails/18.jpg)
Which one is right?
![Page 19: Relations between image coordinates Given coordinates in one image, and the tranformation Between cameras, T = [R t], what are the image coordinates In.](https://reader030.fdocuments.in/reader030/viewer/2022032703/56649d2a5503460f949fe758/html5/thumbnails/19.jpg)
How do we backproject?
€
x'measured = C2x ' xmeasured = C1x
Knowing Ci allows us to determine the undistorted image points :x'= C2
−1x'measured x = C1−1xmeasured
Recalling the projection equations allows to relate the world point and the image points.x'= C2
−1x'measured x = C1−1xmeasured
z' x '= C2−1C2 M2
norm X zx = C1−1C1 I | 0[ ]X
z' x '= M2norm X zx = I | 0[ ]X
![Page 20: Relations between image coordinates Given coordinates in one image, and the tranformation Between cameras, T = [R t], what are the image coordinates In.](https://reader030.fdocuments.in/reader030/viewer/2022032703/56649d2a5503460f949fe758/html5/thumbnails/20.jpg)
Backprojection to 3D
We now know x, x’, R, and tNeed X
![Page 21: Relations between image coordinates Given coordinates in one image, and the tranformation Between cameras, T = [R t], what are the image coordinates In.](https://reader030.fdocuments.in/reader030/viewer/2022032703/56649d2a5503460f949fe758/html5/thumbnails/21.jpg)
Solving…
€
zx i = M norm X i
x i
y i
1
⎡
⎣
⎢ ⎢ ⎢
⎤
⎦
⎥ ⎥ ⎥=
m1t
m2t
m3t
⎡
⎣
⎢ ⎢ ⎢
⎤
⎦
⎥ ⎥ ⎥X i ⇒
z ui = m1t ⋅X i
zv i = m2t ⋅X i
z = m3t ⋅X i
⎧
⎨ ⎪
⎩ ⎪
⇒(m3
t ⋅X i) ui = m1t ⋅X i
(m3t ⋅X i)v i = m2
t ⋅X i
⎧ ⎨ ⎩
⇒(m3
t ⋅X i) ui − m1t ⋅X i = 0
(m3t ⋅X i)v i − m2
t ⋅X i = 0
⎧ ⎨ ⎩
⇒ui(m3
t )− m1t
v i(m3t )− m2
t
⎡
⎣ ⎢
⎤
⎦ ⎥X i = 0
![Page 22: Relations between image coordinates Given coordinates in one image, and the tranformation Between cameras, T = [R t], what are the image coordinates In.](https://reader030.fdocuments.in/reader030/viewer/2022032703/56649d2a5503460f949fe758/html5/thumbnails/22.jpg)
Solving…
€
u'i(2m3
t )−2m1t
v 'i(2m3
t )−2m2t
⎡
⎣ ⎢
⎤
⎦ ⎥X i = 0
Combining 1 & 2 :ui(m3
t )− m1t
v i(m3t )− m2
t
u'i(2m3
t )−2m1t
v 'i(2m3
t )−2m2t
⎡
⎣
⎢ ⎢ ⎢ ⎢
⎤
⎦
⎥ ⎥ ⎥ ⎥
X i = 0
AX i = 0It has a solvable form! Solve using minimum eigenvalue-Eigenvector approach (e.g. Xi = Null(A))
Where 2mti denotes the ith
row of the second camera’s normalized projection matrix.
![Page 23: Relations between image coordinates Given coordinates in one image, and the tranformation Between cameras, T = [R t], what are the image coordinates In.](https://reader030.fdocuments.in/reader030/viewer/2022032703/56649d2a5503460f949fe758/html5/thumbnails/23.jpg)
Finishing up
![Page 24: Relations between image coordinates Given coordinates in one image, and the tranformation Between cameras, T = [R t], what are the image coordinates In.](https://reader030.fdocuments.in/reader030/viewer/2022032703/56649d2a5503460f949fe758/html5/thumbnails/24.jpg)
What else can you do with these methods? Synthesize new views
60 deg!Image 1 Image 2
Avidan & Shashua, 1997
![Page 25: Relations between image coordinates Given coordinates in one image, and the tranformation Between cameras, T = [R t], what are the image coordinates In.](https://reader030.fdocuments.in/reader030/viewer/2022032703/56649d2a5503460f949fe758/html5/thumbnails/25.jpg)
Faugeras and Robert, 1996
![Page 26: Relations between image coordinates Given coordinates in one image, and the tranformation Between cameras, T = [R t], what are the image coordinates In.](https://reader030.fdocuments.in/reader030/viewer/2022032703/56649d2a5503460f949fe758/html5/thumbnails/26.jpg)
Undo Perspective Distortion (for a plane)Original images (left and right)
Transfer and superimposed images
The ``transfer'' image is the left image projectively warped so that points on the plane containing the Chinese text are mapped to their position in the right image.
•The ``superimpose'' image is a superposition of the transfer and right image. The planes exactly coincide. However, points off the plane (such as the mug) do not coincide.
*This is an example of planar projectively induced parallax. Lines joining corresponding points off the plane in the ``superimposed'' image intersect at the epipole.
![Page 27: Relations between image coordinates Given coordinates in one image, and the tranformation Between cameras, T = [R t], what are the image coordinates In.](https://reader030.fdocuments.in/reader030/viewer/2022032703/56649d2a5503460f949fe758/html5/thumbnails/27.jpg)
Its all about point matches
![Page 28: Relations between image coordinates Given coordinates in one image, and the tranformation Between cameras, T = [R t], what are the image coordinates In.](https://reader030.fdocuments.in/reader030/viewer/2022032703/56649d2a5503460f949fe758/html5/thumbnails/28.jpg)
Point match ambiguity in human perception
![Page 29: Relations between image coordinates Given coordinates in one image, and the tranformation Between cameras, T = [R t], what are the image coordinates In.](https://reader030.fdocuments.in/reader030/viewer/2022032703/56649d2a5503460f949fe758/html5/thumbnails/29.jpg)
Traditional Solutions
• Try out lots of possible point matches.
• Apply constraints to weed out the bad ones.
![Page 30: Relations between image coordinates Given coordinates in one image, and the tranformation Between cameras, T = [R t], what are the image coordinates In.](https://reader030.fdocuments.in/reader030/viewer/2022032703/56649d2a5503460f949fe758/html5/thumbnails/30.jpg)
Find matches and apply epipolar uniqueness constraint
![Page 31: Relations between image coordinates Given coordinates in one image, and the tranformation Between cameras, T = [R t], what are the image coordinates In.](https://reader030.fdocuments.in/reader030/viewer/2022032703/56649d2a5503460f949fe758/html5/thumbnails/31.jpg)
Compute lots of possible matches
1) Compute “match strength”2) Find matches with highest strength
Optimization problem with many possible solutions
![Page 32: Relations between image coordinates Given coordinates in one image, and the tranformation Between cameras, T = [R t], what are the image coordinates In.](https://reader030.fdocuments.in/reader030/viewer/2022032703/56649d2a5503460f949fe758/html5/thumbnails/32.jpg)
Example