Lecture 4.3 Estimating homographies from feature correspondences Thomas Opsahl

Homographies induced by central projection

2

𝐻 𝐻

• Homography 𝐻𝒖� = 𝒖�′

𝐻 =ℎ1 ℎ2 ℎ3ℎ4 ℎ5 ℎ6ℎ7 ℎ8 ℎ9

• Point-correspondences can be determined automatically • Erroneous correspondences are common • Robust estimation is required to find 𝐻

1 1 2 2

2 3 1

𝒖 𝒖′

Homographies induced by central projection

3

𝐻

1 2 3

𝐻1←3 𝐻1←2

𝐻 1 2 1 2

• Homography 𝐻𝒖� = 𝒖�′

𝐻 =ℎ1 ℎ2 ℎ3ℎ4 ℎ5 ℎ6ℎ7 ℎ8 ℎ9

• Point-correspondences can be determined automatically • Erroneous correspondences are common • Robust estimation is required to find 𝐻

𝒖 𝒖′

Estimating the homography between overlapping images

• Establish point correspondences 𝒖𝑖 ↔ 𝒖′𝑖 – Find key points 𝒖𝑖 ∈ Img1 and 𝒖′𝑖 ∈ Img2 – Represent key points by suitable descriptors – Determine correspondences 𝒖𝑖 ↔ 𝒖′𝑖 by matching descriptors – Some wrong correspondences are to be expected

• Estimate the homography 𝐻 such that 𝒖′𝑖 = 𝐻𝒖𝑖 ∀𝑖

– Robust estimation with RANSAC – Improved estimation based on RANSAC inliers

• This homography enables us to compose the images into a

larger image – Image mosaicing – Panorama

4

Adaptive RANSAC

Objective To robustly fit a model 𝒚 = 𝑓 𝒙;𝜶 to a data set 𝑆 containing outliers Algorithm 1. Let 𝑁 = ∞, 𝑆𝐼𝐼 = ∅ and #𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖 = 0

2. while 𝑁 > #𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖 repeat 3-5

3. Estimate parameters 𝜶𝑡𝑡𝑡 from a random 𝑖-tuple from 𝑆

4. Determine inlier set 𝑆𝑡𝑡𝑡, i.e. data points within a distance 𝑖 of the model 𝒚 = 𝑓 𝒙;𝜶𝑡𝑡𝑡

5. If 𝑆𝑡𝑡𝑡 > 𝑆𝐼𝐼 , set 𝑆𝐼𝐼 = 𝑆𝑡𝑡𝑡, 𝜶 = 𝜶𝑡𝑡𝑡, 𝜔 = 𝑆𝐼𝐼

𝑆 and 𝑁 = 𝑙𝑙𝑙 1−𝑝

𝑙𝑙𝑙 1−𝜔𝑛 with 𝑝 = 0.99 Increase #𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖 by 1

5

Estimating the homography

• Estimating the homography in a RANSAC scheme requires 1. A basic homography estimation method for 𝑖 point-correspondences 2. A way to determine the inlier set of point-correspondences for a given homography

6

Estimating the homography

• Estimating the homography in a RANSAC scheme requires 1. A basic homography estimation method for 𝒏 point-correspondences 2. A way to determine the inlier set of point-correspondences for a given homography

• The homography has 8 degrees of freedom, but it is custom to treat all 9 entries of the matrix

as unknowns instead of setting one of the entries to 1 which excludes all potential solutions where this entry is 0

• Let us solve the equation 𝐻𝒖� = 𝒖�′ for the entries of the homography matrix 𝐻𝒖� = 𝒖�′ ℎ1 ℎ2 ℎ3ℎ4 ℎ5 ℎ6ℎ7 ℎ8 ℎ9

𝑢𝑣1

=𝑢′𝑣′1

7

Basic homography estimation

8

1

2

3

1 2 3 1 2 3 4

4 5 6 4 5 6 5

7 8 9 7 8 9 6

7

8

9

0 0 0 11 0 0 0

1 1 1 0 0 0

hhh

h h h u u uh vh h u u v v u v v v hh h h v v uh vh h v u v u u u v u hh h h uh vh h v u v v v u u u v u h

hhh

′ ′ ′ ′ ′+ + = − − − ′ ′ ′ ′ ′= ⇔ + + = ⇔ − − − ′ ′ ′ ′ ′ ′ + + = − − −

000

A

= ⇔ =

h 0

Basic homography estimation

Observe that the third row in 𝐴 is a linear combination of the first and second row 𝑖𝑖𝑟3 = −𝑢′ ∙ 𝑖𝑖𝑟1 − 𝑣′ ∙ 𝑖𝑖𝑟2

Hence every correspondence 𝒖𝑖 ↔ 𝒖′𝑖 contribute with 2 equations in the 9 unknown entries

9

1

2

3

1 2 3 1 2 3 4

4 5 6 4 5 6 5

7 8 9 7 8 9 6

7

8

9

0 0 0 11 0 0 0

1 1 1 0 0 0

hhh

h h h u u uh vh h u u v v u v v v hh h h v v uh vh h v u v u u u v u hh h h uh vh h v u v v v u u u v u h

hhh

′ ′ ′ ′ ′+ + = − − − ′ ′ ′ ′ ′= ⇔ + + = ⇔ − − − ′ ′ ′ ′ ′ ′ + + = − − −

000

A

= ⇔ =

h 0

Basic homography estimation

• Since 𝐻 (and thus 𝒉) is homogeneous, we only need the matrix 𝐴 to have rank 8 in order to determine 𝒉 up to scale

• It is sufficient with 4 point correspondences where no 3 points are collinear

• We can calculate the non-trivial solution to the equation 𝐴𝒉 = 𝟎 by SVD

svd 𝐴 = 𝑈𝑆𝑉𝑇

• The solution is given by the right singular vector without a singular value which is the last column of 𝑉, i.e. 𝒉 = 𝒗9

10

1

2

31 1 1 1 1 1 1

41 1 1 1 1 1 1

52 2 2 2 2 2 2

62 2 2 2 2 2 2

7

8

9

0 0 0 1 01 0 0 0 0

0 0 0 1 01 0 0 0 0

hhhu v v u v v vhu v u u u v uhu v v u v v vhu v u u u v uhhhA

′ ′ ′ − − − ′ ′ ′− − − ′ ′ ′ =− − − ′ ′ ′− − −

=

h 0

Basic homography estimation

• Estimating the homography in a RANSAC scheme requires 1. A basic homography estimation method for 𝒏 point-correspondences 2. A way to determine which of the point correspondences that are inliers for a given homography

Direct Linear Transform

1. Build the matrix 𝐴 from at least 4 point-correspondences 𝑢𝑖 ,𝑣𝑖 ↔ 𝑢′𝑖 ,𝑣′𝑖 2. Obtain the SVD of 𝐴: 𝐴 = 𝑈𝑆𝑉𝑇 3. If 𝑆 is diagonal with positive values in descending order along the main diagonal,

then 𝒉 equals the last column of 𝑉 4. Reconstruct 𝐻 from 𝒉

1 1 1 1 1 1 1

1 1 1 1 1 1 1

0 0 0 11 0 0 0

u v v u v v vA u v u u u v u

′ ′ ′− − − ′ ′ ′= − − −

11

Basic homography estimation

• The basic DLT algorithm is never used with more than 4 point-correspondences • This is because the algorithm performs better when all the terms of 𝐴 has a similar scale

– Note that some of the terms will always be of scale 1 • To achieve this, it is common to extend the algorithm with a normalization and a

denormalization step

Normalized Direct Linear Transform 1. Normalize the set of points 𝒖𝑖 = 𝑢𝑖 , 𝑣𝑖 𝑇 by computing a similarity transform 𝑇 that

translates the centroid to the origin and scales such that the average distance from the origin is 2

2. In the same way normalize the set of points 𝒖′𝑖 = 𝑢′𝑖 ,𝑣′𝑖 𝑇 by computing a similarity transform 𝑇′

3. Apply the basic DLT algorithm on the normalized points to obtain a homography 𝐻� 4. Denormalize to get the homography: 𝐻 = 𝑇′−1𝐻�𝑇

12

Basic homography estimation

• Estimating the homography in a RANSAC scheme requires 1. A basic homography estimation method for 𝑖 point-correspondences 2. A way to determine the inlier set of point-correspondences for a given homography

• For a point correspondence 𝑢𝑖 , 𝑣𝑖 ↔ 𝑢′𝑖 , 𝑣′𝑖 and homography 𝐻, we can choose from

several errors – Algebraic error: 𝜀𝑖 = 𝐴𝑖𝒉 where

– Geometric errors: 1. 𝜀𝑖 = 𝑑 𝐻𝒖𝑖 ,𝒖′𝑖 + 𝑑 𝒖𝑖 ,𝐻−1𝒖′𝑖 (Reprojection error) 2. 𝜀𝑖 = 𝑑 𝒖𝑖 ,𝐻−1𝒖′𝑖 3. 𝜀𝑖 = 𝑑 𝐻𝒖𝑖 ,𝒖′𝑖

0 0 0 11 0 0 0

i i i i i i ii

i i i i i i i

u v v u v v vA

u v u u u v u′ ′ ′− − −

= ′ ′ ′− − −

13

Notation

Euclidean distance 𝑑 ∙,∙

Inhomogenous 𝐻𝒖�𝑖 𝐻𝒖𝑖

Inhomogeneous 𝐻−1𝒖�′𝑖 𝐻−1𝒖′𝑖

Robust homography estimation

• Finally we would typically re-estimate 𝐻 from all correspondences in 𝑆𝐼𝐼 – Normalized DLT – Minimize ϵ = ∑𝜖𝑖 in an iterative optimization method like Levenberg Marquardt

RANSAC estimation of homography For a set of point-correspondences S = 𝒖𝒊 ↦ 𝒖′𝑖 , perform 𝑁 iterations, where 𝑁 is determined adaptively

1. Estimate 𝐻𝑡𝑡𝑡 from 4 random correspondences 𝒖𝒊 ↦ 𝒖′𝑖 using the basic DLT algorithm 2. Determine the set of inlier-correspondences 𝑆𝑡𝑡𝑡 = 𝒖𝒊 ↦ 𝒖′𝑖 𝑖𝑢𝑠ℎ 𝑖ℎ𝑖𝑖 𝜖𝑖 < 𝑖

Here one can choose 𝜖𝑖 = 𝑑 𝐻𝒖𝑖 ,𝒖′𝑖 + 𝑑 𝒖𝑖 ,𝐻−1𝒖′𝑖 and 𝑖 = 5.99𝜎 where 𝜎 is the expected uncertainty in key-point positions

3. If 𝑆𝑡𝑡𝑡 > 𝑆𝐼𝐼 update 𝑁, homography and inlier set: 𝐻 = 𝐻𝑡𝑡𝑡, 𝑆𝐼𝐼 = 𝑆𝑡𝑡𝑡

14

Image mosaicing

15

• Let us compose these two images into a larger image

Image mosaicing

• Find key points and represent by descriptors

16

Image mosaicing

• Establish point-correspondences by matching descriptors • Several wrong correspondences

17

Image mosaicing

• Establish point-correspondences by matching descriptors • Several wrong correspondences

18

Image mosaicing

• Estimate homography 𝐻𝒖� = 𝒖�′ – OpenCV

#include "opencv2/calib3d.hpp" cv::findHomography(srcPoints, dstPoints, CV_RANSAC);

– Matlab tform = estimateGeometricTransform(srcPoints,dstPoints,’projective’);

19

𝐻

Image mosaicing

• Represent the images in common coordinates (Note the additional translation!) – OpenCV

#include "opencv2/calib3d.hpp" cv::warpPerspective(img1, img2, H, output_size);

– Matlab img2 = imwarp(img1,tform);

20

Image mosaicing

• Now we can compose the images

21

22

Overwriting

23

Blending with a ramp

24

Blending with a ramp + histogram equalization

SVD How to use • Matlab

[U,S,V] = svd(A); Right singular vectors are columns in V

• OpenCV cv::SVD::compute(A, S, U, Vtranspose, cv::SVD::FULL_UV); Right singular vectors are rows in Vtranspose

• Eigen Eigen::JacobiSVD<Eigen::MatrixXd> svd(A, Eigen::ComputeFullU | Eigen::ComputeFullV); Right singular vectors are columns in svd.matrixV()

25

Singular Value Decomposition The singular value decomposition of a real 𝑚 × 𝑖 matrix 𝐴 is a factorization 𝐴 = 𝑈S𝑉𝑇 Here 𝑈 is a orthogonal 𝑚 × 𝑚 matrix, 𝑉 is a orthogonal 𝑖 × 𝑖 matrix and 𝑆 is a real positive diagonal 𝑚 × 𝑖 matrix The diagonal entries of 𝑆 = diag 𝑖1, … , 𝑖min 𝑚,𝑛 are known as the singular values of 𝐴 and the columns of 𝑈 = 𝒖1, … ,𝒖𝑚 and 𝑉 = 𝒗1, … ,𝒗𝑛 are known as the left and right singular vectors of 𝐴 respectively The nullspace of A is the span of the right singular vectors 𝒗𝑖 that corresponds to a zero singular value 𝑖𝑖 (or does not have a corresponding singular value)

SVD Singular Value Decomposition The singular value decomposition of a real 𝑚 × 𝑖 matrix 𝐴 is a factorization 𝐴 = 𝑈S𝑉𝑇 Here 𝑈 is a orthogonal 𝑚 × 𝑚 matrix, 𝑉 is a orthogonal 𝑖 × 𝑖 matrix and 𝑆 is a real positive diagonal 𝑚 × 𝑖 matrix The diagonal entries of 𝑆 = diag 𝑖1, … , 𝑖min 𝑚,𝑛 are known as the singular values of 𝐴 and the columns of 𝑈 = 𝒖1, … ,𝒖𝑚 and 𝑉 = 𝒗1, … ,𝒗𝑛 are known as the left and right singular vectors of 𝐴 respectively The nullspace of A is the span of the right singular vectors 𝒗𝑖 that corresponds to a zero singular value 𝑖𝑖 (or does not have a corresponding singular value)

Applications of SVD Solving homogeneous linear equations like

𝐴𝒉 = 𝟎 Method For theoretical problems, 𝐡 ∈ null 𝐴 so 𝒉 is a linear combination of the right singular vectors 𝒗𝑖 that correspond to a zero singular value 𝑖𝑖

𝒉 = �𝑘𝑖𝒗𝑖 ; 𝑘𝑖 ∈ ℝ, 𝑖𝑖 = 0 (or missing)

For practical problems, the presence of noise force us to expand the solution by including those right singular vectors that correspond to small singular values 𝑖𝑖 ≈ 0

𝒉 = �𝑘𝑖𝒗𝑖 ; 𝑘𝑖 ∈ ℝ, 𝑖𝑖 ≈ 0 (or missing)

26

SVD Example

From [U,S,V] = svd(A); we get

From this we see that A has: • 2 left singular vectors

• 2 nonzero singular values • 3 right singular vectors Since 𝒗3 does not have a corresponding singular value, 𝒙 = 𝒗3 is a non-trivial solution to A𝒙 = 0 and 𝒙 = 𝑘 ∙ 𝒗3; 𝑘 ∈ ℝ\ 0 is the family of all non-trivial solutions

27

1 2 3 04 5 6 0

Axyz

=

=

x 0

0.3863 0.9224 9.5080 0 00.9224 0.3863 0 0.7729 0

0.4287 0.8060 0.40820.5663 0.1124 0.81650.7039 0.5812 0.4082

U S

V

− − = = −

− = − − − −

1 2 3

0.4287 0.8060 0.40820.5663 0.1124 0.81650.7039 0.5812 0.4082

− = − = = − − −

v v v

1 29.5080 0.7729s s= =

1 2

0.3863 0.92240.9224 0.3863− −

= = − u u

SVD Example

This time singular value decomposition give us the following singular values and right singular vectors:

This time all right singular vectors correspond to a non-zero singular value, so the equation does not have any non-trivial solutions!

If this equation came from a practical problem, instead of looking for solutions to 𝐴𝒙 = 0, we might be looking for the 𝒙 that minimize 𝐴𝒙 Since 𝑖1 ≉ 0, 𝑖2 ≉ 0, 𝑖3 ≈ 0, we would conclude that 𝒙 = 𝒗3 solves the equation in a least-squares sense Check:

28

1.0792 2.0656 3.08494.0959 5.0036 6.0934 01.0679 2.0743 3.0655 04.0758 5.0392 6.0171

A

xyz

=

=

x 0

1 2 3

1 2 3

13.6295 1.0849 0.05060.4336 0.8103 0.39420.5635 0.0975 0.82030.7032 0.5778 0.4143

s s s= = =

− − = − = − = − −

v v v 3

0.00910.42880.01050.0341

A

= − −

v

Summary

• Homography 𝐻𝒖� = 𝒖�′

𝐻 =ℎ1 ℎ2 ℎ3ℎ4 ℎ5 ℎ6ℎ7 ℎ8 ℎ9

• Automatic point-correspondences

• Wrong correspondences are common

• RANSAC estimation

– Basic DLT (Direct Linear Transform) on 4 random correspondences

– Inliers determined from the reprojection error 𝜖𝑖 = 𝑑 𝐻𝒖𝑖 ,𝒖′𝑖 + 𝑑 𝒖𝑖 ,𝐻−1𝒖′𝑖

• Improve estimate by normalized DLT on inliers or iterative methods for an even better estimate

• Additional reading – Szeliski: 6.1.1 – 6.1.3

29