Section 4 -

Jinchao Ye

CS231A Section 4 Camera Models (Problem Set 2)

10/19/2012

Jinchao Ye

1 10/19/2012

Section 4 -

Jinchao Ye

Topics

• Homogeneous Coordinates

• Transformations

• Solving a Homogeneous Linear Equation System • SVD

• Camera Calibration

2 10/19/2012

Section 4 -

Jinchao Ye

Homogeneous Coordinates

• Why?

• Without homogeneous

• With homogeneous

3

Convert to homogeneous

Linear projection

Convert from homogenous

Non-linear operation

3D World 2D Image

3D World 2D Image

3D 4D

10/19/2012

Section 4 -

Jinchao Ye

Homogeneous Coordinates

• Convert to homogeneous

𝑥, 𝑦 ⇒𝑥𝑦1

(image) 𝑥, 𝑦, 𝑧 ⇒

𝑥𝑦𝑧1

(scene)

• Convert from homogeneous

𝑥𝑦𝑤

⇒ 𝑥/𝑤, 𝑦/𝑤

𝑥𝑦𝑧𝑤

⇒ 𝑥/𝑤, 𝑦/𝑤, 𝑧/𝑤

• Keep track of dimensions – Usually, 3 (world) to 2 (image) – Homogeneous: 4(world) to 3(image) – X’ = MX

• **Hint 𝑥𝑦𝑧𝑤

⇒ 𝑥/𝑤, 𝑦/𝑤, 𝑧/𝑤

𝑐𝑥𝑐𝑦𝑐𝑧𝑐𝑤

⇒ 𝑥/𝑤, 𝑦/𝑤, 𝑧/𝑤

10/21/2011 4

Section 4 -

Jinchao Ye

Homogeneous Coordinates of Lines

5 10/19/2012

• The homogeneous coordinate of a line is • The dot product of the homogeneous coordinates of a line and a point on it is zero

• Cross product of two lines gives the homogeneous coordinate of their point of intersection

• Points at infinity (ideal points): • Line at infinity (ideal line): set of ideal points,

Section 4 -

Jinchao Ye

Rotation Matrices

• Examples

𝑅𝑧 =𝑐𝑜𝑠𝜃𝑧 −𝑠𝑖𝑛𝜃𝑧 0𝑠𝑖𝑛𝜃𝑧 𝑐𝑜𝑠𝜃𝑧 00 0 1

𝑅𝑦 =

𝑐𝑜𝑠𝜃𝑦 0 −𝑠𝑖𝑛𝜃𝑦0 1 0

𝑠𝑖𝑛𝜃𝑦 0 𝑐𝑜𝑠𝜃𝑦

𝑅𝑥 =1 0 00 𝑐𝑜𝑠𝜃𝑥 −𝑠𝑖𝑛𝜃𝑥0 𝑠𝑖𝑛𝜃𝑥 𝑐𝑜𝑠𝜃𝑥

𝑅 = 𝑅𝑥𝑅𝑦𝑅𝑧

• Important Properties – 𝑅 = 1

– 𝑅𝑇𝑅 = 𝐼

– 𝑅𝑇 = 𝑅−1

6 10/19/2012

Section 4 -

Jinchao Ye

Homography/Projective/Perspective

• Parallel lines intersect at vanishing points

• World->image

𝑥𝑦𝑤

=

𝑎11 𝑎12 𝑎13𝑎21 𝑎22 𝑎23𝑎31 𝑎32 𝑎33

𝑏1𝑏2𝑏3

𝑋𝑌𝑍1

• Image->image

𝑥𝑦𝑤

=

𝑎11 𝑎12 𝑏1𝑎21 𝑎22 𝑏2𝑎31 𝑎32 𝑏3

𝑋𝑌1

7 10/19/2012

Section 4 -

Jinchao Ye

A Quick Question…

• Do these 2 homography matrices yield the same points in the image space 𝑥 and 𝑥’?

8

𝑥 = 𝐻𝑋 𝑥′ = 𝑐𝐻𝑋

10/19/2012

Section 4 -

Jinchao Ye

Affine Transform

• Special case: weak perspective – simpler math (less computations) • Parallel lines remain parallel • World->image

𝑥𝑦𝑤

=𝑎11 𝑎12 𝑎13𝑎21 𝑎22 𝑎230 0 0

𝑏1𝑏21

𝑋𝑌𝑍1

• Image->image 𝑥𝑦𝑤

=𝑎11 𝑎12 𝑏1𝑎21 𝑎22 𝑏20 0 1

𝑋𝑌1

• |𝐻| ≠ 0 • w = 1 (no division required!)

9 10/19/2012

Section 4 -

Jinchao Ye

Solving For an Affine Transform Matrix

𝑥1𝑦11

𝑥2 𝑦2 1

𝑥3𝑦31…

=𝑎11 𝑎12 𝑎13 𝑏1𝑎21 𝑎22 𝑎23 𝑏20 0 0 1

𝑋1

𝑌1𝑍1

𝑋2

𝑌2 𝑍2

𝑋3

𝑌3 …𝑍3

1 1 1

10

unknowns

10/19/2012

Section 4 -

Jinchao Ye

Solving For an Affine Transform Matrix

𝑥1𝑦11

𝑥2 𝑦2 1

𝑥3𝑦31

=𝑎11 𝑎12 𝑏1𝑎21 𝑎22 𝑏20 0 1

𝑋1

𝑌11

𝑋2

𝑌21

𝑋3

𝑌31

11

unknowns

10/19/2012

Section 4 -

Jinchao Ye

Solving a Homogeneous Linear Equation System

• A standard problem in computer vision is to solve a set of homogeneous linear equations.

• 𝐴𝑥 = 0 – Where x is the vector of 𝑁 unknowns, and 𝐴 is the

matrix of (𝑀 × 𝑁) coefficients. We only consider the situation of 𝑀 ≥ 𝑁, so that we have enough equations to solve for the unknowns.

• 𝒙 = 0 is a trivial solution. – Constraint 𝒙 2 = 1. This is reasonable because 𝑥 is up

to scale.

10/21/2011 12

Section 4 -

Jinchao Ye

Solving a Homogeneous Linear Equation System

• 𝑀 = 𝑁

• 𝑀 > 𝑁

10/21/2011 13

𝑨𝒙 = 0 𝑠. 𝑡. 𝒙 2 = 1

𝑚𝑖𝑛 𝑨𝒙 𝟐 𝑠. 𝑡. 𝒙 2 = 1

Section 4 -

Jinchao Ye

Singular Value Decomposition

• The best numerical way to solve the homogeneous linear equation system is to perform singular value decomposition on matrix 𝑨.

10/21/2011 14

𝐴 = 𝑈Σ𝑉∗ 𝑠𝑖𝑧𝑒 𝐴 = 𝑚, 𝑛 𝑠𝑖𝑧𝑒 𝑈 = 𝑚,𝑚 𝑠𝑖𝑧𝑒 Σ = 𝑚, 𝑛 𝑠𝑖𝑧𝑒 𝑉∗ = 𝑛, 𝑛

𝑈𝑈∗ = 𝐼 𝑉𝑉∗ = 𝐼 Σ: 𝑟𝑒𝑐𝑡𝑎𝑛𝑔𝑙𝑒 𝑑𝑖𝑎𝑔𝑜𝑛𝑎𝑙 𝑚𝑎𝑡𝑟𝑖𝑥 𝑤𝑖𝑡𝑕 𝑛𝑜𝑛𝑛𝑒𝑔𝑎𝑡𝑖𝑣𝑒 𝑟𝑒𝑎𝑙 𝑛𝑢𝑚𝑏𝑒𝑟𝑠

𝑙𝑒𝑓𝑡 𝑠𝑖𝑛𝑔𝑢𝑙𝑎𝑟 𝑣𝑎𝑙𝑢𝑒𝑠 𝑜𝑓 𝐴 𝑎𝑟𝑒 𝑒𝑖𝑔𝑒𝑛𝑣𝑒𝑐𝑡𝑜𝑟𝑠 𝑜𝑓 𝐴𝐴∗ right 𝑠𝑖𝑛𝑔𝑢𝑙𝑎𝑟 𝑣𝑎𝑙𝑢𝑒𝑠 𝑜𝑓 𝐴 𝑎𝑟𝑒 𝑒𝑖𝑔𝑒𝑛𝑣𝑒𝑐𝑡𝑜𝑟𝑠 𝑜𝑓 𝐴𝐴∗

non − zero 𝑠𝑖𝑛𝑔𝑢𝑙𝑎𝑟 𝑣𝑎𝑙𝑢𝑒𝑠 𝑜𝑓 𝐴 𝑎𝑟𝑒 𝑡𝑕𝑒 𝑠𝑞𝑢𝑎𝑟𝑒 𝑟𝑜𝑜𝑡𝑠 𝑜𝑓 𝑡𝑕𝑒 𝑛𝑜𝑛− 𝑧𝑒𝑟𝑜𝑠 𝑒𝑖𝑔𝑒𝑛𝑣𝑎𝑙𝑢𝑒𝑠 𝑜𝑓 𝑏𝑜𝑡𝑕 𝐴𝐴∗ 𝑎𝑛𝑑 𝐴∗𝐴

Section 4 -

Jinchao Ye

Singular Value Decomposition

• The least square solution is given by the column of 𝑽, which corresponds to the smallest diagonal entry of 𝚺.

– 𝐴𝑇𝐴𝑥 = 𝜆2𝑥 && 𝑥 2 = 1

–⇒ 𝐴𝑥 2 = 𝑥𝑇𝐴𝑇𝐴𝑥 = 𝑥𝑇𝜆2𝑥 = 𝜆2 𝑥 2 = 𝜆2

• How to calculate using Matlab

– [U,D, V] = svd(A,0);

– x = V(:, end);

10/21/2011 15

Section 4 -

Jinchao Ye

Camera Calibration

• Maps 3D world scene onto 2D image in homogeneous coordinates.

• M= 𝐾 𝑅 𝑇

– [R T]: rigid transformation

• From world to camera reference

• Extrinsic parameters

– K: camera calibration matrix

• From camera reference to sensor

• intrinsic parameters

16 10/19/2012

Section 4 -

Jinchao Ye

Camera Calibration

10/21/2011 17

Section 4 -

Jinchao Ye

Camera Calibration

10/21/2011 18

Section 4 -

Jinchao Ye

Camera Calibration

10/21/2011 19

Section 4 -

Jinchao Ye

Camera Calibration

• 𝑷𝒎 = 0

• 𝑷: 2n × 12;𝒎: 12 × 1

• Over-determined solution, minimize algebraic

error: 𝑷𝒎

– min 𝑷𝒎 𝑠. 𝑡. 𝒎 = 1

• Every singular value is non-negative.

– The right-singular vector corresponds to the smallest singular value is the solution to the optimization problem.

10/21/2011 20

Section 4 -

Jinchao Ye

Camera Calibration

10/21/2011 21

• Last column of 𝑽 gives 𝒎.

• 𝒎 = 𝒎𝟏𝑻,𝒎𝟐

𝑻,𝒎𝟑𝑻 𝑻

• 𝑴 = [𝒎𝟏;𝒎𝟐;𝒎𝟑]

• 𝑴𝑷𝒊 = 𝒑𝒊

Section 4 -

Jinchao Ye

Camera Calibration

• The above method is called Direct Linear Transformation (DLT).

• In practice, to prevent unexpected numerical error, normalization is necessary. – Use a similarity transformation 𝑇 to normalize the image

points, s.t. centroid of the points is at origin, and their RMS distance from the origin is 2. 𝑝𝑖 = 𝑇𝑝𝑖

– Use a similarity transformation 𝑈 to normalize the space points, s.t. centroid of the points is at origin, and their RMS distance from the origin is 3. 𝑃𝑖 = 𝑈𝑃𝑖

– Use DLT to minimize algebraic error. Get 𝑀

– 𝑀 = 𝑇−1𝑀 𝑈

10/21/2011 22

Section 4 -

Jinchao Ye

Camera Calibration

10/21/2011 23

Section 4 -

Jinchao Ye

Some Notation…

24 10/19/2012

Section 4 -

Jinchao Ye

Epipolar Geometry

25 10/19/2012

Section 4 -

Jinchao Ye

Essential and Fundamental Matrix

26 10/19/2012

Section 4 -

Jinchao Ye

A Simple “Trick”

• The fundamental matrix corresponding to a camera pair, M = 𝐼 0 and M′ = 𝐴 𝑎 is equal to 𝑎 𝑥𝐴

27 10/19/2012