Metric Self Calibration From Screw-Transform Manifolds Russell Manning and Charles Dyer University...

Metric Self Calibration From Screw-Transform Manifolds

Russell Manning and Charles Dyer

University of Wisconsin -- Madison

Goal

• Find internal camera calibration

• Only input is images (views)• Camera…

– has fixed internal parameters – undergoes general motion

Input

assorted views

Output

• Internal calibration

• (Optional) scene reconstruction

reconstruction

Quick Overview

Find pairwise fundamental matrices Fi

F1

F2

Don’t need an initial projective reconstruction

Just need fundamental matrices, which are derived from view pairs

So related to methods based on the Kruppa constraints rather than stratified calibration methods like the modulus constraint

Each fundamental matrix induces a curved surface (manifold) in R5

F1

every manifold point is a legal internal calibration

There is a unique mutual intersection point to all these manifolds

F4

F1

F2

F3

intersection point gives calibration

Details

• What is a manifold?• What is a screw transform?• Screw-transform manifolds and their

coordinate system• Finding intersection point: voting algorithm

Manifold

Surface in Rn that has a coordinate system

-2

-1

0

13

4

5

6 7

8

2

Manifold

grid paper example

kappa

theta

Screw Transformation

A B

A

A B

B


the scene

camera A

camera B (=camera A in arbitrary new position)

screw transformation

Equivalent to Rising Turntable

rising turntable: just like a normal turntable except can also translate (rise) parallel to its rotation axis

*turntable is viewed by a single camera, fixed in position

camera A

scene rising turntable



camera B

translated by gamma

rotated by theta


The Two Camera Matrices

camera A...

is located at position (1,0,0)

is tilted by a rotation matrix R

and has internal calibration K


camera B...is exactly like camera A, but...

the world has first undergone a screw transform S


rotation by theta around the z-axis (== screw axis)

translation by gamma parallel to z-axis


so we can refer to the column vectors of the camera matrices

Fundamental Matrix

screw rotation angle

screw translation columns from camera matrices

H = [ h1 h2 h3 ] = K R

Fundamental Matrix

through this representation, we will be able to find K from F by picking three real numbers: theta, gamma, and kappa !

these real numbers parameterize a “screw-transform manifold”

Screw Transform Manifold

You give me 3 real numbers (kappa, theta, gamma)and a fundamental matrix

I run the numbers through an algorithm (given in the paper)to get a legal internal calibration matrix (K)

Set of legal K’s is the screw-transform manifold

(kappa, theta, gamma) are the coordinate system

Where does manifold live?

Screw Transform Manifold

Manifold lives in R5 because K is 3x3 upper-triangularmatrix:

x x x

00

x0x x

only 5 degrees of freedom since don’t care about scale (e.g., make norm 1)

Mutual Intersection Point

Manifold is a set of points

Each point is a legal internal calibration matrix

Manifold corresponds to fundamental matrix

The mutual intersection point of all the manifolds is a legal internal calibration matrix for every fundamental matrix

Voting Algorithm

Basic Idea

“sketch” the manifolds within initial search region

find approximate region where manifolds seem to intersect

“zoom in”: repeat the process starting in the new, smaller search region

Manifold defined by function f(t) with one underlying real-valued parameter t

In the limited search space shown at left, t might vary between 0 and 1

Pick t at random, say t=0.6, and find f(t)

f(t) is shown as a dotted line because we don’t know it’s shape initially

Continue picking random values for t in range [0,1] to sketch manifold f(t)

To utilize the randomly-selected points (“votes”), first drop a voxel grid over the search space

Mark voxels that receive at least one vote

Sketch the second manifold in the same way

Now consider tabulating the votes

For each voxel, keep track of how many manifolds have voted for that voxel

lighter color means higher number of votes

When some region receives enough votes, zoom in on that region

Now “zoom in” -- start over using the smaller search region as the original search region

“binary search”

Repeat the sketching and voting procedure.

A winner emerges.

Determine the new “zoom in” region

Voxelize the new search space

No consensus!

Properties of Voting Algorithm

Benefits from using lots of fundamental matrices

Noisy manifolds will contribute votes in empty regionsand thus be ignored

Zoom-in step eliminates bad manifolds

Determines a small region where K is rather than anexact point

Quick convergence at end when manifolds becomelinear

Properties of Voting Algorithm

General technique -- use whenever need to intersectmanifolds

(1) What is self calibration?(2) What is a screw transform?(3) Fundamental matrix + choice of three screw

transform parameters internal calibration matrix.

(4) Internal calibration matrix K lives in a 5D space (termed “K-space”).

(5) Each fundamental matrix yields a 3D manifold in K-space, termed “screw-transform manifold.”

(6) Intersection of 3 or more manifolds gives K matrix common to all view pairs (i.e., finds internal calibration).

Map of the Talk

(1) A sample application of self calibration

(2) The “K” matrix (internal calibration matrix).

(3) Relationship between K and “screw transform” that underlies pairwise views.

(5) The screw-transform manifold.

(6) Intersecting screw-transform manifolds.

(7) Algorithm for self calibration.

(8) Voting algorithm to find intersection point.

(4) What is a manifold?

(pictures of me taking photos of scene…leading to self calibration and scene reconstruction)

Map of the Talk









Camera Equation

Consider the equation for how points in 3D get projected onto image plane of camera…

Camera Equation

p == K [ R | t ] P

position P in space…

…gets translated by t and rotated by R…

…then transformed to camera’s internal coordinate system by K

p is projected position on image plane (in homogeneous coords)

Self Calibration

K is an upper triangular, 3x3 matrix:

x x x

00

x0x x

only 5 degrees of freedom since don’t care about scale (e.g., make norm 1)

p == K [ R | t ] P

Goal of self calibration is to find K up to a scale factor.

Map of the Talk










arbitrary movement in space


screw axis

screw rotation


screw axis


the scene

camera A

camera B (=camera A in arbitrary new position)

screw transformation



*is viewed by a single camera, fixed in position

camera A

scene rising turntable




camera B

translated by gamma

rotated by theta


camera A...

is located at position (1,0,0)

is tilted by a rotation matrix R

and has internal calibration K


camera B...is exactly like camera A, but...

the world has first undergone a screw transform S


rotation by theta around the z-axis (== screw axis)

translation by gamma parallel to z-axis


so we can refer to the column vectors of the camera matrices

Fundamental Matrix

screw rotation angle

screw translation columns from camera matrices

H = [ h1 h2 h3 ] = K R

Fundamental Matrix

through this representation, we will be able to find K from F by picking three real numbers: theta, gamma, and kappa !

these real numbers parameterize a “screw-transform manifold”

Finding K from F

F and kappa h3

h3 and theta h1 and h2 up to a scale factor

gamma h1 and h2

h1, h2, and h3 K R = [ h1 h2 h3 ]

K R K since (K R) (K R)T = K R RT KT = K KT,which gives K by Cholesky factorization

Map of the Talk









Manifolds

Example 1: A one-dimensional manifold in 2D space

Manifolds

A 1D manifold in the 2D space

Another 1D manifold

Manifold can be seen as a function f(t) that maps real numbers R into R2

t “parameterizes” the manifold

Part of a 2D space

Manifolds

Manifolds are locally flat

That is, locally a manifold behaves like its underlying parameter space (which is Rn)

In this case, each manifold is locally like a line

Manifolds

Example 2: A two-dimensional manifold in 3D space

Map of the Talk









Screw-Transform Manifold

Using this formulation, it can be shown that once a fundamental matrix between two views is known, then internal calibration K is known up to three parameters (i.e., K has three degrees of freedom).

Earlier it was shown how the fundamental matrix between two views relates to the parameters of the underlying screw transformation.

Screw-Transform Manifold

The set of possible K matrices that are consistent with a given fundamental matrix is a 3D manifold in (5D) K-space.

K-space == space of possible K matrices (of norm 1)

theta = angle of rotation for the screw transform

gamma = translation parallel to the screw axis

kappa = determines location of screw axis

The three underlying parameters:

Map of the Talk









Mutual Intersection Point

Every point on screw-transform manifold i represents a K matrix that is consistent with fundamental matrix i.

Thus a point that lies in the intersection of all the screw transform manifolds is a legal K that is consistent with every fundamental matrix.

At least one such point exists (since the camera has an internal calibration matrix).

Others have shown that, under the right conditions, there is a unique intersection point.

double intersection

all manifolds intersect here

for self calibration, we will want this point of mutual intersection

Map of the Talk









(1) Take several pictures of a scene.

(2) Find all pairwise fundamental matrices.

(3) Each fundamental matrix leads to a screw-transform manifold.

F1

F2

F3

(4) Find K by intersecting the manifolds.

Map of the Talk









Basic Idea

“sketch” the manifolds within initial search region

find approximate region where manifolds seem to intersect

“zoom in”: repeat the process starting in the new, smaller search region

Manifold defined by function f(t) with one underlying real-valued parameter t

In the limited search space shown at left, t might vary between 0 and 1

Pick t at random, say t=0.6, and find f(t)

f(t) is shown as a dotted line because we don’t know it’s shape initially

Continue picking random values for t in range [0,1] to sketch manifold f(t)

To utilize the randomly-selected points (“votes”), first drop a voxel grid over the search space

Mark voxels that receive at least one vote

Sketch the second manifold in the same way

Now consider tabulating the votes

For each voxel, keep track of how many manifolds have voted for that voxel

lighter color means higher number of votes

When some region receives enough votes, zoom in on that region

Now “zoom in” -- start over using the smaller search region as the original search region

“binary search”

Repeat the sketching and voting procedure.

A winner emerges.

Determine the new “zoom in” region

Voxelize the new search space

No consensus!

Experimental Results

Performance

Concluding Remarks

Last Slide

Self Calibration

Voting Algorithm

Tell me again why we want to find the mutual intersection point?

Every point in K-space corresponds to a possible K matrix (where K is the internal calibration that we wish to find).

Each fundamental matrix F induces a “screw-transform manifold” in K-space consisting of all possible K matrices that are consistent with F.

Hence the mutual intersection point is consistent with all the fundamental matrices and must represent the internal calibration of the camera.

3 or more fundamental matrices are necessary for the manifold intersection to be a single point (or finite set of points).

More on K-space

K-space is 5D; each STM is a 3D manifold in this space.


(two photos of me moving camera to different positions)

Metric Self Calibration From Screw-Transform Manifolds Russell Manning and Charles Dyer University...

Documents

Transcript of Metric Self Calibration From Screw-Transform Manifolds Russell Manning and Charles Dyer University...