Multi-linear Systems and Invariant Theory in the Context of Computer Vision and Graphics CS329

Multi-linear Systems and Invariant Theory

in the Context of Computer Vision and Graphics

CS329

Amnon Shashua

Material We Will Cover Today

• The structure of 3D->2D projection matrix

• The homography matrix

• A primer on projective geometry of the plane

The Structure of a Projection Matrix

WVU

Pw

'Ot

nb

U

V

W

X

Y

Z

O

bWnVtUOOOP

'

bntR

TWVU

RZYX

X

Y

Z


x

y

ZYX

P

),( 00 yx

),0,0( f

0xZXfx

0yZYfy


0xZXfx

0yZYfy

1

];[)(100

00

10

0

WVU

TRKTRPKZYX

yfxf

yx

w

PMp 43


Generally,

100sin

0

sincos

0

0

yf

xff

K y

xx

x

y

),( 00 yx is called the “principle point”

sincos

xfs is called the “skew”x

y

ff

is “aspect ratio”

The Camera Center

PMp 43

M has rank=3, thus c such that 0Mc

10];[0];[

TRccTRcTRK

T

Why is c the camera center?

The Camera Center

Why is c the camera center?

cPQ )1()( Consider the “optical ray”

P

c

pMPMPMQ )(

pAll points along the line )(Qare mapped to the same point

)(Q is a ray through the camera center

The Epipolar Points

c

'Mce

'c

cMe ''

Choice of Canonical Frame

PMWWMPp 1

PWWMPMp 1'''

PW 1 is the new world coordinate frame

Choose W such that ]0;[IMW

We have 15 degrees of freedom (16 upto scale)


];[ * mMM Let

/1)/1()(

];[11

*T

T

nmMnmMM

mMMW

mmmnmnI TT )/1()/1(;

0;I

We are left with 4 degrees of freedom (upto scale): ),( Tn


PIp ]0;[

PeHp ]';['

PnI

nI

Ip TT

0

/10

]0;[

'p

P

1yx

p

Pyx

PnI

T

10


p

nI

eHp

eHp T /10

]';['

]';['

')/1(;'/10

]';[ eneHnI

eH TT

';' eneH T

where ),( Tn are free variables

peneHp T ';''

jip

Projection Matrices

Let be the image of point iP at frame number j

ii PIp ]0;[0

ijT

jjji PeneHp ;

where ),( Tn are free variablesjip are known

ijT

jjji PeneHp ;

Family of Homography Matrices

TneHH '

H Stands for the family of 2D projective transformations between two fixed images induced by a plane in space

The remainder of this class is about making the above statement intelligible


1

];[WVU

TRKp

Recall,

3D->2D from Euclidean world frame to image

Let K,K’ be the internal parameters of camera 1,2 and choose canonical

frame in which R=I and T=0 for first camera.

PKp ]0;[

PtRKp ][''

1ZYX

P

TWVU

RZYX

world frame to first camera frame


PKp ]0;[

PtRKp ][''

1ZYX

P

ZYX

KZ

p 1Recall that 3rd row of K is )1,0,0(

tKpRKZKpZK

tRKp ''1

]['' 11

tKZ

pRKKp '1'' 1

tKZ

pRKKp '1'' 1


Assume

ZYX

are on a planar surface

dpZKn T )( 1

dZYX

n T

dpKnZtK

ZpRKKp

T 11 '1''


dpKnZtK

ZpRKKp

T 11 '1''

pKntKd

RKKp T )'1'(' 11

Let ,'' tKe and1 Knn TT 1' RKKH

pned

Hp T )'1('

Image-to-image mapping 'pp where the matching pair ', ppare induced by a planar surface.

Tned

HH '1


n

d

d

first camera frame

when1'

RKKHH

1' RKK is the homography matrix induced by the plane at infinity

Tned

HH '1


c

'Mce

'c

cMe ''

tKe '' is the epipole in the second image

]0;[KM

]['' tRKM

)1,0,0,0(Tc

tKcMe '''

Tned

HH '1

Relationship Between two Homography Matrices

TleHH '21

21 L

c

l

uHuH21 lu

0luT

l is the projection of 21 Lonto the first image

pHp '

pp’

Estimating the Homography Matrix

How many matching points?

phphx T

T

3

1'

phphy T

T

3

2'4 points make a basis for the projective plane

Projective Geometry of the Plane0 cbyax Equation of a line in the 2D plane

Tcbal ),,(The line is represented by the vector and 0lpT Tyxp )1,,(

Correspondence between lines and vectors are not 1-1 because Tcba ),,( represents the same line 0,0)( lpT

Two vectors differing by a scale factor are equivalent.This equivalence class is called homogenous vector. Anyvector Tcba ),,( is a representation of the equivalence class.

The vector T)0,0,0( does represent any line.

Projective Geometry of the Plane

A point

),( yx

lies on the line (coincident with) which is representedTcbal ),,(by iff 0lpT

But also 0)( lp T

0,),,( Tyx represents the pointTxxx ),,( 321 represents the point

),( yx

),(3

2

3

1

xx

xx

The vector T)0,0,0( does represent any point.

Points and lines are dual to each other (only in the 2D case!).

Projective Geometry of the Plane

r s

psrp

0)( rsrrp TT

0)( ssrsp TT

note: ),,det()( cbacba T

p

q

l

qpl

Lines and Points at Infinity

Consider lines Tcbar ),,( Tcbas )',,(

r s

00

)'(0

''

ab

ab

ccacaccbbc

sr

which represents the point )0

,0

( ab with infinitely large coordinates

ba

All meet at the same point

0ab

Lines and Points at Infinity

The points 2121 ,,)0,,( xxxx T lie on a line

0100

)0,,( 21

Txx

Tl )1,0,0(The line is called the line at infinity

The points 2121 ,,)0,,( xxxx T are called ideal points.

A line Tba ),,( meets Tl )1,0,0( at Tab )0,,(

(which is the direction of the line)

A Model of the Projective Plane

1x

2x

3x

0,),,( 321 Txxx

Points are represented as lines (rays) through the originLines are represented as planes through the origin

ideal pointTxx )0,,( 21

Tl )1,0,0(

is the plane 21xx

A Model of the Projective Plane

0],,...,[],...,[:]0,...,0[],...,[ 111 nnnn xxxxxxP

={lines through the origin in 1nR }

={1-dim subspaces of 1nR }

Projective Transformations in 2P

The study of properties of the projective plane that are invariantunder a group of transformations.

Projectivity: 22: PPh

that maps lines to lines (i.e. preserves colinearity)

Any invertible 3x3 matrix is a Projectivity:

Let 321 ,, ppp Colinear points, i.e.

0iT pl 01

iT HpHl

the pointsiHp lie on the line lH T

TH is the dual.His called homography, colineation


perspectivity

A composition of perspectivities from a plane to other planesand back to is a projectivity.

Every projectivity can be represented in this way.


Example, a prespectivity in 1D:Lines adjoining matching points are concurrent

a bc 'a

'b 'c

Lines adjoining matching points (a,a’),(b,b’),(c,c’) are not concurrent


Tl )1,0,0( is not invariant under H:

Points on l are Txx )0,,( 21

3

2

1

22112

1

'''

0 xxx

hxhxxx

H

3'x is not necessarily 0

Parallel lines do not remain parallel !

l is mapped to lH T

Projective Basis

A Simplex in 1nR is a set of n+2 points such that no subsetOf n+1 of them lie on a hyperplane (linearly dependent).

In 2P a Simplex is 4 points

Theorem: there is a unique colineation between any two Simplexes

Why do we need 4 points

Invariants are measurements that remain fixed under colineations

# of independent invariants = # d.o.f of configuration - # d.o.f of trans.

Ex: 1D case pHp x22' H has 3 d.o.f

A point in 1D is represented by 1 parameter.

4 points we have: 4-3=1 invariant (cross ratio)

2D case: H has 8 d.o.f, a point has 2 d.o.f thus 5 points induce 2 invariants


The cross-ratio of 4 points:

abb

d

bdcdacab

24 permutations of the 4 points forming 6 groups:

1

1,1

,1,1,1,


up

y

xz

5 points gives us 10 d.o.f, thus 10-8=2 invariants which represent 2D

uzyx ,,, are the 4 basis points (simplex)

yu

yp

xu yp

xpuz xx ,,,

ypuz yy ,,,

yx pp , are determined uniquely by ,

Point of intersection is preserved under projectivity (exercise)yx pp , uniquely determined p

Multi-linear Systems and Invariant Theory in the Context of Computer Vision and Graphics CS329

Documents

Transcript of Multi-linear Systems and Invariant Theory in the Context of Computer Vision and Graphics CS329