Vision-based Registration for AR

Presented by Diem VuNov 20, 2003

Markerless Tracking using Planar Structure in the Scene. G. Simon, A.W. Fitzgibbon and A. Zisserman, 2000.

Calibration-Free Augmented Reality. K.N Kutulakos and J.R. Vallino, 1998.

Planar-surface tracking.Camera position can be recovered from

planar homography.Planar structure is common in almost all

scenarios.

y

x

z

Hw

World to image homography

jiH

Image to image homography

World to image homographyConsider our tracking plane is the plane

Z=0

y

x

z

Hw

1H

1w YX

yx

Projection matrix

trrr 321KP

trrrr 2121 KP

Projection matrix

1yx

10YX

y

x

z

P

10

K1

2

YX

yx

trrr 31

trrrr 2121 KP

Projection matrix

1yx

10YX

y

x

z

P

1K

1YX

yx

trr 21

If K and Hw are known, then r1, r2 and t can be recovered, thus P.

Question: How to compute Hw?Direct.Indirect.

trr 21KHw

Direct measurement of Hw

Select 4 points {xk} on a rectangle in the scene.

Compute H which maps the unit square to {xk}.

(0,0)

(0,1) (1,1)

((1,0))

Direct measurement of Hw

Select 4 points {xk} on a rectangle in the scene.

Compute H which maps the unit square to {xk}.

s,1)(1,diagH H w

trr 21 s HK -1

s,1)(1,diagK H trr 21

Compute Hw=Hdiag(1,1/s,1)

(0,0)

(0,s) (1,s)

((1,0))

Indirect measurement of Hw

iwH

jiH

? H jw

y

x

z

Indirect measurement of Hw

iwH

jiH

iw

ji

jw HHH

y

x

z

Algorithm summaryCompute (direct measure).For each frame i, compute frame to frame

homography (RANSAC)Compute by:

0wH

1-iw

i1-i

iw HHH

i1-iH

iwH

Other …Using only 2 points in direct method ??Matching the frame i with frame 0 in order

to reduce error.Estimate intrinsic parameters K Hand-off mechanism.

Possible problems?Homography is only up-to-scale?Plain surface (no texture) or moving

objects in the foreground ?Depth order, occlusion ?Speed ?

Affine virtual object representation

Represent virtual objects so that their projection can be computed as a linear combination of the projection of the fiducial points.

Project a point from its affine coordinates

Compute affine coordinates from projection along two viewing

direction

Algorithm Setup the affine basis

Algorithm Setup the affine basis Locate the object in 2 frames.

Algorithm Setup the affine basis Locate the object in 2 frames. Compute the affine coordinates

for each point.

Algorithm Setup the affine basis Locate the object in 2 frames. Compute the affine coordinates

for each point. Compute projection of the object

and render the object in each frame.

Camera viewing direction and are the first and second row of

2x3.The camera viewing direction expressed in

the coordinate frame of the affine basis points: =

Depth order

w is the z-value of point p (x,y,z).

AdvantagesNo need any metric information.Able to use with the existing hardware to

accelerate graphics operations.Can be used to improve tracking.

LimitationAffine constraints.Lost of metric information.