Lecture 22: Camera Motionrtc12/CSE486/lecture22.pdf · Lecture 22: Camera Motion Readings: ......

CSE486, Penn StateRobert Collins

Lecture 22:

Camera Motion

Readings: T&V Sec 8.1 and 8.2


Moving Camera

yaw

pitch

roll

Tx

TyTz

From one time to the next, the camera undergoesrotation (roll, pitch, yaw) and translation (tx,ty,tz)

t t+1

t+2t-1

t-2t-3

t-4

Camera takes a sequence of images (frames)indexed by time t


Motion (Displacement) Fields

Time t

Time t+1

Displacement field


Motion Field vs Optic Flow

Motion Field: projection of 3D relative velocity vectors onto the 2D image plane.

Optic Flow: observed 2D displacements of brightness patterns in the image.

Motion field is what we want to know.Optic flow is what we can estimate.


Motion Field vs Optic FlowSometimes optic flow is a good approximationto the unknown motion flow.

optic flow field

We can then infer relative motion between the camera and objects in the world.


Warning: Optic Flow ≠ Motion Field

Consider a moving light source:

MF = 0 since the points on the scene are not movingOF ≠ 0 since there is a moving pattern in the images


Motion Field

We are going to derive an equation relating 3D scene structure and velocity to the 2D motion flow field.


Motion Field

What is a Field anyways?

Image a vector at each point in space. This is avector field. In 3D space, we will look at the fieldof 3D velocity vectors induced by camera motion.

In 2D, we will be looking at the projections of those3D vectors in the image. There will be a 2D flow vector at each point in the image. This is the 2Dmotion flow field.


Recall : General Projection Equation

Relative R,TProjectionInternalparams

Strategy:1) Assume internal params known (set to identity)2) At time t, set R=I, T=03) At time t+1, movement generates a relative R and T simplifying assumption: small motion --> small rotation

4) Compute 3D velocity vector 5) Compute 2D velocity vector a function of X,Y,Z,R,T,f


Displacement of 3D World Point

Time t: 3D position PTime t+1: 3D position RP+T

3D Displacement = RP+T - P

Now consider short time period (like time betweentwo video frames = 1/30 sec). Can assume a smallrotation angle in that amount of time. Make a smallangle approximation and rewrite displacement. In thelimit (infinitesimal time period), we will get a velocity.

CSE486, Penn StateRobert Collins Write Rotation matrix in terms of Euler Angles


Small Angle Approx

for small angle x

xx ≈sin

1cos ≈x

x

0sinsin ≈yx


3D Velocity

Under small angle approx,displacement = RP+T - P = (I+S)P+T-P = SP+T

Note: SP = [θx, θy, θz]T X P

In limit, displacement becomes a velocity

V = T + ω X P

3D velocity

linear velocityangular velocity

where

Caution: Abuse of notation ==> we are re-using T as a velocity


3D Relative Velocity

ff

PPVV

ppvv

ZZ zz

The relative velocity of P The relative velocity of P wrt wrt camera:camera:

Camps, PSU

Note: we change signs of the velocity to beconsistent with the book. This should notcause concern. It just depends on whetheryou want to think of the motion as being dueto the camera or the scene.



ff

PPVV

ppvv

ZZ zz

The relative velocity of P The relative velocity of P wrt wrt camera:camera:

Camps, PSU



ff

PPVV

ppvv

ZZ zz

Where are we in this lecture?

• We just derived an equationrelating R,T and to the 3D velocity vector at each scenepoint.

• We can think of that velocity as a little vector inthe scene.

• Now ask, what does the projection of that vectorlook like in the image? It is a 2D vector. It is oneof the vectors that make up the Motion Field!


Motion Field: the 2D velocity of p

ff

PPVV

ppvv

ZZ zz

Taking derivative Taking derivative wrt wrt time:time:

Camps, PSU

Perspectiveprojection


Motion Field: the velocity of p

ff

PPVV

ppvv

ZZ zz

Camps, PSU



ff

PPVV

ppvv

ZZ zz

Camps, PSUTHIS IS THE EQUATION WE WANT!!!!



ff

PPVV

ppvv

ZZ zz

TranslationalTranslationalcomponentcomponent

Camps, PSU



ff

PPVV

ppvv

ZZ zz

RotationalRotationalcomponentcomponent

NOTE: The rotational component is independent of depth Z !NOTE: The rotational component is independent of depth Z !

Camps, PSU


Special Case I: Pure Translation

Assume Assume TTzz ≠≠ 0 0

Define:Define:

Camps, PSU



What if What if TTzz == 0 ? 0 ?

All motion field vectors are parallel to each other andAll motion field vectors are parallel to each other andinversely proportional to depth !inversely proportional to depth !

Camps, PSUTIE IN WITH SIMPLE STEREO!



The motion field in this case is RADIAL:The motion field in this case is RADIAL:••It consists of vectors passing through pIt consists of vectors passing through poo = (x = (xoo,,yyoo))••If:If:

•• TTzz > 0, > 0, (camera moving towards object)(camera moving towards object)

•• the vectors point away from p the vectors point away from poo

••ppoo is the is the POINT OF EXPANSIONPOINT OF EXPANSION•• TTzz < 0, < 0, (camera moving away from object)(camera moving away from object)

••the vectors point towards pthe vectors point towards poo

••ppoo is the is the POINT OF CONTRACTIONPOINT OF CONTRACTION

TTzz > 0> 0 TTzz < 0< 0

Camps, PSU

CSE486, Penn StateRobert Collins Pure Translation:

Properties of the MF

• If Tz≠ 0 the MF is RADIAL with all vectorspointing towards (or away from) a singlepoint po. If Tz = 0 the MF is PARALLEL.

• The length of the MF vectors is inverselyproportional to depth Z. If Tz ≠ 0 it is alsodirectly proportional to the distancebetween p and po.

Camps, PSU

CSE486, Penn StateRobert Collins Pure Translation:

Properties of the MF

• po is the vanishing point of the direction oftranslation.

• po is the intersection of the ray parallel tothe translation vector and the image plane.

Camps, PSU


Special Case II: Moving Plane

ff

PPVV

ppvv

ZZ zz

nnPlanar surfaces are commonPlanar surfaces are commonin man-made environmentsin man-made environments

Question: How does the MF of a moving plane look like?Question: How does the MF of a moving plane look like?

Camps, PSU

Self-study



Points on the plane must satisfy thePoints on the plane must satisfy theequation describing the plane.equation describing the plane.

LetLet••nn be the unit vector normal to the plane. be the unit vector normal to the plane.••dd be the distance from the plane to the be the distance from the plane to theorigin.origin.••NOTE:NOTE: If the plane is moving If the plane is moving wrtwrtcamera, camera, nn and and dd are functions of timeare functions of time..

Then:Then:

dd PP

ZZ

nn

OOYY

XX

wherewhere

Camps, PSU

Self-study



Camps, PSU

dd PP

ZZ

nn

OOYY

XX

Using the pinholeUsing the pinholeprojection equation:projection equation:pp

LetLet be the image of Pbe the image of P

Using the plane equation:Using the plane equation:

Solving for Z:Solving for Z:

Self-study



And Plug in:And Plug in:

dd PP

ZZ

nn

OOYY

XX

pp

Now consider the MF equations:Now consider the MF equations:

Camps, PSU

Self-study



dd PP

ZZ

nn

OOYY

XX

pp

The MF equations become:The MF equations become:

wherewhere

Camps, PSU

Self-study



dd PP

ZZ

nn

OOYY

XX

pp

MF equations: MF equations:

Q: What is the significance of this?Q: What is the significance of this?

A: The MF vectors are given by low order (second)A: The MF vectors are given by low order (second)polynomials.polynomials.

••Their Their coeffscoeffs. a. a11 to a to a8 8 (only 8 !) are functions of n, d, T and (only 8 !) are functions of n, d, T and ωω..••That is, can estimate 8That is, can estimate 8 param param global flow rather than 2global flow rather than 2 params params per pixel! Huge savings in time and robustness.per pixel! Huge savings in time and robustness.

Camps, PSU

Self-study


Application: Time to Collision

ff

LL

vv

LL

DDoo

l(t)l(t)

An object of heightAn object of height L L moves with moves withconstant velocity constant velocity v:v:••At time At time t=0t=0 the object is at: the object is at:

•• D(0) = DD(0) = Doo

••At time At time tt it is at it is at••D(t) = DD(t) = Doo –– vt vt

••It will crash into the camera atIt will crash into the camera attime:time:

•• D(D(ττ) = D) = Doo –– v vττ = 0 = 0•• ττ = D = Doo/v/v

t=0t=0tt

D(t)D(t)

Camps, PSU

Self-study


Time to Collision

ff

LL

vv

LL

DDoo

l(t)l(t)

t=0t=0tt

D(t)D(t)

The image of the object has size l(t):The image of the object has size l(t):

Taking derivative Taking derivative wrt wrt time:time:

Camps, PSU

Self-study


Time to Collision

ff

LL

vv

LL

DDoo

l(t)l(t)

t=0t=0tt

D(t)D(t)

Camps, PSU

Self-study


Time to Collision

Camps, PSU

ff

LL

vv

LL

DDoo

l(t)l(t)

t=0t=0tt

D(t)D(t)

And their ratio is:And their ratio is:

Self-study


Time to Collision

Can beCan bedirectlydirectlymeasuredmeasuredfrom imagefrom image

ff

LL

vv

LL

DDoo

l(t)l(t)

t=0t=0tt

D(t)D(t)

And And time to collisiontime to collision::

Can be found, without knowing Can be found, without knowing LL or or DDoo or or vv !! !!

Camps, PSU

Self-study


Application: Decomposition of Vehicle Flow Fieldinto Rotational and Translational Components

Motivation•Estimate steering angles•Computation of scene structure simplifies when rotational motion is removed


Review: Flow Due to Self-Motion

Rotation Translation

Flow:

Rotation component Translation component

Note: I’ve changed sign again!


Approach

Use prior knowledge of car’s likely motion to simplify the problem.

Assume we know approx direction of translation (Tx,Ty,Tz)

[ e.g. if driving forward, we choose (0,0,1) ]

Assume roll angle of motion (wz) is 0

Problem reduces to solving for pitch and yaw angles (wx and wy)

0known


Approach (continued)

For each observed flow vector, form a linear constraint on wx and wy

by taking the dot product of the flow equation with a new vector that

is constructed to “annihilate” the translation component of flow.

This annihilation vector is

Verify:

0

Note: these are allknown values!


Approach (continued)

For each observed flow vector, we annihilate the translation component

to form one linear constraint of the form (ai wx + bi wy + ci ) = 0.

Given the whole set of n observed flow vectors, we seek wx and wy

that simultaneously satisfy the set of n linear equations

Geometric intuition:

image space solution space

wx

wy


Use Robust Estimator

robust solution

least-squares solution

red: outliers blue: inliers

Solving using least-squares will be sensitive to outliers (grossly incorrect data).Instead, we use a robust estimator (Random Sample Consensus – RANSAC).

Lecture 22: Camera Motionrtc12/CSE486/lecture22.pdf · Lecture 22: Camera Motion Readings: ......

Documents

Transcript of Lecture 22: Camera Motionrtc12/CSE486/lecture22.pdf · Lecture 22: Camera Motion Readings: ......