Tracking

Definition of Tracking

• Tracking: – Generate some conclusions about the motion of

the scene, objects, or the camera, given a sequence of images.

– Knowing this motion, predict where things are going to project in the next image, so that we don’t have so much work looking for them.

Why Track?• Detection and

recognition are expensive• If we get an idea of

where an object is in the image because we have an idea of the motion from previous images, we need less work detecting or recognizing the object.

Scene

Image Sequence

Detection+ recognition

Tracking(less detection,no recognition) New

person

A B

C

A B

C

Tracking a Silhouette by Measuring Edge Positions

• Observations are positions of edges along normals to tracked contour

Why not Wait and Process the Set of Images as a Batch?

• In a car system, detecting and tracking pedestrians in real time is important.

• Recursive methods require less computing

Implicit Assumptions of Tracking

• Physical cameras do not move instantly from a viewpoint to another

• Object do not teleport between places around the scene

• Relative position between camera and scene changes incrementally

• We can model motion

Related Fields

• Signal Detection and Estimation• Radar technology

The Problem: Signal Estimation

• We have a system with parameters– Scene structure, camera motion, automatic zoom– System state is unknown (“hidden”)

• We have measurements– Components of stable “feature points” in the

images.– “Observations”, projections of the state.

• We want to recover the state components from the observations

Necessary Models

System DynamicsModel

MeasurementModel (projection)

(u, v)

Previous StateNext State

State Measurement

• We use models to describe a priori knowledge about • the world (including external parameters of camera)• the imaging projection process

State variablea

A Simple Example of Estimationby Least Square Method

• Goal: Find estimate of state such that the least square error between measurements and the state is minimum

a

∑=

−=n

ii axC

1

2)(21

∑∑==

−=−==∂∂ n

ii

n

ii anxax

aC

11

ˆ)ˆ(0

∑=

=n

iix

na

1

1ˆ

a Measurementx

x

txi

a xi-a

ti

Recursive Least Square Estimation• We don’t want to wait until

all data have been collected to get an estimate of the depth

• We don’t want to reprocess old data when we make a new measurement

• Recursive method: data at step i are obtained from data at step i-1

a

State variablea

Measurementx

x

t

axi

1ˆ −ia ia

Recursive Least Square Estimation 2

• Recursive method: data at step i are obtained from data at step i-1

x

t

axi

)ˆ(1ˆˆ 11 −− −+= iiii axi

aa

i

i

kk

i

kki x

ix

ix

ia

111ˆ1

11∑∑

−

==

+==

∑−

=− −

=1

11 1

1ˆi

kki x

ia

iii xi

ai

ia

1ˆ1ˆ 1 +−

= −

Recursive Least Square Estimation 3

)ˆ(1ˆˆ 11 −− −+= iiii axi

aaEstimate at step i

Predicted measure

Innovation

Gain Actual measure

Gain specifies how much do we pay attentionto the differencebetween what we expectedand what we actually get

Least Square Estimation of the State Vector of a Static System

1. Batch method

a

H1 Hi

H2

x1

xi

x2

aH ⋅= iix a

H

HH

=

⇒

nnx

xx

......2

1

2

1

aHx =⇒

measurement equation

XHH)(Ha T1T −=ˆ

Find estimate that minimizes aa)H(Xa)H(X T −−=

21

C

We find

Least Square Estimation of the State Vector

of a Static System 22. Recursive method

a

H1 Hi

H2

x1

xi

x2

)ˆˆˆ -1iiii-1ii aH(XKaa −+=

with

Calculation is similar to calculation of running average

Now we find:

We had: )ˆ(1ˆˆ 11 −− −+= iiii axi

aa

1)( −=

=T

iii

Tiii

HHP

HPKGain matrix

Innovation

Predicted measure

Dynamic System

i

i

i

i

AVX

a

ixwAA

tAVV

tVXX

ii

iii

iii

+=∆+=∆+=

−

−−

−−

1

11

11

+

∆

∆=

−

−

−

wAVX

tt

AVX

i

i

i

i

i

i

00

10010

01

1

1

1

State of rocket

Measurement

waa -1ii +Φ=⇒

[ ] VAVX

x

i

i

i

i +

= 001 Vxi +=⇒ iaH

State equation for rocket

Measurement equation

Noise

Tweak factor

)ˆˆˆ -1iiiii-1iii aH(xKaa Φ−+Φ=

Recursive Least Square Estimation for a Dynamic System

(Kalman Filter)-1i-1iii waa +Φ=

iiii vaHx +=

),0(~ ii Qw N

),0(~ ii Rv N

-1i-1i-1i-1i

-1iTi-1iii

-1i

Tiii

Tiii

)P'HK(IPQPP'

RHP'HHP'K

−=+ΦΦ=

+= )(

State equation

Measurement equation

Tweak factor for model

Measurement noise

Prediction for xi

Prediction for ai

Gain

Covariance matrix for prediction error

Covariance for estimation error

-1i-1i-1i-1i

-1i

T

i-1i

ii

-1i

Tiii

Tiii

)P'HKIP

Qaf

Paf

P'

RHP'HHP'K

−=

+∂∂

∂∂

=

+=

(

)(

))ˆ)ˆ(ˆ -1iiii-1ii af(H(xKafa −+=

Estimation when System Model is Nonlinear

(Extended Kalman Filter)

-1i-1ii wafa += )( iiii vaHx +=State equation Measurement equation

Jacobian Matrix

Differences compared toregular Kalman filter are circled in red

• Predict next state as using previous step and dynamic model

• Predict regions of next measurements using measurement model and uncertainties

• Make new measurements xi in predicted regions

• Compute best estimate of next state

Tracking Steps

(u, v)Measurement

Predictionregion

-1iiaΦ

),ˆ( i-1iii P'aH ΦN

)ˆˆˆ -1iiiii-1iii aH(xKaa Φ−+Φ=“Correction” of predicted state

Recursive Least Square Estimation for a Dynamic System (Kalman Filter)

x

t

Measurement xi

State vector aiEstimation ia

Tracking as a Probabilistic Inference Problem

• Find distributions for state vector ai and for measurement vector xi. Then we are able to compute the expectations and

• Simplifying assumptions (same as for HMM) ix

)|(),,,|( -1ii-1i21i aaaaaa PP =L

)|()|()|()|,,,( ikijiiiji axaxaxaxx PPPP KK =(Conditional independence ofmeasurements given a state)

(Only immediate past matters)

ia

Tracking as Inference

• Prediction

• Correction

• Produces same results as least square approach if distributions are Gaussians: Kalman filter

• See Forsyth and Ponce, Ch. 19

∫= -1i-1i1-1i-1ii-1i1i axxaaaxxa dPPP ),,|()|(),,|( LL

∫=

i-1i1iii

-1i1iiii1i

axxaaxxxaax

xxadPP

PPP

),,|()|(),,|()|(

),,|(LL

L

)ˆ(ˆˆ 11 i-iii-i afhxKafa −+=

Kalman Filter for 1D Signals

11 i-i-i wafa +=

iii vahx +=

)0(~ ,qNwi

)0(~ ,rNvi

111

12

12

')1(

)(

i-ii-

ii

-iii

phKp

qpfp'

rp'hhp'K

−

−

−=+=

+=

State equation

Measurement equation

Tweak factor for model

Measurement noise

Prediction for xi

Prediction for ai (a priori estimate)

Gain

Standard deviation for prediction error

St.d. for estimation error

Applications: Structure from Motion • Measurement vector components:

– Coordinates of corners, “salient points”

• State vector components: – Camera motion parameters – Scene structure

• Is there enough equations?– N corners, 2N measurements– N unknown state components from structure

(distances from first center of projection to 3D points)

– 6 unknown state components from motion (translation and rotation)

– More measurements than unknowns for every frame if N>6 (2N > N + 6)

xi

• Batch methods• Recursive methods

(Kalman filter)

Problems with Tracking

• Initial detection– If it is too slow we will never catch up– If it is fast, why not do detection at every frame?

• Even if raw detection can be done in real time, tracking saves processing cycles compared to raw detection. The CPU has other things to do.

• Detection is needed again if you lose tracking• Most vision tracking prototypes use initial

detection done by hand (see Forsyth and Ponce for discussion)

References

• Kalman, R.E., “A New Approach to Linear Prediction Problems”, Transactions of the ASME--Journal of Basic Engineering, pp. 35-45, March 1960.

• Sorenson, H.W., “Least Squares Estimation: from Gauss to Kalman”, IEEE Spectrum, vol. 7, pp. 63-68, July 1970.

• http://www.cs.unc.edu/~welch/kalmanLinks.html• D. Forsyth and J. Ponce. Computer Vision: A Modern

Approach, Chapter 19. http://www.cs.berkeley.edu/~daf/book3chaps.html

• O. Faugeras.. Three-Dimensional Computer Vision. MIT Press. Ch. 8, “Tracking Tokens over Time”.