Post on 07-Aug-2015
Definition of Tracking
• Tracking: – Generate some conclusions about the motion of
the scene, objects, or the camera, given a sequence of images.
– Knowing this motion, predict where things are going to project in the next image, so that we don’t have so much work looking for them.
Why Track?• Detection and
recognition are expensive• If we get an idea of
where an object is in the image because we have an idea of the motion from previous images, we need less work detecting or recognizing the object.
Scene
Image Sequence
Detection+ recognition
Tracking(less detection,no recognition) New
person
A B
C
A B
C
Tracking a Silhouette by Measuring Edge Positions
• Observations are positions of edges along normals to tracked contour
Why not Wait and Process the Set of Images as a Batch?
• In a car system, detecting and tracking pedestrians in real time is important.
• Recursive methods require less computing
Implicit Assumptions of Tracking
• Physical cameras do not move instantly from a viewpoint to another
• Object do not teleport between places around the scene
• Relative position between camera and scene changes incrementally
• We can model motion
The Problem: Signal Estimation
• We have a system with parameters– Scene structure, camera motion, automatic zoom– System state is unknown (“hidden”)
• We have measurements– Components of stable “feature points” in the
images.– “Observations”, projections of the state.
• We want to recover the state components from the observations
Necessary Models
System DynamicsModel
MeasurementModel (projection)
(u, v)
Previous StateNext State
State Measurement
• We use models to describe a priori knowledge about • the world (including external parameters of camera)• the imaging projection process
State variablea
A Simple Example of Estimationby Least Square Method
• Goal: Find estimate of state such that the least square error between measurements and the state is minimum
a
∑=
−=n
ii axC
1
2)(21
∑∑==
−=−==∂∂ n
ii
n
ii anxax
aC
11
ˆ)ˆ(0
∑=
=n
iix
na
1
1ˆ
a Measurementx
x
txi
a xi-a
ti
Recursive Least Square Estimation• We don’t want to wait until
all data have been collected to get an estimate of the depth
• We don’t want to reprocess old data when we make a new measurement
• Recursive method: data at step i are obtained from data at step i-1
a
State variablea
Measurementx
x
t
axi
1ˆ −ia ia
Recursive Least Square Estimation 2
• Recursive method: data at step i are obtained from data at step i-1
x
t
axi
)ˆ(1ˆˆ 11 −− −+= iiii axi
aa
i
i
kk
i
kki x
ix
ix
ia
111ˆ1
11∑∑
−
==
+==
∑−
=− −
=1
11 1
1ˆi
kki x
ia
iii xi
ai
ia
1ˆ1ˆ 1 +−
= −
Recursive Least Square Estimation 3
)ˆ(1ˆˆ 11 −− −+= iiii axi
aaEstimate at step i
Predicted measure
Innovation
Gain Actual measure
Gain specifies how much do we pay attentionto the differencebetween what we expectedand what we actually get
Least Square Estimation of the State Vector of a Static System
1. Batch method
a
H1 Hi
H2
x1
xi
x2
aH ⋅= iix a
H
HH
=
⇒
nnx
xx
......2
1
2
1
aHx =⇒
measurement equation
XHH)(Ha T1T −=ˆ
Find estimate that minimizes aa)H(Xa)H(X T −−=
21
C
We find
Least Square Estimation of the State Vector
of a Static System 22. Recursive method
a
H1 Hi
H2
x1
xi
x2
)ˆˆˆ -1iiii-1ii aH(XKaa −+=
with
Calculation is similar to calculation of running average
Now we find:
We had: )ˆ(1ˆˆ 11 −− −+= iiii axi
aa
1)( −=
=T
iii
Tiii
HHP
HPKGain matrix
Innovation
Predicted measure
Dynamic System
i
i
i
i
AVX
a
ixwAA
tAVV
tVXX
ii
iii
iii
+=∆+=∆+=
−
−−
−−
1
11
11
+
∆
∆=
−
−
−
wAVX
tt
AVX
i
i
i
i
i
i
00
10010
01
1
1
1
State of rocket
Measurement
waa -1ii +Φ=⇒
[ ] VAVX
x
i
i
i
i +
= 001 Vxi +=⇒ iaH
State equation for rocket
Measurement equation
Noise
Tweak factor
)ˆˆˆ -1iiiii-1iii aH(xKaa Φ−+Φ=
Recursive Least Square Estimation for a Dynamic System
(Kalman Filter)-1i-1iii waa +Φ=
iiii vaHx +=
),0(~ ii Qw N
),0(~ ii Rv N
-1i-1i-1i-1i
-1iTi-1iii
-1i
Tiii
Tiii
)P'HK(IPQPP'
RHP'HHP'K
−=+ΦΦ=
+= )(
State equation
Measurement equation
Tweak factor for model
Measurement noise
Prediction for xi
Prediction for ai
Gain
Covariance matrix for prediction error
Covariance for estimation error
-1i-1i-1i-1i
-1i
T
i-1i
ii
-1i
Tiii
Tiii
)P'HKIP
Qaf
Paf
P'
RHP'HHP'K
−=
+∂∂
∂∂
=
+=
(
)(
))ˆ)ˆ(ˆ -1iiii-1ii af(H(xKafa −+=
Estimation when System Model is Nonlinear
(Extended Kalman Filter)
-1i-1ii wafa += )( iiii vaHx +=State equation Measurement equation
Jacobian Matrix
Differences compared toregular Kalman filter are circled in red
• Predict next state as using previous step and dynamic model
• Predict regions of next measurements using measurement model and uncertainties
• Make new measurements xi in predicted regions
• Compute best estimate of next state
Tracking Steps
(u, v)Measurement
Predictionregion
-1iiaΦ
),ˆ( i-1iii P'aH ΦN
)ˆˆˆ -1iiiii-1iii aH(xKaa Φ−+Φ=“Correction” of predicted state
Recursive Least Square Estimation for a Dynamic System (Kalman Filter)
x
t
Measurement xi
State vector aiEstimation ia
Tracking as a Probabilistic Inference Problem
• Find distributions for state vector ai and for measurement vector xi. Then we are able to compute the expectations and
• Simplifying assumptions (same as for HMM) ix
)|(),,,|( -1ii-1i21i aaaaaa PP =L
)|()|()|()|,,,( ikijiiiji axaxaxaxx PPPP KK =(Conditional independence ofmeasurements given a state)
(Only immediate past matters)
ia
Tracking as Inference
• Prediction
• Correction
• Produces same results as least square approach if distributions are Gaussians: Kalman filter
• See Forsyth and Ponce, Ch. 19
∫= -1i-1i1-1i-1ii-1i1i axxaaaxxa dPPP ),,|()|(),,|( LL
∫=
i-1i1iii
-1i1iiii1i
axxaaxxxaax
xxadPP
PPP
),,|()|(),,|()|(
),,|(LL
L
)ˆ(ˆˆ 11 i-iii-i afhxKafa −+=
Kalman Filter for 1D Signals
11 i-i-i wafa +=
iii vahx +=
)0(~ ,qNwi
)0(~ ,rNvi
111
12
12
')1(
)(
i-ii-
ii
-iii
phKp
qpfp'
rp'hhp'K
−
−
−=+=
+=
State equation
Measurement equation
Tweak factor for model
Measurement noise
Prediction for xi
Prediction for ai (a priori estimate)
Gain
Standard deviation for prediction error
St.d. for estimation error
Applications: Structure from Motion • Measurement vector components:
– Coordinates of corners, “salient points”
• State vector components: – Camera motion parameters – Scene structure
• Is there enough equations?– N corners, 2N measurements– N unknown state components from structure
(distances from first center of projection to 3D points)
– 6 unknown state components from motion (translation and rotation)
– More measurements than unknowns for every frame if N>6 (2N > N + 6)
xi
• Batch methods• Recursive methods
(Kalman filter)
Problems with Tracking
• Initial detection– If it is too slow we will never catch up– If it is fast, why not do detection at every frame?
• Even if raw detection can be done in real time, tracking saves processing cycles compared to raw detection. The CPU has other things to do.
• Detection is needed again if you lose tracking• Most vision tracking prototypes use initial
detection done by hand (see Forsyth and Ponce for discussion)
References
• Kalman, R.E., “A New Approach to Linear Prediction Problems”, Transactions of the ASME--Journal of Basic Engineering, pp. 35-45, March 1960.
• Sorenson, H.W., “Least Squares Estimation: from Gauss to Kalman”, IEEE Spectrum, vol. 7, pp. 63-68, July 1970.
• http://www.cs.unc.edu/~welch/kalmanLinks.html• D. Forsyth and J. Ponce. Computer Vision: A Modern
Approach, Chapter 19. http://www.cs.berkeley.edu/~daf/book3chaps.html
• O. Faugeras.. Three-Dimensional Computer Vision. MIT Press. Ch. 8, “Tracking Tokens over Time”.