CSCE643: Computer Vision Mean-Shift Object Tracking Jinxiang Chai Many slides from Yaron Ukrainitz &...
-
Upload
roger-bridgett -
Category
Documents
-
view
221 -
download
0
Transcript of CSCE643: Computer Vision Mean-Shift Object Tracking Jinxiang Chai Many slides from Yaron Ukrainitz &...
![Page 1: CSCE643: Computer Vision Mean-Shift Object Tracking Jinxiang Chai Many slides from Yaron Ukrainitz & Bernard Sarel & Robert Collins.](https://reader035.fdocuments.in/reader035/viewer/2022062223/551b51d65503465c7e8b5ab0/html5/thumbnails/1.jpg)
CSCE643: Computer VisionMean-Shift Object Tracking
Jinxiang Chai
Many slides from Yaron Ukrainitz & Bernard Sarel & Robert Collins
![Page 2: CSCE643: Computer Vision Mean-Shift Object Tracking Jinxiang Chai Many slides from Yaron Ukrainitz & Bernard Sarel & Robert Collins.](https://reader035.fdocuments.in/reader035/viewer/2022062223/551b51d65503465c7e8b5ab0/html5/thumbnails/2.jpg)
Appearance-based Tracking
Slide from Collins
![Page 3: CSCE643: Computer Vision Mean-Shift Object Tracking Jinxiang Chai Many slides from Yaron Ukrainitz & Bernard Sarel & Robert Collins.](https://reader035.fdocuments.in/reader035/viewer/2022062223/551b51d65503465c7e8b5ab0/html5/thumbnails/3.jpg)
Review: Lucas-Kanade Tracking/Registration
• Key Idea #1: Formulate the tracking/registration as a nonlinear least square minimization problem
x
pxHpxwI 2)());((minarg
![Page 4: CSCE643: Computer Vision Mean-Shift Object Tracking Jinxiang Chai Many slides from Yaron Ukrainitz & Bernard Sarel & Robert Collins.](https://reader035.fdocuments.in/reader035/viewer/2022062223/551b51d65503465c7e8b5ab0/html5/thumbnails/4.jpg)
Review: Lucas-Kanade Tracking/Registration
• Key Idea #2: Solve the problem with Gauss-Newton optimization techniques
x
pxHpxwI 2)());((minarg
![Page 5: CSCE643: Computer Vision Mean-Shift Object Tracking Jinxiang Chai Many slides from Yaron Ukrainitz & Bernard Sarel & Robert Collins.](https://reader035.fdocuments.in/reader035/viewer/2022062223/551b51d65503465c7e8b5ab0/html5/thumbnails/5.jpg)
Review: Gauss-Newton Optimization
Rearrange
xp
xHpp
WIpxwI
2
)());((minarg
![Page 6: CSCE643: Computer Vision Mean-Shift Object Tracking Jinxiang Chai Many slides from Yaron Ukrainitz & Bernard Sarel & Robert Collins.](https://reader035.fdocuments.in/reader035/viewer/2022062223/551b51d65503465c7e8b5ab0/html5/thumbnails/6.jpg)
xp
pxwIxHpp
WI
2
)));(()((minarg
Rearrange
xp
xHpp
WIpxwI
2
)());((minarg
Review: Gauss-Newton Optimization
![Page 7: CSCE643: Computer Vision Mean-Shift Object Tracking Jinxiang Chai Many slides from Yaron Ukrainitz & Bernard Sarel & Robert Collins.](https://reader035.fdocuments.in/reader035/viewer/2022062223/551b51d65503465c7e8b5ab0/html5/thumbnails/7.jpg)
Review:Gauss-Newton Optimization
xp
pxwIxHpp
WI
2
)));(()((minarg
Rearrange
A
xp
xHpp
WIpxwI
2
)());((minarg
b
![Page 8: CSCE643: Computer Vision Mean-Shift Object Tracking Jinxiang Chai Many slides from Yaron Ukrainitz & Bernard Sarel & Robert Collins.](https://reader035.fdocuments.in/reader035/viewer/2022062223/551b51d65503465c7e8b5ab0/html5/thumbnails/8.jpg)
Review:Gauss-Newton Optimization
xp
pxwIxHpp
WI
2
)));(()((minarg
)));(()((()( 1 pxwIxHp
WI
p
WI
p
WIp
T
xx
T
Rearrange
A
ATb
xp
xHpp
WIpxwI
2
)());((minarg
b
(ATA)-1
![Page 9: CSCE643: Computer Vision Mean-Shift Object Tracking Jinxiang Chai Many slides from Yaron Ukrainitz & Bernard Sarel & Robert Collins.](https://reader035.fdocuments.in/reader035/viewer/2022062223/551b51d65503465c7e8b5ab0/html5/thumbnails/9.jpg)
Lucas-Kanade Registration
Initialize p=p0:
Iterate: 1. Warp I with w(x;p) to compute I(w(x;p))
));(()(()( 1 pxwIxHp
WI
p
WI
p
WIp
T
xx
T
![Page 10: CSCE643: Computer Vision Mean-Shift Object Tracking Jinxiang Chai Many slides from Yaron Ukrainitz & Bernard Sarel & Robert Collins.](https://reader035.fdocuments.in/reader035/viewer/2022062223/551b51d65503465c7e8b5ab0/html5/thumbnails/10.jpg)
Lucas-Kanade Registration
Initialize p=p0:
Iterate: 1. Warp I with w(x;p) to compute I(w(x;p))
2. Compute the error image
));(()(()( 1 pxwIxHp
WI
p
WI
p
WIp
T
xx
T
![Page 11: CSCE643: Computer Vision Mean-Shift Object Tracking Jinxiang Chai Many slides from Yaron Ukrainitz & Bernard Sarel & Robert Collins.](https://reader035.fdocuments.in/reader035/viewer/2022062223/551b51d65503465c7e8b5ab0/html5/thumbnails/11.jpg)
Lucas-Kanade Registration
Initialize p=p0:
Iterate: 1. Warp I with w(x;p) to compute I(w(x;p))
2. Compute the error image
3. Warp the gradient with w(x;p)
));(()(()( 1 pxwIxHp
WI
p
WI
p
WIp
T
xx
T
I
![Page 12: CSCE643: Computer Vision Mean-Shift Object Tracking Jinxiang Chai Many slides from Yaron Ukrainitz & Bernard Sarel & Robert Collins.](https://reader035.fdocuments.in/reader035/viewer/2022062223/551b51d65503465c7e8b5ab0/html5/thumbnails/12.jpg)
Lucas-Kanade Registration
Initialize p=p0:
Iterate: 1. Warp I with w(x;p) to compute I(w(x;p))
2. Compute the error image
3. Warp the gradient with w(x;p)
4. Evaluate the Jacobian at (x;p)
));(()(()( 1 pxwIxHp
WI
p
WI
p
WIp
T
xx
T
I
p
w
![Page 13: CSCE643: Computer Vision Mean-Shift Object Tracking Jinxiang Chai Many slides from Yaron Ukrainitz & Bernard Sarel & Robert Collins.](https://reader035.fdocuments.in/reader035/viewer/2022062223/551b51d65503465c7e8b5ab0/html5/thumbnails/13.jpg)
Lucas-Kanade Registration
Initialize p=p0:
Iterate: 1. Warp I with w(x;p) to compute I(w(x;p))
2. Compute the error image
3. Warp the gradient with w(x;p)
4. Evaluate the Jacobian at (x;p)
5. Compute the using linear system solvers
));(()(()( 1 pxwIxHp
WI
p
WI
p
WIp
T
xx
T
I
p
w
p
![Page 14: CSCE643: Computer Vision Mean-Shift Object Tracking Jinxiang Chai Many slides from Yaron Ukrainitz & Bernard Sarel & Robert Collins.](https://reader035.fdocuments.in/reader035/viewer/2022062223/551b51d65503465c7e8b5ab0/html5/thumbnails/14.jpg)
Lucas-Kanade Registration
Initialize p=p0:
Iterate: 1. Warp I with w(x;p) to compute I(w(x;p))
2. Compute the error image
3. Warp the gradient with w(x;p)
4. Evaluate the Jacobian at (x;p)
5. Compute the using linear system solvers
6. Update the parameters
));(()(()( 1 pxwIxHp
WI
p
WI
p
WIp
T
xx
T
I
p
w
p
ppp
![Page 15: CSCE643: Computer Vision Mean-Shift Object Tracking Jinxiang Chai Many slides from Yaron Ukrainitz & Bernard Sarel & Robert Collins.](https://reader035.fdocuments.in/reader035/viewer/2022062223/551b51d65503465c7e8b5ab0/html5/thumbnails/15.jpg)
Object Tracking
• Can we apply Lucas-Kanade techniques to non-rigid objects (e.g., a walking person)?
![Page 16: CSCE643: Computer Vision Mean-Shift Object Tracking Jinxiang Chai Many slides from Yaron Ukrainitz & Bernard Sarel & Robert Collins.](https://reader035.fdocuments.in/reader035/viewer/2022062223/551b51d65503465c7e8b5ab0/html5/thumbnails/16.jpg)
Motivation of Mean Shift Tracking
• To track non-rigid objects (e.g., a walking person), it is hard to specify an explicit 2D parametric motion model.
• Appearances of non-rigid objects can sometimes be modeled with color distributions
![Page 17: CSCE643: Computer Vision Mean-Shift Object Tracking Jinxiang Chai Many slides from Yaron Ukrainitz & Bernard Sarel & Robert Collins.](https://reader035.fdocuments.in/reader035/viewer/2022062223/551b51d65503465c7e8b5ab0/html5/thumbnails/17.jpg)
Mean-Shift Tracking
The mean-shift algorithm is an efficient approach to tracking objects whose appearance is defined by histograms.
- not limited to only color, however.
- Could also use edge orientation, texture motion
Slide from Robert Collins
![Page 18: CSCE643: Computer Vision Mean-Shift Object Tracking Jinxiang Chai Many slides from Yaron Ukrainitz & Bernard Sarel & Robert Collins.](https://reader035.fdocuments.in/reader035/viewer/2022062223/551b51d65503465c7e8b5ab0/html5/thumbnails/18.jpg)
Mean Shift Tracking: Demos
![Page 19: CSCE643: Computer Vision Mean-Shift Object Tracking Jinxiang Chai Many slides from Yaron Ukrainitz & Bernard Sarel & Robert Collins.](https://reader035.fdocuments.in/reader035/viewer/2022062223/551b51d65503465c7e8b5ab0/html5/thumbnails/19.jpg)
Mean Shift
Mean Shift [Che98, FH75, Sil86]
- An algorithm that iteratively shifts a data point to the average of data points in its neighborhood.
- Similar to clustering.
- Useful for clustering, mode seeking, probability density estimation, tracking, etc.
![Page 20: CSCE643: Computer Vision Mean-Shift Object Tracking Jinxiang Chai Many slides from Yaron Ukrainitz & Bernard Sarel & Robert Collins.](https://reader035.fdocuments.in/reader035/viewer/2022062223/551b51d65503465c7e8b5ab0/html5/thumbnails/20.jpg)
Mean Shift Reference
• Y. Cheng. Mean shift, mode seeking, and clustering. IEEE Trans. on Pattern Analysis and Machine Intelligence, 17(8):790–799, 1998.
• K. Fukunaga and L. D. Hostetler. The estimation of the gradient of a density function, with applications in pattern recognition. IEEE Trans. on Information Theory, 21:32–40, 1975.
• B. W. Silverman. Density Estimation for Statistics and Data Analysis. Chapman and Hall, 1986.
![Page 21: CSCE643: Computer Vision Mean-Shift Object Tracking Jinxiang Chai Many slides from Yaron Ukrainitz & Bernard Sarel & Robert Collins.](https://reader035.fdocuments.in/reader035/viewer/2022062223/551b51d65503465c7e8b5ab0/html5/thumbnails/21.jpg)
Mean Shift Theory
![Page 22: CSCE643: Computer Vision Mean-Shift Object Tracking Jinxiang Chai Many slides from Yaron Ukrainitz & Bernard Sarel & Robert Collins.](https://reader035.fdocuments.in/reader035/viewer/2022062223/551b51d65503465c7e8b5ab0/html5/thumbnails/22.jpg)
Intuitive Description
Distribution of identical billiard balls
Region ofinterest
Center ofmass
Mean Shiftvector
Objective : Find the densest region
![Page 23: CSCE643: Computer Vision Mean-Shift Object Tracking Jinxiang Chai Many slides from Yaron Ukrainitz & Bernard Sarel & Robert Collins.](https://reader035.fdocuments.in/reader035/viewer/2022062223/551b51d65503465c7e8b5ab0/html5/thumbnails/23.jpg)
Intuitive Description
Distribution of identical billiard balls
Region ofinterest
Center ofmass
Mean Shiftvector
Objective : Find the densest region
![Page 24: CSCE643: Computer Vision Mean-Shift Object Tracking Jinxiang Chai Many slides from Yaron Ukrainitz & Bernard Sarel & Robert Collins.](https://reader035.fdocuments.in/reader035/viewer/2022062223/551b51d65503465c7e8b5ab0/html5/thumbnails/24.jpg)
Intuitive Description
Distribution of identical billiard balls
Region ofinterest
Center ofmass
Mean Shiftvector
Objective : Find the densest region
![Page 25: CSCE643: Computer Vision Mean-Shift Object Tracking Jinxiang Chai Many slides from Yaron Ukrainitz & Bernard Sarel & Robert Collins.](https://reader035.fdocuments.in/reader035/viewer/2022062223/551b51d65503465c7e8b5ab0/html5/thumbnails/25.jpg)
Intuitive Description
Distribution of identical billiard balls
Region ofinterest
Center ofmass
Mean Shiftvector
Objective : Find the densest region
![Page 26: CSCE643: Computer Vision Mean-Shift Object Tracking Jinxiang Chai Many slides from Yaron Ukrainitz & Bernard Sarel & Robert Collins.](https://reader035.fdocuments.in/reader035/viewer/2022062223/551b51d65503465c7e8b5ab0/html5/thumbnails/26.jpg)
Intuitive Description
Distribution of identical billiard balls
Region ofinterest
Center ofmass
Mean Shiftvector
Objective : Find the densest region
![Page 27: CSCE643: Computer Vision Mean-Shift Object Tracking Jinxiang Chai Many slides from Yaron Ukrainitz & Bernard Sarel & Robert Collins.](https://reader035.fdocuments.in/reader035/viewer/2022062223/551b51d65503465c7e8b5ab0/html5/thumbnails/27.jpg)
Intuitive Description
Distribution of identical billiard balls
Region ofinterest
Center ofmass
Mean Shiftvector
Objective : Find the densest region
![Page 28: CSCE643: Computer Vision Mean-Shift Object Tracking Jinxiang Chai Many slides from Yaron Ukrainitz & Bernard Sarel & Robert Collins.](https://reader035.fdocuments.in/reader035/viewer/2022062223/551b51d65503465c7e8b5ab0/html5/thumbnails/28.jpg)
Intuitive Description
Distribution of identical billiard balls
Region ofinterest
Center ofmass
Objective : Find the densest region
![Page 29: CSCE643: Computer Vision Mean-Shift Object Tracking Jinxiang Chai Many slides from Yaron Ukrainitz & Bernard Sarel & Robert Collins.](https://reader035.fdocuments.in/reader035/viewer/2022062223/551b51d65503465c7e8b5ab0/html5/thumbnails/29.jpg)
What is Mean Shift ?
Non-parametricDensity Estimation
Non-parametricDensity GRADIENT Estimation
(Mean Shift)
Data
Discrete PDF Representation
PDF Analysis
PDF in feature space• Color space• Scale space• Actually any feature space you can conceive• …
A tool for:Finding modes in a set of data samples, manifesting an underlying probability density function (PDF) in RN
![Page 30: CSCE643: Computer Vision Mean-Shift Object Tracking Jinxiang Chai Many slides from Yaron Ukrainitz & Bernard Sarel & Robert Collins.](https://reader035.fdocuments.in/reader035/viewer/2022062223/551b51d65503465c7e8b5ab0/html5/thumbnails/30.jpg)
Non-Parametric Density Estimation
Assumption : The data points are sampled from an underlying PDF
Assumed Underlying PDF Real Data Samples
Data point density implies PDF value !
![Page 31: CSCE643: Computer Vision Mean-Shift Object Tracking Jinxiang Chai Many slides from Yaron Ukrainitz & Bernard Sarel & Robert Collins.](https://reader035.fdocuments.in/reader035/viewer/2022062223/551b51d65503465c7e8b5ab0/html5/thumbnails/31.jpg)
Assumed Underlying PDF Real Data Samples
Non-Parametric Density Estimation
![Page 32: CSCE643: Computer Vision Mean-Shift Object Tracking Jinxiang Chai Many slides from Yaron Ukrainitz & Bernard Sarel & Robert Collins.](https://reader035.fdocuments.in/reader035/viewer/2022062223/551b51d65503465c7e8b5ab0/html5/thumbnails/32.jpg)
Assumed Underlying PDF Real Data Samples
?Non-Parametric Density Estimation
![Page 33: CSCE643: Computer Vision Mean-Shift Object Tracking Jinxiang Chai Many slides from Yaron Ukrainitz & Bernard Sarel & Robert Collins.](https://reader035.fdocuments.in/reader035/viewer/2022062223/551b51d65503465c7e8b5ab0/html5/thumbnails/33.jpg)
Parametric Density Estimation
Assumption : The data points are sampled from an underlying PDF
Assumed Underlying PDF
2
2
( )
2
i
PDF( ) = i
iic e
x-μ
x
Estimate
Real Data Samples
![Page 34: CSCE643: Computer Vision Mean-Shift Object Tracking Jinxiang Chai Many slides from Yaron Ukrainitz & Bernard Sarel & Robert Collins.](https://reader035.fdocuments.in/reader035/viewer/2022062223/551b51d65503465c7e8b5ab0/html5/thumbnails/34.jpg)
Kernel Density Estimation Parzen Windows - Function Forms
1
1( ) ( )
n
ii
P Kn
x x - x A function of some finite number of data pointsx1…xn
DataIn practice one uses the forms:
1
( ) ( )d
ii
K c k x
x or ( )K ckx x
Same function on each dimension Function of vector length only
![Page 35: CSCE643: Computer Vision Mean-Shift Object Tracking Jinxiang Chai Many slides from Yaron Ukrainitz & Bernard Sarel & Robert Collins.](https://reader035.fdocuments.in/reader035/viewer/2022062223/551b51d65503465c7e8b5ab0/html5/thumbnails/35.jpg)
Kernel Density EstimationVarious Kernels
1
1( ) ( )
n
ii
P Kn
x x - x A function of some finite number of data pointsx1…xn
Examples:
• Epanechnikov Kernel
• Uniform Kernel
• Normal Kernel
21 1
( ) 0 otherwise
E
cK
x xx
1( )
0 otherwiseU
cK
xx
21( ) exp
2NK c
x x
Data
![Page 36: CSCE643: Computer Vision Mean-Shift Object Tracking Jinxiang Chai Many slides from Yaron Ukrainitz & Bernard Sarel & Robert Collins.](https://reader035.fdocuments.in/reader035/viewer/2022062223/551b51d65503465c7e8b5ab0/html5/thumbnails/36.jpg)
Kernel Density EstimationGradient
1
1 ( ) ( )
n
ii
P Kn
x x - x Give up estimating the PDF !Estimate ONLY the gradient
2
( ) iiK ck
h
x - xx - x
Using theKernel form:
We get :
1
1 1
1
( )
n
i in ni
i i ni i
ii
gc c
P k gn n g
xx x
Size of window
g( ) ( )kx x
![Page 37: CSCE643: Computer Vision Mean-Shift Object Tracking Jinxiang Chai Many slides from Yaron Ukrainitz & Bernard Sarel & Robert Collins.](https://reader035.fdocuments.in/reader035/viewer/2022062223/551b51d65503465c7e8b5ab0/html5/thumbnails/37.jpg)
Kernel Density EstimationGradient
1
1 1
1
( )
n
i in ni
i i ni i
ii
gc c
P k gn n g
xx x
Computing The Mean Shift
g( ) ( )kx x
![Page 38: CSCE643: Computer Vision Mean-Shift Object Tracking Jinxiang Chai Many slides from Yaron Ukrainitz & Bernard Sarel & Robert Collins.](https://reader035.fdocuments.in/reader035/viewer/2022062223/551b51d65503465c7e8b5ab0/html5/thumbnails/38.jpg)
1
1 1
1
( )
n
i in ni
i i ni i
ii
gc c
P k gn n g
xx x
Computing The Mean Shift
Yet another Kernel density estimation !
Simple Mean Shift procedure:• Compute mean shift vector
• Translate the Kernel window by m(x)
2
1
2
1
( )
ni
ii
ni
i
gh
gh
x - xx
m x xx - x
g( ) ( )kx x
![Page 39: CSCE643: Computer Vision Mean-Shift Object Tracking Jinxiang Chai Many slides from Yaron Ukrainitz & Bernard Sarel & Robert Collins.](https://reader035.fdocuments.in/reader035/viewer/2022062223/551b51d65503465c7e8b5ab0/html5/thumbnails/39.jpg)
Mean Shift Mode Detection
Updated Mean Shift Procedure:• Find all modes using the Simple Mean Shift Procedure• Prune modes by perturbing them (find saddle points and plateaus)• Prune nearby – take highest mode in the window
What happens if wereach a saddle point
?
Perturb the mode positionand check if we return back
![Page 40: CSCE643: Computer Vision Mean-Shift Object Tracking Jinxiang Chai Many slides from Yaron Ukrainitz & Bernard Sarel & Robert Collins.](https://reader035.fdocuments.in/reader035/viewer/2022062223/551b51d65503465c7e8b5ab0/html5/thumbnails/40.jpg)
Mean Shift Strengths & Weaknesses
Strengths :
• Application independent tool
• Suitable for real data analysis
• Does not assume any prior shape (e.g. elliptical) on data clusters
• Can handle arbitrary feature spaces
• Only ONE parameter to choose
• h (window size) has a physical meaning, unlike K-Means
Weaknesses :
• The window size (bandwidth selection) is not trivial
• Inappropriate window size can cause modes to be merged, or generate additional “shallow” modes Use adaptive window size
![Page 41: CSCE643: Computer Vision Mean-Shift Object Tracking Jinxiang Chai Many slides from Yaron Ukrainitz & Bernard Sarel & Robert Collins.](https://reader035.fdocuments.in/reader035/viewer/2022062223/551b51d65503465c7e8b5ab0/html5/thumbnails/41.jpg)
Mean Shift Applications
![Page 42: CSCE643: Computer Vision Mean-Shift Object Tracking Jinxiang Chai Many slides from Yaron Ukrainitz & Bernard Sarel & Robert Collins.](https://reader035.fdocuments.in/reader035/viewer/2022062223/551b51d65503465c7e8b5ab0/html5/thumbnails/42.jpg)
• D. Comaniciu, V. Ramesh, and P. Meer. Real-time tracking of non-rigid objects using mean shift. In IEEE Proc. on Computer Vision and Pattern Recognition, pages 673–678, 2000. (Best paper award)
• Journal version: Kernel-Based Object Tracking, PAMI, 2003.
Mean Shift Object Tracking
![Page 43: CSCE643: Computer Vision Mean-Shift Object Tracking Jinxiang Chai Many slides from Yaron Ukrainitz & Bernard Sarel & Robert Collins.](https://reader035.fdocuments.in/reader035/viewer/2022062223/551b51d65503465c7e8b5ab0/html5/thumbnails/43.jpg)
Non-Rigid Object Tracking
… …
![Page 44: CSCE643: Computer Vision Mean-Shift Object Tracking Jinxiang Chai Many slides from Yaron Ukrainitz & Bernard Sarel & Robert Collins.](https://reader035.fdocuments.in/reader035/viewer/2022062223/551b51d65503465c7e8b5ab0/html5/thumbnails/44.jpg)
Non-Rigid Object TrackingReal-Time
Surveillance Driver AssistanceObject-Based Video
Compression
![Page 45: CSCE643: Computer Vision Mean-Shift Object Tracking Jinxiang Chai Many slides from Yaron Ukrainitz & Bernard Sarel & Robert Collins.](https://reader035.fdocuments.in/reader035/viewer/2022062223/551b51d65503465c7e8b5ab0/html5/thumbnails/45.jpg)
Current frame
… …
Mean-Shift Object TrackingGeneral Framework: Target Representation
Choose a feature space
Represent the model in the
chosen feature space
Choose a reference
model in the current frame
![Page 46: CSCE643: Computer Vision Mean-Shift Object Tracking Jinxiang Chai Many slides from Yaron Ukrainitz & Bernard Sarel & Robert Collins.](https://reader035.fdocuments.in/reader035/viewer/2022062223/551b51d65503465c7e8b5ab0/html5/thumbnails/46.jpg)
Mean-Shift Object TrackingGeneral Framework: Target Localization
Search in the model’s
neighborhood in next frame
Start from the position of the model in the current frame
Find best candidate by maximizing a similarity func.
Repeat the same process in the next pair
of frames
Current frame
… …Model Candidate
![Page 47: CSCE643: Computer Vision Mean-Shift Object Tracking Jinxiang Chai Many slides from Yaron Ukrainitz & Bernard Sarel & Robert Collins.](https://reader035.fdocuments.in/reader035/viewer/2022062223/551b51d65503465c7e8b5ab0/html5/thumbnails/47.jpg)
Mean-Shift Object TrackingTarget Representation
Choose a reference
target model
Quantized Color Space
Choose a feature space
Represent the model by its PDF in the
feature space
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
1 2 3 . . . m
color
Pro
bab
ility
Kernel Based Object Tracking, by Comaniniu, Ramesh, Meer
![Page 48: CSCE643: Computer Vision Mean-Shift Object Tracking Jinxiang Chai Many slides from Yaron Ukrainitz & Bernard Sarel & Robert Collins.](https://reader035.fdocuments.in/reader035/viewer/2022062223/551b51d65503465c7e8b5ab0/html5/thumbnails/48.jpg)
Mean-Shift Object TrackingPDF Representation
,f y f q p y Similarity
Function:
Target Model(centered at 0)
Target Candidate(centered at y)
1..1
1m
u uu mu
q q q
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
1 2 3 . . . m
color
Pro
bab
ility
1..
1
1m
u uu mu
p y p y p
0
0.05
0.1
0.15
0.2
0.25
0.3
1 2 3 . . . m
color
Pro
bab
ility
![Page 49: CSCE643: Computer Vision Mean-Shift Object Tracking Jinxiang Chai Many slides from Yaron Ukrainitz & Bernard Sarel & Robert Collins.](https://reader035.fdocuments.in/reader035/viewer/2022062223/551b51d65503465c7e8b5ab0/html5/thumbnails/49.jpg)
f y
Mean-Shift Object TrackingSmoothness of Similarity Function
Similarity Function: ,f y f p y q
Problem:Target is
represented by color info only
Spatial info is lost
Solution:Mask the target with an isotropic kernel in the spatial domain
f(y) becomes smooth in y
f is not smooth
Gradient-based
optimizations are not robust
Large similarity variations for
adjacent locations
![Page 50: CSCE643: Computer Vision Mean-Shift Object Tracking Jinxiang Chai Many slides from Yaron Ukrainitz & Bernard Sarel & Robert Collins.](https://reader035.fdocuments.in/reader035/viewer/2022062223/551b51d65503465c7e8b5ab0/html5/thumbnails/50.jpg)
Mean-Shift Object TrackingFinding the PDF of the target model
1..i i nx
Target pixel locations
( )k x A differentiable, isotropic, convex, monotonically decreasing kernel• Peripheral pixels are affected by occlusion and background interference
( )b x The color bin index (1..m) of pixel x
2
( )i
u ib x u
q C k x
Normalization factor
Pixel weight
Probability of feature u in model
0
0.05
0.1
0.15
0.2
0.25
0.3
1 2 3 . . . m
color
Pro
bab
ility
Probability of feature u in candidate
0
0.05
0.1
0.15
0.2
0.25
0.3
1 2 3 . . . m
color
Pro
bab
ility
2
( )i
iu h
b x u
y xp y C k
h
Normalization factor Pixel weight
0
model
y
candidate
![Page 51: CSCE643: Computer Vision Mean-Shift Object Tracking Jinxiang Chai Many slides from Yaron Ukrainitz & Bernard Sarel & Robert Collins.](https://reader035.fdocuments.in/reader035/viewer/2022062223/551b51d65503465c7e8b5ab0/html5/thumbnails/51.jpg)
Mean-Shift Object TrackingSimilarity Function
1 , , mp y p y p y
1, , mq q q
Target model:
Target candidate:
Similarity function: , ?f y f p y q
1 , , mp y p y p y
1 , , mq q q
q
p yy
1
1
The Bhattacharyya Coefficient
1
cosT m
y u uu
p y qf y p y q
p y q
![Page 52: CSCE643: Computer Vision Mean-Shift Object Tracking Jinxiang Chai Many slides from Yaron Ukrainitz & Bernard Sarel & Robert Collins.](https://reader035.fdocuments.in/reader035/viewer/2022062223/551b51d65503465c7e8b5ab0/html5/thumbnails/52.jpg)
Mean-Shift Object TrackingTarget Localization Algorithm
Start from the position of the model in the current frame
q
Search in the model’s
neighborhood in next frame
p y
Find best candidate by maximizing a similarity func.
]),([maxarg qypfy
![Page 53: CSCE643: Computer Vision Mean-Shift Object Tracking Jinxiang Chai Many slides from Yaron Ukrainitz & Bernard Sarel & Robert Collins.](https://reader035.fdocuments.in/reader035/viewer/2022062223/551b51d65503465c7e8b5ab0/html5/thumbnails/53.jpg)
Mean-Shift Object TrackingFunction Optimization
]),([maxarg qypfy
Or
]),([1minarg qypfy
![Page 54: CSCE643: Computer Vision Mean-Shift Object Tracking Jinxiang Chai Many slides from Yaron Ukrainitz & Bernard Sarel & Robert Collins.](https://reader035.fdocuments.in/reader035/viewer/2022062223/551b51d65503465c7e8b5ab0/html5/thumbnails/54.jpg)
Mean-Shift Object TrackingFunction Optimization
]),([maxarg qypfy
Or
]),([1minarg qypfy
This is similar to Lucas Kanade registration!
- we can use gradient based optimization techniques to solve the problem.
![Page 55: CSCE643: Computer Vision Mean-Shift Object Tracking Jinxiang Chai Many slides from Yaron Ukrainitz & Bernard Sarel & Robert Collins.](https://reader035.fdocuments.in/reader035/viewer/2022062223/551b51d65503465c7e8b5ab0/html5/thumbnails/55.jpg)
Mean-Shift Object TrackingFunction Optimization
]),([maxarg qypfy
Or
]),([1minarg qypfy
Mean Shift provides an efficient way to optimize the function!
![Page 56: CSCE643: Computer Vision Mean-Shift Object Tracking Jinxiang Chai Many slides from Yaron Ukrainitz & Bernard Sarel & Robert Collins.](https://reader035.fdocuments.in/reader035/viewer/2022062223/551b51d65503465c7e8b5ab0/html5/thumbnails/56.jpg)
01 1 0
1 1
2 2
m mu
u u uu u u
qf y p y q p y
p y
Linear
approx.(around y0)
Mean-Shift Object TrackingApproximating the Similarity Function
0yModel location:yCandidate location:
2
12
nh i
ii
C y xw k
h
2
( )i
iu h
b x u
y xp y C k
h
Independent of y
Density estimate!
(as a function of y)
1
m
u uu
f y p y q
![Page 57: CSCE643: Computer Vision Mean-Shift Object Tracking Jinxiang Chai Many slides from Yaron Ukrainitz & Bernard Sarel & Robert Collins.](https://reader035.fdocuments.in/reader035/viewer/2022062223/551b51d65503465c7e8b5ab0/html5/thumbnails/57.jpg)
Mean-Shift Object TrackingMaximizing the Similarity Function
2
12
nh i
ii
C y xw k
h
The mode of = sought maximum
Important Assumption:
One mode in the searched neighborhood
The target representation
provides sufficient discrimination
![Page 58: CSCE643: Computer Vision Mean-Shift Object Tracking Jinxiang Chai Many slides from Yaron Ukrainitz & Bernard Sarel & Robert Collins.](https://reader035.fdocuments.in/reader035/viewer/2022062223/551b51d65503465c7e8b5ab0/html5/thumbnails/58.jpg)
Mean-Shift Object TrackingApplying Mean-Shift
Original Mean-Shift:
2
0
1
1 2
0
1
ni
ii
ni
i
y xx g
hy
y xg
h
Find mode of
2
1
ni
i
y xc k
h
using
2
12
nh i
ii
C y xw k
h
The mode of = sought maximum
Extended Mean-Shift:
2
0
1
1 2
0
1
ni
i ii
ni
ii
y xx w g
hy
y xw g
h
2
1
ni
ii
y xc w k
h
Find mode of using
![Page 59: CSCE643: Computer Vision Mean-Shift Object Tracking Jinxiang Chai Many slides from Yaron Ukrainitz & Bernard Sarel & Robert Collins.](https://reader035.fdocuments.in/reader035/viewer/2022062223/551b51d65503465c7e8b5ab0/html5/thumbnails/59.jpg)
Mean-Shift Object TrackingAbout Kernels and Profiles
A special class of radially symmetric kernels: 2
K x ck x
The profile of kernel K
Extended Mean-Shift:
2
0
1
1 2
0
1
ni
i ii
ni
ii
y xx w g
hy
y xw g
h
2
1
ni
ii
y xc w k
h
Find mode of using
k x g x
![Page 60: CSCE643: Computer Vision Mean-Shift Object Tracking Jinxiang Chai Many slides from Yaron Ukrainitz & Bernard Sarel & Robert Collins.](https://reader035.fdocuments.in/reader035/viewer/2022062223/551b51d65503465c7e8b5ab0/html5/thumbnails/60.jpg)
Mean-Shift Object TrackingChoosing the Kernel
Epanechnikov kernel:
1 if x 1
0 otherwise
xk x
A special class of radially symmetric kernels: 2
K x ck x
11
1
n
i iin
ii
x wy
w
2
0
1
1 2
0
1
ni
i ii
ni
ii
y xx w g
hy
y xw g
h
1 if x 1
0 otherwiseg x k x
Uniform kernel:
![Page 61: CSCE643: Computer Vision Mean-Shift Object Tracking Jinxiang Chai Many slides from Yaron Ukrainitz & Bernard Sarel & Robert Collins.](https://reader035.fdocuments.in/reader035/viewer/2022062223/551b51d65503465c7e8b5ab0/html5/thumbnails/61.jpg)
Mean-Shift Object TrackingAdaptive Scale
Problem:
The scale of the target
changes in time
The scale (h) of the
kernel must be adapted
Solution:
Run localization 3
times with different h
Choose h that achieves
maximum similarity
![Page 62: CSCE643: Computer Vision Mean-Shift Object Tracking Jinxiang Chai Many slides from Yaron Ukrainitz & Bernard Sarel & Robert Collins.](https://reader035.fdocuments.in/reader035/viewer/2022062223/551b51d65503465c7e8b5ab0/html5/thumbnails/62.jpg)
Mean-Shift Object TrackingResults
Feature space: 161616 quantized RGBTarget: manually selected on 1st frameAverage mean-shift iterations: 4
![Page 63: CSCE643: Computer Vision Mean-Shift Object Tracking Jinxiang Chai Many slides from Yaron Ukrainitz & Bernard Sarel & Robert Collins.](https://reader035.fdocuments.in/reader035/viewer/2022062223/551b51d65503465c7e8b5ab0/html5/thumbnails/63.jpg)
Mean-Shift Object TrackingResults
Partial occlusion Distraction Motion blur
![Page 64: CSCE643: Computer Vision Mean-Shift Object Tracking Jinxiang Chai Many slides from Yaron Ukrainitz & Bernard Sarel & Robert Collins.](https://reader035.fdocuments.in/reader035/viewer/2022062223/551b51d65503465c7e8b5ab0/html5/thumbnails/64.jpg)
Mean-Shift Object TrackingResults
![Page 65: CSCE643: Computer Vision Mean-Shift Object Tracking Jinxiang Chai Many slides from Yaron Ukrainitz & Bernard Sarel & Robert Collins.](https://reader035.fdocuments.in/reader035/viewer/2022062223/551b51d65503465c7e8b5ab0/html5/thumbnails/65.jpg)
Mean-Shift Object TrackingSummary
![Page 66: CSCE643: Computer Vision Mean-Shift Object Tracking Jinxiang Chai Many slides from Yaron Ukrainitz & Bernard Sarel & Robert Collins.](https://reader035.fdocuments.in/reader035/viewer/2022062223/551b51d65503465c7e8b5ab0/html5/thumbnails/66.jpg)
• Key idea #1: Formulate the tracking problem as nonlinear optimization by maximizing appearance/histogram consistency between target and template.
Mean-Shift Object TrackingSummary
]),([maxarg qypfy
q p y
![Page 67: CSCE643: Computer Vision Mean-Shift Object Tracking Jinxiang Chai Many slides from Yaron Ukrainitz & Bernard Sarel & Robert Collins.](https://reader035.fdocuments.in/reader035/viewer/2022062223/551b51d65503465c7e8b5ab0/html5/thumbnails/67.jpg)
• Key idea #2: Solving the optimization problem with mean-shift techniques
Mean-Shift Object TrackingSummary
![Page 68: CSCE643: Computer Vision Mean-Shift Object Tracking Jinxiang Chai Many slides from Yaron Ukrainitz & Bernard Sarel & Robert Collins.](https://reader035.fdocuments.in/reader035/viewer/2022062223/551b51d65503465c7e8b5ab0/html5/thumbnails/68.jpg)
• How to deal with scaling and rotation?
Mean-Shift Object TrackingDiscussion