KOUROSH MESHGI
description
Transcript of KOUROSH MESHGI
![Page 1: KOUROSH MESHGI](https://reader035.fdocuments.in/reader035/viewer/2022062310/56815b99550346895dc99e2b/html5/thumbnails/1.jpg)
K O U R O S HM E S H G IPROGRESS REPORT TOPIC
Occlusion Aware Particle Filter Tracker to Handle Complex and
Persistent Occlusions usingMultiple Feature Fusion
To: Ishii Lab Members,Dr. Shin-ichi Maeda, Dr. Shigeuki Oba,
And Prof. Shin Ishii
9 MAY 2014
![Page 2: KOUROSH MESHGI](https://reader035.fdocuments.in/reader035/viewer/2022062310/56815b99550346895dc99e2b/html5/thumbnails/2.jpg)
TRACKING APPLICATIONS
K O U R O S H M E S H G I – I S H I I L A B - D E C 2 0 1 3 - S L I D E 2
MAIN APPLICATIONS
Surveillance Public Entertainment
Robotics Video Indexing
Action Recog.
![Page 3: KOUROSH MESHGI](https://reader035.fdocuments.in/reader035/viewer/2022062310/56815b99550346895dc99e2b/html5/thumbnails/3.jpg)
TRACKING CHALLENGES
K O U R O S H M E S H G I – I S H I I L A B - D E C 2 0 1 3 - S L I D E 3
MAIN CHALLENGES
Varying ScaleClutterDeformation
OcclusionIlluminationAbrupt Motion
![Page 4: KOUROSH MESHGI](https://reader035.fdocuments.in/reader035/viewer/2022062310/56815b99550346895dc99e2b/html5/thumbnails/4.jpg)
Goal: Define p(Xt|Y1,…,Yt) given p(X1)
BAYESIAN TRACKING
K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 4
X1 X2 … Xt
Y1 Y2 … Yt
States: Target Location and Scale
Observations: Sensory Information
![Page 5: KOUROSH MESHGI](https://reader035.fdocuments.in/reader035/viewer/2022062310/56815b99550346895dc99e2b/html5/thumbnails/5.jpg)
PARTICLE FILTER TR.INTRODUCTION• • • • • • • • • • • • • • • • • • • • • • • •
![Page 6: KOUROSH MESHGI](https://reader035.fdocuments.in/reader035/viewer/2022062310/56815b99550346895dc99e2b/html5/thumbnails/6.jpg)
K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 6
INPUT IMAGEFrame: t
RGB Domain
![Page 7: KOUROSH MESHGI](https://reader035.fdocuments.in/reader035/viewer/2022062310/56815b99550346895dc99e2b/html5/thumbnails/7.jpg)
K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 7
INPUT DEPTH MAPFrame: t
Depth Domain
Close Far
![Page 8: KOUROSH MESHGI](https://reader035.fdocuments.in/reader035/viewer/2022062310/56815b99550346895dc99e2b/html5/thumbnails/8.jpg)
K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 8
SENSORY INFORMATIONFrame: t
Sensory Information
, ,{ , }t rgb t d tI I I
![Page 9: KOUROSH MESHGI](https://reader035.fdocuments.in/reader035/viewer/2022062310/56815b99550346895dc99e2b/html5/thumbnails/9.jpg)
Observation
K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 9
STATE REPRESENTATION & OBSERVATION MODEL
Frame: t
State
{ , , , }t t t t tB x y w h{ }t tX B
( ; )t t tY g I B
w
h
(x,y)
![Page 10: KOUROSH MESHGI](https://reader035.fdocuments.in/reader035/viewer/2022062310/56815b99550346895dc99e2b/html5/thumbnails/10.jpg)
K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 1 0
FEATURES
Feature Set1{ ,..., }MF f f
Color
Shape Edge
Texture
![Page 11: KOUROSH MESHGI](https://reader035.fdocuments.in/reader035/viewer/2022062310/56815b99550346895dc99e2b/html5/thumbnails/11.jpg)
K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 1 1
TEMPLATEFrame: 1
Template1 1,1 ,1{ ,..., }M
f1 fj fM
1 ,1 1{ }Mi i
1 1{ ( )}if Y
… …
![Page 12: KOUROSH MESHGI](https://reader035.fdocuments.in/reader035/viewer/2022062310/56815b99550346895dc99e2b/html5/thumbnails/12.jpg)
K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 1 2
PARTICLES INITIALIZATIONFrame: 1
Particles, ,{ }k t k tX B1,2, ,k N
Initialized Overlapped
![Page 13: KOUROSH MESHGI](https://reader035.fdocuments.in/reader035/viewer/2022062310/56815b99550346895dc99e2b/html5/thumbnails/13.jpg)
K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 1 3
MOTION MODELFrame: t
Motion Model, , ,k t k t k tB B
→ t + 1
![Page 14: KOUROSH MESHGI](https://reader035.fdocuments.in/reader035/viewer/2022062310/56815b99550346895dc99e2b/html5/thumbnails/14.jpg)
K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 1 4
FEATURE EXTRACTIONFrame: t + 1
Feature Vectors , 1( )i k tf Y
f1 f2 fM
X1,t+1
X2,t+1
XN,t+1
…
…
![Page 15: KOUROSH MESHGI](https://reader035.fdocuments.in/reader035/viewer/2022062310/56815b99550346895dc99e2b/html5/thumbnails/15.jpg)
K O U R O S H M E S H G I – I S H I I L A B - M A R 2 0 1 4 - S L I D E 1 5
FEATURE FUSIONFrame: t
Probability of Observation( | , )t t tp Y X ,1
( ( ) | , )Mi i t t i ti
p f Y B ( ( ) | , )t t tp f Y B ,1
( ),Mi i i t i ti
p D f Y ,1
exp ( ),Mi i i t i tiD f Y
,1
exp ( ),Mi i i t i tiD f Y
Each Feature(.)if(.)iD
i Indepen
dence
Assumptio
n
!
![Page 16: KOUROSH MESHGI](https://reader035.fdocuments.in/reader035/viewer/2022062310/56815b99550346895dc99e2b/html5/thumbnails/16.jpg)
K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 1 6
PROB. CALCULATIONFrame: t + 1
Particles
Brighter = More Probable
,k tp
![Page 17: KOUROSH MESHGI](https://reader035.fdocuments.in/reader035/viewer/2022062310/56815b99550346895dc99e2b/html5/thumbnails/17.jpg)
K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 1 7
TARGET ESTIMATIONFrame: t + 1
Feature Vectors 1 1
ˆ | ,...,t t tB B Y Y E
![Page 18: KOUROSH MESHGI](https://reader035.fdocuments.in/reader035/viewer/2022062310/56815b99550346895dc99e2b/html5/thumbnails/18.jpg)
K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 1 8
MODEL UPDATEFrame: t + 1
New Model
Model Update
1ˆ ˆ( ; )t t tY g I B
1ˆ ˆ( )t i tf Y
, 1 , 1
,
ˆ
(1 )i t i i t
i i t
![Page 19: KOUROSH MESHGI](https://reader035.fdocuments.in/reader035/viewer/2022062310/56815b99550346895dc99e2b/html5/thumbnails/19.jpg)
K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 1 9
RESAMPLINGFrame: t + 1
Proportional to Probability
1( | )t tp X X1
2
345
67
![Page 20: KOUROSH MESHGI](https://reader035.fdocuments.in/reader035/viewer/2022062310/56815b99550346895dc99e2b/html5/thumbnails/20.jpg)
PARTICLE FILTER TR.CHALLENGES• • • • • • • • • • • • • • • • • • • • • • • •
![Page 21: KOUROSH MESHGI](https://reader035.fdocuments.in/reader035/viewer/2022062310/56815b99550346895dc99e2b/html5/thumbnails/21.jpg)
PFT ISSUES
K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 2 1
Appearance Changes
Model Drift
Deficient Feature Space
Uninformed Search
Optimized Feature Selection
Approximation of Target
![Page 22: KOUROSH MESHGI](https://reader035.fdocuments.in/reader035/viewer/2022062310/56815b99550346895dc99e2b/html5/thumbnails/22.jpg)
K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 2 2
APPEARANCE CHANGES
Same Color ObjectsBackground ClutterIllumination ChangeShadows, Shades
Use Depth!
![Page 23: KOUROSH MESHGI](https://reader035.fdocuments.in/reader035/viewer/2022062310/56815b99550346895dc99e2b/html5/thumbnails/23.jpg)
K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 2 3
MODEL DRIFT PROBLEM
Templates Corrupted! t
Handle Occlusion!(No Model Update During Them)
![Page 24: KOUROSH MESHGI](https://reader035.fdocuments.in/reader035/viewer/2022062310/56815b99550346895dc99e2b/html5/thumbnails/24.jpg)
DEFICIENT FEATURE SPACE
* Local Optima of Feature Space* Feature Noise* Feature Failures
K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 2 4
RegularizationNon-zero ValuesNormalization
![Page 25: KOUROSH MESHGI](https://reader035.fdocuments.in/reader035/viewer/2022062310/56815b99550346895dc99e2b/html5/thumbnails/25.jpg)
PERSISTENT OCCLUSION
Particles Converge to Local Optima / Remains The Same Region
K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 2 5
Advanced Motion Models(not always feasible)
Restart Tracking(slow occlusion recovery)
Expand Search Area!
![Page 26: KOUROSH MESHGI](https://reader035.fdocuments.in/reader035/viewer/2022062310/56815b99550346895dc99e2b/html5/thumbnails/26.jpg)
DYNAMICS…
* The Search is not Directed* Neither of the Channels have Useful Information* Particles Should Scatter Away from Last Known Position
K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 2 6
Occlusion!
![Page 27: KOUROSH MESHGI](https://reader035.fdocuments.in/reader035/viewer/2022062310/56815b99550346895dc99e2b/html5/thumbnails/27.jpg)
OCCLUSIONdo not address occlusion explicitly
maintain a large set of hypotheses
computationally expensive
direct occlusion detection robust against partial & temp occ. persistent occ. hinder tracking
GENERATIVE MODELS DISCRIMINATIVE MODELS
Dynamic Occlusion: Pixels of other object close to camera
Scene Occlusion: Still objects are closer to camera than the target object
Apparent Occlusion: Due to shape change, silhouette motion, shadows, self-occ
UPDATE MODEL FOR TARGET TYPE OF OCCLUSION IS IMPORTANT KEEP MEMORY VS. KEEP FOCUS ON THE TARGET
Combine
Them!
![Page 28: KOUROSH MESHGI](https://reader035.fdocuments.in/reader035/viewer/2022062310/56815b99550346895dc99e2b/html5/thumbnails/28.jpg)
PTO partial occlusion SAO self- or articulation occlusion TFO temporal full occlusion - shorter than 3
frames PFO persistent full occlusion CPO complex partial occlusion - including “split
and merge” and permanent changes in a key attribute of a part of target
CFO complex full occlusion
OCCLUSION TYPES
K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 2 8
![Page 29: KOUROSH MESHGI](https://reader035.fdocuments.in/reader035/viewer/2022062310/56815b99550346895dc99e2b/html5/thumbnails/29.jpg)
[Zhao & Nevatia, 04] Occlusion Indicator: Ratio of FG/BKG
[Wu & Nevatia, 07] Handle Occlusion using Appearance Model
[de Villiers et al., 12] Switch Tracker in the case of Occlusion
[Song & Xiao, 13] Occlusion Indicator: New Peak in HOD or Reduction of the Size of Main Peak
LITERATURE REVIW
Many other papers handle occlusions as the by-product of their novel trackers
![Page 30: KOUROSH MESHGI](https://reader035.fdocuments.in/reader035/viewer/2022062310/56815b99550346895dc99e2b/html5/thumbnails/30.jpg)
OCCLUSION AWARE PFTSOLUTION• • • • • • • • • • • • • • • • • • • • • • • •
![Page 31: KOUROSH MESHGI](https://reader035.fdocuments.in/reader035/viewer/2022062310/56815b99550346895dc99e2b/html5/thumbnails/31.jpg)
Motion Model
Resampling
Target Estimation
Calculate Likelihood
PRO
POSE
D M
OD
IFIC
ATIO
N
K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 3 1
Initialization
Model Update
Observation
Occlusion Flag?
Constant Likelihood
Occlusion Estimation
Occlusion Threshold>?
YES
YES
NO
NO
![Page 32: KOUROSH MESHGI](https://reader035.fdocuments.in/reader035/viewer/2022062310/56815b99550346895dc99e2b/html5/thumbnails/32.jpg)
Occlusion Flag (for each particle)
Observation Model
No-Occlusion Particles Same as Before
Occlusion-Flagged Particles Uniform Distribution
OCCLUSION AWAREPARTICLE FILTER FRAMEWORK
( | ) ( | , , )t t t t t tp Y X p Y B Z ( | ) (1 ) ( | , 0, ) ( | , 1, )t t t t t t t t t t t tp Y X Z p Y B Z Z p Y B Z
,k tZ
( | , 1, ) 1t t t tp Y B Z
,1( | , 0, ) exp ( ),M
t t t t i i i t i tip Y B Z D f Y
K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 3 2
![Page 33: KOUROSH MESHGI](https://reader035.fdocuments.in/reader035/viewer/2022062310/56815b99550346895dc99e2b/html5/thumbnails/33.jpg)
Position Estimation of the Target
Occlusion State for the Next Box
TARGET ESTIMATION
1 1
, , , ,1
ˆ [ | ,..., ]
( | , , )
t t t occ
Nj t j t j t j t t occj
Z u Z Y Y
u Z p Y B Z
E
K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 3 3
1
, , , ,
ˆ [ | ,..., , 0]
( | , 0, )t t t t
j t j t j t j t tj
B B Y Y Z
B p Y B Z
E
J '
1
0
1
0 a
( )u x
( )u x a0a x
x
![Page 34: KOUROSH MESHGI](https://reader035.fdocuments.in/reader035/viewer/2022062310/56815b99550346895dc99e2b/html5/thumbnails/34.jpg)
Model Update (Separately for each Feature)
Modified Dynamics Model of Particle
UPDATE RULE
11
1 1
ˆ( ) ,( )
ˆ ˆ( ) (1 ) ( ) ,t t occ
t
t t t occ
f Zf
f Y f Z
K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 3 4
1 1 1 1 1( | ) ( , | , ) ( | ) ( | )t t t t t t t t t tp X X p B Z B Z p B B p Z Z
![Page 35: KOUROSH MESHGI](https://reader035.fdocuments.in/reader035/viewer/2022062310/56815b99550346895dc99e2b/html5/thumbnails/35.jpg)
OA-PF DYNAMICS
K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 3 5
Occlusion!
![Page 36: KOUROSH MESHGI](https://reader035.fdocuments.in/reader035/viewer/2022062310/56815b99550346895dc99e2b/html5/thumbnails/36.jpg)
OA-PF DYNAMICS
K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 3 6
Occlusion!
GOTCHA!
![Page 37: KOUROSH MESHGI](https://reader035.fdocuments.in/reader035/viewer/2022062310/56815b99550346895dc99e2b/html5/thumbnails/37.jpg)
OA-PF DYNAMICS
K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 3 7
Quick Occlusion Recovery Low CPE
No Template Corruption
No Attraction to other Object/ Background
![Page 38: KOUROSH MESHGI](https://reader035.fdocuments.in/reader035/viewer/2022062310/56815b99550346895dc99e2b/html5/thumbnails/38.jpg)
CO
LO
R
(HO
C)
TE
XT
UR
E
(LB
P)
ED
GE
(L
OG
)
2D PR
OJ.
(BE
TA)
3D SH
APE
(PC
L Σ)FEATURES
DE
PTH
(H
OD
)
GR
AD
IEN
T (H
OG
)K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 3 8
![Page 39: KOUROSH MESHGI](https://reader035.fdocuments.in/reader035/viewer/2022062310/56815b99550346895dc99e2b/html5/thumbnails/39.jpg)
& DISCUSSIONRESULTS• • • • • • • • • • • • • • • • • • • • • • • •
![Page 40: KOUROSH MESHGI](https://reader035.fdocuments.in/reader035/viewer/2022062310/56815b99550346895dc99e2b/html5/thumbnails/40.jpg)
Princeton Tracking Dataset
DATASET( )
5 Validation Video with Ground Truth95 Evaluation Video
K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 4 0
![Page 41: KOUROSH MESHGI](https://reader035.fdocuments.in/reader035/viewer/2022062310/56815b99550346895dc99e2b/html5/thumbnails/41.jpg)
EXPERIMENT
K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 4 1
OAPFT (Proposed, with different feature sets)
OI+SVM (SVM tracker with Occlusion Indicator)
• State-of-the-art RGBD tracker
ACPF (Adaptive Color Particle Filter)
• Traditional Particle Filter tracker
STRUCK (Structured Output SVM Tracker)
• State-of-the-art RGB tracker, Successful for Occlusion Handling
![Page 42: KOUROSH MESHGI](https://reader035.fdocuments.in/reader035/viewer/2022062310/56815b99550346895dc99e2b/html5/thumbnails/42.jpg)
PASCAL VOC: Overall Performance
CRITERIA I
1
1
*1
* *1 1 1
*1 1
*1 1
ˆ
ˆ ˆ, 0ˆ1 , 1ˆ1 ,
t
t
t
t t t
t t t
t t
B B
B B Z Z
S Z Z
Z Z
0 1ott oS t AUC
toSu
cces
sOverlap Threshold
0
1
1
Area Under Curve
K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 4 2
![Page 43: KOUROSH MESHGI](https://reader035.fdocuments.in/reader035/viewer/2022062310/56815b99550346895dc99e2b/html5/thumbnails/43.jpg)
RESULTS
K O U R O S H M E S H G I – I S H I I L A B - M A R 2 0 1 4 - S L I D E 4 3
1
1
Success Plot
Overlap Threshold
Succ
ess R
ate
1
1
![Page 44: KOUROSH MESHGI](https://reader035.fdocuments.in/reader035/viewer/2022062310/56815b99550346895dc99e2b/html5/thumbnails/44.jpg)
Mean Central Point Error: Localization Success
Mean Scale Adaption Error
CRITERIA II
* 2 * 21
ˆˆ( ) ( )Tt t t tt
w w h hSAE
T
* 2 * 21
ˆ ˆ( ) ( )Tt t t tt
x x y yCPE
T
K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 4 4
ˆˆ ˆ ˆ ˆ{ , , , }t t t t tB x y w h * * * * *{ , , , }t t t t t
B x y w h
Estimated Ground Truth
![Page 45: KOUROSH MESHGI](https://reader035.fdocuments.in/reader035/viewer/2022062310/56815b99550346895dc99e2b/html5/thumbnails/45.jpg)
RESULTSCenter Positioning Error
400
50Frames
CPE
(pix
els)
![Page 46: KOUROSH MESHGI](https://reader035.fdocuments.in/reader035/viewer/2022062310/56815b99550346895dc99e2b/html5/thumbnails/46.jpg)
RESULTSScale Adaptation Error
140
50Frames
SAE
(pix
els)
![Page 47: KOUROSH MESHGI](https://reader035.fdocuments.in/reader035/viewer/2022062310/56815b99550346895dc99e2b/html5/thumbnails/47.jpg)
FP happens when a tracker doesn’t realize that the target is occluded.
MI happens when the target is visible but the tracker fails to track it as if the target is still in an occlusion state
MT the estimated bounding box has nothing in common with ground truth box
FPS execution time in frames per second
CRITERIA III
K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 4 7
![Page 48: KOUROSH MESHGI](https://reader035.fdocuments.in/reader035/viewer/2022062310/56815b99550346895dc99e2b/html5/thumbnails/48.jpg)
RESULTS
K O U R O S H M E S H G I – I S H I I L A B – M A R 2 0 1 4 – S L I D E 4 8
Tracker
AUC
CPE
SAE MI FP MT FP
SBCDEGST (proposed)
76.50
9.59
7.32 0.0 2.4 0.0 0.9
ACPF (Nummiaro ‘03)
27.55
90.38
35.27
12.6 0.0 31.
0 1.4
STRUCK (Hare ‘11)
46.67
68.74
26.61
12.6 0.0 64.
413.4
OI+SVM (Song ‘13)
69.15
9.68
12.04 0.4 20.
0 0.8 0.4
![Page 49: KOUROSH MESHGI](https://reader035.fdocuments.in/reader035/viewer/2022062310/56815b99550346895dc99e2b/html5/thumbnails/49.jpg)
FUTURE WORKS
K O U R O S H M E S H G I – I S H I I L A B – M A Y 2 0 1 4 – S L I D E 4 9
More Resilient Features + Scale
Adaptation
Active Occlusion Handling
Measure the Confidence of
each Data Channel
Adaptive Model Update
![Page 50: KOUROSH MESHGI](https://reader035.fdocuments.in/reader035/viewer/2022062310/56815b99550346895dc99e2b/html5/thumbnails/50.jpg)
QUESTIONS?Thank you for your time…