Local Affine Feature Tracking in Films/Sitcoms Chunhui Gu CS 294-6 Final Presentation Dec. 13, 2006.
-
date post
15-Jan-2016 -
Category
Documents
-
view
216 -
download
0
Transcript of Local Affine Feature Tracking in Films/Sitcoms Chunhui Gu CS 294-6 Final Presentation Dec. 13, 2006.
Local Affine Feature Tracking in Films/Sitcoms
Chunhui GuCS 294-6
Final PresentationDec. 13, 2006
Objective
• Automatically detect and track local affine features in film/sitcom frame sequences.
– Current Dataset: Sex and the City– Why sitcom?
• Simple daily environment• Few or no special effects• Repeated scenes
Outline
• Preprocessing• Tracking Algorithm
– Pairwise local matching– Robust features
• Feature Matching across Shots• Results
– Feature matching vs baseline color histogram– Time complexity– When does tracking fail
Preprocessing
FrameExtraction
(i-1)’th shot i’th shot
ShotDetection
MSER Interest PointDetection
SIFT FeatureExtraction
Tracking Algorithm
• Basic: Pairwise Matching
Frame i Frame j=i+1
imf
,i im mx y
jnf
Tracking Algorithm
• Basic: Pairwise Matching
Frame i Frame j=i+1
imf
,i im mx y
jnf
Tracking Algorithm
• Basic: Pairwise Matching
Frame i Frame j=i+1
imf
,i im mx y
jnf min fd
Thresholding on both minimum distance and ratio
Tracking Algorithm
• Basic: Pairwise Matching
Frame i Frame j=i+1
imf
,i im mx y
jnf
Tracking Algorithm
• Basic: Pairwise Matching
Frame i Frame j=i+1
imf
,i im mx y
jnf
Tracking Algorithm
• Problem of Pairwise Matching– Sensitive to occlusion and feature misdetection
• Solutions:– Use multiple overlapping windows– Backward Matching
• Match features in current frame to features in all previous frames within the shot
• Pruning process (reduce computation time)
• Select a proportion of features that have longer tracking length as robust features
Shot grouping/Scene Retrieval
60
601 2, ,...rf mf x x x
56
561 2, ,...rf mf x x x
10746 10747 10772
Shot 49
10933 10934 10968
Shot 53
11393 11394 11435
Shot 56
Shot 60
11533 11534 11560
Scene 5
49
491 2, ,...rf mf x x x
53
531 2, ,...rf mf x x x
Inter-Shot Matching
Shot I Shot J
1 11 2, ,...I mf x x x
2 21 2, ,...I mf x x x
1 11 2, ,...J nf x x x
2 21 2, ,...J nf x x x
1 2, ,...q qJ nf x x x 1 2, ,...
p pI mf x x x
D
“Confusion Table”
Ground Truth50 55 60 65 70 75
50
55
60
65
70
75
Color Histograms50 55 60 65 70 75
50
55
60
65
70
75
Feature Matching50 55 60 65 70 75
50
55
60
65
70
75
ROC
0 0.2 0.4 0.6 0.8 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
False Alarm
Tru
e D
etec
tion
ROC curve of Feature Matching
When Does Tracking Fail?• Tracking feature outside local window
– Rare when continuous tracking– Happens when occlusion occurs
• Same feature splitting to two or more groups– Long occlusion– Multiple matching in a single frame
Frame i Frame j=i+1
imf
,i im mx y
jnf
Computation Complexity• Everything except for MSER and SIFT algorithms are
implemented in Matlab (slow…)
Complexity Time
Frame Extraction O(N) ~0.3s/frame
Shot Detection O(N*f(B)) ~0.07s/frame (B=16)
MSER Detection O(N) ~0.3s/frame
SIFT Detection O(N) ~0.9s/frame
Feature Tracking O(N*F*W*L) ~0.5s/frame
Matching across shots
O(S2*T2) ~1s/shot pair
N: # of frames; (30,000) B: # of bins for color hist (16) F: ave. # of features per frame; (400) W: Local window size; (15)L: tracking length; (20) T: ave. # of robust trackers per shot; (300)S: # of shots; (35)
Conclusion
• We successfully implemented local affine feature tracking in sitcom “sex and the city”. The tracking method is robust to occlusion and feature misdetection.
• Although no quantitative precision/recall curve (hard to find ground truth), the demonstration shows that precision is almost perfect with good recall performance.
• We show one successful application of using robust features to associate similar shots together for scene retrieval.
Future Work
• Implement algorithm in real-time (C/C++)
• Search unique shots in films/sitcoms
• Separate indoor scenes from outdoor scenes
• Determine context of the scene
Acknowledgement