CAP5415-Computer Vision Lecture 21-Deformable Part Model...
Transcript of CAP5415-Computer Vision Lecture 21-Deformable Part Model...
![Page 1: CAP5415-Computer Vision Lecture 21-Deformable Part Model ...bagci/teaching/computervision16/Lec21.pdf · Pattern Analysis and Machine Intelligence, Vol. 32, No. 9, Sep. 2010. •](https://reader036.fdocuments.in/reader036/viewer/2022081613/5fbccc697281727ba27bc7f5/html5/thumbnails/1.jpg)
CAP5415-ComputerVisionLecture21-DeformablePartModel(DPM)
GuestLecturer:Sarfaraz Husein
1
![Page 2: CAP5415-Computer Vision Lecture 21-Deformable Part Model ...bagci/teaching/computervision16/Lec21.pdf · Pattern Analysis and Machine Intelligence, Vol. 32, No. 9, Sep. 2010. •](https://reader036.fdocuments.in/reader036/viewer/2022081613/5fbccc697281727ba27bc7f5/html5/thumbnails/2.jpg)
Motivation• Objectcategorydetection:
– Detectallobjectswiththesamecategoryinanimage• Forexamplehorsedetection:
2
![Page 3: CAP5415-Computer Vision Lecture 21-Deformable Part Model ...bagci/teaching/computervision16/Lec21.pdf · Pattern Analysis and Machine Intelligence, Vol. 32, No. 9, Sep. 2010. •](https://reader036.fdocuments.in/reader036/viewer/2022081613/5fbccc697281727ba27bc7f5/html5/thumbnails/3.jpg)
ClassificationTask
3
Atleastonebus
100%
![Page 4: CAP5415-Computer Vision Lecture 21-Deformable Part Model ...bagci/teaching/computervision16/Lec21.pdf · Pattern Analysis and Machine Intelligence, Vol. 32, No. 9, Sep. 2010. •](https://reader036.fdocuments.in/reader036/viewer/2022081613/5fbccc697281727ba27bc7f5/html5/thumbnails/4.jpg)
DetectionTask
4
100%
Predictedboundingboxshouldoverlapbyatleast50%withgroundtruth!!!
![Page 5: CAP5415-Computer Vision Lecture 21-Deformable Part Model ...bagci/teaching/computervision16/Lec21.pdf · Pattern Analysis and Machine Intelligence, Vol. 32, No. 9, Sep. 2010. •](https://reader036.fdocuments.in/reader036/viewer/2022081613/5fbccc697281727ba27bc7f5/html5/thumbnails/5.jpg)
Detections“nearmisses”
5
![Page 6: CAP5415-Computer Vision Lecture 21-Deformable Part Model ...bagci/teaching/computervision16/Lec21.pdf · Pattern Analysis and Machine Intelligence, Vol. 32, No. 9, Sep. 2010. •](https://reader036.fdocuments.in/reader036/viewer/2022081613/5fbccc697281727ba27bc7f5/html5/thumbnails/6.jpg)
PascalVOCChallenge-TheObjectClasses
6
Imagesretrieved fromflickerwebsite.
![Page 7: CAP5415-Computer Vision Lecture 21-Deformable Part Model ...bagci/teaching/computervision16/Lec21.pdf · Pattern Analysis and Machine Intelligence, Vol. 32, No. 9, Sep. 2010. •](https://reader036.fdocuments.in/reader036/viewer/2022081613/5fbccc697281727ba27bc7f5/html5/thumbnails/7.jpg)
SuccessfulDetectionMethod
• Jointwinnerin2009PascalVOCchallengewiththeOxfordMethod.
• Awardof"lifetimeachievement“in2010.• Mixtureofdeformablepartmodels.• Eachcomponenthasglobaltemplate+deformableparts– HOGfeaturetemplates.
• Fullytrainedfromboundingboxesalone.
(P.Felzenszwalbetal.)
7
![Page 8: CAP5415-Computer Vision Lecture 21-Deformable Part Model ...bagci/teaching/computervision16/Lec21.pdf · Pattern Analysis and Machine Intelligence, Vol. 32, No. 9, Sep. 2010. •](https://reader036.fdocuments.in/reader036/viewer/2022081613/5fbccc697281727ba27bc7f5/html5/thumbnails/8.jpg)
Recap:DeformableModels
8
Challenge:handlingvariationofappearanceswithinobjectclassesnon-rigidobjects
Idea:Considerobjectsasadeformedversionofatemplate!
![Page 9: CAP5415-Computer Vision Lecture 21-Deformable Part Model ...bagci/teaching/computervision16/Lec21.pdf · Pattern Analysis and Machine Intelligence, Vol. 32, No. 9, Sep. 2010. •](https://reader036.fdocuments.in/reader036/viewer/2022081613/5fbccc697281727ba27bc7f5/html5/thumbnails/9.jpg)
WhatisDPM?• Deformable Part is a discriminatively trained,
multi-scale model for image training that aim at making possible the effective use of more latent information such as hierarchical (grammar) models and models involving latent three dimensional pose.
9
![Page 10: CAP5415-Computer Vision Lecture 21-Deformable Part Model ...bagci/teaching/computervision16/Lec21.pdf · Pattern Analysis and Machine Intelligence, Vol. 32, No. 9, Sep. 2010. •](https://reader036.fdocuments.in/reader036/viewer/2022081613/5fbccc697281727ba27bc7f5/html5/thumbnails/10.jpg)
DPM• Definition:
– Root : Catch Roughly appearance of object– Part : Catch local appearance of object– Spring : spatial connections between parts
• Displacement :– Using minimizing energy function to find the optimal
displacement[1] Pictorial Structures for Object Recognition, Daniel P. Huttenlocher,1973
10
![Page 11: CAP5415-Computer Vision Lecture 21-Deformable Part Model ...bagci/teaching/computervision16/Lec21.pdf · Pattern Analysis and Machine Intelligence, Vol. 32, No. 9, Sep. 2010. •](https://reader036.fdocuments.in/reader036/viewer/2022081613/5fbccc697281727ba27bc7f5/html5/thumbnails/11.jpg)
Model
11
(b1) coarse template (b2)part templates (b3) spatial model (a) person detection Example
The deformable model includes both a coarse global template covering an entire object and higher resolution part templates. The templates represent histogram of gradient features
![Page 12: CAP5415-Computer Vision Lecture 21-Deformable Part Model ...bagci/teaching/computervision16/Lec21.pdf · Pattern Analysis and Machine Intelligence, Vol. 32, No. 9, Sep. 2010. •](https://reader036.fdocuments.in/reader036/viewer/2022081613/5fbccc697281727ba27bc7f5/html5/thumbnails/12.jpg)
ModelingSpatialRelationswithDPM
12
Spring-based models
[Fischler & Elschlager ,’73], [Ramanan et al,’07], [Felszwenwalb et al,’05,’09], [Kumar et al, ‘09]
![Page 13: CAP5415-Computer Vision Lecture 21-Deformable Part Model ...bagci/teaching/computervision16/Lec21.pdf · Pattern Analysis and Machine Intelligence, Vol. 32, No. 9, Sep. 2010. •](https://reader036.fdocuments.in/reader036/viewer/2022081613/5fbccc697281727ba27bc7f5/html5/thumbnails/13.jpg)
Parts Based Model
Vertices – Local Appearance
Edges - Spatial Relationship
JJ
JL
LL
Goal: Assign model parts to image regions preserving both local appearance and spatial
relationships
![Page 14: CAP5415-Computer Vision Lecture 21-Deformable Part Model ...bagci/teaching/computervision16/Lec21.pdf · Pattern Analysis and Machine Intelligence, Vol. 32, No. 9, Sep. 2010. •](https://reader036.fdocuments.in/reader036/viewer/2022081613/5fbccc697281727ba27bc7f5/html5/thumbnails/14.jpg)
MatchingProblem
14
![Page 15: CAP5415-Computer Vision Lecture 21-Deformable Part Model ...bagci/teaching/computervision16/Lec21.pdf · Pattern Analysis and Machine Intelligence, Vol. 32, No. 9, Sep. 2010. •](https://reader036.fdocuments.in/reader036/viewer/2022081613/5fbccc697281727ba27bc7f5/html5/thumbnails/15.jpg)
MatchingProblem
15
![Page 16: CAP5415-Computer Vision Lecture 21-Deformable Part Model ...bagci/teaching/computervision16/Lec21.pdf · Pattern Analysis and Machine Intelligence, Vol. 32, No. 9, Sep. 2010. •](https://reader036.fdocuments.in/reader036/viewer/2022081613/5fbccc697281727ba27bc7f5/html5/thumbnails/16.jpg)
Parts based models - Inference Problem
Inference problem: What is the best scoring assignment f?
Local Appearance term Pairwise Spatial Relationship term
Inference is NP-hard for general graphs
For trees can use belief propagation for exact solution in polytime
![Page 17: CAP5415-Computer Vision Lecture 21-Deformable Part Model ...bagci/teaching/computervision16/Lec21.pdf · Pattern Analysis and Machine Intelligence, Vol. 32, No. 9, Sep. 2010. •](https://reader036.fdocuments.in/reader036/viewer/2022081613/5fbccc697281727ba27bc7f5/html5/thumbnails/17.jpg)
Parts based models - Learning Problem
Linear models:
s.t.
Local Appearance term Pairwise Spatial Relationship term
Convex max-margin objective
Positive examples on one side
Negative examples on the other
[Kumar et al,’09]
Learning linear models: Find weight vectors that best separate positive and negative examples. E.g.,
![Page 18: CAP5415-Computer Vision Lecture 21-Deformable Part Model ...bagci/teaching/computervision16/Lec21.pdf · Pattern Analysis and Machine Intelligence, Vol. 32, No. 9, Sep. 2010. •](https://reader036.fdocuments.in/reader036/viewer/2022081613/5fbccc697281727ba27bc7f5/html5/thumbnails/18.jpg)
FindingMotorbike
18
![Page 19: CAP5415-Computer Vision Lecture 21-Deformable Part Model ...bagci/teaching/computervision16/Lec21.pdf · Pattern Analysis and Machine Intelligence, Vol. 32, No. 9, Sep. 2010. •](https://reader036.fdocuments.in/reader036/viewer/2022081613/5fbccc697281727ba27bc7f5/html5/thumbnails/19.jpg)
HumanTracking
19
![Page 20: CAP5415-Computer Vision Lecture 21-Deformable Part Model ...bagci/teaching/computervision16/Lec21.pdf · Pattern Analysis and Machine Intelligence, Vol. 32, No. 9, Sep. 2010. •](https://reader036.fdocuments.in/reader036/viewer/2022081613/5fbccc697281727ba27bc7f5/html5/thumbnails/20.jpg)
HumanPoseEstimation
20
![Page 21: CAP5415-Computer Vision Lecture 21-Deformable Part Model ...bagci/teaching/computervision16/Lec21.pdf · Pattern Analysis and Machine Intelligence, Vol. 32, No. 9, Sep. 2010. •](https://reader036.fdocuments.in/reader036/viewer/2022081613/5fbccc697281727ba27bc7f5/html5/thumbnails/21.jpg)
HoG:HistogramofGradients
21
![Page 22: CAP5415-Computer Vision Lecture 21-Deformable Part Model ...bagci/teaching/computervision16/Lec21.pdf · Pattern Analysis and Machine Intelligence, Vol. 32, No. 9, Sep. 2010. •](https://reader036.fdocuments.in/reader036/viewer/2022081613/5fbccc697281727ba27bc7f5/html5/thumbnails/22.jpg)
HistogramofGradient1.Compute gradient every pixel2.Group 8*8 pixels into a cell, and 4*4 cells into a
block.Build histogram of each cells about 9 orientation
bins (00~1800)
22
![Page 23: CAP5415-Computer Vision Lecture 21-Deformable Part Model ...bagci/teaching/computervision16/Lec21.pdf · Pattern Analysis and Machine Intelligence, Vol. 32, No. 9, Sep. 2010. •](https://reader036.fdocuments.in/reader036/viewer/2022081613/5fbccc697281727ba27bc7f5/html5/thumbnails/23.jpg)
HistogramofGradient3.A block contain 4*9 feature vector for local
information, and a bounding box which contain n’s blocks has n*4*9 feature vector
4.Training by SVM
23
![Page 24: CAP5415-Computer Vision Lecture 21-Deformable Part Model ...bagci/teaching/computervision16/Lec21.pdf · Pattern Analysis and Machine Intelligence, Vol. 32, No. 9, Sep. 2010. •](https://reader036.fdocuments.in/reader036/viewer/2022081613/5fbccc697281727ba27bc7f5/html5/thumbnails/24.jpg)
Filters
24
![Page 25: CAP5415-Computer Vision Lecture 21-Deformable Part Model ...bagci/teaching/computervision16/Lec21.pdf · Pattern Analysis and Machine Intelligence, Vol. 32, No. 9, Sep. 2010. •](https://reader036.fdocuments.in/reader036/viewer/2022081613/5fbccc697281727ba27bc7f5/html5/thumbnails/25.jpg)
ObjectHypothesis
25
![Page 26: CAP5415-Computer Vision Lecture 21-Deformable Part Model ...bagci/teaching/computervision16/Lec21.pdf · Pattern Analysis and Machine Intelligence, Vol. 32, No. 9, Sep. 2010. •](https://reader036.fdocuments.in/reader036/viewer/2022081613/5fbccc697281727ba27bc7f5/html5/thumbnails/26.jpg)
Training
26
![Page 27: CAP5415-Computer Vision Lecture 21-Deformable Part Model ...bagci/teaching/computervision16/Lec21.pdf · Pattern Analysis and Machine Intelligence, Vol. 32, No. 9, Sep. 2010. •](https://reader036.fdocuments.in/reader036/viewer/2022081613/5fbccc697281727ba27bc7f5/html5/thumbnails/27.jpg)
LearnedModels
27
![Page 28: CAP5415-Computer Vision Lecture 21-Deformable Part Model ...bagci/teaching/computervision16/Lec21.pdf · Pattern Analysis and Machine Intelligence, Vol. 32, No. 9, Sep. 2010. •](https://reader036.fdocuments.in/reader036/viewer/2022081613/5fbccc697281727ba27bc7f5/html5/thumbnails/28.jpg)
Training Models• Latent SVM is used for learning the training images
![Page 29: CAP5415-Computer Vision Lecture 21-Deformable Part Model ...bagci/teaching/computervision16/Lec21.pdf · Pattern Analysis and Machine Intelligence, Vol. 32, No. 9, Sep. 2010. •](https://reader036.fdocuments.in/reader036/viewer/2022081613/5fbccc697281727ba27bc7f5/html5/thumbnails/29.jpg)
Experiment Result• PASCAL VOC 2006,2007,2008 comp3 challenge datasets• Some statistics:
– It takes 2 seconds to evaluate a model in one image(4952 images in the test dataset)
– It takes 4 hours to train a model– MUCH faster than most systems.– All of the experiments were done on a 2.8Ghz 8-core Intel Xeon Mac
Pro computer running Mac OS X 10.5.
![Page 30: CAP5415-Computer Vision Lecture 21-Deformable Part Model ...bagci/teaching/computervision16/Lec21.pdf · Pattern Analysis and Machine Intelligence, Vol. 32, No. 9, Sep. 2010. •](https://reader036.fdocuments.in/reader036/viewer/2022081613/5fbccc697281727ba27bc7f5/html5/thumbnails/30.jpg)
Experiment Result• Measurement: predicted bounding box is correct if it overlaps
more than 50 percent with ground truth bounding box; otherwise, considered false positive
![Page 31: CAP5415-Computer Vision Lecture 21-Deformable Part Model ...bagci/teaching/computervision16/Lec21.pdf · Pattern Analysis and Machine Intelligence, Vol. 32, No. 9, Sep. 2010. •](https://reader036.fdocuments.in/reader036/viewer/2022081613/5fbccc697281727ba27bc7f5/html5/thumbnails/31.jpg)
Experiment Result
![Page 32: CAP5415-Computer Vision Lecture 21-Deformable Part Model ...bagci/teaching/computervision16/Lec21.pdf · Pattern Analysis and Machine Intelligence, Vol. 32, No. 9, Sep. 2010. •](https://reader036.fdocuments.in/reader036/viewer/2022081613/5fbccc697281727ba27bc7f5/html5/thumbnails/32.jpg)
Experiment Result
![Page 33: CAP5415-Computer Vision Lecture 21-Deformable Part Model ...bagci/teaching/computervision16/Lec21.pdf · Pattern Analysis and Machine Intelligence, Vol. 32, No. 9, Sep. 2010. •](https://reader036.fdocuments.in/reader036/viewer/2022081613/5fbccc697281727ba27bc7f5/html5/thumbnails/33.jpg)
Experiment Result• Best Average Precision score in 9 out of 20, second in 8
![Page 34: CAP5415-Computer Vision Lecture 21-Deformable Part Model ...bagci/teaching/computervision16/Lec21.pdf · Pattern Analysis and Machine Intelligence, Vol. 32, No. 9, Sep. 2010. •](https://reader036.fdocuments.in/reader036/viewer/2022081613/5fbccc697281727ba27bc7f5/html5/thumbnails/34.jpg)
Experiment Result
![Page 35: CAP5415-Computer Vision Lecture 21-Deformable Part Model ...bagci/teaching/computervision16/Lec21.pdf · Pattern Analysis and Machine Intelligence, Vol. 32, No. 9, Sep. 2010. •](https://reader036.fdocuments.in/reader036/viewer/2022081613/5fbccc697281727ba27bc7f5/html5/thumbnails/35.jpg)
SliceCreditsandReferences• JonathanHuang,TomaszMalisiewicz,2009.• P.Felzenszwalb,D.McAllester,D.Ramanan.ADiscriminatively Trained,
Multiscale,DeformablePartModel.IEEEConferenceonComputerVisionandPatternRecognition(CVPR),2008
• P.Felzenszwalb,R.Girshick,D.McAllester,D.Ramanan.ObjectDetectionwithDiscriminatively TrainedPartBasedModels. IEEETransactionsonPatternAnalysis andMachineIntelligence,Vol.32,No.9,Sep.2010.
• LevelImageRepresentationforSceneClassificationandSemanticFeatureSparsification.ProceedingsoftheNeuralInformationProcessingSystems(NIPS),2010.
35