Beyond bags of features: Part-based models Many slides adapted from Fei-Fei Li, Rob Fergus, and...
-
date post
19-Dec-2015 -
Category
Documents
-
view
229 -
download
2
Transcript of Beyond bags of features: Part-based models Many slides adapted from Fei-Fei Li, Rob Fergus, and...
![Page 1: Beyond bags of features: Part-based models Many slides adapted from Fei-Fei Li, Rob Fergus, and Antonio Torralba.](https://reader036.fdocuments.in/reader036/viewer/2022062313/56649d2c5503460f94a02a26/html5/thumbnails/1.jpg)
Beyond bags of features:Part-based models
Many slides adapted from Fei-Fei Li, Rob Fergus, and Antonio Torralba
![Page 2: Beyond bags of features: Part-based models Many slides adapted from Fei-Fei Li, Rob Fergus, and Antonio Torralba.](https://reader036.fdocuments.in/reader036/viewer/2022062313/56649d2c5503460f94a02a26/html5/thumbnails/2.jpg)
Implicit shape models• Visual codebook is used to index votes for
object position
B. Leibe, A. Leonardis, and B. Schiele, Combined Object Categorization and Segmentation with an Implicit Shape Model, ECCV Workshop on Statistical Learning in Computer Vision 2004
training image annotated with object localization info
visual codeword withdisplacement vectors
![Page 3: Beyond bags of features: Part-based models Many slides adapted from Fei-Fei Li, Rob Fergus, and Antonio Torralba.](https://reader036.fdocuments.in/reader036/viewer/2022062313/56649d2c5503460f94a02a26/html5/thumbnails/3.jpg)
Implicit shape models• Visual codebook is used to index votes for
object position
B. Leibe, A. Leonardis, and B. Schiele, Combined Object Categorization and Segmentation with an Implicit Shape Model, ECCV Workshop on Statistical Learning in Computer Vision 2004
test image
![Page 4: Beyond bags of features: Part-based models Many slides adapted from Fei-Fei Li, Rob Fergus, and Antonio Torralba.](https://reader036.fdocuments.in/reader036/viewer/2022062313/56649d2c5503460f94a02a26/html5/thumbnails/4.jpg)
Implicit shape models: Training
1. Build codebook of patches around extracted interest points using clustering
![Page 5: Beyond bags of features: Part-based models Many slides adapted from Fei-Fei Li, Rob Fergus, and Antonio Torralba.](https://reader036.fdocuments.in/reader036/viewer/2022062313/56649d2c5503460f94a02a26/html5/thumbnails/5.jpg)
Implicit shape models: Training
1. Build codebook of patches around extracted interest points using clustering
2. Map the patch around each interest point to closest codebook entry
![Page 6: Beyond bags of features: Part-based models Many slides adapted from Fei-Fei Li, Rob Fergus, and Antonio Torralba.](https://reader036.fdocuments.in/reader036/viewer/2022062313/56649d2c5503460f94a02a26/html5/thumbnails/6.jpg)
Implicit shape models: Training
1. Build codebook of patches around extracted interest points using clustering
2. Map the patch around each interest point to closest codebook entry
3. For each codebook entry, store all positions it was found, relative to object center
![Page 7: Beyond bags of features: Part-based models Many slides adapted from Fei-Fei Li, Rob Fergus, and Antonio Torralba.](https://reader036.fdocuments.in/reader036/viewer/2022062313/56649d2c5503460f94a02a26/html5/thumbnails/7.jpg)
Implicit shape models: Testing1. Given test image, extract patches, match to
codebook entry
2. Cast votes for possible positions of object center
3. Search for maxima in voting space
4. Extract weighted segmentation mask based on stored masks for the codebook occurrences
![Page 8: Beyond bags of features: Part-based models Many slides adapted from Fei-Fei Li, Rob Fergus, and Antonio Torralba.](https://reader036.fdocuments.in/reader036/viewer/2022062313/56649d2c5503460f94a02a26/html5/thumbnails/8.jpg)
Source: B. Leibe
Original image
Example: Results on Cows
![Page 9: Beyond bags of features: Part-based models Many slides adapted from Fei-Fei Li, Rob Fergus, and Antonio Torralba.](https://reader036.fdocuments.in/reader036/viewer/2022062313/56649d2c5503460f94a02a26/html5/thumbnails/9.jpg)
Interest points
Example: Results on Cows
Source: B. Leibe
![Page 10: Beyond bags of features: Part-based models Many slides adapted from Fei-Fei Li, Rob Fergus, and Antonio Torralba.](https://reader036.fdocuments.in/reader036/viewer/2022062313/56649d2c5503460f94a02a26/html5/thumbnails/10.jpg)
Example: Results on Cows
Matched patchesSource: B. Leibe
![Page 11: Beyond bags of features: Part-based models Many slides adapted from Fei-Fei Li, Rob Fergus, and Antonio Torralba.](https://reader036.fdocuments.in/reader036/viewer/2022062313/56649d2c5503460f94a02a26/html5/thumbnails/11.jpg)
Example: Results on Cows
Probabilistic votesSource: B. Leibe
![Page 12: Beyond bags of features: Part-based models Many slides adapted from Fei-Fei Li, Rob Fergus, and Antonio Torralba.](https://reader036.fdocuments.in/reader036/viewer/2022062313/56649d2c5503460f94a02a26/html5/thumbnails/12.jpg)
Example: Results on Cows
Hypothesis 1Source: B. Leibe
![Page 13: Beyond bags of features: Part-based models Many slides adapted from Fei-Fei Li, Rob Fergus, and Antonio Torralba.](https://reader036.fdocuments.in/reader036/viewer/2022062313/56649d2c5503460f94a02a26/html5/thumbnails/13.jpg)
Example: Results on Cows
Hypothesis 2Source: B. Leibe
![Page 14: Beyond bags of features: Part-based models Many slides adapted from Fei-Fei Li, Rob Fergus, and Antonio Torralba.](https://reader036.fdocuments.in/reader036/viewer/2022062313/56649d2c5503460f94a02a26/html5/thumbnails/14.jpg)
Example: Results on Cows
Hypothesis 3Source: B. Leibe
![Page 15: Beyond bags of features: Part-based models Many slides adapted from Fei-Fei Li, Rob Fergus, and Antonio Torralba.](https://reader036.fdocuments.in/reader036/viewer/2022062313/56649d2c5503460f94a02a26/html5/thumbnails/15.jpg)
Additional examples
B. Leibe, A. Leonardis, and B. Schiele, Robust Object Detection with Interleaved Categorization and Segmentation, IJCV 77 (1-3), pp. 259-289, 2008.
![Page 16: Beyond bags of features: Part-based models Many slides adapted from Fei-Fei Li, Rob Fergus, and Antonio Torralba.](https://reader036.fdocuments.in/reader036/viewer/2022062313/56649d2c5503460f94a02a26/html5/thumbnails/16.jpg)
Generative part-based models
R. Fergus, P. Perona and A. Zisserman, Object Class Recognition by Unsupervised Scale-Invariant Learning, CVPR 2003
![Page 17: Beyond bags of features: Part-based models Many slides adapted from Fei-Fei Li, Rob Fergus, and Antonio Torralba.](https://reader036.fdocuments.in/reader036/viewer/2022062313/56649d2c5503460f94a02a26/html5/thumbnails/17.jpg)
• Model the probability of an image given a class
What is a generative model?
)|( zebranoimagep)|( zebraimagep vs.
![Page 18: Beyond bags of features: Part-based models Many slides adapted from Fei-Fei Li, Rob Fergus, and Antonio Torralba.](https://reader036.fdocuments.in/reader036/viewer/2022062313/56649d2c5503460f94a02a26/html5/thumbnails/18.jpg)
• Maximum a posteriori (MAP) decision: assign the image to the class that gets the highest posterior probability P(class | image)
Making a decision
![Page 19: Beyond bags of features: Part-based models Many slides adapted from Fei-Fei Li, Rob Fergus, and Antonio Torralba.](https://reader036.fdocuments.in/reader036/viewer/2022062313/56649d2c5503460f94a02a26/html5/thumbnails/19.jpg)
• Maximum a posteriori (MAP) decision: assign the image to the class that gets the highest posterior probability P(class | image)
• Bayes rule:
Making a decision
)(
)()|()|(
imagep
classpclassimagepimageclassp
![Page 20: Beyond bags of features: Part-based models Many slides adapted from Fei-Fei Li, Rob Fergus, and Antonio Torralba.](https://reader036.fdocuments.in/reader036/viewer/2022062313/56649d2c5503460f94a02a26/html5/thumbnails/20.jpg)
Generative vs. discriminative models
• Discriminative methods: model posterior
• Generative methods: model likelihood and prior
)()|()|( classpclassimagepimageclassp
posterior likelihood prior
![Page 21: Beyond bags of features: Part-based models Many slides adapted from Fei-Fei Li, Rob Fergus, and Antonio Torralba.](https://reader036.fdocuments.in/reader036/viewer/2022062313/56649d2c5503460f94a02a26/html5/thumbnails/21.jpg)
Generative vs. discriminative models
Generative Discriminative
Cla
ss d
ensi
ties
Pos
terio
r pr
obab
ilitie
s
![Page 22: Beyond bags of features: Part-based models Many slides adapted from Fei-Fei Li, Rob Fergus, and Antonio Torralba.](https://reader036.fdocuments.in/reader036/viewer/2022062313/56649d2c5503460f94a02a26/html5/thumbnails/22.jpg)
The Naïve Bayes model
• Generative model for a bag of features
• Assume that each feature fi is conditionally independent given the class c:
N
iiN cfpcffpcimagep
11 )|()|,,()|(
![Page 23: Beyond bags of features: Part-based models Many slides adapted from Fei-Fei Li, Rob Fergus, and Antonio Torralba.](https://reader036.fdocuments.in/reader036/viewer/2022062313/56649d2c5503460f94a02a26/html5/thumbnails/23.jpg)
Generative part-based model
Candidate parts
)|(),|(),|(max
)|,,(max
)|,()|(
objecthpobjecthshapepobjecthappearanceP
objecthshapeappearanceP
objectshapeappearancePobjectimageP
h
h
Partdescriptors
Partlocations
![Page 24: Beyond bags of features: Part-based models Many slides adapted from Fei-Fei Li, Rob Fergus, and Antonio Torralba.](https://reader036.fdocuments.in/reader036/viewer/2022062313/56649d2c5503460f94a02a26/html5/thumbnails/24.jpg)
Generative part-based model
h: assignment of features to parts
Part 2
Part 3
Part 1
)|(),|(),|(max
)|,,(max
)|,()|(
objecthpobjecthshapepobjecthappearanceP
objecthshapeappearanceP
objectshapeappearancePobjectimageP
h
h
![Page 25: Beyond bags of features: Part-based models Many slides adapted from Fei-Fei Li, Rob Fergus, and Antonio Torralba.](https://reader036.fdocuments.in/reader036/viewer/2022062313/56649d2c5503460f94a02a26/html5/thumbnails/25.jpg)
Generative part-based model
h: assignment of features to parts
Part 2
Part 3
Part 1
)|(),|(),|(max
)|,,(max
)|,()|(
objecthpobjecthshapepobjecthappearanceP
objecthshapeappearanceP
objectshapeappearancePobjectimageP
h
h
![Page 26: Beyond bags of features: Part-based models Many slides adapted from Fei-Fei Li, Rob Fergus, and Antonio Torralba.](https://reader036.fdocuments.in/reader036/viewer/2022062313/56649d2c5503460f94a02a26/html5/thumbnails/26.jpg)
Generative part-based model
)|(),|(),|(max
)|,()|(
objecthpobjecthshapepobjecthappearanceP
objectshapeappearancePobjectimageP
h
High-dimensional appearance space
Distribution over patchdescriptors
![Page 27: Beyond bags of features: Part-based models Many slides adapted from Fei-Fei Li, Rob Fergus, and Antonio Torralba.](https://reader036.fdocuments.in/reader036/viewer/2022062313/56649d2c5503460f94a02a26/html5/thumbnails/27.jpg)
Generative part-based model
)|(),|(),|(max
)|,()|(
objecthpobjecthshapepobjecthappearanceP
objectshapeappearancePobjectimageP
h
2D image space
Distribution over jointpart positions
![Page 28: Beyond bags of features: Part-based models Many slides adapted from Fei-Fei Li, Rob Fergus, and Antonio Torralba.](https://reader036.fdocuments.in/reader036/viewer/2022062313/56649d2c5503460f94a02a26/html5/thumbnails/28.jpg)
Results: Faces
Faceshapemodel
Patchappearancemodel
Recognitionresults
![Page 29: Beyond bags of features: Part-based models Many slides adapted from Fei-Fei Li, Rob Fergus, and Antonio Torralba.](https://reader036.fdocuments.in/reader036/viewer/2022062313/56649d2c5503460f94a02a26/html5/thumbnails/29.jpg)
Results: Motorbikes and airplanes
![Page 30: Beyond bags of features: Part-based models Many slides adapted from Fei-Fei Li, Rob Fergus, and Antonio Torralba.](https://reader036.fdocuments.in/reader036/viewer/2022062313/56649d2c5503460f94a02a26/html5/thumbnails/30.jpg)
Pictorial structures
P. Felzenszwalb and D. Huttenlocher, Pictorial Structures for Object Recognition, IJCV 61(1), 2005
• Set of parts (oriented rectangles) connected by edges
• Recognition problem: find the most probable part layout l1, …, ln in the image
![Page 31: Beyond bags of features: Part-based models Many slides adapted from Fei-Fei Li, Rob Fergus, and Antonio Torralba.](https://reader036.fdocuments.in/reader036/viewer/2022062313/56649d2c5503460f94a02a26/html5/thumbnails/31.jpg)
Pictorial structures
• MAP formulation: maximize posterior
• Energy-based formulation: minimize minus the log of probability:
Eji
jii
innn llPlPllPllPllP,
111 )|())(Im(),...,(),...,|(ImIm)|,...,(
i ji
jiijiin lldlmllE,
1 ),()(),...,(
Appearance Geometry
Matching cost
Deformation cost
![Page 32: Beyond bags of features: Part-based models Many slides adapted from Fei-Fei Li, Rob Fergus, and Antonio Torralba.](https://reader036.fdocuments.in/reader036/viewer/2022062313/56649d2c5503460f94a02a26/html5/thumbnails/32.jpg)
Pictorial structures: Complexity
• Suppose there are n parts, and each has h possible positions in the image
• What is the complexity of finding the optimal layout?
• Brute force search: O(hn)• Tree-structured model: O(h2n)• Deformation cost is quadratic: O(hn)
i ji
jiijiill lldlmn
,,, ),()(minarg
1