Recognition Using SIFT Features
description
Transcript of Recognition Using SIFT Features
![Page 1: Recognition Using SIFT Features](https://reader033.fdocuments.in/reader033/viewer/2022061602/56815a6e550346895dc7cea8/html5/thumbnails/1.jpg)
Recognition Using SIFT Features
CS491Y/691Y Topics in Computer Vision Dr. George Bebis
![Page 2: Recognition Using SIFT Features](https://reader033.fdocuments.in/reader033/viewer/2022061602/56815a6e550346895dc7cea8/html5/thumbnails/2.jpg)
Object Recognition
• Model-based Object Recognition
• Generic Object Recognition
![Page 3: Recognition Using SIFT Features](https://reader033.fdocuments.in/reader033/viewer/2022061602/56815a6e550346895dc7cea8/html5/thumbnails/3.jpg)
Model-Based Object Recognition
• Recognition relies upon the existence of a set of predefined objects.
![Page 4: Recognition Using SIFT Features](https://reader033.fdocuments.in/reader033/viewer/2022061602/56815a6e550346895dc7cea8/html5/thumbnails/4.jpg)
Generic Object Recognition (or Object Categorization)
![Page 5: Recognition Using SIFT Features](https://reader033.fdocuments.in/reader033/viewer/2022061602/56815a6e550346895dc7cea8/html5/thumbnails/5.jpg)
Feature-based Recognition
(1) Identify a group of features from an unknown scene which approximately match a group of features from a model object (i.e., correspondences).
(2) Recover the geometric transformation that the model object has undergone and look for additional matches.
model scene
![Page 6: Recognition Using SIFT Features](https://reader033.fdocuments.in/reader033/viewer/2022061602/56815a6e550346895dc7cea8/html5/thumbnails/6.jpg)
2D Transformation Spaces
• Rigid transformations (3 parameters)
• Similarity transformations (4 parameters)
• Affine transformations (6 parameters)
![Page 7: Recognition Using SIFT Features](https://reader033.fdocuments.in/reader033/viewer/2022061602/56815a6e550346895dc7cea8/html5/thumbnails/7.jpg)
Matching – Main Steps• Hypothesis generation: the identities of one or more models
are hypothesized.• Hypothesis verification: tests are performed to check if a
given hypothesis is correct or not.
Recognition
![Page 8: Recognition Using SIFT Features](https://reader033.fdocuments.in/reader033/viewer/2022061602/56815a6e550346895dc7cea8/html5/thumbnails/8.jpg)
Hypothesis Verification - Example
Features might correspond to:
(1) curvature extrema orzero-crossings alongthe boundary of anobject .
(2) interest points.
two hypotheses:
![Page 9: Recognition Using SIFT Features](https://reader033.fdocuments.in/reader033/viewer/2022061602/56815a6e550346895dc7cea8/html5/thumbnails/9.jpg)
Object Recognition using SIFT features
1. Match individual SIFT features from an image to a database of SIFT features from known objects (i.e., find nearest neighbors).
2. Find clusters of SIFT features belonging to a single object (hypothesis generation).
![Page 10: Recognition Using SIFT Features](https://reader033.fdocuments.in/reader033/viewer/2022061602/56815a6e550346895dc7cea8/html5/thumbnails/10.jpg)
Object Recognition using SIFT features
3. Estimate object pose (i.e., recover the transformation that the model has undergone) using at least three matches.
4. Verify that additional features agree on object pose (hypothesis verification).
![Page 11: Recognition Using SIFT Features](https://reader033.fdocuments.in/reader033/viewer/2022061602/56815a6e550346895dc7cea8/html5/thumbnails/11.jpg)
Nearest neighbor search
• Linear search: too slow for large database • kD trees: become slow when k > 10
![Page 12: Recognition Using SIFT Features](https://reader033.fdocuments.in/reader033/viewer/2022061602/56815a6e550346895dc7cea8/html5/thumbnails/12.jpg)
Nearest neighbor search (cont’d)
• Approximate nearest neighbor search:– Best-bin-first [Beis et al. 97] (modification to kD-tree algorithm)– Examine only the N closest bins of the kD-tree– Use a heap to identify bins in order by their distance from query.
• Can give speedup by factor of 1000 while finding nearest neighbor (of interest) 95% of the time.
Marius Muja and David G. Lowe, "Fast Approximate Nearest Neighbors with Automatic Algorithm Configuration", International Conference on Computer Vision Theory and Applications, 2009
FLANN - Fast Library for Approximate Nearest Neighbors
![Page 13: Recognition Using SIFT Features](https://reader033.fdocuments.in/reader033/viewer/2022061602/56815a6e550346895dc7cea8/html5/thumbnails/13.jpg)
Estimate object pose
• Now, given feature matches…– Find clusters of features corresponding to a single object.– Solve for transformation (e.g., affine transformation).
model scene
![Page 14: Recognition Using SIFT Features](https://reader033.fdocuments.in/reader033/viewer/2022061602/56815a6e550346895dc7cea8/html5/thumbnails/14.jpg)
Estimate object pose (cont’d)
• Need to consider clusters of size >=3• How do we find three “good” (i.e., true) matches?
![Page 15: Recognition Using SIFT Features](https://reader033.fdocuments.in/reader033/viewer/2022061602/56815a6e550346895dc7cea8/html5/thumbnails/15.jpg)
Estimate object pose (cont’d)• Pose clustering
– Each SIFT feature is associated with four parameters: – For every match (mi, sj), estimate a similarity
transformation between mi and sj
(2D location, scale, orientation)(2D location, scale, orientation)
(tx,ty,s,θ)
vote
![Page 16: Recognition Using SIFT Features](https://reader033.fdocuments.in/reader033/viewer/2022061602/56815a6e550346895dc7cea8/html5/thumbnails/16.jpg)
Estimate object pose (cont’d)
– Transformation space is 4D: (tx, ty, s, θ)
(tx,ty,s,θ) (t’x,t’y,s’,θ’) ….
votes
![Page 17: Recognition Using SIFT Features](https://reader033.fdocuments.in/reader033/viewer/2022061602/56815a6e550346895dc7cea8/html5/thumbnails/17.jpg)
Estimate object pose (cont’d)
– Partial voting: vote for neighboring bins and use
large bin size to better tolerate errors.
– Transformations that accumulate at least three votes are selected (hypothesis generation).
– Using model-scene matches, compute object pose (i.e., affine transformation) and apply verification.
![Page 18: Recognition Using SIFT Features](https://reader033.fdocuments.in/reader033/viewer/2022061602/56815a6e550346895dc7cea8/html5/thumbnails/18.jpg)
Verification• Back-project model (i.e., interest points) on the scene and
look for additional matches.
• Discard outliers (i.e., incorrect matches) by imposing stricter matching constraints (e.g., half error).
• Find additional matches and refine the transformation computed (i.e., iterative affine refinements).
• Repeat until no additional matches can be found.
![Page 19: Recognition Using SIFT Features](https://reader033.fdocuments.in/reader033/viewer/2022061602/56815a6e550346895dc7cea8/html5/thumbnails/19.jpg)
Verification (cont’d)• Additional verification: evaluate probability that match
is correct.
– Use Bayesian (probabilistic) model, to estimate the probability that a model is present based on the matching features.
– Bayesian model takes into account: • Object size in image• Textured regions• Model feature count in database• Accuracy of fit
Lowe, D.G. 2001. Local feature view clustering for 3D object recognition.IEEE Conference on Computer Vision and Pattern Recognition, Kauai, Hawaii, pp. 682–688.
![Page 20: Recognition Using SIFT Features](https://reader033.fdocuments.in/reader033/viewer/2022061602/56815a6e550346895dc7cea8/html5/thumbnails/20.jpg)
Planar recognition
Models
![Page 21: Recognition Using SIFT Features](https://reader033.fdocuments.in/reader033/viewer/2022061602/56815a6e550346895dc7cea8/html5/thumbnails/21.jpg)
Planar recognition (cont’d)
• Reliably recognized at a rotation of 60° away from the camera.
• Affine fit approximates perspective projection.
• Only 3 points are needed for recognition.
Test image
![Page 22: Recognition Using SIFT Features](https://reader033.fdocuments.in/reader033/viewer/2022061602/56815a6e550346895dc7cea8/html5/thumbnails/22.jpg)
3D object recognitionModels
![Page 23: Recognition Using SIFT Features](https://reader033.fdocuments.in/reader033/viewer/2022061602/56815a6e550346895dc7cea8/html5/thumbnails/23.jpg)
3D object recognition
• Only 3 keys are needed for recognition; extra keys provide robustness.
• Affine model is no longer as accurate.
Test images
![Page 24: Recognition Using SIFT Features](https://reader033.fdocuments.in/reader033/viewer/2022061602/56815a6e550346895dc7cea8/html5/thumbnails/24.jpg)
Recognition under occlusion
![Page 25: Recognition Using SIFT Features](https://reader033.fdocuments.in/reader033/viewer/2022061602/56815a6e550346895dc7cea8/html5/thumbnails/25.jpg)
Object Categorization
![Page 26: Recognition Using SIFT Features](https://reader033.fdocuments.in/reader033/viewer/2022061602/56815a6e550346895dc7cea8/html5/thumbnails/26.jpg)
Bag-of-features (BoF) model
![Page 27: Recognition Using SIFT Features](https://reader033.fdocuments.in/reader033/viewer/2022061602/56815a6e550346895dc7cea8/html5/thumbnails/27.jpg)
Origin 1: Texture recognition• Texture is characterized by the repetition of basic elements
or textons.• Many times, it is the identity of the textons, not their
spatial arrangement, that matters.
![Page 28: Recognition Using SIFT Features](https://reader033.fdocuments.in/reader033/viewer/2022061602/56815a6e550346895dc7cea8/html5/thumbnails/28.jpg)
Origin 1: Texture recognition
Universal texton dictionary
histogram
Universal texton dictionary
histogram
![Page 29: Recognition Using SIFT Features](https://reader033.fdocuments.in/reader033/viewer/2022061602/56815a6e550346895dc7cea8/html5/thumbnails/29.jpg)
Origin 2: Document retrieval• Orderless document representation: frequencies of words from a dictionary Salton & McGill (1983)
![Page 30: Recognition Using SIFT Features](https://reader033.fdocuments.in/reader033/viewer/2022061602/56815a6e550346895dc7cea8/html5/thumbnails/30.jpg)
BoF for object categorization
G. Cruska et al., "Visual Categorization with Bags of Keypoints", European Conference on Computer Vision, Czech Republic, 2004
Need a “visual” dictionary!
![Page 31: Recognition Using SIFT Features](https://reader033.fdocuments.in/reader033/viewer/2022061602/56815a6e550346895dc7cea8/html5/thumbnails/31.jpg)
BoF: main steps
Characterize objects in terms of parts or local features
![Page 32: Recognition Using SIFT Features](https://reader033.fdocuments.in/reader033/viewer/2022061602/56815a6e550346895dc7cea8/html5/thumbnails/32.jpg)
BoF: main steps
Step 1: Feature extraction (e.g., SIFT features)
…
![Page 33: Recognition Using SIFT Features](https://reader033.fdocuments.in/reader033/viewer/2022061602/56815a6e550346895dc7cea8/html5/thumbnails/33.jpg)
BoF: main steps (cont’d)
Step 2: Learn “visual” vocabulary
…
“visual” vocabulary
Feature extraction & clustering
![Page 34: Recognition Using SIFT Features](https://reader033.fdocuments.in/reader033/viewer/2022061602/56815a6e550346895dc7cea8/html5/thumbnails/34.jpg)
BoF: main steps (cont’d)
Clustering
…
![Page 35: Recognition Using SIFT Features](https://reader033.fdocuments.in/reader033/viewer/2022061602/56815a6e550346895dc7cea8/html5/thumbnails/35.jpg)
BoF: main steps (cont’d)
Clustering
…
“Visual” vocabulary: cluster centers
![Page 36: Recognition Using SIFT Features](https://reader033.fdocuments.in/reader033/viewer/2022061602/56815a6e550346895dc7cea8/html5/thumbnails/36.jpg)
Example: K-means clustering
•Want to minimize sum of squared Euclidean distances between points xi and their nearest cluster centers mk
![Page 37: Recognition Using SIFT Features](https://reader033.fdocuments.in/reader033/viewer/2022061602/56815a6e550346895dc7cea8/html5/thumbnails/37.jpg)
Example: K-means clustering
Algorithm:•Randomly initialize K cluster centers•Iterate until convergence:
– Assign each data point to the nearest center.– Re-compute each cluster center as the mean of all points
assigned to it.
![Page 38: Recognition Using SIFT Features](https://reader033.fdocuments.in/reader033/viewer/2022061602/56815a6e550346895dc7cea8/html5/thumbnails/38.jpg)
More powerful clustering algorithms
•Affinity propagation http://www.psi.toronto.edu/index.php?q=affinity%20propagation
•Autoclasshttp://ti.arc.nasa.gov/tech/rse/synthesis-projects-applications/autoclass/autoclass-c/
![Page 39: Recognition Using SIFT Features](https://reader033.fdocuments.in/reader033/viewer/2022061602/56815a6e550346895dc7cea8/html5/thumbnails/39.jpg)
BoF: main steps (cont’d)Step 3: Quantize features using “visual” vocabulary (i.e., represent each feature by the closest cluster center).
…
![Page 40: Recognition Using SIFT Features](https://reader033.fdocuments.in/reader033/viewer/2022061602/56815a6e550346895dc7cea8/html5/thumbnails/40.jpg)
BoF: main steps (cont’d)
Step 4: Represent images by frequencies of “visual words” (i.e., bags of features)
![Page 41: Recognition Using SIFT Features](https://reader033.fdocuments.in/reader033/viewer/2022061602/56815a6e550346895dc7cea8/html5/thumbnails/41.jpg)
BoF Object Categorization
• How do we use BoF for object categorization?
![Page 42: Recognition Using SIFT Features](https://reader033.fdocuments.in/reader033/viewer/2022061602/56815a6e550346895dc7cea8/html5/thumbnails/42.jpg)
BoF Object Categorization (cont’d)
(1) Use a Nearest Neighbor (NN) Classifier
![Page 43: Recognition Using SIFT Features](https://reader033.fdocuments.in/reader033/viewer/2022061602/56815a6e550346895dc7cea8/html5/thumbnails/43.jpg)
BoF Object Categorization (cont’d)
• Functions for comparing histograms
![Page 44: Recognition Using SIFT Features](https://reader033.fdocuments.in/reader033/viewer/2022061602/56815a6e550346895dc7cea8/html5/thumbnails/44.jpg)
BoF Object Categorization (cont’d)
(2) Use a K-Nearest Neighbor (KNN) Classifier
Find the k closest points from training data.
Labels of the k points “vote” to classify.
Works well provided there is lots of data and the distance function is good.
![Page 45: Recognition Using SIFT Features](https://reader033.fdocuments.in/reader033/viewer/2022061602/56815a6e550346895dc7cea8/html5/thumbnails/45.jpg)
BoF Object Categorization (cont’d)
(3) Naïve Bayes classifier
![Page 46: Recognition Using SIFT Features](https://reader033.fdocuments.in/reader033/viewer/2022061602/56815a6e550346895dc7cea8/html5/thumbnails/46.jpg)
BoF Object Categorization (cont’d)
(3) Naïve Bayes classifier
![Page 47: Recognition Using SIFT Features](https://reader033.fdocuments.in/reader033/viewer/2022061602/56815a6e550346895dc7cea8/html5/thumbnails/47.jpg)
BoF Object Categorization (cont’d)
(3) Naïve Bayes classifier
![Page 48: Recognition Using SIFT Features](https://reader033.fdocuments.in/reader033/viewer/2022061602/56815a6e550346895dc7cea8/html5/thumbnails/48.jpg)
BoF Object Categorization (cont’d)
(3) Naïve Bayes classifier
or
![Page 49: Recognition Using SIFT Features](https://reader033.fdocuments.in/reader033/viewer/2022061602/56815a6e550346895dc7cea8/html5/thumbnails/49.jpg)
BoF Object Categorization (cont’d)
(4) Use an SVM classifier
SVM
SVM
SVM
![Page 50: Recognition Using SIFT Features](https://reader033.fdocuments.in/reader033/viewer/2022061602/56815a6e550346895dc7cea8/html5/thumbnails/50.jpg)
ExampleCaltech6 dataset
![Page 51: Recognition Using SIFT Features](https://reader033.fdocuments.in/reader033/viewer/2022061602/56815a6e550346895dc7cea8/html5/thumbnails/51.jpg)
Example (cont’d)Caltech6 dataset
Dictionary quality and size are very important parameters!
![Page 52: Recognition Using SIFT Features](https://reader033.fdocuments.in/reader033/viewer/2022061602/56815a6e550346895dc7cea8/html5/thumbnails/52.jpg)
What about spatial information?
Svetlana Lazebnik, Cordelia Schmid, and Jean Ponce, "Beyond Bags of Features: Spatial Pyramid Matching for Recognizing Natural Scene Categories", CVPR 2006.
![Page 53: Recognition Using SIFT Features](https://reader033.fdocuments.in/reader033/viewer/2022061602/56815a6e550346895dc7cea8/html5/thumbnails/53.jpg)
What about spatial information?Scene category dataset
100 training images per class