Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 ·...
Transcript of Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 ·...
![Page 1: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/1.jpg)
Interest Points
Computer Vision
Jia-Bin Huang, Virginia Tech
Many slides from N Snavely, K. Grauman & Leibe, and D. Hoiem
![Page 2: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/2.jpg)
Administrative Stuffs
• HW 1 posted, due 11:59 PM Sept 19• Submission through Canvas
• Frequently Asked Questions for HW 1 posted on piazza
• Reading - Szeliski: 4.1
![Page 3: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/3.jpg)
What have we learned so far?• Light and color
– What an image records
• Filtering in spatial domain• Filtering = weighted sum of neighboring pixels• Smoothing, sharpening, measuring texture
• Filtering in frequency domain• Filtering = change frequency of the input image• Denoising, sampling, image compression
• Image pyramid and template matching• Filtering = a way to find a template• Image pyramids for coarse-to-fine search and
multi-scale detection
• Edge detection• Canny edge = smooth -> derivative -> thin ->
threshold -> link• Finding straight lines, binary image analysis
![Page 4: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/4.jpg)
This module:Correspondence and Alignment• Correspondence:
matching points, patches, edges, or regions across images
≈
Slide credit: Derek Hoiem
![Page 5: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/5.jpg)
This module:Correspondence and Alignment• Alignment/registration:
solving the transformation that makes two things match better
Slide credit: Derek Hoiem
![Page 6: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/6.jpg)
The three biggest problems in computer vision?
1. Registration
2. Registration
3. Registration
Takeo Kanade (CMU)
![Page 7: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/7.jpg)
Example: automatic panoramas
Credit: Matt Brown
![Page 8: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/8.jpg)
AExample: fitting an 2D shape template
Slide from Silvio Savarese
![Page 9: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/9.jpg)
Example: fitting a 3D object model
Slide from Silvio Savarese
![Page 10: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/10.jpg)
Example: estimating “fundamental matrix” that corresponds two views
Slide from Silvio Savarese
![Page 11: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/11.jpg)
Example: tracking points
frame 0 frame 22 frame 49
x xx
![Page 12: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/12.jpg)
Today’s class
• What is interest point?
• Corner detection
• Handling scale and orientation
• Feature matching
![Page 13: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/13.jpg)
Why extract features?
• Motivation: panorama stitching• We have two images – how do we combine them?
![Page 14: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/14.jpg)
Why extract features?
• Motivation: panorama stitching• We have two images – how do we combine them?
Step 1: extract featuresStep 2: match features
![Page 15: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/15.jpg)
Why extract features?
• Motivation: panorama stitching• We have two images – how do we combine them?
Step 1: extract featuresStep 2: match featuresStep 3: align images
![Page 16: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/16.jpg)
Image matching
by Diva Sian
by swashford
![Page 17: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/17.jpg)
Harder case
by Diva Sian by scgbt
![Page 18: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/18.jpg)
Harder still?
![Page 19: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/19.jpg)
NASA Mars Rover imageswith SIFT feature matches
Answer below (look for tiny colored squares…)
![Page 20: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/20.jpg)
Applications
• Keypoints are used for:• Image alignment
• 3D reconstruction
• Motion tracking
• Robot navigation
• Indexing and database retrieval
• Object recognition
![Page 21: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/21.jpg)
Advantages of local features
Locality
• features are local, so robust to occlusion and clutter
Quantity
• hundreds or thousands in a single image
Distinctiveness:
• can differentiate a large database of objects
Efficiency
• real-time performance achievable
![Page 22: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/22.jpg)
Overview of Keypoint Matching
K. Grauman, B. Leibe
AfBf
A1
A2 A3
Tffd BA ),(
1. Find a set of distinctive key-points
3. Extract and normalize the region content
2. Define a region around each keypoint
4. Compute a local descriptor from the normalized region
5. Match local descriptors
![Page 23: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/23.jpg)
Goals for Keypoints
Detect points that are repeatable and distinctive
![Page 24: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/24.jpg)
Key trade-offs
More Repeatable More Points
A1
A2 A3
Detection
More Distinctive More Flexible
Description
Robust to occlusionWorks with less texture
Minimize wrong matches Robust to expected variationsMaximize correct matches
Robust detectionPrecise localization
![Page 25: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/25.jpg)
Choosing interest points
Where would you tell your friend to meet you?
![Page 26: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/26.jpg)
Choosing interest points
Where would you tell your friend to meet you?
![Page 27: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/27.jpg)
Corner Detection: Basic Idea
“flat” region:no change in all directions
“edge”: no change along the edge direction
“corner”:significant change in all directions
• How does the window change when you shift it?
• Shifting the window in any direction causes a big change
Credit: S. Seitz, D. Frolova, D. Simakov
![Page 28: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/28.jpg)
Consider shifting the window W by (u,v)
• how do the pixels in W change?
• compare each pixel before and after bysumming up the squared differences (SSD)
• this defines an SSD “error” E(u,v):
Harris corner detection: the math
W
![Page 29: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/29.jpg)
Taylor Series expansion of I:
If the motion (u,v) is small, then first order approximation is good
Plugging this into the formula on the previous slide…
Small motion assumption
![Page 30: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/30.jpg)
Harris corner detection: the math
Using the small motion assumption, replace I with a linear approximation
W
(Shorthand: )
![Page 31: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/31.jpg)
Corner detection: the math
W
• Thus, E(u,v) is locally approximated as a quadratic form
![Page 32: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/32.jpg)
The surface E(u,v) is locally approximated by a quadratic form.
The second moment matrix
Let’s try to understand its shape.
![Page 33: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/33.jpg)
Horizontal edge:
uv
E(u,v)
![Page 34: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/34.jpg)
Vertical edge:
uv
E(u,v)
![Page 35: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/35.jpg)
General caseThe shape of H tells us something about the distribution of gradients around a pixel
We can visualize H as an ellipse with axis lengths determined by the eigenvalues of H and orientation determined by the eigenvectors of H
direction of the slowest change
direction of the fastest change
(max)-1/2
(min)-1/2
const][
v
uHvu
Ellipse equation:max, min : eigenvalues of H
![Page 36: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/36.jpg)
Quick eigenvalue/eigenvector review
The eigenvectors of a matrix A are the vectors x that satisfy:
The scalar is the eigenvalue corresponding to x• The eigenvalues are found by solving:
• In our case, A = H is a 2x2 matrix, so we have
• The solution:
Once you know , you find x by solving
![Page 37: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/37.jpg)
Corner detection: the math
Eigenvalues and eigenvectors of H
• Define shift directions with the smallest and largest change in error
• xmax = direction of largest increase in E
• max = amount of increase in direction xmax
• xmin = direction of smallest increase in E
• min = amount of increase in direction xmin
xmin
xmax
![Page 38: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/38.jpg)
Corner detection: the mathHow are max, xmax, min, and xmin relevant for feature detection?
• What’s our feature scoring function?
![Page 39: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/39.jpg)
Corner detection: the mathHow are max, xmax, min, and xmin relevant for feature detection?
• What’s our feature scoring function?
Want E(u,v) to be large for small shifts in all directions
• the minimum of E(u,v) should be large, over all unit vectors [u v]
• this minimum is given by the smaller eigenvalue (min) of H
![Page 40: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/40.jpg)
Interpreting the eigenvalues
1
2
“Corner”
1 and 2 are large,
1 ~ 2;
E increases in all
directions
1 and 2 are small;
E is almost constant
in all directions
“Edge”
1 >> 2
“Edge”
2 >> 1
“Flat”
region
Classification of image points using eigenvalues of M:
![Page 41: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/41.jpg)
Corner detection summaryHere’s what you do
• Compute the gradient at each point in the image
• Create the H matrix from the entries in the gradient
• Compute the eigenvalues.
• Find points with large response (min > threshold)
• Choose those points where min is a local maximum as features
![Page 42: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/42.jpg)
Corner detection summaryHere’s what you do
• Compute the gradient at each point in the image
• Create the H matrix from the entries in the gradient
• Compute the eigenvalues.
• Find points with large response (min > threshold)
• Choose those points where min is a local maximum as features
![Page 43: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/43.jpg)
The Harris operator
min is a variant of the “Harris operator” for feature detection
• The trace is the sum of the diagonals, i.e., trace(H) = h11 + h22
• Very similar to min but less expensive (no square root)
• Called the “Harris Corner Detector” or “Harris Operator”
• Lots of other detectors, this is one of the most popular
![Page 44: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/44.jpg)
The Harris operator
Harris operator
![Page 45: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/45.jpg)
Harris detector example
![Page 46: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/46.jpg)
![Page 47: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/47.jpg)
f value (red high, blue low)
![Page 48: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/48.jpg)
Threshold (f > value)
![Page 49: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/49.jpg)
Find local maxima of f
![Page 50: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/50.jpg)
Harris features (in red)
![Page 51: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/51.jpg)
Weighting the derivatives
• In practice, using a simple window W doesn’t work too well
• Instead, we’ll weight each derivative value based on its distance from the center pixel
![Page 52: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/52.jpg)
Harris Detector – Responses [Harris88]
Effect: A very precise corner detector.
![Page 53: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/53.jpg)
Harris Detector – Responses [Harris88]
![Page 54: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/54.jpg)
Harris Detector: Invariance Properties• Rotation
Ellipse rotates but its shape (i.e. eigenvalues) remains the same
Corner response is invariant to image rotation
![Page 55: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/55.jpg)
Harris Detector: Invariance Properties• Affine intensity change: I aI + b
Only derivatives are used => invariance to intensity shift I I + b
Intensity scale: I a I
R
x (image coordinate)
threshold
R
x (image coordinate)
Partially invariant to affine intensity change
![Page 56: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/56.jpg)
Harris Detector: Invariance Properties• Scaling
All points will be classified as edges
Corner
Not invariant to scaling
![Page 57: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/57.jpg)
Scale invariant detectionSuppose you’re looking for corners
Key idea: find scale that gives local maximum of f• in both position and scale• One definition of f: the Harris operator
![Page 58: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/58.jpg)
Automatic Scale Selection
K. Grauman, B. Leibe
)),(( )),((11
xIfxIfmm iiii
How to find corresponding patch sizes?
![Page 59: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/59.jpg)
Automatic Scale Selection• Function responses for increasing scale (scale signature)
K. Grauman, B. Leibe
)),((1
xIfmii
)),((1
xIfmii
![Page 60: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/60.jpg)
Automatic Scale Selection
K. Grauman, B. Leibe
)),((1
xIfmii
)),((1
xIfmii
• Function responses for increasing scale (scale signature)
![Page 61: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/61.jpg)
Automatic Scale Selection
K. Grauman, B. Leibe
)),((1
xIfmii
)),((1
xIfmii
• Function responses for increasing scale (scale signature)
![Page 62: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/62.jpg)
Automatic Scale Selection
K. Grauman, B. Leibe
)),((1
xIfmii
)),((1
xIfmii
• Function responses for increasing scale (scale signature)
![Page 63: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/63.jpg)
Automatic Scale Selection
K. Grauman, B. Leibe
)),((1
xIfmii
)),((1
xIfmii
• Function responses for increasing scale (scale signature)
![Page 64: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/64.jpg)
Automatic Scale Selection
K. Grauman, B. Leibe
)),((1
xIfmii
)),((1
xIfmii
• Function responses for increasing scale (scale signature)
![Page 65: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/65.jpg)
Implementation
• Instead of computing f for larger and larger windows, we can implement using a fixed window size with a Gaussian pyramid
(sometimes need to create in-between levels, e.g. a ¾-size image)
![Page 66: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/66.jpg)
What Is A Useful Signature Function?
• Difference-of-Gaussian = “blob” detector
K. Grauman, B. Leibe
![Page 67: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/67.jpg)
Difference-of-Gaussian (DoG)
K. Grauman, B. Leibe
- =
![Page 68: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/68.jpg)
DoG – Efficient Computation• Computation in Gaussian scale pyramid
K. Grauman, B. Leibe
Original image4
1
2
Sampling withstep 4 =2
![Page 69: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/69.jpg)
Find local maxima in position-scale space of Difference-of-Gaussian
K. Grauman, B. Leibe
)()( yyxx LL
2
3
4
5
List of(x, y, s)
![Page 70: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/70.jpg)
Results: Difference-of-Gaussian
K. Grauman, B. Leibe
![Page 71: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/71.jpg)
T. Tuytelaars, B. Leibe
Orientation Normalization
• Compute orientation histogram
• Select dominant orientation
• Normalize: rotate to fixed orientation
0 2p
[Lowe, SIFT, 1999]
![Page 72: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/72.jpg)
Maximally Stable Extremal Regions [Matas ‘02]
• Based on Watershed segmentation algorithm
• Select regions that stay stable over a large parameter range
K. Grauman, B. Leibe
![Page 73: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/73.jpg)
Example Results: MSER
73 K. Grauman, B. Leibe
![Page 74: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/74.jpg)
Available at a web site near you…
• For most local feature detectors, executables are available online:– http://www.robots.ox.ac.uk/~vgg/research/affine
– http://www.cs.ubc.ca/~lowe/keypoints/
– http://www.vision.ee.ethz.ch/~surf
K. Grauman, B. Leibe
![Page 75: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/75.jpg)
Local Descriptors
• The ideal descriptor should be– Robust
– Distinctive
– Compact
– Efficient
• Most available descriptors focus on edge/gradient information– Capture texture information
– Color rarely used
K. Grauman, B. Leibe
![Page 76: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/76.jpg)
Basic idea:
• Take 16x16 square window around detected feature
• Compute edge orientation (angle of the gradient - 90) for each pixel
• Throw out weak edges (threshold gradient magnitude)
• Create histogram of surviving edge orientations
Scale Invariant Feature Transform
Adapted from slide by David Lowe
0 2p
angle histogram
![Page 77: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/77.jpg)
SIFT descriptorFull version
• Divide the 16x16 window into a 4x4 grid of cells (2x2 case shown below)
• Compute an orientation histogram for each cell
• 16 cells * 8 orientations = 128 dimensional descriptor
Adapted from slide by David Lowe
![Page 78: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/78.jpg)
Local Descriptors: SIFT Descriptor
[Lowe, ICCV 1999]
Histogram of oriented
gradients
• Captures important texture
information
• Robust to small translations /
affine deformationsK. Grauman, B. Leibe
![Page 79: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/79.jpg)
Details of Lowe’s SIFT algorithm
• Run DoG detector– Find maxima in location/scale space– Remove edge points
• Find all major orientations– Bin orientations into 36 bin histogram
• Weight by gradient magnitude• Weight by distance to center (Gaussian-weighted mean)
– Return orientations within 0.8 of peak• Use parabola for better orientation fit
• For each (x,y,scale,orientation), create descriptor:– Sample 16x16 gradient mag. and rel. orientation– Bin 4x4 samples into 4x4 histograms– Threshold values to max of 0.2, divide by L2 norm– Final descriptor: 4x4x8 normalized histograms
Lowe IJCV 2004
![Page 80: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/80.jpg)
SIFT Example
sift
868 SIFT features
![Page 81: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/81.jpg)
Feature matching
Given a feature in I1, how to find the best match in I2?1. Define distance function that compares two descriptors
2. Test all the features in I2, find the one with min distance
![Page 82: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/82.jpg)
Feature distance
How to define the difference between two features f1, f2?• Simple approach: L2 distance, ||f1 - f2 ||
• can give good scores to ambiguous (incorrect) matches
I1 I2
f1 f2
![Page 83: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/83.jpg)
f1 f2f2'
How to define the difference between two features f1, f2?• Better approach: ratio distance = ||f1 - f2 || / || f1 - f2’ ||
• f2 is best SSD match to f1 in I2
• f2’ is 2nd best SSD match to f1 in I2
• gives large values for ambiguous matches
I1 I2
Feature distance
![Page 84: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/84.jpg)
Feature matching example
51 matches
![Page 85: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/85.jpg)
Feature matching example
58 matches
![Page 86: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/86.jpg)
Evaluating the resultsHow can we measure the performance of a feature matcher?
50
75
200
feature distance
![Page 87: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/87.jpg)
True/false positives
The distance threshold affects performance• True positives = # of detected matches that are correct
• Suppose we want to maximize these—how to choose threshold?
• False positives = # of detected matches that are incorrect• Suppose we want to minimize these—how to choose threshold?
50
75
200false match
true match
feature distance
How can we measure the performance of a feature matcher?
![Page 88: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/88.jpg)
0.7
Evaluating the results
0 1
1
false positive rate
true
positive
rate
# true positives
# correctly matched features (positives)
0.1
How can we measure the performance of a feature matcher?
“recall”
# false positives
# incorrectly matched features (negatives)
1 - “precision”
![Page 89: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/89.jpg)
0.7
Evaluating the results
0 1
1
false positive rate
true
positive
rate
0.1
ROC curve (“Receiver Operator Characteristic”)
How can we measure the performance of a feature matcher?
# true positives
# correctly matched features (positives)
“recall”
# false positives
# incorrectly matched features (negatives)
1 - “precision”
![Page 90: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/90.jpg)
Matching SIFT Descriptors
• Nearest neighbor (Euclidean distance)
• Threshold ratio of nearest to 2nd nearest descriptor
Lowe IJCV 2004
![Page 91: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/91.jpg)
SIFT Repeatability
Lowe IJCV 2004
![Page 92: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/92.jpg)
SIFT Repeatability
![Page 93: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/93.jpg)
SIFT Repeatability
Lowe IJCV 2004
![Page 94: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/94.jpg)
Local Descriptors: SURF
K. Grauman, B. Leibe
• Fast approximation of SIFT idea Efficient computation by 2D box filters &
integral images 6 times faster than SIFT
Equivalent quality for object identification
[Bay, ECCV’06], [Cornelis, CVGPU’08]
• GPU implementation available Feature extraction @ 200Hz
(detector + descriptor, 640×480 img)
http://www.vision.ee.ethz.ch/~surf
Many other efficient descriptors are also available
![Page 95: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/95.jpg)
Local Descriptors: Shape Context
Count the number of points inside each bin, e.g.:
Count = 4
Count = 10...
Log-polar binning: more precision for nearby points, more flexibility for farther points.
Belongie & Malik, ICCV 2001K. Grauman, B. Leibe
![Page 96: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/96.jpg)
Local Descriptors: Geometric Blur
Example descriptor
~
Compute edges
at four
orientations
Extract a patch
in each channel
Apply spatially varying
blur and sub-sample
(Idealized signal)
Berg & Malik, CVPR 2001K. Grauman, B. Leibe
![Page 97: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/97.jpg)
Choosing a detector• What do you want it for?
– Precise localization in x-y: Harris– Good localization in scale: Difference of Gaussian– Flexible region shape: MSER
• Best choice often application dependent– Harris-/Hessian-Laplace/DoG work well for many natural categories– MSER works well for buildings and printed things
• Why choose?– Get more points with more detectors
• There have been extensive evaluations/comparisons– [Mikolajczyk et al., IJCV’05, PAMI’05]– All detectors/descriptors shown here work well
![Page 98: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/98.jpg)
Comparison of Keypoint Detectors
Tuytelaars Mikolajczyk 2008
![Page 99: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/99.jpg)
Choosing a descriptor
• Again, need not stick to one
• For object instance recognition or stitching, SIFT or variant is a good choice
![Page 100: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/100.jpg)
Recent advances in interest points
Features from Accelerated Segment Test, ECCV 06
Binary feature descriptors
• BRIEF: Binary Robust Independent Elementary Features, ECCV 10• ORB (Oriented FAST and Rotated BRIEF), CVPR 11• BRISK: Binary robust invariant scalable keypoints, ICCV 11• Freak: Fast retina keypoint, CVPR 12• LIFT: Learned Invariant Feature Transform, ECCV 16
![Page 101: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/101.jpg)
Things to remember• Keypoint detection: repeatable
and distinctive• Corners, blobs, stable regions
• Harris, DoG
• Descriptors: robust and selective• spatial histograms of orientation
• SIFT
![Page 102: Introduction to Computer Vision - Virginia Techjbhuang/teaching/ece5554... · 2016-09-12 · Computer Vision Jia-Bin Huang, Virginia Tech Many slides from N Snavely, K. Grauman &](https://reader031.fdocuments.in/reader031/viewer/2022013021/5f0d77507e708231d43a7ed6/html5/thumbnails/102.jpg)
Thank you
• Next class: feature tracking and optical flow