Local features and image matching October 1 st 2015 Devi Parikh Virginia Tech Disclaimer: Many...
-
Upload
tiffany-davis -
Category
Documents
-
view
213 -
download
0
description
Transcript of Local features and image matching October 1 st 2015 Devi Parikh Virginia Tech Disclaimer: Many...
![Page 1: Local features and image matching October 1 st 2015 Devi Parikh Virginia Tech Disclaimer: Many slides have been borrowed from Kristen Grauman, who may.](https://reader035.fdocuments.in/reader035/viewer/2022070605/5a4d1aee7f8b9ab05997cb90/html5/thumbnails/1.jpg)
Local features and image matching
October 1st 2015Devi Parikh
Virginia Tech
Disclaimer: Many slides have been borrowed from Kristen Grauman, who may have borrowed some of them from others. Any time a slide did not already have a credit on it, I have credited it to Kristen. So there is a chance some of these credits are inaccurate.
![Page 2: Local features and image matching October 1 st 2015 Devi Parikh Virginia Tech Disclaimer: Many slides have been borrowed from Kristen Grauman, who may.](https://reader035.fdocuments.in/reader035/viewer/2022070605/5a4d1aee7f8b9ab05997cb90/html5/thumbnails/2.jpg)
2
Announcements
• PS2 due Monday
• PS3 out– Due October 19th 11:55 pm
Slide credit: Kristen Grauman
![Page 3: Local features and image matching October 1 st 2015 Devi Parikh Virginia Tech Disclaimer: Many slides have been borrowed from Kristen Grauman, who may.](https://reader035.fdocuments.in/reader035/viewer/2022070605/5a4d1aee7f8b9ab05997cb90/html5/thumbnails/3.jpg)
Topics overview• Features & filters• Grouping & fitting• Multiple views and motion
– Homography and image warping– Local invariant features– Image formation– Epipolar geometry– Stereo and structure from motion
• Recognition• Video processing
3Slide credit: Kristen Grauman
![Page 4: Local features and image matching October 1 st 2015 Devi Parikh Virginia Tech Disclaimer: Many slides have been borrowed from Kristen Grauman, who may.](https://reader035.fdocuments.in/reader035/viewer/2022070605/5a4d1aee7f8b9ab05997cb90/html5/thumbnails/4.jpg)
Numerical Issues
• When computing H– Say true match is [50 100 1] [50 100]– [50.5 100 1]
• [50.5 100]– [50 100 1.5]
• [33 67]– Scale co-ordinates to lie between 0 and 2
4
![Page 5: Local features and image matching October 1 st 2015 Devi Parikh Virginia Tech Disclaimer: Many slides have been borrowed from Kristen Grauman, who may.](https://reader035.fdocuments.in/reader035/viewer/2022070605/5a4d1aee7f8b9ab05997cb90/html5/thumbnails/5.jpg)
Topics overview• Features & filters• Grouping & fitting• Multiple views and motion
– Homography and image warping– Local invariant features– Image formation– Epipolar geometry– Stereo and structure from motion
• Recognition• Video processing
5Slide credit: Kristen Grauman
![Page 6: Local features and image matching October 1 st 2015 Devi Parikh Virginia Tech Disclaimer: Many slides have been borrowed from Kristen Grauman, who may.](https://reader035.fdocuments.in/reader035/viewer/2022070605/5a4d1aee7f8b9ab05997cb90/html5/thumbnails/6.jpg)
Last time
• Image mosaics– Fitting a 2D transformation
• Affine, Homography– 2D image warping– Computing an image mosaic
![Page 7: Local features and image matching October 1 st 2015 Devi Parikh Virginia Tech Disclaimer: Many slides have been borrowed from Kristen Grauman, who may.](https://reader035.fdocuments.in/reader035/viewer/2022070605/5a4d1aee7f8b9ab05997cb90/html5/thumbnails/7.jpg)
Robust feature-based alignment
Source: L. Lazebnik
![Page 8: Local features and image matching October 1 st 2015 Devi Parikh Virginia Tech Disclaimer: Many slides have been borrowed from Kristen Grauman, who may.](https://reader035.fdocuments.in/reader035/viewer/2022070605/5a4d1aee7f8b9ab05997cb90/html5/thumbnails/8.jpg)
Robust feature-based alignment
• Extract features
Source: L. Lazebnik
![Page 9: Local features and image matching October 1 st 2015 Devi Parikh Virginia Tech Disclaimer: Many slides have been borrowed from Kristen Grauman, who may.](https://reader035.fdocuments.in/reader035/viewer/2022070605/5a4d1aee7f8b9ab05997cb90/html5/thumbnails/9.jpg)
Robust feature-based alignment
• Extract features• Compute putative matches
Source: L. Lazebnik
![Page 10: Local features and image matching October 1 st 2015 Devi Parikh Virginia Tech Disclaimer: Many slides have been borrowed from Kristen Grauman, who may.](https://reader035.fdocuments.in/reader035/viewer/2022070605/5a4d1aee7f8b9ab05997cb90/html5/thumbnails/10.jpg)
Robust feature-based alignment
• Extract features• Compute putative matches• Loop:
• Hypothesize transformation T (small group of putative matches that are related by T)
Source: L. Lazebnik
![Page 11: Local features and image matching October 1 st 2015 Devi Parikh Virginia Tech Disclaimer: Many slides have been borrowed from Kristen Grauman, who may.](https://reader035.fdocuments.in/reader035/viewer/2022070605/5a4d1aee7f8b9ab05997cb90/html5/thumbnails/11.jpg)
Robust feature-based alignment
• Extract features• Compute putative matches• Loop:
• Hypothesize transformation T (small group of putative matches that are related by T)
• Verify transformation (search for other matches consistent with T)
Source: L. Lazebnik
![Page 12: Local features and image matching October 1 st 2015 Devi Parikh Virginia Tech Disclaimer: Many slides have been borrowed from Kristen Grauman, who may.](https://reader035.fdocuments.in/reader035/viewer/2022070605/5a4d1aee7f8b9ab05997cb90/html5/thumbnails/12.jpg)
Robust feature-based alignment
• Extract features• Compute putative matches• Loop:
• Hypothesize transformation T (small group of putative matches that are related by T)
• Verify transformation (search for other matches consistent with T)
Source: L. Lazebnik
![Page 13: Local features and image matching October 1 st 2015 Devi Parikh Virginia Tech Disclaimer: Many slides have been borrowed from Kristen Grauman, who may.](https://reader035.fdocuments.in/reader035/viewer/2022070605/5a4d1aee7f8b9ab05997cb90/html5/thumbnails/13.jpg)
Today
How to detect which features to match?
![Page 14: Local features and image matching October 1 st 2015 Devi Parikh Virginia Tech Disclaimer: Many slides have been borrowed from Kristen Grauman, who may.](https://reader035.fdocuments.in/reader035/viewer/2022070605/5a4d1aee7f8b9ab05997cb90/html5/thumbnails/14.jpg)
Detecting local invariant features
• Detection of interest points– Harris corner detection– Scale invariant blob detection: LoG (next
time)• Description of local patches (next time)
![Page 15: Local features and image matching October 1 st 2015 Devi Parikh Virginia Tech Disclaimer: Many slides have been borrowed from Kristen Grauman, who may.](https://reader035.fdocuments.in/reader035/viewer/2022070605/5a4d1aee7f8b9ab05997cb90/html5/thumbnails/15.jpg)
Local features: main components1) Detection: Identify the
interest points
2) Description:Extract vector feature descriptor surrounding each interest point.
3) Matching: Determine correspondence between descriptors in two views
],,[ )1()1(11 dxx x
],,[ )2()2(12 dxx x
Kristen Grauman
![Page 16: Local features and image matching October 1 st 2015 Devi Parikh Virginia Tech Disclaimer: Many slides have been borrowed from Kristen Grauman, who may.](https://reader035.fdocuments.in/reader035/viewer/2022070605/5a4d1aee7f8b9ab05997cb90/html5/thumbnails/16.jpg)
Local features: desired properties
• Repeatability– The same feature can be found in several images
despite geometric and photometric transformations • Saliency
– Each feature has a distinctive description• Compactness and efficiency
– Many fewer features than image pixels• Locality
– A feature occupies a relatively small area of the image; robust to clutter and occlusion
![Page 17: Local features and image matching October 1 st 2015 Devi Parikh Virginia Tech Disclaimer: Many slides have been borrowed from Kristen Grauman, who may.](https://reader035.fdocuments.in/reader035/viewer/2022070605/5a4d1aee7f8b9ab05997cb90/html5/thumbnails/17.jpg)
Goal: interest operator repeatability• We want to detect (at least some of) the
same points in both images.
• Yet we have to be able to run the detection procedure independently per image.
No chance to find true matches!
![Page 18: Local features and image matching October 1 st 2015 Devi Parikh Virginia Tech Disclaimer: Many slides have been borrowed from Kristen Grauman, who may.](https://reader035.fdocuments.in/reader035/viewer/2022070605/5a4d1aee7f8b9ab05997cb90/html5/thumbnails/18.jpg)
Goal: descriptor distinctiveness• We want to be able to reliably determine
which point goes with which.
• Must provide some invariance to geometric and photometric differences between the two views.
?
![Page 19: Local features and image matching October 1 st 2015 Devi Parikh Virginia Tech Disclaimer: Many slides have been borrowed from Kristen Grauman, who may.](https://reader035.fdocuments.in/reader035/viewer/2022070605/5a4d1aee7f8b9ab05997cb90/html5/thumbnails/19.jpg)
Local features: main components1) Detection: Identify the
interest points
2) Description:Extract vector feature descriptor surrounding each interest point.
3) Matching: Determine correspondence between descriptors in two views
![Page 20: Local features and image matching October 1 st 2015 Devi Parikh Virginia Tech Disclaimer: Many slides have been borrowed from Kristen Grauman, who may.](https://reader035.fdocuments.in/reader035/viewer/2022070605/5a4d1aee7f8b9ab05997cb90/html5/thumbnails/20.jpg)
• What points would you choose?
![Page 21: Local features and image matching October 1 st 2015 Devi Parikh Virginia Tech Disclaimer: Many slides have been borrowed from Kristen Grauman, who may.](https://reader035.fdocuments.in/reader035/viewer/2022070605/5a4d1aee7f8b9ab05997cb90/html5/thumbnails/21.jpg)
Corners as distinctive interest points
We should easily recognize the point by looking through a small window
Shifting a window in any direction should give a large change in intensity
“edge”:no change along the edge direction
“corner”:significant change in all directions
“flat” region:no change in all directionsSlide credit: Alyosha Efros, Darya Frolova, Denis Simakov
![Page 22: Local features and image matching October 1 st 2015 Devi Parikh Virginia Tech Disclaimer: Many slides have been borrowed from Kristen Grauman, who may.](https://reader035.fdocuments.in/reader035/viewer/2022070605/5a4d1aee7f8b9ab05997cb90/html5/thumbnails/22.jpg)
yyyx
yxxx
IIIIIIII
yxwM ),(
xII x
yII y
yI
xIII yx
Corners as distinctive interest points
2 x 2 matrix of image derivatives (averaged in neighborhood of a point).
Notation:
![Page 23: Local features and image matching October 1 st 2015 Devi Parikh Virginia Tech Disclaimer: Many slides have been borrowed from Kristen Grauman, who may.](https://reader035.fdocuments.in/reader035/viewer/2022070605/5a4d1aee7f8b9ab05997cb90/html5/thumbnails/23.jpg)
First, consider an axis-aligned corner:
What does this matrix reveal?
![Page 24: Local features and image matching October 1 st 2015 Devi Parikh Virginia Tech Disclaimer: Many slides have been borrowed from Kristen Grauman, who may.](https://reader035.fdocuments.in/reader035/viewer/2022070605/5a4d1aee7f8b9ab05997cb90/html5/thumbnails/24.jpg)
First, consider an axis-aligned corner:
This means dominant gradient directions align with x or y axis
Look for locations where both λ’s are large.
If either λ is close to 0, then this is not corner-like.
What does this matrix reveal?
What if we have a corner that is not aligned with the image axes?
![Page 25: Local features and image matching October 1 st 2015 Devi Parikh Virginia Tech Disclaimer: Many slides have been borrowed from Kristen Grauman, who may.](https://reader035.fdocuments.in/reader035/viewer/2022070605/5a4d1aee7f8b9ab05997cb90/html5/thumbnails/25.jpg)
What does this matrix reveal?
Since M is symmetric, we have TXXM
2
1
00
iii xMx
The eigenvalues of M reveal the amount of intensity change in the two principal orthogonal gradient directions in the window.
![Page 26: Local features and image matching October 1 st 2015 Devi Parikh Virginia Tech Disclaimer: Many slides have been borrowed from Kristen Grauman, who may.](https://reader035.fdocuments.in/reader035/viewer/2022070605/5a4d1aee7f8b9ab05997cb90/html5/thumbnails/26.jpg)
Corner response function
“flat” region1 and 2 are small;
“edge”:1 >> 2
2 >> 1
“corner”:1 and 2 are large, 1 ~ 2;
1
2
![Page 27: Local features and image matching October 1 st 2015 Devi Parikh Virginia Tech Disclaimer: Many slides have been borrowed from Kristen Grauman, who may.](https://reader035.fdocuments.in/reader035/viewer/2022070605/5a4d1aee7f8b9ab05997cb90/html5/thumbnails/27.jpg)
Harris corner detector
1) Compute M matrix for each image window to get their cornerness scores.
2) Find points whose surrounding window gave large corner response (f> threshold)
3) Take the points of local maxima, i.e., perform non-maximum suppression
![Page 28: Local features and image matching October 1 st 2015 Devi Parikh Virginia Tech Disclaimer: Many slides have been borrowed from Kristen Grauman, who may.](https://reader035.fdocuments.in/reader035/viewer/2022070605/5a4d1aee7f8b9ab05997cb90/html5/thumbnails/28.jpg)
Example of Harris application
Kristen Grauman
![Page 29: Local features and image matching October 1 st 2015 Devi Parikh Virginia Tech Disclaimer: Many slides have been borrowed from Kristen Grauman, who may.](https://reader035.fdocuments.in/reader035/viewer/2022070605/5a4d1aee7f8b9ab05997cb90/html5/thumbnails/29.jpg)
Compute corner response at every pixel.
Example of Harris application
Kristen Grauman
![Page 30: Local features and image matching October 1 st 2015 Devi Parikh Virginia Tech Disclaimer: Many slides have been borrowed from Kristen Grauman, who may.](https://reader035.fdocuments.in/reader035/viewer/2022070605/5a4d1aee7f8b9ab05997cb90/html5/thumbnails/30.jpg)
Example of Harris application
Kristen Grauman
![Page 31: Local features and image matching October 1 st 2015 Devi Parikh Virginia Tech Disclaimer: Many slides have been borrowed from Kristen Grauman, who may.](https://reader035.fdocuments.in/reader035/viewer/2022070605/5a4d1aee7f8b9ab05997cb90/html5/thumbnails/31.jpg)
Properties of the Harris corner detectorRotation invariant?
Scale invariant?
TXXM
2
1
00
Yes
![Page 32: Local features and image matching October 1 st 2015 Devi Parikh Virginia Tech Disclaimer: Many slides have been borrowed from Kristen Grauman, who may.](https://reader035.fdocuments.in/reader035/viewer/2022070605/5a4d1aee7f8b9ab05997cb90/html5/thumbnails/32.jpg)
Properties of the Harris corner detectorRotation invariant?
Scale invariant?
All points will be classified as edges
Corner !
Yes
No
![Page 33: Local features and image matching October 1 st 2015 Devi Parikh Virginia Tech Disclaimer: Many slides have been borrowed from Kristen Grauman, who may.](https://reader035.fdocuments.in/reader035/viewer/2022070605/5a4d1aee7f8b9ab05997cb90/html5/thumbnails/33.jpg)
Harris Detector: Steps
Slide credit: Kristen Grauman34
![Page 34: Local features and image matching October 1 st 2015 Devi Parikh Virginia Tech Disclaimer: Many slides have been borrowed from Kristen Grauman, who may.](https://reader035.fdocuments.in/reader035/viewer/2022070605/5a4d1aee7f8b9ab05997cb90/html5/thumbnails/34.jpg)
Harris Detector: StepsCompute corner response f
Slide credit: Kristen Grauman35
![Page 35: Local features and image matching October 1 st 2015 Devi Parikh Virginia Tech Disclaimer: Many slides have been borrowed from Kristen Grauman, who may.](https://reader035.fdocuments.in/reader035/viewer/2022070605/5a4d1aee7f8b9ab05997cb90/html5/thumbnails/35.jpg)
Harris Detector: StepsFind points with large corner response: f > threshold
Slide credit: Kristen Grauman 36
![Page 36: Local features and image matching October 1 st 2015 Devi Parikh Virginia Tech Disclaimer: Many slides have been borrowed from Kristen Grauman, who may.](https://reader035.fdocuments.in/reader035/viewer/2022070605/5a4d1aee7f8b9ab05997cb90/html5/thumbnails/36.jpg)
Harris Detector: StepsTake only the points of local maxima of f
Slide credit: Kristen Grauman 37
![Page 37: Local features and image matching October 1 st 2015 Devi Parikh Virginia Tech Disclaimer: Many slides have been borrowed from Kristen Grauman, who may.](https://reader035.fdocuments.in/reader035/viewer/2022070605/5a4d1aee7f8b9ab05997cb90/html5/thumbnails/37.jpg)
Harris Detector: Steps
Slide credit: Kristen Grauman38
![Page 38: Local features and image matching October 1 st 2015 Devi Parikh Virginia Tech Disclaimer: Many slides have been borrowed from Kristen Grauman, who may.](https://reader035.fdocuments.in/reader035/viewer/2022070605/5a4d1aee7f8b9ab05997cb90/html5/thumbnails/38.jpg)
Summary• Image warping to create mosaic, given
homography
• Interest point detection– Harris corner detector– Next time:
• Laplacian of Gaussian, automatic scale selection
![Page 39: Local features and image matching October 1 st 2015 Devi Parikh Virginia Tech Disclaimer: Many slides have been borrowed from Kristen Grauman, who may.](https://reader035.fdocuments.in/reader035/viewer/2022070605/5a4d1aee7f8b9ab05997cb90/html5/thumbnails/39.jpg)
Questions?• See you Tuesday!