SIFT keypoint detectionslazebni.cs.illinois.edu/spring19/lec09_sift.pdfSIFT keypoint detection D....
Transcript of SIFT keypoint detectionslazebni.cs.illinois.edu/spring19/lec09_sift.pdfSIFT keypoint detection D....
![Page 1: SIFT keypoint detectionslazebni.cs.illinois.edu/spring19/lec09_sift.pdfSIFT keypoint detection D. Lowe, Distinctive image features from scale-invariant keypoints, IJCV 60 (2), pp.](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f511e337dd201484f78ca93/html5/thumbnails/1.jpg)
SIFT keypoint detection
D. Lowe, Distinctive image features from scale-invariant keypoints, IJCV 60 (2), pp. 91-110, 2004
![Page 2: SIFT keypoint detectionslazebni.cs.illinois.edu/spring19/lec09_sift.pdfSIFT keypoint detection D. Lowe, Distinctive image features from scale-invariant keypoints, IJCV 60 (2), pp.](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f511e337dd201484f78ca93/html5/thumbnails/2.jpg)
Keypoint detection with scale selection• We want to extract keypoints with
characteristic scales that are covariant w.r.t. the image transformation
![Page 3: SIFT keypoint detectionslazebni.cs.illinois.edu/spring19/lec09_sift.pdfSIFT keypoint detection D. Lowe, Distinctive image features from scale-invariant keypoints, IJCV 60 (2), pp.](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f511e337dd201484f78ca93/html5/thumbnails/3.jpg)
Basic idea
• Convolve the image with a “blob filter”
at multiple scales and look for extrema of
filter response in the resulting scale space
T. Lindeberg, Feature detection with automatic scale selection,
IJCV 30(2), pp 77-116, 1998
![Page 4: SIFT keypoint detectionslazebni.cs.illinois.edu/spring19/lec09_sift.pdfSIFT keypoint detection D. Lowe, Distinctive image features from scale-invariant keypoints, IJCV 60 (2), pp.](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f511e337dd201484f78ca93/html5/thumbnails/4.jpg)
Blob detection
Find maxima and minima of blob filter response in space and scale
* =
maxima
minima
Source: N. Snavely
![Page 5: SIFT keypoint detectionslazebni.cs.illinois.edu/spring19/lec09_sift.pdfSIFT keypoint detection D. Lowe, Distinctive image features from scale-invariant keypoints, IJCV 60 (2), pp.](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f511e337dd201484f78ca93/html5/thumbnails/5.jpg)
Blob filterLaplacian of Gaussian: Circularly symmetric
operator for blob detection in 2D
2
2
2
22
yg
xg
g¶¶
+¶¶
=Ñ
![Page 6: SIFT keypoint detectionslazebni.cs.illinois.edu/spring19/lec09_sift.pdfSIFT keypoint detection D. Lowe, Distinctive image features from scale-invariant keypoints, IJCV 60 (2), pp.](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f511e337dd201484f78ca93/html5/thumbnails/6.jpg)
Recall: Edge detection
gdxd
f *
f
gdxd
Source: S. Seitz
Edge
Derivativeof Gaussian
Edge = maximumof derivative
![Page 7: SIFT keypoint detectionslazebni.cs.illinois.edu/spring19/lec09_sift.pdfSIFT keypoint detection D. Lowe, Distinctive image features from scale-invariant keypoints, IJCV 60 (2), pp.](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f511e337dd201484f78ca93/html5/thumbnails/7.jpg)
Edge detection, Take 2
gdxd
f 2
2
*
f
gdxd2
2
Edge
Second derivativeof Gaussian (Laplacian)
Edge = zero crossingof second derivative
Source: S. Seitz
![Page 8: SIFT keypoint detectionslazebni.cs.illinois.edu/spring19/lec09_sift.pdfSIFT keypoint detection D. Lowe, Distinctive image features from scale-invariant keypoints, IJCV 60 (2), pp.](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f511e337dd201484f78ca93/html5/thumbnails/8.jpg)
From edges to blobs• Edge = ripple• Blob = superposition of two ripples
Spatial selection: the magnitude of the Laplacianresponse will achieve a maximum at the center ofthe blob, provided the scale of the Laplacian is“matched” to the scale of the blob
maximum
![Page 9: SIFT keypoint detectionslazebni.cs.illinois.edu/spring19/lec09_sift.pdfSIFT keypoint detection D. Lowe, Distinctive image features from scale-invariant keypoints, IJCV 60 (2), pp.](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f511e337dd201484f78ca93/html5/thumbnails/9.jpg)
Scale selection• We want to find the characteristic scale of the
blob by convolving it with Laplacians at several scales and looking for the maximum response
• However, Laplacian response decays as scale increases:
increasing σoriginal signal(radius=8)
![Page 10: SIFT keypoint detectionslazebni.cs.illinois.edu/spring19/lec09_sift.pdfSIFT keypoint detection D. Lowe, Distinctive image features from scale-invariant keypoints, IJCV 60 (2), pp.](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f511e337dd201484f78ca93/html5/thumbnails/10.jpg)
Scale normalization• The response of a derivative of Gaussian filter to a
perfect step edge decreases as σ increases:
• To keep response the same (scale-invariant), must multiply Gaussian derivative by σ
• Laplacian is the second Gaussian derivative, so it must be multiplied by σ2
ps 21
![Page 11: SIFT keypoint detectionslazebni.cs.illinois.edu/spring19/lec09_sift.pdfSIFT keypoint detection D. Lowe, Distinctive image features from scale-invariant keypoints, IJCV 60 (2), pp.](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f511e337dd201484f78ca93/html5/thumbnails/11.jpg)
Effect of scale normalization
Scale-normalized Laplacian response
Unnormalized Laplacian responseOriginal signal
maximum
![Page 12: SIFT keypoint detectionslazebni.cs.illinois.edu/spring19/lec09_sift.pdfSIFT keypoint detection D. Lowe, Distinctive image features from scale-invariant keypoints, IJCV 60 (2), pp.](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f511e337dd201484f78ca93/html5/thumbnails/12.jpg)
Blob detection in 2D• Scale-normalized Laplacian of Gaussian:
÷÷ø
öççè
涶
+¶¶
=Ñ 2
2
2
222
norm yg
xg
g s
![Page 13: SIFT keypoint detectionslazebni.cs.illinois.edu/spring19/lec09_sift.pdfSIFT keypoint detection D. Lowe, Distinctive image features from scale-invariant keypoints, IJCV 60 (2), pp.](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f511e337dd201484f78ca93/html5/thumbnails/13.jpg)
Blob detection in 2D• At what scale does the Laplacian achieve a maximum
response to a binary circle of radius r?
r
image Laplacian
![Page 14: SIFT keypoint detectionslazebni.cs.illinois.edu/spring19/lec09_sift.pdfSIFT keypoint detection D. Lowe, Distinctive image features from scale-invariant keypoints, IJCV 60 (2), pp.](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f511e337dd201484f78ca93/html5/thumbnails/14.jpg)
Blob detection in 2D• At what scale does the Laplacian achieve a maximum
response to a binary circle of radius r?• To get maximum response, the zeros of the Laplacian
have to be aligned with the circle• The Laplacian is given by (up to scale):
• Therefore, the maximum response occurs at
r
image
222 2/)(222 )2( ss yxeyx +--+.2/r=s
circle
Laplacian
0
![Page 15: SIFT keypoint detectionslazebni.cs.illinois.edu/spring19/lec09_sift.pdfSIFT keypoint detection D. Lowe, Distinctive image features from scale-invariant keypoints, IJCV 60 (2), pp.](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f511e337dd201484f78ca93/html5/thumbnails/15.jpg)
Scale-space blob detector1. Convolve image with scale-normalized
Laplacian at several scales
![Page 16: SIFT keypoint detectionslazebni.cs.illinois.edu/spring19/lec09_sift.pdfSIFT keypoint detection D. Lowe, Distinctive image features from scale-invariant keypoints, IJCV 60 (2), pp.](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f511e337dd201484f78ca93/html5/thumbnails/16.jpg)
Scale-space blob detector: Example
![Page 17: SIFT keypoint detectionslazebni.cs.illinois.edu/spring19/lec09_sift.pdfSIFT keypoint detection D. Lowe, Distinctive image features from scale-invariant keypoints, IJCV 60 (2), pp.](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f511e337dd201484f78ca93/html5/thumbnails/17.jpg)
Scale-space blob detector: Example
![Page 18: SIFT keypoint detectionslazebni.cs.illinois.edu/spring19/lec09_sift.pdfSIFT keypoint detection D. Lowe, Distinctive image features from scale-invariant keypoints, IJCV 60 (2), pp.](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f511e337dd201484f78ca93/html5/thumbnails/18.jpg)
Scale-space blob detector1. Convolve image with scale-normalized
Laplacian at several scales2. Find maxima of squared Laplacian response
in scale-space
![Page 19: SIFT keypoint detectionslazebni.cs.illinois.edu/spring19/lec09_sift.pdfSIFT keypoint detection D. Lowe, Distinctive image features from scale-invariant keypoints, IJCV 60 (2), pp.](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f511e337dd201484f78ca93/html5/thumbnails/19.jpg)
Scale-space blob detector: Example
![Page 20: SIFT keypoint detectionslazebni.cs.illinois.edu/spring19/lec09_sift.pdfSIFT keypoint detection D. Lowe, Distinctive image features from scale-invariant keypoints, IJCV 60 (2), pp.](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f511e337dd201484f78ca93/html5/thumbnails/20.jpg)
• Approximating the Laplacian with a difference of Gaussians:
( )2 ( , , ) ( , , )xx yyL G x y G x ys s s= +
( , , ) ( , , )DoG G x y k G x ys s= -
(Laplacian)
(Difference of Gaussians)
Efficient implementation
![Page 21: SIFT keypoint detectionslazebni.cs.illinois.edu/spring19/lec09_sift.pdfSIFT keypoint detection D. Lowe, Distinctive image features from scale-invariant keypoints, IJCV 60 (2), pp.](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f511e337dd201484f78ca93/html5/thumbnails/21.jpg)
Efficient implementation
David G. Lowe. "Distinctive image features from scale-invariant keypoints.” IJCV 60 (2), pp. 91-110, 2004.
![Page 22: SIFT keypoint detectionslazebni.cs.illinois.edu/spring19/lec09_sift.pdfSIFT keypoint detection D. Lowe, Distinctive image features from scale-invariant keypoints, IJCV 60 (2), pp.](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f511e337dd201484f78ca93/html5/thumbnails/22.jpg)
Eliminating edge responses• Laplacian has strong response along edges
![Page 23: SIFT keypoint detectionslazebni.cs.illinois.edu/spring19/lec09_sift.pdfSIFT keypoint detection D. Lowe, Distinctive image features from scale-invariant keypoints, IJCV 60 (2), pp.](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f511e337dd201484f78ca93/html5/thumbnails/23.jpg)
Eliminating edge responses• Laplacian has strong response along edges
• Solution: filter based on Harris response function over neighborhoods containing the “blobs”
![Page 24: SIFT keypoint detectionslazebni.cs.illinois.edu/spring19/lec09_sift.pdfSIFT keypoint detection D. Lowe, Distinctive image features from scale-invariant keypoints, IJCV 60 (2), pp.](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f511e337dd201484f78ca93/html5/thumbnails/24.jpg)
From feature detection to feature description• To recognize the same pattern in multiple
images, we need to match appearance “signatures” in the neighborhoods of extracted keypoints
• But corresponding neighborhoods can be related by a scale change or rotation
• We want to normalize neighborhoods to make signatures invariant to these transformations
![Page 25: SIFT keypoint detectionslazebni.cs.illinois.edu/spring19/lec09_sift.pdfSIFT keypoint detection D. Lowe, Distinctive image features from scale-invariant keypoints, IJCV 60 (2), pp.](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f511e337dd201484f78ca93/html5/thumbnails/25.jpg)
Finding a reference orientation• Create histogram of local gradient directions in
the patch• Assign reference orientation at peak of
smoothed histogram
0 2 p
![Page 26: SIFT keypoint detectionslazebni.cs.illinois.edu/spring19/lec09_sift.pdfSIFT keypoint detection D. Lowe, Distinctive image features from scale-invariant keypoints, IJCV 60 (2), pp.](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f511e337dd201484f78ca93/html5/thumbnails/26.jpg)
SIFT features• Detected features with characteristic scales
and orientations:
David G. Lowe. "Distinctive image features from scale-invariant keypoints.” IJCV 60 (2), pp. 91-110, 2004.
![Page 27: SIFT keypoint detectionslazebni.cs.illinois.edu/spring19/lec09_sift.pdfSIFT keypoint detection D. Lowe, Distinctive image features from scale-invariant keypoints, IJCV 60 (2), pp.](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f511e337dd201484f78ca93/html5/thumbnails/27.jpg)
From keypoint detection to feature description
Detection is covariant:features(transform(image)) = transform(features(image))
Description is invariant:features(transform(image)) = features(image)
![Page 28: SIFT keypoint detectionslazebni.cs.illinois.edu/spring19/lec09_sift.pdfSIFT keypoint detection D. Lowe, Distinctive image features from scale-invariant keypoints, IJCV 60 (2), pp.](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f511e337dd201484f78ca93/html5/thumbnails/28.jpg)
SIFT descriptors• Inspiration: complex neurons in the primary
visual cortex
D. Lowe, Distinctive image features from scale-invariant keypoints, IJCV 60 (2), pp. 91-110, 2004
![Page 29: SIFT keypoint detectionslazebni.cs.illinois.edu/spring19/lec09_sift.pdfSIFT keypoint detection D. Lowe, Distinctive image features from scale-invariant keypoints, IJCV 60 (2), pp.](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f511e337dd201484f78ca93/html5/thumbnails/29.jpg)
Properties of SIFTExtraordinarily robust detection and description technique
• Can handle changes in viewpoint– Up to about 60 degree out-of-plane rotation
• Can handle significant changes in illumination– Sometimes even day vs. night
• Fast and efficient—can run in real time• Lots of code available
Source: N. Snavely
![Page 30: SIFT keypoint detectionslazebni.cs.illinois.edu/spring19/lec09_sift.pdfSIFT keypoint detection D. Lowe, Distinctive image features from scale-invariant keypoints, IJCV 60 (2), pp.](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f511e337dd201484f78ca93/html5/thumbnails/30.jpg)
A hard keypoint matching problem
NASA Mars Rover images
![Page 31: SIFT keypoint detectionslazebni.cs.illinois.edu/spring19/lec09_sift.pdfSIFT keypoint detection D. Lowe, Distinctive image features from scale-invariant keypoints, IJCV 60 (2), pp.](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f511e337dd201484f78ca93/html5/thumbnails/31.jpg)
NASA Mars Rover imageswith SIFT feature matchesFigure by Noah Snavely
Answer below (look for tiny colored squares…)
![Page 32: SIFT keypoint detectionslazebni.cs.illinois.edu/spring19/lec09_sift.pdfSIFT keypoint detection D. Lowe, Distinctive image features from scale-invariant keypoints, IJCV 60 (2), pp.](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f511e337dd201484f78ca93/html5/thumbnails/32.jpg)
What about 3D rotations?
![Page 33: SIFT keypoint detectionslazebni.cs.illinois.edu/spring19/lec09_sift.pdfSIFT keypoint detection D. Lowe, Distinctive image features from scale-invariant keypoints, IJCV 60 (2), pp.](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f511e337dd201484f78ca93/html5/thumbnails/33.jpg)
What about 3D rotations?• Affine transformation approximates viewpoint
changes for roughly planar objects and roughly orthographic cameras
![Page 34: SIFT keypoint detectionslazebni.cs.illinois.edu/spring19/lec09_sift.pdfSIFT keypoint detection D. Lowe, Distinctive image features from scale-invariant keypoints, IJCV 60 (2), pp.](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f511e337dd201484f78ca93/html5/thumbnails/34.jpg)
Affine adaptation
RRIIIIII
yxwMyyx
yxx
yxúû
ùêë
é=
úúû
ù
êêë
é= -å
2
112
2
, 00
),(l
l
direction of the slowest
change
direction of the fastest change
(lmax)-1/2(lmin)-1/2
Consider the second moment matrix of the window containing the blob:
const][ =úû
ùêë
évu
Mvu
Recall:
This ellipse visualizes the “characteristic shape” of the window
![Page 35: SIFT keypoint detectionslazebni.cs.illinois.edu/spring19/lec09_sift.pdfSIFT keypoint detection D. Lowe, Distinctive image features from scale-invariant keypoints, IJCV 60 (2), pp.](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f511e337dd201484f78ca93/html5/thumbnails/35.jpg)
Affine adaptation
K. Mikolajczyk and C. Schmid, Scale and affine invariant interest point detectors, IJCV 60(1):63-86, 2004
![Page 36: SIFT keypoint detectionslazebni.cs.illinois.edu/spring19/lec09_sift.pdfSIFT keypoint detection D. Lowe, Distinctive image features from scale-invariant keypoints, IJCV 60 (2), pp.](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f511e337dd201484f78ca93/html5/thumbnails/36.jpg)
Keypoint detectors/descriptors for recognition: A retrospective
S. Lazebnik, C. Schmid, and J. Ponce, A Sparse Texture Representation Using Affine-Invariant Regions, CVPR 2003
Detected features
![Page 37: SIFT keypoint detectionslazebni.cs.illinois.edu/spring19/lec09_sift.pdfSIFT keypoint detection D. Lowe, Distinctive image features from scale-invariant keypoints, IJCV 60 (2), pp.](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f511e337dd201484f78ca93/html5/thumbnails/37.jpg)
Keypoint detectors/descriptors for recognition: A retrospective
R. Fergus, P. Perona, and A. Zisserman, Object Class Recognition by Unsupervised Scale-Invariant Learning, CVPR 2003 – winner of 2013 Longuet-Higgins Prize
Detected features
![Page 38: SIFT keypoint detectionslazebni.cs.illinois.edu/spring19/lec09_sift.pdfSIFT keypoint detection D. Lowe, Distinctive image features from scale-invariant keypoints, IJCV 60 (2), pp.](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f511e337dd201484f78ca93/html5/thumbnails/38.jpg)
Keypoint detectors/descriptors for recognition: A retrospective
S. Lazebnik, C. Schmid, and J. Ponce, Beyond Bags of Features: Spatial Pyramid
Matching for Recognizing Natural Scene Categories, CVPR 2006 – winner of 2016 Longuet-Higgins Prize
![Page 39: SIFT keypoint detectionslazebni.cs.illinois.edu/spring19/lec09_sift.pdfSIFT keypoint detection D. Lowe, Distinctive image features from scale-invariant keypoints, IJCV 60 (2), pp.](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f511e337dd201484f78ca93/html5/thumbnails/39.jpg)
level 0 level 1 level 2
Keypoint detectors/descriptors for recognition: A retrospective
S. Lazebnik, C. Schmid, and J. Ponce, Beyond Bags of Features: Spatial Pyramid Matching for Recognizing Natural Scene Categories, CVPR 2006 – winner of 2016
Longuet-Higgins Prize
![Page 40: SIFT keypoint detectionslazebni.cs.illinois.edu/spring19/lec09_sift.pdfSIFT keypoint detection D. Lowe, Distinctive image features from scale-invariant keypoints, IJCV 60 (2), pp.](https://reader034.fdocuments.in/reader034/viewer/2022042612/5f511e337dd201484f78ca93/html5/thumbnails/40.jpg)
Keypoint detectors/descriptors for recognition: A retrospective