1
Local features:detection and description
Tuesday October 8Devi Parikh
Virginia Tech
Slide credit: Kristen Grauman
Disclaimer: Most slides have been borrowed from Kristen Grauman, who may have borrowed some of them from others. Any time a slide did not already have a credit on it, I have credited it to Kristen. So there is a chance some of these credits are inaccurate.
2
Announcements
• Project proposal grades out
• PS3 out today
Slide credit: Devi Parikh
3
Last time
• Image warping based on homography• Detecting corner-like points in an image
Slide credit: Kristen Grauman
Today
• Local invariant features– Detection of interest points
• (Harris corner detection)• Scale invariant blob detection: LoG
– Description of local patches• SIFT : Histograms of oriented gradients
Slide credit: Kristen Grauman4
Local features: main components1) Detection: Identify the
interest points
2) Description: Extract vector feature descriptor surrounding each interest point.
3) Matching: Determine correspondence between descriptors in two views
],,[ )1()1(11 dxx x
],,[ )2()2(12 dxx x
Kristen Grauman 5
Goal: interest operator repeatability• We want to detect (at least some of) the
same points in both images.
• Yet we have to be able to run the detection procedure independently per image.
No chance to find true matches!
Slide credit: Kristen Grauman6
Goal: descriptor distinctiveness• We want to be able to reliably determine
which point goes with which.
• Must provide some invariance to geometric and photometric differences between the two views.
?
Slide credit: Kristen Grauman7
Local features: main components1) Detection: Identify the
interest points
2) Description:Extract vector feature descriptor surrounding each interest point.
3) Matching: Determine correspondence between descriptors in two views
Slide credit: Kristen Grauman8
yyyx
yxxx
IIIIIIII
yxwM ),(
xII x
yII y
yI
xIII yx
Recall: Corners as distinctive interest points
2 x 2 matrix of image derivatives (averaged in neighborhood of a point).
Notation:
Slide credit: Kristen Grauman 9
Since M is symmetric, we have TXXM
2
1
00
iii xMx
The eigenvalues of M reveal the amount of intensity change in the two principal orthogonal gradient directions in the window.
Recall: Corners as distinctive interest points
Slide credit: Kristen Grauman10
“flat” region1 and 2 are small;
“edge”:1 >> 2
2 >> 1
“corner”:1 and 2 are large, 1 ~ 2;
One way to score the cornerness:
Recall: Corners as distinctive interest points
Slide credit: Kristen Grauman11
Harris corner detector
1) Compute M matrix for image window surrounding each pixel to get its cornerness score.
2) Find points with large corner response (f > threshold)
3) Take the points of local maxima, i.e., perform non-maximum suppression
Slide credit: Kristen Grauman12
Harris Detector: Steps
Slide credit: Kristen Grauman13
Harris Detector: StepsCompute corner response f
Slide credit: Kristen Grauman14
Harris Detector: StepsFind points with large corner response: f > threshold
Slide credit: Kristen Grauman 15
Harris Detector: StepsTake only the points of local maxima of f
Slide credit: Kristen Grauman 16
Harris Detector: Steps
Slide credit: Kristen Grauman17
Properties of the Harris corner detectorRotation invariant?
Scale invariant?
TXXM
2
1
00
Yes
Slide credit: Kristen Grauman18
Properties of the Harris corner detectorRotation invariant?
Scale invariant?
All points will be classified as edges
Corner !
Yes
No
Slide credit: Kristen Grauman19
Scale invariant interest pointsHow can we independently select interest points in each image, such that the detections are repeatable across different scales?
Slide credit: Kristen Grauman20
Automatic scale selectionIntuition: • Find scale that gives local maxima of some function f
in both position and scale.
f
region size
Image 1f
region size
Image 2
s1 s2Slide credit: Kristen Grauman 21
What can be the “signature” function?
Slide credit: Kristen Grauman22
Recall: Edge detection
gdxdf
f
gdxd
Source: S. Seitz
Edge
Derivativeof Gaussian
Edge = maximumof derivative
23
gdxdf 2
2
f
gdxd
2
2
Edge
Second derivativeof Gaussian (Laplacian)
Edge = zero crossingof second derivative
Source: S. Seitz
Recall: Edge detection
24
From edges to blobs• Edge = ripple• Blob = superposition of two ripples
Spatial selection: the magnitude of the Laplacianresponse will achieve a maximum at the center ofthe blob, provided the scale of the Laplacian is“matched” to the scale of the blob
maximum
Slide credit: Lana Lazebnik
25
Blob detection in 2DLaplacian of Gaussian: Circularly symmetric
operator for blob detection in 2D
2
2
2
22
yg
xgg
Slide credit: Kristen Grauman26
Blob detection in 2D: scale selection
Laplacian-of-Gaussian = “blob” detector 2
2
2
22
yg
xgg
fil
ter s
cale
s
img1 img2 img3Bastian Leibe
27
Blob detection in 2DWe define the characteristic scale as the scale
that produces peak of Laplacian response
characteristic scale
Slide credit: Lana Lazebnik
28
Example
Original image at ¾ the size
Kristen Grauman 29
Original image at ¾ the size
Kristen Grauman
30
Kristen Grauman
31
Kristen Grauman
32
Kristen Grauman
33
Kristen Grauman
34
Kristen Grauman
35
)()( yyxx LL
1
2
3
4
5
List of (x, y, σ)
scale
Scale invariant interest pointsInterest points are local maxima in both position
and scale.
Squared filter response maps
Slide credit: Kristen Grauman
36
Scale-space blob detector: Example
Image credit: Lana Lazebnik
37
We can approximate the Laplacian with a difference of Gaussians; more efficient to implement.
2 ( , , ) ( , , )xx yyL G x y G x y
( , , ) ( , , )DoG G x y k G x y
(Laplacian)
(Difference of Gaussians)
Technical detail
Slide credit: Kristen Grauman
38
Local features: main components1) Detection: Identify the
interest points
2) Description:Extract vector feature descriptor surrounding each interest point.
3) Matching: Determine correspondence between descriptors in two views
],,[ )1()1(11 dxx x
],,[ )2()2(12 dxx x
Slide credit: Kristen Grauman
39
Geometric transformations
e.g. scale, translation, rotation
Slide credit: Kristen Grauman40
Photometric transformations
Figure from T. Tuytelaars ECCV 2006 tutorial
Slide credit: Kristen Grauman
41
Raw patches as local descriptors
The simplest way to describe the neighborhood around an interest point is to write down the list of intensities to form a feature vector.
But this is very sensitive to even small shifts, rotations.
Slide credit: Kristen Grauman42
SIFT descriptor [Lowe 2004] • Use histograms to bin pixels within sub-patches
according to their orientation.
0 2p
Why subpatches?Why does SIFT have some illumination invariance?
Slide credit: Kristen Grauman43
CSE 576: Computer Vision
Making descriptor rotation invariant
Image from Matthew Brown
• Rotate patch according to its dominant gradient orientation
• This puts the patches into a canonical orientation.Slide credit: Kristen Grauman
44
• Extraordinarily robust matching technique• Can handle changes in viewpoint
• Up to about 60 degree out of plane rotation• Can handle significant changes in illumination
• Sometimes even day vs. night (below)• Fast and efficient—can run in real time• Lots of code available
• http://people.csail.mit.edu/albert/ladypack/wiki/index.php/Known_implementations_of_SIFT
Steve Seitz
SIFT descriptor [Lowe 2004]
45
Example
NASA Mars Rover images
Slide credit: Kristen Grauman
46
NASA Mars Rover imageswith SIFT feature matchesFigure by Noah Snavely
Example
Slide credit: Kristen Grauman
47
SIFT properties• Invariant to
– Scale – Rotation
• Partially invariant to– Illumination changes– Camera viewpoint– Occlusion, clutter
Slide credit: Kristen Grauman48
Local features: main components1) Detection: Identify the
interest points
2) Description:Extract vector feature descriptor surrounding each interest point.
3) Matching: Determine correspondence between descriptors in two views
Slide credit: Kristen Grauman
49
Matching local features
Kristen Grauman50
Matching local features
?
To generate candidate matches, find patches that have the most similar appearance (e.g., lowest SSD)Simplest approach: compare them all, take the closest (or closest k, or within a thresholded distance)
Image 1 Image 2
Kristen Grauman51
Ambiguous matches
At what SSD value do we have a good match?To add robustness to matching, can consider ratio : distance to best match / distance to second best matchIf low, first match looks good.If high, could be ambiguous match.
Image 1 Image 2
? ? ? ?
Kristen Grauman52
Matching SIFT Descriptors• Nearest neighbor (Euclidean distance)• Threshold ratio of nearest to 2nd nearest descriptor
Lowe IJCV 2004Slide credit: Kristen Grauman
53
Recap: robust feature-based alignment
Source: L. Lazebnik
54
Recap: robust feature-based alignment
• Extract features
Source: L. Lazebnik
55
Recap: robust feature-based alignment
• Extract features• Compute putative matches
Source: L. Lazebnik
56
Recap: robust feature-based alignment
• Extract features• Compute putative matches• Loop:
• Hypothesize transformation T (small group of putative matches that are related by T)
Source: L. Lazebnik
57
Recap: robust feature-based alignment
• Extract features• Compute putative matches• Loop:
• Hypothesize transformation T (small group of putative matches that are related by T)
• Verify transformation (search for other matches consistent with T)
Source: L. Lazebnik
58
Recap: robust feature-based alignment
• Extract features• Compute putative matches• Loop:
• Hypothesize transformation T (small group of putative matches that are related by T)
• Verify transformation (search for other matches consistent with T)
Source: L. Lazebnik
59
Applications of local invariant features
• Wide baseline stereo• Motion tracking• Panoramas• Mobile robot navigation• 3D reconstruction• Recognition• …
Slide credit: Kristen Grauman
60
Automatic mosaicing
http://www.cs.ubc.ca/~mbrown/autostitch/autostitch.html
Slide credit: Kristen Grauman
61
Wide baseline stereo
[Image from T. Tuytelaars ECCV 2006 tutorial]
Slide credit: Kristen Grauman 62
Recognition of specific objects, scenes
Rothganger et al. 2003 Lowe 2002
Schmid and Mohr 1997 Sivic and Zisserman, 2003
Kristen Grauman
63
Summary• Interest point detection
– Harris corner detector– Laplacian of Gaussian, automatic scale selection
• Invariant descriptors– Rotation according to dominant gradient direction– Histograms for robustness to small shifts and
translations (SIFT descriptor)
64
Numerical Issues
• When computing H– Say true match is [50 100 1] [50 100]– [50.5 100 1]
• [50.5 100]– [50 100 1.5]
• [33 67]– Scale co-ordinates to lie between 0 and 2
65
Questions?
See you Thursday!
66
Slide credit: Devi Parikh
Top Related