Post on 14-Apr-2018
7/30/2019 Feature Detection and Matching
1/80
Feature Detection and Matching
Reading: Szeliski Chapter 4
Pages: 208~266
http://staff.ustc.edu.cn/~xjchen99/teaching/teaching.html
7/30/2019 Feature Detection and Matching
2/80
Image Matching
by Diva Sian
by swashford
http://www.flickr.com/photos/diaphanus/136915456/http://www.flickr.com/photos/swashford/428567562/http://www.flickr.com/photos/swashford/428567562/http://www.flickr.com/photos/diaphanus/136915456/7/30/2019 Feature Detection and Matching
3/80
Harder Case
by Diva Sian by scgbt
http://www.flickr.com/photos/diaphanus/136915456/http://www.flickr.com/photos/scpgt/328570837/http://www.flickr.com/photos/scpgt/328570837/http://www.flickr.com/photos/diaphanus/136915456/7/30/2019 Feature Detection and Matching
4/80
Even Harder Case
How the Afghan Girl was Identified by Her Iris Patterns Read the story
http://www.cl.cam.ac.uk/~jgd1000/afghan.htmlhttp://www.cl.cam.ac.uk/~jgd1000/afghan.html7/30/2019 Feature Detection and Matching
5/80
Harder still?
NASA Mars Rover images
7/30/2019 Feature Detection and Matching
6/80
NASA Mars Rover images
with SIFT feature matchesFigure by Noah Snavely
Answer below (look for tiny colored squares)
7/30/2019 Feature Detection and Matching
7/80
Features
All is Vanity, by C. Allan Gilbert, 1873-1929Readings
Szeliski, Ch 4.1
(optional) K. Mikolajczyk, C. Schmid, A performance evaluation of local
descriptors. In PAMI 27(10):1615-1630- http://www.robots.ox.ac.uk/~vgg/research/affine/det_eval_files/mikolajczyk_
http://www.robots.ox.ac.uk/~vgg/research/affine/det_eval_files/mikolajczyk_pami2004.pdfhttp://www.robots.ox.ac.uk/~vgg/research/affine/det_eval_files/mikolajczyk_pami2004.pdfhttp://upload.wikimedia.org/wikipedia/en/c/c9/Allisvanity.jpg7/30/2019 Feature Detection and Matching
8/80
Image Matching
7/30/2019 Feature Detection and Matching
9/80
Image Matching
7/30/2019 Feature Detection and Matching
10/80
Invariant Local Features
Find features that are invariant to transformations
geometric invariance: translation, rotation, scale
photometric invariance: brightness, exposure,
Feature Descriptors
7/30/2019 Feature Detection and Matching
11/80
Advantages of Local Features
Locality
features are local, so robust to occlusion and clutter
Distinctiveness:
can differentiate a large database of objects
Quantity
hundreds or thousands in a single image
Efficiency
real-time performance achievableGenerality
exploit different types of features in different situations
7/30/2019 Feature Detection and Matching
12/80
More Motivation
Feature points are used for:
Image alignment (e.g., mosaics)
3D reconstruction
Motion tracking Object recognition
Indexing and database retrieval
Robot navigation
other
7/30/2019 Feature Detection and Matching
13/80
Features
Point/patch,
Edge/curve
Region
7/30/2019 Feature Detection and Matching
14/80
Want Uniqueness
Look for image regions that are unusual
Lead to unambiguous matches in other images
How to define unusual?
7/30/2019 Feature Detection and Matching
15/80
7/30/2019 Feature Detection and Matching
16/80
Corner Detection
Basic idea: Find points where two edges meeti.e., high gradient in twodirections
Cornerness is undefined at a single pixel, because theres only onegradient per point
Look at the gradient behavior over a small window
Categories image windows based on gradient statistics Constant: Little or no brightness change
Edge: Strong brightness change in single direction
Flow: Parallel stripes
Corner/spot: Strong brightness changes in orthogonal directions
7/30/2019 Feature Detection and Matching
17/80
Matching Criterion
(weighted) Summed Square Difference SSD
Compare an image patch against itself
7/30/2019 Feature Detection and Matching
18/80
Local Measures of Uniqueness
Suppose we only consider a small window of pixels
What defines whether a feature is a good or bad candidate?
Slide adapted from Darya Frolova, Denis Simakov, Weizmann Institute.
7/30/2019 Feature Detection and Matching
19/80
Feature Detection
flat region:
no change in all
directions
edge:
no change along
the edge direction
corner:
significant change
in all directions
Local measure of feature uniqueness
How does the window change when you shift it? Shifting the window in any direction causes a big change
Slide adapted from Darya Frolova, Denis Simakov, Weizmann Institute.
7/30/2019 Feature Detection and Matching
20/80
Consider shifting the window W by (u,v)
how do the pixels in W change?
compare each pixel before and after by
summing up the squared differences
(SSD) this defines an SSD error ofE(u,v):
Feature Detection: Mathematics
W
7/30/2019 Feature Detection and Matching
21/80
Harris Detector: Mathematics
2
,
( , ) ( , ) ( , ) ( , )x y
E u v w x y I x u y v I x y
Change of intensity for the shift [u,v]:
IntensityShifted
intensityWindowfunction
orWindow function w(x,y) =
Gaussian1 in window, 0 outside
Auto-correlationfunction
7/30/2019 Feature Detection and Matching
22/80
Auto-Correlation Function
7/30/2019 Feature Detection and Matching
23/80
Auto-Correlation Function
Good unique minimum 1D aperture problem No good peak
7/30/2019 Feature Detection and Matching
24/80
Taylor Series expansion of I:
If the motion (u,v) is small, then first order approx is good
Small Motion Assumption
7/30/2019 Feature Detection and Matching
25/80
Feature Detection: Mathematics
Image gradient
Harris detector with a [-2,-1,0,1,2] filter for Ix
Gaussian filter
7/30/2019 Feature Detection and Matching
26/80
Plugging this into the formula on
Small Motion Assumption
2
,
( , ) ( , ) ( , ) ( , )x y
E u v w x y I x u y v I x y
7/30/2019 Feature Detection and Matching
27/80
Feature Detection: Mathematics
W
7/30/2019 Feature Detection and Matching
28/80
Feature Detection: MathematicsThis can be rewritten:
x-
x+
Auto-correlation matrix
7/30/2019 Feature Detection and Matching
29/80
Feature Detection: Mathematics
For the example above
You can move the center of the blue window to anywhere on the
blue unit circle
Which directions will result in the largest and smallest E values?
We can find these directions by looking at the eigenvectors ofH
7/30/2019 Feature Detection and Matching
30/80
Quick eigenvalue/eigenvector review
The eigenvectors of a matrix A are the vectors x that satisfy:
The scalar is the eigenvalue corresponding to x
The eigenvalues are found by solving:
In our case, A = H is a 2x2 matrix, so we have
The solution:
Once you know , you find x by solving
7/30/2019 Feature Detection and Matching
31/80
Feature Detection: MathematicsThis can be rewritten:
Eigenvalues and eigenvectors of H
Define shifts with the smallest and largest change (E value)
x+ = direction of largest increase in E.
+ = amount of increase in direction x+
x- = direction of smallest increase in E.
- = amount of increase in direction x+
x-
x+
7/30/2019 Feature Detection and Matching
32/80
Feature Detection: Mathematics
( , ) ,u
E u v u v H
v
Intensity change in shifting window: eigenvalue analysis
1, 2eigenvalues ofH
(max)-1/2
(min)-1/2
EllipseE(u,v) = const
If we try every possible orientation n,
the max. change in intensity is 2
7/30/2019 Feature Detection and Matching
33/80
Feature Detection: Mathematics
1
2
Corner
1 and 2 are large,
1 ~
2;Eincreases in all
directions
1 and 2 are small;
Eis almost constant
in all directions
Edge
1 >> 2
Edge
2 >> 1
Flat
region
Classification of image
points using eigenvalues
ofM:
7/30/2019 Feature Detection and Matching
34/80
Feature Detection: MathematicsHow are +, x+, -, and x+ relevant for feature detection?
Whats our feature scoring function?
7/30/2019 Feature Detection and Matching
35/80
Feature Detection: MathematicsHow are
+, x
+,
-, and x
+relevant for feature detection?
Whats our feature scoring function?
Want E(u,v) to be large for small shifts in alldirections
the minimum ofE(u,v) should be large, over all unit vectors [u v]
this minimum is given by the smaller eigenvalue (
-) ofH
7/30/2019 Feature Detection and Matching
36/80
Feature DetectionHeres what you do
Compute the gradient at each point in the image
Create the Hmatrix from the entries in the gradient
Compute the eigenvalues.
Find points with large response (- > threshold)
Choose those points where - is a local maximum as features
7/30/2019 Feature Detection and Matching
37/80
Feature DetectionHeres what you do
Compute the gradient at each point in the image
Create the Hmatrix from the entries in the gradient
Compute the eigenvalues.
Find points with large response (- > threshold)
Choose those points where - is a local maximum as features
[Shi and Tomasi 1994]
7/30/2019 Feature Detection and Matching
38/80
Harris Detector
Measure of corner response:
2
det traceR M k M
1 2
1 2
det
trace
M
M
(kempirical constant, k= 0.04-0.06)
Harris and Stephens 1988
The trace is the sum of the diagonals, i.e., trace(H) = h11 + h22 Very similar to - but less expensive (no square root)
7/30/2019 Feature Detection and Matching
39/80
Harris Detector: Mathematics
1
2 Corner
Edge
Edge
Flat
R depends only on eigenvalues
of H
R is large for a corner
R is negative with large
magnitude for an edge
|R| is small for a flat region
R > 0
R < 0
R < 0|R| small
7/30/2019 Feature Detection and Matching
40/80
Harris Detector
The Algorithm:
Find points with large corner response function R
(R > threshold)
Take the points of local maxima ofR
7/30/2019 Feature Detection and Matching
41/80
Harmonic Mean
Smoother function in the region where
Brown, M., Szeliski, R., and Winder, S. (2005).
0 1
0 1
det
trf
H
H
0 1
7/30/2019 Feature Detection and Matching
42/80
Isocontours of Response
Harris Detector: Workflow
7/30/2019 Feature Detection and Matching
43/80
Harris Detector: Workflow
Harris Detector: Workflow
7/30/2019 Feature Detection and Matching
44/80
Harris Detector: Workflow
Compute corner responseR
Harris Detector: Workflow
7/30/2019 Feature Detection and Matching
45/80
Harris Detector: Workflow
Find points with large corner response:R>threshold
Harris Detector: Workflow
7/30/2019 Feature Detection and Matching
46/80
Harris Detector: Workflow
Take only the points of local maxima ofR
Harris Detector: Workflow
7/30/2019 Feature Detection and Matching
47/80
Harris Detector: Workflow
7/30/2019 Feature Detection and Matching
48/80
Example: Gradient Covariances
Full imageDetail of image with gradient covar-
iance ellipses for 3 x 3 windows
from Forsyth & Ponce
Corners are whereboth eigenvalues are big
7/30/2019 Feature Detection and Matching
49/80
Example: Corner Detection
(for camera calibration)
courtesy of B. Wilburn
7/30/2019 Feature Detection and Matching
50/80
Example: Corner Detection
courtesy of S. Smith
SUSAN corners
7/30/2019 Feature Detection and Matching
51/80
Harris Detector: Summary
Average intensity change in direction [u,v] can be expressed as a bilinearform:
Describe a point in terms of eigenvalues ofM:
measure of corner response
A good (corner) point should have a large intensity change in all directions, i.e.R should be large positive
( , ) ,u
E u v u v Mv
2
1 2 1 2R k
7/30/2019 Feature Detection and Matching
52/80
Outline of Feature Detection
1. Compute the horizontal and vertical derivatives of the image Ix and Iy byconvolving the original image with derivatives of Gaussians (Section3.2.3).
2. Compute the three images corresponding to the outer products of thesegradients. (The matrix H is symmetric, so only three entries are needed.)
3. Convolve each of these images with a larger Gaussian.
4. Compute a scalar interest measure using one of the formulas discussedabove.
5. Find local maxima above a certain threshold and report them asdetected feature point locations.
Ad ti N M i l S i
7/30/2019 Feature Detection and Matching
53/80
Adaptive Non-Maximal Suppression
(ANMS)
Local maxima
Response value should be significantly (10%)larger than all of its neighbors within a radius
(r)
(a) Strongest 250 (b) Strongest 500
Ad ti N M i l S i
7/30/2019 Feature Detection and Matching
54/80
Adaptive Non-Maximal Suppression
(ANMS)
(a) Strongest 250 (b) Strongest 500
(c) ANMS 250, r = 24 (d) ANMS 500, r = 16
7/30/2019 Feature Detection and Matching
55/80
Invariance
Suppose you rotate the image by some angle
Will you still pick up the same features?
What if you change the brightness?
Scale?
7/30/2019 Feature Detection and Matching
56/80
Invariance
Repeatability of feature detector:
frequency with which keypoints are detected in one image are found
within (=1.5) pixels of the corresponding location in a transformed
image
7/30/2019 Feature Detection and Matching
57/80
Scale Invariance
Detect features at a
variety of scales
Multiple resolutions ina pyramid
Matching in allpossible levels
7/30/2019 Feature Detection and Matching
58/80
Multi-Scale Oriented Patches
7/30/2019 Feature Detection and Matching
59/80
Scale invariant detection
Suppose youre looking for corners
7/30/2019 Feature Detection and Matching
60/80
Feature Detection
Key idea: find scale that gives local maximum of f
f is a local maximum in both position and scale
Common definition of f: Laplacian
Difference between two Gaussian filtered images
with different sigmas
7/30/2019 Feature Detection and Matching
61/80
Feature Detection
Stable features in both location and scale
Laplacian of Gaussian (LoG)
Difference of Gaussian2 2 2 2
2 2
1 2
1 2
1 2
2 22 2
1 1 12
x y x y
DoG G G e e
7/30/2019 Feature Detection and Matching
62/80
Slide from Tinne Tuytelaars
Lindeberg et al, 1996
Slide from Tinne Tuytelaars
Lindeberg et al., 1996
7/30/2019 Feature Detection and Matching
63/80
7/30/2019 Feature Detection and Matching
64/80
7/30/2019 Feature Detection and Matching
65/80
7/30/2019 Feature Detection and Matching
66/80
7/30/2019 Feature Detection and Matching
67/80
7/30/2019 Feature Detection and Matching
68/80
7/30/2019 Feature Detection and Matching
69/80
7/30/2019 Feature Detection and Matching
70/80
Scale-Space Feature Detection
Empirical number of levels in each octave = 31.6
7/30/2019 Feature Detection and Matching
71/80
SIFT Feature Detection
Will be explained next week
Rotation Invariance and orientation
7/30/2019 Feature Detection and Matching
72/80
Rotation Invariance and orientation
estimation
Dominant orientation
Average gradient within a region around the keypoint(larger window than detection window to make it more
reliable)
Orientation histogram
7/30/2019 Feature Detection and Matching
73/80
Affine Invariance
For wide baseline stereo matching or objectrecognition
7/30/2019 Feature Detection and Matching
74/80
Affine Invariance
For a small enough patch, any continuous image wrapping canbe well approximated by an affine deformation
Eigenvalue analysis for Hessian matrix or auto-correlation:
using the principle axis and ratios to fit the affine coordinateframe1 1 2 2
, , , x x
(Lindeberg and Garding 1997; Baumberg 2000; Mikolajczyk and Schmid 2004; Mikolajczyk,
Tuytelaars, Schmid et al. 2005; Tuytelaars and Mikolajczyk 2007)
7/30/2019 Feature Detection and Matching
75/80
Affine Invariance
Maximally Stable Extremal Region (MSER) Matas, Chum, Urban et al. (2004)
Algorithm
Binary regions by thresholding the image at all possiblegray levels
Region area changes as the threshold changes
D_area/D_t is minimalmaximal stable
7/30/2019 Feature Detection and Matching
76/80
Keypoint Detection
Very active area (Xiao and Shah 2003; Koethe 2003; Carneiro and Jepson 2005; Kenney, Zuliani, and
Manjunath 2005; Bay, Tuytelaars, and Van Gool 2006; Platel, Balmachnova, Floracket al. 2006; Rosten and Drummond 2006
Invariance to transformation such as scaling, rotations,noise, and blur
These experimental results, code, and pointers to the surveyed
papers can be found on their Web site athttp://www.robots.ox.ac.uk/vgg/research/affine/
(Mikolajczyk, Tuytelaars, Schmid et al. 2005)
http://www.robots.ox.ac.uk/vgg/research/affine/http://www.robots.ox.ac.uk/vgg/research/affine/7/30/2019 Feature Detection and Matching
77/80
4.1.2 Feature Descriptor
TBD
7/30/2019 Feature Detection and Matching
78/80
SIFT Feature
Scale Invariant Feature Transform (SIFT)
Detect features that densely cover the image
over the full range of scales and locations
Distinctive Image Features from Scale-Invariant Keypoints, David G. Lowe
d h
7/30/2019 Feature Detection and Matching
79/80
Points and Patches
Two approaches to finding feature points andtheir correspondences
detect in one, then track
For video sequence application, tracking, stereo
Detect independently, then match
For image stitching, recognition
d h
7/30/2019 Feature Detection and Matching
80/80
Keypoint Detection and Matching
Four steps:
Feature detection
Feature description
Feature matching
Feature tracking