Surface Modeling with Oriented Particle System Szeliski and Tonnesen Siggraph 1992.
Feature Detection and Matching Reading: Szeliski Chapter 4 Pages: 208~266...
-
Upload
april-malone -
Category
Documents
-
view
223 -
download
4
Transcript of Feature Detection and Matching Reading: Szeliski Chapter 4 Pages: 208~266...
![Page 1: Feature Detection and Matching Reading: Szeliski Chapter 4 Pages: 208~266 xjchen99/teaching/teaching.html.](https://reader030.fdocuments.in/reader030/viewer/2022020117/56649eb45503460f94bbd083/html5/thumbnails/1.jpg)
Feature Detection and Matching
Reading: Szeliski Chapter 4 Pages: 208~266
http://staff.ustc.edu.cn/~xjchen99/teaching/teaching.html
![Page 2: Feature Detection and Matching Reading: Szeliski Chapter 4 Pages: 208~266 xjchen99/teaching/teaching.html.](https://reader030.fdocuments.in/reader030/viewer/2022020117/56649eb45503460f94bbd083/html5/thumbnails/2.jpg)
How to UNDERSTAND?
COMPARE with WHAT we learn before?
![Page 3: Feature Detection and Matching Reading: Szeliski Chapter 4 Pages: 208~266 xjchen99/teaching/teaching.html.](https://reader030.fdocuments.in/reader030/viewer/2022020117/56649eb45503460f94bbd083/html5/thumbnails/3.jpg)
Image Matching
by Diva Sian
by swashford
![Page 4: Feature Detection and Matching Reading: Szeliski Chapter 4 Pages: 208~266 xjchen99/teaching/teaching.html.](https://reader030.fdocuments.in/reader030/viewer/2022020117/56649eb45503460f94bbd083/html5/thumbnails/4.jpg)
Harder Case
by Diva Sian by scgbt
![Page 5: Feature Detection and Matching Reading: Szeliski Chapter 4 Pages: 208~266 xjchen99/teaching/teaching.html.](https://reader030.fdocuments.in/reader030/viewer/2022020117/56649eb45503460f94bbd083/html5/thumbnails/5.jpg)
Even Harder Case
“How the Afghan Girl was Identified by Her Iris Patterns” Read the story
![Page 6: Feature Detection and Matching Reading: Szeliski Chapter 4 Pages: 208~266 xjchen99/teaching/teaching.html.](https://reader030.fdocuments.in/reader030/viewer/2022020117/56649eb45503460f94bbd083/html5/thumbnails/6.jpg)
Harder still?
NASA Mars Rover images
![Page 7: Feature Detection and Matching Reading: Szeliski Chapter 4 Pages: 208~266 xjchen99/teaching/teaching.html.](https://reader030.fdocuments.in/reader030/viewer/2022020117/56649eb45503460f94bbd083/html5/thumbnails/7.jpg)
NASA Mars Rover imageswith SIFT feature matchesFigure by Noah Snavely
Answer below (look for tiny colored squares…)
![Page 8: Feature Detection and Matching Reading: Szeliski Chapter 4 Pages: 208~266 xjchen99/teaching/teaching.html.](https://reader030.fdocuments.in/reader030/viewer/2022020117/56649eb45503460f94bbd083/html5/thumbnails/8.jpg)
Features
All is Vanity, by C. Allan Gilbert, 1873-1929Readings
• Szeliski, Ch 4.1• (optional) K. Mikolajczyk, C. Schmid, A performance evaluation of local
descriptors. In PAMI 27(10):1615-1630- http://www.robots.ox.ac.uk/~vgg/research/affine/det_eval_files/mikolajczyk_
![Page 9: Feature Detection and Matching Reading: Szeliski Chapter 4 Pages: 208~266 xjchen99/teaching/teaching.html.](https://reader030.fdocuments.in/reader030/viewer/2022020117/56649eb45503460f94bbd083/html5/thumbnails/9.jpg)
Image Matching
![Page 10: Feature Detection and Matching Reading: Szeliski Chapter 4 Pages: 208~266 xjchen99/teaching/teaching.html.](https://reader030.fdocuments.in/reader030/viewer/2022020117/56649eb45503460f94bbd083/html5/thumbnails/10.jpg)
Image Matching
![Page 11: Feature Detection and Matching Reading: Szeliski Chapter 4 Pages: 208~266 xjchen99/teaching/teaching.html.](https://reader030.fdocuments.in/reader030/viewer/2022020117/56649eb45503460f94bbd083/html5/thumbnails/11.jpg)
Invariant Local FeaturesFind features that are invariant to transformations
– geometric invariance: translation, rotation, scale– photometric invariance: brightness, exposure, …
Feature Descriptors
![Page 12: Feature Detection and Matching Reading: Szeliski Chapter 4 Pages: 208~266 xjchen99/teaching/teaching.html.](https://reader030.fdocuments.in/reader030/viewer/2022020117/56649eb45503460f94bbd083/html5/thumbnails/12.jpg)
Advantages of Local Features
Locality – features are local, so robust to occlusion and clutter
Distinctiveness: – can differentiate a large database of objects
Quantity– hundreds or thousands in a single image
Efficiency– real-time performance achievable
Generality– exploit different types of features in different situations
![Page 13: Feature Detection and Matching Reading: Szeliski Chapter 4 Pages: 208~266 xjchen99/teaching/teaching.html.](https://reader030.fdocuments.in/reader030/viewer/2022020117/56649eb45503460f94bbd083/html5/thumbnails/13.jpg)
More Motivation…
Feature points are used for:– Image alignment (e.g., mosaics)– 3D reconstruction– Motion tracking– Object recognition– Indexing and database retrieval– Robot navigation– … other
![Page 14: Feature Detection and Matching Reading: Szeliski Chapter 4 Pages: 208~266 xjchen99/teaching/teaching.html.](https://reader030.fdocuments.in/reader030/viewer/2022020117/56649eb45503460f94bbd083/html5/thumbnails/14.jpg)
Features
• Point/patch,• Edge/curve• Region
![Page 15: Feature Detection and Matching Reading: Szeliski Chapter 4 Pages: 208~266 xjchen99/teaching/teaching.html.](https://reader030.fdocuments.in/reader030/viewer/2022020117/56649eb45503460f94bbd083/html5/thumbnails/15.jpg)
Want Uniqueness
• Look for image regions that are unusual– Lead to unambiguous matches in other images
• How to define “unusual”?
![Page 16: Feature Detection and Matching Reading: Szeliski Chapter 4 Pages: 208~266 xjchen99/teaching/teaching.html.](https://reader030.fdocuments.in/reader030/viewer/2022020117/56649eb45503460f94bbd083/html5/thumbnails/16.jpg)
![Page 17: Feature Detection and Matching Reading: Szeliski Chapter 4 Pages: 208~266 xjchen99/teaching/teaching.html.](https://reader030.fdocuments.in/reader030/viewer/2022020117/56649eb45503460f94bbd083/html5/thumbnails/17.jpg)
Corner Detection• Basic idea: Find points where two edges meet—i.e., high gradient in two
directions
• “Cornerness” is undefined at a single pixel, because there’s only one gradient per point– Look at the gradient behavior over a small window
• Categories image windows based on gradient statistics– Constant: Little or no brightness change– Edge: Strong brightness change in single direction– Flow: Parallel stripes– Corner/spot: Strong brightness changes in orthogonal directions
![Page 18: Feature Detection and Matching Reading: Szeliski Chapter 4 Pages: 208~266 xjchen99/teaching/teaching.html.](https://reader030.fdocuments.in/reader030/viewer/2022020117/56649eb45503460f94bbd083/html5/thumbnails/18.jpg)
Matching Criterion
• (weighted) Summed Square Difference – SSD
• Compare an image patch against itself
![Page 19: Feature Detection and Matching Reading: Szeliski Chapter 4 Pages: 208~266 xjchen99/teaching/teaching.html.](https://reader030.fdocuments.in/reader030/viewer/2022020117/56649eb45503460f94bbd083/html5/thumbnails/19.jpg)
Local Measures of Uniqueness
Suppose we only consider a small window of pixels– What defines whether a feature is a good or bad
candidate?
Slide adapted from Darya Frolova, Denis Simakov, Weizmann Institute.
![Page 20: Feature Detection and Matching Reading: Szeliski Chapter 4 Pages: 208~266 xjchen99/teaching/teaching.html.](https://reader030.fdocuments.in/reader030/viewer/2022020117/56649eb45503460f94bbd083/html5/thumbnails/20.jpg)
Feature Detection
“flat” region:no change in all directions
“edge”: no change along the edge direction
“corner”:significant change in all directions
Local measure of feature uniqueness– How does the window change when you shift it?– Shifting the window in any direction causes a big change
Slide adapted from Darya Frolova, Denis Simakov, Weizmann Institute.
![Page 21: Feature Detection and Matching Reading: Szeliski Chapter 4 Pages: 208~266 xjchen99/teaching/teaching.html.](https://reader030.fdocuments.in/reader030/viewer/2022020117/56649eb45503460f94bbd083/html5/thumbnails/21.jpg)
Consider shifting the window W by (u,v)• how do the pixels in W change?• compare each pixel before and after by
summing up the squared differences (SSD)
• this defines an SSD “error” of E(u,v):
Feature Detection: Mathematics
W
![Page 22: Feature Detection and Matching Reading: Szeliski Chapter 4 Pages: 208~266 xjchen99/teaching/teaching.html.](https://reader030.fdocuments.in/reader030/viewer/2022020117/56649eb45503460f94bbd083/html5/thumbnails/22.jpg)
Harris Detector: Mathematics
2
,
( , ) ( , ) ( , ) ( , )x y
E u v w x y I x u y v I x y
Change of intensity for the shift [u,v]:
IntensityShifted
intensityWindow function
orWindow function w(x,y) =
Gaussian1 in window, 0 outside
Auto-correlation
function
![Page 23: Feature Detection and Matching Reading: Szeliski Chapter 4 Pages: 208~266 xjchen99/teaching/teaching.html.](https://reader030.fdocuments.in/reader030/viewer/2022020117/56649eb45503460f94bbd083/html5/thumbnails/23.jpg)
Auto-Correlation Function
![Page 24: Feature Detection and Matching Reading: Szeliski Chapter 4 Pages: 208~266 xjchen99/teaching/teaching.html.](https://reader030.fdocuments.in/reader030/viewer/2022020117/56649eb45503460f94bbd083/html5/thumbnails/24.jpg)
Auto-Correlation Function
Good unique minimum 1D aperture problem No good peak
![Page 25: Feature Detection and Matching Reading: Szeliski Chapter 4 Pages: 208~266 xjchen99/teaching/teaching.html.](https://reader030.fdocuments.in/reader030/viewer/2022020117/56649eb45503460f94bbd083/html5/thumbnails/25.jpg)
Taylor Series expansion of I:
If the motion (u,v) is small, then first order approx is good
Small Motion Assumption
![Page 26: Feature Detection and Matching Reading: Szeliski Chapter 4 Pages: 208~266 xjchen99/teaching/teaching.html.](https://reader030.fdocuments.in/reader030/viewer/2022020117/56649eb45503460f94bbd083/html5/thumbnails/26.jpg)
Feature Detection: Mathematics
• Image gradient– Harris detector with a [-2,-1,0,1,2] filter for Ix– Gaussian filter
![Page 27: Feature Detection and Matching Reading: Szeliski Chapter 4 Pages: 208~266 xjchen99/teaching/teaching.html.](https://reader030.fdocuments.in/reader030/viewer/2022020117/56649eb45503460f94bbd083/html5/thumbnails/27.jpg)
Plugging this into the formula on
Small Motion Assumption
2
,
( , ) ( , ) ( , ) ( , )x y
E u v w x y I x u y v I x y
![Page 28: Feature Detection and Matching Reading: Szeliski Chapter 4 Pages: 208~266 xjchen99/teaching/teaching.html.](https://reader030.fdocuments.in/reader030/viewer/2022020117/56649eb45503460f94bbd083/html5/thumbnails/28.jpg)
Feature Detection: Mathematics
W
![Page 29: Feature Detection and Matching Reading: Szeliski Chapter 4 Pages: 208~266 xjchen99/teaching/teaching.html.](https://reader030.fdocuments.in/reader030/viewer/2022020117/56649eb45503460f94bbd083/html5/thumbnails/29.jpg)
Feature Detection: MathematicsThis can be rewritten:
x-
x+
Auto-correlation matrix
2
2
( ) ( ) ( )( )
( ) ( ) ( )x i x i y i
ii x i y i y i
I I IA
I I I
x x xx
x x x
![Page 30: Feature Detection and Matching Reading: Szeliski Chapter 4 Pages: 208~266 xjchen99/teaching/teaching.html.](https://reader030.fdocuments.in/reader030/viewer/2022020117/56649eb45503460f94bbd083/html5/thumbnails/30.jpg)
Feature Detection: Mathematics
For the example above• You can move the center of the blue window to anywhere on the
blue unit circle• Which directions will result in the largest and smallest E values?• We can find these directions by looking at the eigenvectors of H
![Page 31: Feature Detection and Matching Reading: Szeliski Chapter 4 Pages: 208~266 xjchen99/teaching/teaching.html.](https://reader030.fdocuments.in/reader030/viewer/2022020117/56649eb45503460f94bbd083/html5/thumbnails/31.jpg)
Quick eigenvalue/eigenvector reviewThe eigenvectors of a matrix A are the vectors x that satisfy:
The scalar is the eigenvalue corresponding to x– The eigenvalues are found by solving:
– In our case, A = H is a 2x2 matrix, so we have
– The solution:
Once you know , you find x by solving
![Page 32: Feature Detection and Matching Reading: Szeliski Chapter 4 Pages: 208~266 xjchen99/teaching/teaching.html.](https://reader030.fdocuments.in/reader030/viewer/2022020117/56649eb45503460f94bbd083/html5/thumbnails/32.jpg)
Feature Detection: MathematicsThis can be rewritten:
Eigenvalues and eigenvectors of H=A
• Define shifts with the smallest and largest change (E value)• x+ = direction of largest increase in E.
• + = amount of increase in direction x+
• x- = direction of smallest increase in E.
• - = amount of increase in direction x+
x-
x+
![Page 33: Feature Detection and Matching Reading: Szeliski Chapter 4 Pages: 208~266 xjchen99/teaching/teaching.html.](https://reader030.fdocuments.in/reader030/viewer/2022020117/56649eb45503460f94bbd083/html5/thumbnails/33.jpg)
Feature Detection: Mathematics
( , ) ,u
E u v u v Hv
Intensity change in shifting window: eigenvalue analysis
1, 2 – eigenvalues of H
(max)-1/2
(min)-1/2
Ellipse E(u,v) = const
If we try every possible orientation n,the max. change in intensity is 2
![Page 34: Feature Detection and Matching Reading: Szeliski Chapter 4 Pages: 208~266 xjchen99/teaching/teaching.html.](https://reader030.fdocuments.in/reader030/viewer/2022020117/56649eb45503460f94bbd083/html5/thumbnails/34.jpg)
Feature Detection: Mathematics
1
2
“Corner”1 and 2 are large, 1 ~ 2;E increases in all directions
1 and 2 are small;E is almost constant in all directions
“Edge” 1 >> 2
“Edge” 2 >> 1
“Flat” region
Classification of image points using eigenvalues of H:
![Page 35: Feature Detection and Matching Reading: Szeliski Chapter 4 Pages: 208~266 xjchen99/teaching/teaching.html.](https://reader030.fdocuments.in/reader030/viewer/2022020117/56649eb45503460f94bbd083/html5/thumbnails/35.jpg)
Feature Detection: MathematicsHow are +, x+, -, and x+ relevant for feature detection?
• What’s our feature scoring function?
![Page 36: Feature Detection and Matching Reading: Szeliski Chapter 4 Pages: 208~266 xjchen99/teaching/teaching.html.](https://reader030.fdocuments.in/reader030/viewer/2022020117/56649eb45503460f94bbd083/html5/thumbnails/36.jpg)
Feature Detection: MathematicsHow are +, x+, -, and x+ relevant for feature detection?
• What’s our feature scoring function?
Want E(u,v) to be large for small shifts in all directions• the minimum of E(u,v) should be large, over all unit vectors [u v]
• this minimum is given by the smaller eigenvalue (-) of H
![Page 37: Feature Detection and Matching Reading: Szeliski Chapter 4 Pages: 208~266 xjchen99/teaching/teaching.html.](https://reader030.fdocuments.in/reader030/viewer/2022020117/56649eb45503460f94bbd083/html5/thumbnails/37.jpg)
Feature DetectionHere’s what you do
• Compute the gradient at each point in the image• Create the H matrix from the entries in the gradient• Compute the eigenvalues. • Find points with large response (- > threshold)
• Choose those points where - is a local maximum as features
![Page 38: Feature Detection and Matching Reading: Szeliski Chapter 4 Pages: 208~266 xjchen99/teaching/teaching.html.](https://reader030.fdocuments.in/reader030/viewer/2022020117/56649eb45503460f94bbd083/html5/thumbnails/38.jpg)
Feature DetectionHere’s what you do
• Compute the gradient at each point in the image• Create the A matrix from the entries in the gradient• Compute the eigenvalues. • Find points with large response (- > threshold)
• Choose those points where - is a local maximum as features
[Shi and Tomasi 1994]
![Page 39: Feature Detection and Matching Reading: Szeliski Chapter 4 Pages: 208~266 xjchen99/teaching/teaching.html.](https://reader030.fdocuments.in/reader030/viewer/2022020117/56649eb45503460f94bbd083/html5/thumbnails/39.jpg)
Harris Detector
Measure of corner response:
2det traceR H k H
1 2
1 2
det
trace
H
H
(k – empirical constant, k = 0.04-0.06)
Harris and Stephens 1988
• The trace is the sum of the diagonals, i.e., trace(H) = h11 + h22
• Very similar to - but less expensive (no square root)
![Page 40: Feature Detection and Matching Reading: Szeliski Chapter 4 Pages: 208~266 xjchen99/teaching/teaching.html.](https://reader030.fdocuments.in/reader030/viewer/2022020117/56649eb45503460f94bbd083/html5/thumbnails/40.jpg)
Harris Detector: Mathematics
1
2 “Corner”
“Edge”
“Edge”
“Flat”
• R depends only on eigenvalues of H
• R is large for a corner
• R is negative with large magnitude for an edge
• |R| is small for a flat region
R > 0
R < 0
R < 0|R| small
![Page 41: Feature Detection and Matching Reading: Szeliski Chapter 4 Pages: 208~266 xjchen99/teaching/teaching.html.](https://reader030.fdocuments.in/reader030/viewer/2022020117/56649eb45503460f94bbd083/html5/thumbnails/41.jpg)
Harris Detector
• The Algorithm:– Find points with large corner response function R
(R > threshold)– Take the points of local maxima of R
![Page 42: Feature Detection and Matching Reading: Szeliski Chapter 4 Pages: 208~266 xjchen99/teaching/teaching.html.](https://reader030.fdocuments.in/reader030/viewer/2022020117/56649eb45503460f94bbd083/html5/thumbnails/42.jpg)
Harmonic Mean
• Smoother function in the region where
Brown, M., Szeliski, R., and Winder, S. (2005).
0 1
0 1
det
trf
H
H
0 1
![Page 43: Feature Detection and Matching Reading: Szeliski Chapter 4 Pages: 208~266 xjchen99/teaching/teaching.html.](https://reader030.fdocuments.in/reader030/viewer/2022020117/56649eb45503460f94bbd083/html5/thumbnails/43.jpg)
Isocontours of Response
![Page 44: Feature Detection and Matching Reading: Szeliski Chapter 4 Pages: 208~266 xjchen99/teaching/teaching.html.](https://reader030.fdocuments.in/reader030/viewer/2022020117/56649eb45503460f94bbd083/html5/thumbnails/44.jpg)
Harris Detector: Workflow
![Page 45: Feature Detection and Matching Reading: Szeliski Chapter 4 Pages: 208~266 xjchen99/teaching/teaching.html.](https://reader030.fdocuments.in/reader030/viewer/2022020117/56649eb45503460f94bbd083/html5/thumbnails/45.jpg)
Harris Detector: Workflow
Compute corner response R
![Page 46: Feature Detection and Matching Reading: Szeliski Chapter 4 Pages: 208~266 xjchen99/teaching/teaching.html.](https://reader030.fdocuments.in/reader030/viewer/2022020117/56649eb45503460f94bbd083/html5/thumbnails/46.jpg)
Harris Detector: Workflow
Find points with large corner response: R>threshold
![Page 47: Feature Detection and Matching Reading: Szeliski Chapter 4 Pages: 208~266 xjchen99/teaching/teaching.html.](https://reader030.fdocuments.in/reader030/viewer/2022020117/56649eb45503460f94bbd083/html5/thumbnails/47.jpg)
Harris Detector: Workflow
Take only the points of local maxima of R
![Page 48: Feature Detection and Matching Reading: Szeliski Chapter 4 Pages: 208~266 xjchen99/teaching/teaching.html.](https://reader030.fdocuments.in/reader030/viewer/2022020117/56649eb45503460f94bbd083/html5/thumbnails/48.jpg)
Harris Detector: Workflow
![Page 49: Feature Detection and Matching Reading: Szeliski Chapter 4 Pages: 208~266 xjchen99/teaching/teaching.html.](https://reader030.fdocuments.in/reader030/viewer/2022020117/56649eb45503460f94bbd083/html5/thumbnails/49.jpg)
Example: Gradient Covariances
Full imageDetail of image with gradient covar-
iance ellipses for 3 x 3 windows
from Forsyth & Ponce
Corners are where both eigenvalues are big
![Page 50: Feature Detection and Matching Reading: Szeliski Chapter 4 Pages: 208~266 xjchen99/teaching/teaching.html.](https://reader030.fdocuments.in/reader030/viewer/2022020117/56649eb45503460f94bbd083/html5/thumbnails/50.jpg)
Example: Corner Detection (for camera calibration)
courtesy of B. Wilburn
![Page 51: Feature Detection and Matching Reading: Szeliski Chapter 4 Pages: 208~266 xjchen99/teaching/teaching.html.](https://reader030.fdocuments.in/reader030/viewer/2022020117/56649eb45503460f94bbd083/html5/thumbnails/51.jpg)
Example: Corner Detection
courtesy of S. Smith
SUSAN corners
![Page 52: Feature Detection and Matching Reading: Szeliski Chapter 4 Pages: 208~266 xjchen99/teaching/teaching.html.](https://reader030.fdocuments.in/reader030/viewer/2022020117/56649eb45503460f94bbd083/html5/thumbnails/52.jpg)
Harris Detector: Summary• Average intensity change in direction [u,v] can be
expressed as a bilinear form:
• Describe a point in terms of eigenvalues of H:measure of corner response
• A good (corner) point should have a large intensity change in all directions, i.e. R should be large positive
( , ) ,u
E u v u v Hv
2
1 2 1 2R k
![Page 53: Feature Detection and Matching Reading: Szeliski Chapter 4 Pages: 208~266 xjchen99/teaching/teaching.html.](https://reader030.fdocuments.in/reader030/viewer/2022020117/56649eb45503460f94bbd083/html5/thumbnails/53.jpg)
Outline of Feature Detection
1. Compute the horizontal and vertical derivatives of the image Ix and Iy by convolving the original image with derivatives of Gaussians
2. Compute the three images corresponding to the outer products of these gradients. (The matrix H is symmetric, so only three entries are needed.)
3. Convolve each of these images with a larger Gaussian.4. Compute a scalar interest measure using one of the formulas
discussed above.5. Find local maxima above a certain threshold and report
them as detected feature point locations.
![Page 54: Feature Detection and Matching Reading: Szeliski Chapter 4 Pages: 208~266 xjchen99/teaching/teaching.html.](https://reader030.fdocuments.in/reader030/viewer/2022020117/56649eb45503460f94bbd083/html5/thumbnails/54.jpg)
Adaptive Non-Maximal Suppression (ANMS)
• Local maxima• Response value should be significantly (10%)
larger than all of its neighbors within a radius (r)
(a) Strongest 250 (b) Strongest 500
![Page 55: Feature Detection and Matching Reading: Szeliski Chapter 4 Pages: 208~266 xjchen99/teaching/teaching.html.](https://reader030.fdocuments.in/reader030/viewer/2022020117/56649eb45503460f94bbd083/html5/thumbnails/55.jpg)
Adaptive Non-Maximal Suppression (ANMS)
(a) Strongest 250 (b) Strongest 500
(c) ANMS 250, r = 24 (d) ANMS 500, r = 16
![Page 56: Feature Detection and Matching Reading: Szeliski Chapter 4 Pages: 208~266 xjchen99/teaching/teaching.html.](https://reader030.fdocuments.in/reader030/viewer/2022020117/56649eb45503460f94bbd083/html5/thumbnails/56.jpg)
Invariance
Suppose you rotate the image by some angle– Will you still pick up the same features?
What if you change the brightness?
Scale?
![Page 57: Feature Detection and Matching Reading: Szeliski Chapter 4 Pages: 208~266 xjchen99/teaching/teaching.html.](https://reader030.fdocuments.in/reader030/viewer/2022020117/56649eb45503460f94bbd083/html5/thumbnails/57.jpg)
Invariance
• Repeatability of feature detector: frequency with which keypoints are detected in one image are found
within ε (ε=1.5) pixels of the corresponding location in a transformed image
![Page 58: Feature Detection and Matching Reading: Szeliski Chapter 4 Pages: 208~266 xjchen99/teaching/teaching.html.](https://reader030.fdocuments.in/reader030/viewer/2022020117/56649eb45503460f94bbd083/html5/thumbnails/58.jpg)
Scale Invariance
• Detect features at a variety of scales
• Multiple resolutions in a pyramid
• Matching in all possible levels
![Page 59: Feature Detection and Matching Reading: Szeliski Chapter 4 Pages: 208~266 xjchen99/teaching/teaching.html.](https://reader030.fdocuments.in/reader030/viewer/2022020117/56649eb45503460f94bbd083/html5/thumbnails/59.jpg)
Multi-Scale Oriented Patches
![Page 60: Feature Detection and Matching Reading: Szeliski Chapter 4 Pages: 208~266 xjchen99/teaching/teaching.html.](https://reader030.fdocuments.in/reader030/viewer/2022020117/56649eb45503460f94bbd083/html5/thumbnails/60.jpg)
Scale invariant detection
Suppose you’re looking for corners
![Page 61: Feature Detection and Matching Reading: Szeliski Chapter 4 Pages: 208~266 xjchen99/teaching/teaching.html.](https://reader030.fdocuments.in/reader030/viewer/2022020117/56649eb45503460f94bbd083/html5/thumbnails/61.jpg)
Feature Detection
Key idea: find scale that gives local maximum of f– f is a local maximum in both position and scale– Common definition of f: Laplacian– Difference between two Gaussian filtered images
with different sigmas
![Page 62: Feature Detection and Matching Reading: Szeliski Chapter 4 Pages: 208~266 xjchen99/teaching/teaching.html.](https://reader030.fdocuments.in/reader030/viewer/2022020117/56649eb45503460f94bbd083/html5/thumbnails/62.jpg)
Feature Detection
• Stable features in both location and scale• Laplacian of Gaussian (LoG)
• Difference of Gaussian2 2 2 2
2 21 2
1 2
1 2
2 2
2 2
1 1 1
2
x y x y
DoG G G e e
![Page 63: Feature Detection and Matching Reading: Szeliski Chapter 4 Pages: 208~266 xjchen99/teaching/teaching.html.](https://reader030.fdocuments.in/reader030/viewer/2022020117/56649eb45503460f94bbd083/html5/thumbnails/63.jpg)
Slide from Tinne Tuytelaars
Lindeberg et al, 1996
Slide from Tinne Tuytelaars
Lindeberg et al., 1996
![Page 64: Feature Detection and Matching Reading: Szeliski Chapter 4 Pages: 208~266 xjchen99/teaching/teaching.html.](https://reader030.fdocuments.in/reader030/viewer/2022020117/56649eb45503460f94bbd083/html5/thumbnails/64.jpg)
![Page 65: Feature Detection and Matching Reading: Szeliski Chapter 4 Pages: 208~266 xjchen99/teaching/teaching.html.](https://reader030.fdocuments.in/reader030/viewer/2022020117/56649eb45503460f94bbd083/html5/thumbnails/65.jpg)
![Page 66: Feature Detection and Matching Reading: Szeliski Chapter 4 Pages: 208~266 xjchen99/teaching/teaching.html.](https://reader030.fdocuments.in/reader030/viewer/2022020117/56649eb45503460f94bbd083/html5/thumbnails/66.jpg)
![Page 67: Feature Detection and Matching Reading: Szeliski Chapter 4 Pages: 208~266 xjchen99/teaching/teaching.html.](https://reader030.fdocuments.in/reader030/viewer/2022020117/56649eb45503460f94bbd083/html5/thumbnails/67.jpg)
![Page 68: Feature Detection and Matching Reading: Szeliski Chapter 4 Pages: 208~266 xjchen99/teaching/teaching.html.](https://reader030.fdocuments.in/reader030/viewer/2022020117/56649eb45503460f94bbd083/html5/thumbnails/68.jpg)
![Page 69: Feature Detection and Matching Reading: Szeliski Chapter 4 Pages: 208~266 xjchen99/teaching/teaching.html.](https://reader030.fdocuments.in/reader030/viewer/2022020117/56649eb45503460f94bbd083/html5/thumbnails/69.jpg)
![Page 70: Feature Detection and Matching Reading: Szeliski Chapter 4 Pages: 208~266 xjchen99/teaching/teaching.html.](https://reader030.fdocuments.in/reader030/viewer/2022020117/56649eb45503460f94bbd083/html5/thumbnails/70.jpg)
![Page 71: Feature Detection and Matching Reading: Szeliski Chapter 4 Pages: 208~266 xjchen99/teaching/teaching.html.](https://reader030.fdocuments.in/reader030/viewer/2022020117/56649eb45503460f94bbd083/html5/thumbnails/71.jpg)
Scale-Space Feature Detection
Empirical number of levels in each octave = 31.6
![Page 72: Feature Detection and Matching Reading: Szeliski Chapter 4 Pages: 208~266 xjchen99/teaching/teaching.html.](https://reader030.fdocuments.in/reader030/viewer/2022020117/56649eb45503460f94bbd083/html5/thumbnails/72.jpg)
SIFT Feature Detection
• Scale Invariant Feature Transform
![Page 73: Feature Detection and Matching Reading: Szeliski Chapter 4 Pages: 208~266 xjchen99/teaching/teaching.html.](https://reader030.fdocuments.in/reader030/viewer/2022020117/56649eb45503460f94bbd083/html5/thumbnails/73.jpg)
Feature Detector
• Harris and DoG detectors tend to respond at complementary locations.
Sample image Harris response DoG response
![Page 74: Feature Detection and Matching Reading: Szeliski Chapter 4 Pages: 208~266 xjchen99/teaching/teaching.html.](https://reader030.fdocuments.in/reader030/viewer/2022020117/56649eb45503460f94bbd083/html5/thumbnails/74.jpg)
Rotation Invariance and orientation estimation
• Dominant orientation– Average gradient within a region around the keypoint
(larger window than detection window to make it more reliable)
– Orientation histogram
![Page 75: Feature Detection and Matching Reading: Szeliski Chapter 4 Pages: 208~266 xjchen99/teaching/teaching.html.](https://reader030.fdocuments.in/reader030/viewer/2022020117/56649eb45503460f94bbd083/html5/thumbnails/75.jpg)
Affine Invariance
• For wide baseline stereo matching or object recognition
![Page 76: Feature Detection and Matching Reading: Szeliski Chapter 4 Pages: 208~266 xjchen99/teaching/teaching.html.](https://reader030.fdocuments.in/reader030/viewer/2022020117/56649eb45503460f94bbd083/html5/thumbnails/76.jpg)
Affine Invariance
• For a small enough patch, any continuous image wrapping can be well approximated by an affine deformation
• Eigenvalue analysis for Hessian matrix or auto-correlation: using the principle axis and ratios to fit the affine coordinate frame
1 1 2 2, , , x x
(Lindeberg and Garding 1997; Baumberg 2000; Mikolajczyk and Schmid 2004; Mikolajczyk, Tuytelaars, Schmid et al. 2005; Tuytelaars and Mikolajczyk 2007)
![Page 77: Feature Detection and Matching Reading: Szeliski Chapter 4 Pages: 208~266 xjchen99/teaching/teaching.html.](https://reader030.fdocuments.in/reader030/viewer/2022020117/56649eb45503460f94bbd083/html5/thumbnails/77.jpg)
Affine Invariance
• Maximally Stable Extremal Region (MSER)– Matas, Chum, Urban et al. (2004)
• Algorithm – Binary regions by thresholding the image at all possible
gray levels– Region area changes as the threshold changes– D_area/D_t is minimal maximal stable
![Page 78: Feature Detection and Matching Reading: Szeliski Chapter 4 Pages: 208~266 xjchen99/teaching/teaching.html.](https://reader030.fdocuments.in/reader030/viewer/2022020117/56649eb45503460f94bbd083/html5/thumbnails/78.jpg)
Keypoint Detection
• Very active area– (Xiao and Shah 2003; Koethe 2003; Carneiro and Jepson 2005; Kenney, Zuliani, and
Manjunath 2005; Bay, Tuytelaars, and Van Gool 2006; Platel, Balmachnova, Florack et al. 2006; Rosten and Drummond 2006
• Invariance to transformation such as scaling, rotations, noise, and blur
• These experimental results, code, and pointers to the surveyed papers can be found on their Web site at http://www.robots.ox.ac.uk/vgg/research/affine/
(Mikolajczyk, Tuytelaars, Schmid et al. 2005)
![Page 79: Feature Detection and Matching Reading: Szeliski Chapter 4 Pages: 208~266 xjchen99/teaching/teaching.html.](https://reader030.fdocuments.in/reader030/viewer/2022020117/56649eb45503460f94bbd083/html5/thumbnails/79.jpg)
4.1.2 Feature Descriptor
• TBD
![Page 80: Feature Detection and Matching Reading: Szeliski Chapter 4 Pages: 208~266 xjchen99/teaching/teaching.html.](https://reader030.fdocuments.in/reader030/viewer/2022020117/56649eb45503460f94bbd083/html5/thumbnails/80.jpg)
SIFT Feature
• Scale Invariant Feature Transform (SIFT)• Detect features that densely cover the image
over the full range of scales and locations
Distinctive Image Features from Scale-Invariant Keypoints, David G. Lowe
![Page 81: Feature Detection and Matching Reading: Szeliski Chapter 4 Pages: 208~266 xjchen99/teaching/teaching.html.](https://reader030.fdocuments.in/reader030/viewer/2022020117/56649eb45503460f94bbd083/html5/thumbnails/81.jpg)
Points and Patches
• Two approaches to finding feature points and their correspondences – detect in one, then track
• For video sequence application, tracking, stereo…
– Detect independently, then match • For image stitching, recognition …
![Page 82: Feature Detection and Matching Reading: Szeliski Chapter 4 Pages: 208~266 xjchen99/teaching/teaching.html.](https://reader030.fdocuments.in/reader030/viewer/2022020117/56649eb45503460f94bbd083/html5/thumbnails/82.jpg)
Keypoint Detection and Matching
• Four steps:– Feature detection– Feature description– Feature matching– Feature tracking