Feature detectors and descriptors · 2008-10-05 · 2nd moment(s) Tuytelaars, ‘00 2nd moment(s)...
Transcript of Feature detectors and descriptors · 2008-10-05 · 2nd moment(s) Tuytelaars, ‘00 2nd moment(s)...
![Page 1: Feature detectors and descriptors · 2008-10-05 · 2nd moment(s) Tuytelaars, ‘00 2nd moment(s) Mikolajczyk & Schmid ’01, ‘02 2nd moment(s) Harris corner others others Matas,](https://reader033.fdocuments.in/reader033/viewer/2022060516/5f8f9138297f2177275f0b62/html5/thumbnails/1.jpg)
Feature detectorsand descriptors
Fei-Fei Li
![Page 2: Feature detectors and descriptors · 2008-10-05 · 2nd moment(s) Tuytelaars, ‘00 2nd moment(s) Mikolajczyk & Schmid ’01, ‘02 2nd moment(s) Harris corner others others Matas,](https://reader033.fdocuments.in/reader033/viewer/2022060516/5f8f9138297f2177275f0b62/html5/thumbnails/2.jpg)
Feature Detection
Feature Description
Matching / Indexing /
Recognition
local descriptors –(invariant) vectors
detected points – (~300)coordinates, neighbourhoods
database of local descriptors
e.g. DoG
e.g. SIFT
e.g. Mahalanobis distance + Voting algorithm[Mikolajczyk & Schmid ’01]
![Page 3: Feature detectors and descriptors · 2008-10-05 · 2nd moment(s) Tuytelaars, ‘00 2nd moment(s) Mikolajczyk & Schmid ’01, ‘02 2nd moment(s) Harris corner others others Matas,](https://reader033.fdocuments.in/reader033/viewer/2022060516/5f8f9138297f2177275f0b62/html5/thumbnails/3.jpg)
Some of the challenges…
• Geometry– Rotation– Similarity (rotation + uniform scale)
– Affine (scale dependent on direction)valid for: orthographic camera, locally planar object
• Photometry– Affine intensity change (I → a I + b)
![Page 4: Feature detectors and descriptors · 2008-10-05 · 2nd moment(s) Tuytelaars, ‘00 2nd moment(s) Mikolajczyk & Schmid ’01, ‘02 2nd moment(s) Harris corner others others Matas,](https://reader033.fdocuments.in/reader033/viewer/2022060516/5f8f9138297f2177275f0b62/html5/thumbnails/4.jpg)
2nd
moment(s)Tuytelaars,
‘00
2nd
moment(s)Mikolajczyk& Schmid’01, ‘02
2nd
moment(s)Harris corner
othersothers
Matas, ‘02
Kadir & Brady, 01
SIFT, PCA-SIFT
Lowe ’99 (DoG)
AffineScaleRotationIntensityDescrip-tor
Detector
![Page 5: Feature detectors and descriptors · 2008-10-05 · 2nd moment(s) Tuytelaars, ‘00 2nd moment(s) Mikolajczyk & Schmid ’01, ‘02 2nd moment(s) Harris corner others others Matas,](https://reader033.fdocuments.in/reader033/viewer/2022060516/5f8f9138297f2177275f0b62/html5/thumbnails/5.jpg)
2nd
moment(s)Tuytelaars,
‘00
2nd
moment(s)Mikolajczyk& Schmid’01, ‘02
2nd
moment(s)Harris corner
othersothers
Matas, ‘02
Kadir & Brady, 01
SIFT, PCA-SIFT
Lowe ’99 (DoG)
AffineScaleRotationIntensityDescriptorDetector
![Page 6: Feature detectors and descriptors · 2008-10-05 · 2nd moment(s) Tuytelaars, ‘00 2nd moment(s) Mikolajczyk & Schmid ’01, ‘02 2nd moment(s) Harris corner others others Matas,](https://reader033.fdocuments.in/reader033/viewer/2022060516/5f8f9138297f2177275f0b62/html5/thumbnails/6.jpg)
An introductory example:
Harris corner detector
C.Harris, M.Stephens. “A Combined Corner and Edge Detector”. 1988
![Page 7: Feature detectors and descriptors · 2008-10-05 · 2nd moment(s) Tuytelaars, ‘00 2nd moment(s) Mikolajczyk & Schmid ’01, ‘02 2nd moment(s) Harris corner others others Matas,](https://reader033.fdocuments.in/reader033/viewer/2022060516/5f8f9138297f2177275f0b62/html5/thumbnails/7.jpg)
Harris Detector: Basic Idea
“flat” region:no change in all directions
“edge”:no change along the edge direction
“corner”:significant change in all directions
![Page 8: Feature detectors and descriptors · 2008-10-05 · 2nd moment(s) Tuytelaars, ‘00 2nd moment(s) Mikolajczyk & Schmid ’01, ‘02 2nd moment(s) Harris corner others others Matas,](https://reader033.fdocuments.in/reader033/viewer/2022060516/5f8f9138297f2177275f0b62/html5/thumbnails/8.jpg)
Harris Detector: Mathematics
[ ]2
,
( , ) ( , ) ( , ) ( , )x y
E u v w x y I x u y v I x y= + + −∑
Change of intensity for the shift [u,v]:
IntensityShifted intensity
Window function
orWindow function w(x,y) =
Gaussian1 in window, 0 outside
![Page 9: Feature detectors and descriptors · 2008-10-05 · 2nd moment(s) Tuytelaars, ‘00 2nd moment(s) Mikolajczyk & Schmid ’01, ‘02 2nd moment(s) Harris corner others others Matas,](https://reader033.fdocuments.in/reader033/viewer/2022060516/5f8f9138297f2177275f0b62/html5/thumbnails/9.jpg)
Harris Detector: Mathematics
[ ]( , ) ,u
E u v u v Mv⎡ ⎤
≅ ⎢ ⎥⎣ ⎦
For small shifts [u,v] we have a bilinear approximation:
2
2,
( , ) x x y
x y x y y
I I IM w x y
I I I⎡ ⎤
= ⎢ ⎥⎢ ⎥⎣ ⎦
∑
where M is a 2×2 matrix computed from image derivatives:
![Page 10: Feature detectors and descriptors · 2008-10-05 · 2nd moment(s) Tuytelaars, ‘00 2nd moment(s) Mikolajczyk & Schmid ’01, ‘02 2nd moment(s) Harris corner others others Matas,](https://reader033.fdocuments.in/reader033/viewer/2022060516/5f8f9138297f2177275f0b62/html5/thumbnails/10.jpg)
Harris Detector: Mathematics
[ ]( , ) ,u
E u v u v Mv⎡ ⎤
≅ ⎢ ⎥⎣ ⎦
Intensity change in shifting window: eigenvalue analysis
λ1, λ2 – eigenvalues of M
direction of the slowest change
direction of the fastest change
(λmax)-1/2
(λmin)-1/2
Ellipse E(u,v) = const
![Page 11: Feature detectors and descriptors · 2008-10-05 · 2nd moment(s) Tuytelaars, ‘00 2nd moment(s) Mikolajczyk & Schmid ’01, ‘02 2nd moment(s) Harris corner others others Matas,](https://reader033.fdocuments.in/reader033/viewer/2022060516/5f8f9138297f2177275f0b62/html5/thumbnails/11.jpg)
Harris Detector: Mathematics
λ1
λ2
“Corner”λ1 and λ2 are large,λ1 ~ λ2;E increases in all directions
λ1 and λ2 are small;E is almost constant in all directions
“Edge”λ1 >> λ2
“Edge”λ2 >> λ1
“Flat”region
Classification of image points using eigenvalues of M:
![Page 12: Feature detectors and descriptors · 2008-10-05 · 2nd moment(s) Tuytelaars, ‘00 2nd moment(s) Mikolajczyk & Schmid ’01, ‘02 2nd moment(s) Harris corner others others Matas,](https://reader033.fdocuments.in/reader033/viewer/2022060516/5f8f9138297f2177275f0b62/html5/thumbnails/12.jpg)
Harris Detector: Mathematics
Measure of corner response:
( )2det traceR M k M= −
1 2
1 2
dettrace
MM
λ λλ λ
== +
(k – empirical constant, k = 0.04-0.06)
![Page 13: Feature detectors and descriptors · 2008-10-05 · 2nd moment(s) Tuytelaars, ‘00 2nd moment(s) Mikolajczyk & Schmid ’01, ‘02 2nd moment(s) Harris corner others others Matas,](https://reader033.fdocuments.in/reader033/viewer/2022060516/5f8f9138297f2177275f0b62/html5/thumbnails/13.jpg)
Harris Detector: Mathematics
λ1
λ2 “Corner”
“Edge”
“Edge”
“Flat”
• R depends only on eigenvalues of M
• R is large for a corner
• R is negative with large magnitude for an edge
• |R| is small for a flatregion
R > 0
R < 0
R < 0|R| small
![Page 14: Feature detectors and descriptors · 2008-10-05 · 2nd moment(s) Tuytelaars, ‘00 2nd moment(s) Mikolajczyk & Schmid ’01, ‘02 2nd moment(s) Harris corner others others Matas,](https://reader033.fdocuments.in/reader033/viewer/2022060516/5f8f9138297f2177275f0b62/html5/thumbnails/14.jpg)
Harris Detector
• The Algorithm:– Find points with large corner response
function R (R > threshold)– Take the points of local maxima of R
![Page 15: Feature detectors and descriptors · 2008-10-05 · 2nd moment(s) Tuytelaars, ‘00 2nd moment(s) Mikolajczyk & Schmid ’01, ‘02 2nd moment(s) Harris corner others others Matas,](https://reader033.fdocuments.in/reader033/viewer/2022060516/5f8f9138297f2177275f0b62/html5/thumbnails/15.jpg)
Harris Detector: Workflow
![Page 16: Feature detectors and descriptors · 2008-10-05 · 2nd moment(s) Tuytelaars, ‘00 2nd moment(s) Mikolajczyk & Schmid ’01, ‘02 2nd moment(s) Harris corner others others Matas,](https://reader033.fdocuments.in/reader033/viewer/2022060516/5f8f9138297f2177275f0b62/html5/thumbnails/16.jpg)
Harris Detector: WorkflowCompute corner response R
![Page 17: Feature detectors and descriptors · 2008-10-05 · 2nd moment(s) Tuytelaars, ‘00 2nd moment(s) Mikolajczyk & Schmid ’01, ‘02 2nd moment(s) Harris corner others others Matas,](https://reader033.fdocuments.in/reader033/viewer/2022060516/5f8f9138297f2177275f0b62/html5/thumbnails/17.jpg)
Harris Detector: WorkflowFind points with large corner response: R>threshold
![Page 18: Feature detectors and descriptors · 2008-10-05 · 2nd moment(s) Tuytelaars, ‘00 2nd moment(s) Mikolajczyk & Schmid ’01, ‘02 2nd moment(s) Harris corner others others Matas,](https://reader033.fdocuments.in/reader033/viewer/2022060516/5f8f9138297f2177275f0b62/html5/thumbnails/18.jpg)
Harris Detector: WorkflowTake only the points of local maxima of R
![Page 19: Feature detectors and descriptors · 2008-10-05 · 2nd moment(s) Tuytelaars, ‘00 2nd moment(s) Mikolajczyk & Schmid ’01, ‘02 2nd moment(s) Harris corner others others Matas,](https://reader033.fdocuments.in/reader033/viewer/2022060516/5f8f9138297f2177275f0b62/html5/thumbnails/19.jpg)
Harris Detector: Workflow
![Page 20: Feature detectors and descriptors · 2008-10-05 · 2nd moment(s) Tuytelaars, ‘00 2nd moment(s) Mikolajczyk & Schmid ’01, ‘02 2nd moment(s) Harris corner others others Matas,](https://reader033.fdocuments.in/reader033/viewer/2022060516/5f8f9138297f2177275f0b62/html5/thumbnails/20.jpg)
Harris Detector: Summary
• Average intensity change in direction [u,v] can be expressed as a bilinear form:
• Describe a point in terms of eigenvalues of M:measure of corner response
• A good (corner) point should have a large intensity change in all directions, i.e. R should be large positive
[ ]( , ) ,u
E u v u v Mv⎡ ⎤
≅ ⎢ ⎥⎣ ⎦
( )21 2 1 2R kλ λ λ λ= − +
![Page 21: Feature detectors and descriptors · 2008-10-05 · 2nd moment(s) Tuytelaars, ‘00 2nd moment(s) Mikolajczyk & Schmid ’01, ‘02 2nd moment(s) Harris corner others others Matas,](https://reader033.fdocuments.in/reader033/viewer/2022060516/5f8f9138297f2177275f0b62/html5/thumbnails/21.jpg)
Harris Detector: Some Properties
• Rotation invariance
Ellipse rotates but its shape (i.e. eigenvalues) remains the same
Corner response R is invariant to image rotation
![Page 22: Feature detectors and descriptors · 2008-10-05 · 2nd moment(s) Tuytelaars, ‘00 2nd moment(s) Mikolajczyk & Schmid ’01, ‘02 2nd moment(s) Harris corner others others Matas,](https://reader033.fdocuments.in/reader033/viewer/2022060516/5f8f9138297f2177275f0b62/html5/thumbnails/22.jpg)
Harris Detector: Some Properties
• Partial invariance to affine intensitychange
Only derivatives are used => invariance to intensity shift I → I + b
Intensity scale: I → a I
R
x (image coordinate)
threshold
R
x (image coordinate)
![Page 23: Feature detectors and descriptors · 2008-10-05 · 2nd moment(s) Tuytelaars, ‘00 2nd moment(s) Mikolajczyk & Schmid ’01, ‘02 2nd moment(s) Harris corner others others Matas,](https://reader033.fdocuments.in/reader033/viewer/2022060516/5f8f9138297f2177275f0b62/html5/thumbnails/23.jpg)
Harris Detector: Some Properties
• But: non-invariant to image scale!
All points will be classified as edges
Corner !
![Page 24: Feature detectors and descriptors · 2008-10-05 · 2nd moment(s) Tuytelaars, ‘00 2nd moment(s) Mikolajczyk & Schmid ’01, ‘02 2nd moment(s) Harris corner others others Matas,](https://reader033.fdocuments.in/reader033/viewer/2022060516/5f8f9138297f2177275f0b62/html5/thumbnails/24.jpg)
Harris Detector: Some Properties
• Quality of Harris detector for different scale changes
Repeatability rate:# correspondences
# possible correspondences
C.Schmid et.al. “Evaluation of Interest Point Detectors”. IJCV 2000
![Page 25: Feature detectors and descriptors · 2008-10-05 · 2nd moment(s) Tuytelaars, ‘00 2nd moment(s) Mikolajczyk & Schmid ’01, ‘02 2nd moment(s) Harris corner others others Matas,](https://reader033.fdocuments.in/reader033/viewer/2022060516/5f8f9138297f2177275f0b62/html5/thumbnails/25.jpg)
2nd
moment(s)Tuytelaars,
‘00
2nd
moment(s)Mikolajczyk& Schmid’01, ‘02
NoNoYesYes2nd
moment(s)Harris corner
othersothers
Matas, ‘02
Kadir & Brady, 01
SIFT, PCA-SIFT
Lowe ’99 (DoG)
AffineScaleRotationIntensityDescriptorDetector
![Page 26: Feature detectors and descriptors · 2008-10-05 · 2nd moment(s) Tuytelaars, ‘00 2nd moment(s) Mikolajczyk & Schmid ’01, ‘02 2nd moment(s) Harris corner others others Matas,](https://reader033.fdocuments.in/reader033/viewer/2022060516/5f8f9138297f2177275f0b62/html5/thumbnails/26.jpg)
2nd
moment(s)Tuytelaars,
‘00
2nd
moment(s)Mikolajczyk& Schmid’01, ‘02
NoNoYesYes2nd
moment(s)Harris corner
othersothers
Matas, ‘02
Kadir & Brady, 01
SIFT, PCA-SIFT
Lowe ’99 (DoG)
AffineScaleRotationIntensityDescriptorDetector
![Page 27: Feature detectors and descriptors · 2008-10-05 · 2nd moment(s) Tuytelaars, ‘00 2nd moment(s) Mikolajczyk & Schmid ’01, ‘02 2nd moment(s) Harris corner others others Matas,](https://reader033.fdocuments.in/reader033/viewer/2022060516/5f8f9138297f2177275f0b62/html5/thumbnails/27.jpg)
Interest point detectorsHarris-Laplace [Mikolajczyk & Schmid ’01]• Adds scale invariance to Harris points
– Set si = λsd
– Detect at several scales by varying sd
– Only take local maxima (8-neighbourhood) of scale adapted Harris points
– Further restrict to scales at which Laplacian is local maximum
![Page 28: Feature detectors and descriptors · 2008-10-05 · 2nd moment(s) Tuytelaars, ‘00 2nd moment(s) Mikolajczyk & Schmid ’01, ‘02 2nd moment(s) Harris corner others others Matas,](https://reader033.fdocuments.in/reader033/viewer/2022060516/5f8f9138297f2177275f0b62/html5/thumbnails/28.jpg)
• Selected scale determines size of support region• Laplacian justified experimentally
– compared to gradient squared & DoG– [Lindeberg ’98] gives thorough analysis of scale-space
Interest point detectorsHarris-Laplace [Mikolajczyk & Schmid ’01]
![Page 29: Feature detectors and descriptors · 2008-10-05 · 2nd moment(s) Tuytelaars, ‘00 2nd moment(s) Mikolajczyk & Schmid ’01, ‘02 2nd moment(s) Harris corner others others Matas,](https://reader033.fdocuments.in/reader033/viewer/2022060516/5f8f9138297f2177275f0b62/html5/thumbnails/29.jpg)
Interest point detectorsHarris-Affine [Mikolajczyk & Schmid ’02]• Adds invariance to affine image transformations• Initial locations and isotropic scale found by
Harris-Laplace• Affine invariant neighbourhood evolved iteratively
using the 2nd moment matrix μ:
))),())(,((()(),,( TDDIDI LLg Σ∇Σ∇⊗Σ=ΣΣ xxxμ
)2
exp(2
1)(1xx −Σ
−Σ
=ΣT
gπ
)()(),( xx IgL ⊗Σ=Σ
![Page 30: Feature detectors and descriptors · 2008-10-05 · 2nd moment(s) Tuytelaars, ‘00 2nd moment(s) Mikolajczyk & Schmid ’01, ‘02 2nd moment(s) Harris corner others others Matas,](https://reader033.fdocuments.in/reader033/viewer/2022060516/5f8f9138297f2177275f0b62/html5/thumbnails/30.jpg)
Interest point detectorsHarris-Affine [Mikolajczyk & Schmid ’02]
RL Axx =
LLDLIL M=ΣΣ ),,( ,,xμ 1,
−=Σ LLI tM
For affinely related points:
1,
−=Σ LLD dM
RRDRIR M=ΣΣ ),,( ,,xμ 1,
−=Σ RRI tM 1,
−=Σ RRD dM
If
Then by normalising:
LLL M xx 2/1−→′ RRR M xx 2/1−→′
and
andWe get:
RL Rxx ′→′so the normalised regions are related by a pure rotation
See also [Lindeberg & Garding ’97] and [Baumberg ’00]
![Page 31: Feature detectors and descriptors · 2008-10-05 · 2nd moment(s) Tuytelaars, ‘00 2nd moment(s) Mikolajczyk & Schmid ’01, ‘02 2nd moment(s) Harris corner others others Matas,](https://reader033.fdocuments.in/reader033/viewer/2022060516/5f8f9138297f2177275f0b62/html5/thumbnails/31.jpg)
Interest point detectorsHarris-Affine [Mikolajczyk & Schmid ’02]• Algorithm iteratively adapts
– shape of support region– spatial location x(k)
– integration scale σI (based on Laplacian)– derivation scale σD = sσI
![Page 32: Feature detectors and descriptors · 2008-10-05 · 2nd moment(s) Tuytelaars, ‘00 2nd moment(s) Mikolajczyk & Schmid ’01, ‘02 2nd moment(s) Harris corner others others Matas,](https://reader033.fdocuments.in/reader033/viewer/2022060516/5f8f9138297f2177275f0b62/html5/thumbnails/32.jpg)
2nd
moment(s)Tuytelaars,
‘00
YesYesYesYes2nd
moment(s)Mikolajczyk& Schmid’01, ‘02
NoNoYesYes2nd
moment(s)Harris corner
othersothers
Matas, ‘02
Kadir & Brady, 01
SIFT, PCA-SIFT
Lowe ’99 (DoG)
AffineScaleRotationIntensityDescriptorDetector
![Page 33: Feature detectors and descriptors · 2008-10-05 · 2nd moment(s) Tuytelaars, ‘00 2nd moment(s) Mikolajczyk & Schmid ’01, ‘02 2nd moment(s) Harris corner others others Matas,](https://reader033.fdocuments.in/reader033/viewer/2022060516/5f8f9138297f2177275f0b62/html5/thumbnails/33.jpg)
2nd
moment(s)Tuytelaars,
‘00
YesYesYesYes2nd
moment(s)Mikolajczyk& Schmid’01, ‘02
NoNoYesYes2nd
moment(s)Harris corner
othersothers
Matas, ‘02
Kadir & Brady, 01
SIFT, PCA-SIFT
Lowe ’99 (DoG)
AffineScaleRotationIntensityDescriptorDetector
![Page 34: Feature detectors and descriptors · 2008-10-05 · 2nd moment(s) Tuytelaars, ‘00 2nd moment(s) Mikolajczyk & Schmid ’01, ‘02 2nd moment(s) Harris corner others others Matas,](https://reader033.fdocuments.in/reader033/viewer/2022060516/5f8f9138297f2177275f0b62/html5/thumbnails/34.jpg)
Affine Invariant Detection
• Take a local intensity extremum as initial point• Go along every ray starting from this point and stop
when extremum of function f is reached
T.Tuytelaars, L.V.Gool. “Wide Baseline Stereo Matching Based on Local, Affinely Invariant Regions”. BMVC 2000.
0
10
( )( )
( )t
ot
I t If t
I t I dt
−=
−∫
f
points along the ray
• We will obtain approximately corresponding regions
![Page 35: Feature detectors and descriptors · 2008-10-05 · 2nd moment(s) Tuytelaars, ‘00 2nd moment(s) Mikolajczyk & Schmid ’01, ‘02 2nd moment(s) Harris corner others others Matas,](https://reader033.fdocuments.in/reader033/viewer/2022060516/5f8f9138297f2177275f0b62/html5/thumbnails/35.jpg)
Affine Invariant Detection
• The regions found may not exactly correspond, so we approximate them with ellipses
• Geometric Moments:
2
( , )p qpqm x y f x y dxdy= ∫
¡
Fact: moments mpq uniquely determine the function f
Taking f to be the characteristic function of a region (1 inside, 0 outside), moments of orders up to 2 allow to approximate the region by an ellipse
This ellipse will have the same moments of orders up to 2 as the original region
![Page 36: Feature detectors and descriptors · 2008-10-05 · 2nd moment(s) Tuytelaars, ‘00 2nd moment(s) Mikolajczyk & Schmid ’01, ‘02 2nd moment(s) Harris corner others others Matas,](https://reader033.fdocuments.in/reader033/viewer/2022060516/5f8f9138297f2177275f0b62/html5/thumbnails/36.jpg)
Affine Invariant Detection
q Ap=
2 1TA AΣ = Σ
12 1Tq q−Σ =
2 region 2
TqqΣ =
• Covariance matrix of region points defines an ellipse:
11 1Tp p−Σ =
1 region 1
TppΣ =
( p = [x, y]T is relative to the center of mass)
Ellipses, computed for corresponding regions, also correspond!
![Page 37: Feature detectors and descriptors · 2008-10-05 · 2nd moment(s) Tuytelaars, ‘00 2nd moment(s) Mikolajczyk & Schmid ’01, ‘02 2nd moment(s) Harris corner others others Matas,](https://reader033.fdocuments.in/reader033/viewer/2022060516/5f8f9138297f2177275f0b62/html5/thumbnails/37.jpg)
Affine Invariant Detection• Algorithm summary (detection of affine invariant region):
– Start from a local intensity extremum point– Go in every direction until the point of extremum of
some function f– Curve connecting the points is the region boundary– Compute geometric moments of orders up to 2 for this
region– Replace the region with ellipse
T.Tuytelaars, L.V.Gool. “Wide Baseline Stereo Matching Based on Local, Affinely Invariant Regions”. BMVC 2000.
![Page 38: Feature detectors and descriptors · 2008-10-05 · 2nd moment(s) Tuytelaars, ‘00 2nd moment(s) Mikolajczyk & Schmid ’01, ‘02 2nd moment(s) Harris corner others others Matas,](https://reader033.fdocuments.in/reader033/viewer/2022060516/5f8f9138297f2177275f0b62/html5/thumbnails/38.jpg)
YesNo (Yes ’04 ?)
YesYes2nd
moment(s)Tuytelaars,
‘00
YesYesYesYes2nd
moment(s)Mikolajczyk& Schmid’01, ‘02
NoNoYesYes2nd
moment(s)Harris corner
othersothers
Matas, ‘02
Kadir & Brady, 01
SIFT, PCA-SIFT
Lowe ’99 (DoG)
AffineScaleRotationIntensityDescriptorDetector
![Page 39: Feature detectors and descriptors · 2008-10-05 · 2nd moment(s) Tuytelaars, ‘00 2nd moment(s) Mikolajczyk & Schmid ’01, ‘02 2nd moment(s) Harris corner others others Matas,](https://reader033.fdocuments.in/reader033/viewer/2022060516/5f8f9138297f2177275f0b62/html5/thumbnails/39.jpg)
YesNo (Yes ’04 ?)
YesYes2nd
moment(s)Tuytelaars,
‘00
YesYesYesYes2nd
moment(s)Mikolajczyk& Schmid’01, ‘02
NoNoYesYes2nd
moment(s)Harris corner
othersothers
Matas, ‘02
Kadir & Brady, 01
SIFT, PCA-SIFT
Lowe ’99 (DoG)
AffineScaleRotationIntensityDescriptorDetector
![Page 40: Feature detectors and descriptors · 2008-10-05 · 2nd moment(s) Tuytelaars, ‘00 2nd moment(s) Mikolajczyk & Schmid ’01, ‘02 2nd moment(s) Harris corner others others Matas,](https://reader033.fdocuments.in/reader033/viewer/2022060516/5f8f9138297f2177275f0b62/html5/thumbnails/40.jpg)
Interest point detectorsDifference of Gaussians [Lowe ’99]
• Difference of Gaussians in scale-space– detects ‘blob’-like features
• Can be computed efficiently with image pyramid
• Approximates Laplacian for correct scale factor
• Invariant to rotation and scale changes
![Page 41: Feature detectors and descriptors · 2008-10-05 · 2nd moment(s) Tuytelaars, ‘00 2nd moment(s) Mikolajczyk & Schmid ’01, ‘02 2nd moment(s) Harris corner others others Matas,](https://reader033.fdocuments.in/reader033/viewer/2022060516/5f8f9138297f2177275f0b62/html5/thumbnails/41.jpg)
Key point localization• Detect maxima and minima of
difference-of-Gaussian in scale space (Lowe, 1999)
• Fit a quadratic to surrounding values for sub-pixel and sub-scale interpolation (Brown & Lowe, 2002)
• Taylor expansion around point:
• Offset of extremum (use finite differences for derivatives):
Blur
Res ample
Subtra ct
![Page 42: Feature detectors and descriptors · 2008-10-05 · 2nd moment(s) Tuytelaars, ‘00 2nd moment(s) Mikolajczyk & Schmid ’01, ‘02 2nd moment(s) Harris corner others others Matas,](https://reader033.fdocuments.in/reader033/viewer/2022060516/5f8f9138297f2177275f0b62/html5/thumbnails/42.jpg)
Select canonical orientation• Create histogram of local
gradient directions computed at selected scale
• Assign canonical orientation at peak of smoothed histogram
• Each key specifies stable 2D coordinates (x, y, scale, orientation) 0 2π
![Page 43: Feature detectors and descriptors · 2008-10-05 · 2nd moment(s) Tuytelaars, ‘00 2nd moment(s) Mikolajczyk & Schmid ’01, ‘02 2nd moment(s) Harris corner others others Matas,](https://reader033.fdocuments.in/reader033/viewer/2022060516/5f8f9138297f2177275f0b62/html5/thumbnails/43.jpg)
Example of keypoint detectionThreshold on value at DOG peak and on ratio of principle curvatures (Harris approach)
(a) 233x189 image(b) 832 DOG extrema(c) 729 left after peak
value threshold(d) 536 left after testing
ratio of principlecurvatures
![Page 44: Feature detectors and descriptors · 2008-10-05 · 2nd moment(s) Tuytelaars, ‘00 2nd moment(s) Mikolajczyk & Schmid ’01, ‘02 2nd moment(s) Harris corner others others Matas,](https://reader033.fdocuments.in/reader033/viewer/2022060516/5f8f9138297f2177275f0b62/html5/thumbnails/44.jpg)
Creating features stable to viewpoint change
• Edelman, Intrator & Poggio (97) showed that complex cell outputs are better for 3D recognition than simple correlation
![Page 45: Feature detectors and descriptors · 2008-10-05 · 2nd moment(s) Tuytelaars, ‘00 2nd moment(s) Mikolajczyk & Schmid ’01, ‘02 2nd moment(s) Harris corner others others Matas,](https://reader033.fdocuments.in/reader033/viewer/2022060516/5f8f9138297f2177275f0b62/html5/thumbnails/45.jpg)
SIFT vector formation• Thresholded image gradients are sampled over 16x16
array of locations in scale space• Create array of orientation histograms• 8 orientations x 4x4 histogram array = 128 dimensions
![Page 46: Feature detectors and descriptors · 2008-10-05 · 2nd moment(s) Tuytelaars, ‘00 2nd moment(s) Mikolajczyk & Schmid ’01, ‘02 2nd moment(s) Harris corner others others Matas,](https://reader033.fdocuments.in/reader033/viewer/2022060516/5f8f9138297f2177275f0b62/html5/thumbnails/46.jpg)
Feature stability to noise• Match features after random change in image scale &
orientation, with differing levels of image noise• Find nearest neighbor in database of 30,000 features
![Page 47: Feature detectors and descriptors · 2008-10-05 · 2nd moment(s) Tuytelaars, ‘00 2nd moment(s) Mikolajczyk & Schmid ’01, ‘02 2nd moment(s) Harris corner others others Matas,](https://reader033.fdocuments.in/reader033/viewer/2022060516/5f8f9138297f2177275f0b62/html5/thumbnails/47.jpg)
Feature stability to affine change• Match features after random change in image scale &
orientation, with 2% image noise, and affine distortion• Find nearest neighbor in database of 30,000 features
![Page 48: Feature detectors and descriptors · 2008-10-05 · 2nd moment(s) Tuytelaars, ‘00 2nd moment(s) Mikolajczyk & Schmid ’01, ‘02 2nd moment(s) Harris corner others others Matas,](https://reader033.fdocuments.in/reader033/viewer/2022060516/5f8f9138297f2177275f0b62/html5/thumbnails/48.jpg)
YesNo (Yes ’04 ?)
YesYes2nd
moment(s)Tuytelaars,
‘00
YesYesYesYes2nd
moment(s)Mikolajczyk& Schmid’01, ‘02
NoNoYesYes2nd
moment(s)Harris corner
othersothers
Matas, ‘02
Kadir & Brady, 01
YesYesYesYesSIFT, PCA-SIFT
Lowe ’99 (DoG)
AffineScaleRotationIntensityDescriptorDetector
![Page 49: Feature detectors and descriptors · 2008-10-05 · 2nd moment(s) Tuytelaars, ‘00 2nd moment(s) Mikolajczyk & Schmid ’01, ‘02 2nd moment(s) Harris corner others others Matas,](https://reader033.fdocuments.in/reader033/viewer/2022060516/5f8f9138297f2177275f0b62/html5/thumbnails/49.jpg)
YesNo (Yes ’04 ?)
YesYes2nd
moment(s)Tuytelaars,
‘00
YesYesYesYes2nd
moment(s)Mikolajczyk& Schmid’01, ‘02
NoNoYesYes2nd
moment(s)Harris corner
othersothers
Matas, ‘02
Kadir & Brady, 01
YesYesYesYesSIFT, PCA-SIFT
Lowe ’99 (DoG)
AffineScaleRotationIntensityDescriptorDetector
![Page 50: Feature detectors and descriptors · 2008-10-05 · 2nd moment(s) Tuytelaars, ‘00 2nd moment(s) Mikolajczyk & Schmid ’01, ‘02 2nd moment(s) Harris corner others others Matas,](https://reader033.fdocuments.in/reader033/viewer/2022060516/5f8f9138297f2177275f0b62/html5/thumbnails/50.jpg)
Other interest point detectorsScale Saliency [Kadir & Brady ’01, ’03]
![Page 51: Feature detectors and descriptors · 2008-10-05 · 2nd moment(s) Tuytelaars, ‘00 2nd moment(s) Mikolajczyk & Schmid ’01, ‘02 2nd moment(s) Harris corner others others Matas,](https://reader033.fdocuments.in/reader033/viewer/2022060516/5f8f9138297f2177275f0b62/html5/thumbnails/51.jpg)
Other interest point detectorsScale Saliency [Kadir & Brady ’01, ’03]
• Uses entropy measure of local pdf of intensities:
• Takes local maxima in scale• Weights with ‘change’ of distribution with scale:
• To get saliency measure:
∫∈
−=Dd
D dsdpsdpsH d).,,(log),,(),( 2 xxx
∫∈ ∂
∂=
DdD dsdp
sssW d.),,(),( xx
),(),(),( xxx sWsHsY DDD ×=
![Page 52: Feature detectors and descriptors · 2008-10-05 · 2nd moment(s) Tuytelaars, ‘00 2nd moment(s) Mikolajczyk & Schmid ’01, ‘02 2nd moment(s) Harris corner others others Matas,](https://reader033.fdocuments.in/reader033/viewer/2022060516/5f8f9138297f2177275f0b62/html5/thumbnails/52.jpg)
Other interest point detectorsScale Saliency [Kadir & Brady ’01, ’03]
HD(s,x) WD(s,x)
Most salient parts detected
Just usingHD(s,x)
UsingYD(s,x) = HDWD
![Page 53: Feature detectors and descriptors · 2008-10-05 · 2nd moment(s) Tuytelaars, ‘00 2nd moment(s) Mikolajczyk & Schmid ’01, ‘02 2nd moment(s) Harris corner others others Matas,](https://reader033.fdocuments.in/reader033/viewer/2022060516/5f8f9138297f2177275f0b62/html5/thumbnails/53.jpg)
Other interest point detectorsmaximum stable extremal regions [matas et al. 02]
– Sweep threshold of intensity from black to white
– Locate regions based on stability of region with respect to change of threshold
![Page 54: Feature detectors and descriptors · 2008-10-05 · 2nd moment(s) Tuytelaars, ‘00 2nd moment(s) Mikolajczyk & Schmid ’01, ‘02 2nd moment(s) Harris corner others others Matas,](https://reader033.fdocuments.in/reader033/viewer/2022060516/5f8f9138297f2177275f0b62/html5/thumbnails/54.jpg)
YesNo (Yes ’04 ?)
YesYes2nd
moment(s)Tuytelaars,
‘00
YesYesYesYes2nd
moment(s)Mikolajczyk& Schmid’01, ‘02
NoNoYesYes2nd
moment(s)Harris corner
othersothers
?YesYesYesMatas, ’02
?YesYesYesKadir & Brady, 01
YesYesYesYesSIFT, PCA-SIFT
Lowe ’99 (DoG)
AffineScaleRotationIntensityDescriptorDetector
![Page 55: Feature detectors and descriptors · 2008-10-05 · 2nd moment(s) Tuytelaars, ‘00 2nd moment(s) Mikolajczyk & Schmid ’01, ‘02 2nd moment(s) Harris corner others others Matas,](https://reader033.fdocuments.in/reader033/viewer/2022060516/5f8f9138297f2177275f0b62/html5/thumbnails/55.jpg)
YesNo (Yes ’04 ?)
YesYes2nd
moment(s)Tuytelaars,
‘00
YesYesYesYes2nd
moment(s)Mikolajczyk& Schmid’01, ‘02
NoNoYesYes2nd
moment(s)Harris corner
othersothers
?YesYesYesMatas, ’02
?YesYesYesKadir & Brady, 01
YesYesYesYesSIFT, PCA-SIFT
Lowe ’99 (DoG)
AffineScaleRotationIntensityDescriptorDetector
![Page 56: Feature detectors and descriptors · 2008-10-05 · 2nd moment(s) Tuytelaars, ‘00 2nd moment(s) Mikolajczyk & Schmid ’01, ‘02 2nd moment(s) Harris corner others others Matas,](https://reader033.fdocuments.in/reader033/viewer/2022060516/5f8f9138297f2177275f0b62/html5/thumbnails/56.jpg)
Affine-invariant texture recognition
• Texture recognition under viewpoint changes and non-rigid transformations
• Use of affine-invariant regions– invariance to viewpoint changes– spatial selection => more compact representation, reduction of
redundancy in texton dictionary
[A sparse texture representation using affine-invariant regions, S. Lazebnik, C. Schmid and J. Ponce, CVPR 2003]
![Page 57: Feature detectors and descriptors · 2008-10-05 · 2nd moment(s) Tuytelaars, ‘00 2nd moment(s) Mikolajczyk & Schmid ’01, ‘02 2nd moment(s) Harris corner others others Matas,](https://reader033.fdocuments.in/reader033/viewer/2022060516/5f8f9138297f2177275f0b62/html5/thumbnails/57.jpg)
Overview of the approach
![Page 58: Feature detectors and descriptors · 2008-10-05 · 2nd moment(s) Tuytelaars, ‘00 2nd moment(s) Mikolajczyk & Schmid ’01, ‘02 2nd moment(s) Harris corner others others Matas,](https://reader033.fdocuments.in/reader033/viewer/2022060516/5f8f9138297f2177275f0b62/html5/thumbnails/58.jpg)
Harris detector
Laplace detector
Region extraction
![Page 59: Feature detectors and descriptors · 2008-10-05 · 2nd moment(s) Tuytelaars, ‘00 2nd moment(s) Mikolajczyk & Schmid ’01, ‘02 2nd moment(s) Harris corner others others Matas,](https://reader033.fdocuments.in/reader033/viewer/2022060516/5f8f9138297f2177275f0b62/html5/thumbnails/59.jpg)
Descriptors – Spin images
![Page 60: Feature detectors and descriptors · 2008-10-05 · 2nd moment(s) Tuytelaars, ‘00 2nd moment(s) Mikolajczyk & Schmid ’01, ‘02 2nd moment(s) Harris corner others others Matas,](https://reader033.fdocuments.in/reader033/viewer/2022060516/5f8f9138297f2177275f0b62/html5/thumbnails/60.jpg)
Spatial selection
clustering each pixel clustering selected pixels
![Page 61: Feature detectors and descriptors · 2008-10-05 · 2nd moment(s) Tuytelaars, ‘00 2nd moment(s) Mikolajczyk & Schmid ’01, ‘02 2nd moment(s) Harris corner others others Matas,](https://reader033.fdocuments.in/reader033/viewer/2022060516/5f8f9138297f2177275f0b62/html5/thumbnails/61.jpg)
Signature and EMD
• Hierarchical clustering=> Signature :
• Earth movers distance
– robust distance, optimizes the flow between distributions
– can match signatures of different size– not sensitive to the number of clusters
SS = { ( m1 , w1 ) , … , ( mk , wk ) }
D( SS , SS’’ ) = [∑i,j fij d( mi , m’j)] / [∑i,j fij ]
![Page 62: Feature detectors and descriptors · 2008-10-05 · 2nd moment(s) Tuytelaars, ‘00 2nd moment(s) Mikolajczyk & Schmid ’01, ‘02 2nd moment(s) Harris corner others others Matas,](https://reader033.fdocuments.in/reader033/viewer/2022060516/5f8f9138297f2177275f0b62/html5/thumbnails/62.jpg)
Database with viewpoint changes
20 samples of 10 different textures
![Page 63: Feature detectors and descriptors · 2008-10-05 · 2nd moment(s) Tuytelaars, ‘00 2nd moment(s) Mikolajczyk & Schmid ’01, ‘02 2nd moment(s) Harris corner others others Matas,](https://reader033.fdocuments.in/reader033/viewer/2022060516/5f8f9138297f2177275f0b62/html5/thumbnails/63.jpg)
Results
Spin images Gabor-like filters
![Page 64: Feature detectors and descriptors · 2008-10-05 · 2nd moment(s) Tuytelaars, ‘00 2nd moment(s) Mikolajczyk & Schmid ’01, ‘02 2nd moment(s) Harris corner others others Matas,](https://reader033.fdocuments.in/reader033/viewer/2022060516/5f8f9138297f2177275f0b62/html5/thumbnails/64.jpg)
Feature detectorsand descriptors
![Page 65: Feature detectors and descriptors · 2008-10-05 · 2nd moment(s) Tuytelaars, ‘00 2nd moment(s) Mikolajczyk & Schmid ’01, ‘02 2nd moment(s) Harris corner others others Matas,](https://reader033.fdocuments.in/reader033/viewer/2022060516/5f8f9138297f2177275f0b62/html5/thumbnails/65.jpg)
Widely used descriptors
• SIFT• Gray-scale intensity values• Steerable filters• GLOH• Shape context & geometric blur
![Page 66: Feature detectors and descriptors · 2008-10-05 · 2nd moment(s) Tuytelaars, ‘00 2nd moment(s) Mikolajczyk & Schmid ’01, ‘02 2nd moment(s) Harris corner others others Matas,](https://reader033.fdocuments.in/reader033/viewer/2022060516/5f8f9138297f2177275f0b62/html5/thumbnails/66.jpg)
SIFT
![Page 67: Feature detectors and descriptors · 2008-10-05 · 2nd moment(s) Tuytelaars, ‘00 2nd moment(s) Mikolajczyk & Schmid ’01, ‘02 2nd moment(s) Harris corner others others Matas,](https://reader033.fdocuments.in/reader033/viewer/2022060516/5f8f9138297f2177275f0b62/html5/thumbnails/67.jpg)
Gray-scale intensity
11x11 patchNormalize
Projection ontoPCA basis
c1
c2
c15
……
…..
![Page 68: Feature detectors and descriptors · 2008-10-05 · 2nd moment(s) Tuytelaars, ‘00 2nd moment(s) Mikolajczyk & Schmid ’01, ‘02 2nd moment(s) Harris corner others others Matas,](https://reader033.fdocuments.in/reader033/viewer/2022060516/5f8f9138297f2177275f0b62/html5/thumbnails/68.jpg)
Steerable filters
![Page 69: Feature detectors and descriptors · 2008-10-05 · 2nd moment(s) Tuytelaars, ‘00 2nd moment(s) Mikolajczyk & Schmid ’01, ‘02 2nd moment(s) Harris corner others others Matas,](https://reader033.fdocuments.in/reader033/viewer/2022060516/5f8f9138297f2177275f0b62/html5/thumbnails/69.jpg)
Steerable filters
![Page 70: Feature detectors and descriptors · 2008-10-05 · 2nd moment(s) Tuytelaars, ‘00 2nd moment(s) Mikolajczyk & Schmid ’01, ‘02 2nd moment(s) Harris corner others others Matas,](https://reader033.fdocuments.in/reader033/viewer/2022060516/5f8f9138297f2177275f0b62/html5/thumbnails/70.jpg)
GLOH
![Page 71: Feature detectors and descriptors · 2008-10-05 · 2nd moment(s) Tuytelaars, ‘00 2nd moment(s) Mikolajczyk & Schmid ’01, ‘02 2nd moment(s) Harris corner others others Matas,](https://reader033.fdocuments.in/reader033/viewer/2022060516/5f8f9138297f2177275f0b62/html5/thumbnails/71.jpg)
Shape context
Belongie et al. 2002
![Page 72: Feature detectors and descriptors · 2008-10-05 · 2nd moment(s) Tuytelaars, ‘00 2nd moment(s) Mikolajczyk & Schmid ’01, ‘02 2nd moment(s) Harris corner others others Matas,](https://reader033.fdocuments.in/reader033/viewer/2022060516/5f8f9138297f2177275f0b62/html5/thumbnails/72.jpg)
Geometric blur
Berg et al. 2001
![Page 73: Feature detectors and descriptors · 2008-10-05 · 2nd moment(s) Tuytelaars, ‘00 2nd moment(s) Mikolajczyk & Schmid ’01, ‘02 2nd moment(s) Harris corner others others Matas,](https://reader033.fdocuments.in/reader033/viewer/2022060516/5f8f9138297f2177275f0b62/html5/thumbnails/73.jpg)
Geometric blur