Top-points in Image Matching
description
Transcript of Top-points in Image Matching
![Page 1: Top-points in Image Matching](https://reader036.fdocuments.in/reader036/viewer/2022062305/5681516e550346895dbf9f7d/html5/thumbnails/1.jpg)
Top-points in Image Matching
Bram PlatelEvgenya Balmashnova
Luc FlorackBart ter Haar Romeny
![Page 2: Top-points in Image Matching](https://reader036.fdocuments.in/reader036/viewer/2022062305/5681516e550346895dbf9f7d/html5/thumbnails/2.jpg)
Introduction• Question: can top-points be used
for object-retrieval tasks?
![Page 3: Top-points in Image Matching](https://reader036.fdocuments.in/reader036/viewer/2022062305/5681516e550346895dbf9f7d/html5/thumbnails/3.jpg)
Introduction to Scale Space and Deep Structure
![Page 4: Top-points in Image Matching](https://reader036.fdocuments.in/reader036/viewer/2022062305/5681516e550346895dbf9f7d/html5/thumbnails/4.jpg)
Importance of Scalein Image Analysis
Painting by Dali
• Objects in images exist at different ranges of scale.
• Usually it is not known a priory at what scale to look.
![Page 5: Top-points in Image Matching](https://reader036.fdocuments.in/reader036/viewer/2022062305/5681516e550346895dbf9f7d/html5/thumbnails/5.jpg)
• At the original scale of a dithered image we cannot calculate a derivative.
• We need to observe the image at a certain scale.
BLUR
![Page 6: Top-points in Image Matching](https://reader036.fdocuments.in/reader036/viewer/2022062305/5681516e550346895dbf9f7d/html5/thumbnails/6.jpg)
Solution?• Look at all scales
simultaneously Scale
x
y
Scale Space
![Page 7: Top-points in Image Matching](https://reader036.fdocuments.in/reader036/viewer/2022062305/5681516e550346895dbf9f7d/html5/thumbnails/7.jpg)
Scale Space in Human Vision
• The human visual system is a multi-scale sampling device
• The retina contains receptive fields; groups of receptors assembled in such a way that they form a set of apertures of widely varying size.
![Page 8: Top-points in Image Matching](https://reader036.fdocuments.in/reader036/viewer/2022062305/5681516e550346895dbf9f7d/html5/thumbnails/8.jpg)
Practical Implementation
• Convolve the image with a Gaussian Kernel
2
221
2/22
)2(1),(
D
D
xx
exG
![Page 9: Top-points in Image Matching](https://reader036.fdocuments.in/reader036/viewer/2022062305/5681516e550346895dbf9f7d/html5/thumbnails/9.jpg)
We can Calculate Derivatives and Combinations of them at all Scales
22yx LL
yyxx LL
Gradient Magnitude
Laplacian
Original Image
![Page 10: Top-points in Image Matching](https://reader036.fdocuments.in/reader036/viewer/2022062305/5681516e550346895dbf9f7d/html5/thumbnails/10.jpg)
Main Topic
• In this presentation we will show how we can exploit the deep structure of images to define invariant interest points and features which can be used for matching problems in computer vision.
• We consider only grey-value images.
![Page 11: Top-points in Image Matching](https://reader036.fdocuments.in/reader036/viewer/2022062305/5681516e550346895dbf9f7d/html5/thumbnails/11.jpg)
Interest Points
• The locations of particularly characteristic points are called the interest points or key points.
• These interest points have to be as invariant as possible, but at the same time they have to carry a lot of distinctive information.
![Page 12: Top-points in Image Matching](https://reader036.fdocuments.in/reader036/viewer/2022062305/5681516e550346895dbf9f7d/html5/thumbnails/12.jpg)
Why Interest Points in Scale Space?• Information in interest points is
defined by their neighborhood. But how big should we choose this neighborhood?
• Let’s take the corners of the mouth as interest points.
• The red circles are the areas in which the information is gathered.
• If we make the picture bigger, the size of the neighborhood is too small.
• The neighborhood should scale with the image
![Page 13: Top-points in Image Matching](https://reader036.fdocuments.in/reader036/viewer/2022062305/5681516e550346895dbf9f7d/html5/thumbnails/13.jpg)
Why Interest Points in Scale Space?• When the interest points are
detected in scale space they do not only have spatial coordinates x and y, but also a scale .
• This scale tells us how big the neighborhood should be.
![Page 14: Top-points in Image Matching](https://reader036.fdocuments.in/reader036/viewer/2022062305/5681516e550346895dbf9f7d/html5/thumbnails/14.jpg)
Which Interest Points to Use?
• Our interest points have to be detected in scale space.
• They also have to…– …contain a lot of information– …be reproducible– …be stable– …be well understood
![Page 15: Top-points in Image Matching](https://reader036.fdocuments.in/reader036/viewer/2022062305/5681516e550346895dbf9f7d/html5/thumbnails/15.jpg)
We Suggest Top-Points
• The points we introduce have these desired properties.
![Page 16: Top-points in Image Matching](https://reader036.fdocuments.in/reader036/viewer/2022062305/5681516e550346895dbf9f7d/html5/thumbnails/16.jpg)
Critical Points, Paths and Top-Points
Maxima
Minimum
SaddlesL=0Critical Points
![Page 17: Top-points in Image Matching](https://reader036.fdocuments.in/reader036/viewer/2022062305/5681516e550346895dbf9f7d/html5/thumbnails/17.jpg)
Critical Points, Paths and Top-Points
Maxima
Minimum
SaddlesL=0Critical Points Det(H)=0
Top-Points
![Page 18: Top-points in Image Matching](https://reader036.fdocuments.in/reader036/viewer/2022062305/5681516e550346895dbf9f7d/html5/thumbnails/18.jpg)
Possible to calculate them for every Function of the Image L(x,y,)
Original Gradient Magnitude
Laplacian Det(H)
![Page 19: Top-points in Image Matching](https://reader036.fdocuments.in/reader036/viewer/2022062305/5681516e550346895dbf9f7d/html5/thumbnails/19.jpg)
Detecting Critical Paths
• Since for a critical path L=0
• Intersection of Level Surfaces Lx=0 with Ly=0
• Will give the critical paths.
![Page 20: Top-points in Image Matching](https://reader036.fdocuments.in/reader036/viewer/2022062305/5681516e550346895dbf9f7d/html5/thumbnails/20.jpg)
Detecting Top-Points
• Since for a top-point both L=0 and Det[H]=Lxx Lyy-Lxy2=0
• We can find them by intersecting the paths with the level surface Det[H]=0
![Page 21: Top-points in Image Matching](https://reader036.fdocuments.in/reader036/viewer/2022062305/5681516e550346895dbf9f7d/html5/thumbnails/21.jpg)
![Page 22: Top-points in Image Matching](https://reader036.fdocuments.in/reader036/viewer/2022062305/5681516e550346895dbf9f7d/html5/thumbnails/22.jpg)
Invariance of Top Points
• Top-points are invariant to certain transformations.
• By invariant we mean that they move according to the transformation.
Allo
wed
Tra
ns.
![Page 23: Top-points in Image Matching](https://reader036.fdocuments.in/reader036/viewer/2022062305/5681516e550346895dbf9f7d/html5/thumbnails/23.jpg)
Reconstruction
• It is possible to make a reconstruction of the original image from its top-points.
• We can generate reconstructed images which give the same (plus more) top-points as the original image.
• This reconstruction resembles the original image.
![Page 24: Top-points in Image Matching](https://reader036.fdocuments.in/reader036/viewer/2022062305/5681516e550346895dbf9f7d/html5/thumbnails/24.jpg)
Original Image
Top-Points and Features
Reconstruction
![Page 25: Top-points in Image Matching](https://reader036.fdocuments.in/reader036/viewer/2022062305/5681516e550346895dbf9f7d/html5/thumbnails/25.jpg)
Metameric Class
OriginalBy adjusting boundary and smoothness constraints we can improve the visual performance.
For this 300x300 picture 1000 top-points with 6 features were used.
![Page 26: Top-points in Image Matching](https://reader036.fdocuments.in/reader036/viewer/2022062305/5681516e550346895dbf9f7d/html5/thumbnails/26.jpg)
• This tells us that the top-points indeed contain a lot of information about the image.
![Page 27: Top-points in Image Matching](https://reader036.fdocuments.in/reader036/viewer/2022062305/5681516e550346895dbf9f7d/html5/thumbnails/27.jpg)
Localization of Top-Points
• For points close to top-points it is possible to calculate a vector pointing towards the position of the top-point.
x
y
Approximated Top-Points
Displacement Vectors
Real Locations
![Page 28: Top-points in Image Matching](https://reader036.fdocuments.in/reader036/viewer/2022062305/5681516e550346895dbf9f7d/html5/thumbnails/28.jpg)
Localization of Top-Points
• For points close to top-points it is possible to calculate a vector pointing towards the position of the top-point.
• This enables us to use fast top-point detection algorithms which do not have to be very accurate.
![Page 29: Top-points in Image Matching](https://reader036.fdocuments.in/reader036/viewer/2022062305/5681516e550346895dbf9f7d/html5/thumbnails/29.jpg)
Stability of Top-Points
• The locations of top-points change when noise is added to the image.
![Page 30: Top-points in Image Matching](https://reader036.fdocuments.in/reader036/viewer/2022062305/5681516e550346895dbf9f7d/html5/thumbnails/30.jpg)
![Page 31: Top-points in Image Matching](https://reader036.fdocuments.in/reader036/viewer/2022062305/5681516e550346895dbf9f7d/html5/thumbnails/31.jpg)
Stability of Top-Points
• We can calculate the variance of the displacement of top- points under noise.
• We need 4th order derivatives in the top-points for that.
![Page 32: Top-points in Image Matching](https://reader036.fdocuments.in/reader036/viewer/2022062305/5681516e550346895dbf9f7d/html5/thumbnails/32.jpg)
Selecting Stable Paths
• Large instabilities are found in flat areas in the image. Areas with a lot of structure result in more stable top points.
• In the article we used an adapted version of the TV-norm (Total Variation) to measure the flatness around a top point.
• We can give top points a weight in the EMD algorithm based on the stability of the point.
![Page 33: Top-points in Image Matching](https://reader036.fdocuments.in/reader036/viewer/2022062305/5681516e550346895dbf9f7d/html5/thumbnails/33.jpg)
Calculating the Stability Norm
dV
dVxuTV
22def )()(
)h.o.t.()(Tr4
)( 244 HTV
Taylor Expansion (2nd order)
)(Tr 24 Htv
Limiting Procedure
Koenderink’s deviation from flatness
222: cxx
xc
x
Integration Area
![Page 34: Top-points in Image Matching](https://reader036.fdocuments.in/reader036/viewer/2022062305/5681516e550346895dbf9f7d/html5/thumbnails/34.jpg)
Thresholding on Stability
Stable Paths Unstable Paths
![Page 35: Top-points in Image Matching](https://reader036.fdocuments.in/reader036/viewer/2022062305/5681516e550346895dbf9f7d/html5/thumbnails/35.jpg)
1 2 3 4 5 87 96
Database Image Retrieval
Data
ba
seQuery Image
![Page 36: Top-points in Image Matching](https://reader036.fdocuments.in/reader036/viewer/2022062305/5681516e550346895dbf9f7d/html5/thumbnails/36.jpg)
Experiments• A simple image retrieval task.• Using a small version of the Olivetti
Faces Database.• Consisting of 200 images of 20
different people (10 p.p.)
![Page 37: Top-points in Image Matching](https://reader036.fdocuments.in/reader036/viewer/2022062305/5681516e550346895dbf9f7d/html5/thumbnails/37.jpg)
The Deep Structure of an Image
Look at all scales simultaneously
Scale
x
y
![Page 38: Top-points in Image Matching](https://reader036.fdocuments.in/reader036/viewer/2022062305/5681516e550346895dbf9f7d/html5/thumbnails/38.jpg)
Critical Points, Paths and Top Points
Maxima
Minimum
Saddlesu=0Critical Points Det(H)=0
Top Points
![Page 39: Top-points in Image Matching](https://reader036.fdocuments.in/reader036/viewer/2022062305/5681516e550346895dbf9f7d/html5/thumbnails/39.jpg)
![Page 40: Top-points in Image Matching](https://reader036.fdocuments.in/reader036/viewer/2022062305/5681516e550346895dbf9f7d/html5/thumbnails/40.jpg)
Comparing Top Points of Images
CompareEMD
![Page 41: Top-points in Image Matching](https://reader036.fdocuments.in/reader036/viewer/2022062305/5681516e550346895dbf9f7d/html5/thumbnails/41.jpg)
Earth Movers Distance (EMD)
wi fij cij
A Buj
miwf
njuf
njmif
i
n
j ij
j
m
i ij
ij
,,1
,,1
,,1,,1 ,0
:sConstraint
1
1
m
i
n
j ijij fc1 1:Minimize
n
j
m
i ij uw1 1
:DemandSupply
n
j j
m
i
n
j ijij
u
fc
1
1 1u)EMD(w,
[*]Rubner, Tomasi, Guibas, 1998, IEEE Conf. on Computer Vision
Piles Holes
![Page 42: Top-points in Image Matching](https://reader036.fdocuments.in/reader036/viewer/2022062305/5681516e550346895dbf9f7d/html5/thumbnails/42.jpg)
Calculating the EMD
• To calculate the earth movers distance we need:– Weights for our top points (e.g. more
important points could contain more weight).
– A cost function for transporting the weights between point sets (we incorporated a distance function between top points).
MMA
![Page 43: Top-points in Image Matching](https://reader036.fdocuments.in/reader036/viewer/2022062305/5681516e550346895dbf9f7d/html5/thumbnails/43.jpg)
Measuring Distance in Scale Space• Scale Space is not a
Euclidean Space.• The simple Euclidean
Metric no longer holds.Scale
x
y
2222 yxs
![Page 44: Top-points in Image Matching](https://reader036.fdocuments.in/reader036/viewer/2022062305/5681516e550346895dbf9f7d/html5/thumbnails/44.jpg)
Eberly’s Distance Measure in Scale SpaceFor an infinitely small step in scale space the following distance function holds:
Given this distance function we can find a unique geodesic path connecting two points in scale space.
The distance between two points is measured along the this geodesic path connecting these points.
2
2
2
2
2
22 1
ddydx
ds 0 relates the importance of scale- to spatial measurements
![Page 45: Top-points in Image Matching](https://reader036.fdocuments.in/reader036/viewer/2022062305/5681516e550346895dbf9f7d/html5/thumbnails/45.jpg)
Distance Between Points in Scale Space• By parameterizing along
the geodesic path the distance function can be derived by solving an ordinary differential equation.
• This yields the following equation describing the distance between two points in scale space:
221
221
2221
22
2
12
)()(
1,2i, 2
)1(
1,2i, )1ln(
yyxxR
R
Rb
bbr
rrS
i
i
i
iii
![Page 46: Top-points in Image Matching](https://reader036.fdocuments.in/reader036/viewer/2022062305/5681516e550346895dbf9f7d/html5/thumbnails/46.jpg)
• We now have a distance measure which we can use in our EMD algorithm.
• We have introduced a tunable parameter (>0).
• We now need to give a weight to our points.
![Page 47: Top-points in Image Matching](https://reader036.fdocuments.in/reader036/viewer/2022062305/5681516e550346895dbf9f7d/html5/thumbnails/47.jpg)
Results1 2 3 4 5 6 7 8 9
a 93% 78% 68% 62% 58% 54% 49% 45% 43%
b 93% 82% 90% 73% 70% 68% 63% 59% 56%
c 95% 88% 83% 76% 73% 66% 62% 59% 58%
d 100%
97% 96% 95% 92% 88% 86% 84% 81%
a. Using Euclidean Distance
b. Using Eberly Distance
c. As b. including stability norm
d. As c. including 2nd order derivatives.
![Page 48: Top-points in Image Matching](https://reader036.fdocuments.in/reader036/viewer/2022062305/5681516e550346895dbf9f7d/html5/thumbnails/48.jpg)
Distinctive Features
• To distinguish top-points from each other a set of distinctive features are needed in every top-point.
• These local features describe the neighborhood of the top-point.
![Page 49: Top-points in Image Matching](https://reader036.fdocuments.in/reader036/viewer/2022062305/5681516e550346895dbf9f7d/html5/thumbnails/49.jpg)
Differential Invariants
• We use the complete set of irreducible 3rd order differential invariants.
• These features are rotation and scaling invariant.
![Page 50: Top-points in Image Matching](https://reader036.fdocuments.in/reader036/viewer/2022062305/5681516e550346895dbf9f7d/html5/thumbnails/50.jpg)
Matching
• With the top-points as interest points and the differential invariants as the descriptors we can now start matching them across images.
![Page 51: Top-points in Image Matching](https://reader036.fdocuments.in/reader036/viewer/2022062305/5681516e550346895dbf9f7d/html5/thumbnails/51.jpg)
The Task
• We have a scene and from that scene we want to retrieve the location of the query object.
![Page 52: Top-points in Image Matching](https://reader036.fdocuments.in/reader036/viewer/2022062305/5681516e550346895dbf9f7d/html5/thumbnails/52.jpg)
• The first step of the matching algorithm is to find the location of our highly invariant interest points, the top-points.
![Page 53: Top-points in Image Matching](https://reader036.fdocuments.in/reader036/viewer/2022062305/5681516e550346895dbf9f7d/html5/thumbnails/53.jpg)
• The top-points and differential invariants are calculated for the query object and the scene.
![Page 54: Top-points in Image Matching](https://reader036.fdocuments.in/reader036/viewer/2022062305/5681516e550346895dbf9f7d/html5/thumbnails/54.jpg)
• The location in scale-space (x,y,), the gradient angle and the differential invariant features (D1,D2,D3,D4,D5,D6) are stored for every top-point.
![Page 55: Top-points in Image Matching](https://reader036.fdocuments.in/reader036/viewer/2022062305/5681516e550346895dbf9f7d/html5/thumbnails/55.jpg)
Distance between Feature Vectors• A sensible distance between feature
vectors is essential.• We have used Euclidean distance on
‘normalized’ differential invariants.
• We tried Mahalanobis distance obtained from a training set.
![Page 56: Top-points in Image Matching](https://reader036.fdocuments.in/reader036/viewer/2022062305/5681516e550346895dbf9f7d/html5/thumbnails/56.jpg)
Drawbacks of these Methods
• Euclidean distance is not optimal since the normalization of the features still does not make them Euclidean.
• The covariance matrix C of the Mahalanobis distance is global and not defined for each feature vector and training is required.
![Page 57: Top-points in Image Matching](https://reader036.fdocuments.in/reader036/viewer/2022062305/5681516e550346895dbf9f7d/html5/thumbnails/57.jpg)
Similarity Measure
• We can calculate the propagation of noise in scale space*
• This enables us to calculate a covariance matrix for each feature vector.
• The dissimilarity (“distance”) measure is expressed as:
*Topological and Geometrical Aspects of Image Structure, Johan Blom
![Page 58: Top-points in Image Matching](https://reader036.fdocuments.in/reader036/viewer/2022062305/5681516e550346895dbf9f7d/html5/thumbnails/58.jpg)
Improvement
• Improvement e.g. for 5% noise addition.
Tru
e P
osi
tive R
ate
False Positive Rate
![Page 59: Top-points in Image Matching](https://reader036.fdocuments.in/reader036/viewer/2022062305/5681516e550346895dbf9f7d/html5/thumbnails/59.jpg)
• We now compare the differential invariant features.
compare
distance = 0.5distance = 0.2distance = 0.3
![Page 60: Top-points in Image Matching](https://reader036.fdocuments.in/reader036/viewer/2022062305/5681516e550346895dbf9f7d/html5/thumbnails/60.jpg)
The vectors with the smallest distance are paired.smallest
distancedistance = 0.2
![Page 61: Top-points in Image Matching](https://reader036.fdocuments.in/reader036/viewer/2022062305/5681516e550346895dbf9f7d/html5/thumbnails/61.jpg)
• A set of coordinates is formed from the differences in scale (Log(o1)- Log(s2)) and in angles (o1- s2).
(, )
![Page 62: Top-points in Image Matching](https://reader036.fdocuments.in/reader036/viewer/2022062305/5681516e550346895dbf9f7d/html5/thumbnails/62.jpg)
Important Clusters
For these clusters we calculate the mean and
Clustering (,)
• If these coordinates are plotted in a scatter plot clusters can be identified.
• In this scatter plot we find two dense clusters
![Page 63: Top-points in Image Matching](https://reader036.fdocuments.in/reader036/viewer/2022062305/5681516e550346895dbf9f7d/html5/thumbnails/63.jpg)
• The stability criterion removes much of the scatter
![Page 64: Top-points in Image Matching](https://reader036.fdocuments.in/reader036/viewer/2022062305/5681516e550346895dbf9f7d/html5/thumbnails/64.jpg)
• After finding and for each cluster we transform the (x,y,) parameters of the object top-points to match the scene.
![Page 65: Top-points in Image Matching](https://reader036.fdocuments.in/reader036/viewer/2022062305/5681516e550346895dbf9f7d/html5/thumbnails/65.jpg)
Rotate and scale according to the cluster means.
![Page 66: Top-points in Image Matching](https://reader036.fdocuments.in/reader036/viewer/2022062305/5681516e550346895dbf9f7d/html5/thumbnails/66.jpg)
• After this step we can find the translation of the points in scene. By again doing the clustering but now for coordinates (x,y).
![Page 67: Top-points in Image Matching](https://reader036.fdocuments.in/reader036/viewer/2022062305/5681516e550346895dbf9f7d/html5/thumbnails/67.jpg)
The translations we find correspond to the location of the objects in the scene.
![Page 68: Top-points in Image Matching](https://reader036.fdocuments.in/reader036/viewer/2022062305/5681516e550346895dbf9f7d/html5/thumbnails/68.jpg)
In this example we have two clusters of correctly matched points.
C1
C2
![Page 69: Top-points in Image Matching](https://reader036.fdocuments.in/reader036/viewer/2022062305/5681516e550346895dbf9f7d/html5/thumbnails/69.jpg)
• The transformation of each object in the scene matching to the query object is known from the clustering.
![Page 70: Top-points in Image Matching](https://reader036.fdocuments.in/reader036/viewer/2022062305/5681516e550346895dbf9f7d/html5/thumbnails/70.jpg)
We can transform the outline of the query object and project it on the scene image.
![Page 71: Top-points in Image Matching](https://reader036.fdocuments.in/reader036/viewer/2022062305/5681516e550346895dbf9f7d/html5/thumbnails/71.jpg)
Object Retrieval Results
![Page 72: Top-points in Image Matching](https://reader036.fdocuments.in/reader036/viewer/2022062305/5681516e550346895dbf9f7d/html5/thumbnails/72.jpg)
![Page 73: Top-points in Image Matching](https://reader036.fdocuments.in/reader036/viewer/2022062305/5681516e550346895dbf9f7d/html5/thumbnails/73.jpg)
Invariance to Sampling
![Page 74: Top-points in Image Matching](https://reader036.fdocuments.in/reader036/viewer/2022062305/5681516e550346895dbf9f7d/html5/thumbnails/74.jpg)
![Page 75: Top-points in Image Matching](https://reader036.fdocuments.in/reader036/viewer/2022062305/5681516e550346895dbf9f7d/html5/thumbnails/75.jpg)
Conclusion
• Top-points have proved to be invariant interest points which are useful for matching.
• The differential invariants have shown to be very distinctive.
• Experiments show good results.