Special Topic on Image Retrieval Local Feature Matching Verification.
-
Upload
beatrice-harrington -
Category
Documents
-
view
217 -
download
0
Transcript of Special Topic on Image Retrieval Local Feature Matching Verification.
Geometric Verification
• Motivation– Remove false matches by checking geometric
consistency
2
Red line: geometric consistent matchBlue line: geometric inconsistent match
Global Verification: RANSAC
• Take RANSAC as an example– Check geometric consistency from matched feature pairs.
2
1
43
21
t
t
v
u
mm
mm
y
x
Random sampling
Local Geometric-Verification
• Locally nearest neighbors ( Video Goole, cvpr’03)– Matched regions should have a similar spatial layout.– For each match define its search area– Region in the search area that also matches casts a vote
for the image– Reject matches with no support
• Drawback– Sensitive to clutter
Hamming Embedding (ECCV’08)• Introduced as an extension of BOV [Jegou 08]
– Combination of– A partitioning technique (k-means)– A binary code that refine the descriptor
• Representation of a descriptor x– Vector-quantized to q(x) as in standard BOV– short binary vector b(x) for an additional localization in the Voronoi cell
• Two descriptors x and y match iif
Hamming Embedding
• Binary signature generation– Off-line learning
• Random matrix generation• Descriptor projection and assignment• Median values of projected descriptors
– On-line binarization• Quantization assignment• Descriptor projection• Computing the signature:
Local Geometric-Verification
• Bundled feature (CVPR’09)– Group local features in local MSER region.– Increase discriminative power of visual words.– Allowed to have large overlap error.
• Bundle comparison:– Mm(q; p): number of common visual words between two bundles
– Mg(q; p): inconsistency of geometric order in x- and y- direction.
• Drawbacks: Infeasible for rotated bundles.
– Visual words are bundled in MSER regions.– Spatial consistency for bundled features is utilized to weight visual
words. ( ; ) ( ; ) ( ; )m gM q p M q p M q p
Z. Wu, J. Sun, and Q. Ke, “Bundling Features for Large Scale Partial-Duplicate Web Image Search,” CVPR 09
),( pqMvv idftf
# of shared visual words
Spatial consistency
– Great performance for partial-dup detection in over 1 M database– Drawbacks: Infeasible for rotated bundles.
Local Geometric-VerificationBundled feature (CVPR’09)
Global Verification: RANSAC RANSAC: remove outliers by inlier classification
Inliers: true matched features Outliers: false matched features
Assumption of RANdom SAmple Consensus (RANSAC) The original data consists of inliers and outliers. A subset of inliers can estimate a model to optimally explain the inliers.
Estimate the affine transformation by RANSAC
Procedure: Iteratively select a random subset as hypothetical inliers 1. A model is fitted to the hypothetical inliers.2. All other data are tested against the fitted model for inlier classification.3. The model is re-estimated from all hypothetical inliers.4. The model is evaluated by estimating the error of the inliers relative to the model.
Drawbacks: Computationally expensive, not scalableFischler, et al., RANdom SAmple Consensus: a paradigm for model fitting with applications to image analysis and automated
cartography, Comm. of the ACM, 24:381-395, 1981
2
1
43
21
t
t
v
u
mm
mm
y
x
Spatial Coding for Geometric Verification (ACM MM’ 10)
• Motivation– Encode local features’ relative positions into compact binary maps– Check spatial consistency of local matches for geometric verification
• Spatial coding maps– Relative spatial positions between local features.– Very efficient and high precision
1000
1100
1110
1111
1000
1111
1011
1001
Ymap
Xmap
Zhou & Tian, Spatial Coding for large scale partial-duplicate image search. ACM Multimedia 2010.
11
Spatial Map Generation
Rotate 45 degree counterclockwise
In previous case, each quadrant has one part Consider each quadrant is uniformly divided into two parts.
=
12
Spatial Map Generation Generalized spatial map: GX and GY
Each quadrant is uniformly divided into r parts.
… …
r
k
2
k=r-1
k=1
k=0 X-map
X-map
X-map
Y-map
Y-map
Y-map
GX GY
13
Generalized Spatial Coding
1,,2,1,0 ,2
rkr
k
i
iki
ki
y
x
y
x)cos()sin(
)sin()cos(
k
jki
kj
ki
xx
xxkjiGX
if1
if0),,(
k
jki
kj
ki
yy
yykjiGY
if1
if0),,(
Spatial coding maps: Each quadrant uniformly divided into r parts. Decompose the division into r sub-division. Rotate each sub-division to align the axis.
New feature locations after rotation :
Generalized spatial maps:
14
Spatial Verification Verification with spatial maps GX and GY
Compare the spatial maps of matched features:
k=0, …, r-1; i, j=1, …, N; N: number of matched features Find and delete the most inconsistent matched pair,
recursively:
),,(),,(),,(
),,(),,(),,(
kjiGYkjiGYkjiV
kjiGXkjiGXkjiV
mqy
mqx
1
0 1
),,()(r
k
N
jxx kjiViS
1
0 1
),,()(r
k
N
jyy kjiViS
)(max arg* iSi xi
)(max arg* jSj yj
Vx: inconsistent degree in X-map
Vy: inconsistent degree in Y-map
Identify i* and remove
15
,
01111
10000
10000
10000
10000
,
01011
10000
00000
10000
10000
x
y
2
4
3
1
2
43
1
5
5
11111
01000
01100
01110
01111
10000
11000
11100
11110
11111
10000
11000
11111
11011
11001
11011
01000
11111
01011
01001
YmYqVy
XmXqVx
4
1
1
1
1
3
1
0
1
1
Sy
Sx
,
01111
10000
10000
10000
10000
,
01011
10000
00000
10000
10000
11111
01000
01100
01110
01111
10000
11000
11100
11110
11111
10000
11000
11111
11011
11001
11011
01000
11111
01011
01001
YmYqVy
XmXqVx
4
1
1
1
1
3
1
0
1
1
Sy
Sx
Geometric Verification with Coding Maps
16
SUM
x
y 2
4
3
1
5
2
4
3
1
5
17
Image Plane Division (TOMCCAP’ 10)
2
1
3
5
4
(a)
2
1
3
5
4
(d)
2
1
3
5
4
(c)
2
1
3
5
4
(b)
2
1
3
5
4
(e)
2
1
3
5
4
(f)
18
Geometric Square Coding
• Coordinate adjustment
• Square coding map
Mjiy
x
y
x
j
j
ii
iiij
ij
,0 ,
)cos()sin(
)sin()cos()(
)(
otherwise
syy,xxjiSmap i
ii
ij
ii
ij
0
max if1),(
)()()()(
i
ii
ij
ii
ij
s
yy,xxjiGS
)max(),(
)()()()( Generalized map:
21
3
5
4
2
1
3
5
4
19
Geometric Fan Coding
• Fan coding maps
• Coordinate adjustment
• Generalized coding maps
)()(
)()(
if1
if0),( i
ii
j
ii
ij
xx
xxjiHmap
ii
ij
ii
ij
yy
yyjiVmap
if1
if0),(
j
j
ki
ki
ki
ki
kij
kij
y
x
y
x
)cos()sin(
)sin()cos()()(
)()(
),(
),(
),(),(
),(),(
if1
if0),,( ki
iki
j
kii
kij
xx
xxkjiGH
),(),(
),(),(
if1
if0),,( ki
iki
j
kii
kij
yy
yykjiGV