IIIT HYDERABAD

Towards Exhaustive Pairwise Matching In Large Image Collections

Kumar Srijan : IIIT – Hyderabad C.V. Jawahar : IIIT - Hyderabad

IIIT HYDERABAD

Community Photo Collections

• Anyone can take photographs!

• Sharing photographs is easy!

• Searching for photographs is easy!

• How to organize these collections ?

IIIT HYDERABAD

CPCs - Characteristics

• Totally unstructured collections!– Different resolutions– Different viewpoints…

• Contents unknown– Difficult to find relevant information

• Opportunity – Discover relationships between images which are

hard to find manually

IIIT HYDERABAD

Matching Images

• Establish correspondences – Ex: Matching SIFT features

• Discover transitive relationships– Discover Popular Images / Locations / Paths…– Filter irrelevant images

• Applications : – Automatic Annotation transfer – 3D reconstruction…

IIIT HYDERABAD

Representing Matching Images

• Image Match Graph – Images Nodes – Image Match Edges– Allows queries such as :• Connected components• Shortest path

– Image Match : Ascertained by verification• RANSAC based epipolar geometry estimation• Cosegmentation .. etc.• Verification is Expensive

IIIT HYDERABAD

Creating Match Graphs

• Collection of N images– Verify N x N matches : Ideally!– Impractical!

• Practical– Filter– Verify only promising candidates– Successful!

• Exhaustive Pairwise matching – A Step!

IIIT HYDERABAD

Current Techniques

• Image Retrieval– Quantization Image Features– Inverted Index Database Images– Filtering Reject lot of Matches– Verification On Top scoring matches– O(N) time for a single querying

• Min Hash – Grow seeds by Image Retrieval

• Image Webs – Compute cluster skeletons by Image Retrieval

IIIT HYDERABAD

Exhaustive Pairwise Matching - Why?

• Image matches beyond human scope!– No ground truth for large collections!

• Discover low scoring matches!

• No effort spent in Designing filter!

IIIT HYDERABAD

Exhaustive Pairwise Matching – How?• Image Retrieval• ‘ Query each image in turn ’ – Verify all potential matches• No shortlists

– Verification doable from Index retrievals• Index Geometry • More discriminative features

– IF : Constant time queries• THEN : Linear time complexity

IIIT HYDERABAD

High Order Features

• High Order Features– Combine nearby features• Primary with Secondary Features• Encode Affine Invariants

– Relative Orientation and Scale– Normalized distance– Baseline orientation

– Tuple• <VWp,VWs,g1,g2,g3,g4>

– Huge Feature Space• Helps in faster queries!

IIIT HYDERABAD

Constant Time Queries

• Regular Inverted Index – Fixed size : # Visual words• Posting lists grow with Database

• Our Case : Huge Feature Space– Hash Functions Reproject to custom size• Database size• Constant sized posting lists• Constant time queries

IIIT HYDERABAD

Bloom Filters

• Bloom Filter– Set Membership• Present• Not Present

– Bit array(m) – Hash Functions(k)– Elements(n)– False Positives• C is not present• D is a False Positive

IIIT HYDERABAD

Inverted Index over Bloom Filters• Database of N images• N Bloom Filters – Index and Query N images

• Inverted Index – Bloom filter size(m) Proportional to N• Use one Hash function (k = 1) No list Intersections!• Constant time queries

IIIT HYDERABAD

Spatial Verification

IIIT HYDERABAD

Results

• UK benchmark• 73.2 % recall

• Oxford Buildings– 78 Mn Hof– 2^25 index size– Extracting features : 27 minutes – Querying features : 2 minutes

IIIT HYDERABAD

• 317 clusters with 1375 images• Small Clusters

• Errors

IIIT HYDERABAD

Effect of doubling inverted index size

• On oxford 105k (100k + 5k Oxford Buildings)– 1480 Mn HoF

– While using 16 x 2^25 = query time = 2.5 hours– While using 32 x 2^25 = query time = 1.8 hours– 2147 clusters 7198 images– Largest 2265

IIIT HYDERABAD

Conclusions

• Indexing geometry gives a Bigger feature space

• Bloom Filters can be used for Match Graph construction

• Exhaustive Pairwise Matching is Feasible with efficient indexing