Lost in Quantization: Improving Particular Object Retrieval in Large Scale Image Databases CVPR 2008...
-
date post
19-Dec-2015 -
Category
Documents
-
view
217 -
download
2
Transcript of Lost in Quantization: Improving Particular Object Retrieval in Large Scale Image Databases CVPR 2008...
![Page 1: Lost in Quantization: Improving Particular Object Retrieval in Large Scale Image Databases CVPR 2008 James Philbin Ondˇrej Chum Michael Isard Josef Sivic.](https://reader038.fdocuments.in/reader038/viewer/2022103022/56649d2c5503460f94a023a2/html5/thumbnails/1.jpg)
Lost in Quantization: Improving Particular Object Retrieval in Large Scale Image Databases
CVPR 2008James PhilbinOndˇrej ChumMichael Isard
Josef SivicAndrew Zisserman
[7] O. Chum, J. Philbin, J. Sivic, M. Isard, and A. Zisserman. Total recall: Automatic query expansion with a generative feature model for object retrieval. In Proc. ICCV, 2007.
![Page 2: Lost in Quantization: Improving Particular Object Retrieval in Large Scale Image Databases CVPR 2008 James Philbin Ondˇrej Chum Michael Isard Josef Sivic.](https://reader038.fdocuments.in/reader038/viewer/2022103022/56649d2c5503460f94a023a2/html5/thumbnails/2.jpg)
Outline
• Introduction• Methods in this paper• Experiment & Result• Conclusion
![Page 3: Lost in Quantization: Improving Particular Object Retrieval in Large Scale Image Databases CVPR 2008 James Philbin Ondˇrej Chum Michael Isard Josef Sivic.](https://reader038.fdocuments.in/reader038/viewer/2022103022/56649d2c5503460f94a023a2/html5/thumbnails/3.jpg)
Outline
• Introduction• Methods in this paper• Experiment & Result• Conclusion
![Page 4: Lost in Quantization: Improving Particular Object Retrieval in Large Scale Image Databases CVPR 2008 James Philbin Ondˇrej Chum Michael Isard Josef Sivic.](https://reader038.fdocuments.in/reader038/viewer/2022103022/56649d2c5503460f94a023a2/html5/thumbnails/4.jpg)
Introduction
• Goal– Specific object retrieval from an image database
• For large database– It’s achieved by systems that are inspired by text retrieval
(visual words).
![Page 5: Lost in Quantization: Improving Particular Object Retrieval in Large Scale Image Databases CVPR 2008 James Philbin Ondˇrej Chum Michael Isard Josef Sivic.](https://reader038.fdocuments.in/reader038/viewer/2022103022/56649d2c5503460f94a023a2/html5/thumbnails/5.jpg)
Flow
1. Get features– SIFT
2. Cluster– Approximate k-means
3. Feature quantization– Visual word– Soft-assignment (query)
4. Re-ranked– RANSAC
5. Query expansion– Average query expansion
![Page 6: Lost in Quantization: Improving Particular Object Retrieval in Large Scale Image Databases CVPR 2008 James Philbin Ondˇrej Chum Michael Isard Josef Sivic.](https://reader038.fdocuments.in/reader038/viewer/2022103022/56649d2c5503460f94a023a2/html5/thumbnails/6.jpg)
Outline
• Introduction• Methods in this paper• Experiment & Result• Conclusion
![Page 7: Lost in Quantization: Improving Particular Object Retrieval in Large Scale Image Databases CVPR 2008 James Philbin Ondˇrej Chum Michael Isard Josef Sivic.](https://reader038.fdocuments.in/reader038/viewer/2022103022/56649d2c5503460f94a023a2/html5/thumbnails/7.jpg)
Feature
• SIFT
![Page 8: Lost in Quantization: Improving Particular Object Retrieval in Large Scale Image Databases CVPR 2008 James Philbin Ondˇrej Chum Michael Isard Josef Sivic.](https://reader038.fdocuments.in/reader038/viewer/2022103022/56649d2c5503460f94a023a2/html5/thumbnails/8.jpg)
8
Quantization (visual word)
• Point List = [(2,3), (5,4), (9,6), (4,7), (8,1), (7,2)]• Sorted List = [(2,3), (4,7), (5,4), (7,2), (8,1),(9,6)]
![Page 9: Lost in Quantization: Improving Particular Object Retrieval in Large Scale Image Databases CVPR 2008 James Philbin Ondˇrej Chum Michael Isard Josef Sivic.](https://reader038.fdocuments.in/reader038/viewer/2022103022/56649d2c5503460f94a023a2/html5/thumbnails/9.jpg)
Soft-assignment of visual words
• Matching two image features in bag-of-visual-words in hard-assignment– Yes if assigned to the same visual word– No otherwise
• Sort-assignment– A weighted combination of visual words
![Page 10: Lost in Quantization: Improving Particular Object Retrieval in Large Scale Image Databases CVPR 2008 James Philbin Ondˇrej Chum Michael Isard Josef Sivic.](https://reader038.fdocuments.in/reader038/viewer/2022103022/56649d2c5503460f94a023a2/html5/thumbnails/10.jpg)
Soft-assignment of visual words
A~E represent cluster centers (visual words)points 1–4 are features
![Page 11: Lost in Quantization: Improving Particular Object Retrieval in Large Scale Image Databases CVPR 2008 James Philbin Ondˇrej Chum Michael Isard Josef Sivic.](https://reader038.fdocuments.in/reader038/viewer/2022103022/56649d2c5503460f94a023a2/html5/thumbnails/11.jpg)
Soft-assignment of visual words
• – d is the distance from the cluster center to the
descriptor• In practice is chosen so that a substantial
weight is only assigned to few cells• The essential parameters– the spatial scale – r, nearest neighbors considered
![Page 12: Lost in Quantization: Improving Particular Object Retrieval in Large Scale Image Databases CVPR 2008 James Philbin Ondˇrej Chum Michael Isard Josef Sivic.](https://reader038.fdocuments.in/reader038/viewer/2022103022/56649d2c5503460f94a023a2/html5/thumbnails/12.jpg)
Soft-assignment of visual words
• the weights to the r nearest neighbors, the descriptor is represented by an r-vector, which is then L1 normalized
![Page 13: Lost in Quantization: Improving Particular Object Retrieval in Large Scale Image Databases CVPR 2008 James Philbin Ondˇrej Chum Michael Isard Josef Sivic.](https://reader038.fdocuments.in/reader038/viewer/2022103022/56649d2c5503460f94a023a2/html5/thumbnails/13.jpg)
TF–IDF weighting
• Standard index architecture
•
![Page 14: Lost in Quantization: Improving Particular Object Retrieval in Large Scale Image Databases CVPR 2008 James Philbin Ondˇrej Chum Michael Isard Josef Sivic.](https://reader038.fdocuments.in/reader038/viewer/2022103022/56649d2c5503460f94a023a2/html5/thumbnails/14.jpg)
TF–IDF weighting
• tf– 100 vocabularies in a document, ‘a’ 3 times– 0.03 (3/100)
• idf– 1,000 documents have ‘a’, total number of
documents 10,000,000– 9.21 ( ln(10,000,000 / 1,000) )
• if-idf = 0.28( 0.03 * 9.21)
![Page 15: Lost in Quantization: Improving Particular Object Retrieval in Large Scale Image Databases CVPR 2008 James Philbin Ondˇrej Chum Michael Isard Josef Sivic.](https://reader038.fdocuments.in/reader038/viewer/2022103022/56649d2c5503460f94a023a2/html5/thumbnails/15.jpg)
TF–IDF weighting
• In this paper– For the term frequency(tf)• we simply use the normalized weight value for each
visual word.
– For the inverse document(idf)• feature measure, we found that counting an occurrence
of a visual word as one, no matter how small its weight, gave the best results
![Page 16: Lost in Quantization: Improving Particular Object Retrieval in Large Scale Image Databases CVPR 2008 James Philbin Ondˇrej Chum Michael Isard Josef Sivic.](https://reader038.fdocuments.in/reader038/viewer/2022103022/56649d2c5503460f94a023a2/html5/thumbnails/16.jpg)
Re-ranking
• RANSAC– Affine transform Θ : Y = AX+b
• Algorithm– 1. Randomly choose n points– 2. Use n points to find Θ – 3. Input N-n points to Θ– 4. How many inlier– Repeat 1~4 K times– Pick the best Θ
![Page 17: Lost in Quantization: Improving Particular Object Retrieval in Large Scale Image Databases CVPR 2008 James Philbin Ondˇrej Chum Michael Isard Josef Sivic.](https://reader038.fdocuments.in/reader038/viewer/2022103022/56649d2c5503460f94a023a2/html5/thumbnails/17.jpg)
Re-ranking
• In this paper– No only counting the number of inlier
correspondences ,but also scoring function, or cosine =
![Page 18: Lost in Quantization: Improving Particular Object Retrieval in Large Scale Image Databases CVPR 2008 James Philbin Ondˇrej Chum Michael Isard Josef Sivic.](https://reader038.fdocuments.in/reader038/viewer/2022103022/56649d2c5503460f94a023a2/html5/thumbnails/18.jpg)
Average query expansion
• Obtain top (m < 50) verified results of original query• Construct new query using average of these results•
– where d0 is the normalized tf vector of the query region
– di is the normalized tf vector of the i-th result
• Requery once
![Page 19: Lost in Quantization: Improving Particular Object Retrieval in Large Scale Image Databases CVPR 2008 James Philbin Ondˇrej Chum Michael Isard Josef Sivic.](https://reader038.fdocuments.in/reader038/viewer/2022103022/56649d2c5503460f94a023a2/html5/thumbnails/19.jpg)
Outline
• Introduction• Methods in this paper• Experiment & Result• Conclusion
![Page 20: Lost in Quantization: Improving Particular Object Retrieval in Large Scale Image Databases CVPR 2008 James Philbin Ondˇrej Chum Michael Isard Josef Sivic.](https://reader038.fdocuments.in/reader038/viewer/2022103022/56649d2c5503460f94a023a2/html5/thumbnails/20.jpg)
Dataset
• Crawled from Flickr & high resolution(1024x768)• Oxford buildings– About 5,062 high resolution(1024x768) images– using 11 landmarks as queries
• Paris– Used for quantization– 6,300 images
• Flickr1– 145 most popular tags– 99,782 images
![Page 21: Lost in Quantization: Improving Particular Object Retrieval in Large Scale Image Databases CVPR 2008 James Philbin Ondˇrej Chum Michael Isard Josef Sivic.](https://reader038.fdocuments.in/reader038/viewer/2022103022/56649d2c5503460f94a023a2/html5/thumbnails/21.jpg)
Dataset
![Page 22: Lost in Quantization: Improving Particular Object Retrieval in Large Scale Image Databases CVPR 2008 James Philbin Ondˇrej Chum Michael Isard Josef Sivic.](https://reader038.fdocuments.in/reader038/viewer/2022103022/56649d2c5503460f94a023a2/html5/thumbnails/22.jpg)
Dataset
• Query– 55 queries: 5 queries for each of 11 landmarks
![Page 23: Lost in Quantization: Improving Particular Object Retrieval in Large Scale Image Databases CVPR 2008 James Philbin Ondˇrej Chum Michael Isard Josef Sivic.](https://reader038.fdocuments.in/reader038/viewer/2022103022/56649d2c5503460f94a023a2/html5/thumbnails/23.jpg)
Baseline
• Follow the architecture of previous work [15]• A visual vocabulary of 1M words is generated
using an approximate k-means
[15] J. Philbin, O. Chum, M. Isard, J. Sivic, and A. Zisserman. Object retrieval with large vocabularies and fast spatial matching. In Proc. CVPR, 2007
![Page 24: Lost in Quantization: Improving Particular Object Retrieval in Large Scale Image Databases CVPR 2008 James Philbin Ondˇrej Chum Michael Isard Josef Sivic.](https://reader038.fdocuments.in/reader038/viewer/2022103022/56649d2c5503460f94a023a2/html5/thumbnails/24.jpg)
24
Evaluation
• Compute Average Precision (AP) score for each of the 5 queries for a landmark– Area under the precision-recall curve• Precision = RPI / TNIR• Recall = RPI / TNPCRPI = retrieved positive images
TNIR = total number of images retrieved
TNPC = total number of positives in the corpus• Average these to obtain a Mean Average
Precision (MAP)
Recall
Precision
![Page 25: Lost in Quantization: Improving Particular Object Retrieval in Large Scale Image Databases CVPR 2008 James Philbin Ondˇrej Chum Michael Isard Josef Sivic.](https://reader038.fdocuments.in/reader038/viewer/2022103022/56649d2c5503460f94a023a2/html5/thumbnails/25.jpg)
Evaluation
• Dataset– Only the Oxford (D1) 5,062 images– Oxford (D1) + Flickr1 (D2) 104,844 images
• Vector quantizers– Oxford or Paris
![Page 26: Lost in Quantization: Improving Particular Object Retrieval in Large Scale Image Databases CVPR 2008 James Philbin Ondˇrej Chum Michael Isard Josef Sivic.](https://reader038.fdocuments.in/reader038/viewer/2022103022/56649d2c5503460f94a023a2/html5/thumbnails/26.jpg)
Result
[14] D. Nister and H. Stewenius. Scalable recognition with a vocabulary tree.CVPR, 2006.
[18] T. Tuytelaars and C. Schmid. Vector quantizing feature space with a regular lattice. ICCV, 2007.
[15] J. Philbin, O. Chum, M. Isard, J. Sivic, and A. Zisserman. Object retrieval with large vocabularies and fast spatial matching. CVPR, 2007.
Parameter variation Comparison with other methods
![Page 27: Lost in Quantization: Improving Particular Object Retrieval in Large Scale Image Databases CVPR 2008 James Philbin Ondˇrej Chum Michael Isard Josef Sivic.](https://reader038.fdocuments.in/reader038/viewer/2022103022/56649d2c5503460f94a023a2/html5/thumbnails/27.jpg)
Result
Effect of vocabulary size
Spatial verification
![Page 28: Lost in Quantization: Improving Particular Object Retrieval in Large Scale Image Databases CVPR 2008 James Philbin Ondˇrej Chum Michael Isard Josef Sivic.](https://reader038.fdocuments.in/reader038/viewer/2022103022/56649d2c5503460f94a023a2/html5/thumbnails/28.jpg)
Result
Query expansion
Scaling-up to 100K images
![Page 29: Lost in Quantization: Improving Particular Object Retrieval in Large Scale Image Databases CVPR 2008 James Philbin Ondˇrej Chum Michael Isard Josef Sivic.](https://reader038.fdocuments.in/reader038/viewer/2022103022/56649d2c5503460f94a023a2/html5/thumbnails/29.jpg)
Result
![Page 30: Lost in Quantization: Improving Particular Object Retrieval in Large Scale Image Databases CVPR 2008 James Philbin Ondˇrej Chum Michael Isard Josef Sivic.](https://reader038.fdocuments.in/reader038/viewer/2022103022/56649d2c5503460f94a023a2/html5/thumbnails/30.jpg)
Result
ashmolean_3 goes from 0.626 AP to 0.874 APchrist_church_5 increases from 0.333 to 0.813 AP
![Page 31: Lost in Quantization: Improving Particular Object Retrieval in Large Scale Image Databases CVPR 2008 James Philbin Ondˇrej Chum Michael Isard Josef Sivic.](https://reader038.fdocuments.in/reader038/viewer/2022103022/56649d2c5503460f94a023a2/html5/thumbnails/31.jpg)
Outline
• Introduction• Methods in this paper• Experiment & Result• Conclusion
![Page 32: Lost in Quantization: Improving Particular Object Retrieval in Large Scale Image Databases CVPR 2008 James Philbin Ondˇrej Chum Michael Isard Josef Sivic.](https://reader038.fdocuments.in/reader038/viewer/2022103022/56649d2c5503460f94a023a2/html5/thumbnails/32.jpg)
Conclusion
• A new method of visual word assignment was introduced:– descriptor-space soft-assignment
• It improves that descriptor lost in the quantization step of previously published methods.