Similar image search
-
Upload
aliaishang -
Category
Documents
-
view
118 -
download
0
description
Transcript of Similar image search
Similar Image Search with a Tiny Bag-of-Delegates
Representation
Weiwen Tu, Rong Pan, Jingdong Wang
Similar Image Search
• text-based image search• context-based image search
Tiny Bag-of-Delegates
Given a large image set ,
our goal is to find a bag-of-delegates to represent each image.
is represented by M delegates,
Tiny Bag-of-Delegates
So we need to build M vocabularies ,
where with J being the size of each vocabulary.
From each vocabulary an element is select as a delegate for each image.
Tiny Bag-of-Delegates
To conduct search, we organize the image set using M inverted indices ,each corresponding to a vocabulary respectively.
An inverted index is composed of J list,
each list is corresponding to a word, consists of images that are mapped to the word ( corresponds to )
Example for Tiny Bag-of-Delegates
𝐼 1→(𝑣1 ,𝑣3 ,𝑣5)
𝐼 2→(𝑣1 ,𝑣4 ,𝑣6)
𝐼 3→(𝑣2 ,𝑣3 ,𝑣6)
𝑉 3𝑉 2𝑉 1
𝑇 1
𝑇 2
𝑇 3
Query:
Tiny Bag-of-Delegates
Given an image whose bag-of-delegates is defined as , we find the lists
where is a function mapping from the delegate w to the list in the inverted index . Then regard all the images in these lists as the candidate similar images, and finally order these images according to their similarities computed with their features.
Spatial Partition Tree
• Spatial partition tree is a method for recursively subdividing a space into subsets by hyperplanes.
• Popular data structures: Kd-tree, PCA-tree, VP-tree.
Example for Spatial Partition Tree
The left figure shows a spatial partitioning induced by an RP tree. The cross is q query point and the lines means the partition hyperplanes.
𝒗𝟏
𝒗𝟐
𝒗𝟑𝒗𝟒
𝒗𝟓
𝒗𝟔𝒗𝟕
𝒗𝟖
Tree Construction
procedure MAKETREE(S)if(S) < MinSize
then return(Leaf)else;
return ([Rule, LeftTree, RightTree])
We define the split method as ChooseRule. The core tree-building is called MakeTree, and takes as input a data set .
Problem
A larger amount of trees yield a better performance, while requiring more storage to save the inverted indices and accordingly more query time
Problem:Can we use a small amount of trees but with the search performance guaranteed?
Vocabulary Construction
Our idea is to use supervised information in constructing trees. In similar image search, the supervised information is the true nearest neighbors of each image.
We denote the true neighbors of an image by aList ,
where is the index of a similar image.
Criterion
The nearest neighbor candidates discovered from the bag-of-delegates representation can be written as
The recall of :The average recall :
Criterion
Given a set of Z candidate inverted indices
the goal is to find inverted indices such that the recall is maximized. The objective function is written as
where .
Forward selection scheme
We progressively generate the candidate inverted indices for each step. At the beginning, we randomly generate a set of candidate inverted indices
,We evaluate each inverted index to compute the recall, and then identify the first inverted index that corresponds to the largest recall. Denote the identified inverted index as and the current solution is
Forward selection scheme
The later steps sequentially find the inverted indices oneby one. Considering the step, we have found t indices .We generate a set of new candidates
and form the whole candidates
The objective of identifying the (t+ 1)-th index is as follows,
Random selection vs. forward selection
Random selection
Forward selection
Adaptive scheme
We maintain a set
where and are truly neighboring points but not appear in the same bucket in the previously-identified indices .
Compute top r sparse principal directions from sample points,
Adaptive scheme
Each direction can form a partition ,
where v is the current node with data points
Then we judge a pair is in the same side
Adaptive scheme
Finally, the objective function of the direction is,
Adaptive forward selection
1. Initialization: 𝑄←𝐿,𝑡 ←0,𝑒←ȁ𝑄ȁ,𝑅←∅
2. Repeat
3. Candidate proposal
Randomly generate spatial partition tree 𝒯ത per Q
4. Candidate selection
Choose the spatial partition tree T from 𝒯ത that keeps the large number of Q
5. Update
Discard all the pairs of points lying in the same bucket in T from Q 𝑡 ←𝑡+ 1,𝑒←ȁ𝑄ȁ,𝑅←𝑅∪{𝑇}
6. Until 𝑒≤ 𝜀 && 𝑡 ≥ 𝜏
7. Return R;
EXPERIMENTS
• Data set– 32 ×32 color images– 1M tiny images to form around 80M
images– global GIST descriptor(384 D vector)
• Evaluation criteria(average accuracy score)
Evaluation
Given an image , the accuracy is computed as,
: true neighbors list: the top K images found from M inverted indices, The whole average accuracy is computed as
Accuracy vs indices
Bucket size: the maximum number of points in a leaf node;g: the number of target NNs.
Accuracy vs indices
bucket size = 300
bucket size = 100
accuracy vs accessed images
Out-of-sample Test
Visual search result
13 : 9 22 : 9
Thank you!