Similarity-Based Comparisons for Interactive Fine-Grained...
1
Qualitative Results Overview Approach A. Collect grid-based similarity comp- arisons that do not require prior expertise B. Broadcast grid-based comparisons to triplet comparisons Results Similarity Comparisons INTERACTIVE CATEGORIZATION • Compute per-class probabilities as: where Efficient computation • Approximate per-class probabilities as: i.e. sum of weights of examples of class where enumerates training examples • Weight ݓ represents how likely ܢ is true location ܢ: such that Incorporating Users • ܦis grid of images for each question Incorporate independent user response as: Method Avg. #Qs CV, Color Similarity 2.70 CV, Shape Similarity 2.67 CV, Pattern Similarity 2.67 CV, Color/Shape/Pattern Similarity 2.64 No CV, Color/Shape/Pattern Similarity 4.21 Multiple Metrics • System supports multiple similarity metrics as different types of questions • Simulate perceptual spaces using CUB-200-2011 attribute annotations Learned Embedding • Learn category-level embedding of = 200 nodes • Category-level embedding requires much fewer comparisons compared to at the instance-level Catherine Wah 1 Grant Van Horn 1 Steve Branson 2 Subhransu Maji 3 Pietro Perona 2 Serge Belongie 4 1 University of California, San Diego vision.ucsd.edu {cwah@cs,gvanhorn@}ucsd.edu 2 California Institute of Technology vision.caltech.edu {sbranson,perona}@caltech.edu 3 Toyota Technological Institute at Chicago ttic.edu [email protected] 4 Cornell Tech tech.cornell.edu [email protected] TRAIN TIME TEST TIME 2) Computer Vision | ݔ3) Learn Perceptual Embedding 2) Collect Similarity Comparisons ? 1) Image Database w/ Class Labels Mallard Cardinal 1) Query Image 3) Human-in-the-Loop Categorization | ݔ, ௧ ? Problem • Parts and attributes exhibit weaknesses ¾ Scalability issues; costly; reliance on experts, but experts are scarce Proposed Solution • Use relative similarity comparisons to reduce dependence on expert- derived part and attribute vocabularies Contributions • We present an efficient, flexible, and scalable system for interactive fine-grained visual categorization ¾ Based on perceptual similarity ¾ Combines similarity metrics and computer vision methods in a unified framework • Outperforms state-of-the-art relevance feedback-based and part/attribute-based approaches Is this more similar to… ݔ This one? ݔ Or this one? ݔ ݏ, > ݏ, ݏ, > ݏ, ݏ, > ݏ, ݏ, > ݏ, ݏ, > ݏ, ݏ, > ݏ, ݏ, > ݏ, ݏ, > ݏ, … ? ݏ( , ): perceptual similarity between images ݔ and ݔ A B Q1: Most Similar? Query Image Q2: Most Similar? Vermilion Fly- catcher Q1: Most Similar By Color? Q2: Most Similar By Pattern? Hooded Merganser Query Image Interactive Categorization • Similarity comparisons are advantageous compared to part/attribute questions • Using computer vision reduces the burden on the user • Intelligently selecting image displays reduces effort • The system is robust to user noise Deterministic users No computer vision Deterministic users With computer vision Simulated noisy users With computer vision Learning a Metric • Given set of triplet comparisons , learn embedding ܈of training images with stochastic triplet embedding [van der Maaten & Weinberger 2012] • From ܈, generate similarity matrix א × Selecting the Display • Approximate solution: maximizes expected information gain in terms of entropy of , ܢ , ௧ | ݔ• Group images into equal-weight clusters [Fang & Geman 2005] • From each cluster, select image with largest ݓ ௧ = , , ݔ more similar to ݔ than ݔ ݔQuery image Class ݐTime step ௧ User responses at ݐ ܢTrue location of ݔin perceptual space Computer Vision • Easy to map off-the-shelf CV algorithms into framework, e.g., multiclass classification scores , ݔܢ ן | ݔ ,| ݔ, ௧ ן , ௧ | ݔ= න , ܢ, ௧ | ݔ ܢ ܢ ݓ௧ = , ܢ, ௧ | ݔ= ௧ | , ܢ, ݔ , ݔܢ ܢݑ= ߶ ݏ(ܢ, ܢ ) σ ߶ ݏ(ܢ, ܢ ) א ,| ݔ, ௧ ൎ σ ݓ ௧ , ೖ ୀ σ ݓ ௧ ݓ ௧ = , ܢ , ௧ | ݔ= ௧ | , ܢ , ݔ , ܢ ݔ ݓ ௧ାଵ = ݑ௧ାଵ ܢ ݓ ௧ = ߶ σ ߶ א ݓ ௧ Efficient update rule: ඹ Initialize weights ݓ = , ܢ ݔය Update weights ݓ ௧ାଵ when user answers a similarity question ර Update per-class probabilities A A B B C C D D D D Similarity Comparisons for Interactive Fine-Grained Categorization
Transcript of Similarity-Based Comparisons for Interactive Fine-Grained...
Qualitative Results
Overview Approach
A. Collect grid-based similarity comp-arisons that do not require prior expertise
B. Broadcast grid-based comparisons to triplet comparisons
Results
Similarity Comparisons
INTERACTIVE CATEGORIZATION • Compute per-class probabilities as:
where
Efficient computation • Approximate per-class probabilities as:
i.e. sum of weights of examples of class where enumerates training examples
• Weight ݓ represents how likely ܢ is true location ܢ: such that
Incorporating Users • is grid of images for each question ܦ
Incorporate independent user response as:
Method Avg. #Qs
CV, Color Similarity 2.70
CV, Shape Similarity 2.67
CV, Pattern Similarity 2.67
CV, Color/Shape/Pattern Similarity 2.64
No CV, Color/Shape/Pattern Similarity 4.21
Multiple Metrics • System supports multiple similarity
metrics as different types of questions
• Simulate perceptual spaces using CUB-200-2011 attribute annotations
Learned Embedding • Learn category-level embedding of
= 200 nodes • Category-level embedding requires
much fewer comparisons compared to at the instance-level
Catherine Wah1 Grant Van Horn1 Steve Branson2 Subhransu Maji3 Pietro Perona2 Serge Belongie4 1University of California, San Diego
vision.ucsd.edu {cwah@cs,gvanhorn@}ucsd.edu
2California Institute of Technology vision.caltech.edu
{sbranson,perona}@caltech.edu
3 Toyota Technological Institute at Chicago ttic.edu
4 Cornell Tech tech.cornell.edu
TRAI
N T
IME
TEST
TIM
E
2) Computer Vision
|ݔ
3) Learn Perceptual Embedding
2) Collect Similarity Comparisons
?
1) Image Database w/ Class Labels
Mallard
Cardinal
1) Query Image 3) Human-in-the-Loop Categorization
|,ݔ
௧
?
Problem • Parts and attributes exhibit weaknesses ¾ Scalability issues; costly; reliance on experts, but experts are scarce
Proposed Solution • Use relative similarity comparisons to reduce dependence on expert-
derived part and attribute vocabularies
Contributions • We present an efficient, flexible, and scalable system for interactive
fine-grained visual categorization ¾ Based on perceptual similarity ¾ Combines similarity metrics and computer vision methods in a
unified framework • Outperforms state-of-the-art relevance feedback-based and
part/attribute-based approaches
Is this more similar to…
ݔThis one?
ݔ
Or this one?
ݔ
ݏ , > ݏ , ݏ , > ݏ , ݏ , > ݏ , ݏ , > ݏ , ݏ , > ݏ , ݏ , > ݏ , ݏ , > ݏ , ݏ , > ݏ , …
?
,)ݏ ): perceptual similarity between images ݔ and ݔ
A
B
Q1: Most Similar? Query Image
Q2: Most Similar?
Vermilion Fly-
catcher
Q1: Most Similar By Color? Q2: Most Similar By Pattern?
Hooded Merganser
Query Image
Interactive Categorization • Similarity comparisons are advantageous compared to part/attribute questions • Using computer vision reduces the burden on the user • Intelligently selecting image displays reduces effort • The system is robust to user noise
Deterministic users No computer vision
Deterministic users With computer vision
Simulated noisy users With computer vision
Learning a Metric • Given set of triplet comparisons , learn
embedding ܈ of training images with stochastic triplet embedding [van der Maaten & Weinberger 2012]
• From ܈, generate similarity matrix א ×
Selecting the Display • Approximate solution: maximizes
expected information gain in terms of entropy of , ܢ , ௧|ݔ
• Group images into equal-weight clusters [Fang & Geman 2005]
• From each cluster, select image with largest ݓ௧
= , , ݔ more similar toݔ than ݔ
Query image ݔ Class
Time step ݐ௧ User responses at ݐ
in ݔ True location of ܢperceptual space
Computer Vision • Easy to map off-the-shelf CV
algorithms into framework, e.g., multiclass classification scores
, ܢ ݔ ן ݔ|
, ,ݔ| ௧ ן , ௧|ݔ = න , ,ܢ ௧|ݔ ܢܢ
௧ݓ = , ,ܢ ௧|ݔ = ௧| , ,ܢ ݔ , ܢ ݔ
ݑ ܢ = ߶ ,ܢ)ݏ (ܢ
σ ߶ ,ܢ)ݏ א)ܢ
, ,ݔ| ௧ ൎσ ௧,ೖୀݓσ ௧ݓ
௧ݓ = , ܢ , ௧|ݔ = ௧| , ܢ , ݔ , ܢ ݔ
௧ାଵݓ = ௧ାଵݑ ܢ ௧ݓ
=߶
σ ߶ א௧ݓ
Efficient update rule: ඹ Initialize weights ݓ = , ܢ ݔ ය Update weights ݓ௧ାଵ when user answers
a similarity question ර Update per-class probabilities
A
A
B
B
C
C
D
D
D
D
Similarity Comparisons for Interactive Fine-Grained Categorization
Subhransu Maji
