Part-based Object Retrieval with Binary Partition Trees
-
Upload
xavi-giro -
Category
Technology
-
view
181 -
download
0
description
Transcript of Part-based Object Retrieval with Binary Partition Trees
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 1
Part-based Object Retrievalwith Binary Partition Trees
Phd thesis dissertation by Xavier Giró-i-Nieto
Supervised by Professor Ferran Marqués Acosta (UPC)
and Professor Shih-Fu Chang (Columbia)
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 2
AcknowledgementsJury GPI @ UPC
DVMM @ Columbia Students @ UPC
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 3
Outline
1. Introduction ←
2. Hierarchical Image Partitions
3. Interactive segmentation
4. Object Tree
5. Region Features
6. Codebooks
7. Fusion of Visual Modalities
8. Partition Tree Matching
9. Part-based Object Model
10.Conclusions
Part I
Part II
Part III
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 4
Problem statement
Content-Based Image Retrieval
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 5
Problem statement
Semantic gap between user query and machine data.
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 6
Challenges
Visual diversity within the same semantic class
anchor
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 7
Challenges
Visual diversity within the same instance
Thisanchor
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 8
Challenges
Semantics located at multiple scales
TV studioNews
Anchor
Jacket
Button
Head
Button
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 9
Problem statement
This thesis focuses in the object retrieval with one example...
Queryobject
Targetdataset
Ranked list
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 10
Problem statement
...and multiple examples.
Queryobjects
Target dataset
Ranked list
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 11
Outline
1. Introduction
2. Hierarchical Image Partitions ←
3. Interactive segmentation
4. Object Tree
5. Region Features
6. Codebooks
7. Fusion of Visual Modalities
8. Partition Tree Matching
9. Part-based Object Model
10.Conclusions
Part I
Part II
Part III
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 12
2. Hierchical Image Partition
Sliding windows
Image partition
Local scale
Interestpoints
Object analysis
ObjectSegmentation ? ✔✘✘
[Viola-Jones'01] SIFT [Lowe'04]
SURF [Bay'06]
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 13
✔Hierarchicalpartition
2. Hierchical Image Partition
SpatialPyramidof points
Multi-scale analysis
Multiplesegmentations
Semantics at multiple scales
[Lazebnik'06][Bosch'07]
[Ravinovich'07]
[Vieux'10]
[Adamek'06]
[Gu'09]
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 14
Adopted Solution: Binary Partition Trees (BPTs) [Garrido, Salembier 2000].
2. Hierchical Image Partition
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 15
Perceptual BPT
Semantic Object “head”
Split
Thesistopic
2. Hierchical Image Partition
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 16
Outline
1. Introduction
2. Hierarchical Image Partitions
3. Interactive segmentation ←
4. Object Tree
5. Region Features
6. Codebooks
7. Fusion of Visual Modalities
8. Partition Tree Matching
9. Part-based Object Model
10.Conclusions
Part I
Part II
Part III
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 17
3. Interactive segmentationMotivations: Ground truth generation and retrieval interface.
Demo @ http://upseek.upc.edu/gat & http://upseek.upc.edu/gos
Graphical Annotation Tool (GAT) Graphical Object Searcher (GOS)
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 18
Outline
1. Introduction
2. Hierarchical Image Partitions
3. Interactive segmentation
4. Object Tree ←
5. Region Features
6. Codebooks
7. Fusion of Visual Modalities
8. Partition Tree Matching
9. Part-based Object Model
10.Conclusions
Part I
Part II
Part III
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 19
Related work: “Definition Hierarchy”, Jaimes and Chang (2001)
4. Object Tree
Thesisapproach
BPTnodes
BPTleaves
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 20
Assumption: All objects are connected
Interactive segmentation BPT sub-roots selection
4. Object Tree
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 21
User selection of sub-roots on the
BPT.
Automatic top-down expansion through the
BPT
4. Object Tree
The Object Tree (OT) represents an instance of the object as a hierarchy of regions.
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 22
Outline
1. Introduction
2. Hierarchical Image Partitions
3. Interactive segmentation
4. Object Tree
5. Region Features ←
6. Codebooks
7. Fusion of Visual Modalities
8. Partition Tree Matching
9. Part-based Object Model
10.Conclusions
Part I
Part II
Part III
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 23
MPEG-7 and many more, “Visual Descriptors (VD)” (2001)
5. Region Features
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 24
5. Region Features
➔ Dominant Colors➔ Contour Shape➔ Edge Histogram
→ Region features in BPT
NOT Thesistopic
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 25
Outline
1. Introduction
2. Hierarchical Image Partitions
3. Interactive segmentation
4. Object Tree
5. Region Features ←
6. Codebooks
7. Fusion of Visual Modalities
8. Partition Tree Matching
9. Part-based Object Model
10.Conclusions
Part I
Part II
Part III
Evaluation
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 26
Evaluation
How to evaluate an object retrieval system ?
1. Image Retrieval2. Object Detection3. Object Segmentation4. Search time
Annotated dataset
+ Metrics
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 27
Evaluation (in Part II)
Annotated Dataset ETHZ: Category #1 “Apple logos”
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 28
Annotated Dataset ETHZ: Category #2 “Bottles”
Evaluation (in Part II)
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 29
Annotated Dataset ETHZ: Category #3 “Giraffes”
Evaluation (in Part II)
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 30
Annotated Dataset ETHZ: Category #4 “Mugs”
Evaluation (in Part II)
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 31
Annotated Dataset ETHZ: Category #5 “Swans”
Evaluation (in Part II)
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 32
Evaluation
1. Image Retrieval (global scale)
User Query (local) Ranked list of images (global)
Only global scale is considered.
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 33
Evaluation
P (k )=∣{relevant}∩{retrieved (k )}∣
k
R(k )=∣{relevant }∩{retrieved (k )}∣
∣relevant∣
1. Image Retrieval (global scale)
Mean Average Precision (MAP) combines Precision and Recall for a set of queries.
Plotted by consideringdifferent k's
Precision
Recall
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 34
5. Region features
Image Retrieval (global scale)
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 35
Evaluation
2. Object Detection (local scale-rough)
User Query (local) Retrieved boxes in relevant images
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 36
Evaluation
Pascal criterion requires a minimum of 50% overlapping in the union of detected object box and ground truth box.
2. Object Detection (local scale-rough)
Detection rate=∣{correct detections }∣
∣{relevant }∣
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 37
Object Detection (local scale-rough)
5. Region features
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 38
Evaluation
3. Object Segmentation (local scale-precise)
User Query (local)Retrieved regions
from correct detections
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 39
Evaluation
Jaccard index-J(A,B) compares retrieved region and ground truth mask at the pixel level.
J̄=1
∣D∣∑i∈D
J i
J (A , B)=∣A∩B∣∣A∪B∣
3. Object Segmentation (local scale-precise)
A B
Mean Jaccard Index combines the J's obtained by a set of correct detections D.
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 40
Segmentation (local scale-precise)
5. Region features
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 41
Outline
1. Introduction
2. Hierarchical Image Partitions
3. Interactive segmentation
4. Object Tree
5. Region Features
6. Codebooks ←
7. Fusion of Visual Modalities
8. Partition Tree Matching
9. Part-based Object Model
10.Conclusions
Part I
Part II
Part III
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 42
6. Codebooks
Visual Word (VW): Vector quantization of observation (point, region...) onto a codebook [Sivic, Zissermann 2003].
Codebook
Visual Words
CodeRegion
Regions
(a1,a
2,a
3,a
4) (b
1,b
2,b
3,b
4) (c
1,c
2,c
3,c
4)
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 43
6. Codebooks
Code Regions (CR) define the basis of the Parts Space.
“Regions described by their sub-parts”
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 44
6. Codebooks
Codebook
Visual Words
Regions
(1,0,0,0) ? (1,0,0,0) (0,1,0,0) ?
(a) Hard quantization
✘ ✘
SIFT [Sivic 2003]
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 45
6. Codebooks
Codebook
Visual Words
Regions
(0.25,0.25,0.25,0.25) (1,0,0,0)✘ ✘
(b) Probabilistic quantization
(0.25,0.25,0.25,0.25)
[Alpaydin 1998]
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 46
6. Codebooks
Codebook
Visual Words
Regions
(0,0,0,0) (1,0,0,0) ✘(0,0,0,0)
(c) Possibilistic quantization
✔
[Bezdek 1995]
Similarity to CRs
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 47
6. Codebooks
How to define the Parts Space with a given codebook ?
Possibilistic quantization alone characterizes regions, but not their sub-parts.
Region-basedPossibilistic
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 48
6. Codebooks
How to define the Parts Space with a given codebook ?
Bottom-up Max Pooling through BPT ✔ [Boureau'10]
FastExtraction
(max)
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 49
BPT is exploited to describe sub-trees.
→ Hierarchy of Bags of Regions (HBoR)
6. Codebooks
FastExtraction
(max)
EfficientAnalysis(resilency to splits)
HBoR
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 50
6. Codebooks
How to generate a codebook ? (1/2)
Train set
K-Means Clustering
Object parts
Codebook
BPT-based decomposition
CR1
CR2
CR3
CR4
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 51
a) Foreground and background regions - “generic”
b) Only object regions - “frgd”
6. Codebooks
How to generate a codebook ? (2/2)
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 52
1. Image Retrieval (global scale) - Texture
6. Codebooks
HBoR features outstand region-based for shape & texture (not dominant colors).
Little differences between generic vs frgd codebooks (5 classes in ETHZ).
HBoR
Codebook size
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 53
Outline
1. Introduction
2. Hierarchical Image Partitions
3. Interactive segmentation
4. Object Tree
5. Region Features ←
6. Codebooks
7. Fusion of Visual Modalities ←
8. Partition Tree Matching
9. Part-based Object Model
10.Conclusions
Part I
Part II
Part III
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 54
Problem: Similarity scores between descriptors are not comparable.
Solution:
d color
x̄color
d shape d texture
x̄ shape x̄texture
x̄
Normalization (z-scores, SVM, w-scores)
Fusion (average, product, max/min)
5. Region Features + 7. Fusion
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 55
Outline
1. Introduction
2. Hierarchical Image Partitions
3. Interactive segmentation
4. Object Tree
5. Region Features
6. Codebooks ←
7. Fusion of Visual Modalities ←
8. Partition Tree Matching
9. Part-based Object Model
10.Conclusions
Part I
Part II
Part III
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 56
6. Codebooks + 7. Fusion
Separate codebooks
When working with codebooks, when to apply fusion ?
Quantization
(a) Fusion after quantization - “nofused”
Cosinedistance
[colorshape
texture]
[ VW color
VW shape
VW texture]
Fusion[ x̄colorx̄ shapex̄ texture
] x
Queryregion
Targetregion
[ VW color
VW shape
VW texture]
[colorshape
texture]
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 57
Fused codebook
When working with codebooks, when to apply fusion ?
(b) Fusion during quantization - “fused”
6. Codebooks + 7. Fusion
Cosinedistance
[colorshape
texture]
[VW ]
x
Queryregion
Targetregion
[VW ]
[colorshape
texture]Quantization
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 58
6. Codebooks + 7. Fusion
1. Image Retrieval (global scale)
w-scoresz-scores
When working with codebooks, when to apply fusion ?
W-scores outperform z-scores when working with HBoR.
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 59
Outline
1. Introduction
2. Hierarchical Image Partitions
3. Interactive segmentation
4. Object Tree
5. Region Features
6. Codebooks
7. Fusion of Visual Modalities
8. Partition Tree Matching ←
9. Part-based Object Model ←
10.Conclusions
Part I
Part II
Part III Evaluation
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 60
CCMA News dataset.
Topology diversity on 11 anchors objects.
Evaluation (in Part III)
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 61
Outline
1. Introduction
2. Hierarchical Image Partitions
3. Interactive segmentation
4. Object Tree
5. Region Features
6. Codebooks
7. Fusion of Visual Modalities
8. Partition Tree Matching ←
9. Part-based Object Model
10.Conclusions
Part I
Part II
Part III
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 62
Reminder: The Object Retrieval Problem (single instance)
Queryobject
Targetdataset
Ranked list
8. Partition Tree Matching
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 63
Object Tree (OT) Target BPT
?Q2Q1
Q0
Q4Q3
Challenges:
(a) Distance for a given OT-BPT correspondence
(b) Matching algorithm
8. Partition Tree Matching
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 64
Proposal (ad-hoc): Weighted sum of part matches
8. Partition Tree Matching: Distance
αi=1
max (1,∣Ci∣)
area(i)
area(Pi)
Example:Children of part i
Parents of part i
Q2Q1
Q0
d=∑i=0
N
wi di wi=i ∏j∈Ancestors Qi
j=i⋅wParent Qi, where
α0=1 /2=w0α1=area(Q1)/area (Q0)α2=area(Q2)/area(Q0)
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 65
Two proposals:
8. Partition Tree Matching: Algorithm
Object Tree (OT) Target BPT
?Q2Q1
Q0
Q4Q3
(a) Top-down Matching
(b) Bottom-Up Matching
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 66
Assumption: All OT nodes are contained in a single sub-BPT.
Assumption: OT nodes are spread through different sub-BPTs.
- common
+ common(b) Bottom-Up Matching
8. Partition Tree Matching: Algorithm
(a) Top-down Matching
Sets a baseline
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 67
Q2Q1
Q0
Assumption: The best match for the OT is contained in a single sub-tree of the target BPT.
Target BPTOT
Q2Q1
Q0
8. Partition Tree Matching: Algorithm
(b) Top-down QT Matching
1. Set a match for Q0
2. Find matches for Q1 and Q2
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 68
What is the impact of decomposing the query into parts ?
8. Partition Tree Matching
Retrieval gain until the “correct” amount of parts (3).
Image Retrieval (global scale)
MPEG-7Region Top-down
match
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 69
What is the impact of decomposing the query into parts ?
8. Partition Tree Matching
Exponential growth of search time with the amount of parts.
Search time
Top-downmatch
MPEG-7Region
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 70
Assumption: All OT nodes are contained in a single sub-BPT.
Assumption: OT nodes are spread through different sub-BPTs.
- common
+ common(b) Bottom-Up Matching
8. Partition Tree Matching: Algorithm
(a) Top-down Matching
Sets a baseline
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 71
Q2Q1
Q0
Weaker assumption that in the top-down case:
This regionis not part
of the target BPT
→ OT leaves are among the nodes of the target BPT
(c) Bottom-UP QT Matching
8. Partition Tree Matching: Algorithm
Q2Q1
Q0
1. Set a match for Q1 & Q2
2. Assess the union o f candidates for Q1 and Q2 to match Q0
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 72
How to rapidly describe the merged region generated during search time ?
(c) Bottom-UP QT Matching
8. Partition Tree Matching: Algorithm
(1, 1, 1, 1)
→ Approximate HBoR satisfies this requirement.
Q2Q1
Q0
Object Tree
HBoR
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 73
What is the impact of introducing the bottom-up match ?
8. Partition Tree Matching
A clear gain, the object splits in the target BPTs are solved.
RegionVW
HBoR
Bottom-upgain
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 74
Outline
1. Introduction
2. Hierarchical Image Partitions
3. Interactive segmentation
4. Object Tree
5. Region Features
6. Codebooks
7. Fusion of Visual Modalities
8. Partition Tree Matching
9. Part-based Object Model ←
10.Conclusions
Part I
Part II
Part III
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 75
Queryobjects
Target dataset
Ranked list
9. Part-based Object Model
Reminder: The Object Retrieval Problem (multiple instances)
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 76
Query objects Model
Training
Target image
Test
Retrieved object
Model-based retrieval
9. Part-based Object Model
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 77
9. Part-based Object Model
Parts definition
Tra
inin
g
# Parts
Inclusion classifier
Detector classifier
Fusion classifier
# Parts
ContextualfilteringT
est
Fusion classifier
Inclusion classifier
Detector classifier
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 78
9. Part-based Object Model
Parts definition
Tra
inin
g
# Parts
Inclusion classifier
Detector classifier
Fusion classifier
# Parts
ContextualfilteringT
est
Fusion classifier
Inclusion classifier
Detector classifier
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 79
9. Part-based Object Model
Parts definition
Unsupervised clustering with Quality Threshold [Heyer 1999].
Two parameters: - Mininum amount of elements in cluster- Maximum radius of a cluster
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 80
9. Part-based Object Model
Parts definition
Tra
inin
g
# Parts
Inclusion classifier
Detector classifier
Fusion classifier
# Parts
ContextualfilteringT
est
Fusion classifier
Inclusion classifier
Detector classifier
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 81
Positive: Parts and their ancestorsNegative: Parts' children and positive node's siblings.
9. Part-based Object Model
Inclusion classifier (training)
HBoR
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 82
9. Part-based Object Model
Parts definition
Tra
inin
g
# Parts
Inclusion classifier
Detector classifier
Fusion classifier
# Parts
ContextualfilteringT
est
Fusion classifier
Inclusion classifier
Detector classifier
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 83
Efficient top-down exploration of the target BPT.
9. Part-based Object Model
Inclusion classifier (test)
Not explored(efficiency)
HBoR
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 84
9. Part-based Object Model
Parts definition
Tra
inin
g
# Parts
Inclusion classifier
Detector classifier
Fusion classifier
# Parts
ContextualfilteringT
est
Fusion classifier
Inclusion classifier
Detector classifier
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 85
Positive: Parts.Negative: Random sampling (balanced set).
9. Part-based Object Model
Detector classifier (training)
Region-basedPossibilistic
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 86
9. Part-based Object Model
Parts definition
Tra
inin
g
# Parts
Inclusion classifier
Detector classifier
Fusion classifier
# Parts
ContextualfilteringT
est
Fusion classifier
Inclusion classifier
Detector classifier
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 87
Positive training samples represent valid combinations of parts.Negative training samples represent invalid combinations.
Parts of object i Parts of object j
Type 1 Type 2 Type 3 Type 4
(1, 1, 0, 0) (0, 1, 1, 1)
9. Part-based Object Model
Fusion classifier (training)
Parts definition
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 88
9. Part-based Object Model
Parts definition
Tra
inin
g
# Parts
Inclusion classifier
Detector classifier
Fusion classifier
# Parts
ContextualfilteringT
est
Fusion classifier
Inclusion classifier
Detector classifier
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 89
9. Part-based Object Model
How important is a correct estimation of the types of parts ?
The performance is dependent from the Quality Threshold parameters.
CodeBooksize
Image
Retrieval
Min # elems in cluster
Max radius for cluster
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 90
9. Part-based Object Model
What is the impact of codebook size in terms of time ?
Search time does not grow with codebook size (Efficient exploration ??).
Training
time
Search
time
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 91
Outline
1. Introduction
2. Hierarchical Image Partitions
3. Interactive segmentation
4. Object Tree
5. Region Features
6. Codebooks
7. Fusion of Visual Modalities
8. Partition Tree Matching
9. Part-based Object Model
10.Conclusions ←
Part I
Part II
Part III
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 92
Conclusions
BPT presents several opportunities for local analysis:
Three main contributions in this thesis:
→ Interactive segmentation→ Semantic analysis
Contribution#1
Contribution#2
Contribution#3
Hierarchy of Bags of Regions:● Possibilistic quantization● Max pooling throug BPT
Object Tree - BPT matching● Top-down● Bottom-up
Part-based model for objects● Automatic determination of parts● Efficient BPT analysis.
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 93
Open work
Test in public benchmark.
→ Instance Search task @ TRECVID 2012
Richer image representations
→ Amount of BPT leaves adapted to content→ Discard uninformative merges & allow multiple options
Enrich region descriptors
Better model for combination of parts representing object
→ Introduce interest points→ Significance estimation
Local segmentation of objects from global annotations
Smarter codebooks
→ Unsupervised clustering→ TF-IDF weights & Stop words
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 94
Outline
1. Introduction
2. Hierarchical Image Partitions
3. Interactive segmentation
4. Object Tree
5. Region Features
6. Codebooks
7. Fusion of Visual Modalities
8. Partition Tree Matching
9. Part-based Object Model
10.Conclusions
Part I
Part II
Part III
Gru
p d
e P
rocess
am
en
t d
e la Im
atg
eU
niv
ers
itat
Po
litè
cn
ica d
e C
ata
lun
ya (
UP
C)
X.Giró-i-Nieto, “Part-based Object Retrieval with Binary Partition Trees” 31/5/2012 @ UPC 95
Thank you, moltes gràcies !