Li-J ia LiYongwhan LimLi Fe i -Fe iChong WangDavid M. Ble i

BUILDING AND USING A SEMANTIVISUAL IMAGE

HIERARCHYCVPR, 2010

IntroductionBuilding the hierarchy

Graphical modal Learning

Semantivisual image hierarchy Implementation Visualizing the semantivisual hierarchy Quantitative evaluation

Application Annotation Labeling Classification

OUTLINE

For images, a meaningful image hierarchy can make image organization, browsing and searching more convenient and effective

Good image hierarchies can serve as knowledge ontology for end tasks such as image retrieval, annotation or classification.

Language-basedLow-level visual feature based

INTRODUCTION

Use a multi-modal model to represent images and textual tags on the semantivisual hierarchy

Each image is associated with a path of the hierarchy, where the image regions can be assigned to different nodes of the path

BULIDING THE HIERARCHY

• Each image is decomposed into a set of over-segmented regions R = [R1…Rr…RN]

• each of the N regions is characterized by four appearance features

Graphical model Each image-text pair (R,W) is assigned to a path C c = [Cc1,…,Ccl,…,CcL]

BULIDING THE HIERARCHY

Learning the semantivisual image hierarchy Given a set of unorganized images and user tags associated with them Gibbs sampling : samples concept index Z, coupling variable S and path C

Sampling Z Depend on 1) the likelihood of the region appearance

2) the likelihood of tags associated with this region3) the concept indices of the other regions in the same image-text pair

..

BULIDING THE HIERARCHY

Sampling S Its conditional distribution solely depends on the likelihood of the tag

Sampling C Influenced by the previous arrangement of the hierarchy and the

likelihood of the image-text pair

BULIDING THE HIERARCHY

Prior probability induced by nCRP

likelihood

4000 user upload images and 538 unique user tags

Each image is divided into small patches of 10×10 pixels. Each patch is assigned to a codeword in a codebook of 500 visual word

obtained by K-means Obtain 4 region codebook for color(HSV histogram), location, texture,

normalized SIFT histogram To speed up learning, we initialize the levels in a path according to tf-idf score . We obtain a hierarchy of 121 nodes, 4 levels and 53 paths.

A SEMANTIVISUAL IMAGE HIERARCHY- - I mpleme ntati on

A SEMANTIVISUAL IMAGE HIERARCHY- - V i s ua l i z ing the Sema nti v is ua l H ierarchy

• General-to-specific relationship• Purely visual information cannot provide meaningful

image hierarchy• Purely language-based hierarchy would miss close

connection

Good clustering of images that share similar concepts ,i.e., image along the same path, should be more or less annotated with similar tags.

Good hierarchical structure given path, i.e., images and their associated tags at different levels of the path, should demonstrate good general-to-specific relationships.

A SEMANTIVISUAL IMAGE HIERARCHY- - A Quanti tati ve Eva luati on Of Im age H ierarch ies

A path of L levels is selected from the hierarchy.

Given our learned image ontology, we can propose a hierarchical annotation of an unlabeled query image.

nCRP cannot perform well on sparse tag words. I ts proposed hierarchy has many words assigned to the root node, resul t ing in very few paths.

A simple cluster ing algori thm such as KNN cannot find a good association between the test images and the training images in our chal lenging dataset with large visual divers ity.

In contrast , our model learns an accurate associat ion of visual and text data s imultaneously

APPLICATION- - H ierarch ica l A nnotatio n of I mage

Serving as an image and text knowledge ontology, our semantivisual hierarchy and model can be used for image labeling without a hierarchical relation.

APPLICATION- - I ma ge L abe l ing

Collect the top 5 predicted words of each image

Our model captures the hierarchical structure of image and tags !!

Another 4000 image are held out as test images.

APPLICATION- - I ma ge C las s ificati on

By encoding semantic meaning tothe hierarchy, our semantivisual hierarchy delivers a moredescriptive structure, which could be helpful for classification.

Use image and their tags to construct a meaningful hierarchy that organizes images in a general-to-specific structure.

Our quantitative evaluation by human subjects shows that our hierarchy is more meaningful and accurate than others.

CONCLUSION