ACM MM 2000, LA, USA1

Giving Meanings to WWW Images

Heng Tao Shen

Beng Chin Ooi

Kian Lee Tan

ACM MM 20002

Outline

Image Representation Model

Semantic Measure Model

Relevance Feedback

Experiments

ACM MM 20003

Background

Image: indispensable component in WWW– 1 image = 1000 words

WWW: rich resource of images– Some 100 billions?

Tradition: poor performance– Keywords

Content_based: no enough semantic– Like object, event, and relationship

Not effective for images from WWW

ACM MM 20004

Cont

Semantics of embedded images in HTML– Image Title, ALT, Page Title, Image Caption ->

ChainNet model Similarity between query and image

– List space model Relevance feedback:

– Improve precision further

ACM MM 20005

Weight ChainNet model

Lexical chain(LC) – A sentence that carries certain semantics by its words

6 types of LC– TLC: Title Lexical Chain– PLC: Page Lexical Chain– ALC: Alt Lexical Chain– SLC: Sentence Lexical Chain– RSLC: Reconstructed Sentence Lexical Chain– CLC: Caption Lexical Chain

ACM MM 20006

Title

Caption

7

8

9144

5

ALT

SLC: 1->2->3->4->5

RSLC: 1->2->8->9

CLC: 1->2->…->14

3

2

1 4

Page

Title

ACM MM 20007

Semantic measure model

Computing similarity between two LCs– List space model

MatchScalesizelistsizelist

weighteweighte

Similarity

sizelist

i

sizelist

jji

listlist *().2*().1

.*.

().1

0

().2

02,1

Where ei and ej are matched terms in list 1 and list 2 respectively.

ACM MM 20008

Semantic measure model

– Match scale: closeness in view of match order

||2||*||1||

212,1 vv

vvleMatchedSca vv

Here v1 and v2 represent the children of first and second original lists respectively.

Inspired from the angle between two vectors

().1

1

2*121sizev

iji vvvv

Where v2j is the matched word in v2 for v1i in v1

ACM MM 20009

Semantic measure model

LC Match Level(LC1, LC2): the number of distinct matched words by two LCs

– Match level threshold: The minimum match level for LC to keep its original semantic

– LC Semantic similarity: similarity(list1, list2) in its LC Match Level

ACM MM 200010

Semantic measure model

Image Match Level(image, query) = MAX ( TLC.weight * LCMatchLevel( TLC, QLC),

ALC.weight * LCMatchLevel( ALC, QLC),

PLC.weight * LCMatchLevel( PLC, QLC),

SLC.weight * LCMatchLevel( SLC, QLC),

RSLC.weight *LCMatchLevel( RSLC, QLC),

CLC.weight * LCMatchLevel( CLC, QLC) )

ACM MM 200011

Relevance Feedback

Semantic Accumulation

– Choose one best image as feedback– Accumulate the previous feedback images’ semantics to

construct a new QLC– Results are more close to the specific image selected– More noise

ACM MM 200012

Semantic accumulation

Weight F/Q ChainNet

QLCNew query

Last feedback image

Image ALT

Image Title

Image Caption

Page Title

ACM MM 200013

Semantic Integration and Differentiation

Semantic Integration and Differentiation

– Choose several Good and Bad images as feedback– Integrate Good semantics to construct new query– Differentiate irrelevant images by Bad images– Results are more diverse and less noise

ACM MM 200014

Semantic integration and differentiation

Similar weight F/Q ChainNet

QLCNew query

Good feedback images

Image iImage 3Image 2Image 1

LC1 LC2 LC3 LCi

Most related LC

ACM MM 200015

Experiments

Set up

– Web crawler to collect images– 5232 images from over 2000 URLs– 12 general queries

ACM MM 200016

0

0.2

0.4

0.6

0.8

1

0 0.2 0.4 0.6 0.8 1

Recall

Pre

cis

ion

TLC ALC PLC SLC

RSLC CLC Opt

Tuning the LC Weights

ACM MM 200017

Tune the match level

Chart1: Precision vs Coef

0

0.2

0.4

0.6

0.8

1

1.2

0 0.2 0.4 0.5 0.6 0.8 1

Coef

Prec

isio

n

Initial Try

Chart2: Recall vs Coef

0

0.2

0.4

0.6

0.8

1

1.2

0 0.2 0.4 0.5 0.6 0.8 1

Coef

Reca

ll

Initial Try

MatchLevel Threshold= coef * query.length()+ constant

ACM MM 200018

Impact of match scale

– explore the importance of match order

ACM MM 200019

Feedback Mechanisms

0

0.2

0.4

0.6

0.8

1

0 0.2 0.4 0.6 0.8 1

Recall

Pre

cisi

on I&D

Accu

Opt

ACM MM 200020

Feedback Mechanisms

One-step feedback of Accu and I&D for Q1.

ACM MM 200021

Conclusion

– Inner semantic structure of surrounding text is explored well for good precision achievement

– ChainNet model and list space model work well

– RF techniques help to return more accurate results

ACM MM 200022

Future work

– Explore LC meanings by AI technique

– Extract semantics from visual content, then integrate with our system to construct a more advanced semantic retrieval system

– Object-oriented detection

ACM MM 200023

DEMO ON THURSDAY

SEE YOU THEN…

http://efoto.geofoto.com