Contextual Wisdom

Social Relations and Correlations for Multimedia Event Annotation

Amit Zunjarwad, Hari Sundaram and Lexing Xie

I don’t want to spend time annotating :( help!

April 18, 2023@NUS 2

Talk Outline

Observations

Events

Generalization:Sum of Partial Observations

Similarity, Co-Occurrence and Trust

@I2R April 18, 2023 3

Experiments:compare against SVM

Conclusions

An Annotation Puzzle

@NUS April 18, 2023 4

April 18, 2023 5@NUS

Observing Flickr Data

The pool statistics reveal a power law distribution• Less than 11% of the tags have more than 10 photos• There are not enough instances to learn most of the

concepts! The global flickr pool is interesting:

April 18, 2023 6@NUS

Learnability

April 18, 2023 7@NUS

Learnability

The pool statistics reveal a power law distribution• Less than 11% of the photos have more than 10

instances• There are not enough instances to learn most of the

concepts! The global flickr pool is interesting: • Most of the tags have over 100 instances• Photos reveal very high visual diversity

The Power law is a fundamental property of online networks – cannot be wished away.

April 18, 2023 8@NUS

Learnability

Singapore People Walking Orchard rd. After MRT Experimenting Walking Day Outdoor..

April 18, 2023 9@NUS

Scalability

The assumption of consensual semantics Search for “yamagata”

April 18, 2023 10@NUS

The Role of context

April 18, 2023@NUS 11

What if the answer didn’t completely lie in the pixels?

EventsWhat are they?

April 18, 2023@NUS 12

An event refers to a real-world occurrence, spread over space and time.

Observations form event meta data [Westermann / Jain 2007]• Images / text / sounds describe events

April 18, 2023 13@NUS

Defining Events

when

where

who

what

author

image

Event context refers to the set of attributes that help in understanding the semantics • Images / Who / Where / When / What / Why / How

Context is always application dependent • Ubiquitous computing community – location, identity

and time are main considerations

April 18, 2023 14@NUS

Context

[Mani and Sundaram 2007]

Event archival – events involve people, places and artifacts

Exploit different forms of knowledge: • (Global) Similarity – media, events, people. • (Personal) Co-occurrence – what are the joint statistics

of occurrence?• (Social) Trust – determining whom to trust for effective

annotation?

April 18, 2023 15@NUS

Four Problems

SimilarityGlobal, Systemic knowledge

April 18, 2023@NUS 16

A bottom up approach • Edge, color and texture histograms for images. • Rely on ConceptNet for text tags

Why ConceptNet and not WordNet?• Expands on pure lexical terms, to compound terms –

“buy food”• Expands on number of relations – from three to twenty• Contains practical knowledge – we can infer that a

student is near a library.

April 18, 2023 17@NUS

Event similarity

ConceptNet provides three functions:• GetContext(node): the neighborhood of the concept “book” includes “knowledge”, “library”

• GetAnalogousConcepts(node): concepts that share incoming relations; analogous concepts for the concept “people” are “human”, “person”, “man”

• FindPathsBetweenNodes(node1,node2) – returns a set of paths.

Our similarity measure is built using these functions.

April 18, 2023 18@NUS

A base similarity measure

The similarity between two concepts (e,f) is defined as follows:

We current use a uniform weighting on all three as the composite measure

April 18, 2023 19@NUS

Concept similarity

fe

fCeC

eA fA

context

analogous

path based1

1 1( , )

N

pi i

s e fN h

The distance between two concept sets is a modified Haussdorf similarity.

April 18, 2023 20@NUS

Computing similarity between sets

A

B

| |

1

1( , | ) max{ ( , )}

| |

A

H k ii

k

S A B m m a bA

Similarity between facets are computed using a weighted sum of frequency and the concept similarity measure:

Time distance is based on text tags, not actual time data – allows for temporal descriptions as “summer”, “holidays” etc.

Only frequency is used for “who” facet.April 18, 2023 21@NUS

Facet similarity (4w)

1 21 2 1 2

2

1 | |( , ) , | ,

2 | | H

L Ls L L S L L CS

L

Color, texture and edges are computed• 166 bin HSV color histogram• 71 bin edge histogram• 3 texture features

Euclidean distance on the composite feature vector.

The distance between two events is then a weighted sum of distances across all event facets.

April 18, 2023 22@NUS

Image facet similarity

April 18, 2023 23@NUS

The global similarity matrix Ms

Co-occurrencePersonal, statistical knowledge

April 18, 2023@NUS 24

The concept co-occurrences are just frequency counts.

(i= fun , j = new york) then the index (i,j) contains the number of occurrences of this tuple.

Notes:• Each concept is given a globally unique index• Co-occurrence matrixes are locally compact

Each user k, has a co-occurrence matrix Mck

associated with the user.

April 18, 2023 25@NUS

Statistics are computed per person

TrustPeople we like

April 18, 2023@NUS 26

Narrow understanding of “trust” a priori value is important Computing trust:• Compute event-event similarity

Trust propagation• Biased PageRank algorithm

• Trust vectors are row normalized

April 18, 2023 27@NUS

Activity based trust

activity matrix apriori

(1 ) k t = A t + p

The recommendation algorithm

April 18, 2023@NUS 28

The framework is event centric We know:

How to combine the three?

April 18, 2023 29@NUS

A review of what we know

similarity co-occurrence trust vectors

global personal social

1. Compute the social network trust vector (t) for the current user.

2. Compute the trusted, global co-occurrence matrix, for all tuples.

3. Iterate:

April 18, 2023 30@NUS

details

1

( , ) ( ) ( , ),N

k kc c

i

a b t i a b

M M

who where whatwhen image event

query

,

,c

s

y M x q

x M y q

Experiments

April 18, 2023@NUS 31

Developed and event based archival system

8 graduate students 58 events, 250 images,

over two weeks SVM – baseline

comparison Two cases• Uniform trust (global)• Personal trust

April 18, 2023 32@NUS

Details

Training is difficult – very small pool.• Modified bagging strategy • Train five symmetric classifiers• Pick one which maximizes the F-score

April 18, 2023 33@NUS

SVM training

Global Case:• 31 classifiers (who:8, when: 6, where: 10, what: 7)• Minimum number of images: 10 • Tested on 50 images (why?)

April 18, 2023 34@NUS

Uniform trust

Facets SVM CM (uniform)

H M X U H M

Who 13 23 5 9 22 28

When 11 20 6 13 24 26

Where 12 19 3 16 23 27

What 13 21 8 8 31 19

Event 10 12 22 6 22 28

H Hits

M Misses

X Unknown

U Undecidable

Trained classifiers per person• Very small pool• Min images – 5• 28 classifiers (who:9, when: 4, where: 6, what: 9)

April 18, 2023 35@NUS

Personal Network

Facets SVM CM (network)

H M X U H M

Who 45 81 62 62 183 67

When 51 96 73 30 167 83

Where 62 76 59 53 179 71

What 72 89 23 66 204 46

Events 0 0 250 0 153 97

H Hits

M Misses

X Unknown

U Undecidable

April 18, 2023 36@NUS

Positive examplesSVM ‘sky diving’

Social Network based method ‘fun’

The Sum of Partial ObservationsBeyond web 2.0 hype

April 18, 2023@NUS 37

Which media object summarizes “my trip to Singapore?”

April 18, 2023 38@NUS

Experiential fragments

April 18, 2023 39@NUS

A reconsideration of a traditional idea

@NUS

The Creation of participatory knowledge

April 18, 2023 40

Conclusions

April 18, 2023@NUS 41

An event based annotation system• Media are event meta-data• Issues: learnability, scalability, context

Employ three kinds of knowledge• Global – conceptnet, image similarity• Personal – statistical co-occurrence • Social – trust

Recommendations• Employ iterative schemes (HITS / PageRank)

Results:• Outperform SVM in small pools

April 18, 2023 42@NUS

summary

Power law tag distribution• Data pool will remain small for most tags• Fundamental issue

Participatory knowledge is powerful – trust within context is important issue.

Future work: • Careful math analysis of coupling equations• Event structure / relationships need to be incorporated • Multi-source (email / Calendar / IM / blogs)

integration.

April 18, 2023 43@NUS

Conclusions

Thanks!Esp. Dick Bulterman, Mohan

April 18, 2023@NUS 44