Dmitry Ustalov — TagBag: Annotating a Foreign Language Lexical Resource with Pictures

Annotating a Foreign Language Lexical

Resource with Pictures

Dmitry UstalovIMM UB RAS / UrFUYekaterinburg, Russia

Outline

• Introduction•Related Work•Approach•Evaluation•Results•Discussion•Conclusion

2

Introduction

•The problem of mapping images to the word senses is quite important:•multimedia search,• text illustration,• quality assessment.

• It is also interesting to assess the Yet Another RussNet lexical resource.(Braslavski et al, 2014).

3

Related Work

• PicNet, a proprietary resource(Mihalcea & Leong, 2008).• ImageNet annotates WordNet with

pictures & bounding boxes(Deng et al., 2009).• Intersection with WordNet.ru is negligible.

• ImageCLEF creates software and datasets for image indexing (Mül̈ler et al., 2010).

4

Related Work: Flickr

•Single-query image retrieval(Reiter et al., 2007).•Semantic Web-based approach

(Trojahn et al., 2008).•Wikipedia-based approach

(Stampouli et al., 2010).•Flickr tags with visual saliency of images (Jiang et al., 2014).

5

Problem

Given an annotated image I, a bilingual dictionary B, and a lexical resource S, produce a mapping Is.

“cat”, “tomcat”, “kitten” →«кот, кошка, котёночек»

6

TagBag: Assumptions

•The most image tags are nouns.•Tags may be polysemous and the redundant tags may be present.• “crane” is «журавль» or «кран»?

•The image has a “main” object.

7

TagBag

•Tag. Initialize an empty vector.• Iterate over image tags and retrieve all

the translations for each tag.• Add each occurrence to a dimension.

•Bag. Prune that vector.• Remove the low frequency dimensions

with the cut-off value.• Return the resulting vector.

8

TagBag: Pseudocode

9

Evaluation

•The present approach is pretty simple.Let’s evaluate it empirically.•Take the top 1500 English nouns and search for Flickr photos.

http://www.talkenglish.com/Vocabulary/Top-1500-Nouns.aspx

•Get the V.K. Mueller’s dictionary.http://ustalov.imm.uran.ru/pub/mueller.tar.gz

10

Experimental Setup

•Yet Another RussNet (CC BY-SA).http://russianword.net/

•Similarity measures: • cosine similarity,• Jaccard index.

•Ask threeannotators tosubmitjudgements.

11

призрак, тень, намёк

12https://www.flickr.com/photos/127324269@N03/16217604730

труд, работа, занятие


мужчина, парень, юноша


футбол


пища, провизия, питание, корм


Results

•The accuracy is moderately high and the agreement level is good.•Both measures demonstrate the same performance.

17

http://ustalov.imm.uran.ru/pub/tagbag-aist.tar.gz

Discussion

•Some mappings are the same w.r.t. the similarity measures and 13 of 43 of these mapping are wrong.•Three sources of errors:• sloppy image tags (7 of 13),• actual mapping errors (3 of 13),• batch uploads (3 of 13).

18

Conclusion

•TagBag is an unsupervised approach for mapping images to synsets.•The performance depends both on image tags and ontology bias.•Visual saliency and spam filtering may increase the quality.

19

Further Work

20

Thank you!

Dmitry Ustalova post-graduate student @IMM UB RAS, Yekaterinburg, Russia.

https://ustalov.name/[email protected]

The present work is supported by the Russian Foundation for the Humanities, project no. 13-04-12020.

21

Dmitry Ustalov — TagBag: Annotating a Foreign Language Lexical Resource with Pictures

Science

Transcript of Dmitry Ustalov — TagBag: Annotating a Foreign Language Lexical Resource with Pictures