The CLEF 2005 Cross-Language Image Retrieval Track
Organised by
Paul Clough, Henning Müller, Thomas Deselaers, Michael Grubinger, Thomas Lehmann, Jeffery Jensen and William Hersh
22/09/05 ImageCLEF: cross-language image retrieval at CLEF2005
Overview
• Image Retrieval and CLEF
• Motivations
• Tasks in 2005• Ad-hoc retrieval of historic photographs and medical
images
• Automatic annotation of medical images
• Interactive task
• Summary and future work
22/09/05 ImageCLEF: cross-language image retrieval at CLEF2005
Image Retrieval and CLEF
• Cross-language image retrieval• Images often accompanied by text (used for retrieval)
• Began in 2003 as pilot experiment
• Aims of ImageCLEF• Investigate retrieval combining visual features and
associated text
• Promote the exchange of ideas
• Provide resources for IR evaluation
22/09/05 ImageCLEF: cross-language image retrieval at CLEF2005
Motivations
• Image retrieval a good application for CLIR• Assume images are language-independent
• Many images have associated text (e.g. captions, metadata, Web page links)
• CLIR has potential benefits for image vendors and users
• Image retrieval can be performed using• Low-level visual features (e.g. texture, colour and shape)
• Abstracted features expressed using text
• Combining both visual and textual approaches
22/09/05 ImageCLEF: cross-language image retrieval at CLEF2005
ImageCLEF 2005• 24 participants from 11 countries
• Specific domains and tasks• Retrieval of historic photographs (St Andrews)
• Retrieval and annotation of medical images (medImageCLEF and IRMA)
• Additional co-ordinators• William Hersh and Jeffrey Jensen (OHSU)
• Thomas Lehmann and Thomas Deselaers (Aachen)
• Michael Grubinger (Melbourne)
• Links with MUSCLE NoE including pre-CLEF workshop• http://muscle.prip.tuwien.ac.at/workshops.php
22/09/05 ImageCLEF: cross-language image retrieval at CLEF2005
Ad-hoc retrieval from historic photographs
Paul Clough (University of Sheffield)
Michael Grubinger (Victoria University)
22/09/05 ImageCLEF: cross-language image retrieval at CLEF2005
From: St Andrews Library historic photographic collectionhttp://specialcollections.st-and.ac.uk/photo/controller
イングランドにある灯台の写真
Изображения английских маяков
Fotos de faros ingleses
Pictures of English lighthouses
Kuvia englantilaisista majakoista
Bilder von englischen Leuchttürmen
انجليزيه لمنارات صور
St Andrewsimage collection
22/09/05 ImageCLEF: cross-language image retrieval at CLEF2005
Topics• 28 search tasks (topics)
• Consist of title, narrative and example images
• Topics more general than 2004 and more “visual” • e.g. waves breaking on beach, dog in sitting position
• Topics translated by native speakers• 8 languages for title & narrative (e.g. German, Spanish, Chinese,
Japanese)• 25 languages for title (e.g. Russian, Bulgarian, Norwegian,
Hebrew, Croatian)
• 2004 topics and qrels used as training data
22/09/05 ImageCLEF: cross-language image retrieval at CLEF2005
Relevance judgements
• Staff from Sheffield University were assessors
• Assessors judged topic pools• Top 50 images from all 349 runs
• Average of 1,376 images per pool
• 3 assessments per image (inc. topic creator)
• Ternary relevance judgements
• Qrels: images judged as relevant/partially relevant by topic creator and at least one other assessor
22/09/05 ImageCLEF: cross-language image retrieval at CLEF2005
Submissions & Results (1)
11 groups (5 new*)CEA*NII*AlicanteCUHK*DCUGenevaIndonesia*MiracleNTUJaen*UNED
Dimension Type #Runs (%) Avg. MAP
Language English 119 (34%) 0.2084
Non-English 230 (66%) 0.2009
Run type Automatic 349 (100%) 0.2399
Feedback (QE) Yes 142 (41%) 0.2399
No 207 (59%) 0.2043
Modality Image 4 (1%) 0.1500
Text 318 (91%) 0.2121
Text + Image 27 (8%) 0.3086
Initial Query Image 4 (1%) 0.1418
Title 274 (79%) 0.2140
Narrative 6 (2%) 0.1313
Title + Narr 57 (16%) 0.2314
Title + Image 4 (1%) 0.4016
All 4 (1%) 0.3953
22/09/05 ImageCLEF: cross-language image retrieval at CLEF2005
Submissions & Results (2)Language #Runs Max. MAP Group Initial
QueryFeedback Modality
English 70 0.4135 CUHK Title+img Yes Text+img
Chinese (trad.) 8 0.3993 NTU Title+narr Yes Text+img
Spanish (Lat. Am.)
36 0.3447 Alicante/Jaen
Title Yes Text
Dutch 15 0.3435 Alicante/Jaen
Title Yes Text
Visual 3 0.3425 NTU Visual Yes Image
German 29 0.3375 Alicante/Jaen
Title Yes Text
Spanish (Euro.) 28 0.3175 UNED Title Yes Text
Portuguese 12 0.3073 Miracle Title No Text
Greek 9 0.3024 DCU Title Yes Text
French 17 0.2864 Jaen Title+narr Yes Text
Japanese 16 0.2811 Alicante Title Yes Text
Russian 15 0.2798 DCU Title Yes Text
Italian 19 0.2468 Miracle Title Yes Text
22/09/05 ImageCLEF: cross-language image retrieval at CLEF2005
Submissions & Results (3)Language #Runs Max. MAP Group Initial
QueryFeedback Modality
Chinese (simpl.) 21 0.2305 Alicante Title Yes Text
Indonesian 9 0.2290 Indonesia Title No Text+img
Turkish 5 0.2225 Miracle Title No Text
Swedish 7 0.2074 Jaen Title No Text
Norwegian 5 0.1610 Miracle Title No Text
Filipino 5 0.1486 Miracle Title No Text
Polish 5 0.1558 Miracle Title No Text
Romanian 5 0.1429 Miracle Title No Text
Bulgarian 2 0.1293 Miracle Title No Text
Czech 2 0.1219 Miracle Title No Text
Croatian 2 0.1187 Miracle Title No Text
Finnish 2 0.1114 Miracle Title No Text
Hungarian 2 0.0968 Miracle Title no Text
22/09/05 ImageCLEF: cross-language image retrieval at CLEF2005
Summary
• Most groups focused on text retrieval• Fewer combined runs than 2004
But still gives highest average MAP
• Translation main focus for many groups13 languages have at least 2 groups
• More use of title & narrative than 2004• As Relevance feedback (QE) improves results
• Topics still dominated by semantics• But typical of searches in this domain
22/09/05 ImageCLEF: cross-language image retrieval at CLEF2005
Ad-hoc medical retrieval task
Henning Müller (University Hospitals Geneva)
William Hersh, Jeffrey Jensen (OHSU)
22/09/05 ImageCLEF: cross-language image retrieval at CLEF2005
Collection
• 50,000 medical images• 4 sub-collections with heterogeneous annotation
• Radiographs, photographs, Powerpoint slides and illustrations
• Mixed languages for annotations (French, German and English)
• In 2004 only 9,000 images available
22/09/05 ImageCLEF: cross-language image retrieval at CLEF2005
Search topics• Topics based on 4 axes
• Modality (e.g. x-ray, CT, MRI)• Anatomic region shown in image (e.g. head, arm)• Pathology (disease) shown in image• Abnormal visual observation (e.g. enlarged heart)
• Different types of topic identified from survey• Visual (11) – visual approaches only expected to perform well• Mixed (11) – text and visual approaches expected to perform well• Semantic (3) – visual approaches not expected to perform well
• Topics consist of annotation in 3 languages and 1-3 query images
22/09/05 ImageCLEF: cross-language image retrieval at CLEF2005
An example (topic # 20 - mixed)
Show me microscopic pathologies of cases with chronic myelogenous leukemia.
Zeige mir mikroskopische Pathologiebilder von chronischer Leukämie.
Montre-moi des images de la leucémie chronique myélogène.
22/09/05 ImageCLEF: cross-language image retrieval at CLEF2005
Relevance assessments
• Medical doctors made relevance judgements• Only one per topic for money and time constraints
Some additional to verify consistency
• Relevant/partially relevant/non relevantFor ranking only relevant vs. non-relevant
• Image pools created from submissions• Top 40 images from 134 runs
• Average of 892 images per topic to assess
22/09/05 ImageCLEF: cross-language image retrieval at CLEF2005
Submissions
• 13 groups submitted runs (24 registered)• Resources very interesting but lack of manpower
• 134 runs submitted
• Several categories for submissions• Manual vs. Automatic
• Data source usedVisual/textual/mixedAll languages could be used or a single one
22/09/05 ImageCLEF: cross-language image retrieval at CLEF2005
Results (1)• Mainly automatic and mixed
submissions• some further to be
classified as manual
• Large variety of text/visual retrieval approaches• Ontology-based
• Simple tf/idf weighting
• Manual classification before visual retrieval
Query types Automatic Manual
Visual 28 3
Textual 14 1
Mixed 86 2
22/09/05 ImageCLEF: cross-language image retrieval at CLEF2005
Results (2) – highest MAP
Query types Automatic Manual
Visual I2Rfus.txt
0.146
i2r-vk-avg.txt
0.092
Textual IPALI2R_Tn
0.208
OHSUmanual.txt
0.212
Mixed IPALI2R_TIan
0.282
OHSUmanvis.txt
0.160
22/09/05 ImageCLEF: cross-language image retrieval at CLEF2005
Average results per topic type
0.0000
0.1000
0.2000
0.3000
0.4000
0.5000
0.6000
0.7000
Avera
ge
Visual
Mixe
d
Seman
tic
Weight
ed
P10Avg
Topic Type
MA
P
Auto-Mixed
Auto-Text
Auto-Vis
Man-Mixed
Man-Text
Man-Vis
22/09/05 ImageCLEF: cross-language image retrieval at CLEF2005
Summary
• Text-only approaches perform better than image-only• But some visual systems have high early precision• Depends on the topics formulated
Visual systems very bad on semantic queries
• Best overall systems use combined approaches• GIFT as a baseline system used by many participants
and still best visual completely automatic
• Few manual runs
22/09/05 ImageCLEF: cross-language image retrieval at CLEF2005
Automatic Annotation Task
Thomas Deselaers, Thomas Lehmann
(RWTH Aachen University)
22/09/05 ImageCLEF: cross-language image retrieval at CLEF2005
Automatic annotation
• Goal• Compare state-of-the-art classifiers for medical image
annotation task
• Purely visual task
• Task• 9,000 training & 1,000 test medical images from Aachen
University Hospital
• 57 classes identifying modality, body orientation, body region and biological system (IRMA code)
e.g. 01: plain radiography, coronal, cranuim, musculosceletal system
• Classes in English and German and unevenly distributed
22/09/05 ImageCLEF: cross-language image retrieval at CLEF2005
Example of IRMA code
• Example: 1121-127-720-500Example: 1121-127-720-500• rradiography, plain, analog, overviewadiography, plain, analog, overview• ccoronal, AP, supineoronal, AP, supine• aabdomen, middlebdomen, middle• uuropoetic systemropoetic system
22/09/05 ImageCLEF: cross-language image retrieval at CLEF2005
Example Images
http://irma-project.org
22/09/05 ImageCLEF: cross-language image retrieval at CLEF2005
Participants
• Groups• 26 registered
• 12 submitted runs
• Runs• In total 41 submitted
• CEA (France)
• CINDI (Montreal,CA)
• medGift (Geneva, CH)
• Infocomm (Singapore, SG)
• Miracle (Madrid, ES)
• Umontreal (Montreal, CA)
• Mt. Holyoke College (Mt. Hol., US)
• NCTU-DBLAB (TW)
• NTU (TW)
• RWTH Aachen CS (Aachen, DE)
• IRMA Group (Aachen, DE)
• U Liège (Liège, BE)
22/09/05 ImageCLEF: cross-language image retrieval at CLEF2005
Results
...
...• Baseline error rate: 36.8%
22/09/05 ImageCLEF: cross-language image retrieval at CLEF2005
Conclusions
• Continued global participation from variety of research communities
• Improvements in ad-hoc medical task• Realistic topics
• Larger medical image collection
• Introduction of medical annotation task
• Overall combining text and visual approaches works well for ad-hoc task
22/09/05 ImageCLEF: cross-language image retrieval at CLEF2005
ImageCLEF2006 and beyond …
22/09/05 ImageCLEF: cross-language image retrieval at CLEF2005
ImageCLEF 2006 …
• New ad-hoc• IAPR collection of 25,000 personal photographs
• Annotations in English, German and Spanish
• Medical ad-hoc• Same data; new topics
• Medical annotation• Larger collection; more fine-grained classification
• New interactive task• Using Flickr.com … more in iCLEF talk
22/09/05 ImageCLEF: cross-language image retrieval at CLEF2005
… and beyond
• Image annotation task• Annotate general images with simple concepts
• Using the LTU 80,000 Web images (~350 categories)
• MUSCLE collaboration• Create visual queries for ad-hoc task (IAPR)
• Funding workshop in 2006
• All tasks involve cross-language in some way
Top Related