MediaEval 2012 Placing Task Overview
-
Upload
adam-rae -
Category
Technology
-
view
1.305 -
download
0
Transcript of MediaEval 2012 Placing Task Overview
Placing TaskOrganisers: Adam Rae (Yahoo! Research)
Pascal Kelm (Technische Universität Berlin)
Smile!
?
!
Task Description
• Given a video, how accurately can it be placed on a map and be given latitude and longitude coordinates?
METADATA
Task Overview
• Automatic location annotation of online videos
• 7 teams submitted results (17% up)– 5 veterans– 2 new participants
• First year for code sharing– GitHub (currently)
Data
• Provided– Textual metadata: tags, titles, descriptions– Visual: 9 visual features extracted for key frames
every 4 seconds– Additional media: images with textual and
visual feature data• Available (external)– Up to the participant, but controlled according to
run submission
Data
• Training– 15,563 videos (combination of last year’s training
and test data)– 3,185,258 additional Flickr images
• Test– 4,182 videos
2012 Training
2011 Training
2010 Training
2010 Test2011 Test
2012 Test
Evaluation
• Take the latitude + longitude suggested by participants for each video
• Compute Haversine distance between that and the ‘true’ location
• We group results into buckets of increasing radii, e.g. 1km, 10km, 20km, etc.
Overall Best Results
CEALIST
IRISA
UNICAMP
GENT
TUB
ICSI
TUD
0% 5% 10% 15% 20% 25% 30%
Percentage of correct locations @ 1km
Organiser-connected team
1 10 100 1000 10000 1000000
500
1000
1500
2000
2500
3000
3500
4000
4500
Only Restriction: No new material, gazetteer permitted
ICSI TUD UG-CUUNICAMP CEA_LIST London Baseline
Distance from Ground Truth
Corr
ect T
est V
ideo
s
1 10 100 1000 10000 1000000
500
1000
1500
2000
2500
3000
3500
4000
4500
Restriction: Visual Only
CEA_LIST ICSI IRISA UG-CU UNICAMP TUB
Distance from Ground Truth
Corr
ect T
est V
ideo
s
Detected trends and activity of note
• What classes of approaches were taken (has this change since last year?)– Textual, visual– Graph modelling– User modelling– …combinations of above
• Challenging Assumptions– Spatial locality visual stability?
• Absolute performance lower than last year – but…– Different data set– Less textual metadata in general
Future of the task
• Still room for improvement• Still a valuable task?• Standard of science improving
• Need new organisers! Talk to Pascal and me