RECOD @ Placing Task of MediaEval 2015
L. T. Li1, J. A. V. Muñoz1, J. Almeida1,3,R. T. Calumby1,4, O. A. B. Penatti1,2, I. C. Dourado1,
K. Nogueira6, P. R. Mendes Júnior1,D. C. G. Pedronette1,5, J. A. dos Santos6,
M. A. Gonçalves6, and R. S. Torres1
Daniel MoreiraOn behalf of the authors.
1. UNICAMP – 2. UNIFESP – 3. SAMSUNG – 4. UEFS – 5. UNESP – 6. UFMG– BRAZIL –
2015 Participation
● We focused on the Localization subtask;● Innovations concerning the last year (2014)
– Rank agreggation based on Genetic Programming;
– Geocoding improvement with Ranked List Density Analysis (RLDA).
Submitted Runs
● Run 1Textual Features with Genetic Programming– Image and Video
Descriptors
BM25*TF-IDF*IBS*LMD*
Ranks
Ranked list 1Ranked list 2Ranked list 3Ranked list 4
Rank Aggregation
Single ranked list(GP-Agg - based)
* All from Lucene package (http://lucene.apache.org/core/)
GP-Agg Framework
Parameter Value
Number of generations
30
Genetics operators
Reproduction, Mutation, Crossover
Fitness functions
FFP1, WAS, MAP, NDCG
Rank Agg. methods
CombMAX, CombMIN, CombSUM, CombMED, CombANZ, CombMNZ,
RLSim, BordaCount, RRF, MRA
Submitted Runs
● Run 1Textual Features with Genetic Programming– Image and Video
Descriptors Ranks
Ranked list 1Ranked list 2Ranked list 3Ranked list 4
Rank Aggregation
Single ranked list(GP-Agg - based)
BM25*TF-IDF*IBS*LMD*
* All from Lucene package (http://lucene.apache.org/core/)
Submitted Runs
● Run 2Visual Features with RLDA– Image
Descriptor
BIC
Rank
BICRanked list
GeocodingImprovement
Improved geocode(top 100 RLDA-based)
Ranked List Density Analysis
Ranked list 1~N
ID1 LAT1 LONG1ID2 LAT2 LONG2… … ...IDX LATX LONGX
Top X items' lat/long defined as points of a OPF cluster
Node v: (lat1,long1)
edge (v, u): connect k-nn d(v, u)
d(v, u): geo-distance between v and uk=3
u: (lat2,long2)
Submitted Runs
● Run 2Visual Features with RLDA– Image
Descriptor
BIC
Rank
BICRanked list
Improved geocode(top 100 RLDA-based)
GeocodingImprovement
Submitted Runs
● Run 2Visual Features with RLDA– Video
Descriptors
LIRE 1...LIRE nHMP
Ranks
Ranked list 1...Ranked list nRanked list n+1
Rank Aggregation
Single ranked list(GP-Agg - based)
Improved geocode(top 100 RLDA-based)
GeocodingImprovement
Submitted Runs
● Run 3Multimodal Solution with RLDA– Image
Descriptors
BM25TF-IDFBSLMDBIC...SCD
Ranks
Ranked list 1Ranked list 2 Ranked list 3Ranked list 4Ranked list 5…Ranked list 8
Rank Aggregation
Single ranked list(GP-Agg - based)
Submitted Runs
● Run 3Multimodal Solution with RLDA– Video
Descriptors
BM25TF-IDFBSLMDHMP...MFCC
Ranks
Ranked list 1Ranked list 2 Ranked list 3Ranked list 4Ranked list 5…Ranked list n
Rank Aggregation
Single ranked list(GP-Agg - based)
Submitted Runs
● Run 4Textual with RLDA– Image and Video
Descriptors
BM25TF-IDFBSLMD
Ranks
Ranked list 1Ranked list 2 Ranked list 3Ranked list 4
Rank Aggregationand Improvement
Single ranked list(top 5 RLDA-based)
Results – 2015 Global
Run 1
Run 2
Run 3
Run 4
0 5 10 15 20 25 30 35 40 45 50
0,15
0
0,14
0,12
0,54
0,01
0,53
0,62
5,49
0,09
5,35
6,44
19,75
0,44
19,11
21,74
36,6
1,99
35,31
38,38
44,89
3,57
43,26
46,91
58,97
20,38
57,67
63,22
1m10m100m1km10km100km1,000km
GP-Agg Combined Textual
GP-Agg Combined Non-textual
GP-Agg Multimodal
Textual RLDL
Results – Median Distance
With Metadata Runs
Run4
Run3
Run1
50 150 250 350 450 550 650 750
309.86
394.89
196.01
Conclusions
● GP-Agg automatically combines lists and aggregation functions.
● Top-N RLDA improves even GP-Agg results.● RLDA was better (see Run 4) than using GP-
Agg alone.● There is room for improving the GP-Agg
approach.
Future Work
● Develop other fitness fuctions in the GP-Agg approach.
● Use more visual descriptors.● Evaluate different clustering strategies.
Acknowledgments
● MediaEval 2015● FAPESP● CNPq ● CAPES● Samsung
Thank You!
{lintzyli,pedro.mendes,luis.pereira,rtorres}@ic.unicamp.br,[email protected],[email protected],
[email protected],[email protected],[email protected],[email protected],
{keiller.nogueira,jefersson, mgoncalv}@dcc.ufmg.br
L. T. Li, J. A. V. Muñoz, J. Almeida,R. T. Calumby, O. A. B. Penatti, I. C. Dourado,
K. Nogueira, P. R. Mendes Júnior,D. C. G. Pedronette, J. A. dos Santos,
M. A. Gonçalves, and R. S. Torres
Support Slides
Run1 Run3 Run4 Min. : 0.000 Min. : 0.000 Min. : 0.000 1st Qu.: 1.897 1st Qu.: 2.124 1st Qu.: 1.482 Median : 309.865 Median : 394.889 Median : 196.008 Mean : 2913.598 Mean : 2976.632 Mean : 2483.614 3rd Qu.: 5573.930 3rd Qu.: 5766.894 3rd Qu.: 3798.622 Max. :19959.808 Max. :19959.808 Max. :19954.130
Basic analysis of distances (km) in Test set:from Predicted to Expected lat/long
Run2 Min. : 0 1st Qu.: 1240 Median : 5883 Mean : 5597 3rd Qu.: 8637 Max. :19972
Videos-only Test Results (%)
Run 1: GP-Agg TextualRun 2: GP-Agg only visual (HMP+ all Lire) + RLDA (top100) Run 3: GP-Agg multimodal (text, visual, audio)Run 4: BM25_RLDA (top 5)
Run 1 Run 2 Run 3 Run 4
1m 0.08 0 0.08 0.06
10m 0.4 0 0.37 0.41
100m 5.46 0.01 5.13 5.79
1km 17.62 0.02 16.74 17.89
10km 32.44 0.11 32.1 32.24
100km 40.27 3.8 39.69 39.92
1,000km 54.13 20.39 53.67 55.68
10,000km 90.57 91.97 91.5 93.16
Video-only Summary
Run1 Run2 Run3 Min. : 0.000 Min. : 0.05 Min. : 0.000 1st Qu.: 2.914 1st Qu.: 1304.13 1st Qu.: 3.138 Median : 619.747 Median : 6351.71 Median : 660.864 Mean : 3191.163 Mean : 5709.17 Mean : 3158.724 3rd Qu.: 6035.939 3rd Qu.: 8463.49 3rd Qu.: 6154.147 Max. :19656.624 Max. :19596.38 Max. :19524.628
Run4 Min. : 0.000 1st Qu.: 2.992 Median : 547.648 Mean : 2879.899 3rd Qu.: 5548.780 Max. :19656.624
GP-Agg Individual example – Run 1
Input: bm25 (description, fusion, tags, title), tf-idf(description, fusion, tags, title), lmd_fusion, ibs_fusion
Individual:
CombSUM( MRA( CombMNZ( RRF(ibs_fusion,lmd_fusion),CombMNZ(td-idf_fusion, bm25_fusion), RLSim(lmd_fusion, bm25_tags, tf-idf_fusion)), CombMNZ(CombSUM(tf-idf_tags, tf-idf_fusion, tf-idf_tags), CombMIN(tf-idf_description, bm25_description, tf-idf_tags), RLSim(bm25_title, tf-idf_title, tf-idf_title)), RRF(CombMAX(bm25_fusion, bm25_tags), RRF(tf-idf_fusion, ibs_fusion, tf-idf_fusion), BordaCount(tf-idf_fusion, bm25_fusion, tf-idf_tags))), RLSim(CombSUM(CombSUM(tf-idf_tags, tf-idf_tags, lmd_fusion), bm25_fusion), bm25_fusion))
Visual Features (HMP): Extracting
● Histograms of Motion Patterns● Keyframes: Not used● Applying an algorithm to compare video
sequence(1)partial decoding; (2)feature extraction; (3)signature generation.
“Comparison of video sequences with histograms of motion patterns”, J. Almeida et al. ICIP, 2011.
Visual Features (HMP): overview
[Almeida et al., Comparison of video sequences with histograms of motion patterns. ICIP 2011]
HMP: Comparing Video
OPF Density formula
● d(s,t) distânce from s to t (used haversine dist.)● A(s): list of adjacency of s. ● directed graph
Top Related