LAPI @ 2017 Retrieving Diverse Social Images Task:A Pseudo-Relevance Feedback Diversification Perspective

Bogdan Boteanu, Mihai Gabriel Constantin, Bogdan IonescuLAPI - University ”Politehnica” of Bucharest, 061071, RomaniaEmail: {bboteanu,mgconstantin,bionescu}@alpha.imag.pub.ro

UniversityPOLITEHNICAof Bucharest

§ HC pseudo-relevance feedback (HC-RF)

1. selection of positive and negative examples

2. hierarchical clustering & pruning scheme withfeedback determined automatically from initial data

3. diversification is achieved by traversing HC image clusters with respect to the Flickr initial ranking

Proposed approach (1)

MediaEval 2017, Dublin, Ireland 1/10

1. Selection of positive and negative examples

Proposed approach (2)

I 1 I 2 I N

Image Database(Flickr’s rank)

I 3 I N-1 Np+Nn <= N…Nn

un-relevantNp

relevant

MediaEval 2017, Dublin, Ireland 2/10

2. HC clustering and pruning

Proposed approach (3)

Hierarchical Clustering

cut point

I 1

I 2

Class 1I N

I N-1

Class k(un-relevant)

I 3…

MediaEval 2017, Dublin, Ireland 3/10

3. Diversification

Proposed approach (4)

I 1 I 3 … I 4

Class 1 Class 2 Class n

I 9

I 8

I 2

I 7

I 5

I 15

... ... ...1

2

4

9

3

158 7

5Output

MediaEval 2017, Dublin, Ireland 4/10

Parameter tuning

§ positive examples (Np): 100 – 280 with a step of 20

§ negative examples (Nn): 0 – 20 with a step of 10

§ inconsistency coefficient (Nc - no. of classes): 0.5 – 1.3 with a step of 0.2

ü Best combination of Np-Nn-Nc (highest F1@20)

MediaEval 2017, Dublin, Ireland 5/10

Results - devsetRuns P@20 CR@20 F1@20

1 . all visual 0.575 0.3969 0.4473

2. all text 0.575 0.3969 0.4473

3. all vis - all text 0.6136 0.4234 0.4773

4. CNN 0.575 0.3969 0.4473

5. cred. 0.575 0.3969 0.4473

Flickr init. res. 0.5864 0.3646 0.42277

MediaEval 2017, Dublin, Ireland 6/10

Results - testsetRuns P@20 CR@20 F1@20

1 . all visual 0.6333 0.5791 0.5753

2. all text 0.6214 0.5794 0.5733

3. all vis - all text 0.6196 0.5729 0.5741

4. CNN 0.5845 0.5216 0.5253

5. cred. 0.6018 0.6045 0.5777

MediaEval 2017, Dublin, Ireland 7/10

Results - Visual Example (Flickr Initial)

MediaEval 2017, Dublin, Ireland 8/10

Easter Eggs

P@20=0.75 CR@20=0.4 F1@20=0.52

X X XX X

MediaEval 2017, Dublin, Ireland 9/10

Results - Visual Example (best run)Easter Eggs

P@20=0.9 CR@20=0.53 F1@20=0.67 X X

Conclusions

• credibility information was useful in the context of overalldiversification (Run5 - CR@20 = 0.6045), with more than 2% overother types of descriptors

• in terms of F1 metric score, the use of credibility information, (Run5- F1@20 = 0.5777), allows for better performance over visual andtextual descriptors by more than 3% and by more than 5% overCNN descriptors

MediaEval 2017, Dublin, Ireland 10/10

Thank you!