AUTOMATIC KEYFRAME SELECTION

BASED ON

MUTUAL REINFORCEMENT ALGORITHM

C. Ventura, X. Giró-i-Nieto, V. Vilaplana et al

1. Introducing the problem

Automatic selection of the representative

keyframe

2

1.1. What is the application?3

1.2. Current implementation4

ARBITRARY SAMPLING

MANUAL SELECTION

BY

PROFESSIONAL

2. Designing the system

2 scenarios:

Intra-clip mode

Inter-clip mode

Database

Textual search

to retrieve

related videos

5

2.1. General scheme6

Reranking

Frame

extraction

visual

features

Textual

search

Similarity

graph

Mutual

Reinforcement

Inter-clip

mode

21

3

12

3

2.2. Intra-clip mode

Mutual Reinforcement Algorithm (Joshi04)

Gets the frame with maximum coverage

(Joshi04) D. Joshi et al. The story picturing engine: finding elite

images to illustrate a story using mutual reinforcement. In MIR ‘04

7

2.3. Inter-clip mode

Reranking (based on Liu11)

(Liu11) C. Liu et al. Query sensitive dynamic web

video thumbnail generation. In ICIP ‘11

8

INTRA INTER

INPUT FRAMESREPRESENTATIVE KEYFRAMES

FROM TEXTUAL SEARCHER

FRAME HAAR WAVELET MORPHOLOGY

2.4. Post-processing block

9 Text filtering

Goal: To avoid representative keyframes with

textual captions

3. Experiments

Qualitative evaluation

Quantitative evaluation

MOS Test

Experimental dataset

10

3.1. Qualitative evaluation

Intra-clip mode

11

3.1. Qualitative evaluation

Inter-clip mode

Ranking scores

after mutual

reinforcement

(INTRA-CLIP

MODE)

Representative

keyframes of the

retrieved videos

12

3.1. Qualitative evaluation

Inter-clip mode

Final ranking scores after reranking:

13

3.2. Quantitative evaluation

MOS (Mean Opinion Score) test

Performed by TVC professionals

Scores

14

EXCELLENTGOODBAD

NON

ACCEPTABLEACCEPTABLE

NEWS DOMAIN

3.2. Quantitative evaluation

Database

15

POLITICS

INTERNATIONAL

ECONOMY

MORNING SHOW DOMAIN

INTERVIEW

DISCUSSION

3.2. Quantitative evaluation

MOS test

4 different approaches

Intra-clip

Inter-clip

Random

Current

16

3.2. Quantitative evaluation17

NEWS

MORNING

SHOW

3.2. Quantitative evaluation18

NEWSMORNING

SHOW

3. Experiments

Database and results are available on:

imatge.upc.edu

19

4. Conclusions

Keyframe selection based on mutual reinforcement algorithm To get the frame with maximum coverage within the

video in the intra-clip approach

Inter-clip approach Textual similarity to retrieve related videos

Linear fusion to get the new ranking scores

MOS test The semi-manual system (from TVC) can be

replaced by the automatic approach.

Inter-clip approach outperforms intra-clip in controlled environments.

20

21

2.3. Inter-clip mode

Textual search

2 modalities:

Textual searcher binary

TF-IDF descriptors

22