Noboru Babaguchi
Osaka University
Joint Work with Prof. N. Nitta
ICME2013 Co-located WS MMIX13, Keynote San Jose, July 18, 2013
Definitions, Background, Related Work
Multimedia Remixing Support System
Video Clip Sequence Creation
Music Clip Selection
Shot Extraction
Conclusion and Future Work
From wikipedia… A remix is a song that has been edited to sound different from the original version. The person who remixed it might have changed the pitch of the singers' voice, changed the tempo and speed and has made the song shorter or longer, or instead of hearing just one person singing they might have duplicated the voice to make it sound like two people are singing, or make the voice echo.
Remixes should not be confused with edits, which usually involve shortening a final stereo master for marketing or broadcasting purposes. … A remix song recombines audio pieces from a recording to create an altered version of the song.
In recent years the concept of the remix has been applied analogously to other media. …. Scary Movie series is famous for its comic remix of various well-known horror movies such as Ring, Scream, and Saw.
Video Remix: a video clip made by recombining various media components to create an altered version of the original videos.
Video transition effects (Cut, fade-in/out,
dissolve, etc.)
Audio clips (music, sound effects,
voices, etc.)
Original video clips
Video remixes (e.g. movie trailers)
Video clip selection & arrangement
Multimedia stream Combination
How can we create video remixes of good quality?
from “The School of Rock” (2003)
Semantic Aspect:
What should we present? (Semantic Content)
Highlights of Sports Games, etc.
Affective Aspect:
How should we present the video content?
(Aesthetic Compatibility, Film Syntax)
Commercial Films,Movie Trailers, etc.
How to arrange video clips or what music clip to augment to enhance the expressive quality
Two aspects in video remixing
Video Summarization
Video Remix
Scene-Music Relation
Shot-Scene Relation A sequence of L video shots
A sequence of D music clips
A video scene
Problem of Video Remixing
A music clip
= A sequence of D video scenes
An excerpt from a video clip
To maintain the feeling of continuity in a scene
Hitchcock[Girgensohn2001]
Template-based Editing[Davis2003]
Lazycut[Hua2005]
Emotion-based[Canini2010]
Video-Music Mixing [Mulhem2003][Hua2004][Wang2005][Yoon2009][Cristani2010]
Video clip selection and arrangement
Focused on how various types of video clips are arranged in sequence.
For example… • A scene has to have at least three video clips[Sundaram01]. • Two video shots of extremely different shot sizes should not be connected[Kumano02]. • The duration of a shot recorded with the camera fixed is up to 15 seconds[Kumano02].
Film Syntax
[Sundaram01] H. Sundaram, et al., “Condensing computable scenes using visual complexity and film syntax analysis,” Proc. ICME,
pp.389-392, 2001.
[Kumano02] M. Kumano, et al., “Video editing support system based on video content analysis,” Proc. ACCV, pp.628-633, 2002.
[Canini10] L. Canini, et al., “Interactive video mashup based on emotional identity,” Proc. European Signal Processing Conf., pp.1499-1503, 2010.
Aesthetic Compatibility
•Shots with similar emotional impact should be connected[Canini10].
Music clip selection
Focused on which types of music clips are mixed with video shots.
For example… • dynamic, motion, and pitch of image and audio streams coincide with each other[Mulhem03]. • novelty, velocity, and brightness of image and audio streams coincide with each other[Yoon09].
Aesthetic Compatibility
[Mulhem03] P. Mulhem, et al., “Pivot vector space approach for audio-video mixing,” IEEE Multimedia, 10(2), pp.28-40, 2003
[Yoon09] J.-C. Yoon, et al., “Automated music video generation using multi-level feature-based segmentation,” MTAP, 41(2), pp.197-214, 2009
[Cristani10] M. Cristani, et al., “Toward an automatically generated soundtrack from low-level cross-modal correlations for automotive scenarios,”
Proc. ACM Multimedia, pp.551-559, 2010
Determined heuristically • brightness of image and audio streams and rhythm of audio stream and optical flow in image stream coincide with each other[Cristani10]
Determined statistically
Multimedia Remixing Support System
It is difficult to explicitly defining the rules and know-how about how the video and music clips should be arranged, considering the aesthetic compatibility.
The rules and structures commonly used in professionally created examples can be modeled by standard machine learning techniques.
Non-professional users can be supported on their interface based on the models which implicitly describe shot-scene and scene-music relations considering aesthetic compatibility.
A Set of Video Remix Examples
Professionally Created Video Remixes
A Set of Video Remix Examples
Target: Remixing original video clips based on Examples
A Set of Music Clips
A Set of Original Video Clips
video remix
video remix
I) Video Clip Sequence Creation Interface
II) Music Clip Selection
III) Shot Extraction (Video and Music Synchronization)
User
・・・
・・・
A set of video clips:
A set of music clips:
A Set of Video Remix Examples
・・・ ・・
・・・
Video Remix Template
Shot
Scene
Video Clip Suggestions
N. Nitta and N. Babaguchi, “Example-based video remixing,” Multimedia Tools and Applications,
51(2), pp.649-673, 2011
N. Nitta and N. Babaguchi, “Example-based home video remixing,” Proc. ICME, 2011
Video Remix Examples
Symbol Sequence
Home (Personal) Videos
Video Clips
Segmentation
Suitability[Nitta2011] To Template
Perceived Quality[Tao2007]
B A B C G E
Template
Interface
Overview of Procedure I)
Template Generation
T. Mei, et al., "Home Video Visual Quality Assessment With Spatiotemporal Factors," IEEE Trans. Circuits and Systems for Video Technology, vol.17, no.6, pp.699-706, 2007.
Video Remix Examples
Slow Scene
Active Scene
HMM
Example-based Template Generation
Shot Length Brightness
Motion Intensity w/wo Camera Work
w/wo Human Objects
Low-level Features
Feature Extraction
・・・
Sequences of video shots
Shot
i h g f e d c b a
Symbolization
Symbol Sequence
Video Remix Template (New Symbol Sequence & State Sequence)
GA
A Sequence of L Shots
A Sequence of D Scenes
Video Clip 1 Video Clip 2 Video Clip 3
A Home Video
Suitability to Template 0.3 0.2 0.7
Perceived Quality 0.7 0.5 0.6
From Shot to Video Clip
Shots in target video are divided into video clips based on the camerawork
Video clip selection
Video Remix Template
Interface
3D book-style video clip presentation
Timeline Presentation
Suitability To Template
Perceived Quality
◎
× △
▲ spine
Fore edge Fore edge
Interface
Video remix examples: 61 action movie trailers
Video clips: 265 home (personal) video clips recording a sports field day held by a kindergarten
Subjective evaluation by 8 subjects
Compare with video clip sequence created by considering only the perceived quality of video clips
Subjective Score: 3.5 Subjective Score: 3
With Template* Without Template
* Selected video clips are shortened according to the template
Created Video Clip Sequence
Using action movie trailers as examples resulted in creating a sequence of many short video clips
N. Nitta and N. Babaguchi, “Example-based video remixing support system,”
Proc. ACM Multimedia, pp.563-572, 2011
Video Clip Sequence (Scene)
Overview of Procedure II) A Set of Video Remix Examples (Scenes)
A set of Music Clips
visually similar video remix examples
similar music clips
Evaluate the compatibility among video scenes and music clips by their distances in the video scene and music feature spaces
Learn non-linear mapping of music feature space so that the distances among video scenes and the mixed music clips would be correlated [Suzuki07]
Music Clip Feature Space
(Music Clips
Mixed to Example Video Scenes)
Video scene feature space
(Example Video Scenes)
Expected Music clip feature space
(Music Clips
Mixed to Example Video Scenes)
[Suzuki07] K. Suzuki, et al., “A similarity-based neural network for facial expression analysis,” Pattern Recognition Letters, 28(9), pp.1104-1111, 2007
Music Clip Selection
Video Scenes・・・Visual Features
Music Clips・・・Audio Features
[Zettl99]
Emotion-based Music Classification
[Zettl99] H. Zettl, “Sight Sound Motion: Applied Media Aesthetics,” Wadsworth Publishing, 1999
Consists of 2 Neural Networks
Input: Audio Features xAi and xB
i of Music Clips A and B
Output: Transformed Audio Features yAj and yB
j of Music Clips A and B
Learn the weights wl,m of Neural Network so that the differences between the distances of yA
j and yBj and the distances of the video scenes mixed with music clips A and B would be
minimized.
wl,m: Weight for the edge between nodes I and m.
・・・ ・・・
・・・
・・・ ・・・
・・・
TAB
dAB
Teacher
(Distances of Video Scenes
Mixed with Music Clips A and B )
Input A
Input B
xAi
xBi
Neural Network A
Neural Network B
yAj
yBj Distance calclulation
Interface
Video Remix Examples: 61 Action Movie Trailers
Video Scene Examples :45 Scenes
Music Clips:180 Music Clips of Various Genres (Movie Soundtracks, Classical Music, Japanese-pop, Western-pop, etc.)
Video Clips: Shots extracted from Original Movies
265 Home Video Clips recording a sports field day held by a kindergarten
Video Clip Sequence: Made by Procedure I)
Input:10 Video Scenes randomly extracted from movie trailers (without Audio Stream )
10 subjects rated (1: very bad – 10: very good) 10 video scenes mixed with
Video1) 3 Music Clips Selected by Proposed Approach
Video2) Music Clips most similar to the music excerpts mixed with the 3 least similar video scenes
Video3) Music Clip mixed with the video scenes in movie trailers (baseline: professional)
Video4) 3 Music Clips selected in the same way as for Video 1) without music feature space transformation
Video5) 3 Music Clips selected in the same way as for Video 2) without music feature space transformation
Video1 – Video 2 = 1.72±0.34
(95% confidence interval)
⇒indicates the effectiveness of
similarity-based music clip selection
Video1 – Video 4 = 1.11±0.35
⇒indicates the effectiveness of
music feature space transformation
Video 1 → closest to Video 3
⇒selected music clips are subjectively
closest to professionally selected ones 0
1
2
3
4
5
6
7
8
Video1
Video2
Video3
Video4
Video5
Average Subjective Scores
6.1
4.4
7.2
5.0 4.5
Video1
6.8
Video2
2.7
Video3
8.3
Video4
6.4
Video5
3.0
Score
Video1
8.5
Video2
2.1
Video3
5.5
Video4
5.5
Video5
2.3
Score
Subjective Score: 3.8 Subjective Score: 5.3
With Template* Without Template
Video Clip Sequence after Music Mixing
Subjective score improved largely after music mixing
Created video clip sequence and selected music clips are synergetic in improving the expressive quality.
* Selected video clips are shortened according to the template
Y. Kurihara, N. Nitta, and N. Babaguchi, “Automatic appropriate segment extraction from shots
based on learning from example videos,” Proc. PSIVT, pp.1082-1093, 2009
Y. Kurihara, N. Nitta, and N. Babaguchi, “Appropriate segment extraction from shots based on
temporal patterns of example videos,” Proc. MMM, pp.253-264, 2008
Vid
eo C
lip
Seq
uen
ce
Vid
eo R
emix
Video Clip 1
Shot 1 Shot 2 Shot 3
Video Clip 3 Video Clip 2
A video clip needs to be shortened. A video clip contains redundant parts.
Which part of a video clip should be extracted as a shot?
Shot Extraction from Selected Video Clip
k frames
Discarded part (Non-shot)
Selected Part (Shot)
Video Clip Example Video Clip
Shot Extraction
Feature Extraction
Pattern Scan for the k frames
which best matches the shot HMM
Feature Extraction
Shot Symbolization
Symbol Sequence
Shot HMM
Non-shot HMM
Overview of Procedure III)
•Shot Classification action and conversation
•Feature extraction Shot
Action Conversation Scenery ・・・
※VSTD : Volume Standard Deviation, LVFR : Low Volume Signal Ratio,ERSB : Energy Ratio of Ferquency SubBand ZCR : Zero Crossing Ratio
Each type of shot is characterized by different features
Examples:Movies+Trailers
Video Clips:Shots in Movies
Shots:Shots in Trailers
Shot extraction from 69 video clips (shots in movies)
Shot Length (k) = Length of corresponding shots in trailers
(32.3% ×video clips on average)
22 47 Test
12 10 Training
Conversation Action
Experiments
Objective Evaluation
Video Clip (Action)
Ground Truth (Shot in Trailer)
Extracted Shot
82 frames
k= 9 frames
Difference:3 frames(0.3sec)
•Compare Extracted Shot with Ground Truth •1 frame=0.1 sec
107 frames
Extracted Shot
Ground Truth
k= 17 frames
Difference:3 frames (0.3sec)
Video Clip (Conversation)
-25
-20
-15
-10
-5
0
5
10
0 5 10 15 20 25 30 35 40 45 50
フレーム#
LogP
f(n);編集区間モデル
g(n);非編集区間モデル
f(n)-g(n)
Extracted Shot Ground Truth
hk(f)-gk(f)
Correctly Extracted Shot
47 frames
k = 5 frames Difference:3 frames
Video Clip (Action)
Extracted Shot
Ground Truth
Shot HMM
Non-Shot HMM
frame
-30
-25
-20
-15
-10
-5
0
5
0 5 10 15 20 25 30 35
フレーム#
logP
f(n);編集区間モデル
g(n);非編集区間モデル
f(n)-g(n)
Extracted Shot Ground Truth
hk(f)-gk(f)
Incorrectly Extracted Shot
35 frames
k = 7 frames Difference:26 frames
Extracted Shot
Ground Truth
Shot HMM
Non-Shot HMM
frame
Objective Evaluation
clips videoof #
extractioncorrect of # accuracy
※Correct Extraction : Shot was extracted within T-frame Difference 1 frame = 0.1 sec
Correct shots were extracted from 72.5%(50/69) of video clips when T=5
73%(16/22) 72%(34/47) T=5
64%(14/22) 60%(28/47) T=3
50%(11/22) 53%(25/47) T=2
Action Conversation
14 subjects watch original long video clips, and then three kinds of shortly extracted shots:
①Ground Truth
②Extracted Shot
③Random Shot
in random order and rank them.
(There can be a tie)
or
or
Ground Truth:③ Extracted Shot:② Random Shot:①
Video Clip (36 frames) ①
②
③ k = 15 frames
Ground Truth
Extracted Shot Random Shot
・・・Rank 1
・・・Rank 2
・・・Rank 3 69.1%
26.9%
4.0%
53.9% 38.9%
7.2% 7.1%
12.9% 80.0%
Action:18 video clips Conversation:13 video clips
Subjective Evaluation
Extracted Shot ≒Ground Truth >> Random Shot
Subjective Score: 6.2 Subjective Score: 3.9
Without Template With Template
Created Video Remix
Proposed Comparative
I II III I II III
Length (min:sec)
0:36 0:43 10:56 10:59
score 3 5.3 6.2 3.5 3.8 3.9
Introduced an example-based approach for video remixing
Video Clip Sequence Creation
Music Clip Selection
Shot Extraction
Interface
Experiments using movie trailers as remix examples and movies and home videos as video clips
Verified the effectiveness of using remix examples
With Support(6.2), Without Support(3.9)
Conclusion
Improvement of Interface
More investigations using various types/genres of video remix examples
How many examples do we need?
Good examples can reduce the number of examples.
Top Related