11/4/1999ACM Multimedia 991 Auto-Summarization of Audio-Video Presentations Li-wei He, Elizabeth...
-
Upload
katelyn-skinner -
Category
Documents
-
view
219 -
download
0
Transcript of 11/4/1999ACM Multimedia 991 Auto-Summarization of Audio-Video Presentations Li-wei He, Elizabeth...
![Page 1: 11/4/1999ACM Multimedia 991 Auto-Summarization of Audio-Video Presentations Li-wei He, Elizabeth Sanocki Anoop Gupta, Jonathan Grudin Collaboration and.](https://reader035.fdocuments.in/reader035/viewer/2022062511/55147644550346b2598b4580/html5/thumbnails/1.jpg)
11/4/1999 ACM Multimedia 99 1
Auto-Summarization of Audio-Video Presentations
Li-wei He, Elizabeth Sanocki
Anoop Gupta, Jonathan Grudin
Collaboration and Multimedia Group
Microsoft Research
![Page 2: 11/4/1999ACM Multimedia 991 Auto-Summarization of Audio-Video Presentations Li-wei He, Elizabeth Sanocki Anoop Gupta, Jonathan Grudin Collaboration and.](https://reader035.fdocuments.in/reader035/viewer/2022062511/55147644550346b2598b4580/html5/thumbnails/2.jpg)
11/4/1999 ACM Multimedia 99 2
Motivation
• On-demand multimedia is becoming pervasive– Corporate training and communication
• At Microsoft, over 360 courses online in two years
– Research seminars• Microsoft Research archives about 2 talks daily
![Page 3: 11/4/1999ACM Multimedia 991 Auto-Summarization of Audio-Video Presentations Li-wei He, Elizabeth Sanocki Anoop Gupta, Jonathan Grudin Collaboration and.](https://reader035.fdocuments.in/reader035/viewer/2022062511/55147644550346b2598b4580/html5/thumbnails/3.jpg)
11/4/1999 ACM Multimedia 99 3
Motivation (Cont.)
• Effective summarization and browsing techniques can help viewers utilize time better– Audio-video different from text
– Many approaches possible• Time-compression, indexes, highlights, …
• This talk focuses on:– Informational presentations
– Automatic summarization methods
![Page 4: 11/4/1999ACM Multimedia 991 Auto-Summarization of Audio-Video Presentations Li-wei He, Elizabeth Sanocki Anoop Gupta, Jonathan Grudin Collaboration and.](https://reader035.fdocuments.in/reader035/viewer/2022062511/55147644550346b2598b4580/html5/thumbnails/4.jpg)
11/4/1999 ACM Multimedia 99 4
What Is a Video Summary?
• Assembled from segments of the original
O rig ina l
S um m ary
![Page 5: 11/4/1999ACM Multimedia 991 Auto-Summarization of Audio-Video Presentations Li-wei He, Elizabeth Sanocki Anoop Gupta, Jonathan Grudin Collaboration and.](https://reader035.fdocuments.in/reader035/viewer/2022062511/55147644550346b2598b4580/html5/thumbnails/5.jpg)
11/4/1999 ACM Multimedia 99 5
The 4 C’s of a Good Summary
• Conciseness: as short as possible
• Coverage: covers key points
• Context: defines terms before using them
• Coherence: flows naturally and fluidly
![Page 6: 11/4/1999ACM Multimedia 991 Auto-Summarization of Audio-Video Presentations Li-wei He, Elizabeth Sanocki Anoop Gupta, Jonathan Grudin Collaboration and.](https://reader035.fdocuments.in/reader035/viewer/2022062511/55147644550346b2598b4580/html5/thumbnails/6.jpg)
11/4/1999 ACM Multimedia 99 6
Talk Outline• Introduction
• Automatic summarization– Sources of information in A/V presentations– Three algorithms
• Evaluation
![Page 7: 11/4/1999ACM Multimedia 991 Auto-Summarization of Audio-Video Presentations Li-wei He, Elizabeth Sanocki Anoop Gupta, Jonathan Grudin Collaboration and.](https://reader035.fdocuments.in/reader035/viewer/2022062511/55147644550346b2598b4580/html5/thumbnails/7.jpg)
11/4/1999 ACM Multimedia 99 7
Sources of Information
• Audio and video– Pitch and pause information
• Speaker actions– Slide-transition points
• End-user actions– Video segments watched by earlier viewers
![Page 8: 11/4/1999ACM Multimedia 991 Auto-Summarization of Audio-Video Presentations Li-wei He, Elizabeth Sanocki Anoop Gupta, Jonathan Grudin Collaboration and.](https://reader035.fdocuments.in/reader035/viewer/2022062511/55147644550346b2598b4580/html5/thumbnails/8.jpg)
11/4/1999 ACM Multimedia 99 8
Auto-summarization Methods
Method 1
(S)
Method 2
(P)
Method 3
(SPU)
Slide transition
X X
Pitch analysis
X X
User access log
X
![Page 9: 11/4/1999ACM Multimedia 991 Auto-Summarization of Audio-Video Presentations Li-wei He, Elizabeth Sanocki Anoop Gupta, Jonathan Grudin Collaboration and.](https://reader035.fdocuments.in/reader035/viewer/2022062511/55147644550346b2598b4580/html5/thumbnails/9.jpg)
11/4/1999 ACM Multimedia 99 9
1. Slide-based Method (S)
• Rationale– Beginning of a slide marks a new topic– Time devoted to slide indicates its importance
• Algorithm– First N% of video for each slide
![Page 10: 11/4/1999ACM Multimedia 991 Auto-Summarization of Audio-Video Presentations Li-wei He, Elizabeth Sanocki Anoop Gupta, Jonathan Grudin Collaboration and.](https://reader035.fdocuments.in/reader035/viewer/2022062511/55147644550346b2598b4580/html5/thumbnails/10.jpg)
11/4/1999 ACM Multimedia 99 10
2. Pitch-based Method (P)
• Rationale– Pitch activity indicates the speaker’s emphasis
• Algorithm (based on Arons ISSLP 94)– Compute pitch for every 1ms frame – Count the number of frames above a threshold
in 15 second windows– Select the windows with the most count
![Page 11: 11/4/1999ACM Multimedia 991 Auto-Summarization of Audio-Video Presentations Li-wei He, Elizabeth Sanocki Anoop Gupta, Jonathan Grudin Collaboration and.](https://reader035.fdocuments.in/reader035/viewer/2022062511/55147644550346b2598b4580/html5/thumbnails/11.jpg)
11/4/1999 ACM Multimedia 99 11
3. Combined Method (SPU)• The amount of time that previous viewers
spent on a slide indicates importance
010203040506070
0 10 20 30 40 50 60 70 80 90
Nth minute into the talk
User
count
A B
![Page 12: 11/4/1999ACM Multimedia 991 Auto-Summarization of Audio-Video Presentations Li-wei He, Elizabeth Sanocki Anoop Gupta, Jonathan Grudin Collaboration and.](https://reader035.fdocuments.in/reader035/viewer/2022062511/55147644550346b2598b4580/html5/thumbnails/12.jpg)
11/4/1999 ACM Multimedia 99 12
3. Combined Method (SPU)
• Algorithm– Compute importance measure for each slide
– Allocate summary time for each slide according to the importance measure
– Use pitch-based algorithm to pick the segments in each slide
Average Viewer Count of Slide N
Average Viewer Count of Slide N-1
Importance of Slide N =
![Page 13: 11/4/1999ACM Multimedia 991 Auto-Summarization of Audio-Video Presentations Li-wei He, Elizabeth Sanocki Anoop Gupta, Jonathan Grudin Collaboration and.](https://reader035.fdocuments.in/reader035/viewer/2022062511/55147644550346b2598b4580/html5/thumbnails/13.jpg)
11/4/1999 ACM Multimedia 99 13
Talk Outline• Introduction
• Automatic summarization
• Evaluation– Experimental design– Results
![Page 14: 11/4/1999ACM Multimedia 991 Auto-Summarization of Audio-Video Presentations Li-wei He, Elizabeth Sanocki Anoop Gupta, Jonathan Grudin Collaboration and.](https://reader035.fdocuments.in/reader035/viewer/2022062511/55147644550346b2598b4580/html5/thumbnails/14.jpg)
11/4/1999 ACM Multimedia 99 14
Experimental Design
• To compare summarization techniques– Original presenters (authors) created summaries (A) as
gold standard
– Authors wrote quiz questions that covered the content of summaries
– Objective measure: quiz score improvement after watching a summary
– Subjective measures: user survey
![Page 15: 11/4/1999ACM Multimedia 991 Auto-Summarization of Audio-Video Presentations Li-wei He, Elizabeth Sanocki Anoop Gupta, Jonathan Grudin Collaboration and.](https://reader035.fdocuments.in/reader035/viewer/2022062511/55147644550346b2598b4580/html5/thumbnails/15.jpg)
11/4/1999 ACM Multimedia 99 15
Experimental Design (Cont.)
• 4 summary types (S, P, SPU, A)
• 4 talks chosen from Microsoft training site
• 24 Microsoft employees were subjects – Summary types and talks are counter-balanced
within each subject
![Page 16: 11/4/1999ACM Multimedia 991 Auto-Summarization of Audio-Video Presentations Li-wei He, Elizabeth Sanocki Anoop Gupta, Jonathan Grudin Collaboration and.](https://reader035.fdocuments.in/reader035/viewer/2022062511/55147644550346b2598b4580/html5/thumbnails/16.jpg)
11/4/1999 ACM Multimedia 99 16
Demo Summary
![Page 17: 11/4/1999ACM Multimedia 991 Auto-Summarization of Audio-Video Presentations Li-wei He, Elizabeth Sanocki Anoop Gupta, Jonathan Grudin Collaboration and.](https://reader035.fdocuments.in/reader035/viewer/2022062511/55147644550346b2598b4580/html5/thumbnails/17.jpg)
11/4/1999 ACM Multimedia 99 17
Quiz Score Improvement• As expected, author-created summaries did best• No significant difference among the automatic
methods
0
1
2
3
4
5
6
7
8
A SPU P S
Qu
iz S
co
re
pre-study scores
post-summary scores
![Page 18: 11/4/1999ACM Multimedia 991 Auto-Summarization of Audio-Video Presentations Li-wei He, Elizabeth Sanocki Anoop Gupta, Jonathan Grudin Collaboration and.](https://reader035.fdocuments.in/reader035/viewer/2022062511/55147644550346b2598b4580/html5/thumbnails/18.jpg)
11/4/1999 ACM Multimedia 99 18
Survey Rating Results
• A >> SPU > P = S
Context
(1-7)
Concise
(1-7)
Coherent
(1-7)
Coverage
(%)
A 5.39 5.57 5.30 75
SPU 4.30 4.52 3.48 63
P 4.00 4.13 3.48 63
S 4.26 4.17 3.64 58
![Page 19: 11/4/1999ACM Multimedia 991 Auto-Summarization of Audio-Video Presentations Li-wei He, Elizabeth Sanocki Anoop Gupta, Jonathan Grudin Collaboration and.](https://reader035.fdocuments.in/reader035/viewer/2022062511/55147644550346b2598b4580/html5/thumbnails/19.jpg)
11/4/1999 ACM Multimedia 99 19
Percent of Value Derived
• From slide content: 46%
• From audio content: 36%
• From video content: 18%
![Page 20: 11/4/1999ACM Multimedia 991 Auto-Summarization of Audio-Video Presentations Li-wei He, Elizabeth Sanocki Anoop Gupta, Jonathan Grudin Collaboration and.](https://reader035.fdocuments.in/reader035/viewer/2022062511/55147644550346b2598b4580/html5/thumbnails/20.jpg)
11/4/1999 ACM Multimedia 99 20
Interesting Sequence Effect
Order Clear Choppy Overall
1 4.04 6.00 3.65
2 4.39 5.09 4.09
3 4.39 4.70 4.00
4 5.13 3.91 5.18
![Page 21: 11/4/1999ACM Multimedia 991 Auto-Summarization of Audio-Video Presentations Li-wei He, Elizabeth Sanocki Anoop Gupta, Jonathan Grudin Collaboration and.](https://reader035.fdocuments.in/reader035/viewer/2022062511/55147644550346b2598b4580/html5/thumbnails/21.jpg)
11/4/1999 ACM Multimedia 99 21
Conclusions
• Ability to skim/browse will be key to wide use
• Automated methods can add significant value– Add domain knowledge is important– Increasing acceptance over time
• Evaluation is a key but very difficult
![Page 22: 11/4/1999ACM Multimedia 991 Auto-Summarization of Audio-Video Presentations Li-wei He, Elizabeth Sanocki Anoop Gupta, Jonathan Grudin Collaboration and.](https://reader035.fdocuments.in/reader035/viewer/2022062511/55147644550346b2598b4580/html5/thumbnails/22.jpg)
11/4/1999 ACM Multimedia 99 22
Conclusions (Cont.)
• Getting the human into the loop– Speakers– End-users as a group
• E.g. collaborative filtering
– End-users as an individual• E.g. interactive browsing
• Visit us at: http://research.microsoft.com/coet
![Page 23: 11/4/1999ACM Multimedia 991 Auto-Summarization of Audio-Video Presentations Li-wei He, Elizabeth Sanocki Anoop Gupta, Jonathan Grudin Collaboration and.](https://reader035.fdocuments.in/reader035/viewer/2022062511/55147644550346b2598b4580/html5/thumbnails/23.jpg)
11/4/1999 ACM Multimedia 99 23
Interface of a Typical Talk
Table of content
Video
Slides
VCR-likecontrols
![Page 24: 11/4/1999ACM Multimedia 991 Auto-Summarization of Audio-Video Presentations Li-wei He, Elizabeth Sanocki Anoop Gupta, Jonathan Grudin Collaboration and.](https://reader035.fdocuments.in/reader035/viewer/2022062511/55147644550346b2598b4580/html5/thumbnails/24.jpg)
11/4/1999 ACM Multimedia 99 24
Summary Characteristics
• Talks were from MS internal training site– UI Design, Internet Explorer, Dynamic HTML,
Microsoft Transaction Server
• Average length– 20% to 25% of the original– 10 to 14 minutes
• Overlap with author-created summaries was no better than chance
![Page 25: 11/4/1999ACM Multimedia 991 Auto-Summarization of Audio-Video Presentations Li-wei He, Elizabeth Sanocki Anoop Gupta, Jonathan Grudin Collaboration and.](https://reader035.fdocuments.in/reader035/viewer/2022062511/55147644550346b2598b4580/html5/thumbnails/25.jpg)
11/4/1999 ACM Multimedia 99 25
Survey on the Summary Just Watched
• Concise: It captured the essence of the talk without using too many sentences
• Coverage: My confidence that it covered the key points of the talk is …
• Context: It is clear and easy to understand
• Coherent: It provided reasonable context, transitions, and sentence flow
![Page 26: 11/4/1999ACM Multimedia 991 Auto-Summarization of Audio-Video Presentations Li-wei He, Elizabeth Sanocki Anoop Gupta, Jonathan Grudin Collaboration and.](https://reader035.fdocuments.in/reader035/viewer/2022062511/55147644550346b2598b4580/html5/thumbnails/26.jpg)
11/4/1999 ACM Multimedia 99 26
Survey Rating Results
• A >> SPU > P = S
1
2
3
4
5
6
7
Context Concise Coherent Coverage
A
SPU
P
S
![Page 27: 11/4/1999ACM Multimedia 991 Auto-Summarization of Audio-Video Presentations Li-wei He, Elizabeth Sanocki Anoop Gupta, Jonathan Grudin Collaboration and.](https://reader035.fdocuments.in/reader035/viewer/2022062511/55147644550346b2598b4580/html5/thumbnails/27.jpg)
11/4/1999 ACM Multimedia 99 27
Information Not Used
• Spoken text content
• Speaker gestures
![Page 28: 11/4/1999ACM Multimedia 991 Auto-Summarization of Audio-Video Presentations Li-wei He, Elizabeth Sanocki Anoop Gupta, Jonathan Grudin Collaboration and.](https://reader035.fdocuments.in/reader035/viewer/2022062511/55147644550346b2598b4580/html5/thumbnails/28.jpg)
11/4/1999 ACM Multimedia 99 28
Talk Outline• Introduction
– Motivation– Definition of a video summary– Attributes of a good summary
• Automatic summarization
• Evaluation
![Page 29: 11/4/1999ACM Multimedia 991 Auto-Summarization of Audio-Video Presentations Li-wei He, Elizabeth Sanocki Anoop Gupta, Jonathan Grudin Collaboration and.](https://reader035.fdocuments.in/reader035/viewer/2022062511/55147644550346b2598b4580/html5/thumbnails/29.jpg)
11/4/1999 ACM Multimedia 99 29
Viewers Over Time for One Talk• Viewer number decreases overall and
within each slide
010203040506070
0 10 20 30 40 50 60 70 80 90
Nth minute into the talk
User
count
A B
![Page 30: 11/4/1999ACM Multimedia 991 Auto-Summarization of Audio-Video Presentations Li-wei He, Elizabeth Sanocki Anoop Gupta, Jonathan Grudin Collaboration and.](https://reader035.fdocuments.in/reader035/viewer/2022062511/55147644550346b2598b4580/html5/thumbnails/30.jpg)
11/4/1999 ACM Multimedia 99 30
Importance Measure
Average Viewer Count of Slide N
Average Viewer Count of Slide N-1
Importance of Slide N =
![Page 31: 11/4/1999ACM Multimedia 991 Auto-Summarization of Audio-Video Presentations Li-wei He, Elizabeth Sanocki Anoop Gupta, Jonathan Grudin Collaboration and.](https://reader035.fdocuments.in/reader035/viewer/2022062511/55147644550346b2598b4580/html5/thumbnails/31.jpg)
11/4/1999 ACM Multimedia 99 31
Author-created Summary (A)
• Original presenters (authors) were asked to produce summaries of the talks
– Author marked the text transcript
– Video summaries were generated manually by aligning the video with the marked portions
![Page 32: 11/4/1999ACM Multimedia 991 Auto-Summarization of Audio-Video Presentations Li-wei He, Elizabeth Sanocki Anoop Gupta, Jonathan Grudin Collaboration and.](https://reader035.fdocuments.in/reader035/viewer/2022062511/55147644550346b2598b4580/html5/thumbnails/32.jpg)
11/4/1999 ACM Multimedia 99 32
Summary
• Automatic algorithms performed respectably – “That’s pretty cool for a computer. I thought
someone had sat down and made them”– SPU was preferred over S and P
• Will viewers get used to auto summary?
![Page 33: 11/4/1999ACM Multimedia 991 Auto-Summarization of Audio-Video Presentations Li-wei He, Elizabeth Sanocki Anoop Gupta, Jonathan Grudin Collaboration and.](https://reader035.fdocuments.in/reader035/viewer/2022062511/55147644550346b2598b4580/html5/thumbnails/33.jpg)
11/4/1999 ACM Multimedia 99 33
Future Work
• Compare audio/video and text summaries
• Interactive and intelligent video browser
• Visit us at http://research.microsoft.com/coet