WEB + TV = Multimedia Information Fusion Victor Kulesh and Valery Petrushin.
-
date post
21-Dec-2015 -
Category
Documents
-
view
216 -
download
0
Transcript of WEB + TV = Multimedia Information Fusion Victor Kulesh and Valery Petrushin.
Outline
• Information Retrieval on the Web and Event Tracking• Multimedia data analysis• Story matching• Story synchronization across multimedia streams• Demos:
– Video story extraction
– Event tracking in TV broadcasts
• Applications of the technology
Information Retrieval on the Web
• The goal of IR is to find documents that are relevant to the user’s query
• IR is a well-studied field• A branch of IR called Information Filtering (IF)
deals with static type of queries called ‘user profile’ and dynamic information
• Event Tracking is a sub-problem of Information Filtering for tracking specific events rather than general topics
Text document representation
• Newswire articles are converted from HTML to ASCII text• Most common words are removed (stop-list operation)• Stemming is applied• Term-frequency-inverse-document-frequency (TFIDF)
vectors are computed• When more than one document is available as seeds for an
event then their vectors are averaged• Event tracking is done by computed similarity between
documents
Detection Tradeoff Error (DET) Curves in 5 minutes
Distribution of Detection ScoresDET curves: ----- for A ----- for B
A
B
Web-based Event Tracking Performance Results
• The first two figures show the performance of the tracking system for Cosine (red), Dice (green) and Jaccard(blue) measures when 1 and 4 seed documents are used in the profile for each event
Similarity Measures
n
i ik
n
i ij
n
i ikijkj
ww
wwdd
1
2
1
2
1
*
*),cos(
:Cosine
kj
kj
kjdd
dddddice
*2),(
:Dice
kjkj
kjkj
dddd
ddddjaccardext
*
*),(_
:Jaccard Extended
22
Web-based Event Tracking Performance Results
• Performance of Cosine measure with 1, 2, 3, and 4 seed documents
DET curves: ----- 1 seed document ----- 2 seed documents ----- 3 seed documents ----- 4 seed documents
Multimedia data analysis
• Three types of data:– Closed captioning (CC)– Audio– Video
• Commercial detection and recognition• All three carry the same message, however, in
different formats and poorly synchronized• CC is easy to work with but AV is what we
really want
Commercial Detection/Recognition
• Commercial Detection– Video: frequent change of shots
– Audio: music + speech
• Commercial Recognition– Video: Hidden Markov Model (HMM)
– Audio: Gaussian Mixture Model (GMM)
• Composite score: HMM + GMM
Commercial Recognition Results
• These are preliminary results and we are working on improving the algorithms for commercials recognition
DET curves for commercial recognitionUsing audio, video and composite approaches
Miss Probability (%)
False Alarm(%)
Video only 14.29 6.73
Audio only 3.57 1.24
Video and Audio
3.57 0.16
Optimal operating points for ad recognition
Processing CC streams
• Closed captioning data has two good properties:– It is already divided into stories and speaker turns by
inclusion of special markers (>>> and >>)
– It is more accurate than ASR data
• The bad part is the fact that CC is very loosely synchronized with AV streams
• CC is used as a link between text documents from the internet for finding relevant stories and for extracting separate stories from AV streams
Story matching• Given a list of text documents and a list of stories from CC stream classical IR
techniques are applied to find pairs of documents that describe the same events
First -- lawyers for timothy mcveigh spoke with the condemned oklahoma city bomber today. They say mcveigh did not get a fair trial and he will seek a stay of execution. Mcveigh's lawyers accused the federal government of committing "a fraud upon the court." They say the supreme court has an old doctrine that when such fraud occurs any judgment that court has made is void. Attorney robert nigh says it's mcveigh's belief a stay of the june 11th scheduled execution is necessary in order to promote the integrity of the criminal justice system. Mr. Mcveigh has given us permission to seek a stay of execution on his behalf. This decision was not easy for mr. Mcveigh. He has prepared to die. He had previously indicated he preferred death to life in prison without the possibility of release. The attorneys are basing their claim on the basis of documents that were inadvertently withheld from them by the fbi. It is their contention the fbi is knowingly withholding further documents.
McVeigh to Decide About Appeal
DENVER (AP) - Oklahoma City bomber Timothy McVeigh will meet with lawyers this week and is likely to file a request to block his execution, his attorneys said Wednesday.The request would be based on about 4,000 documents the FBI turned over to McVeigh's attorneys earlier this month, just days before he had been scheduled to be executed for carrying out the 1995 blast that killed 168 people and injured hundreds more.At his office in Tulsa, Okla., Rob Nigh said he plans to meet with McVeigh on Thursday at the federal penitentiary in Terre Haute, Ind., and will seek his approval on a request to block the execution.Nigh declined to comment on the contents of the group of documents he would show McVeigh.But he added: ``You can certainly anticipate it will request a stay.''Nathan Chambers, McVeigh's Denver-based attorney, said McVeigh believes the information is worthy of judicial review.``If he gives us permission to file something, we'll probably file something tomorrow,'' he said Wednesday. ``We're in the process of drafting the paperwork.''McVeigh told a federal judge in December that he would not appeal his death sentence.In early May, the FBI gave McVeigh's attorneys thousands of documents that it said had accidentally not been turned over to the defense. Attorney General John Ashcroft then postponed McVeigh's execution from May 16 to June 11.In a statement Wednesday, Ashcroft reiterated that the government would fight any further delay, saying that failure to carry out the sentence ``would deny justice for the victims of this crime and for the American people.''Meanwhile, a former FBI agent who worked on the case reportedly told a Republican member of the Senate Judiciary Committee last year that the FBI ignored evidence that might have helped the defense.Ricardo Ojeda, a former special agent in Oklahoma City, wrote Sen. Charles Grassley, R-Iowa, in March 2000, complaining of corruption and discrimination in the FBI's field office, according to CBS' ``60 Minutes II.''``I am also aware of instances in other cases, including the Oklahoma City bombing, where exculpatory evidence was ignored and not documented. Including exculpatory information I personally gathered from leads assigned me in the case,'' Ojeda wrote.Nigh said Ojeda's allegations should have an impact on the case.``That information should, at minimum, change the course of this case in the near future,'' Nigh said on ``60 Minutes II.''The FBI said Ojeda's records were turned over to McVeigh's lawyers, but that none of his investigation was used at trial. Ojeda said he was fired from the FBI after testifying in a discrimination hearing against FBI management.``Because he is no longer on the rolls, former Agent Ojeda would not know that his concerns are unfounded,'' FBI Deputy Director Tom Pickard said in a statement. ``Thousands of agents worked on this case but, in the end, most did not have their work presented at trial.''Ojeda could not be reached by The Associated Press; there was no answer at his home in Oklahoma, and a message left at his wife's business was not returned.On the Net:Justice Department: http://www.usdoj.govBombing memorial: http://www.oklahomacitynationalmemorial.orgN
ewsw
ire
arti
cle
CC
Audio stream analysis
• Silence detection; non-silent parts of at least 4 seconds are further processed
• Using a 1 second window classification into speech/music is performed for non-silent parts of the audio stream
• The segments containing loud music are either commercials or previews
• Speech segments are transcribed using ViaVoice ASR engine
Audio processing examples
Speech segments and their ASR transcripts*Lawyers for convicted Oklahoma City bomber Timothy McVeigh saidin a motion for a stay of execution
Of a final decision will be made until they meetwith him tomorrow if and his attorneys said they didn't have enough time to review four thousand pages of withheld FBI documents
The Justice Department's says the documents don't change any thank attorney-general John Ashcroft is not support granting anotherthe lack
*Text in red is transcribed incorrectly
CC and Audio alignment
• For each CC we know the story boundaries (in words)• For each segment of transcription we know its location
in the audio stream (1 second precision)• By aligning CC and transcripts using local alignment
algorithm we find the story boundaries in the audio stream
• The beginning of the left-most audio segment and the end of the right-most audio segment are used for locating the video boundaries
CC and ASR text alignment example
Local Alignment results for: ASR story3.txtLocal Alignment Number: 1 Similarity Score: 3444Match Percentage: 75%Number of Matches: 380Number of Mismatches: 19Total Length of Gaps: 107Begins at: 878 1 Ends at: 1295 487
878 Lawyers for convicted Oklahoma City bomber Timothy McVeigh said in a motion for a stay of execution |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ----||||||||||||||||||||||||||||||||| 1 Lawyers for convicted oklahoma city bomber timothy mcveigh say a motion for a stay of execution
978 O f a final decision w |-|-----|-----||-------------------------------------------------------------------|||||||||||||||| 97 could be filed as early as tomorrow. Right now he 's scheduled to die june 11th. No final decision w
1000 ill be made until they meet with him tomorrow if and his attorneys said they didn't have enoug |||||||||||||||||||||||||||||||||||||||||||||-| -| ||||||||||||-----||||||||||||||||||||||||||| 197 ill be made until they meet with him tomorrow. Mcveig h's attorneys have said they didn't have enoug
1094 h time to review four thousand pages of withheld FBI documents The Justice Department's says the do ||||||||||||||||| |---------|||||||||||||||||||||||||||||||-|||||||||||||||||||||||--|||||||||||| 296 h time to review 4000 pages of withheld fbi documents. The justice department says the do
1193 cuments don't change any thank attorney-general John Ashcroft i s not support granting another the ||||||||||||||||||||||||-|| | -||||||||| |||||||||||||||||||||| --||||||||||||||||||||||||||||||| -| 385 cuments don't change any thing. Attorney general john ashcroft does not support granting another d e
1290 lack -|| | 483 lay.
Extracting AV clips
• Audio is synchronized with video fairly well • Observation: video slightly precedes audio• Shot cuts in video are detected using intensity
histogram difference of neighboring frames• The true boundaries of the AV clip are
computed by searching for video shot cuts that are closest to the left and right audio story boundaries in a local neighborhood of 10 frame radius
Preliminary Accuracy Results
Date
(clip # )
True video cuts
Start End
Predicted video cuts
Start End
May 30
(clip 1) 1364 1787 1365 1787
May 31
(clip 2) 500 1545 509 1545
Jun 01
(clip 3) 1647 2226 1649 2388
Miss error rate: 0.58%False alarm error rate: 7.41%
Perseus: Personal Multimedia News Portal
• Collects news on the Web– User’s profile
– Event tracking
• Extracts relevant video clips• Creates personalized video summaries
(for the grandma)
How about commercialization?
• Web service companies– Providing video clips on demand
– Personalized video news summaries
• Electronics manufacturers– Embedding Web and AV analysis into
a product (WebTV + TiVo box)
• Broadcast companies– AV indexing