CS257 2006 ghLecture 6
-
Upload
winter-fann -
Category
Documents
-
view
216 -
download
0
Transcript of CS257 2006 ghLecture 6
-
7/29/2019 CS257 2006 ghLecture 6
1/33
CS257 Modelling Multimedia Information
LECTURE 6
-
7/29/2019 CS257 2006 ghLecture 6
2/33
Introduction
See beginning of Lecture 5
-
7/29/2019 CS257 2006 ghLecture 6
3/33
-
7/29/2019 CS257 2006 ghLecture 6
4/33
Queries to Video Databases
User may want to specify a temporal
sequence of events, e.g. find me video
where this happens then this happens
while that happens
[More on this in PART 2]
-
7/29/2019 CS257 2006 ghLecture 6
5/33
Queries to Video Databases
How to express queries / How to describe
content can be considered two sides of
the same coin; both require dealing with
the same kinds of issues
-
7/29/2019 CS257 2006 ghLecture 6
6/33
Creating Metadata for Video Data
Content-descriptive metadata for video oftenneeds to be manually annotated
However, in some cases the process can be
automated (partially) by: Video segmentation Feature recognition, e.g. to detect faces, explosions, etc.
Extracting keywords from time-aligned collateral texts,e.g. subtitles and audio description
-
7/29/2019 CS257 2006 ghLecture 6
7/33
Overview of LECTURE 6
PART 1:Need to be able to formally describe video content interms of objects and events in order to make a query to a videodatabase, e.g. specify who is doing what.
Subrahmanians Video SQL
PART 2: May wish to specify temporal and / or causalrelationships between events, e.g. X happens before Y, A causes Bto happenAllens temporal logic
Roths system for video browsing by causal links
LABBring coursework questions;
-
7/29/2019 CS257 2006 ghLecture 6
8/33
PART 1:Querying Video Content
Four kinds of retrieval according to Subrahmanian (1998)
Segment Retrieval: find all video segments where an
exchange of a briefcase took place at Johns house
Object Retrieval: find all the people in the video
sequence (v,s,e)
Activity Retrieval: what was happening in the video
sequence (v,s,e)
Property-based Retrieval: find all segments where
somebody is wearing a blue shirt
-
7/29/2019 CS257 2006 ghLecture 6
9/33
Querying Video Content
Subrahmanian (1998) proposes an extension toSQL in order to express a users informationneed when querying a video database Based on video functions
Recall that SQL is a database query languagefor relational databases; queries expressed interms of:
SELECT (which attributes)FROM (which table)
WHERE (these conditions hold)
-
7/29/2019 CS257 2006 ghLecture 6
10/33
Subrahmanians
Video FunctionsFindVideoWithObject(o)
FindVideoWithActivity(a)
FindVideoWithActivityandProp(a,p,z)
FindVideoWithObjectandProp(o,p,z)
-
7/29/2019 CS257 2006 ghLecture 6
11/33
Subrahmanians
Video Functions (continued)FindObjectsInVideo(v,s,e)
FindActivitiesInVideo(v,s,e)
FindActivitiesAndPropsInVideo(v,s,e)
FindObjectsAndPropsInVideo(v,s,e)
-
7/29/2019 CS257 2006 ghLecture 6
12/33
A Query Language for Video
SELECT may contain
Vid_Id : [s,e]
FROM may contain
video : WHERE condition allows statements like
term IN func_call
(term can be variable, object, activity or property valuefunc_callis a video function)
-
7/29/2019 CS257 2006 ghLecture 6
13/33
EXAMPLE 1
Find all video sequences from the library
CrimeVidLib1 that contain Denis Dopeman
SELECT vid : [s,e]
FROM video : CrimeVidLib1
WHERE
(vid,s,e) IN FindVideoWithObjects(Denis Dopeman)
-
7/29/2019 CS257 2006 ghLecture 6
14/33
EXAMPLE 2
Find all video sequences from the library
CrimeVidLib1 that show Jane Shady giving
Denis Dopeman a suitcase
-
7/29/2019 CS257 2006 ghLecture 6
15/33
EXAMPLE 2
SELECT vid : [s,e]
FROM video : CrimeVidLib1
WHERE
(vid,s,e) IN FindVideoWithObjects(Denis Dopeman) AND
(vid,s,e) IN FindVideoWithObjects(Jane Shady) AND
(vid,s,e) IN FindVideoWithActivityandProp(ExchangeObject, Item, Briefcase) AND
(vid,s,e) IN FindVideoWithActivityandProp(ExchangeObject, Giver, Jane Shady) AND
(vid,s,e) IN FindVideoWithActivityandProp(ExchangeObject, Receiver, Denis Dopeman)
-
7/29/2019 CS257 2006 ghLecture 6
16/33
EXAMPLE 3
Which people have been seen with Denis
Dopeman in CrimeVidLib1
-
7/29/2019 CS257 2006 ghLecture 6
17/33
EXAMPLE 3
SELECT vid : [s,e], Object
FROM video : CrimeVidLib1
WHERE
(vid,s,e) IN FindVideoWithObject(Denis Dopeman) AND
Object IN FindObjectsInVideo(vid,s,e) AND
Object = Denis Dopeman AND
type of (Object, Person)
-
7/29/2019 CS257 2006 ghLecture 6
18/33
Exercise 6-1
Given a video database of old sports broadcasts, calledSportsVidLib, express the following users information needsusing the extended SQL as best as possible. You shouldcomment on how well the extended SQL is able to captureeach users information need and discuss alternative ways of
expressing the information need more fully.
Bob wants to see all the video sequences with Michael Owen kicking a ball
Tom wants to see all the video sequences in which Vinnie Jones is
tackling Paul Gascoigne
Mary wants to see all the video sequences in which Roy Keane is arguingwith the referee, because Jose Reyes punched Gary Neville, while ThierryHenry scores a goal, and then Roy Keane is sent off.
-
7/29/2019 CS257 2006 ghLecture 6
19/33
Bob wants to see all the video sequences
with Michael Owen kicking a ball
-
7/29/2019 CS257 2006 ghLecture 6
20/33
Tom wants to see all the video sequences in which
Vinnie Jones is tackling Paul Gascoigne
-
7/29/2019 CS257 2006 ghLecture 6
21/33
Mary wants to see all the video sequences inwhich Roy Keane is arguing with the referee,
because Jose Reyes punched Gary Neville, while
Thierry Henry scores a goal, and then Roy Keane
is sent off.
-
7/29/2019 CS257 2006 ghLecture 6
22/33
Think about
What metadata would be required in
order to execute these kinds of video
query?
How could this be stored and searched
most efficiently?
-
7/29/2019 CS257 2006 ghLecture 6
23/33
Part 2: Enriching Video Data
Models and Queries More sophisticated queries to video databases
can be supported by considering: Temporal relationships between video intervals
Causal relationships between events
Need to be able to describe temporalrelationships between intervals formallyandmake inferencesabout temporal sequences
-
7/29/2019 CS257 2006 ghLecture 6
24/33
Temporal Relationships
between Intervals Allens (1983) work on temporal logic is often discussed
in the video database literature (and in other computingdisciplines)
13 temporal relationships that describe the possible
temporal relationships that can hold between temporalintervals (e.g. intervals or events in video) these canbe used to formulate video queries
A transitivity table allows a system to infer the relationshipbetween A r C, if A r B and B r C are known (where r
stands for one temporal relationship, and A, B, C areintervals)
SEE MODULE WEB-PAGE FOR EXTRA NOTES ON THIS
X equal Y = = XXXXX
-
7/29/2019 CS257 2006 ghLecture 6
25/33
X equal Y = = XXXXX
YYYYY
X before Y < > XXXX YYYY
X meets Y m mi XXXXYYYY
X overlaps Y o oi XXXXX
YYYYY
X during Y d di XXX
YYYYYYYYY
X starts Y s si XXXXYYYYYYYY
X finishes Y f fi XXXXX
YYYYYYYYYY
-
7/29/2019 CS257 2006 ghLecture 6
26/33
Temporal Relationships
between Intervals
Crucial aspect of Allens work is the transitivity
table that enables inferences to be made about
temporal sequences
Inferences take the form:If A r B, and B r C, then r1, r2, r3 may hold
between A and C
For example:If A < B and B < C, then A < C
-
7/29/2019 CS257 2006 ghLecture 6
27/33
Another Example
If A contains B, and B < C then whatrelationships can hold between A and C?
BBBBB ?CC? ?CCCC? ?CCCCC?AAAAAAAAAAAAA?CCCCC?
?CCCCC?
Possibilities:A < C ; A overlaps C; A meets
C; A contains C; A is finished by C
-
7/29/2019 CS257 2006 ghLecture 6
28/33
Modelling the Relationships between
Entities and Events in Film
Some temporal relationships might beinterpreted as causal relationships
Roth (1999) proposed the use of a semanticnetwork to represent the relationships between
entities and events in a movie includingcausal relations
The user can then browse between scenes ina movie, e.g. if they are watching the scene of
an explosion, they may browse to the scene inwhich a bomb was planted, via the semanticnetwork (extra note on semantic network willbe on the module website).
-
7/29/2019 CS257 2006 ghLecture 6
29/33
-
7/29/2019 CS257 2006 ghLecture 6
30/33
Organising and Querying Video
Content Should consider
Which aspects of the video are likely to be ofinterest to the users who access the video
archive? How to store relevant information about the
video efficiently?
How to express and process queries?
What scope of automatic content extraction?
-
7/29/2019 CS257 2006 ghLecture 6
31/33
EXERCISE 6-2 For an video database application domain of your
choosing write five video queries that use some of
Allens 13 temporal relationships
If event A is before (
-
7/29/2019 CS257 2006 ghLecture 6
32/33
LECTURE 6:
LEARNING OUTCOMES
After the lecture, you should be able to:
Express a users query to a video database
using Subrahmanians VideoSQL and discuss
the limitations of this formalism Explain how and why temporal and causal
relationships between events are represented in
metadata for video databases
-
7/29/2019 CS257 2006 ghLecture 6
33/33
OPTIONAL READING
Dunckley (2003), pages 38-39; 393-395.For details of the extended video SQL, see:
Subrahmanian (1998). Principles of Multimedia Databases- pages 191-195. IN LIBRARY ARTICLE COLLECTION
For temporal relationships:Allen (1983). J. F. Allen, Maintaining Knowledge About TemporalIntervals. Communications of the ACM26 (11), pp. 832-843.Especially Figure 2 for the 13 relationships and Figure 4 for the fulltransitivity table. [In Library on shelf]
For causal relationships:
Roth (1999). Volker Roth, Content-based retrieval from digital video.Image and Vision Computing17, pp. 531-540. [Available onlinethrough library eJournals]