CS257 Modelling Multimedia Information LECTURE 5.

CS257 Modelling Multimedia InformationCS257 Modelling Multimedia Information

LECTURE 5LECTURE 5

IntroductionIntroduction

• Let’s review…– What we’re dealing with (video data)– What kinds of moving images might be stored

in a digital video database– What kinds of queries people might want to

make to a video database

Techniques for modelling and querying video content

Digital Video Data is…Digital Video Data is…

• A sequence of frames, where each frame is an image (typically 25 frames per second)

• May include a soundtrack: Speech (commentary / monologue / dialogue); Music; Sound effects

• May include text, i.e. subtitles / closed captions• Video data has temporal aspects as well as spatial

aspects: the temporal organisation of the moving images conveys meaning in its own right – change the order of scenes change the meaning of a film

• Cinematic techniques (pan, zoom, etc.) and editing effects can also convey information

Moving Images include…Moving Images include…

• Television• Cinema• Meteorological images• Surveillance cameras• Medical images• Biomechanical images• Dance

Diverse range of video database applications, users and information needs

What is Video Content?What is Video Content?

• Generally speaking video content can be said to comprise:– Objects (including people) with properties

AND– Activities (actions, events) involving 0 or more objects– Organised in space and time

• (This is an attempt at a generic approach, i.e. one that is applicable to all kinds of moving images)

Example queries to film Example queries to film archivesarchives

“Transport”“Meercats”“Portillo”“Ferries departing at night”“Animal rights protesters outside Hillingdon cat

farm, wearing masks”

From Enser and Sandom (2002).

Queries to Video DatabasesQueries to Video Databases

• User may not want to see the whole of a video file (e.g. a whole film), may just want certain parts or intervals (e.g. shots or scenes)

Need to attach metadata to different parts of the video [More on this in Part 1 of this Lecture]


• Users may want to query for a particular event involving particular people, e.g. “find me video with Bill hitting Tom” – why not use a list of keywords [hit, Bill, Tom] for query and to represent film content?

Need more structured descriptions of what’s happening (both for queries and for video metadata), i.e. who is doing what to whom with what and why. [More on this in Lecture 6]


• User may want to specify a temporal sequence of events, e.g. “find me video where this happens then this happens while that happens”

[More on this in Lecture 6]

Overview of LECTURE 5Overview of LECTURE 5

• PART 1: Need to be able to index (attach metadata to) and retrieve parts of video data files: Video Data Models

• PART 2: How developments in the MPEG standards have enabled new ways to access and create digital video

• LAB – Finish previous exercises; bring coursework questions

PART 1:PART 1: Video Data Models Video Data Models

• Must decide how to model (i.e. structure) video data so that metadata can be attached appropriately – must consider potential user information needs

• Video data can be modelled as:– BLOb’s (Binary Large Objects)– Frames– Intervals (discrete, hierarchical, overlapping); – Object-based schemes (cf. MPEG-4)

• NOTE, this does not entail chopping up the video data file

Modelling Video Data as a BLObModelling Video Data as a BLOb

• Metadata may be associated with the video data as a whole

• The kinds of metadata for visual information discussed in Lecture 4 apply equally well to moving images; but note – ideally only metadata that is true for the whole video data file should be associated with a BLOb

Modelling Video Data as a BLObModelling Video Data as a BLOb

Time

Attribute1:

Attribute2:

Attribute3:

AttributeX…

Modelling Video Data as FramesModelling Video Data as Frames

• An exhaustive metadata description of a video data file would include details for each and every frame (remember each frame is a still image)

• However, with 25-30 fps, the cost of this is usually prohibitive and there are few applications where it would be beneficial

Modelling Video Data as FramesModelling Video Data as Frames

Time

Frame: 1

Attribute A:

Attribute B:

Frame: 2

Attribute A:

Attribute B:

Frame: 3

Attribute A:

Attribute B:

Modelling Video Data as IntervalsModelling Video Data as Intervals

• It is more usual to model video data as ‘meaningful’ intervals – where ‘meaningful’ depends on the particular domain and application

• The intervals may be discrete or overlapping• The intervals may be arranged in a hierarchy so

that metadata descriptions can be inherited

Discrete IntervalsDiscrete Intervals

Time

Interval: 1

Start Time / Frame: 1

End Time / Frame: 103

Attribute A:

Attribute B:

Attribute C:

Interval: 2



Attribute A:

Attribute B:

Attribute C:

Interval 1 Interval 2 Interval 3 Interval 4

Overlapping IntervalsOverlapping Intervals

Time

Interval 1

Interval 2

Interval 3

Interval 4

Interval: 2



Attribute A:

Attribute B:

Attribute C:

Interval: 3



Attribute A:

Attribute B:

Attribute C:

Hierarchical IntervalsHierarchical Intervals

Time

Interval 1 Interval 2 Interval 3 Interval 4 Interval 5 Int 6Interval 7 Interval 8 Interval 9

Interval 10

Interval: 10



Attribute A:

Attribute B:

Attribute C:

Hierarchical IntervalsHierarchical Intervals• One example of video data that can modelled as hierarchical intervals are news broadcasts, made up of separate parts (main news, sports news, weather) and each part has separate stories, and each story is made up of shots and scenes

• Then attach metadata to each shot and scene

From: Corridoni et al (1996), ‘Multi-perspective Navigation of Movies’, Journal of Visual Languages and Computing 7, 445-466. [Available through Library eJournals].

Exercise 5-1Exercise 5-1

• Metadata about both audio and video data streams can be organised by modelling the data as a Binary Large Object (BLOb) or in terms of intervals. Intervals may be modelled as single- or multi-levelled, they may be discrete or overlapping, and they may be organised in a hierarchy. Give an example of a video stream and an example of an audio stream which would be best modelled as:

• (i) A Binary Large Objects (BLOb)• (ii) Overlapping multi-level intervals• (iii) A hierarchy of discrete intervals

• For each of (i), (ii) and (iii) you should discuss the information needs of potential users and give examples of the metadata you would attach to the data streams

Part 2: Developments in MPEG Part 2: Developments in MPEG standards for digital videostandards for digital video

Video Data: In the beginning….• Before the mid-1980s, digitising, storing and displaying still images,

never mind moving ones, required dedicated high power and high cost hardware

• Now video data is commonplace on home PCs; and throughout the TV and film industries

• Moreover, the ways in which video data can be accessed have become increasingly intuitive and flexible. In a large part these developments have been driven by successive MPEG standards

The MPEG standards: some hows and whys….

Moving Pictures Expert GroupMoving Pictures Expert Group• A committee comprising 100s of international

experts• MISSION: “Development of international

standards for compression, decompression, processing and coded representation of moving pictures, audio, and their combination, in order to satisfy a wide variety of applications”

Each standard developed with a specific aim

MPEG-1: why?MPEG-1: why?

Compression: video data at 1.5 Mbits/s• e.g. for storage on a CD-ROM and playback on

a low-spec PC • then for distribution of video data over the

Internet (not necessarily in real time) – progressive rendering

Started 1988; finished 1992(MPEG-1 Audio Layer III MP3)

MPEG-1: how?MPEG-1: how?

Combines visual, audio and timing information into one data stream suitable for digital transmission

Removes redundancy• Inter-frame redundancy (cf. still images)• Intra-frame redundancy – i.e. regions that do not

change (significantly) -> store some frames and interpolate others

Quality issues: fidelity of colour map; pixel resolution; frame resolution


Television requirements• higher quality• “efficient transmission over error-prone delivery

systems”Set top boxes and DVDsTV and film production

Started 1990; finished 1996


Digital Storage Media Command and Control: protocols for managing bitstreams

Requirements• Higher bandwidth• Dedicated hardware for coding (far from real

time); professional systems still cost many £K

(?MPEG-3?)


Next generation mobile communication• Further compression• PLUS… Reuse / Personlisation /

Customisation / Interactivity / Creativity

Requirements– Composition of real and synthetic media– Multiplex and synchronise objects over networks– Interact with audiovisual scene that is generated at the receiver’s

end

(Started 1993; parts finished in 2000/1)


**FROM FRAMES TO AUDIO-VISUAL OBJECTS**

Coding of audio-visual objects - real or synthetic• Still images (e.g. a fixed background)• Video objects (e.g. talking person); need not be

rectangular frames• Audio objects (e.g. person’s speech)• Text and graphics – inc. VR• Synthetic heads, speech and sound


Standard scene description (cf. VRML):• Place media objects in coordinate system• Apply transforms to change geometry / acoustics• Form compound media objects• Apply streamed data to objects to modify

attributes• Interactively change user’s viewing and listening

points

MPEG-7MPEG-7

Indexing and retrieval

“Multimedia Content Description Interface”

“different from previous MPEG standards… what is represented is not the information itself but information about the information”

i.e. it is a metadata standard.

MPEG-7MPEG-7

Description Definition Language– for low-level visual features; – for low-level audio features;– for ‘semantic content’, e.g. descriptions

of the narrative world depicted by a piece of media

• Started 1997; official standard in 2001• MPEG-7 cameras developed to capture

some features at time of recording

EXERCISE 5-2EXERCISE 5-2• Consider the following people who are each developing a different

multimedia system to handle digital video data. Which of the MPEG standards would be of interest to each person and why?

– Annabel: she has collected a digital library of historical documentaries which she would like people to be able to access via a high-speed intranet by making a range of queries about semantic content

– Brian: he is distributing a video browser with his home-made cartoons on a CD-ROM

– Claire: she is setting up a pay-per-view digital television service– Dave: he is an experimental filmmaker who is developing a system to help him

edit video material in new and interesting ways

• It can be argued that the format in which multimedia data is coded has an impact on the potential functionality of systems that will store, retrieve and manipulate that data. Discuss, with explanations, the impact on potential system functionality that you think could arise from the following choices of coding format:

– Coding a moving image in MPEG-1 or coding it in MPEG-4– Coding a piece of music in ‘.wav’ or coding it in ‘.midi’

LECTURE 5:LECTURE 5:LEARNING OUTCOMESLEARNING OUTCOMES

• After the lecture, you should be able to:– Select an appropriate video data model given

a specific kind of video data and some user information needs, and explain / justify your decision

– Describe and explain how each MPEG standard enabled new ways to access and create digital video

OPTIONAL READINGOPTIONAL READING

• For information about all the MPEG standards, see: www.chiariglione.org/mpeg/index.htm

• For an overview of MPEG-7: www.chiariglione.org/mpeg/standards/mpeg-7/mpeg-7.htm


• Benitez et al. (2002), “Description of a Single Multimedia Document”, in: Manjunath, Salembier and Sikora (eds.), Introduction to MPEG-7: multimedia content description interface.

There are 4 copies of this article in the Library Article Collection.

• Note, focus on pages 113-4 and 124-138, and particularly on Figs. 8.9, 8.11, 8.14 and 8.16.


Multimedia Standards in Television / Media Production• Issues of multimedia standards are discussed by a

number of other professional groups, including those interested in mxf (multimedia exchange format) and aaf (advanced authoring format).

For a short article about mxf and aaf, see: www.snellwilcox.com/knowledgecenter/mxf_aaf.html

Homepages;mxf: www.pro-mpeg.org/aaf: www.aafassociation.org/


• The following articles are available for download via the University Library’s e-journals section. Many of the authors are from the MPEG-7 committee but the papers are more concise than the official MPEG technical documents.

Martinez, J.M., Koenen, R., and Pereira, F. (2002), “MPEG-7: the generic multimedia content description standard, part 1.” IEEE Multimedia 9 (2), April-June 2002, 78–87

Martinez, J.M. (2002), “Standards - MPEG-7 overview of MPEG-7 description tools, part 2. ” IEEE Multimedia 9 (3), July-Sept. 2002, 83 -93.

Chang, S.-F., Sikora, T. and Purl (2001), “An Overview of the MPEG-7 Standard”, IEEE Transactions on Circuits and Systems for Video Technology, 11 (6), 688-695.

Nack, F. and Lindsay, A.T. (1999), “Everything you wanted to know about MPEG-7. Part 1.” IEEE Multimedia 6 (3), July-Sept. 1999, 65 –77.

Nack, F. and Lindsay, A.T. (1999), “Everything you wanted to know about MPEG-7. Part 2.” IEEE Multimedia 6 (4), Oct.-Dec. 1999 , 64 –73

Today’s LabToday’s Lab

• Continue with image retrieval

CS257 Modelling Multimedia Information LECTURE 5.

Documents

Transcript of CS257 Modelling Multimedia Information LECTURE 5.