E.G.M. PetrakisThe MPEG Standard1 MPEG-1 (1992) actually a video player plays out audio/video...
-
Upload
elizabeth-hensley -
Category
Documents
-
view
218 -
download
1
Transcript of E.G.M. PetrakisThe MPEG Standard1 MPEG-1 (1992) actually a video player plays out audio/video...
E.G.M. Petrakis The MPEG Standard 1
The MPEG Standard
MPEG-1 (1992) actually a video player plays out audio/video streamssame type of access as home VCR
MPEG-2 (1995) introduced for compression and transmission of digital TV signalsstill limited interactivity
MPEG-4 (1999) is completely different high level of interactivity
MPEG-7 (2002) for the description of metadata only
E.G.M. Petrakis The MPEG Standard 2
MPEG-4
MPEG-4 addresses the need towardsMixing of natural and synthetic audiovisual
informationHigh interactivity in the presentation of
multimedia contentDeployment of communication systems for real-
time or broadcast delivery of coded data streamsA new approach for describing, coding and
presenting a sceneMPEG-4 combines different coding tools for
Audio/video Synthetic objects and graphics
E.G.M. Petrakis The MPEG Standard 3
MPEG-4 Objects
The audio/video components of MPEG-4Objects are coded, transmitted separately
and composed at the decoder siteThey can exist independentlyMultiple objects can be grouped together to
form complex objectsVideo and audio can be easily manipulatedPermits choosing appropriate coding tools
for audio, video and graphics objects
E.G.M. Petrakis The MPEG Standard 4
MPEG-4 Object Based Coding
E.G.M. Petrakis The MPEG Standard 5
MPEG-4 Coding
The scene is composed and rendered at the sender sitevideo frames, audio are coded, multiplexed
and transmitted tools for coding arbitrarily shaped objects
At the receiver the stream is demultiplexedvideo and audio are decoded, composed,
synchronized and presented as defined at the senders site
E.G.M. Petrakis The MPEG Standard 6
Object Coding
Objects are described mathematically (e.g. by their positions)similarly for audio and graphics objectsan object need only be defined oncethe viewer can change their positiontransmit calculations to update the scene
at the receiverthis is a critical feature when the response
has to be fast and bit-rate is limited
E.G.M. Petrakis The MPEG Standard 7
Binary Format for Scenes (BIFS)
MPEG-4’s language for describing and dynamically changing a scene
Borrows concepts from VRMLBoth define representations of the same dataVRML defines objects and actions in textBIFS code is binary (10-15 times shorter)Unlike VRML, MPEG-4 uses BIFS for real-time
streaming: a scene can be built-up and played on the fly
VRML and BIFS evolve consistently
E.G.M. Petrakis The MPEG Standard 8
scene graph
E.G.M. Petrakis The MPEG Standard 9
The Scene Graph
Represents a scene as independent or compound objects e.g., father and child the audio track of his voicefloor and walls (sprites: for
backgrounds)the web site the synthetic image of the furniturea synthetic HDTV set playing a movie
from the families DVD library
E.G.M. Petrakis The MPEG Standard 10
Elementary Streams (ES)The scheme for preparing content for
transmission, storage and decodingObjects are placed in ESsProbably two or more ESs per objectA sound track or a video may have a single
ESScalable objects way have one ES for basic
quality information + one or more enhancement layers for improved quality (e.g., finer detail, faster motion)
ESs are split into packets and sent along with timing information for proper synchronization
E.G.M. Petrakis The MPEG Standard 11
Object Descriptors (OD)
MPEG-4s mechanism that informs the system which ES belongs to a certain objectOD contain Elementary Stream
Descriptors (ESD) which tell the system which decoders to use
ODs are sent in their own stream which allows them to be added or deleted as the scene changes
E.G.M. Petrakis The MPEG Standard 12
Profiles and Levels
MPEG-4 provides a set of tools for coding multimedia contentsan application may use only subsets of these
toolsProfiles: MPEG-4s definitions of these
subsets for audio, visual, graphics information
Levels: define the computational complexity of the profile’s tool subset
Certain combinations of profiles fit well together
E.G.M. Petrakis The MPEG Standard 13
MPEG-4 Profiles
E.G.M. Petrakis The MPEG Standard 14
MPEG-4 Visual Objects
Arbitrarily shaped objects are coded apart from their background
Binary shape coding: a pixel is or is not part of an object simple, crude technique, suitable for low-bit
rates, suffers from aliasing
Alpha shape (gray scale) coding: each pixel is assigned a value for its transparencyobjects can be smoothly blended into a
background or with other objects
E.G.M. Petrakis The MPEG Standard 15
Visual Objects
Rectangular natural images and scenes are coded using MPEG-1, 2
Texture is coded separately by a DCT, block based coding scheme or wavelets
E.g., weather reports: the weatherman’s image seems to be standing in front of a map which is actually generated elsewhere
E.G.M. Petrakis The MPEG Standard 16
Object Segmentation
MPEG does not specify how objects are extracted video object segmentation is difficult e.g., record weatherman’s image in
front of a color background MPEG-4 specifies decoding
implementation of encoding is left to the industry to decide
E.G.M. Petrakis The MPEG Standard 17
MPEG-4 Applications
MPEG-4 makes video possible even at very low bit-rates (e.g., 10 kb/s)mobile devices, internet
Scalable objects for low bit-ratesa base layer conveys all the information
in some basic quality one of more enhancement layers can be
sent to get better qualitysend only the most important objects
E.G.M. Petrakis The MPEG Standard 18
Sprites
For coding unchanged backgrounds The background is defined and coded
only onceMust be updated for each change
(e.g., when the viewing angles changes)
The sprite is sent only onceNew views are created by sending the
new positions
E.G.M. Petrakis The MPEG Standard 19
Advanced Features
Map images into computer generated shapesa 2D or 3D mesh may have an image mapped onto ita few parameters to deform the mesh generate the
impression of a moving picture rather than sending new images for each change,
send commands and parameters to the viewerpre-defined faces are particularly interesting meshesthe appearance of a face may be left to the decoder
(e.g., custom facial models can be downloaded)
E.G.M. Petrakis The MPEG Standard 20
MPEG-4 Faces
Images laid over a wire-frame face
Send wire-frame plus parameters
Image reconstruction at receivers site
Speech is generated from text in steps with motions of the mouth, eyes and lips
E.G.M. Petrakis The MPEG Standard 21
MPEG-7MPEG-7 (2002) focuses on description of
multimedia contentmodalities: image, speech, video, graphics and
their combinationsMPEG-7 complements existing MPEG
standards and is applicable even to non-MPEG formats (compressed or uncompressed)
MPEG-7 is driven by trends in technology, market and user needs
Applications: VideoOnDemand, NewsOnDemand, InteractiveTV, multimedia information systems etc.
E.G.M. Petrakis The MPEG Standard 22
Scope of the StandardProvides the means for indexing,
searching, filtering and managing audio-visual contentbroadcast media selection (e.g.,
personalized TV)multimedia editing (e.g., personalized
news service)MPEG-7 interoperable interface
defines syntax and semantics tools may be designed for specific
modalities, aspects or applications
E.G.M. Petrakis The MPEG Standard 23
Interoperable Servicesand Applications
E.G.M. Petrakis The MPEG Standard 24
MPEG-7 Main Tasks
Multimedia: generate customized program guides or summaries of broadcast audio-visual content
Archive: generate descriptions of audio-visual content (or elements)
Adaptation: filter and transform multimedia streams in low bit-rate environments (e.g., mobile users)
E.G.M. Petrakis The MPEG Standard 25
MPEG-7 Specific Tasks
Music/audio: play a few notes and return music with similar music/audio
Images/graphics: draw a sketch and return images with similar graphics
Movement: describe movements and return video clips with the specified temporal and spatial relations
Scenario: describe actions and return scenarios where similar actions take place
E.G.M. Petrakis The MPEG Standard 26
MPEG-7 Elements
1. Descriptors (D) : define syntax and semantics of features of audio-visual content
Application independent Low level: shape, motion, color,
camera motion, harmonicity, timbre for audio ...
Semantic level: events, concepts ...
E.G.M. Petrakis The MPEG Standard 27
MPEG-7 Elements (cont.d)
2. Description Schemes (DS): specify the structure and semantics of the relationships among the constituent Ds or DSs e.g.,
Video DS specify syntax and semantics for segment decomposition, attributes, their relationships
DS related to creation, production, and access of content (e.g., property rights, parental rating, etc.)
E.G.M. Petrakis The MPEG Standard 28
MPEG-7 Elements (cont.d)
3. Description Definition Language (DDL): allows flexible definition of Ds and DSs based on XML schema
Ds and DSs are application independent
DLLs to define specialized tools
E.G.M. Petrakis The MPEG Standard 29
MPEG-7 Descriptions
MPEG-7 allows descriptions at different levels of abstractionslow level features extracted automaticallysemantic features with human interaction
or textual annotationMPEG-7 does not specify how features
are extracted or used (e.g., filtering, retrieval)their representation must conform to the
MPEG-7 standard
E.G.M. Petrakis The MPEG Standard 30
MPEG-7 Parts
Systems: specifies functionality at system level Preparation of descriptions for efficient
transport and storage synchronization of content and descriptorsdevelopment of decoders
Description Definition Language (DDL): language for specifying new Ds and DSsextension of XML schema
E.G.M. Petrakis The MPEG Standard 31
MPEG-7 Visual
Specifies a set of standardized visual Ds and DSsColor descriptors: color space, quantizationTexture descriptors: homogeneous texture,
texture browsing, edge histogram ...Shape descriptors: for regions or contoursMotion descriptors: camera motion,
trajectories, motion activity ...Face recognition
E.G.M. Petrakis The MPEG Standard 32
MPEG-7 Audio
Specifies standardized audio descriptors and descriptor schemes for pure music, pure speech, sound effects, soundtrackssilence descriptor spoken content descriptors sound effects descriptorsmelody contour descriptors
E.G.M. Petrakis The MPEG Standard 33
Multimedia Description Schemes
Specify a framework that allows generic description of all kinds of multimedia data
basic elements: data types, structures, Ds content management: content from
several viewpoints (creation, usage etc.) organization of content by collections,
classification navigation and access user interaction
E.G.M. Petrakis The MPEG Standard 34
Multimedia Description Schemes
E.G.M. Petrakis The MPEG Standard 35
MPEG-7 Reference Software
Reference implementation of the relevant parts of the MPEG-7 standardThe focus is on creating bit-streams of
descriptors and description schemes (DDL parser, DDL validation, multimedia description schemes)
Some software for extracting descriptors is also included (visual, audio descriptors)
E.G.M. Petrakis The MPEG Standard 36
References1. “MPEG-4 Multimedia for our Time” R. Koenen,
IEEE Spectrum, Feb. 1999, pp. 26-332. “
Applying and Implementing the MPEG-4 Multimedia Standard”, J. Kneip et.al. IEEE Micro, Nov-Dec 1999, pp. 64-74
3. “Overview of the MPEG-7 Standard”, S.-Fu Chang, T. Sikora and A. Puri, IEEE Transactions on Circuits and Systems for Video Technology, special issue on MPEG-7, June 2001
4. “Everything You Wanted to Know about MPEG-7” F. Nack and A.T. Lindsay, Part I, II, IEEE Multimedia, Aug-Dec1999