WWW9 Amsterdam Streaming Multimedia Metadata Frank Nack & Jane Hunter CWI, Amsterdam DSTC, Uni. Of...

WWW9Amsterdam

Streaming MultimediaMetadata

Frank Nack & Jane Hunter

CWI, Amsterdam DSTC, Uni. Of Qld

[email protected] [email protected]

WWW9Amsterdam

DescriptionGeneration

MPEG7Description

MPEG7Coded

DescriptionEncoder Decoder

Search /QueryEngine

HumanOrSystem

MPEG-7 System Architecture

XMLSchemaParser

WWW9Amsterdam

MPEG-7 Applications

• Indexing and retrieval of audiovisual archives (pull)

• Broadcast media selection (push media filtering e.g. news, radio, tv)

• Journalism, Entertainment

• Education

• Surveillance and remote sensing

• Telemedicine, Bio-medical applications

WWW9Amsterdam

When Is Metadata Streaming Required

• Applications performing in real timee.g. sports, news

• Interactive Applications e.g. education, VR games, etc.

WWW9Amsterdam

?

MPEG-7 Video Camera

WWW9Amsterdam

Types of Metadata Required

• Content Description

• Rights

• Formatting

• Administrative (financial, usage, rating)

• Delivery

• Security

• EPGs

Metadata andContent Referencing

Provider

MetadataActors

ServiceProvider

NetworkProvider

PDR Consumer

Input

Output

TargetActors

• Descriptive information• Applications• EPG• Copyright mgmt• Service mgmt • Content Referencing• Security

• Copyright mgmt• PDR Storage management• Payment models

• Descriptive information • Content Format• Content mgmt• Payment Models• Usage History• Content Reference• Security

Capturing Local storing Consuming

Content Creation & PackagingDelivery Storing •Viewing

•Copying

RelatedProcesses

• Original descriptive information• Creator IPR

Creator

Content

•Service Mgt in PDR•Location and Access Criteria•Payment Models•Billing•Security

•Content mgt (bookmarking etc)•Descriptive information•Payment•User Preferences•Usage History

• Descriptive information• Content Format • Content referencing• Copyright protections• Payment models

• Advanced Applications, EPG• Content management in PDR

• Copyright management• Storage management• Billing

METADATA FLOW IN THE TV-ANYTIME SYSTEM (TV037.DOC)

WWW9Amsterdam

Descriptors:Descriptors:(Syntax & semantic(Syntax & semanticof feature representation)of feature representation)

D7

D2

D5

D6D4

D1

D9

D8

D10

Description Definition extension extensionLanguage

DefinitionDefinition

101011 0

Encoding&

Delivery

TagsTags

<scene id=1> <time> .... <camera>.. <annotation</scene>

InstantiationInstantiation

D3

Description SchemesDescription Schemes

D1

D3D2

D5D4D6

DS2

DS3

DS1

DS4StructuringStructuring

MPEG-7 Components

WWW9Amsterdam

Definition: The Description Definition Language (DDL) is a language that allows the creation of new Description Schemes and, possibly, Descriptors. It also allows the extension and modification of existing Description Schemes.

(The MPEG-7 DDL is XML Schema Language with some extensions.)

Description Definition Language (DDL)

WWW9Amsterdam

UML of MediaInformationDS

WWW9Amsterdam

Definition: An MPEG-7 Description consists of a DS (structure) and the set of Descriptor Values (instantiations) that describe the Data.

MPEG-7 descriptions are XML but are there any non-XML metadata standards?

Description

<MediaInformation> <MediaProfile> <MediaIdentification> <Identifier IdOrg=’MPEG’ IdName=’MPEG7ContentSet’> mpeg7_content:news1</Identifier> <Locator> <MediaURL>file://D:/Mpeg7_17/news1.mpg</MediaURL> </Locator> </MediaIdentification> <MediaFormat>

<FileFormat>MPEG-1</FileFormat><System>PAL</System><Medium>CD</Medium><Color>color</Color><Sound>mono</Sound><FileSize>666.478.608</FileSize><Duration><m>38</m><s>18</s></Duration>

<AudioChannels>1</AudioChannels> </MediaFormat> <MediaCoding>

<FrameWidth>352</FrameWidth><FrameHeight>288</FrameHeight><FrameRate>25</FrameRate><CompressionFormat>MPEG-1</CompressionFormat>

</MediaCoding> </MediaProfile></MediaInformation>

WWW9Amsterdam

Definition: A Coded Description is a Description that has been encoded to fulfil relevant requirements such as compression efficiency, error resilience, random access, etc.

Coded Description

WWW9Amsterdam

Streaming MPEG-7

Two possible approaches

1 Multiplex descriptive metadata on top of video stream - packets interleaved• only one stream to parse• don’t know what is lost - metadata or essence?

2 Separate streams - one for video content, another for the XML metadata description(s)

• can prioritize streams, reserve bandwidth, specify QoS

Delivery-ready Description

DescriptionRef. To Schemas +

Instantiation(Text or binary)

TransportSync. & encapsulation

data for storage or streaming

(MPEG-2, MPEG-4)

Delivery-ready Media

MPEG-1MPEG-2MPEG-4

RealMedia, QT, AVI??

TransportSync. & encapsulation

data for storage or streaming

(MPEG-2, MPEG-4)

Stand-alone Elementary Streams (ESs)

Stand-alone Media Stream

Delivery-ready Described Media

Metadata

Media Data

Tran

spo

rt

Multiplex

WWW9Amsterdam

Metadata Categoriesrealtime vs mediatime

Specific metadata is application-dependent, however 4 relevant categories:

• Temporal delivery - metadata needs to be delivered at a certain real time - before or synchronized with the audiovisual delivery (content descriptions for filtering)

• Atemporal delivery - metadata can be delivered anytime (before, during, after - the billing information?)

• Temporal meaning - metadata linked by time stamp to video content e.g. keyframe.

• Atemporal meaning - e.g. title, director

WWW9Amsterdam

Elementary Streams• Separate metadata into separate streams according to type,

error sensitivity, temporal aspects

• Do we need to send schema definitions with the descriptions?

• Store schema defns. on set top box - update periodically by downloading from URLs

Access Units - MPEG-7 data packets

Synchronization layer - defines buffer model for MPEG-7 data plus interface to enable synchronous consumption

MPEG-7 Systems Approach

Description Layer

MPEG-7 Elementary StreamFlattened and compressedmetadata description

MPEG-7 AUES segmented intopackets with timing

Des_ES_ID

Media_ES_IDMedia Elementary Stream

Media AUES segmented intopackets with timing

Description AU Link

Media Layer

WWW9Amsterdam

Encoding Metadata for Streaming

Encoder has to:

• Flatten nested descriptions;

• Determine optimum packet size;

• Resend header/contextual info periodically to cover errors, packet loss and mid-way entry;

• Add time stamps to support real time delivery and synchronizn;

• Binary encode/compress packets;

WWW9Amsterdam

Linking

Need to be able to link from MPEG-7 descriptions to media or essence:

• MPEG-2 systems layer

• MPEG-4

• MPEG-4 file format on top of MPEG-2 or MPEG-4

WWW9Amsterdam

Effect of Lossiness

Built-in error checking but...

• What happens to timestamps/links to essence in the description when essence packets are lost? Link from metadata points to nothing.

• What happens if description packets are lost? Users miss lost ‘highlights’

WWW9Amsterdam

Streaming Choices

• Define a transport-independent MPEG-7 content stream and then define the carriage on transport mechanism

• MPEG-7 on MPEG-2 (transport stream private data)

• MPEG-7 on MPEG-4

• MPEG-7 on file (extension of MP4 file format)

• Or Use WAP/WML to define the transport-independent MPEG-7 content stream

WWW9Amsterdam

MPEG 1/2 Internal Metadata

• Image Size

• Frame Rate

• Motion Vectors

• User Data

• Private Data Streams

• Copyright Markers

• Color Separation (RGB -> YUV)

• Timing and Sync Information

WWW9Amsterdam

MPEG 2 Meta Data

Program Specific Information (PSI)

(ISO 13818-1/ITU-T Rec. H222.0, ETS 468)

• Program Association Table (PAT)

• Program Map Table (PMT)

• Conditional Access Table (CAT)

• Network Information Table (NIT)

Service Information (SI by DVB)(ETS 468, TM 1324)

• Bouquet Assotiation Table (BAT)

• Service Description Table (SDT)

• Event Information Table (EIT)

• Time and Date Table (TDT)

• Running Status Table (RST) Video Text (ETS 472)

Closed Caption (TM 1398)

WWW9Amsterdam

MPEG 4Media Object Description

• Audio• TextToSpeech• Score Driven Synthesis: SAOL (Structured Audio Orchestra

Language)

• Video• Parametric descriptions of

• a) a synthetic description of human face and body

• b) animation streams of the face and body

• Static and Dynamic Mesh Coding with texture mapping • Texture Coding for View Dependent applications • Shape and alpha coding

WWW9Amsterdam

MPEG 4Object Content Information

Data stream to send textual information along with MPEG-4 content

Possibilities:

• classify content according to pre-defined tables (defined outside of MPEG)

• giving unique labels to content

• storing camera parameters

WWW9Amsterdam

MPEG 4Scene Description

BIFS:• Binary Format for Scenes• to compose a set of objects into a scene• coded and transmitted together with the media

objects• follows a hierarchical structure• Each node of the graph is a media object• audiovisual objects have both a spatial and a

temporal extent• parameters to the composition layer

(e.g. pitch of a sound, the color for a synthetic object)

WWW9Amsterdam

MPEG 4 Scene Description II

BIFS: Scene Graph

WWW9Amsterdam

MPEG 4Stream Concept

WWW9Amsterdam

MPEG 4Scene Composition

WWW9Amsterdam

WAP/WML

Wireless Application Protocol (WAP) - specifn. that enables mobile users with wireless devices to easily access and interact with information and services

Wireless Markup Language (WML) - designed to send XML over low bandwidth with limited memory and CPU

• lightweight, subset of HDML (Hand-held Device Markup)

• specifies presentation/layout of text and images

• specifies user interaction

• linear, non-nested flat context

Binary XML Content Format Specification (WBML) - compact binary representation of XML documents

WWW9Amsterdam

References

MPEG-7

• http://www.cselt.it/mpeg/

• Document N3292 Systems Working Draft 0.3

MPEG-1, MPEG-2, MPEG-4

• http://www.cselt.it/mpeg/

WAP/WML

• http://www.oasis-open.org/cover/wap-wml.html

• http://www.wapforum.org/

• http://www.colorline.no/wap-faq/

WWW9 Amsterdam Streaming Multimedia Metadata Frank Nack & Jane Hunter CWI, Amsterdam DSTC, Uni. Of...

Documents

Transcript of WWW9 Amsterdam Streaming Multimedia Metadata Frank Nack & Jane Hunter CWI, Amsterdam DSTC, Uni. Of...