Automatic Manifestation of Composite Multimedia Objects

Ahmed Gomaa

Ph.D. Student

Rutgers University - CIMIC

Outline• Introduction to CMO• Types of constraints• Universal Access• Related Research• Framework

– Petri net layer– SMIL layer

• Architecture• Research issues

– Uniform Model – Incorporating User interactivity – Modality conversion: Determining minimal cost paths – Enforcing Security Constraints

– MMCTPN– SMIL 2.0 Implementation

– Conflict Identification and Resolution

Introduction

• Composite multimedia Object (CMO)– Comprised of different media component

such as text, image, video, audio, shape files,...

– Variety of relationships among components.

– Relationships represent different types of constraints (e.g., temporal), that must be adhered to when rendering it.

– Associated with each component is a set of parameters, such as modality and playback duration.

Types of Constraints

• Synchronization constraints– Map A and Video B appear

simultaneously

• Spatial Constraints– Image I is above Text T

• Modality Constraints– Video B needs a player to be

rendered

• Fidelity Constraints– Map A must be displayed at a

resolution of at least 800*600

• Security Constraints– Sales representative audio and video

are accessed by subscribers only.

CMO

Over Any Network

To Any One with varying expertise, capabilities, and preferences

To Any Device

Universal Access

• When a subject requests a CMO, he may not be able to view the entire CMO.– For example, due to the limitations of his

appliances, or due to lack of his credentials to satisfy the security requirements.

• Need to facilitate access to desired CMO according to the various user’s:

• Capabilities ( e.g, devices)

• Characteristics (e.g, expertise)

• Credentials (e.g, subscribers)

Related Research

Multimedia Object Rendering:• Adam and Atluri et al.(2001) Presented the Universal Access

problem and the Oblet approach. – Uses Petri Net to present temporal, spatial, modality and fidelity

constraints.

• Bertino et al.(2000) present a system called MPGS - Multimedia Presentation Generator System.– Enables specification of synchronization and spatial constraints

– Capable of analyzing the consistency among these two types of constraints.

None of those papers addressed the security constraints in a CMO

Related Research

Security:• Damiani et al.(2000),Bertino et Al.(2000) and Kudo et al (2000) addressed

access control models in XML.

• We adopted Kudo et al. approach in presenting a new Petri Net model that incorporate the security constraints in the CMO.

Related ResearchUser interaction:• User interaction research within CMO may be divided into three categories:

– Formal modeling

– Programming approach

– Statistical techniques

• Formal modeling, using Petri net :– Song et al.(1996) presented Timed Petri Net (TPN).

– Guan et al. (1998) presented a distributed object composition Petri net (DOCPN).

– Prabhakaran et al.(1993) presented a dynamic timed Petri nets (DTPN) model.

– All models are restrictive in types of interactions.

– They only deal with the synchronization constraints.

Related Research• Programming approach

– Partially support user interactions for online CMO.

– LimSee(2002), Grins (2002), Yang (2001).

– All of which based on time line events

– which leads to dead times if the CMO is adaptable.

• Statistical techniques – Predict online user interactions

– Hollfelder et al. (2000) where they model the user behavior as a Continuous Time Markov chain (CTMC).

– Boll et al.(2001) implementing adaptive streaming of MPEG videos for interactive internet applications to support jumping to bookmarks within the MPEG

Framework• CMO layer comprises of the formal specification

of the CMO, such that specifying each components in the CMO along with its associated parameter and the relationships between different components.

• Petri Net layer is a conceptual tool for automatic manifestation, description, analyzing, visualization and validating of the CMO.

• SMIL layer is for implementation and automatic rendering purpose.

Why Petri net layer

• Ensure the satisfaction of different types of constraints.

– Synchronization constraints shall be satisfied to preserve the semantics of the CMO.

– Harder to satisfy synchronization constraints if different components resides on different agents over the network.

Why Petri net layer

• Conflict identification and resolution

– For example, a fidelity constraint to display an image only at a certain resolution may:

• Conflict with spatial constraints.

• Conflict with security constraints .

– Need to present all constraints in one model.

– Need to resolve identified conflicts

Why SMIL layer

• SMIL-based rich multimedia presentations integrate several types of media.

• SMIL is XML-based, thus flexible and extensible.

• SMIL is W3C supported, thus enjoys cross industry support.

• SMIL is supported by readily available tools and parsers.

Architecture

• The server comprises of an object manager that handles the requests from the clients.

• It may also provide tools to author multimedia objects.

• We represent the security, synchronization,fidelity and spatial constraints along with the playback duration of each component in the object plan.

Architecture

• Oblet A small piece of software installed on the client side, to support objects with ability to decide on their own renderings (manifestations)

• The object manifestation is comprised of object plan modification,object delivery and object rendition.

• Minimize the responsibilities of the server in order to avoid congestion and overloading at the server.

Research Issues

• Uniform Model• A Uniform model that incorporate the different types of constraints

• Easy to understand presentation.

• Enforcing Security Constraints• Automatic manifestation of the object based on user capabilities and

characteristics.

• Ensure consistent specification of the CMO.

• Secure the plan on the client side ( certificates, relational transducer).

Research Issues

• Incorporating User interactivity• Identify and model the different types of user interactions.• Make sure that different constraints are still valid.• Visualize and analyze the effect of the user interaction on different

constraints.

• Modality conversion• Determining minimal cost paths• convert some MM formats to others ( e.g.,text to audio) • Broker server for converting to different formats. • Route selection for converting the MM presentation• What will be the cost function?

• Conflict Identification and Resolution

Enforcing Security Constraints

• Petri Net based model – To model a multimedia object plan– Called MMCTPN (Multimedia color-time Petri-Net)

MMCTPN Model for a CMO plan

• MMCTPN (Multimedia color-time Petri-Net) consists of– Color-time Petri-Net

– Color sets that represent:

• Types of Fidelity (e.g.,the resolution of the MM component)– Absence of fidelity capabilities is represented as Holes

• Types of credentials (e.g., credentials for researchers)– Absence of credentials of the subject requesting the object is represented as

Holes• Types of Modality (e.g., image, video,text, audio)

– Absence of modality capabilities is represented as Holes

– Playback duration• The length of time the multimedia component is played.

– Null place in each subMMCTPN (A part of the net between two subsequent transitions)

MMCTPN

• Credentials and capabilities are represented as tokens and absence of them are represented as holes.

• The initial place is marked with these tokens to start with.

• Transitions are enabled once any token or hole is available.

• Transition t fires at time 0.

Before Firing

Pa , dur:10Pa , dur:10 Pa , dur:15Pa , dur:15

tt

After Firing

Pa , dur:10Pa , dur:10 Pa , dur:15Pa , dur:15

tt

MMCTPN – Object Manifestation

 

 

 

 

 

Dur = 10 S

AA

CC

 

Dur = 0 S

BB

P2

P1

Ps Pf

P5

P3

null

P9

P6

null

P4

P8

P7

null

<or,gy,br> dur :40s

<all> dur :40s

<re,gy,br> dur :40s

<ye,bc,br> dur :15s <ye,bc,br> dur :10s

<bl,wh,pp> dur :15s

<all> dur :15s

<bl,wh,pp> dur :15s

<gr,br,aq> dur :15s

<re,gy,br,aq> dur :15s

<bl,wh,br> dur :15s

<all> dur :15s

CMO Plan

<or,gy,br> dur :40s

P2

P1

Ps Pf

P5

P3

null

P9

P6

null

P4

P8

P7

null

<all> dur :40s

<re,gy,br> dur :40s

<ye,bc,br> dur :15s <ye,bc,br> dur :10s

<bl,wh,pp> dur :15s

<all> dur :15s

<bl,wh,pp> dur :15s

<gr,br,aq> dur :15s

<re,gy,br,aq> dur :15s

<bl,wh,br> dur :15s

<all> dur :15s

MMCTPN execution

Pf

P5

P3

null

P9

P6

null

P4

<re,gy,br> dur :40s

<ye,bc,br> dur :15s <ye,bc,br> dur :10s

<bl,wh,pp> dur :15s

<all> dur :15s

<bl,wh,pp> dur :15s

<bl,wh,br> dur :15s

<all> dur :15s

Modified CMO Plan

P

s

SMIL 2.0 Implementation

we have adopted the security model and the language (XACL) in specifying the security constraints on the multimedia object. We have used the XACL visual tool,tool from IBM XML Security Suite.

Incorporating User interactivity

• Identify and model the different types of user interactions.

• Make sure that different constraints are still valid.

• Visualize and analyze the effect of the user interaction on different constraints.

User interaction categories• Temporal Interaction:

– continuous component, ( ex. video )• ''play'', '‘fast forward''

– non-continuous component(ex. image) • "begin", "end", "pause".

• Spatial interaction:• "move “, "Zoom" ,” maximize”, "change font".

• Undo – Redo

• The user may want to interact with the CMO as one object, e.g., pause , begin,…

User Interaction example

• If a user wants to stop the fly-over video, what will be the effect on the following text? (rendering)– Appear immediately after stop.

– Appear after a defined amount of time.

• If a user wants to maximize the map, what will be the effect on rendering the following image? ( conflict)– Rendered behind the map.

– Rendered after another action.

• The CMO author need to:– Visualize

– Analyze

User Interaction Problems

How to model different types of user interaction on the composite multimedia object plan?

How can the author visualize and analyze the impact of potential user interaction on the rendering of the CMO?

How to detect potential conflicts between different types of constraints?

How to resolve different conflicts between constraints?

Security, Interaction, Modality,

Conflict resolution, Uniform Model

PETRI-NET Modeling for validation

and analysis

Implementation with

user/web friendly with flexibility

and interoperability (SMIL 2.0)