03 synchronized distance learning

Synchronized Distance Learning

Timothy K. Shih

Outline

• White Board and Chat Room– Designing Shared Whiteboard

• Multimedia Synchronization– Microsoft ASF– SMIL

• Synchronization Models– Petri Net– Interval Temporal Logic (optional)

White Board and Chat Room

• Chat Room– Almost Everywhere

• White Board– On-Line Annotation with Floor Control

Text

Acknowledgement

Timothy K. Shih and Yi-Chun Liao, MINE Lab, Tamkang University, Taiwan

• Text and simple graphic drawing• Annotation message broadcasting via multiple uni-casting• NetMeeting 3.0, COM (Component Object Model), ActiveX

control• Acknowledgement from Remote Stations is included• Floor Control: Control Modes, Channel, and Group

On-Line Annotation

Request Button

Request to SpeakErrorAcknowledgementQuestion Button

Speaking

Return Control

Interface and Demo

Demo

Annotation with Feedbacks

• 30 Client connections• Heterogeneous Network

Demo

Video

• The control of “Who to Speak” in a communication environment

• Concepts:– Channel: communication modal (text, audio, video, etc.)– Mode: communication style of a channel (Free Access vs. Equal

Right)– Group: the IP domain of a channel

• Example: 4 Channels, 3 Groups, 10 Persons, 2 Modes

Video Conferencing (ER) and Chat Room Tool (FA)

Chat Room Tool (FA)

Annotation Tool (ER)

Floor Control

Chat RoomAnnotation

Virtual PanoramaShared Notebook

Video

Audio

Video-on-Demand

Multi-modal and Multi-channel Communication

• Enhance the clearness of video presentation

• Add bitmapped animation to the video

Augmented Video Conferencing Tool

Augmented Video Presentation

Screen CaptureVideo Control

Video + Audio

MPEG-2

Using Direct Show

Implementation

• Configuration• Five Control

Modes– Normal– Resized

Screen– Resized Video– Full Video– Full Screen

• Audio and Video Devices and Codec

• Screen and CCD Camera Resolutions

Device and Layout Configuration

Video Control Modes

Normal Mode

Resized Screen Mode

Resized Video Mode

Full Video Mode

Full Screen Mode

• Problems to Solve– Distance Precision –

Space Discontinuity– Video Precision –

Eye Contact– Object Tracking

• The instructor• The whiteboard

– Interaction Protocol • Floor Control

– Performance

Challenges

DemoDemo

Multimedia Synchronization

• Why Multimedia Synchronization• Synchronization Methods

– Microsoft ASF– SMIL

• Synchronization Models (Theories)– Petri Net– Interval Temporal Logic

Video

Music

AudioSlide

MessageAnnotation

ASF Records

Timothy K. Shih, Y. R. Liu and Sheng-En Yeh, MINE Lab, Tamkang University, Taiwan

• Automatic Recording• Use Commercial Codec and the Advanced Streaming

Format (ASF) from Microsoft• Video and Slide Synchronization• Text and Drawing Annotation Synchronization• Slide Selection• Presentation Editing• Automatic Loading

Multimedia Synchronization Using ASF

• Event markers are embedded in a ASF file• Arbitrary activation of markers• Do not record the entire stream as one video clip• Video streaming depends on Codec

Video Stream

Text

User Interrupt User Interrupt User Interrupt

Slide Slide Slide

MS Office XP – Presentation Broadcast provides video and slide synchronization (PowerPoint and IE)

Event-based Synchronization

Choose Device

Choose Codec

Presentation Recording Tool

Select Output

Slide Selection

Presentation Playback Control Functions

Presentation Tool

• Hard to record the entire lecture without making a mistake

• Different presentation needs different level of details

• Lecture summarization and editing• Atomic presentation object – a PowerPoint

Slide (with its video and annotation events)

Text

AtomicObject

Adaptive Presentation

Three Levels of Details

Automatic Loading Presentation Editing

Multi-Level Presentation

Demo

Video Demonstration

Synchronized MultimediaIntegration Language

(SMIL)• Recommended by W3C• XML-based language• Designed for Internet use• Easy to write similar to HTML• Integrates several media formats• Plain text documents with a “.smi” or

“.smil” suffixPart of slides about SMIL are summarized from: http://service.real.com/help/library/guides/production/realpgd.htm

History

• Dec 1995 - SMIL designing started• Nov 1997 - SMIL 1.0 Recommendation• Mar 1998 - First SMIL implementation• Aug 1999 - First public draft of SMIL 2.0• Aug 2001 - SMIL 2.0 Recommendation

Support for SMIL in Players / Browsers

2.0 • X-Smiles 0.40: SMIL 2.0 Basic Profile2.0 • Oratrix GRiNS player: SMIL 2.0 Language2.0 • IE 5: many SMIL 2.0 modules1.0 • QuickTime Player 4.11.0 • RealPlayer 81.0 • Compaq HPAS1.0 • Productivity Works Lp Player1.0 • SOJA from Helio1.0 • S2M2 by NIST1.0 • Schmunzel by SunTREC Salzburg

Use Cases for SMIL

• Slideshows• Advertisements• Internet TV• Education• Corporate communications• Product information• User’s Guides• Net meetings• etc.

General Rules

• Documents start with <smil> and end with </smil>

• <head> section, which is optional; and a <body> section which is required

• Tags and Attributes in lowercase only• All elements must have closing tags or /• Attribute values must be enclosed in double-

quotes• HTML-style comments are allowed.

–

Typical SMIL Elements

<head> - Head element

<meta> - Meta data, such as author, copyright...

<layout> - Layout of the presentation

<region> - Region describing positioning

<body> - Body element

<seq> - Sequential time container

<par> - Parallel time container

<img>, <audio>, <video> - Media to be played

<switch> - Selects between elements

<a>, <area> - Linking

A SMIL Example<smil>

<head> <layout> <region id=”reg1” top=”0” left=”0” width=”300” height=”200” /> <region id=”reg2” left=”300” width=”300” height=”200” /> </layout></head><body>

<seq> <img id=”id1” src=”intro.jpg” region=”reg1” dur=”4s” /> <audio id=”id2” src=”music.wav” dur=”2s” /> <par> <video id=”id3” src=”movie5s.mpg” region=”reg1” /> <img id=”id4” src=”intro.jpg” begin=”3s” region=”reg2”

/> </par></seq>

</body></smil>

<head> element

• Information not related to the temporal presentation of the material

• May contain the following children– <meta>– <switch>– <layout>

<layout> element

• Probably the most important element• Specify the layout of the document, as

well as the layout of the multimedia elements in the document

• If not specified, how document looks will depend on the player

• Two children allowed here– <root-layout>– <region>

<root-layout> element

• An empty element• Defines the size of the main window• You cannot specify media elements in

here• There can only be one <root-layout>

<root-layout> attributes

• background-color– Can use named or hexadecimal values.

• height, width– All specified in pixels

• title

<region> element

• Areas within <root-layout> in which media elements will appear

• May define as many as you want• Very similar to frames in HTML• Are allowed to overlap — controlled by z-

order• Any regions that lay outside the <root-

layout> will be cut off

Required attributes

• id– A unique text name for the region

• top, left, width, height– Pixels or percentages or mixed

Example

• <head> <root-layout width="480" height="300"/> <region id=“someid" top="60" left="120" width="240" height="180"/></head>

Screen

(60, 120)

240

180

<body> element

• Information related to the temporal and linking behavior of the document

• Children include– Synchronization elements

• <par>, <seq>

– Media elements• <animation>, <audio>, <img>, <text>, <textstream>,

<video>, <ref>

Synchronization elements

• <par> element — children of a <par> element occur in parallel

• <seq>element — children of a <seq> element appear in sequential order

• Valid children of <seq> and <par> elements include– <seq> & <par>– Any of the media elements

Combining <seq> and <par> tags

<seq>clip 1<par>

clip 2

clip 3</par>clip 4

</seq>

<par>clip 1<seq>

clip 2

clip 3</seq>clip 4

</par>

12

34

3

1

2

4

Media elements

• The various pieces of multimedia that go into the SMIL document

• Continuous media– <animation>, <audio>, <textstream>, <video>– Have an intrinsic duration

• Discrete media– <text>, <img>– Have no intrinsic duration

• The actual tag name doesn’t matter• Type attribute is used to determine media type• Could use <ref> tag

Media element attributes

• All media elements have certain attributes in common

• You use these attributes to specify how the children of <seq> and <par> elements will appear in the presentation

Specifying media and location

• src– Specifies the source of the media element– Identical to the src attribute of the <img> tag

in html• region

– The area of the root-layout in which the media element is to be displayed

– e.g., <video src="hal.mov" region="video_region" />– If you don’t specify a region, it’s

implementation dependent

Setting begin and end time

• begin & end attributes• Specifies the explicit beginning and

ending time of the element– e.g., <video src="hal.mov" begin="10.5s" end="45.5s"

/>

• Relative to parent element regardless of own length

• May use with <par> and <seq> elements to affect entire group

Setting internal begin & end times

• clip-begin & clip-end attributes• Specifies the internal beginning and

ending time of a clip– E.g., <video src="hal.mov" clip-begin="2.5s" clip-

end="15.8s" />

• May combine with begin and end attributes

Setting a duration

• dur attribute• Explicit duration of the media element• Time value, or “indefinite”• Should not use dur with end attributes

Filling up time

• fill attribute– Specifies what happens to the clip when it is

done– Freeze or remove

• repeat– Specifies how many times the clip should

repeat itself– Any integer is valid– Can add repeat attribute to the <par> and

<seq> elements

Synchronizing elements

• By default, <par> elements end when all elements finish

• Can be modified with the end attribute to force an end time for the <par> element

• Or you can use the endsync attribute of the <par> element to synchronize the ending of its clips

• Three values for the endsync attribute– first, last, id(element id)

Linking• SMIL provides two ways to link to other

documents• <a> tag

– Identical to the <a> tag in html– Specify an href attribute with a URL– Can surround and turn an entire media

element into a link• <anchor> tag

– Defines a “hot spot” inside a media element – Similar to HTML image maps

<anchor> tag

• Specified as a child of a media element– <video src="video.rm" region="video_region">

<anchor href="url here" coords="20, 40, 80, 120">

</video>

• Where the coords attribute specifies the location within the media element that is the hot spot

• May use pixels or percentages

Control Structure

• SMIL provides a <switch> tag• Can use it to allow player to choose one

of several test-attributes• Format

<switch> <choice1 test-attribute="value1"/> <choice2 test-attribute="value2"/> …</switch>

• Various test-attributes have been defined

Using <switch> for language

• You can use the <switch> tag to allow player to choose the appropriate language

• <par> <video src="video/seattle.rm" /> <switch> <audio src="french/seattle.rm" system-language="fr"/> <audio src="german/seattle.rm" system-language="de"/> <audio src="spanish/seattle.rm" system-language="es"/> <audio src="english/seattle.rm"/> </switch></par>

Using <switch> for system bitrates

• <switch> <par system-bitrate="75000">  <audio src="audio/newsong1.rm"/> <video src="video/newsong1.rm"/> </par> <par system-bitrate="4700">  <audio src="audio/newsong2.rm"/> <video src="video/newsong2.rm"/> </par> <par system-bitrate="20000">  <audio src="audio/newsong3.rm"/> <video src="video/newsong3.rm"/> </par></switch>

For information

• W3C’s Synchronized Multimedia Group– http://www.w3.org/AudioVideo

• Just SMIL– http://smw.internet.com/smil

Petri Net

• What are Petri Nets– A tool for system study

• System Modeling– Structure– Dynamic Behaviors

• System Evaluation

Source: Feng-Cheng Yang

These slides are modified based on Feng-Cheng Yang’s presentation

Using Petri Net

• Modeling– Representation for a System– Often in Mathematical Terms– Manipulated to reveal information

• Example of Discrete System Analysis– Bank System

• Teller Components– Idle state– Busy state

• Customer Components– Idle state (waiting in the line)– Busy state (being served)

Multimedia PresentationSMIL<par>

clip 1<seq>

clip 2

clip 3</seq>clip 4

</par>3

1

2

4

Petri Net (simplified)

3

1

2

4

Features of a System

• Components– Subsystems– Separated and Interacting

• States– Depend on the past history– Change over time

System Behaviors

• Concurrency (Parallelism)– Activities of different components occur

simultaneously– Difficult to model

• Sequencing– Serial activities occur one by one

between components– Easy to model

History

• Basic concept was created in 1962 by Carl

Adam Petri in his doctoral thesis

(W. Germany)– Communication between asynchronous

components of a computer system

• Early theory, notation, and representation

are developed by the Information System

Theory Project (Applied Data Research, US)– Conduct by A.W. Holt (1968)

History (cont.)

• Considerable research and publication in 1970– Directed by Prof. Jack B. Dennis at M.I.T.

• Conference on Petri Nets and Related Methods in 1975 at M.I.T.

• Workshop on Petri Nets held in Paris in 1977• In 1979 an advanced course on General Net

Theory is held in Hamburg• A Special Interest Group on Petri nets was

formed in Germany

Petri Net Structure (Mathematical Model)

• Four parts– P: a set of places– T: a set of transitions– I: an input function

• A mapping from a transition tj to a collection

of places I(tj )• The places are input places of tj

– O: an output function• A mapping from a transition tj to a collection

of places O(tj )• The places are output places of tj

Pi

tj

Pi

tj

Definition 1

• A Petri net structure is a four-tuple:C = (P,T,I,O)

• P = { p1, p2,…, pn }, n ≥ 0• T= { t1, t2,…, tm }, m≥0• P∩T = ∅• I:T→P∞, a mapping from transitions to bags

of places• O:T → P∞, a mapping from transitions to

bags of places

Petri Net Representation

• A place pi is an input place of a transition tj

if pi ∈ I(tj)• A place pi is an output place of a transition tj

if pi ∈ O(tj)• A bag is a generalization of sets which

allows multiple occurrences of an elements

in a bag.– E.g.: O(tj) = { p1, p1, p1, p2, p3, p4, p4, p5 }

Pi

tj

Pi

tj

Example 1

P2

P4P1 P5

P3

t1 t2

t3

t4

Example 1

• C = ( P,T,I,O )• P = { p1 , p2 , p3 , p4 , p5 }, n = 5• T = { t1 , t2 , t3, t4 }, m = 4• I(t1 ) = { p1 } O(t1 ) = { p2 , p3 , p5 }• I(t2 ) = { p2 , p3 , p5 } O(t2 ) = { p5 }• I(t3 ) = { p3 } O(t3 ) = { p4 }• I(t4 ) = { p4 } O(t4 ) = { p2 , p3 }

P2

P4P1 P5

P3

t1 t2

t3

t4

Multiplicity

• #(a, A) : The number of occurrences of

element a in bag A.• The multiplicity of an input place pi for a

transition tj is #( pi, I(tj) )• A transition tj is an input of a place pi, if pi is

an output of tj • A transition tj is an output of a place pi, if pi

is an input of tj

Pi

tj

Pi

tj

# = 3

Definition 2• Input and output functions are extended as

follows:– I:T→P∞, O:T → P∞ , such that

• #( tj, I(pi) ) = #( pi, O(tj) ) and• #( pi, I(tj) ) = #( tj, O(pi) )

Pi

tj

Pi

tj

Example 2

P2

P4P1

P5

P3

t1

t2

t3

t4

P6

t5

Example 2

• C = ( P,T,I,O )• P = { p1 , p2 , p3 , p4 , p5 , p6 }, n = 6• T = { t1 , t2 , t3, t4, t5 }, m = 5• I(t1 ) = { p1 } O(t1 ) = { p2 , p3 }• I(t2 ) = { p3 } O(t2 ) = { p3 , p5 , p5 }• I(t3 ) = { p2, p3 } O(t3 ) = { p2, p4 }• I(t4 ) = { p4 , p5 , p5 , p5 } O(t4 ) = { p4 }• I(t5 ) = { p2 } O(t5 ) = { p6 }

P2

P4P1

P5

P3

t1

t2

t3

t4

P6

t5

Example 3

P2

P4

P1

P5

P3

t1

t2

t3

t4

P6

t5P7

P8

P9t6

Example 3

• C = ( P,T,I,O )• P = { p1 , p2 , p3 , p4 , p5 , p6 , p7 , p8 , p9 }, n = 9• T = { t1 , t2 , t3, t4, t5 , t6 }, m = 6• I(t1 ) = { p1 } O(t1 ) = { p2 , p3 }• I(t2 ) = { p8 } O(t2 ) = { p1 , p7 }• I(t3 ) = { p2 , p5 } O(t3 ) = { p6 }• I(t4 ) = { p3 } O(t4 ) = { p4 }• I(t5 ) = { p6 , p7 } O(t5 ) = { p9 }• I(t6 ) = { p4 , p9 } O(t6 ) = { p5 , p8 }

P2

P4

P1

P5

P3

t1

t2

t3

t4

P6

t5P7

P8

P9t6

Petri Net Graphs

• Bipartite directed multigraph• Vertices:

– Circles (places) and Bars (transitions)• Links:

– Directed arcs (represent the relationsbetween transitions and places)• Represented by functions I and O

• Multiple arcs directed from a place to a transition is allowed

Definition 3

• A Petri net graph G is a bipartite directedmultigraph, G = ( V, A ), whereV = { v1, v2, …, vs } is a set of vertices andA = {a1, a2, …, ar} is a bag of directed arcs,ai = (vj , vk ), with vj, vk ∈ V

• The set V can be partitioned into twodisjoint sets P and T such that V = P ∪ T,P ∩ T = ∅ , and for each directed arc,ai ∈ A, if ai = (vj , vk ), then eithervj ∈ P and vk ∈ T or vj ∈ T and vk ∈ P

Definition 4

• Define V = P ∪ T.• Define A as a bag of directed arcs such that

for all pi ∈ P and tj ∈ T ,– #( (pi , tj ), A ) = #( pi , I(tj) ) and– #( (tj , pi ), A ) = #( pi , O(tj) )

• G = ( V, A ) is a Petri net graph which is

equivalent to the Petri net structure

C = ( P, T, I, O )

The Dual of a Petri Net

• The dual of a Petri net C = ( P, T, I, O ) is the Petri net C’ = (T, P, I, O ) which results from interchanging places and transitions

The Dual of Example 1

P2 P4P1

P3

t1 t5

t3

t4

t2

P2

P4P1 P5

P3

t1 t2

t3

t4

The Inverse Petri Net

• The inverse Petri net for a Petri net C = ( P, T, I, O ) is defined by interchanging the input and output functions,

-C = ( P, T, O , I ).

The Inverse of the Dual of Example 1

P2 P4P1

P3

t1 t5

t3

t4

t2P2 P4P1

P3

t1 t5

t3

t4

t2

Petri Net Markings

• A marking μ is an assignment of tokens to the places of a Petri net

• The number and position of tokens may change during the execution of a Petri net

• The tokens are used to define the execution of a Petri net

Definition 5

• A marking μ of a Petri net C = ( P,T,I,O ) is a

function from the set of places P to the

nonnegative integers N; μ : P → N• The marking can also be defined as an n-vector,

μ = [ μ1 μ2 …μn ]• The number of tokens in place pi is μi, i =

1, …, n– μ(pi ) = μi

Marked Petri Nets

• M = ( C, μ ) is a Petri net structure and a marking μ– M = (P,T,I,O, μ )

• Tokens are represented by small dots in thecircles which represent the places of a Petrinet

• The number of tokens assigned to a placeis unbounded

• The number of possible markings of a Petrinet is infinitive

A Marked Petri Net of Example 1

• The marking μ = [ 1 2 0 0 1 ]

t1 t2

t3

t4

P1 P5

P2

P4

P3

Another Marked Petri Net of Example 1

• The marking μ = [ 0 2 0 5 1 ]

t1 t2

t3

t4

P5

P2

P4P1

P3

Execution Rules for Petri Nets

• Tokens reside in places and control theexecution of the transitions of the net

• A Petri net executes by firing transition• A transition fires by removing tokens from

its input places and creating new tokenswhich are distributed to its output places– The tokens are not moving from one place to

the next place– They are consumed by the transition and

regenerated from the transition to the nextplace

• A transition may fire if it is enabled

Execution Rules for Petri Nets (Conti.)

• A transition is enabled if each of its inputplaces has at least as many tokens in it asarcs from the place to the transition

• The tokens in the input places which enablea transition are its enabling tokens

• A transition fires by removing all of itsenabling tokens from its input places andthen depositing into each of its outputplaces one token for each arc from thetransition to the place

Definition 6

• A transition tj ∈ T in a marked Petri net

C = ( P,T,I,O ) with marking μ is enabled if

for all places pi , μ(pi) ≥ #( pi, I(tj) )

tj 1 token ≥ one arc

3 tokens ≥ three arcs

2 tokens ≥ two arcs

Definition 7

• A transition tj in a marked Petri net withmarking μ may fire whenever it is enabled

• Firing an enabled transition tj results in a new marking μ’ defined by

• μ’(pi ) = μ (pi ) - #(pi , I(tj) ) + #(pi , O(tj) )

tj

Pi

tj

Pi

Remove 2 tokens (for 2 arcs)Remove 3 tokens (for 3 arcs)

Add 1 token (for 1 arc) Add 1 token (for 1 arc)

Firing Examples

• Example a:– I(t3 ) = { p2} O(t3 ) = {p7, p13 }– t3 is enabled whenever there is at least one

token in place p2• Transition t3 fires by removing one token

from p2 and depositing one token in place p7 and one token in place p13

t3 t3

P2

P7

P13

P7

P13P7P2

Firing Examples (Cont.)

• Example b:– I(t2 ) = {p21, p23} O( t2 ) = {p23, p25, p25 }– t2 is enabled whenever there is at least one token in place

p21 and at least one token in place p23– Transition t2 fires by removing one token from p21 and one

token from p23 and then deposits one token in place p23 and two tokens in place p25

t2 t2

P21

P23

P25

P23

P25

P21

Firing Examples (Cont.)

• t1 , t3 , and t4 are enabled

t1 t2

t3

t4

P4

P1

P3

P5P2

• t4 is fired

t1 t2

t3

t4

P4

P1

P3

P5P2

• t1 is fired

t1 t2

t3

t4

P4

P1

P3

P5P2

• t3 is fired

t1 t2

t3

t4

P4

P1

P3

P5P2

Summary

• Petri Net Definitions• Petri Net Examples

• For multimedia presentations– Transitions are used for synchronization– Places can represent multimedia resources

• Related to Interval Temporal Logic

3

1

2

4

Multimedia Presentation Based on Petri Net

• Defining a operation model of multimedia presentation by features of Petri Net

• Constructing an algorithm based on Timed Petri Net for multimedia presentations

Background

• Petri net is a graph and a mathematical tool• Multimedia presentations includes timing

control and user interaction• Describe multimedia objects by Timed Petri

Net• Create new definitions based on Petri Net for

interactions• Implement a tool for multimedia presentation

designs

An operation example of Petri net

Place

Arc

TokensTransition

Arc

Arc

Place

Place

Tokens

Place

Place

PlaceArc

Arc

Arc

Transition

Token

Token

Token

(a) A initial marking before transition firing (b) A new marking after transition firing

Interpretations of places and transitions

Tansitions AttributesInput Places Output Places

Input Places Transition Attributes Output Places

Data Processing Information

Request Available Service

Resource Permission Approach

Precondition Event Postcondition

Multimdia Timing Playing

Two special cases in Petri net

(a) A sink transition of Petri Net

(End of Presentation)

(b) A source transition of Petri Net

(Begin of Presentation)

Self-loop and Weights

1

1

1

2

1

(a) A self_loop of Petri Net (b) Weights of synchronous arcs of Petri Net

New Components for Multimedia Objects

: Place and token is for playing a multimedia resource.

: User transition is for accepting a message and causingthe activation of connected transitions.

: User arc is for connecting from a user transition to atransition.

: Synchronous arc is for connecting from a place to atransition or from a transition to a place.

: Transition is for synchronous control.

New

New

Example of Multimedia Petri net

image

text

video

sound

music

Syn. arc

Syn. arc

Syn. arc

Syn. arc

Syn. arc

Syn. arc

Syn. arc

Syn. arc

Syn. arc

Syn. arc

User arc

User arc

navigationmessage

navigationmessage

user transition

transition

transition

transition

transition

Definitions of place, transition and user transition in the Multimedia Petri net

Object attributes

File format

Duration

File name

Execution tool

File attributes

Place node

Transition

User Transition

Attributes

Messages

Timing

Object relations

Interrupt

Pause

Stop

Change

Algorithm: Compute_Node_Time

• Construct a Transition_Graph reachable from one of the starting • Navigation_Messages;

• Initialize the node pointed by the Navigation_Message to • Node_Time = 0;

• Sort the Transition_Graph to a Transition_List by Topological sort;

• For each node N in the Transition_List, except the first, • Begin• For each incoming edge, compute the Estimated_Time = • Node_Time + Edge_Duration;• Set Node_Time of node N to the maximum of all Estimated_Time • of incoming edges;• End

Algorithm: Run_Petri_Net– Begin– For each node N in the Transition_List Do Begin– Collect places pointed by Sync_Arcs from node N in Sync_Set(N);– Sort the Transition_List according to Node_Time in a non-– decreasing order;– End– CoBegin– Process1:– For each node N in the Transition_List Do Begin– Sequentially play resources in Sync_Set(N) at Node_Time(N) – concurrently;– End– Process2:– LoopBegin– If Process1 ends, Then Process2 ends;– If there is a Navigation_Message of a User_Transition Then– Exits with the next starting Navigation_Message;– If there is a Selection, Assignment, or Condition Then– Perform the action accordingly;– LoopEnd– CoEnd– End

– Algorithm: Run_Presentation– Begin– LoopBegin– If there is a starting Navigation_Message Then– Compute_Node_Time;– Run_Petri_Net;– Else– End Presentation– LoopEnd– End

A initial graph of Multimedia Petri Net

Text

Video

Wave

Sound

Bmp

Midi

Transition graph for simplified Multimedia Petri Net

Sound

Bmp

Text

Video

User messageWave

User message

Midi

An example for the Compute-Node-Time algorithm

70

35

20

1515

20

40

30

20

ab

c

d e

Topological sort of the transition nodes

a

c

d

e

b

20

70

152030

3515

40

20

A Multimedia Presentation Example

sound

text

musicanimation

video

image

a

b

c

d

Presentation Schedule of the Example

Animation

Sound

Text

Music

Image

Video

TimeTransition "a" Transition "b" Transition "c" Transition "d"

User transition

Synchronous Sets for the Order of Transitions Firing

a

b

c

d

b c

d

Case1 : Without firing of user transition

: {animation,sound,text}

: {music,image,text}

: {music,video}

: End

Case2 : With firing of user transition

: {music,image,video}

: End

Layered Multimedia Presentation

Subnet

Subnet

Subnet

Transition Transition

User Transition

Implementation Example

Implementation Example (cont’)

Summary

• Provide a model for the multimedia presentation design by extending concepts of Petri Net

• Could be extended for layered multimedia presentations

• Petri Net is used as an example of synchronization control

Collaborative Learning with Petri Net

• Different students learn different contents and in different speed

• Team members have to learn skill and gathered together to solve a mission

start

C

B

A

Dispatch Discuss Grade

The Same Structure but Different Skill

Learner A

Cooperative Learning

Learner B

Learner C

L2L1

L3 L4

L1,L3,L4

L1,L2,L3,L4

L1,L2,L3,L4

Capability & AssessmentLearner A

Learner B

Learner C

L1,L3,L4

L1,L2,L3,L4

Exam

Give different color tokens

condition

Learner D

L1,L2,L3,L4

L1,L2,L3,L4

Learner E

L1,L3,L4

C.L.

Cooperative Learning Environment

Enter

Communication

Vote

Suspend

ExitExam

Interval Temporal Logic• Temporal Intervals• Fast Interval Relation Composition• Solving the Composition Conflicts• Interval Relation Distances• Qualitative Spatial Knowledge• The Applications

• James F. Allen, "Maintaining Knowledge about Temporal Intervals," Communications of the ACM, Vol. 26, No. 11, 1983.

• Relations between two time intervals• Relation composition• Can be used in many research areas

Temporal Intervals

Allen’s 13 Temporal Interval Relations

Composition of the 13 Interval Relations

From Point Relations to Interval Relations

Revised 18 Temporal Interval Relations

Fast Interval Relation Composition

• Relation composition involves set operations• The amount of relation sets is finite

• Table13

The Relation Composition Algorithm

Algorithm : RelComp

Input : rs1 29RelSet, rs2 29RelSet

Output : rs 29RelSet

Preconditions : true

Postconditions : true

Steps :

1. rs = r1rs1 , r2rs2 (r1 , r2) rs1 rs2

Table13 (r1 , r2)

Computing All Possible Relations

Algorithm : ComputeTable29

Input : Table13

Output : Table29

Preconditions : true

Postconditions : relation composition is closed under I

Steps:

1. Construct a set of 13 atomic sets from the 13 relations, assuming that this set is called I,

which is an index set for table look up.

2. Let Table29( i, j ) = Table13( i, j ), i I, j I

3. Table29( i, j ), iI , j I , do

3.1: if k = Table29( i, j ) I then

3.1.1 : I = I Table29( i, j )

3.1.2 : m I ,do　 3.1.2.1 Table29( k, m ) = Relcomp( k, m )

3.1.2.2 Table29( m, k ) = Relcomp(m, k )

The 29 Possible Relation Sets

IDs Relation Sets IDs Relation Sets

------------------------------------------------------------------------------------------

1 { < } 16 { o, d ,s }

2 { > } 17 { oi, di, si }

3 { d } 18 { <, o, m }

4 { di } 19 { >, oi, mi }

5 { o } 20 { f, fi, e }*

6 { oi } 21 { s, si, e }*

7 { m } 22 { <, o, m, d, s }

8 { mi } 23 { >, oi, mi, di, si }

9 { s } 24 { <, o, m, di, fi }

10 { si } 25 { > , oi, mi, d, f }

11 { f } 26 { o, oi, d, di, s, si, f, fi, e }*

12 { fi } 27 { <, m, d, di, o, oi, f, fi, s, si, e }

13 { e }* 28 { > , mi, di , d, oi, o, fi, f , si, s, e }

14 { o, di, fi } 29 { < , > ,m, mi, di , d, oi, o, fi, f , si, s, e }*

15 { oi, d, f }

rs rs-1 rs rs-1

-------------------------------------------

1 2 18 19

3 4 20 20

5 6 21 21

7 8 22 23

9 10 24 25

11 12 26 26

13 13 27 28

14 15 29 29

16 17

The Inverse Relations

Table29 o | 01 02 03 04 05 06 07 08

09 10 11 12 13 14 15 1617 18 19 20 21 22 23 2425 26 27 28 29

-------------------------------------------------------------------------------------------------------------------------------------------01 | 01 29 22 01 01 02 01 22

01 01 22 01 01 01 22 2222 01 29 22 01 22 29 0129 22 22 29 29

02 | 29 02 25 02 25 02 25 0225 02 02 02 02 25 25 2502 29 02 02 25 29 02 2925 25 29 25 29

03 | 01 02 03 29 22 25 01 0203 25 03 22 03 29 25 2229 22 25 22 25 22 29 2925 29 29 29 29

04 | 24 23 26 04 14 17 14 1714 04 17 04 04 14 26 2617 24 23 17 14 27 23 2428 26 27 28 29

05 | 01 23 16 24 18 26 01 1705 14 16 18 05 24 26 2227 18 28 22 14 22 29 2428 27 27 29 29

06 | 24 02 15 23 26 19 14 0215 19 06 17 06 28 25 2623 27 19 17 25 27 23 2925 28 29 28 29

07 | 01 23 16 01 01 16 01 2007 07 16 01 07 01 16 2222 01 28 22 07 22 29 0128 22 22 29 29

08 | 24 02 15 02 15 02 21 0215 02 08 08 08 25 25 1502 27 02 08 25 27 02 2925 25 29 25 29

09 | 01 02 03 24 18 15 01 0809 21 03 18 09 24 15 2227 18 25 22 21 22 29 2425 27 27 29 29

10 | 24 02 15 04 14 06 14 0821 10 06 04 10 14 15 2617 24 19 17 21 27 23 2425 26 27 28 29

11 | 01 02 03 23 16 19 07 0203 19 11 20 11 28 25 1623 22 19 20 25 22 23 2925 28 29 28 29

12 | 01 23 16 04 05 17 07 1705 04 20 12 12 14 26 1617 18 23 20 14 22 23 2428 26 27 28 29

13 | 01 02 03 04 05 06 07 0809 10 11 12 13 14 15 1617 18 19 20 21 22 23 2425 26 27 28 29

14 | 24 23 26 24 24 26 24 1714 14 26 24 14 24 26 2727 24 28 27 14 27 29 2428 27 27 29 29

15 | 24 02 15 29 27 25 24 0215 25 15 27 15 29 25 2729 27 25 27 25 27 29 2925 29 29 29 29

16 | 01 23 16 29 22 28 01 2316 28 16 22 16 29 28 2229 22 28 22 28 22 29 2928 29 29 29 29

17 | 24 23 26 23 26 23 14 2326 23 17 17 17 28 28 2623 27 23 17 28 27 23 2928 28 29 28 29

18 | 01 29 22 24 18 27 01 2718 24 22 18 18 24 27 2227 18 29 22 24 22 29 2429 27 27 29 29

19 | 29 02 25 23 28 19 28 0225 19 19 23 19 28 25 2823 29 19 23 25 29 23 2925 28 29 28 29

20 | 01 23 16 23 16 23 07 2316 23 20 20 20 28 28 1623 22 23 20 28 22 23 2928 28 29 28 29

21 | 24 02 15 24 24 15 24 0821 21 15 24 21 24 15 2727 24 25 27 21 27 29 2425 27 27 29 29

22 | 01 29 22 29 22 29 01 2922 29 22 22 22 29 29 2229 22 29 22 29 22 29 2929 29 29 29 29

23 | 29 23 28 23 28 23 28 2328 23 23 23 23 28 28 2823 29 23 23 28 29 23 2928 28 29 28 29

24 | 24 29 27 24 24 27 24 2724 24 27 24 24 24 27 2727 24 29 27 24 27 29 2429 27 27 29 29

25 | 29 02 25 29 29 25 29 0225 25 25 29 25 29 25 2929 29 25 29 25 29 29 2925 29 29 29 29

26 | 24 23 26 29 27 28 24 2326 28 26 27 26 29 28 2729 27 28 27 28 27 29 2928 29 29 29 29

27 | 24 29 27 29 27 29 24 2927 29 27 27 27 29 29 2729 27 29 27 29 27 29 2929 29 29 29 29

28 | 29 23 28 29 29 28 29 2328 28 28 29 28 29 28 2929 29 28 29 28 29 29 2928 29 29 29 29

29 | 29 29 29 29 29 29 29 2929 29 29 29 29 29 29 2929 29 29 29 29 29 29 2929 29 29 29 29

Solving the Composition Conflicts

• Unknown derivation If X < Y and Y > Z, then X ? Z

• Multiple derivationIf X < Y and Y d Z, then X {<, o, m, d, s} Z

• Conflict derivationIf X < Y, Y < Z, and X > Z are declared by the

user,

then there exists a conflict

The Domain of Relations

• An user edge denotes a relation between a pair of objects defined by the user. The relation may be reasonable or non-reasonable.

• A derived edge holds a non-empty set of reasonable relations derived by our algorithm. The relation of the two objects connected by the derived edge can be any reasonable relation in the set.

• complete relation domain (a complete graph): contains user edges and derived edges, with possible cycles and possible conflicts.

• reasonable relation domain (a graph): contains user edges and derived edges, with possible cycles but no conflict.

• reduced relation domain (a graph): contains only user edges, with possible cycles and possible conflicts.

• restricted relation domain (a tree): contains only user edges, without cycle.

Relation among Domains

Computes the Reasonable RD from the Reduced RDAlgorithm : ComputeRD1Input : G = ( GV,GE )

Output: Kn = ( KnV, KnE )Preconditions : true

Postconditions : GV = KnV GE KnESteps : 1 : G = EliminateConflicts (G)

2 : Kn = G pl = 2

3 : repeat until | KnE | = | KnV | * ( | KnV | -1 ) / 2

3.1 : for each e = ( a, b ) e KnE a KnV b KnV such that there is a path of user edges from a to b , with path length = pl

3.2 : suppose (( n1, n2 ), ( n2, n3 ),…, ( nk-1, nk ) )

is a path with a = n1 b = nk k = pl + 1

3.3 : set e.rs = Table29 (( a, nk-1 ).rs, (nk-1, b ).rs )

3.4 : KnE = KnE { e } 3.5 : pl = pl + 1

Eliminate Conflicts

Algorithm : EliminateConflicts Input : G = ( GV, GE )Output : G = ( GV, GE)Preconditions : G contains only user edges G = G Postconditions : G = G , but the reasonable sets of edges in G may be

changed.Steps :

1. for each P = ((n1, n2), (n2, n3),, (nk-1, nk) ) in G with n1 = nk k >3 1.1 : for each i, 1 i k-2

1.1.1 : set (ni, ni+2).rs = Table29 ((ni, ni+1).rs, (ni+1, ni+2).rs )

1.2 : rs = Table29 ((nk, nk-2).rs, (nk-2, nk-1).rs )

1.3 : if (nk, nk-1).r rs then 1.3.1 : ask user to choose a r rs 1.3.2 : set (nk, nk-1).r = r

An Example

User Edges and Derived Edges

User edges:

( A, B ) = { < } = [1]

( B, C ) = { m } = [7]

( C, D ) = { d } = [3]

( C, E ) = { s } = [9]

( F, D ) = { < } = [1]

2. Path Length = 3

(A, E) = (A, B ) o (B, C) o (C, E) = (A, C ) o (C, E)

= [1] o [9] = [1] = { < }

(A, D) = (A, B ) o (B, C) o (C, D) = (A, C ) o (C, D)

= [1] o [3] = [22] = { <, o, m, d, s }

(B, F) = (B, C ) o (C, D) o (D, F) = (B, D ) o (D, F)

= [16] o [1]-1 = [23] = { >, oi, mi, di, si }

(E, F) = (E, C ) o (C, D) o (D, F) = (E, D ) o (D, F)

=[14] -1 o [2] = [15] o [2] = [2] = { > }

3. Path Length = 4

(A, F) = (A, B ) o (B, C) o (C, D ) o (D, F)

= ( (A, B ) o (B, C) ) o ( (C, D ) o (D, F) )

= (A, C ) o (C, F) = [1] o [2] = [29]

= { <, >, d, di, o, oi, m, mi, f, fi, s, si, e }

Derived edges:1. Path Length = 2

(A, C) = (A, B ) o (B, C) = [1] o [7] = [1] = { < }

(B, D) = (B, C ) o (C, D) = [7] o [3] = [16] = {o, d, s }

(C, F) = (C, D ) o (D, F) = [3] o [1]-1 = [3] o [2] = { > }

(D, E) = (D, C ) o (C, E) = [4] o [9] = [14] = {o, di, fi }

(B, E) = (B, C ) o (C, E) = [7] o [9] = [7] = { m }

Adds User Edges to the Reasonable RD

Algorithm : AddUERD

Input : l = (a, b ), Kn = ( KnV, KnE )

Output : Kn+1 = ( Kn+1V, Kn+1E )

Preconditions : l KnE a KnV b KnV

Postconditions : | Kn+1V | = | KnV | + 1 | Kn+1E | = | KnE | + n Steps :

1: Kn+1E = KnE { l }2: for each e = ( c, b ) c a c KnV

2.1: e.rs = d KnV, ( c, d ) KnE, (d, b ) KnE ( Table29(( c, d ).rs, ( d, b ).rs ))

2.2: Kn+1E = Kn+1E {e}

3: Kn+1V = KnV { b }

Adding Edges

Adding an user edge to Example 4.1:

Add ( G, E ) = { f } = [11]

Derived edges :

Derive ( E, G ) = ( G, E )-1 = [11]-1 = [12] = { fi } ( inverse )

Derive ( A, G ) = ( A, E ) o ( E, G ) = [1] o [12] = [1] = { < } ( association )

Derive ( B, G ) = ( B, E ) o ( E, G ) = [7] o [12] = [1] = { < } ( association )

Derive ( C, G ) = ( C, E ) o ( E, G ) = [9] o [12] = [18] = {<, o, m }

Derive ( D, G ) = ( D, E ) o ( E, G ) = [14] o [12] = [24] = { <, o, m, di, fi } ( association )

Derive ( F, G ) = ( F, E ) o ( E, G ) = ( E, F )-1 o ( E, G ) = [1] o [12] = [1] = { < } ( inverse, association )

Interval Relation Distances

• Use relation distances to estimate the similarity of object positions

• There are three types of relation distances– The Point Relation Distance– The Interval Relation Distance– The Extended Point-Interval Relation

Distance

The Point Relation Distance

The Interval Relation Distance

The Extended Point-Interval Relation Distance

The Distance Graph

<

ols

m

loe

o

s

d

fi

e

f

di

si

oi mi

ole

los

>

oo

Qualitative Spatial Knowledge

• Relation Embedding– two points on a line– two points on a plan– two line segments on

a line– two line segments on

a plan– two 2-D objects on a

plan

Projection of Two Line Segments on a 2-D Plan

Projection of Two 2-D Objects on a 2-D Plan

Generalized Spatial Relation

f 1 = 29RelSet o 29RelSet 29RelSet

f 2 = 29RelSet o 29RelSet 29RelSet o 29RelSet 29RelSet 29RelSet

f 3 = 29RelSet o 29RelSet 29RelSet o 29RelSet 29RelSet o 29RelSet 29RelSet 29RelSet 29RelSet

where 29RelSet 29RelSet { {<} {<}, {<} {>}, … , { = } { = } }

29RelSet 29RelSet 29RelSet { {<}{<}{<}, {<}{<}{>},…, {=}{=}{=}}

i1 j1, i2 j2 P (29RelSet 29RelSet )

f 2 (i1 j1, i2 j2 ) = f 1 (i1 , i2 ) f 1 ( j1 , j2 )

i1 j1 k1 , i2 j2 k2 P (29RelSet 29RelSet 29RelSet )

f 3 (i1 j1 k1, i2 j2 k2) = f 1 (i1 , i2 ) f 1 ( j1 , j2 ) f 1 (k1 , k2 )

where A B = { a b | a A, b B }

A B C = { a b c | a A, b B, c C }

The Applications

• Multimedia Presentation Generation• Shape-Based Image Retrieval (not

discussed)

Multimedia Presentation Generation

The Presentation System

Notes• Synchronization of Multimedia Objects• Computation Efficiency• Extension toward N-Dimension• Multimedia Presentation Scheduling

Summary

• White Board and Chat Room• Multimedia Synchronization• SMIL• Petri Net• Interval Temporal Logic

03 synchronized distance learning

Technology

Transcript of 03 synchronized distance learning