Synchronizationcs778/ralf/10c-sync.pdf · 10C-sync.fm215.February.01 Scope Contents ©R....

72
10C-sync.fm 1 15.February.01 Scope Contents http://www.kom.e-technik.tu-darmstadt.de http://www.tk.informatik.tu-darmstadt.de © R. Steinmetz, M. Mühlhäuser Multimedia-Systems: Synchronization Prof. Dr.-Ing. Ralf Steinmetz Prof. Dr. Max Mühlhäuser MM: TU Darmstadt - Darmstadt University of Technology, Dept. of of Computer Science TK - Telecooperation, Tel.+49 6151 16-3709, Alexanderstr. 6, D-64283 Darmstadt, Germany, [email protected] Fax. +49 6151 16-3052 RS: TU Darmstadt - Darmstadt University of Technology, Dept. of Electrical Engineering and Information Technology, Dept. of Computer Science KOM - Industrial Process and System Communications, Tel.+49 6151 166151, Merckstr. 25, D-64283 Darmstadt, Germany, [email protected] Fax. +49 6151 166152 GMD -German National Research Center for Information Technology httc - Hessian Telemedia Technology Competence-Center e.V

Transcript of Synchronizationcs778/ralf/10c-sync.pdf · 10C-sync.fm215.February.01 Scope Contents ©R....

Page 1: Synchronizationcs778/ralf/10c-sync.pdf · 10C-sync.fm215.February.01 Scope Contents   ©R. Steinmetz, M. Mühlhäuser Scope

u-darmstadt.de Fax. +49 6151 16-3052

, Dept. of Computer Science51 166151,M.tu-darmstadt.de Fax. +49 6151 166152

ogy

10C-sync.fm 1 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser Multimedia-Systems:

Synchronization

Prof. Dr.-Ing. Ralf SteinmetzProf. Dr. Max MühlhäuserMM: TU Darmstadt - Darmstadt University of Technology,

Dept. of of Computer ScienceTK - Telecooperation, Tel.+49 6151 16-3709,Alexanderstr. 6, D-64283 Darmstadt, Germany, [email protected]

RS: TU Darmstadt - Darmstadt University of Technology,Dept. of Electrical Engineering and Information Technology

KOM - Industrial Process and System Communications, Tel.+49 61Merckstr. 25, D-64283 Darmstadt, Germany, Ralf.Steinmetz@KOGMD -German National Research Center for Information Technolhttc - Hessian Telemedia Technology Competence-Center e.V

Page 2: Synchronizationcs778/ralf/10c-sync.pdf · 10C-sync.fm215.February.01 Scope Contents   ©R. Steinmetz, M. Mühlhäuser Scope

User Interfaces

.Synchro-nization

GroupCommuni-

cations

Programming

ms Communications

ice Networks

ession

Video Audio

10C-sync.fm 2 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Scope

Usa

ge Applications

Learning & Teaching Design

Ser

vice

s ContentProcess-

ing

Docu-ments

Security ..S

yste

ms Databases

Media-Server Operating Syste

Opt. Memories Quality of Serv

Bas

ics Computer

Archi-tectures

Compr

Image &Graphics

Animation

Page 3: Synchronizationcs778/ralf/10c-sync.pdf · 10C-sync.fm215.February.01 Scope Contents   ©R. Steinmetz, M. Mühlhäuser Scope

zation?

zation

nt

10C-sync.fm 3 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Contents

1. Motivation - What is (Multimedia) Synchroni

2. Synchronization and Multimedia

3. Reference Model for Multimedia - Synchroni

4. Synchronization in a Distributed Environme

5. Synchronization Techniques

6. Case Studies

Page 4: Synchronizationcs778/ralf/10c-sync.pdf · 10C-sync.fm215.February.01 Scope Contents   ©R. Steinmetz, M. Mühlhäuser Scope

Synchronization?

ion 4

ntent

10C-sync.fm 4 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

1. Motivation - What is (Multimedia)

We concentrate on:• Temporal relationship

• New flavor in the context of multimedia• Essential for communications

Applicat

Application 2

Application 3

Application 1

Space

Time

Co

Page 5: Synchronizationcs778/ralf/10c-sync.pdf · 10C-sync.fm215.February.01 Scope Contents   ©R. Steinmetz, M. Mühlhäuser Scope

heetame data

d update

e related views.

the use of

10C-sync.fm 5 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Content Relations

Dependence of media objects on data

Examples:• A graphic that visualizes data from a spreads• Two graphics showing different views of the s

Explicit definition of dependences for automate• Only the data are edited• All views of the data are

• generated automatically• cannot be edited directly

• An update of the data triggers an update of th

Implementation of content relations is based on• common data structures or• common object interfaces

Page 6: Synchronizationcs778/ralf/10c-sync.pdf · 10C-sync.fm215.February.01 Scope Contents   ©R. Steinmetz, M. Mühlhäuser Scope

tation of a media object

entation

gned to this frament may be

cts:d in a windowon a stereo output device

vatars are positioned inith talking avatar!)

10C-sync.fm 6 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Spatial Relations

Usually known as layout relationships• Define the space which is used for the presen

• on an output device• at a certain point of time in a multimedia pres

Typically expressed in layout frames:• A layout frame is placed and a content is assi• The positioning of a layout frame in a docume

• fixed to a position in a document• fixed to a position on a page• relative to the positioning of other frames

Spatial relations for time-dependent media obje• A frame or a group of frames may be presente• Note: an audio presentation can be positioned

(imagine, e.g., a virtual meeting room where aa VRML space: audio should be positioned w

Page 7: Synchronizationcs778/ralf/10c-sync.pdf · 10C-sync.fm215.February.01 Scope Contents   ©R. Steinmetz, M. Mühlhäuser Scope

existed before)

tandardization

st correspond to the temporal

10C-sync.fm 7 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Temporal Relations

Importance for time-dependent media objects• new in context of Multimedia (content, spatial• essential for communications, scheduling• non-trivial yet "general" issue --> subject of s• --> to be emphasized in this chapter

Example for temporal relations:• Temporal relation between

• video object• audio object

recorded during a concertAt presentation time:

• temporal relation of the two media objects murelation at the recording moment

Relations:• at the same time• independently• in sequel

Page 8: Synchronizationcs778/ralf/10c-sync.pdf · 10C-sync.fm215.February.01 Scope Contents   ©R. Steinmetz, M. Mühlhäuser Scope

e relation betweenndent media object

ideo sequence.d each of the frames has to

ws a jumping ball

t

10C-sync.fm 8 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Intra-object Synchronization

Intra-object Synchronization

Intraobject synchronization defines the timvarious presentation units of one time-depe

Example:• Time relation between the single frames of a v

For a video with a rate of 25 frames per seconbe displayed for 40 ms

Frames of a video sequence that sho

40ms

Page 9: Synchronizationcs778/ralf/10c-sync.pdf · 10C-sync.fm215.February.01 Scope Contents   ©R. Steinmetz, M. Mühlhäuser Scope

nchronization between

n

dio sequence

o2

tion

t

10C-sync.fm 9 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Inter-object Synchronization

Inter-object Synchronization

Inter-object synchronization defines the symedia objects

Example:• Time relations of a multimedia synchronizatio

that starts with an audio/video sequence,followed by several picturesand an animation that is commented by an au

Audio1

Video

P1 P2 P3

Audi

Anima

Page 10: Synchronizationcs778/ralf/10c-sync.pdf · 10C-sync.fm215.February.01 Scope Contents   ©R. Steinmetz, M. Mühlhäuser Scope

ence of information

bjects

phony

10C-sync.fm 10 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Logical Data Units

Logical Data Unit

An information unit - usually part of a sequunits - of a time-dependent media object

Granularity levels of LDUs• Application dependent• Imply a hierarchical decomposition of media o• Example:

The LDU hierarchy of a Sym

Symphony

Movements

Notes

Samples

Page 11: Synchronizationcs778/ralf/10c-sync.pdf · 10C-sync.fm215.February.01 Scope Contents   ©R. Steinmetz, M. Mühlhäuser Scope

f continuous media like

ation execution

or a microphone, or

10C-sync.fm 11 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Classification of LDUs

Closed LDUs• have a predictable duration

Example:• LDUs that are parts of stored media objects o

• audio and video or• stored media objects with a fixed duration.

Open LDUs• duration is not predictable before the present

Example:• Open LDUs typically represent

• input from a life source, for example a camera• media objects that include a user interaction

Page 12: Synchronizationcs778/ralf/10c-sync.pdf · 10C-sync.fm215.February.01 Scope Contents   ©R. Steinmetz, M. Mühlhäuser Scope

ce as LDUs

s except lowest)

equence as LDUs

Us, open block-LDU

t

Pic.n

t

Audio n

cy (e.g., 1/8000 s)

(e.g., = 0.064 s)

t

10C-sync.fm 12 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Classification of LDUs (cont.)

Physical units as LDUs:

Physical units of a video sequen

Blocks of samples as LDUs: (all hierarchy level

Blocked physical units of an audio s

note: e.g., life-stream video has closed phys-LD

Pic. 1 Pic. 2 Pic. 3 Pic. 4

1/30 s

Audio 1 Audio 2 Audio 3 Audio 4

physical frame duration = 1/sample frequen

duration of a Logical Data Unit of 512 Bytes

Page 13: Synchronizationcs778/ralf/10c-sync.pdf · 10C-sync.fm215.February.01 Scope Contents   ©R. Steinmetz, M. Mühlhäuser Scope

n

generated media objects.

ser selected size

rded user interactions.

em with varying durations

Frame n

e (= e.g.1/15 s)t

tion 4

t

10C-sync.fm 13 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Classification of LDUs (cont.)

User-defined vs. capturing-defined LDU duratio

Fixed vs. varying LDU duration

• e.g., fixed&user-defined: typical for computer

LDUs of an animation sequence of u

• e.g., fixed&capturing-defined: typical for reco

LDUs of an user interaction in a window syst

Frame1 Frame 2 Frame 3 Frame 4

duration of Logical Data Units of selected siz

Action 1 Action 2 AcA.3

LDUs of a recorded user interaction

Page 14: Synchronizationcs778/ralf/10c-sync.pdf · 10C-sync.fm215.February.01 Scope Contents   ©R. Steinmetz, M. Mühlhäuser Scope

rationknown duration

er interaction

DUs

l processingt

t

10C-sync.fm 14 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Classification of LDUs (cont.)

Note: open LDUs always with unpredictable du• Typically a user interaction with in advance un

An open LDU representing an us

Note: Timers represented as stream of empty L

LDUs of a timer

LDU duration not known before the actua

User selection

LDUs of 10 ms duration

Page 15: Synchronizationcs778/ralf/10c-sync.pdf · 10C-sync.fm215.February.01 Scope Contents   ©R. Steinmetz, M. Mühlhäuser Scope

ed by

User

nimation,Timer

Userteraction

10C-sync.fm 15 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Classification of LDUs - Overview

• open vs. closed• open: always unknown duration• closed: fixed or variable duration

• fixed vs. variable duration• user-defined vs. capturing-related• physical --> blocked

Duration defin

Capturing

Fixed Duration of LDU Audio, Video A

Variable or unknownDuration of LDU

Recordedinteraction in

Page 16: Synchronizationcs778/ralf/10c-sync.pdf · 10C-sync.fm215.February.01 Scope Contents   ©R. Steinmetz, M. Mühlhäuser Scope

eo streams withs

zation

ow

Frame n

Audio n

ms advance

e 3 Slide 4

10C-sync.fm 16 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Synchronization Examples

Lip synchronization• demands for a tight coupling of audio and vid• a limited skew between the two media stream

LDU view of lip synchroni

Slide show with audio comment:

LDU view of a slide sh

Frame 1 Frame 2 Frame 3 Frame 4

Audio 1 Audio 2

Audio max. 80 ms advance Video max. 80

Audio sequence

Slide 1 Slide 2 Slid

Page 17: Synchronizationcs778/ralf/10c-sync.pdf · 10C-sync.fm215.February.01 Scope Contents   ©R. Steinmetz, M. Mühlhäuser Scope

udio1 and Video) is follo-ion (RI), a slidetion) which is partiallyo2). Starting the anima-n is presented to thection, a final picture

ple

Interaction

Audio2

Animation

P4

t

10C-sync.fm 17 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Application ExampleExample used further in the course:

A lip synchronized audio video sequence (Awed by a replay of a recorded user interactsequence (P1 - P3) and an animation (Animacommented using an audio sequence (Audition presentation, a multiple choice questiouser (Interaction). If the user has made a sele(P4) is shown.

Synchronization exam

Audio1

Video

RI

P1 P2 P3

Page 18: Synchronizationcs778/ralf/10c-sync.pdf · 10C-sync.fm215.February.01 Scope Contents   ©R. Steinmetz, M. Mühlhäuser Scope

ser Interface

Sync. QoSProvision

10C-sync.fm 18 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

2. Synchronization and Multimedia

Typical Requirements:• hard vs. soft

Strong Interaction with QoS• acceptable, too bad ...

Variety of Mechanisms• Live Synchronisation• Live Synchronisation with Storage• Synthetical Synchronisation

MMOperatingSystemEnvironment

ProgrammingEnvironment

Application

MM U

MM

MM Sync. QoSRequirements

Page 19: Synchronizationcs778/ralf/10c-sync.pdf · 10C-sync.fm215.February.01 Scope Contents   ©R. Steinmetz, M. Mühlhäuser Scope

ed by Skew (QoS)?????????

r Interface

Sync.QosProvision

10C-sync.fm 19 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Why Multimedia-Synchronization?

Media Influenc

Video

Animation

?

AudioImageText

Audio

AnimationAudioImageText

Pointer

MMOperatingSystemEnvironment

ProgrammingEnvironment

Application

MM Use

MM

MMSync.

QoSRequirements

Page 20: Synchronizationcs778/ralf/10c-sync.pdf · 10C-sync.fm215.February.01 Scope Contents   ©R. Steinmetz, M. Mühlhäuser Scope

iments

ation requirements for:

equipment

10C-sync.fm 20 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Multimedia Synchronization - Exper

Example:• at IBM ENC Heidelberg to quantify synchroniz

• Audio/video synchronization• Audio/pointer synchronization

Selection of material:• Duration

• 30s in experiments• 5s would have been sufficient

• Reuse of same material for all tests

Introduction of artificial skew:• By media composition with professional video• With frame based granularity

Test conditions:• Huge set of test candidates

• Professional: cutter at TV studios• Casual: every day “user”

• Awareness of the synchronization issues• Set of tests with different skews lasted 45 min

Page 21: Synchronizationcs778/ralf/10c-sync.pdf · 10C-sync.fm215.February.01 Scope Contents   ©R. Steinmetz, M. Mühlhäuser Scope

any artifact or

ords. (-)

f or behind the

or.o watch all your(.)

10C-sync.fm 21 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Feedback - Questionnaire

1 While watching this video clip, did you detectstrange effect?

If so, please try to describe it in a few w

....

2 Are you able to identify if audio was ahead omoving pictures? (.)

a) Yes, I identify audio to be played aheadof video

b) Yes, I identify audio to be played behindvideo

c)No, I notice that audio is out of sync with

respect to video but, I am not sure ifaudio is played ahead of or behind video.

3 You noticed a synchronization errHow would you qualify this error if you have t

TV programs with such an error?

a) I would not mind, the error is acceptable

b) I dislike it, the error is annoying

c) I am not sure if I would accept such anerror or if I would really dislike it

Page 22: Synchronizationcs778/ralf/10c-sync.pdf · 10C-sync.fm215.February.01 Scope Contents   ©R. Steinmetz, M. Mühlhäuser Scope

encing Factors

(hammer and nails)

iew)

10C-sync.fm 22 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Lip Synchronization: Major Influ

Video:• Content

• Continuous (talking head) vs. discrete events• Background (no distraction)

• Resolution and quality• View mode (head view, shoulder view, body v

Audio:• Content• Background noise or music• Language and articulation

Page 23: Synchronizationcs778/ralf/10c-sync.pdf · 10C-sync.fm215.February.01 Scope Contents   ©R. Steinmetz, M. Mühlhäuser Scope

tection

ahead of video

10C-sync.fm 23 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Lip Synchronization: Level of De

Areas:• In sync QoS: +/- 80 ms• Transient• Out of sync

audio behind video audio

Page 24: Synchronizationcs778/ralf/10c-sync.pdf · 10C-sync.fm215.February.01 Scope Contents   ©R. Steinmetz, M. Mühlhäuser Scope

noyance

o ahead of video

10C-sync.fm 24 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Lip Synch.: Level of Accuracy/An

Some observations:• Asymmetry• Additional tests with long movie

• +/- 80 ms: no distraction• -240 ms, +160 ms: disturbing

audio behind video audi

Page 25: Synchronizationcs778/ralf/10c-sync.pdf · 10C-sync.fm215.February.01 Scope Contents   ©R. Steinmetz, M. Mühlhäuser Scope

etch”)n map”)

10C-sync.fm 25 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Pointer Synchronization

Fundamental CSCW shared workspace issue

Analysis of CSCW scenarios:• Discrete pointer movement (e.g. “technical sk• Continuous pointer movements (e.g. “route o

Most challenging probes:• Short audio• Fast pointer movement

Page 26: Synchronizationcs778/ralf/10c-sync.pdf · 10C-sync.fm215.February.01 Scope Contents   ©R. Steinmetz, M. Mühlhäuser Scope

etection

ahead of pointer

10C-sync.fm 26 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Pointer Synchronization: Level of D

Observations:• Difficult to detect “out of sync”:

• i.e., other magnitude than lip sync• Asymmetry:

• According to every day experience

audio behind pointer audio

Page 27: Synchronizationcs778/ralf/10c-sync.pdf · 10C-sync.fm215.February.01 Scope Contents   ©R. Steinmetz, M. Mühlhäuser Scope

nnoyance

ahead of pointer

10C-sync.fm 27 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Pointer Synchronization: Level of A

Areas:• In sync: QoS -500 ms, +750 ms• Transient• Out of sync

audio behind pointer audio

Page 28: Synchronizationcs778/ralf/10c-sync.pdf · 10C-sync.fm215.February.01 Scope Contents   ©R. Steinmetz, M. Mühlhäuser Scope

n QoS+/- 120 ms

+/- 80 ms

+/- 240 ms

+/-500 ms

+/- 240 ms

+/-500 ms

+/- 80 ms

+/- 11 µs

with various +/- 120 ms

d music) +/- 500 ms

otes) +/- 5 ms

) +/- 500 ms

+/- 240 ms

-500ms+750 ms

10C-sync.fm 28 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Synchronization QoS: Skew Values

Media Mode, Applicatio

Video

Animation correlated

Audio lip synchronization

Imageoverlay

non-overlay

Textoverlay

non-overlay

Audio

Animation event correlation (e.g., dancing)

Audio

tightly coupled (stereo)

loosely coupled (dialogue modeparticipants)

loosely coupled (e.g., backgroun

Imagetightly coupled (e.g., music with n

loosely coupled (e.g., slide show

Text Anmerkungen zu Text

Pointer Audio Related to the Itlem

Page 29: Synchronizationcs778/ralf/10c-sync.pdf · 10C-sync.fm215.February.01 Scope Contents   ©R. Steinmetz, M. Mühlhäuser Scope

Synchronization

isms

ynchronization systems

10C-sync.fm 29 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

3. Reference Model for Multimedia -

A reference model is needed

• to understand the various requirements formultimedia synchronization

• to identify and structure runtime mechanisms

• to identify interfaces between runtime mechan

• to compare system solutions for multimedia s

Page 30: Synchronizationcs778/ralf/10c-sync.pdf · 10C-sync.fm215.February.01 Scope Contents   ©R. Steinmetz, M. Mühlhäuser Scope

t

Timecompo-sition

Deriva-

Inter-pretation

tion

10C-sync.fm 30 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Existing Synchronization Models

Classification methods:• Physical, system, and human level• Intra-stream (fine-grain) synchronization and

inter-stream (coarse-grain) synchronization• Live and synthetical synchronization• Mapping

Uninterpreted

Media Objects

Bytestream

DerivedMedia Objects

MultimediaObject

Page 31: Synchronizationcs778/ralf/10c-sync.pdf · 10C-sync.fm215.February.01 Scope Contents   ©R. Steinmetz, M. Mühlhäuser Scope

chniques

on

10C-sync.fm 31 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Classification of Synchronization Te

Inter-object synchronization techniques:• used to control jitter between media streams• classification according to the type and locati

of the synchronization control

Distributed synchronization control:• distributed control based on protocols• distribution based on servers• distribution on nodes without server structure

Local synchronization control:• control on several layers• use of local servers

Page 32: Synchronizationcs778/ralf/10c-sync.pdf · 10C-sync.fm215.February.01 Scope Contents   ©R. Steinmetz, M. Mühlhäuser Scope

Ab

stractio

n

low

highlayer

10C-sync.fm 32 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Four Layer Model

Media layer

Stream layer

Object layer

Multimedia application

Specification

Page 33: Synchronizationcs778/ralf/10c-sync.pdf · 10C-sync.fm215.February.01 Scope Contents   ©R. Steinmetz, M. Mühlhäuser Scope

cluded objects

ct

ject synchronizations

ject synchronizations

ynchronization

10C-sync.fm 33 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Synchronization Specification

Synchronization specification• describes all temporal dependencies of the in

in a multimedia object• is part of the description of a multimedia obje• should comprise:

• Intra-object synchronization specificationsfor the media objects of the presentation

• Quality of service descriptions for the intra-ob• Inter-object synchronization specifications

between the media objects of the presentation• Quality of service descriptions for the inter-ob

• may comprise:• alternative presentation forms

• must be explicitly described in a synthetical s• is implicitly defined in a live synchronization

Page 34: Synchronizationcs778/ralf/10c-sync.pdf · 10C-sync.fm215.February.01 Scope Contents   ©R. Steinmetz, M. Mühlhäuser Scope

Tasksto edit

to format

map the quality of service values tothe object layer

Plan and coordinate presentationexecution

tiate presentation of time-dependentmedia objects

itiate presentation of time-indepen-dent media objects

source Reservation and schedulingof LDU processing

File and device access

10C-sync.fm 34 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Layered Model - Summary

Layer Interface AbstractionSpecification The tools performing the tasks of this

layer have interfaces, the layer itself hasno upper interface

to

Object Objects that hide types of enclosedmedia

Synchronization Specification

Media-oriented quality of service (Interms like allowed skew, allowed jitter)

Ini

In

Stream Streams and groups of streams

Guarantees for intra-streamsynchronization

Guarantees for inter-stream synchroni-zation of streams in a group

Re

Media Device independent access to LDUs.

Guarantees for single LDU processing

Page 35: Synchronizationcs778/ralf/10c-sync.pdf · 10C-sync.fm215.February.01 Scope Contents   ©R. Steinmetz, M. Mühlhäuser Scope

Environment

ation

10C-sync.fm 35 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

4. Synchronization in a Distributed

More complex than in a local environment:

• Distributed storage of synchronization inform

• Distributed storage of media objects

• Communication delays and jitter

• Multi-party communication patterns

Page 36: Synchronizationcs778/ralf/10c-sync.pdf · 10C-sync.fm215.February.01 Scope Contents   ©R. Steinmetz, M. Mühlhäuser Scope

fication

ation

Sink

ation

Sink

10C-sync.fm 36 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Transport of Synchronization Speci

Delivery of the complete synchronization informbefore the start of the presentation

Use of an additional synchronization channel:

Use of a multiplexed channel:

Source

Video

Audio

Synchronization

Source

Video

Audio

Synchroniz

multiplexed

Page 37: Synchronizationcs778/ralf/10c-sync.pdf · 10C-sync.fm215.February.01 Scope Contents   ©R. Steinmetz, M. Mühlhäuser Scope

ions

ink

Sinkidth

and

10C-sync.fm 37 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Location of Synchronization Operat

Source SBandwidthdemand

Synchronization at the sink node

Synchronization at the source node

Source Bandwdem

Page 38: Synchronizationcs778/ralf/10c-sync.pdf · 10C-sync.fm215.February.01 Scope Contents   ©R. Steinmetz, M. Mühlhäuser Scope

ing of the presentation:

Ta=Tav - Nla - SaTv=Tav - Nlb - Sv.skew

Sink AVTav

10C-sync.fm 38 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Clock Synchronization

Problem: Skew between the clocks

• Clock skew has to be considered in the plann• Tav: Start of presentation• Nla: Known net delay• Sa,Sv as Skews• Start time of audio transmission of Source A:• Start time of video transmission of Source V:

• Restricted buffer size demands for restricted

Source A

Source V

Audio

Video

Ta, Sa

Tv, Sv

Nla

Nlv

Page 39: Synchronizationcs778/ralf/10c-sync.pdf · 10C-sync.fm215.February.01 Scope Contents   ©R. Steinmetz, M. Mühlhäuser Scope

Sinks

10C-sync.fm 39 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Multiple Communication Relations

1:1

n:1

1:n

n:m

Sources

Sources : Sinks

Page 40: Synchronizationcs778/ralf/10c-sync.pdf · 10C-sync.fm215.February.01 Scope Contents   ©R. Steinmetz, M. Mühlhäuser Scope

o the network

10C-sync.fm 40 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Multi Step Synchronization

• Synchronization during object acquirement

• Synchronization of retrieval

• Synchronization during delivery of the LDUs t

• Synchronization during the transport

• Synchronization at the sink

• Synchronization within the output device

Page 41: Synchronizationcs778/ralf/10c-sync.pdf · 10C-sync.fm215.February.01 Scope Contents   ©R. Steinmetz, M. Mühlhäuser Scope

en two objects

A B

AB

AB

A meets B

A during B

A finishes B

10C-sync.fm 41 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

5. Synchronization Techniques

Specification of great importance

• Simple specification method:

Types of temporal relations betwe

B

AB

A

BA

AB

A before B

A overlaps B

A starts B

A equals B

Page 42: Synchronizationcs778/ralf/10c-sync.pdf · 10C-sync.fm215.February.01 Scope Contents   ©R. Steinmetz, M. Mühlhäuser Scope

ts

nance of specificationsnit in the specificationbe easily describable

ia objectsribableation

10C-sync.fm 42 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Specification Methods - Requiremen

Complex specification comprise:• multiple object synchronization• user interactions

Requirements for a specification method:• Support of object consistency and for mainte• Media objects should be kept as one logical u• All types of synchronization relations should• Support for the integration

of time-dependent and time-independent med• Quality of service requirements must be desc• Support of hierarchical levels of a synchroniz

Page 43: Synchronizationcs778/ralf/10c-sync.pdf · 10C-sync.fm215.February.01 Scope Contents   ©R. Steinmetz, M. Mühlhäuser Scope

either a single-media object

ronization operators

Video

onous audio/videoesentation

10C-sync.fm 43 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Basic Hierarchical Specification

Two main synchronization operations:• Serial synchronization of actions• Parallel synchronization of actions

Action:• an atomic action handles the presentation of

or a user input or a delay.• a compound action is a combination of synch

and atomic actions.

Serial and parallel presentations

Pic. 1 Pic. 2 Pic. 3

Slide sequence

Audio

Lip synchrpr

Page 44: Synchronizationcs778/ralf/10c-sync.pdf · 10C-sync.fm215.February.01 Scope Contents   ©R. Steinmetz, M. Mühlhäuser Scope

synchronization.

(RI = Recorded Interaction, Pic.teraction)

Pic.3

Ani.3 UI Pic.4

10C-sync.fm 44 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Basic Hierarchical Specification

Delay as a possible action allows for• delays in serial presentations and• delayed presentations of objects in a parallel

Application example:

Hierarchical specification of the application example= Picture, Aud. = Audio, Ani. = Animation, UI = User In

Audio1 Video

RI Pic.1 Pic.2

Ani.1 Ani.2 Aud.2

Page 45: Synchronizationcs778/ralf/10c-sync.pdf · 10C-sync.fm215.February.01 Scope Contents   ©R. Steinmetz, M. Mühlhäuser Scope

not be described

of skewessary

ependent mediamust be added

ia objects foroses

r media object

A3

B3

C3

10C-sync.fm 45 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Basic Hierarchical Specification

Not all relations can be described.

Example of a synchronization that can

Advantages Disadvantages

Easy tounderstand

Additional descriptionquality of service nec

Natural supportof

hierarchies

For the presentation of time-indobjects presentation durations

Integration ofinteractive

objects is easy

Need for the splitting of medsynchronization purp

No adequate abstractions focontents

A1 A2

B1 B2

C1 C2

Page 46: Synchronizationcs778/ralf/10c-sync.pdf · 10C-sync.fm215.February.01 Scope Contents   ©R. Steinmetz, M. Mühlhäuser Scope

rios can not be

10C-sync.fm 46 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Some synchronization scenadescribed

Advantages Disadvantages

Page 47: Synchronizationcs778/ralf/10c-sync.pdf · 10C-sync.fm215.February.01 Scope Contents   ©R. Steinmetz, M. Mühlhäuser Scope

Timer

xis

the application example

teraction

io2

ation P4

?t

10C-sync.fm 47 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Synchronization Based on a Global

All single-media objects are attached to a time a

Application example

Time axis synchronization specification of

Audio1

Video

Record. UI In

P1 P2 P3

Aud

Anim

Page 48: Synchronizationcs778/ralf/10c-sync.pdf · 10C-sync.fm215.February.01 Scope Contents   ©R. Steinmetz, M. Mühlhäuser Scope

Timer

vantages

nknown durationgrated, extensionsel are required

f service has to beirectly by using the

axis or additionalvice specificationsto be given

d environment thebal unique time hasbe kept

10C-sync.fm 48 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Synchronization Based on a Global

Advantages Disad

Easy to understandObjects of u

cannot be inteto the mod

Support of hierarchies easy torealize

Skew quality ospecified indcommon timequality of ser

have

Easy to maintain because of themutual independence of objects

In a distributenotion of a glo

to

Good abstraction for mediacontents

Integration of time-independentobjects is easy

Page 49: Synchronizationcs778/ralf/10c-sync.pdf · 10C-sync.fm215.February.01 Scope Contents   ©R. Steinmetz, M. Mühlhäuser Scope

xes

ement unitsate space

xes

at

10C-sync.fm 49 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Synchronization Based on Virtual A

Specification using• coordinate systems with user defined measur• several virtual axes to create a virtual coordin

Example:

Music notes as example for virtual axes

At runtime the virtual axes are mapped to real a

Virtual axis with measurement unit be

Virtual axes with measurement unit pitch

Page 50: Synchronizationcs778/ralf/10c-sync.pdf · 10C-sync.fm215.February.01 Scope Contents   ©R. Steinmetz, M. Mühlhäuser Scope

xes

ple

teraction

io2

ation

P4

t

t

10C-sync.fm 50 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Synchronization Based on Virtual A

Application example:.

Virtual time axis specification of the application exam

Audio1

Video

Record. UI

In

P1 P2 P3

Aud

Anim

Time axis

Time axis

Interaction axis

Page 51: Synchronizationcs778/ralf/10c-sync.pdf · 10C-sync.fm215.February.01 Scope Contents   ©R. Steinmetz, M. Mühlhäuser Scope

xes

vantages

ty of service onlyor by additionaltions definable

on may becomeby many axes

xes at runtime maynd time consuming

10C-sync.fm 51 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Synchronization Based on Virtual A

Advantages Disad

Easy to understandSkew quali

indirectlyspecifica

Often specification according tothe problem space possible

Specificaticomplex

Good possibility for buildinghierarchies

Mapping of abe complex a

Easy maintainable, becauseobjects are kept as units andmutual independent objects

Good abstraction for mediacontent

Easy integration of timeindependent media objects

Interactive objects can beincluded using specialized axes

Page 52: Synchronizationcs778/ralf/10c-sync.pdf · 10C-sync.fm215.February.01 Scope Contents   ©R. Steinmetz, M. Mühlhäuser Scope

e Points

rded as sequences of

ation of a media objectent media objects

it is reachedlicit reference to time.

10C-sync.fm 52 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Synchronization based on Referenc

Synchronization via reference points:• Time-dependent single-media objects are rega

LDUs• Reference points:

• the start time and the stop time of the present• the start-times of the subunits of time-depend

• Synchronization point:• A set of connected reference points• Defines the synchronization between objects• LDUs have to be started resp. stopped when

• Description of temporal relations without exp

Page 53: Synchronizationcs778/ralf/10c-sync.pdf · 10C-sync.fm215.February.01 Scope Contents   ©R. Steinmetz, M. Mühlhäuser Scope

e Points

ation model

eference point model

3 Slide 4

10C-sync.fm 53 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Synchronization based on Referenc

Lip synchronization example

Lip synchronization in the reference point synchroniz

Slide show example.

Example of a slide show with audio sequence in the r

Video

Audio

Slide 1 Slide 2 Slide

Audio

Page 54: Synchronizationcs778/ralf/10c-sync.pdf · 10C-sync.fm215.February.01 Scope Contents   ©R. Steinmetz, M. Mühlhäuser Scope

e Points

e-independent media objectssynchronization specification

Interaction

dio

mation

P4

10C-sync.fm 54 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Synchronization based on Referenc

Application example

Example of the integration of time-dependent and timas well as closed and open LDUs in a reference point

Audio

Video

Record. UI

P1 P2 P3 Au

Ani

Page 55: Synchronizationcs778/ralf/10c-sync.pdf · 10C-sync.fm215.February.01 Scope Contents   ©R. Steinmetz, M. Mühlhäuser Scope

e Points

vantages

to change becausect description ofndencies

omplex hierarchies

10C-sync.fm 55 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Synchronization based on Referenc.

Advantages Disad

Easy to understandMore difficult

of the diredepe

Natural description oftemporal relations Sometimes c

Easy integration of open LDUs

Integrated description of skewquality of service

Easy integration of time-independent objects

Time axis are special case of thereference point synchronization

model

Page 56: Synchronizationcs778/ralf/10c-sync.pdf · 10C-sync.fm215.February.01 Scope Contents   ©R. Steinmetz, M. Mühlhäuser Scope

ns

hat reaches a specific LDU

10C-sync.fm 56 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Event-based Synchronization

Event-based synchronization presentation actioare initiated by the synchronization events

Typical presentation actions are:• to start a presentation• to stop a presentation• to prepare a presentation

Events• may be external,

e.g. generated by a timer, or• internal to the presentation

generated by a time-dependent media object t

Page 57: Synchronizationcs778/ralf/10c-sync.pdf · 10C-sync.fm215.February.01 Scope Contents   ©R. Steinmetz, M. Mühlhäuser Scope

r1.dy ...

p

rt

10C-sync.fm 57 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Event-based Synchronization

Application example:

ActionEvent

Start Audio.stop Timerea

Audio1 start

Video start

Pic.1 start sto

Timer1 start(3)

Pic.2 sta

...

Page 58: Synchronizationcs778/ralf/10c-sync.pdf · 10C-sync.fm215.February.01 Scope Contents   ©R. Steinmetz, M. Mühlhäuser Scope

vantages

sy to handle

specification

to maintain

f time-independentng additional timers

riptions of quality ofnecessary

e of hierarchies

10C-sync.fm 58 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Event-based Synchronization

Advantages Disad

Easy integration of interactiveobjects Not ea

Easy extensible by new events Complex

Flexible because any eventcan be specified Hard

Integration oobjects by usi

Separate descservice

Difficult us

Page 59: Synchronizationcs778/ralf/10c-sync.pdf · 10C-sync.fm215.February.01 Scope Contents   ©R. Steinmetz, M. Mühlhäuser Scope

on blocking token.n is removed and to each

that is assigned to its new

ion

F4

A1

ms 11 ms

33ms

F5

10C-sync.fm 59 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Petri net-based synchronization

The rules for a timed petri net are:• A transition fires if all input places contain a n• If a transition fires, on each input place a toke

output place a token is added• After firing, a token is blocked for the duration

place

Lip synchronization example.

Petri net based synchronization of a lip synchronizat

F2 F3

A1

11 ms 11 ms 11 ms

33ms

11

F1

Page 60: Synchronizationcs778/ralf/10c-sync.pdf · 10C-sync.fm215.February.01 Scope Contents   ©R. Steinmetz, M. Mühlhäuser Scope

F3

11 ms D

0 ms

10C-sync.fm 60 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Petri-Net based synchronization

Example for a hierarchy:

F1 F2

A1

11 ms

33ms

11 msD

0 ms

S

33 ms

Page 61: Synchronizationcs778/ralf/10c-sync.pdf · 10C-sync.fm215.February.01 Scope Contents   ©R. Steinmetz, M. Mühlhäuser Scope

A3nimationubnet 3

10C-sync.fm 61 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Petri-Net based synchronization

Application example:

Petri net specification of the application example

S

60 s

Audio-Videosubnet

R

20 s

Record.UI

P

15 s

Picturesubnet

PI

A1

Au

A2Animation-subnet 1

As

Animation subnet 2and Audio subnet

P4

User interaction Closing picture

Page 62: Synchronizationcs778/ralf/10c-sync.pdf · 10C-sync.fm215.February.01 Scope Contents   ©R. Steinmetz, M. Mühlhäuser Scope

vantages

sy to handle

specification

f media objects

bstraction for thehe media objects

10C-sync.fm 62 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Petri-Net based synchronization.

Advantages Disad

Hierarchies can be created Not ea

Easy integration of time-independent objects Complex

Easy integration of interactiveobjects Splitting o

Integrated skew quality of service Insufficient acontent of t

Page 63: Synchronizationcs778/ralf/10c-sync.pdf · 10C-sync.fm215.February.01 Scope Contents   ©R. Steinmetz, M. Mühlhäuser Scope

ariots

y the timing operations

10C-sync.fm 63 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Scripts

Script• Textual description of a synchronization scen• Elements of scripts are activities and subscrip• Three main operations in a script are

• serial presentation• parallel presentation• repeated presentation of a media object

• Often full programming languages extended b

Page 64: Synchronizationcs778/ralf/10c-sync.pdf · 10C-sync.fm215.February.01 Scope Contents   ©R. Steinmetz, M. Mühlhäuser Scope

;

.rec”);eg”);eg”);eg”);eg”);;on.au”);on.ani”);

;

ple {(

}

10C-sync.fm 64 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Scriptsactivity DigAudioAudio(“video.au”)activity SMPVideo(“video.smp”);activity XRecorderRecorder(“windowactivity PicturePicture1(“bild1.jpactivity PicturePicture2(“bild2.jpactivity PicturePicture3(“bild3.jpactivity PicturePicture4(“bild4.jpactivity StartIntercationSelectionactivity DigAudioAniAudio(“animatiactivity RTAnimaAnimation(“animatiscript Picture_sequence 3Pictures=

Picture1.Duration(5) >>Picture2.Duration(5) >>Picture3.Duration(5);

script Lipsynch AV = Audio & Videoscript AniComment AA = Animation &AniAudio.Translate(2);script Multimedia Application_exam

AV >> Record. UI >> 3Pictures >>( (Selection >> Picture4) & AA )

Page 65: Synchronizationcs778/ralf/10c-sync.pdf · 10C-sync.fm215.February.01 Scope Contents   ©R. Steinmetz, M. Mühlhäuser Scope

vantages

sy to handle

specification

age of commonnecessary

structs for skewervice necessary

10C-sync.fm 65 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Scripts

Advantages Disad

Easy hierarchies Not ea

Logical structure of objectscan be kept Complex

Easy integration of time-independent objects

Implicit ustimers

Easy integration of interactiveobjects

Special conquality of s

Skew quality of service canbe specified

Easy extensible by newsynchronization constructs

Flexible because programmable

Page 66: Synchronizationcs778/ralf/10c-sync.pdf · 10C-sync.fm215.February.01 Scope Contents   ©R. Steinmetz, M. Mühlhäuser Scope

reated further here:

vironment)

AND HYPERMEDIA OBJECTS

tion objects to bed services

at are part of the

p a set of MHEG objectsctions to be performed

oral and conditional

10C-sync.fm 66 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

6. Case Studies

1. Some "historical" or academic systems not t• HyTime• FireFly• MODE (Multimedia Objects in a Distributed En• Multimedia Tele-Orchestra• ACME (Abstractions for Continuous MEdia)• Apple QuickTime

2. MHEG: CODED REPRESENTATION OF MULTIMEDIA

• representation and encoding of• final form multimedia and hypermedia informa

interchanged within or across applications an

MHEG objects• Content objects encapsulate media objects th

presentation• Composite objects act as a ‘container’ to grou• Action objects are used to exchange sets of a

between objects• Link objects are used to specify spatial, temp

relations between MHEG objects

Page 67: Synchronizationcs778/ralf/10c-sync.pdf · 10C-sync.fm215.February.01 Scope Contents   ©R. Steinmetz, M. Mühlhäuser Scope

tent objectsneric units of time=altitude) of finite length

inate space

are sent to an object

ables the object

10C-sync.fm 67 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

Synchronization in MHEG

Generic space:• provides a virtual coordinate system• used to specify the layout and relation of con

• one time axis of infinite length measured in ge• three spatial axes (X=latitude, Y=longtitude, Z

in an interval of [-32768,+32767]• MHEG maps from the virtual to the real coord

Presentation of content objects:• Based on the exchange of action objects that• Examples for actions are:

• prepare for setting the object in a state that ento present itself

• run to start the presentation• stop to stop the presentation

Page 68: Synchronizationcs778/ralf/10c-sync.pdf · 10C-sync.fm215.February.01 Scope Contents   ©R. Steinmetz, M. Mühlhäuser Scope

elayed sequential actions.

...

10C-sync.fm 68 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

MHEG action lists

Action list:• Combination of actions to action lists• Parallel action lists are executed in parallel• Each list is composed of a delay followed by d

Links:• used to create event-based synchronizations• can trigger events

delay delay

action action

Parallel lists of actions

Del

ayed

sequ

entia

l

actio

ns

Page 69: Synchronizationcs778/ralf/10c-sync.pdf · 10C-sync.fm215.February.01 Scope Contents   ©R. Steinmetz, M. Mühlhäuser Scope

Gen

eric

Pre

sent

atio

nS

ervi

ces AV-Sub-

UserInterface

system

Services

10C-sync.fm 69 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

MHEG Engine

Application

MHEG Engine

Object Manager

Link Processor Interpreter

Operating System

Page 70: Synchronizationcs778/ralf/10c-sync.pdf · 10C-sync.fm215.February.01 Scope Contents   ©R. Steinmetz, M. Mühlhäuser Scope

ultimedia object

Engine

User Interface

bsystem

Service

10C-sync.fm 70 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

MHEG Relation to Reference Model

Media layer

Stream layer

Object layer

Specification layer

Object layer interface

Stream layer interface

Media layer interface

MHEG coded m

MHEGInterpreter

AV Su

Object ManagerLink Processor

Page 71: Synchronizationcs778/ralf/10c-sync.pdf · 10C-sync.fm215.February.01 Scope Contents   ©R. Steinmetz, M. Mühlhäuser Scope

guage

)

ia Synchronization

ay (cf. Flash, Shockwave)

ontents not accessible

b?

10C-sync.fm 71 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

SMIL

SMIL: Synchronized Multimedia Integration Lan

part of W3C standards, part of XML&friends

treated in lectures about Web Engineering

based on "Amsterdam Model" (CWI Amsterdam

major contribution of recent years

goal: integrate Hypertext concept and Multimed

remember: multimedia possible on the Web tod

but:• requires plug-in• MM presentation is "one node" in hypertext, c• no links into MM presentation, no links out• same text used within flash and as part of We

--> duplicate --> two different elements!

SMIL introduces MM synchronization in XML• similar to hierarchical specification• plus support for spatial arrangements

Page 72: Synchronizationcs778/ralf/10c-sync.pdf · 10C-sync.fm215.February.01 Scope Contents   ©R. Steinmetz, M. Mühlhäuser Scope

MM node B

TML

t

MM node B

TML

t

ave been, e.g.:

" duration="0.5" target=2>e" in <img> tag)

left=... top.. height.. width../>

10C-sync.fm 72 15.February.01

Scope

Contents

htt

p:/

/ww

w.k

om

.e-t

ech

nik

.tu

-da

rmst

adt

.de

htt

p:/

/ww

w.t

k.in

form

atik

.tu

-da

rmst

adt

.de

©R

.S

tein

me

tz,

M.

hlh

äu

ser

SMIL

Example for temporal spec:<par>

<img src="logo.rp" region="logo"/><text src="info.rt" region="info"/><video src="pk.rm" region="video"/>

</par>

<seq><audio src="music1.ra"/><audio src="music2.ra"/>

</seq>

link

in H

MM node A

single mediumwithin A

t

link

in H

MM node A

single mediumwithin A

t

prior definitions, might h

<fadein start="0.2(target: -> "handl

<region id="logo"