Media Processing in the Network

Wei Tsang Ooi

Research Area How to build multimedia

network applications ?

Outline Dalí Multimedia Library Problems and related work Research proposal

Question How to build multimedia

network applications ?

Dalí

Current Solutions black box library code

easy to write but too high level need finer optimization, resource

control write from scratch

too time consuming

What is Dalí A small set of simple, fast,

reusable primitives and abstractions for image, audio and video processing

Example: Black Box Code gd GIF programming library 1.3

gdImagePtr im;FILE *in;in = fopen("mygif.gif", "rb");im = gdImageCreateFromGif(in);fclose(in);gdImageRectangle(im,0,0,8,8,red);

Example: Dalí codef = fopen(argv[1], "r");BitStreamFileRead(bs,f,0);BitParserWrap(bp, bs);GifSeqHdrParse(bp, sh);rmap = ImageMapNew();gmap = ImageMapNew();bmap = ImageMapNew();n = GifSeqHdrGetCtSize(sh);GifCtParse(bp,n,rmap,gmap,

bmap); ih = GifImgHdrNew();ph = PnmHdrNew();status = GifImgFind(bp);GifImgHdrParse(bp, ih);

h = GifImgHdrGetHeight(ih);w = GifImgHdrGetWidth(ih);r = ByteNew(w, h);g = ByteNew(w, h);b = ByteNew(w, h);i = ByteNew(w, h);GifImgParse(bp,sh,ih,i);ImageMapApply(rmap,i,r);ImageMapApply(gmap,i,g);ImageMapApply(bmap,i,b);

Dalí : Abstraction Byte Image Audio Buffer Bitstream MPEG Headers JPEG Headers Lookup Table

Bit Image DCT Image Vector Image Convolution

Kernel

Dalí : Abstraction Byte Image Audio Buffer Bitstream MPEG

Headers JPEG Headers Lookup Table

Bit Image DCT Image Vector Image Convolution

Kernel

Byte Image Two dimensional array of bytes Can be either physical or

virtual

physicalimage

virtualimage

Composable Abstraction Byte images can be used to

represent: Gray scale images RGB images YUV images Alpha channels

Example Code : Fractal

smaller = byte_new (neww, newh);byte_shrink_2x2(image,smaller);

image

smaller

neww

newh

target = byte_clip(image,0,0,dx,2*dy);byte_set(target,0);target = byte_clip(image,3*dx,0,dx,2*dy);byte_set(target,0);

image

smaller

neww

newh

dx

2dy

target

image

smaller

neww

newh

dx

2dy

target

target = byte_clip(image,dx,0,neww,newh);byte_copy(smaller,target);

image

smaller

neww

newhtarget = byte_clip(image,2*dx,2*dy,neww,newh);byte_copy(smaller,target);target = byte_clip(image,0,2*dy,neww,newh);byte_copy(smaller,target);

Dalí strategies specialized primitives

byte_shrink_2x2 memory sharing

byte_clip explicit memory allocation

byte_new

gop

Dalí : MPEG headers

gopseqhdr ...gop

hdrgophdr

seqendgop

picpichdr

pichdr pic ...

gop

Dalí : MPEG headers

gop ... seqend

seqhdr

gophdr

gophdr gop

picpichdr

pichdr pic ...

Basic Header Primitives find parse skip dump encode

Example Skip to the n-th frame

for (i = 0; i < n; i++) { mpeg_pic_hdr_find(bs); mpeg_pic_hdr_skip(bs);

}

Dalí: Bitstream

parser1011001

chunk of memory

Mode of Operations File i/o

static memory

fread/fwrite

parser

Mode of Operations Network

static memory

socket

parser

Mode of Operations Memory map

parser

entire file

Strategies expose structure

MPEG headers explicit I/O

Bitstream abstraction

Novel Features of Dalí explicit I/O explicit memory management predictable performance programs are highly reconfigurable push semantics to programmer composable primitives, abstractions

Other Features of Dalí fast easily extensible support wide ranges of formats

(GIF, JPEG, MPEG-1, WAV, AVI, ..)

Dalí : Status Currently in alpha release C, Tcl and Java binding Better performance than other

tools (e.g. Rivl, mpeg_play)

Back to the question .. How to build a multimedia

network applications ?

Breaking it up ..

capture process compress store

capture process compress

displayde-compres

s

source receiver

on onemachine

on twomachines

Optimization

capture process compress store

capture process compress

displayde-compres

s

8 bit colors, 10 frame/s

source receiver

on onemachine

on twomachines

Optimization

capture process compress store

capture process compress

displayde-compres

s

8 bit colors, 10 frame/s

8 bit colors, 10 frame/s

on onemachine

on twomachines

Optimization What if there are more than

one receiver, and they have conflicting requirements ?

Problem and Current Solutions

Problem Network heterogeneity

S R1

R2

R3

100Mbps

36kbps

128kbps

Suggested Solution Layered Multicast

streams are hierarchically encoded each layer is multicast into a group incrementally combine the layers

to refine the quality of the streams

ExampleLayer 1

Layer 2 Layer 3

Layered Multicast

S R1

R2

R3

100Mbps

36kbps

128kbps

layer 1,2,3,4,5

layer 1

layer 1,2

Layered Multicast Receivers find out if they can

join the next higher layer by experiments

Join the layer, if packet drops/low throughput detected, leave it

Problems Needs layered encoding More groups means more

states to maintained Uses a lot of multicast

addresses

Alternative Approach Move computation into the

network where to run ? when to run ? how to specify the computation ?

Related Work QOS Filters (Yeadon 96)

where to run ? when to run ? how to specify the computation ?

Filters transform one or more input

streams into an output stream Examples :

drop frames decode frames mix input audio transcode to lower quality

Architecture

S

R1

R2

session mgr

flow spec

filters filters

Architecture

S

R1

R2

session mgr

filters filters

ack

Architecture

S

R1

R2

session mgr

filters filters

data

Filter Propagation

Filter Propagation

Answers QOS Filters

where to run ? filter servers

when to run ? when resource(out) < resource(in)

how to specify the computation ?

pick a pre-determine filter

Related Work Active Service (Amir et.al 98)

where to run ? when to run ? how to specify the computation ?

Active Services

MeGa

ActiveServers

Servlets

ClientsVic need MeGa

Servlets : MeGa An application level video

gateway Transcodes MJPEG to H.261 Written using Mash

next generation multimedia network toolkit

built on tcl

AS Location

R1

R3

R2

S1

S3

S2bottleneck

MeGa

Active Service Where to run ?

active server at bottleneck When to run ?

when receiver requested it How to specify a

computation ? request for pre-determine servent

Problems How to specify computation ?

Not easily extensible Significant extension requires

manual installation of servlets/filter Where to run ?

AS : statically located

Proposal

The answers should be How to specify a

computation ? receivers can inject customized

programs into network Where to run ?

can be specified by users or dynamically place itself

Thesis Proposal active media network

(Medan) allow customizable media

processing code to be injected into the network

application: self-adaptive transcoder (SAT)

Medan

source receiver

Medan node

prog

Medan

source receiver

Medan node

prog

Medan Components media processing language active network kernel user interface

Language Requirements Safe & portable Support common format/ops Fine grain operations

distributed processing possible Explicit resource usage

resource management and admission control

Dalí : How does it fit ? Features of Java Dalí:

safe and portable Features of Dalí :

thin primitives predictable explicit memory allocation explicit I/O

Claim: Dalí is a suitable programming

language for Medan

Medan Components media processing language

Dalí active network kernel

??? user interface

???

Medan Components media processing language

Dalí active network kernel

JKernel, SANE, etc. user interface

???

Medan Components media processing language

Dalí active network kernel

JKernel, SANE, etc. user interface

Mash

Medan: Todo Engineering issues

make Mash, Dalí, and active network environment talk to each other

Mash

source

Mash objects

sinkvideo capture,network

video display,network

encoder, decoder, etc.

Mash + Dalí

Dalí Dalí

Mash + Dalí + AN

Dalí

Safe environment

Medan:Problems to be solved Resource usage determination Load balancing Deployment Security*

Resource management*

Medan Application: Self-adaptive Transcoder (SAT) Injected into the network when

a session starts Perform bandwidth adaptation

by transcoding session’s packets

SAT Scenario

SAT Scenario

congested

SAT Scenario

congested

SAT Features transparent to receivers one group per session lower latency

Basic algorithm each host who joined a group

will receive at an initial rate each node will report it’s state

(congested, under-utilized) to it’s parent

parent will adapt sending rate accordingly

Problems to be solved Detail algorithm Scalability Reflect receiver’s interest Interaction between programs Deployment

More Problems Flexible transcoding algorithm Congestion detection

Other Medan Applications Remote Sensing Multimedia Database Video Conference Auto-Editing Receiver Heterogeneity

Contributions Dalí : a media processsing

library Medan : a framework for active

media processing in the network SAT : apply Medan to solve the

network heterogeneity problems

Summaries How to build multimedia apps Adding the word “network” Current solutions and problems Proposal

Where Medan is going.. Computation can organized

itself (divided, pipelined, factored) into sub-computations

Packets can carry computation Capsules Mobile Code

The End