Media Processing in the Network

Wei Tsang Ooi

Research Area

How to build multimedia network applications ?

Outline

Dalí Multimedia Library Problems and related work Research proposal

Question

How to build multimedia network applications ?

Dalí

Current Solutions

black box library code easy to write but too high level need finer optimization, resource

control write from scratch

too time consuming

What is Dalí

A small set of simple, fast, reusable primitives and abstractions for image, audio and video processing

Example: Black Box Code gd GIF programming library 1.3

gdImagePtr im;FILE *in;in = fopen("mygif.gif", "rb");im = gdImageCreateFromGif(in);fclose(in);gdImageRectangle(im,0,0,8,8,red);

Example: Dalí code

f = fopen(argv[1], "r");BitStreamFileRead(bs,f,0);BitParserWrap(bp, bs);GifSeqHdrParse(bp, sh);rmap = ImageMapNew();gmap = ImageMapNew();bmap = ImageMapNew();n = GifSeqHdrGetCtSize(sh);GifCtParse(bp,n,rmap,gmap,

bmap); ih = GifImgHdrNew();ph = PnmHdrNew();status = GifImgFind(bp);GifImgHdrParse(bp, ih);

h = GifImgHdrGetHeight(ih);w = GifImgHdrGetWidth(ih);r = ByteNew(w, h);g = ByteNew(w, h);b = ByteNew(w, h);i = ByteNew(w, h);GifImgParse(bp,sh,ih,i);ImageMapApply(rmap,i,r);ImageMapApply(gmap,i,g);ImageMapApply(bmap,i,b);

Dalí : Abstraction

Byte Image Audio Buffer Bitstream MPEG Headers JPEG Headers Lookup Table

Bit Image DCT Image Vector Image Convolution

Kernel

Dalí : Abstraction

Byte Image Audio Buffer Bitstream MPEG

Headers JPEG Headers Lookup Table

Bit Image DCT Image Vector Image Convolution

Kernel

Byte Image

Two dimensional array of bytes Can be either physical or

virtual

physicalimage

virtualimage

Composable Abstraction Byte images can be used to

represent: Gray scale images RGB images YUV images Alpha channels

Example Code : Fractal

smaller = byte_new (neww, newh);byte_shrink_2x2(image,smaller);

image

smaller

neww

newh

target = byte_clip(image,0,0,dx,2*dy);byte_set(target,0);target = byte_clip(image,3*dx,0,dx,2*dy);byte_set(target,0);

image

smaller

neww

newh

dx

2dy

target

image

smaller

neww

newh

dx

2dy

target

target = byte_clip(image,dx,0,neww,newh);byte_copy(smaller,target);

image

smaller

neww

newhtarget = byte_clip(image,2*dx,2*dy,neww,newh);byte_copy(smaller,target);target = byte_clip(image,0,2*dy,neww,newh);byte_copy(smaller,target);

Dalí strategies

specialized primitives byte_shrink_2x2

memory sharing byte_clip

explicit memory allocation byte_new

gop

Dalí : MPEG headers

gopseqhdr ...gop

hdrgophdr

seqendgop

picpichdr

pichdr pic ...

gop

Dalí : MPEG headers

gop ... seqend

seqhdr

gophdr

gophdr gop

picpichdr

pichdr pic ...

Basic Header Primitives

find parse skip dump encode

Example

Skip to the n-th framefor (i = 0; i < n; i++) { mpeg_pic_hdr_find(bs); mpeg_pic_hdr_skip(bs);

}

Dalí: Bitstream

parser1011001

chunk of memory

Mode of Operations

File i/ostatic memory

fread/fwrite

parser

Mode of Operations

Networkstatic memory

socket

parser

Mode of Operations

Memory map

parser

entire file

Strategies

expose structure MPEG headers

explicit I/O Bitstream abstraction

Novel Features of Dalí

explicit I/O explicit memory management predictable performance programs are highly reconfigurable push semantics to programmer composable primitives,

abstractions

Other Features of Dalí

fast easily extensible support wide ranges of formats

(GIF, JPEG, MPEG-1, WAV, AVI, ..)

Dalí : Status

Currently in alpha release C, Tcl and Java binding Better performance than other

tools (e.g. Rivl, mpeg_play)

Back to the question ..

How to build a multimedia network applications ?

Breaking it up ..

capture process compress store

capture process compress

displayde-

compress

source receiver

on onemachine

on twomachines

Optimization

capture process compress store

capture process compress

displayde-

compress

8 bit colors, 10 frame/s

source receiver

on onemachine

on twomachines

Optimization

capture process compress store

capture process compress

displayde-

compress

8 bit colors, 10 frame/s

8 bit colors, 10 frame/s

on onemachine

on twomachines

Optimization

What if there are more than one receiver, and they have conflicting requirements ?

Problem and Current Solutions

Problem

Network heterogeneity

S R1

R2

R3

100Mbps

36kbps

128kbps

Suggested Solution

Layered Multicast streams are hierarchically

encoded each layer is multicast into a

group incrementally combine the layers

to refine the quality of the streams

ExampleLayer 1

Layer 2 Layer 3

Layered Multicast

S R1

R2

R3

100Mbps

36kbps

128kbps

layer 1,2,3,4,5

layer 1

layer 1,2

Layered Multicast

Receivers find out if they can join the next higher layer by experiments

Join the layer, if packet drops/low throughput detected, leave it

Problems

Needs layered encoding More groups means more

states to maintained Uses a lot of multicast

addresses

Alternative Approach

Move computation into the network where to run ? when to run ? how to specify the computation ?

Related Work

QOS Filters (Yeadon 96) where to run ? when to run ? how to specify the computation ?

Filters

transform one or more input streams into an output stream

Examples : drop frames decode frames mix input audio transcode to lower quality

Architecture

S

R1

R2

session mgr

flow spec

filters filters

Architecture

S

R1

R2

session mgr

filters filters

ack

Architecture

S

R1

R2

session mgr

filters filters

data

Filter Propagation

Filter Propagation

Answers

QOS Filters where to run ?

filter servers when to run ?

when resource(out) < resource(in) how to specify the

computation ? pick a pre-determine filter

Related Work

Active Service (Amir et.al 98) where to run ? when to run ? how to specify the computation ?

Active Services

MeGa

ActiveServers

Servlets

Clients

Vic need MeGa

Servlets : MeGa

An application level video gateway

Transcodes MJPEG to H.261 Written using Mash

next generation multimedia network toolkit

built on tcl

AS Location

R1

R3

R2

S1

S3

S2

bottleneck

MeGa

Active Service

Where to run ? active server at bottleneck

When to run ? when receiver requested it

How to specify a computation ? request for pre-determine servent

Problems

How to specify computation ? Not easily extensible Significant extension requires

manual installation of servlets/filter Where to run ?

AS : statically located

Proposal

The answers should be

How to specify a computation ? receivers can inject customized

programs into network Where to run ?

can be specified by users or dynamically place itself

Thesis Proposal

active media network (Medan) allow customizable media

processing code to be injected into the network

application: self-adaptive transcoder (SAT)

Medan

source receiver

Medan node

prog

Medan

source receiver

Medan node

prog

Medan Components

media processing language active network kernel user interface

Language Requirements Safe & portable Support common format/ops Fine grain operations

distributed processing possible Explicit resource usage

resource management and admission control

Dalí : How does it fit ?

Features of Java Dalí: safe and portable

Features of Dalí : thin primitives predictable explicit memory allocation explicit I/O

Claim:

Dalí is a suitable programming language for Medan

Medan Components

media processing language Dalí

active network kernel ???

user interface ???

Medan Components

media processing language Dalí

active network kernel JKernel, SANE, etc.

user interface ???

Medan Components

media processing language Dalí

active network kernel JKernel, SANE, etc.

user interface Mash

Medan: Todo

Engineering issues make Mash, Dalí, and active

network environment talk to each other

Mash

source

Mash objects

sink

video capture,network

video display,network

encoder, decoder, etc.

Mash + Dalí

Dalí Dalí

Mash + Dalí + AN

Dalí

Safe environment

Medan:Problems to be solved Resource usage determination Load balancing Deployment Security*

Resource management*

Medan Application:

Self-adaptive Transcoder (SAT) Injected into the network when

a session starts Perform bandwidth adaptation

by transcoding session’s packets

SAT Scenario

SAT Scenario

congested

SAT Scenario

congested

SAT Features

transparent to receivers one group per session lower latency

Basic algorithm

each host who joined a group will receive at an initial rate

each node will report it’s state (congested, under-utilized) to it’s parent

parent will adapt sending rate accordingly

Problems to be solved

Detail algorithm Scalability Reflect receiver’s interest Interaction between programs Deployment

More Problems

Flexible transcoding algorithm Congestion detection

Other Medan Applications Remote Sensing Multimedia Database Video Conference Auto-Editing Receiver Heterogeneity

Contributions

Dalí : a media processsing library

Medan : a framework for active media processing in the network

SAT : apply Medan to solve the network heterogeneity problems

Summaries

How to build multimedia apps Adding the word “network” Current solutions and problems Proposal

Where Medan is going..

Computation can organized itself (divided, pipelined, factored) into sub-computations

Packets can carry computation Capsules Mobile Code

The End