Media Processing in the Network
description
Transcript of Media Processing in the Network
Media Processing in the Network
Wei Tsang Ooi
Research Area How to build multimedia
network applications ?
Outline Dalí Multimedia Library Problems and related work Research proposal
Question How to build multimedia
network applications ?
Dalí
Current Solutions black box library code
easy to write but too high level need finer optimization, resource
control write from scratch
too time consuming
What is Dalí A small set of simple, fast,
reusable primitives and abstractions for image, audio and video processing
Example: Black Box Code gd GIF programming library 1.3
gdImagePtr im;FILE *in;in = fopen("mygif.gif", "rb");im = gdImageCreateFromGif(in);fclose(in);gdImageRectangle(im,0,0,8,8,red);
Example: Dalí codef = fopen(argv[1], "r");BitStreamFileRead(bs,f,0);BitParserWrap(bp, bs);GifSeqHdrParse(bp, sh);rmap = ImageMapNew();gmap = ImageMapNew();bmap = ImageMapNew();n = GifSeqHdrGetCtSize(sh);GifCtParse(bp,n,rmap,gmap,
bmap); ih = GifImgHdrNew();ph = PnmHdrNew();status = GifImgFind(bp);GifImgHdrParse(bp, ih);
h = GifImgHdrGetHeight(ih);w = GifImgHdrGetWidth(ih);r = ByteNew(w, h);g = ByteNew(w, h);b = ByteNew(w, h);i = ByteNew(w, h);GifImgParse(bp,sh,ih,i);ImageMapApply(rmap,i,r);ImageMapApply(gmap,i,g);ImageMapApply(bmap,i,b);
Dalí : Abstraction Byte Image Audio Buffer Bitstream MPEG Headers JPEG Headers Lookup Table
Bit Image DCT Image Vector Image Convolution
Kernel
Dalí : Abstraction Byte Image Audio Buffer Bitstream MPEG
Headers JPEG Headers Lookup Table
Bit Image DCT Image Vector Image Convolution
Kernel
Byte Image Two dimensional array of bytes Can be either physical or
virtual
physicalimage
virtualimage
Composable Abstraction Byte images can be used to
represent: Gray scale images RGB images YUV images Alpha channels
Example Code : Fractal
smaller = byte_new (neww, newh);byte_shrink_2x2(image,smaller);
image
smaller
neww
newh
target = byte_clip(image,0,0,dx,2*dy);byte_set(target,0);target = byte_clip(image,3*dx,0,dx,2*dy);byte_set(target,0);
image
smaller
neww
newh
dx
2dy
target
image
smaller
neww
newh
dx
2dy
target
target = byte_clip(image,dx,0,neww,newh);byte_copy(smaller,target);
image
smaller
neww
newhtarget = byte_clip(image,2*dx,2*dy,neww,newh);byte_copy(smaller,target);target = byte_clip(image,0,2*dy,neww,newh);byte_copy(smaller,target);
Dalí strategies specialized primitives
byte_shrink_2x2 memory sharing
byte_clip explicit memory allocation
byte_new
gop
Dalí : MPEG headers
gopseqhdr ...gop
hdrgophdr
seqendgop
picpichdr
pichdr pic ...
gop
Dalí : MPEG headers
gop ... seqend
seqhdr
gophdr
gophdr gop
picpichdr
pichdr pic ...
Basic Header Primitives find parse skip dump encode
Example Skip to the n-th frame
for (i = 0; i < n; i++) { mpeg_pic_hdr_find(bs); mpeg_pic_hdr_skip(bs);
}
Dalí: Bitstream
parser1011001
chunk of memory
Mode of Operations File i/o
static memory
fread/fwrite
parser
Mode of Operations Network
static memory
socket
parser
Mode of Operations Memory map
parser
entire file
Strategies expose structure
MPEG headers explicit I/O
Bitstream abstraction
Novel Features of Dalí explicit I/O explicit memory management predictable performance programs are highly reconfigurable push semantics to programmer composable primitives, abstractions
Other Features of Dalí fast easily extensible support wide ranges of formats
(GIF, JPEG, MPEG-1, WAV, AVI, ..)
Dalí : Status Currently in alpha release C, Tcl and Java binding Better performance than other
tools (e.g. Rivl, mpeg_play)
Back to the question .. How to build a multimedia
network applications ?
Breaking it up ..
capture process compress store
capture process compress
displayde-compres
s
source receiver
on onemachine
on twomachines
Optimization
capture process compress store
capture process compress
displayde-compres
s
8 bit colors, 10 frame/s
source receiver
on onemachine
on twomachines
Optimization
capture process compress store
capture process compress
displayde-compres
s
8 bit colors, 10 frame/s
8 bit colors, 10 frame/s
on onemachine
on twomachines
Optimization What if there are more than
one receiver, and they have conflicting requirements ?
Problem and Current Solutions
Problem Network heterogeneity
S R1
R2
R3
100Mbps
36kbps
128kbps
Suggested Solution Layered Multicast
streams are hierarchically encoded each layer is multicast into a group incrementally combine the layers
to refine the quality of the streams
ExampleLayer 1
Layer 2 Layer 3
Layered Multicast
S R1
R2
R3
100Mbps
36kbps
128kbps
layer 1,2,3,4,5
layer 1
layer 1,2
Layered Multicast Receivers find out if they can
join the next higher layer by experiments
Join the layer, if packet drops/low throughput detected, leave it
Problems Needs layered encoding More groups means more
states to maintained Uses a lot of multicast
addresses
Alternative Approach Move computation into the
network where to run ? when to run ? how to specify the computation ?
Related Work QOS Filters (Yeadon 96)
where to run ? when to run ? how to specify the computation ?
Filters transform one or more input
streams into an output stream Examples :
drop frames decode frames mix input audio transcode to lower quality
Architecture
S
R1
R2
session mgr
flow spec
filters filters
Architecture
S
R1
R2
session mgr
filters filters
ack
Architecture
S
R1
R2
session mgr
filters filters
data
Filter Propagation
Filter Propagation
Answers QOS Filters
where to run ? filter servers
when to run ? when resource(out) < resource(in)
how to specify the computation ?
pick a pre-determine filter
Related Work Active Service (Amir et.al 98)
where to run ? when to run ? how to specify the computation ?
Active Services
MeGa
ActiveServers
Servlets
ClientsVic need MeGa
Servlets : MeGa An application level video
gateway Transcodes MJPEG to H.261 Written using Mash
next generation multimedia network toolkit
built on tcl
AS Location
R1
R3
R2
S1
S3
S2bottleneck
MeGa
Active Service Where to run ?
active server at bottleneck When to run ?
when receiver requested it How to specify a
computation ? request for pre-determine servent
Problems How to specify computation ?
Not easily extensible Significant extension requires
manual installation of servlets/filter Where to run ?
AS : statically located
Proposal
The answers should be How to specify a
computation ? receivers can inject customized
programs into network Where to run ?
can be specified by users or dynamically place itself
Thesis Proposal active media network
(Medan) allow customizable media
processing code to be injected into the network
application: self-adaptive transcoder (SAT)
Medan
source receiver
Medan node
prog
Medan
source receiver
Medan node
prog
Medan Components media processing language active network kernel user interface
Language Requirements Safe & portable Support common format/ops Fine grain operations
distributed processing possible Explicit resource usage
resource management and admission control
Dalí : How does it fit ? Features of Java Dalí:
safe and portable Features of Dalí :
thin primitives predictable explicit memory allocation explicit I/O
Claim: Dalí is a suitable programming
language for Medan
Medan Components media processing language
Dalí active network kernel
??? user interface
???
Medan Components media processing language
Dalí active network kernel
JKernel, SANE, etc. user interface
???
Medan Components media processing language
Dalí active network kernel
JKernel, SANE, etc. user interface
Mash
Medan: Todo Engineering issues
make Mash, Dalí, and active network environment talk to each other
Mash
source
Mash objects
sinkvideo capture,network
video display,network
encoder, decoder, etc.
Mash + Dalí
Dalí Dalí
Mash + Dalí + AN
Dalí
Safe environment
Medan:Problems to be solved Resource usage determination Load balancing Deployment Security*
Resource management*
Medan Application: Self-adaptive Transcoder (SAT) Injected into the network when
a session starts Perform bandwidth adaptation
by transcoding session’s packets
SAT Scenario
SAT Scenario
congested
SAT Scenario
congested
SAT Features transparent to receivers one group per session lower latency
Basic algorithm each host who joined a group
will receive at an initial rate each node will report it’s state
(congested, under-utilized) to it’s parent
parent will adapt sending rate accordingly
Problems to be solved Detail algorithm Scalability Reflect receiver’s interest Interaction between programs Deployment
More Problems Flexible transcoding algorithm Congestion detection
Other Medan Applications Remote Sensing Multimedia Database Video Conference Auto-Editing Receiver Heterogeneity
Contributions Dalí : a media processsing
library Medan : a framework for active
media processing in the network SAT : apply Medan to solve the
network heterogeneity problems
Summaries How to build multimedia apps Adding the word “network” Current solutions and problems Proposal
Where Medan is going.. Computation can organized
itself (divided, pipelined, factored) into sub-computations
Packets can carry computation Capsules Mobile Code
The End