Building Scalable and High Efficient

Java Multimedia Collaboration

Wenjun Wu, Tao Huang, Geoffrey Fox

Community Grids Computing Laboratory, Indiana University, USA

Global-MMCS & XGSP XGSP

• A common AV signaling protocol

• A core conference control mechanism

• Collaboration Web-Services Global Multimedia Collaboration System (Global-

MMCS) • to support scalable web-service based interoperable

collaborations

• integrates various services including videoconferencing, instant messaging and streaming, and supports multiple videoconferencing technologies such as H.323, SIP and Access Grid clients

Global-MMCS & XGSP

SIP H323 Access Grid Native XGSPAdmire

Gateways convert to uniform XGSP Messaging

High Performance (RTP)and XML/SOAP and ..

Media ServersFilters

Session ServerXGSP-based Control

NaradaBrokeringAll Messaging

Use Multiple Media servers to scale to many codecs and manyversions of audio/video mixing

NB Scales asdistributed

WebServices

NaradaBrokering

Java Media Framework Java Media Framework (JMF)

• A unified framework for managing the acquisition, processing, and delivery of time-based media data

• A common cross-platform Java API for accessing underlying media frameworks.

• “Write Once, Run Anywhere” to multimedia developers

JMF for multimedia collaboration JMF is the key building block for the implementation

of Global-MMCSFor both client-side and server-side

Building multimedia services such as video, audio mixing, snapshot generation

Implementing Global-MMCS AVportlet, a native XGSP audiovisual client

Performance Issues in JMFthe performance optimization of Java Media Framework has to be

made especially for the conferencing application

Multimedia Programming FrameworksVideo Capture Video

RenderingAudio

Windows VFW or WDM Direct Draw Direct Sound

Linux Video-for-Linux XLib ALSA

Mac OS Sequence Grabber Quick Draw Core Audio

Portability vs Performance : Platform-dependent multimedia devices and codec management

a portable multimedia framework must define a good platform-independent

part to cover the heterogeneous native multimedia platforms.

Multimedia Programming Frameworks the similar multimedia filter pipeline architecture Filter & Pipe design pattern

cascaded filter chain Support high efficient media processing

• High-volume multimedia data is typically generated by a source filter objects and piped through one or more filter objects.  Eventually, the media reaches a sink filter object and is consumed.

Multimedia Programming Frameworks Tcl/Tk based

CMT [7], VIC [8], VuSystem [9]

low-overhead control functionality implemented in a scripting language like Tcl

performance-critical data handling implemented in a compiled language like C/C++

Windows DirectShow

C++ COM API

only enables Windows applications for multimedia programming

JMF

RTPDataSource

RTPDataSource

NetworkMoni tor

Vi deoDecoder

Vi deoRenderer

Audi oDecoder

Audi oRenderer

Vi deoEncoder

Raw VideoDataSource

Raw AudioDataSource

Audi oEncoder

VideoEncoded

DataSource

RTP Vi deo TXSessi onManager

RTP Audi o RXSessi onManager

AudioEncoded

DataSource

Network

RTP Vi deo RXSessi onManager

RTP Audi o RXSessi onManager

JMF Filter Architecture

Access Grid Access Grid : a large scale

audio/videoconference based on a multicast network

provides the group-to-group collaborations among 150 nodes connected to Internet 2 world wide.

Use improved MBONE audiovisual tools VIC and RAT

Depends upon high-speed network ( each node needs 20Mbps )

Issues for JMF in collaboration

Very good cross-platform framework No optimization for conferencing application

Compared to fully performance tuned Tcl/Tk package

( Access Grid ) No implementation for Mac OS

Enhanced Java Media Framework

Improving JMF performance by exposing the interfaces of the JMF filters and rewriting a faster one

Adding the state-of-art codecs in JMF and screen capture

Extending the JMF to the Mac-OS platform to extend its portability

Video Rendering JMF filter pipeline is composed of filter Modules and

linking Connectors streaming protocols between a OutputConnector and

InputConnector:

Safe protocol

Push protocol JMF render implementation:

safe protocol , blit the whole video frame

Our optimization approach:

“Direct Write Through” approach

Direct-Write-Through optimization in JMF rendering

VideoDecoder

DirectDraw

Renderer

OffScreenYUV Surface

PrimaryYUV Surface

OutputConnector

InputConnector

BLT

Mark

Data

CircularBuffer

Video Capturing a video data source to abstract the real capture

services which usually have different APIs in modern operating systems.

added Mac video capturing into JMF package Added the desktop capture as a video data source

to support remote desktop sharing

Add New Codec

JMF 2.0 package only supports H.261 decoder, H263 and JPEG. We added H.261 encoder and MPEG-4 video codec (DivX) based on JMF codec interface. In addition, new RTP format and payload for MPEG-4 video are also added through JMF RTPManager.

Global-MMCS AVPortlet

NaradaConnector

RTPEventTransport

Vi deo / Audi oProcessor

Vi deo / Audi oPl ayer

RTP/ RTCPSessi on

NaradaConnector

RTPEventTransport

XGSPAV Sessi onControl l er

NaradaBrokerJ MS

Transport

NaradaBri dge

Vi deo / Audi oProcessor

Vi deo / Audi oPl ayer

XGSPAV Sessi on

Protocol Stack

RTP/ RTCPSessi on

Global-MMCS AVPortlet

GlobalMMCS AVPortlet, to fully make use of the services provided by Global-MMCS. The tool which can run on multiple desktop platforms such as windows, Linux and Mac OSX, integrates audio and video collaboration together

Extending JMF RTP Transport over Publish/Subscribe Overlay

XGSP Audiovisual Session Management

Performance Evaluation Compare the video rendering performance between JMF and

VIC

Testing video streams are captured from the same desktop of the sending machine, and multicasted to the receiver machine

which runs Access Grid VIC, two JMF clients based on Sun’s JMF implementation and our improved version.

two test scenarios with different source streams• the CIF-size still image of the desktop area is sent to the clients. Each

stream is encoded in H.261, and has average bandwidth 20kbps.

• the CIF-size video sequence from a 30-second movie with a lot of motions is streamed to the clients. Each stream is encoded in H.261, and has average bandwidth 500kbps.

Performance Evaluation

0

10

20

30

40

50

60

1 2 3 4 5 6 7 8

AG VIC

SunJMF

FastJMF

0

10

20

30

40

50

60

70

1 2 3 4 5 6 7 8

AG VIC

SunJMF

FastJMF

Video Rendering performance (left: still desktop, right: movie sequence)

Q&A