OpenMAX AL, OpenSL ES - Khronos Group · 2014-04-08 · © Copyright Khronos Group, 2011 - Page 10...

© Copyright Khronos Group, 2011 -

OpenMAX AL, OpenSL ES Native Multimedia in Android

Erik Noreke Chair of OpenMAX AL and OpenSL ES Working Groups


Why Create Khronos Multimedia APIs?

Khronos APIs

•Conformance Tests

•Portable

•IP Protection

•Profiles & Components

•Predictable set of functionality

Open Source Framework

•Free to change

•Every implementation unique

•GPL license

•Plugins

•Functionality dependent on

implementation


&

Application Access Layer


The Need for Application Access

• Hardware and low level access is powerful but

complex

- More power than most application developers

require

• Most application developers just want to

playback and record media

- Specifying where the content comes from and is

rendered to

- Manipulate a few playback controls

- Have simple configurability


The Need for Application Access

• That are what OpenMAX AL and OpenSL ES

provide…

- Simple high-level multimedia APIs for playback

and recording use cases

- With hardware acceleration capabilities built in

• …With Cross-platform portability

- No need to rewrite an application for every

platform


OpenMAX AL Focused on Streaming Media

• Media playback and recording

- Playback and recording of everything from video files to

radio to camera input

• Full range of video effects and controls – including

playback rate, post processing, and image

manipulation

- Apply advanced effects to captured images and video

before sending to friends


OpenMAX AL Focused on Streaming Media

• OpenMAX AL gives easy access to

streaming media

- Allowing time to be spent on developing

applications, not using frameworks

• Designed as a complete set of

multimedia functionality

- Unlike any other streaming media API


OpenSL ES – Powerful Mobile Audio

• Full range of effects and controls,

including advanced 3D effects such as

Doppler and virtualization

- Experience rich, enhanced sound from

locations other than the handset, even

moving, for the ultimate multimedia

experience

• Create theater quality audio

experience

- In a mobile device!


OpenSL ES – Powerful Mobile Audio

• Full 3D audio functionality enhances

any gaming experience

- Perfect companion to OpenGL ES

• Designed for implementation by either a

hardware or software solution

- Take full advantage of system capabilities


Object Oriented Media Access

• Simplified object-oriented streaming media

- Built to enable common use cases

- But also has access to advanced functionality

• Media Objects enable PLAY and RECORD of content

- Media Objects input and output to Devices

- Perform some operation on an input and emit the result as output

• Objects have control interfaces

- Play, Record, Seek, Rate

Player Object URI

Memory

Content pipe

Media Input

Camera (AL)

URI

Memory

Content pipe

Audio Mix

Display Window (AL)

DSrc DSnk


OpenMAX AL & OpenSL ES


OpenSL ES and OpenMAX AL • Working groups collaborated to define the common API functionality

• Independent, Compatible, Consistent and Distinct

3D Audio

Audio Effects

Advanced MIDI

Audio Playback

Audio Recording

Basic MIDI

Video playback

Camera

Video recording

Image capture & display Buffer

queues

Radio and RDS


Why Profiles?

• Functionality divided into basic groups

- Most common groupings

• Profiles reduce fragmentation

- Not all devices support all functionality

- Only support the profiles you need

• Applications can query available profiles

- To adapt to the current device

- Facilitates porting of applications


Why Profiles?

• Profiles can be combined

- To best suit any product

• Optional features

- Some features are optional in all profiles

- APIs are consistent when feature is available


OpenSL ES and OpenMAX AL • Independent

- But designed to work together

- No dependency between the APIs – either one can exist by itself

• Compatible

- Working groups collaborated to make sure the APIs work together well


OpenSL ES and OpenMAX AL • Consistent

- Identical API architecture

- Identical APIs for same functionality

• Distinct

- OpenMAX AL represents basic multimedia functionality (audio, video, and image)

- OpenSL ES represents advanced audio functionality


Complete Multimedia Use-Case

Multimedia Processing

OpenGL ES GL_TEXURE_EXTERNAL_OES texture

3D Audio Rendering

Application on CPU

Video stream to GPU

EGLStream

Much more flexibility than just “show a video”

EGLSurface OpenGL ES

Audio stream to DSP


Android NDK for Multimedia • NDK – Native Development Kit

• Introduced in

Android 2.3 Gingerbread

Mobile 3D

Advanced Audio


Why the NDK?

• Better control of fuctionality and timing

• Access to hardware acceleration

• Multimedia is the primary reason for the NDK

• Increases application complexity

• Requires compilation for each platform family

- ARMv5TE

- ARMv7-A

- x86 instructions


Focus is on Standardized APIs

• Earlier internal APIs not stable enough…

• Avoid fragmentation

• Well tested and documented

• Focus on functionality

- Not on implementation

• Access to hardware acceleration


Native Multimedia APIs in Android 2.3

• OpenGL ES 2.0

• EGL 1.4

• OpenSL ES 1.0.1

- Partial implementation

- Almost complete Phone + Music profiles

- Buffer queues from 1.1 as Android extension

- Google version compliant as far as implemented

- Other vendors offer fully compliant solution

Mobile 3D

Advanced Audio Graphics Context Management


Ice Cream Sandwich – Android 4.0 • Same OS for both phones

and tablets

• Better multitasking

• Better support for

hardware accelerated

graphics

• DRM

• Partial OpenMAX AL 1.0.1

implementation

- Streaming use-case only

- Does not interoperate with

OpenSL ES

Streaming Media


Android & OpenMAX IL

• Android uses OpenMAX IL in their

Multimedia framework

- Codecs only

- No graph building

• Main Multimedia framework is still

Stagefright

- Introduced in Éclair

• Allows for both software

implemented and hardware

accelerated codecs

• Not part of NDK

- Not intended to be accessible from

applications


What is Google saying?


Where is Android Headed?

• More Devices

- TVs?

- Netbooks?

- Chrome OS?

• What does the future hold?

- Compliant OpenSL ES?

- Better OpenMAX AL support?

- OpenMAX AL EGL Stream Extension?

- Support for OpenCL?

- Content Pipes?


Thank You

OpenMAX AL, OpenSL ES - Khronos Group · 2014-04-08 · © Copyright Khronos Group, 2011 - Page 10...

Documents

Transcript of OpenMAX AL, OpenSL ES - Khronos Group · 2014-04-08 · © Copyright Khronos Group, 2011 - Page 10...