Integrating a Hardware Video Codec into Android Stagefright ...
Transcript of Integrating a Hardware Video Codec into Android Stagefright ...
![Page 1: Integrating a Hardware Video Codec into Android Stagefright ...](https://reader030.fdocuments.in/reader030/viewer/2022021416/586987201a28abc2638bf3b5/html5/thumbnails/1.jpg)
Integrating a Hardware Video Codec into Android Stagefright using OpenMAX IL
Damian Hobson-Garcia(Igel), Katsuya Matsubara (Igel), Takanari Hayama (Igel),
Hisao Munakata (Renesas Electronics)
![Page 2: Integrating a Hardware Video Codec into Android Stagefright ...](https://reader030.fdocuments.in/reader030/viewer/2022021416/586987201a28abc2638bf3b5/html5/thumbnails/2.jpg)
Introduction
Android Stagefright media server
– Handles video/audio playback/recording on an Android device (in 2.3 – Gingerbread)
– Built in software codecs
• Enabled by default
Hardware codec
– Faster than software codec
– Frees up the CPU for other tasks (eg. UI)
• Require integration
2/36
![Page 3: Integrating a Hardware Video Codec into Android Stagefright ...](https://reader030.fdocuments.in/reader030/viewer/2022021416/586987201a28abc2638bf3b5/html5/thumbnails/3.jpg)
Why bother?
We want to play video at 1920x1080 @ 30fps from a mobile platform
Stagefright S/W decoder won’t play certain high resolution videos– certain features not supported
3/36
![Page 4: Integrating a Hardware Video Codec into Android Stagefright ...](https://reader030.fdocuments.in/reader030/viewer/2022021416/586987201a28abc2638bf3b5/html5/thumbnails/4.jpg)
Getting H/W into Stagefright
How can we get H/W codec into Stagefright?
We integrated an AVC (H.264) decoder into Stagefright using OpenMAX IL
Here’s what we found out.
OpenMAX IL
4/36
![Page 5: Integrating a Hardware Video Codec into Android Stagefright ...](https://reader030.fdocuments.in/reader030/viewer/2022021416/586987201a28abc2638bf3b5/html5/thumbnails/5.jpg)
Overview
Hardware
OpenMAX IL/Bellagio
Android Stagefright Integration
– Video Decoder Specific Considerations
5/36
![Page 6: Integrating a Hardware Video Codec into Android Stagefright ...](https://reader030.fdocuments.in/reader030/viewer/2022021416/586987201a28abc2638bf3b5/html5/thumbnails/6.jpg)
Platform
• Renesas SH 7372 SoC (ARM Cortex-A8 @ 800MHz on board)
http://www.renesas.com/prod/as
sp/mobile/ap4.html
• Hardware assist IP• Video codec (AVC, MPEG)
• Audio codec (AAC, MP3)
• Image processing (scaling, rotation, colour conversion, filtering)
• JPEG codec
• etc.6/36
![Page 7: Integrating a Hardware Video Codec into Android Stagefright ...](https://reader030.fdocuments.in/reader030/viewer/2022021416/586987201a28abc2638bf3b5/html5/thumbnails/7.jpg)
Hardware Acceleration
Video Processing Unit (VPU)
• Video AVC/MPEG codec– H.264 High/Main/Baseline
Profile codec
– H.263 codec
– 1920x1080 @ 30fps throughput
– YCbCr 4:2:0 color format
Video Engine Unit (VEU)
• Image processing– RGB <-> YCbCr (planar)
– Rotation
– Scaling
7/36
+ necessary drivers/libraries
![Page 8: Integrating a Hardware Video Codec into Android Stagefright ...](https://reader030.fdocuments.in/reader030/viewer/2022021416/586987201a28abc2638bf3b5/html5/thumbnails/8.jpg)
Overview
Hardware
OpenMAX IL/Bellagio
Android Stagefright Integration
– Video Decoder Specific Considerations
8/36
![Page 9: Integrating a Hardware Video Codec into Android Stagefright ...](https://reader030.fdocuments.in/reader030/viewer/2022021416/586987201a28abc2638bf3b5/html5/thumbnails/9.jpg)
OpenMAX IL
“The OpenMAX IL (Integration Layer) API defines a standardized media component interface to enable developers and platform providers to integrate and communicate with multimedia codecs implemented in hardware or software”
Khronos Group - http://www.khronos.org/openmax/
Application
OpenMAX IL implementation core
Hardware/Software Codec Implementation
OpenMAXcomponent
OpenMAXcomponent
OpenMAXcomponent
9/36
![Page 10: Integrating a Hardware Video Codec into Android Stagefright ...](https://reader030.fdocuments.in/reader030/viewer/2022021416/586987201a28abc2638bf3b5/html5/thumbnails/10.jpg)
Sample Configuration
Target application could be GStreamer , Android or something else
Use Bellagio OpenMAX IL implementation as the core
Application
OpenMAX IL implementation core
Hardware/Software Codec Implementation
OpenMAX component
OpenMAX component
OpenMAX component
OpenMAX IL Client
OpenMAX IL core implementation
VPU
RendererDecoderParser
VEU
to
framebufferdata
data pointer10/36
![Page 11: Integrating a Hardware Video Codec into Android Stagefright ...](https://reader030.fdocuments.in/reader030/viewer/2022021416/586987201a28abc2638bf3b5/html5/thumbnails/11.jpg)
Bellagio – What and Why?
Open source (LGPL) OpenMAX IL library
– http://omxil.sourceforge.net/
OpenMAX IL core, component base and framework provided
Provides example components, simple test programs
– ffmpeg, camera input, jpeg, etc.11/36
![Page 12: Integrating a Hardware Video Codec into Android Stagefright ...](https://reader030.fdocuments.in/reader030/viewer/2022021416/586987201a28abc2638bf3b5/html5/thumbnails/12.jpg)
Anatomy of a component
001101010010101101
ComponentCore
port (input)port (output)
input buffers output buffers
OpenMAX IL access mainly through component ports 12/36
![Page 13: Integrating a Hardware Video Codec into Android Stagefright ...](https://reader030.fdocuments.in/reader030/viewer/2022021416/586987201a28abc2638bf3b5/html5/thumbnails/13.jpg)
Making an OpenMAX IL component
1. Look at one of the Bellagio components
– lots to reuse– /src/base/omx_base_*
2. Configuration interface
3. Data interface
4. Buffer allocation (if necessary)
5. Bellagio specific setup
13/36
![Page 14: Integrating a Hardware Video Codec into Android Stagefright ...](https://reader030.fdocuments.in/reader030/viewer/2022021416/586987201a28abc2638bf3b5/html5/thumbnails/14.jpg)
OpenMAX IL functions to implement Component represented as struct OMX_COMPONENTTYPE
Need to implement/customize (at a minimum):
Configuration Interface– (*SetParameter)(…) - Set component properties
– (*GetParameter)(…) - Get component properties
– (*SetCallbacks)(…) - Set callbacks to use
Data Interface– (*EmptyThisBuffer)(…) - Process an input buffer
– (*FillThisBuffer)(…) - Process an output buffer
Buffer allocation (if necessary)– (*UseBuffer)(…) - Application allocated buffer
– (*AllocateBuffer)(…) - Component allocated buffer
All prototypes in include/OMX_Component.h 14/36
![Page 15: Integrating a Hardware Video Codec into Android Stagefright ...](https://reader030.fdocuments.in/reader030/viewer/2022021416/586987201a28abc2638bf3b5/html5/thumbnails/15.jpg)
Component Implementation:Configuration Interface
Application callbacks
– Callback when errors or other events occure.g. OMX_EventPortSettingsChanged
• can be used to inform application of changes to decoded frame size, etc.
15/36
![Page 16: Integrating a Hardware Video Codec into Android Stagefright ...](https://reader030.fdocuments.in/reader030/viewer/2022021416/586987201a28abc2638bf3b5/html5/thumbnails/16.jpg)
Component Implementation: Data Interface
FillThisBuffer()/EmptyThisBuffer()called from application
FillBufferDone()/EmptyBufferDone()event from component
Bellagio default implementation (need to customize) through BufferMgmtFunction()
Application Input Port Output Port Application
Component
16/36
![Page 17: Integrating a Hardware Video Codec into Android Stagefright ...](https://reader030.fdocuments.in/reader030/viewer/2022021416/586987201a28abc2638bf3b5/html5/thumbnails/17.jpg)
Component Implementation:Port Buffer Allocation
Buffer Allocation
– OMX_UseBuffer() – use application allocated buffers to transfer data
– OMX_AllocateBuffer()– Ask component to allocate the buffers and return pointers to application
Bellagio base will malloc() buffers, but you can tailor to your H/W requirements
17/36
![Page 18: Integrating a Hardware Video Codec into Android Stagefright ...](https://reader030.fdocuments.in/reader030/viewer/2022021416/586987201a28abc2638bf3b5/html5/thumbnails/18.jpg)
Bellagio Specific
1. Compile Bellagio
2. library_entry_point.c
– Component name should start with “OMX.”
– Component role what does the component do?
3. Compile component into mycomponent.so
4. Copy mycomponent.so to /lib/bellagio
5. Run omxregister-bellagio to create ~/.omxregister
18/36
![Page 19: Integrating a Hardware Video Codec into Android Stagefright ...](https://reader030.fdocuments.in/reader030/viewer/2022021416/586987201a28abc2638bf3b5/html5/thumbnails/19.jpg)
And finally (for this section)
Access from application
– via component name from library_entry_point.c
Possible applications
– Bellagio sample application
– GStreamer via GstOpenMAX
– or Android Stagefright
19/36
![Page 20: Integrating a Hardware Video Codec into Android Stagefright ...](https://reader030.fdocuments.in/reader030/viewer/2022021416/586987201a28abc2638bf3b5/html5/thumbnails/20.jpg)
Overview
Hardware
OpenMAX IL/Bellagio
Android Stagefright Integration
– Video Decoder Specific Considerations
20/36
![Page 21: Integrating a Hardware Video Codec into Android Stagefright ...](https://reader030.fdocuments.in/reader030/viewer/2022021416/586987201a28abc2638bf3b5/html5/thumbnails/21.jpg)
OpenMAX IL Client
Bellagio
VPU
RendererDecoderParser
VEUdatadata pointer
Stagefright Application
Data input, parsing, and output are supplied natively by Stagefright.
Link Bellagio to Stagefright through libstagefrighthw.so
Bellagio
Stagefright
Parser AVC Codec
VPU
to FBRenderer
BellagioParser
AVC Codec
VPU
to FBRenderer
libstagefrighthw
OMXCodec
Stagefright
21/36
![Page 22: Integrating a Hardware Video Codec into Android Stagefright ...](https://reader030.fdocuments.in/reader030/viewer/2022021416/586987201a28abc2638bf3b5/html5/thumbnails/22.jpg)
Linking Bellagio to Stagefright
Edit OMXCodec.cpp only – Register component
– Configure component
– Configure Stagefright– frameworks/base/media/
libstagefright/OMXCodec.cpp
Create libstagefrighthw.so– Access to Bellagio
– see hardware/xxx/libstagefrighthw
Bellagio
AVC Codec
VPU
libstagefrighthw
OMXCodec
Stagefright
Edit
Create
22/36
![Page 23: Integrating a Hardware Video Codec into Android Stagefright ...](https://reader030.fdocuments.in/reader030/viewer/2022021416/586987201a28abc2638bf3b5/html5/thumbnails/23.jpg)
OMXCodec.cpp:Component registration
Update component list in OMXCodec.cpp
const CodecInfo kDecoderInfo[] = {
...
{MEDIA_MIMETYPE_VIDEO_AVC,”OMX.mydecode.avc”},
{MEDIA_MIMETYPE_VIDEO_AVC,”OMX.another.avc”},
{MEDIA_MIMETYPE_VIDEO_AVC,”AVCDecoder”},
...
}
Codec name must start with “OMX.” so Stagefright
knows it’s an external codec
23/36
![Page 24: Integrating a Hardware Video Codec into Android Stagefright ...](https://reader030.fdocuments.in/reader030/viewer/2022021416/586987201a28abc2638bf3b5/html5/thumbnails/24.jpg)
OMXCodec.cpp:Component Configuration
Additional component settings– Stagefright configures most settings automatically
– Sometimes we need some extra settings
– Before sending data Stagefright calls OMXCodec::configureCodec()
– Edit OMXCodec::configureCodec() to add any codec specific initialization you like
24/36
![Page 25: Integrating a Hardware Video Codec into Android Stagefright ...](https://reader030.fdocuments.in/reader030/viewer/2022021416/586987201a28abc2638bf3b5/html5/thumbnails/25.jpg)
OMXCodec.cpp:Stagefright Configuration
Customize Stagefright behaviour– return value of OMXCodec::getComponentQuirks(…)
– quirks: properties of your component that Stagefright can adapt to.
– bitmap constants defined in:
frameworks/base/include/media/stagefright/OMXCodec.h
25/36
![Page 26: Integrating a Hardware Video Codec into Android Stagefright ...](https://reader030.fdocuments.in/reader030/viewer/2022021416/586987201a28abc2638bf3b5/html5/thumbnails/26.jpg)
Stagefright configuration (cont):Example quirks
Allocate buffers with OMX_AllocateBuffer() instead of OMX_UseBuffer() kRequiresAllocateBufferOnOutputPorts
No data (pointer) or buffer post-processing req’d. kOutputBuffersAreUnreadable
Output buffers allocated after frame size determined kDefersOutputBufferAllocation
26/36
![Page 27: Integrating a Hardware Video Codec into Android Stagefright ...](https://reader030.fdocuments.in/reader030/viewer/2022021416/586987201a28abc2638bf3b5/html5/thumbnails/27.jpg)
libstagefrighthw.so: OMX plugin
Create libstagefrighthw.so with override ofclass OMXPluginBase {
virtual makeComponentInstance(…);
virtual destroyComponentInstance(…);
virtual enumerateComponents(…);
virtual getRolesOfComponent(…);
}
Define class factory functionOMXPluginBase *createOMXPlugin() {
return new myOMXPlugin;
}
Make a component
Destroy a component
List available components
Get component roles
27/36
Bellagio
AVC Codec
VPU
libstagefrighthw
OMXCodec
Stagefright
Create
![Page 28: Integrating a Hardware Video Codec into Android Stagefright ...](https://reader030.fdocuments.in/reader030/viewer/2022021416/586987201a28abc2638bf3b5/html5/thumbnails/28.jpg)
Prepare Bellagio for Stagefright
Compile Bellagio core/component on Android
– Must use Android build environment
Stagefright and Bellagio versions must match
– include/bellagio/omxcore.h
• SPECVERSIONMAJOR = 1
• SPECVERSIONMINOR = 0
Component registry
– copy .omxregistry to Android rootfs (e.g. /system/etc)– export OMX_BELLAGIO_REGISTRY=/<path>/.omxregistry
28/36
![Page 29: Integrating a Hardware Video Codec into Android Stagefright ...](https://reader030.fdocuments.in/reader030/viewer/2022021416/586987201a28abc2638bf3b5/html5/thumbnails/29.jpg)
Integration complete (maybe)
Can’t we get rid of all this output copying?
Can we process the video fast enough?
BellagioParser
AVC Codec
VPU
to FBRenderer
libstagefrighthw
OMXCodec
Stagefright
001101010010
29/36
![Page 30: Integrating a Hardware Video Codec into Android Stagefright ...](https://reader030.fdocuments.in/reader030/viewer/2022021416/586987201a28abc2638bf3b5/html5/thumbnails/30.jpg)
Video Decoder Considerations
Custom renderer
– Default render path requires data copying
– Custom renderer may avoid copying
– Might have other uses
T/L conversion (hardware dependant)
– Increase memory efficiency and decode speed
– (Need a custom renderer to use this)
30/36
![Page 31: Integrating a Hardware Video Codec into Android Stagefright ...](https://reader030.fdocuments.in/reader030/viewer/2022021416/586987201a28abc2638bf3b5/html5/thumbnails/31.jpg)
Bypassing default renderer:Custom Renderer
– direct to output device (reduce copying)
– H/W scaling, color conversion
– process custom frame data
to FB
BellagioParser
AVC Codec
VPU
Renderer
libstagefrighthw
OMXCodec
Stagefright
BellagioParser
AVC Codec
VPU
OMXCodec
Stagefright
libstagefrighthw
Renderer
VEUto FBdata
data pointer
31/36
![Page 32: Integrating a Hardware Video Codec into Android Stagefright ...](https://reader030.fdocuments.in/reader030/viewer/2022021416/586987201a28abc2638bf3b5/html5/thumbnails/32.jpg)
Custom Renderer (cont’d)
Also in libstagefrighthw.so
Renderer is NOT an OMX component
Override
class VideoRenderer {
virtual VideoRenderer(…);
virtual render(…,void *platformPrivate);
}
Implement class factory function
Passed up from OpenMAX decoder with each buffer
VideoRenderer *createRenderer(...) {
return new MyVideoRenderer(...);
}
Called to render each decoded frame
32/36
![Page 33: Integrating a Hardware Video Codec into Android Stagefright ...](https://reader030.fdocuments.in/reader030/viewer/2022021416/586987201a28abc2638bf3b5/html5/thumbnails/33.jpg)
Faster video processing: Tiling/Linear conversion Tiling/Linear conversion faster memory access
when coding macroblocks
Normal byte order
T/L conversion
Bytes from the same macroblock may be spread all over memory
Bytes from the same macroblock stay together faster access (caching, burst memory transfers, etc)
33/36
![Page 34: Integrating a Hardware Video Codec into Android Stagefright ...](https://reader030.fdocuments.in/reader030/viewer/2022021416/586987201a28abc2638bf3b5/html5/thumbnails/34.jpg)
T/L conversion and thumbnails
Problem– When using T/L conversion (or other H/W features) buffers
are unreadable by S/W
(kOutputBuffersAreUnreadable)
– Stagefright needs to make thumbnails?!?!
BellagioParser
AVC Codec
VPU
OMXCodec
Stagefright
libstagefrighthw
Renderer
VEUto FB
for thumbnails
34/36
data
data pointer
![Page 35: Integrating a Hardware Video Codec into Android Stagefright ...](https://reader030.fdocuments.in/reader030/viewer/2022021416/586987201a28abc2638bf3b5/html5/thumbnails/35.jpg)
T/L conversion and thumbnails
Solution– Thumbnail mode: Stagefright calls OMXCodec::configureCodec()with kClientNeedsFramebuffer flag set
– Codec settings can then be adjusted
eg. T/L conversion disabled, necessary data copied, etc
BellagioParser
AVC Codec
VPU
OMXCodec
Stagefright
libstagefrighthw
Renderer
VEUto FB
for thumbnails
data
data pointer 35/36
![Page 36: Integrating a Hardware Video Codec into Android Stagefright ...](https://reader030.fdocuments.in/reader030/viewer/2022021416/586987201a28abc2638bf3b5/html5/thumbnails/36.jpg)
Summary
External video and audio codecs are linked to Stagefright through OpenMAX IL
Bellagio is a reasonable implementation to use
Use quirks to help with integration
Check out the examples in the Android source
BellagioParser
AVC Codec
VPU
OMXCodec
Stagefright
libstagefrighthw
Renderer
VEUto FB
for thumbnails
36/36
data
data pointer