Mastering the Android Media Framework
-
Upload
best-tech-videos -
Category
Documents
-
view
144 -
download
1
description
Transcript of Mastering the Android Media Framework
![Page 1: Mastering the Android Media Framework](https://reader035.fdocuments.in/reader035/viewer/2022081716/5466a545b4af9f06758b47e8/html5/thumbnails/1.jpg)
![Page 2: Mastering the Android Media Framework](https://reader035.fdocuments.in/reader035/viewer/2022081716/5466a545b4af9f06758b47e8/html5/thumbnails/2.jpg)
Noisy Androids Mastering the Android Media FrameworkDave Sparks27-May-2009
![Page 3: Mastering the Android Media Framework](https://reader035.fdocuments.in/reader035/viewer/2022081716/5466a545b4af9f06758b47e8/html5/thumbnails/3.jpg)
3
Agenda
• Frank Lloyd Android: Media Framework architecture• Sweet Android: What's new in Cupcake V1.5?• Busted Android: Common problems• Curious Android: Q & A
![Page 4: Mastering the Android Media Framework](https://reader035.fdocuments.in/reader035/viewer/2022081716/5466a545b4af9f06758b47e8/html5/thumbnails/4.jpg)
Architecture
![Page 5: Mastering the Android Media Framework](https://reader035.fdocuments.in/reader035/viewer/2022081716/5466a545b4af9f06758b47e8/html5/thumbnails/5.jpg)
5
Design Goals
• Simplify application development• Share resources in multi-tasked environment• Strong security model• Room for future growth
![Page 6: Mastering the Android Media Framework](https://reader035.fdocuments.in/reader035/viewer/2022081716/5466a545b4af9f06758b47e8/html5/thumbnails/6.jpg)
6
Media Framework
Application MediaServer
SurfaceFlinger
AudioFlinger
![Page 7: Mastering the Android Media Framework](https://reader035.fdocuments.in/reader035/viewer/2022081716/5466a545b4af9f06758b47e8/html5/thumbnails/7.jpg)
7
Typical Stack for Media Function Call
DVMProxy JNI Native
ProxyBinderProxy
BinderNative
NativeImplementation
![Page 8: Mastering the Android Media Framework](https://reader035.fdocuments.in/reader035/viewer/2022081716/5466a545b4af9f06758b47e8/html5/thumbnails/8.jpg)
8
Media Server Process
Media Player ServiceMediaRecorderService
CameraService
OpenCORE Vorbis MIDI OpenCORE Camera
![Page 9: Mastering the Android Media Framework](https://reader035.fdocuments.in/reader035/viewer/2022081716/5466a545b4af9f06758b47e8/html5/thumbnails/9.jpg)
9
Media Playback
Application MediaServer
SurfaceFlinger
AudioFlinger
source URI
ISurface
ISurface
audio type audio type
![Page 10: Mastering the Android Media Framework](https://reader035.fdocuments.in/reader035/viewer/2022081716/5466a545b4af9f06758b47e8/html5/thumbnails/10.jpg)
10
Media Recorder
Application MediaServer
SurfaceFlinger
AudioFlinger
setCamera
ISurface
ISurface
selectAudio audio
CameraService
Camera
![Page 11: Mastering the Android Media Framework](https://reader035.fdocuments.in/reader035/viewer/2022081716/5466a545b4af9f06758b47e8/html5/thumbnails/11.jpg)
Codecs
![Page 12: Mastering the Android Media Framework](https://reader035.fdocuments.in/reader035/viewer/2022081716/5466a545b4af9f06758b47e8/html5/thumbnails/12.jpg)
12
H.263 Video
• Originally designed for low bit-rate video conferencing• Simple encoder• 3GPP standard• Used by many streaming sites for low bit-rate video
![Page 13: Mastering the Android Media Framework](https://reader035.fdocuments.in/reader035/viewer/2022081716/5466a545b4af9f06758b47e8/html5/thumbnails/13.jpg)
13
MPEG4-SP (Simple Profile) Video
• Designed as replacement for MPEG1/2 codecs• Simple encoder• Is it really an improvement over H.263?
![Page 14: Mastering the Android Media Framework](https://reader035.fdocuments.in/reader035/viewer/2022081716/5466a545b4af9f06758b47e8/html5/thumbnails/14.jpg)
14
H.264 AVC (Advanced Video Codec)
• Better compression (e.g. multiple reference frames)• Better quality (e.g. mandatory in-loop deblocking filter)• Different profiles target different applications and devices• Uses include digital cinema, HDTV broadcast, and mobile• More complex than H.263 or MPEG4-SP
![Page 15: Mastering the Android Media Framework](https://reader035.fdocuments.in/reader035/viewer/2022081716/5466a545b4af9f06758b47e8/html5/thumbnails/15.jpg)
15
MP3
• First generation psycho-acoustic compression• Approximately 10:1 compression @ 128kbps• Sonic transparency 192Kbps
![Page 16: Mastering the Android Media Framework](https://reader035.fdocuments.in/reader035/viewer/2022081716/5466a545b4af9f06758b47e8/html5/thumbnails/16.jpg)
16
AAC (Advanced Audio Codec)
• Psycho-acoustic compression like MP3• Better compression than MP3• Sonic transparency 128Kbps• Commonly used in MPEG-4 streams
![Page 17: Mastering the Android Media Framework](https://reader035.fdocuments.in/reader035/viewer/2022081716/5466a545b4af9f06758b47e8/html5/thumbnails/17.jpg)
17
Ogg Vorbis
• Psycho-acoustic compression like MP3• Better compression than MP3• Low-overhead player
– Lower latency– Uses less memory
• Can loop seamlessly (unlike MP3)
![Page 18: Mastering the Android Media Framework](https://reader035.fdocuments.in/reader035/viewer/2022081716/5466a545b4af9f06758b47e8/html5/thumbnails/18.jpg)
18
Adaptive Multi-rate (AMR) Audio
• Two flavors: Narrow band and wide band• Narrow band is 8KHz input, bit-rates 4.75K to 12.2K bps• Wide band to 16KHz input, bit-rates 6.6K to 23.85K bps• Used in 3GP streams
![Page 19: Mastering the Android Media Framework](https://reader035.fdocuments.in/reader035/viewer/2022081716/5466a545b4af9f06758b47e8/html5/thumbnails/19.jpg)
Streams
![Page 20: Mastering the Android Media Framework](https://reader035.fdocuments.in/reader035/viewer/2022081716/5466a545b4af9f06758b47e8/html5/thumbnails/20.jpg)
20
Typical 3GPP Stream
• Lower quality• H.263 video codec• AMR-NB audio codec• Bit rates up to 192K bps
![Page 21: Mastering the Android Media Framework](https://reader035.fdocuments.in/reader035/viewer/2022081716/5466a545b4af9f06758b47e8/html5/thumbnails/21.jpg)
21
Typical MPEG-4 Stream
• Usually higher quality than 3GPP• H.264 video codec• AAC audio codec• Bit rates up to 500K bps
![Page 22: Mastering the Android Media Framework](https://reader035.fdocuments.in/reader035/viewer/2022081716/5466a545b4af9f06758b47e8/html5/thumbnails/22.jpg)
22
What container/codec type should I use?
• Authoring for Android device, use MPEG4 container with H.264 video (HVGA up to 500Kbps) and AAC 96Kbps
• Creating content on Android device for other Android devices: Use 3GPP container with H.263 video (CIF up to 384Kbps) and AMR-NB audio
• Creating content on Android device for other devices: Use 3GPP container with H.263 video (QCIF up to 192Kbps) and AMR-NB audio
• Make sure that ‘moov’ atom is located before ‘mdat’ atom for HTTP progressive streaming
![Page 23: Mastering the Android Media Framework](https://reader035.fdocuments.in/reader035/viewer/2022081716/5466a545b4af9f06758b47e8/html5/thumbnails/23.jpg)
New Features in V1.5
![Page 24: Mastering the Android Media Framework](https://reader035.fdocuments.in/reader035/viewer/2022081716/5466a545b4af9f06758b47e8/html5/thumbnails/24.jpg)
24
New Media Features For Cupcake V1.5
• Video recording• AudioTrack• AudioRecord• JET interactive MIDI engine
![Page 25: Mastering the Android Media Framework](https://reader035.fdocuments.in/reader035/viewer/2022081716/5466a545b4af9f06758b47e8/html5/thumbnails/25.jpg)
25
AudioTrack and AudioRecord
• Expose raw PCM audio streams to applications• AudioTrack: Write PCM audio directly to mixer engine• AudioRecord: Read PCM audio directly from mic• Callback mechanism for threaded applications• Static buffer for playing sound effects
![Page 26: Mastering the Android Media Framework](https://reader035.fdocuments.in/reader035/viewer/2022081716/5466a545b4af9f06758b47e8/html5/thumbnails/26.jpg)
26
JET Interactive MIDI Engine
• Based on MIDI - file sizes can be small• DLS support allows for better quality instruments• Precise synchronization for layered MIDI tracks• Native code - very efficient• Synchronization callbacks to applications for rhythm games• Open source engine and creation tools• VST plugin - use it inside your favorite DAW tool
![Page 27: Mastering the Android Media Framework](https://reader035.fdocuments.in/reader035/viewer/2022081716/5466a545b4af9f06758b47e8/html5/thumbnails/27.jpg)
Common Problems
![Page 28: Mastering the Android Media Framework](https://reader035.fdocuments.in/reader035/viewer/2022081716/5466a545b4af9f06758b47e8/html5/thumbnails/28.jpg)
28
Problem: Volume control behavior is inconsistent
• Volume control is overloaded• AudioManager has a default strategy for controlling volume:
– If in-call, adjust in-call volume– If ringing, mute ringer– If media track is active, adjust media volume– Otherwise, adjust ringtone volume
• In an application that plays sounds periodically, the volume behavior is not consistent
![Page 29: Mastering the Android Media Framework](https://reader035.fdocuments.in/reader035/viewer/2022081716/5466a545b4af9f06758b47e8/html5/thumbnails/29.jpg)
29
Solution: Set the default stream type
import android.app.Activity; import android.media.AudioManager; ... public void onCreate(Bundle icicle) { super.onCreate(icicle); setVolumeControlStream(AudioManager.STREAM_MUSIC); ... }
![Page 30: Mastering the Android Media Framework](https://reader035.fdocuments.in/reader035/viewer/2022081716/5466a545b4af9f06758b47e8/html5/thumbnails/30.jpg)
30
Problem: Unable to play file from resource?
MediaPlayer mp = new MediaPlayer(); try { mp.setDataSource(“res:com.myapp.raw.test”); mp.prepare(); } catch (IOException e) { Log.e(“Error ” + e.print() + “ opening media player”); }
![Page 31: Mastering the Android Media Framework](https://reader035.fdocuments.in/reader035/viewer/2022081716/5466a545b4af9f06758b47e8/html5/thumbnails/31.jpg)
31
Solution: Use AssetFileDescriptor
AssetFileDescriptor afd = context.getResources().openRawResourceFd(resId);
set.DataSource(afd.getFileDescriptor(), afd.getStartOffset(), afd.getLength());
afd.close();
![Page 32: Mastering the Android Media Framework](https://reader035.fdocuments.in/reader035/viewer/2022081716/5466a545b4af9f06758b47e8/html5/thumbnails/32.jpg)
32
Problem: Out of MediaPlayers!
MediaPlayer mpArray = new Object[50]; for (int i = 0; i < 50; i++) { MediaPlayer mp = MediaPlayer.create(soundName[i]); mp.prepare(); mpArray[i] = mp; }
![Page 33: Mastering the Android Media Framework](https://reader035.fdocuments.in/reader035/viewer/2022081716/5466a545b4af9f06758b47e8/html5/thumbnails/33.jpg)
33
Solution: Reuse MediaPlayer
• Call release() and set to null• Or call reset(), then setDataSource()• Limit to 2 or 3 maximum
![Page 34: Mastering the Android Media Framework](https://reader035.fdocuments.in/reader035/viewer/2022081716/5466a545b4af9f06758b47e8/html5/thumbnails/34.jpg)
34
Problem: CPU Overloaded
MediaPlayer sound1 = new MediaPlayer(); MediaPlayer sound2 = new MediaPlayer(); MediaPlayer sound3 = new MediaPlayer(); // call setDataSource, prepare, etc. ... // later on... sound1.start(); sound2.start(); sound3.start(); ... // CPU is bogging down here...
![Page 35: Mastering the Android Media Framework](https://reader035.fdocuments.in/reader035/viewer/2022081716/5466a545b4af9f06758b47e8/html5/thumbnails/35.jpg)
35
Solution: Use SoundPool
import android.media.SoundPool;import android.media.AudioSystem;
Context context = getApplicationContext();sp = new SoundPool(maxStreams, AudioSystem.STREAM_MUSIC, 0);int snd1 = sp.load(context, res.Raw.pow);int snd2 = sp.load(context, res.Raw.blam);int snd2 = sp.load(context, res.Raw.biff);...sp.play(snd1, leftVol, rightVol, priority, loop, rate);...sp.play(snd2, leftVol, rightVol, priority, loop, rate);...sp.play(snd3, leftVol, rightVol, priority, loop, rate);
![Page 36: Mastering the Android Media Framework](https://reader035.fdocuments.in/reader035/viewer/2022081716/5466a545b4af9f06758b47e8/html5/thumbnails/36.jpg)
Q & A
![Page 37: Mastering the Android Media Framework](https://reader035.fdocuments.in/reader035/viewer/2022081716/5466a545b4af9f06758b47e8/html5/thumbnails/37.jpg)
![Page 38: Mastering the Android Media Framework](https://reader035.fdocuments.in/reader035/viewer/2022081716/5466a545b4af9f06758b47e8/html5/thumbnails/38.jpg)
38
![Page 39: Mastering the Android Media Framework](https://reader035.fdocuments.in/reader035/viewer/2022081716/5466a545b4af9f06758b47e8/html5/thumbnails/39.jpg)
Security
![Page 40: Mastering the Android Media Framework](https://reader035.fdocuments.in/reader035/viewer/2022081716/5466a545b4af9f06758b47e8/html5/thumbnails/40.jpg)
40
Media Server Security
• Isolate file parsers and codecs in a separate process• Centralize hardware resource management and security• Enforce permissions on camera and audio record• Media server runs at reduced privileges• Applications grant file access on as-needed basis
![Page 41: Mastering the Android Media Framework](https://reader035.fdocuments.in/reader035/viewer/2022081716/5466a545b4af9f06758b47e8/html5/thumbnails/41.jpg)
41
Media Server Security (continued)
• Media server has read/write access to SD card only• Applications typically pass in an open file descriptor• Only the media server process has direct access to hardware
resources• Applications must include permissions in their manifest to
access camera and microphone
![Page 42: Mastering the Android Media Framework](https://reader035.fdocuments.in/reader035/viewer/2022081716/5466a545b4af9f06758b47e8/html5/thumbnails/42.jpg)
42
How does sandboxing defeat a hypothetical attack?
• Assume we have a buffer overrun vulnerability in a codec• Hacker creates a malicious stream that exploits vulnerability• Assume that hacker can run arbitrary code in the media server
process• Media server can access SD card, but not data or system
partition• Exposed data from data partition is limited to files that
applications have explicitly opened for the media server• DOS attacks are possible, but limited in scope due to
sandboxing• No data from application is vulnerable
![Page 43: Mastering the Android Media Framework](https://reader035.fdocuments.in/reader035/viewer/2022081716/5466a545b4af9f06758b47e8/html5/thumbnails/43.jpg)
![Page 44: Mastering the Android Media Framework](https://reader035.fdocuments.in/reader035/viewer/2022081716/5466a545b4af9f06758b47e8/html5/thumbnails/44.jpg)
44
![Page 45: Mastering the Android Media Framework](https://reader035.fdocuments.in/reader035/viewer/2022081716/5466a545b4af9f06758b47e8/html5/thumbnails/45.jpg)
45
Why Do I Need to Use This Template?
• We all want to present the same picture of Google– As a unified company with a single brand promise
• By standardizing on a common template, slides can be shared and moved between files with no disparity in their look and feel
• By standardizing on a common template, slides can be shared and moved between files with no disparity in their look and feel
![Page 46: Mastering the Android Media Framework](https://reader035.fdocuments.in/reader035/viewer/2022081716/5466a545b4af9f06758b47e8/html5/thumbnails/46.jpg)
46
Presentation Bullet Slide Layout
• Use this template to create your presentation• Use the provided color palette, box and arrow graphics,
and chart styles• Instructions are provided to assist you in using this presentation
template effectively• At all times, strive to maintain Google’s corporate
look and feel
![Page 47: Mastering the Android Media Framework](https://reader035.fdocuments.in/reader035/viewer/2022081716/5466a545b4af9f06758b47e8/html5/thumbnails/47.jpg)
47
Bullet Slide With Subtitle Placeholder
• Select the subtitle text box above, copy, and paste it on to the slide that requires a subtitle– Vertical position setting for subtitle is 1.01"– Vertical position setting for bullet text is 1.5"– This way the subtitles will be in the exact position throughout the
presentation and will not jump from slide to slide
• Subtitle capitalization is sentence case• Subtitles never have a period at the end
Chart title or subtitle placeholder
![Page 48: Mastering the Android Media Framework](https://reader035.fdocuments.in/reader035/viewer/2022081716/5466a545b4af9f06758b47e8/html5/thumbnails/48.jpg)
48
Code Slide With Subtitle PlaceholderChart title or subtitle placeholder
var map = new GMap2(document.getElementById("map")); map.setCenter(new GLatLng(37.442, -122.142), 13); var bounds = map.getBounds(); var lngSpan = northEast.lng() - southWest.lng(); var latSpan = northEast.lat() - southWest.lat(); for (var i = 0; i < 10; i++) { var point = new GLatLng(southWest.lat() + latSpan * Math.random(), southWest.lng() + lngSpan * Math.random()); map.addOverlay(new GMarker(point)); }
![Page 49: Mastering the Android Media Framework](https://reader035.fdocuments.in/reader035/viewer/2022081716/5466a545b4af9f06758b47e8/html5/thumbnails/49.jpg)
49
Color Palette
Primary
Neutral
Picker
R 0G 102B 204
R 125G 125B 125
R 0G 138B 53
R 255G 0B 0
R 255G 211B 25
![Page 50: Mastering the Android Media Framework](https://reader035.fdocuments.in/reader035/viewer/2022081716/5466a545b4af9f06758b47e8/html5/thumbnails/50.jpg)
50
Graphic Element Styles and Arrows
![Page 51: Mastering the Android Media Framework](https://reader035.fdocuments.in/reader035/viewer/2022081716/5466a545b4af9f06758b47e8/html5/thumbnails/51.jpg)
51
Pie Chart ExampleChart title or subtitle placeholder
Google48%
MSN19%
Yahoo33%
Source: Insight Express Survey of 1000 home Internet users (June 2004)
![Page 52: Mastering the Android Media Framework](https://reader035.fdocuments.in/reader035/viewer/2022081716/5466a545b4af9f06758b47e8/html5/thumbnails/52.jpg)
52
Column Chart Example
• Chart is positioned to the left of the text box aligned to left guide
• Depending on size of chart and amount of text content, positioning of text box may vary
• Font size can be decreased
Source: Insight Express Survey of 1000 home Internet users (June 2004)
0
12.5
25.0
37.5
50.0
Google Yahoo MSN
Chart title or subtitle placeholder
![Page 53: Mastering the Android Media Framework](https://reader035.fdocuments.in/reader035/viewer/2022081716/5466a545b4af9f06758b47e8/html5/thumbnails/53.jpg)
53
Line Chart Example
0%
225%
450%
675%
900%
1 2 3 4 5 6 7 8 9 10 11 12
• Chart is positioned to the left of the text box aligned to left guide
• Depending on size of chart and amount of text content, positioning of text box may vary
• Font size can be decreased
Source: Insight Express Survey of 1000 home Internet users (June 2004)
Chart title or subtitle placeholder
![Page 54: Mastering the Android Media Framework](https://reader035.fdocuments.in/reader035/viewer/2022081716/5466a545b4af9f06758b47e8/html5/thumbnails/54.jpg)
54
Attract Customers
Build Customer
Loyalty
Develop New
Revenue Streams
Monetize Your Digital AssetsMonetize Your Digital Assets
WebSearch
AdSense for Search
AdSense for Content
Matrix ExampleChart title or subtitle placeholder
![Page 55: Mastering the Android Media Framework](https://reader035.fdocuments.in/reader035/viewer/2022081716/5466a545b4af9f06758b47e8/html5/thumbnails/55.jpg)
Segue Slide
![Page 56: Mastering the Android Media Framework](https://reader035.fdocuments.in/reader035/viewer/2022081716/5466a545b4af9f06758b47e8/html5/thumbnails/56.jpg)
“The perfect search engine would understand exactly what you mean and give back exactly what you want.”
Larry Page, Google Co-Founder