Efficient multimediasupport in QtWebKit
on Raspberry Pi
Talk outline
• Raspberry Pi
• WebKit architecture(s) with emphasis on Media playback
• Platform-level improvements for QtWebKit
Raspberry Pi - Model B
• 700MHz single-core ARMv6 CPU• Broadcom BCM2835 VideoCore IV• 512 MB RAM (partly allocated for GPU)
GStreamer OpenMax (gst-omx)
• Hardware decoders for H.264, VP8, Theora
• meta:EGLImage
• Custom audio sinks: HDMI and analog
• Very good integration within playbin
Premises of the project
• Fullscreen browser, no tabs, no WM
• Focus on media consumption web-apps and Youtube TV
• OpenGL for rendering
• Qt5 platform, EGLFS QPA
WebKit rendering - from HTML to the screen
WebKit rendering - the forest
WebKit1 rendering (with OpenGL)
• GraphicsLayers are backed by GL textures
• The video player has its own GraphicsLayer
• RenderLayers are traversed and painted recursively (to their GraphicsLayers): children below -> self -> children above
• Composition: GraphicsLayers are stacked to produce the final result
WebKit2 rendering (OpenGL) - processes
• Rendering split into 2 processes: UI and Web
• Web process renders the layers (similar to WebKit1 rendering)
• UI process composites and shows them
• GraphicsLayers shared using GraphicsSurfaces
• A GraphicsSurface encapsulates a method to share gl textures among processes (EGLImages)
WebKit2 rendering (OpenGL) - Coordinated graphics
The MediaPlayer in WebCore
MediaPlayer - platform level
Video rendering - the inefficient way
• Sink negotiates xRGB or ARGB video caps
• Optional buffer copy in ::render, conversion to Cairo's ARGB pre-multiplied alpha
• Buffer data copy in MediaPlayerPrivate::paint to a WebCore::BitmapImage
• BitmapImage rendered in a QPixmap
• Another approach: GLTextureUpload meta (CPU -> GPU buffer copy)
WebKit2 video rendering (inefficient)
Video sink - improved integration withinthe platform
• Sink negotiates meta:EGLImage with upstream video decoder
• Buffer pool and EGLImage allocator (like in glimagesink)
• EGLImage encapsulated in a GstBuffer passed to MediaPlayerPrivate
• Efficient rendering in the player, different approaches for WebKit1 and WebKit2
Improvements in WebKit2 rendering
Improving media resources loading
Using libsoup within QtWebKit
• The Qt-based resource loader doesn't cope well with large media resources
• The libsoup-based resource loader is more memory efficient and better integrated in WebCore
• Pre-allocation of GstBuffers within our HTTP source element
WebRTC / getUserMedia
Local devices listing/playback
• In JS: webkitGetUserMedia({audio: true}, successCallback, failureCallback);
• function successCallback(stream) { video.src = URL.createObjectURL(stream); video.play(); }
• Pipeline should also allow encoding and RTP payloading if required by webapp (WebRTC PeerConnection)
WebRTC - RPiCam integration
• rpicamsrc: emits H.264 byte stream
• Leverage H.264 hardware rendering
• rpicamsrc ! h264parse ! omxh264dec ! sink
• with a tee downstream rpicamsrc to allow encoding/streaming later on
WebAudio
WebAudio - basics
• JS API for processing and synthesizing audio in web applications
• Pipeline based. Examples of nodes: BufferSource, GainNode, BiQuadFilter, Destination...
• Head-related transfer function: tricking the human ear to spatial sound
WebAudio - decoding incoming files
• First scenario: local file. Leverage decodebin and appsink to create an in-memory representation of the audio file PCM data (FloatArray).
• Second scenario: memory buffer. Decodebin again! With giostreamsrc
• No RPi-specific optimisations
WebAudio - playback
• Playback of WebAudio FloatArrays PCM samples to soundcard
• Custom GStreamer source element encoding internal WebCore audio buffers to WAV
• Again, no RPi-specific optimisations
So which part of the backend needed optimisations?
WebAudio - HRTF Database
• Loaded at AudioContext creation, from a separate thread
• 240 very small (256 frames) WAV files => lots of context switches
• Improvement: concatenate all samples and internally split when building the database
• => only one file to read, less context switches!
WebAudio - FFT experiments
• default WebAudio FFTFrame backend using GstFFT (itself based on KissFFT)
• math functions hot on perf profiles
• GPUFFT library (from rpi-userland): leveraging VideoCore for FFT calculations
• Major drawback: requires allocated RAM, so less RAM for the other libraries
Next tasks
• video/canvas and video/webgl integration
• WebRTC
• ORC ARMv6 backend?
Demo rendering from youtube at 720p
Demo rendering from youtube at 1080p
Flickr pictures
• http://bit.ly/1p8YHJA
• http://bit.ly/1yz8ctT
• http://bit.ly/1v8nwLh
This is the last slide, let's have lunch now!
Top Related