(New) Video Rendering on Frontend and Backend
Transcript of (New) Video Rendering on Frontend and Backend
![Page 1: (New) Video Rendering on Frontend and Backend](https://reader033.fdocuments.in/reader033/viewer/2022060317/62948d7a00630357855fa177/html5/thumbnails/1.jpg)
Video Rendering on Frontend and Backend
![Page 2: (New) Video Rendering on Frontend and Backend](https://reader033.fdocuments.in/reader033/viewer/2022060317/62948d7a00630357855fa177/html5/thumbnails/2.jpg)
2
Lightricks is the creator of numerous award-winning photo and video editing apps. Our goal is to build fun and powerful tools that reinvent the way content is created all over the world.
Established 2013 HQ in Jerusalem, branches in Haifa & London
~400 Employees
12 Apps We’re a Unicorn!
![Page 4: (New) Video Rendering on Frontend and Backend](https://reader033.fdocuments.in/reader033/viewer/2022060317/62948d7a00630357855fa177/html5/thumbnails/4.jpg)
The Playloop
Decode frame
centerX = 0.5 + time * 0.2
Interpolate values
Render frame
Display
Encode to video file
* Links to more info about value interpolation are in the last slide
![Page 5: (New) Video Rendering on Frontend and Backend](https://reader033.fdocuments.in/reader033/viewer/2022060317/62948d7a00630357855fa177/html5/thumbnails/5.jpg)
Can we do that in the browser?
Decode:
HTMLVideoElement
Interpolate:
pure logic
Render:
WebGL
Encode:
Software codec
5
![Page 6: (New) Video Rendering on Frontend and Backend](https://reader033.fdocuments.in/reader033/viewer/2022060317/62948d7a00630357855fa177/html5/thumbnails/6.jpg)
But...
● HTMLVideoElement
○ Can’t give source frame rate
○ Can’t seek to frame
○ Is geared for real-time playing
■ will skip frames during heavy workload
● We can code around it
○ Working against the intended usage
○ Will work on some browsers
6
![Page 7: (New) Video Rendering on Frontend and Backend](https://reader033.fdocuments.in/reader033/viewer/2022060317/62948d7a00630357855fa177/html5/thumbnails/7.jpg)
Alternative
● Software emulation
● FFmpeg library
○ either back-and-forth in backend
○ or large import in frontend
○ Both are slow
7
![Page 8: (New) Video Rendering on Frontend and Backend](https://reader033.fdocuments.in/reader033/viewer/2022060317/62948d7a00630357855fa177/html5/thumbnails/8.jpg)
Let’s split!Browser:
Decode frame with HTMLVideoElement
Render frame Display
Server:
Encode to
video file
Decode frame
with FFmpeg Render frame
![Page 9: (New) Video Rendering on Frontend and Backend](https://reader033.fdocuments.in/reader033/viewer/2022060317/62948d7a00630357855fa177/html5/thumbnails/9.jpg)
Stacks
9
Browser Server
Language Javascript C++
Decoding HTMLVideoElement FFmpeg
![Page 10: (New) Video Rendering on Frontend and Backend](https://reader033.fdocuments.in/reader033/viewer/2022060317/62948d7a00630357855fa177/html5/thumbnails/10.jpg)
Rendering
10
OpenGL / WebGL as API
GL - Graphics Library
Same functionality, different structure
OpenGL - global state machine
WebGL - object oriented
![Page 11: (New) Video Rendering on Frontend and Backend](https://reader033.fdocuments.in/reader033/viewer/2022060317/62948d7a00630357855fa177/html5/thumbnails/11.jpg)
Stacks
11
Browser Server
Language Javascript C++
Decoding HTMLVideoElement FFmpeg
Rendering WebGL OpenGL
![Page 12: (New) Video Rendering on Frontend and Backend](https://reader033.fdocuments.in/reader033/viewer/2022060317/62948d7a00630357855fa177/html5/thumbnails/12.jpg)
Stacks
12
Browser Server
Language Javascript C++
Decoding HTMLVideoElement FFmpeg
Rendering WebGL OpenGL
Text Shaping ... ...
![Page 13: (New) Video Rendering on Frontend and Backend](https://reader033.fdocuments.in/reader033/viewer/2022060317/62948d7a00630357855fa177/html5/thumbnails/13.jpg)
Stacks
13
Browser Server
Language Javascript C++
Decoding HTMLVideoElement FFmpeg
Rendering WebGL OpenGL
Text Shaping ... ...
And more ... ...
![Page 14: (New) Video Rendering on Frontend and Backend](https://reader033.fdocuments.in/reader033/viewer/2022060317/62948d7a00630357855fa177/html5/thumbnails/14.jpg)
WebAssembly
14
WASM in short
Available in browsers
Standard for secure, performant, cross- platform computing
Compiled from any language
* Links to more info about WebAssembly are in the last slide
![Page 15: (New) Video Rendering on Frontend and Backend](https://reader033.fdocuments.in/reader033/viewer/2022060317/62948d7a00630357855fa177/html5/thumbnails/15.jpg)
Emscripten - compiler toolchain
15
Compile code to WASM
OpenGL + Cpp = WASM + JavaScript + WebGL
Create JavaScript bindings Abstract interfaces to use JavaScript objects
![Page 16: (New) Video Rendering on Frontend and Backend](https://reader033.fdocuments.in/reader033/viewer/2022060317/62948d7a00630357855fa177/html5/thumbnails/16.jpg)
Stacks
16
Browser Server
Language Javascript & C++ C++
Decoding HTMLVideoElement FFmpeg
Rendering OpenGL
Text Shaping ...
And more ...
![Page 17: (New) Video Rendering on Frontend and Backend](https://reader033.fdocuments.in/reader033/viewer/2022060317/62948d7a00630357855fa177/html5/thumbnails/17.jpg)
Let’s combine code!
Decode frame with HTMLVideoElement
Render frame
Display
Browser:
Encode to video file
Decode frame
with FFmpeg
Server:
![Page 18: (New) Video Rendering on Frontend and Backend](https://reader033.fdocuments.in/reader033/viewer/2022060317/62948d7a00630357855fa177/html5/thumbnails/18.jpg)
Story Time!
18
![Page 19: (New) Video Rendering on Frontend and Backend](https://reader033.fdocuments.in/reader033/viewer/2022060317/62948d7a00630357855fa177/html5/thumbnails/19.jpg)
WebAssembly — What does security mean?
WASM has separate memory
19
This means that everything must be copied
Can’t access runtime memory WASI — WebAssembly System Interface
![Page 20: (New) Video Rendering on Frontend and Backend](https://reader033.fdocuments.in/reader033/viewer/2022060317/62948d7a00630357855fa177/html5/thumbnails/20.jpg)
Loading frames to textures
20
Textures represented by opaque JS objects
Has API for fast loading of textures from HTMLVideoElement
What’s a HTMLVideoElement?
Textures represented by IDs
![Page 21: (New) Video Rendering on Frontend and Backend](https://reader033.fdocuments.in/reader033/viewer/2022060317/62948d7a00630357855fa177/html5/thumbnails/21.jpg)
The Problem
21
renderer code uses OpenGL, frontend code uses WebGL
Copy to WebGL texture — not available in C++
How to efficiently pass the frame data to WASM
Frame data — copy to WASM’s memory
![Page 22: (New) Video Rendering on Frontend and Backend](https://reader033.fdocuments.in/reader033/viewer/2022060317/62948d7a00630357855fa177/html5/thumbnails/22.jpg)
Restating the Problem
22
Data in JavaScript, needs to be consumed in WASM
Data copy is expensive Abstraction layer separates between WASM and JavaScript
Can’t use the same objects on both sides
![Page 23: (New) Video Rendering on Frontend and Backend](https://reader033.fdocuments.in/reader033/viewer/2022060317/62948d7a00630357855fa177/html5/thumbnails/23.jpg)
Misusing Emscripten for fun & profit
23
Emscripten calls WebGL with OpenGL code
Emscripten gives OpenGL texture identifiers to WebGL textures
getNewId — a function that inserts to array, and returns index as identifier
TexturesFrame buffers
WebGL objects
![Page 24: (New) Video Rendering on Frontend and Backend](https://reader033.fdocuments.in/reader033/viewer/2022060317/62948d7a00630357855fa177/html5/thumbnails/24.jpg)
Solution
24
Create textures
Use texture identifier
Create texture identifier using Emscripten’s getNewId
Load frame data Explicitly remove texture identifier
![Page 25: (New) Video Rendering on Frontend and Backend](https://reader033.fdocuments.in/reader033/viewer/2022060317/62948d7a00630357855fa177/html5/thumbnails/25.jpg)
All’s well that ends well
25
We’ve reached 60 FPS We wrote the engine once, used it twice
The bridging code is minimal
![Page 26: (New) Video Rendering on Frontend and Backend](https://reader033.fdocuments.in/reader033/viewer/2022060317/62948d7a00630357855fa177/html5/thumbnails/26.jpg)
Is This Smart?
26
Relying on internal API which might change — Bad
Bridging abstraction gap — Necessary
Manually Managing texture lifetime isn’t great
Final score: 🤷🤷🤷🤷
![Page 27: (New) Video Rendering on Frontend and Backend](https://reader033.fdocuments.in/reader033/viewer/2022060317/62948d7a00630357855fa177/html5/thumbnails/27.jpg)
Bottom line
27
We can write C++ code and run it everywhere
Everything is more complicated when rendering :(
Emscripten bridges a lot of platform differences
Some hacking might still be required
![Page 28: (New) Video Rendering on Frontend and Backend](https://reader033.fdocuments.in/reader033/viewer/2022060317/62948d7a00630357855fa177/html5/thumbnails/28.jpg)
Links!
28
Check out the app
My talk about value interpolation
Lin Clark’s excellent intro to WebAssembly
My talk about the history of efficient computinting on the web
Contact me
![Page 29: (New) Video Rendering on Frontend and Backend](https://reader033.fdocuments.in/reader033/viewer/2022060317/62948d7a00630357855fa177/html5/thumbnails/29.jpg)
We are hiring!Lightricks.com/careers