FOSDEM 2016 - Creating rich WebRTC Applications with Kurento
Kurento Real Time Media Stream Processing · •WebRTC, RTP, HTTP (video tag), etc. •Standard...
Transcript of Kurento Real Time Media Stream Processing · •WebRTC, RTP, HTTP (video tag), etc. •Standard...
![Page 1: Kurento Real Time Media Stream Processing · •WebRTC, RTP, HTTP (video tag), etc. •Standard capabilities •Transcoding, MCU, recording •Advanced capabilities •Computer vision,](https://reader030.fdocuments.in/reader030/viewer/2022041021/5ed11763bd38ed35b92cd538/html5/thumbnails/1.jpg)
Kurento Real Time Media Stream
Processing
Juan Ángel Fuentes
Software Developer. Stream Oriented GE
![Page 2: Kurento Real Time Media Stream Processing · •WebRTC, RTP, HTTP (video tag), etc. •Standard capabilities •Transcoding, MCU, recording •Advanced capabilities •Computer vision,](https://reader030.fdocuments.in/reader030/viewer/2022041021/5ed11763bd38ed35b92cd538/html5/thumbnails/2.jpg)
Introducing multimedia infrastructures
1
Peer-to-Peer Application (without media infrastructure)
WebRTC video stream
Application with media infrastructure
Media infrastructure
![Page 3: Kurento Real Time Media Stream Processing · •WebRTC, RTP, HTTP (video tag), etc. •Standard capabilities •Transcoding, MCU, recording •Advanced capabilities •Computer vision,](https://reader030.fdocuments.in/reader030/viewer/2022041021/5ed11763bd38ed35b92cd538/html5/thumbnails/3.jpg)
Multimedia infrastructures for the
Future Internet
2
Media is
hereMedia got
there
Media got
there
Analyse
Transform
Store
Transcoding
MCU
Recording
Enrich
Augment
Adapt
Sensors
Context
Events
Media is
here
![Page 4: Kurento Real Time Media Stream Processing · •WebRTC, RTP, HTTP (video tag), etc. •Standard capabilities •Transcoding, MCU, recording •Advanced capabilities •Computer vision,](https://reader030.fdocuments.in/reader030/viewer/2022041021/5ed11763bd38ed35b92cd538/html5/thumbnails/4.jpg)
Enrich, augment, adapt, analyse, transform,
store: what’s the problem?
3
Complexity
![Page 5: Kurento Real Time Media Stream Processing · •WebRTC, RTP, HTTP (video tag), etc. •Standard capabilities •Transcoding, MCU, recording •Advanced capabilities •Computer vision,](https://reader030.fdocuments.in/reader030/viewer/2022041021/5ed11763bd38ed35b92cd538/html5/thumbnails/5.jpg)
The Stream Oriented Generic Enabler –
SO GE
4
• Interoperable media exchange (multiplatform/multiprotocol)
• WebRTC, RTP, HTTP (video tag), etc.
• Standard capabilities
• Transcoding, MCU, recording
• Advanced capabilities
• Computer vision, augmented reality, mixing, blending, etc.
Multimedia infrastructure
• REST API
• JavaScript API
• Java API
APIs
• LGPL 2.1
Is distributed through a flexible FOSS license
![Page 6: Kurento Real Time Media Stream Processing · •WebRTC, RTP, HTTP (video tag), etc. •Standard capabilities •Transcoding, MCU, recording •Advanced capabilities •Computer vision,](https://reader030.fdocuments.in/reader030/viewer/2022041021/5ed11763bd38ed35b92cd538/html5/thumbnails/6.jpg)
Kurento: The equation
5
Future Internet
Multimedia Infrastructur
e
Simple Developmen
t APIs
5
![Page 7: Kurento Real Time Media Stream Processing · •WebRTC, RTP, HTTP (video tag), etc. •Standard capabilities •Transcoding, MCU, recording •Advanced capabilities •Computer vision,](https://reader030.fdocuments.in/reader030/viewer/2022041021/5ed11763bd38ed35b92cd538/html5/thumbnails/7.jpg)
General Architecture
6
![Page 8: Kurento Real Time Media Stream Processing · •WebRTC, RTP, HTTP (video tag), etc. •Standard capabilities •Transcoding, MCU, recording •Advanced capabilities •Computer vision,](https://reader030.fdocuments.in/reader030/viewer/2022041021/5ed11763bd38ed35b92cd538/html5/thumbnails/8.jpg)
77
![Page 9: Kurento Real Time Media Stream Processing · •WebRTC, RTP, HTTP (video tag), etc. •Standard capabilities •Transcoding, MCU, recording •Advanced capabilities •Computer vision,](https://reader030.fdocuments.in/reader030/viewer/2022041021/5ed11763bd38ed35b92cd538/html5/thumbnails/9.jpg)
Key Concepts: Media Elements
Functional unit performing a specific action on a media stream
Developers abstract from the low level implementation
Able to receive media form other elements
Able to send media to other elements
8
![Page 10: Kurento Real Time Media Stream Processing · •WebRTC, RTP, HTTP (video tag), etc. •Standard capabilities •Transcoding, MCU, recording •Advanced capabilities •Computer vision,](https://reader030.fdocuments.in/reader030/viewer/2022041021/5ed11763bd38ed35b92cd538/html5/thumbnails/10.jpg)
Key Concepts: Types of Media Elements
Input Endpoints: injects media
Filters: Transform or analyze media
Hubs: Manage multiple media flows
Output Endpoints: take the media stream out
9
![Page 11: Kurento Real Time Media Stream Processing · •WebRTC, RTP, HTTP (video tag), etc. •Standard capabilities •Transcoding, MCU, recording •Advanced capabilities •Computer vision,](https://reader030.fdocuments.in/reader030/viewer/2022041021/5ed11763bd38ed35b92cd538/html5/thumbnails/11.jpg)
Key Concepts: Media Pipeline
Media Pipeline: is a chain of media elements, where the output
stream generated by one element (source) is fed into one or more
other elements input streams (sinks)
10
![Page 12: Kurento Real Time Media Stream Processing · •WebRTC, RTP, HTTP (video tag), etc. •Standard capabilities •Transcoding, MCU, recording •Advanced capabilities •Computer vision,](https://reader030.fdocuments.in/reader030/viewer/2022041021/5ed11763bd38ed35b92cd538/html5/thumbnails/12.jpg)
Key concepts: media elements and
pipelines
11
Media Element
• Provides a specific media functionality
› Send/receive media. These are the Endpoints
› Process media
› Transform media
• Ready to be used
• New media elements can be added
Media pipeline
• Chain of media elements implementing the
desired media logic
• The Media Server provides the capability
of creating media pipelines by joining
media elements of the toolbox
Media Element
Sin
k
SR
C
Media Pipeline
![Page 13: Kurento Real Time Media Stream Processing · •WebRTC, RTP, HTTP (video tag), etc. •Standard capabilities •Transcoding, MCU, recording •Advanced capabilities •Computer vision,](https://reader030.fdocuments.in/reader030/viewer/2022041021/5ed11763bd38ed35b92cd538/html5/thumbnails/13.jpg)
1212
![Page 14: Kurento Real Time Media Stream Processing · •WebRTC, RTP, HTTP (video tag), etc. •Standard capabilities •Transcoding, MCU, recording •Advanced capabilities •Computer vision,](https://reader030.fdocuments.in/reader030/viewer/2022041021/5ed11763bd38ed35b92cd538/html5/thumbnails/14.jpg)
Agnostic Media adapter
In charge of making possible that Stream Oriented GE APIS allow
developers to combine media elements to create the desired
pipeline
Fully abstracts all the complexities of media codecs and formats
13
![Page 15: Kurento Real Time Media Stream Processing · •WebRTC, RTP, HTTP (video tag), etc. •Standard capabilities •Transcoding, MCU, recording •Advanced capabilities •Computer vision,](https://reader030.fdocuments.in/reader030/viewer/2022041021/5ed11763bd38ed35b92cd538/html5/thumbnails/15.jpg)
Developers create applications just connecting
elementsApplication 2
Application 3
Application 1
Toolbox of media elements
14
![Page 16: Kurento Real Time Media Stream Processing · •WebRTC, RTP, HTTP (video tag), etc. •Standard capabilities •Transcoding, MCU, recording •Advanced capabilities •Computer vision,](https://reader030.fdocuments.in/reader030/viewer/2022041021/5ed11763bd38ed35b92cd538/html5/thumbnails/16.jpg)
Media Server (KMS): The nucleus of
Kurento
15
•KMS is a middleware for media streams:
•Receives the stream
•Process the stream
• Issues the stream
Send
Receive
Analyze
Augment
Enrich
Transform
Transcode
Record
Process
Replicate
Media
SourceMedia
Sink
KMS
![Page 17: Kurento Real Time Media Stream Processing · •WebRTC, RTP, HTTP (video tag), etc. •Standard capabilities •Transcoding, MCU, recording •Advanced capabilities •Computer vision,](https://reader030.fdocuments.in/reader030/viewer/2022041021/5ed11763bd38ed35b92cd538/html5/thumbnails/17.jpg)
Media plane and Signaling plane
16
![Page 18: Kurento Real Time Media Stream Processing · •WebRTC, RTP, HTTP (video tag), etc. •Standard capabilities •Transcoding, MCU, recording •Advanced capabilities •Computer vision,](https://reader030.fdocuments.in/reader030/viewer/2022041021/5ed11763bd38ed35b92cd538/html5/thumbnails/18.jpg)
How to control the media server?
17
•REST API
•JSON RPC
•Websocket interface in KMS
![Page 19: Kurento Real Time Media Stream Processing · •WebRTC, RTP, HTTP (video tag), etc. •Standard capabilities •Transcoding, MCU, recording •Advanced capabilities •Computer vision,](https://reader030.fdocuments.in/reader030/viewer/2022041021/5ed11763bd38ed35b92cd538/html5/thumbnails/19.jpg)
API Implementations
18
For now…
![Page 20: Kurento Real Time Media Stream Processing · •WebRTC, RTP, HTTP (video tag), etc. •Standard capabilities •Transcoding, MCU, recording •Advanced capabilities •Computer vision,](https://reader030.fdocuments.in/reader030/viewer/2022041021/5ed11763bd38ed35b92cd538/html5/thumbnails/20.jpg)
Multimedia Application Architecture in SO GE
Web application three layers parallelism:
Presentation layer: Multimedia presentation and capturing
Application logic: Multimedia logic in charge of building the pipeline
Service Layer: Kurento Media Server
19
![Page 21: Kurento Real Time Media Stream Processing · •WebRTC, RTP, HTTP (video tag), etc. •Standard capabilities •Transcoding, MCU, recording •Advanced capabilities •Computer vision,](https://reader030.fdocuments.in/reader030/viewer/2022041021/5ed11763bd38ed35b92cd538/html5/thumbnails/21.jpg)
20
Kurento Protocol
JSON-RPC over WS
Media Traffic
Application code
Media Traffic
Client Kurento
Media ServerApplication
Server
Java Kurento
Client
JavaScript
Kurento
Client
Application codeApplication code
Kurento
Protocol
JSON-RPC
over WS
Signaling
protocol
Kurento
Media Server
Client
Kurento Applications
![Page 22: Kurento Real Time Media Stream Processing · •WebRTC, RTP, HTTP (video tag), etc. •Standard capabilities •Transcoding, MCU, recording •Advanced capabilities •Computer vision,](https://reader030.fdocuments.in/reader030/viewer/2022041021/5ed11763bd38ed35b92cd538/html5/thumbnails/22.jpg)
Application execution flow
21
Client
Code
Kurento API Media
Server (KMS)
I want this media in this type
Commands requesting
the creation of a pipeline
(JSON-RPC request)
What you want is here
Pipeline
creation
phase
Media
exchange
phase
1
2
Specific logic
(e.g. AAA)
Media
pipeline
creation
Media exchange between client and server
![Page 23: Kurento Real Time Media Stream Processing · •WebRTC, RTP, HTTP (video tag), etc. •Standard capabilities •Transcoding, MCU, recording •Advanced capabilities •Computer vision,](https://reader030.fdocuments.in/reader030/viewer/2022041021/5ed11763bd38ed35b92cd538/html5/thumbnails/23.jpg)
Campus Party Brazil
22
![Page 24: Kurento Real Time Media Stream Processing · •WebRTC, RTP, HTTP (video tag), etc. •Standard capabilities •Transcoding, MCU, recording •Advanced capabilities •Computer vision,](https://reader030.fdocuments.in/reader030/viewer/2022041021/5ed11763bd38ed35b92cd538/html5/thumbnails/24.jpg)
23
Campus Party Brazil
![Page 25: Kurento Real Time Media Stream Processing · •WebRTC, RTP, HTTP (video tag), etc. •Standard capabilities •Transcoding, MCU, recording •Advanced capabilities •Computer vision,](https://reader030.fdocuments.in/reader030/viewer/2022041021/5ed11763bd38ed35b92cd538/html5/thumbnails/25.jpg)
24
![Page 26: Kurento Real Time Media Stream Processing · •WebRTC, RTP, HTTP (video tag), etc. •Standard capabilities •Transcoding, MCU, recording •Advanced capabilities •Computer vision,](https://reader030.fdocuments.in/reader030/viewer/2022041021/5ed11763bd38ed35b92cd538/html5/thumbnails/26.jpg)
Let’s get started!
25
Tutorials
http://doc-kurento.readthedocs.io/en/stable/tutorials.html
![Page 27: Kurento Real Time Media Stream Processing · •WebRTC, RTP, HTTP (video tag), etc. •Standard capabilities •Transcoding, MCU, recording •Advanced capabilities •Computer vision,](https://reader030.fdocuments.in/reader030/viewer/2022041021/5ed11763bd38ed35b92cd538/html5/thumbnails/27.jpg)
What do we need?
26
An instance of the Kurento Media Server
•FIWARE-LAB
•Local installation
Maven
NPM
Bower
Your favorite IDE
![Page 28: Kurento Real Time Media Stream Processing · •WebRTC, RTP, HTTP (video tag), etc. •Standard capabilities •Transcoding, MCU, recording •Advanced capabilities •Computer vision,](https://reader030.fdocuments.in/reader030/viewer/2022041021/5ed11763bd38ed35b92cd538/html5/thumbnails/28.jpg)
27
Tutorial 1 - Hello world
![Page 29: Kurento Real Time Media Stream Processing · •WebRTC, RTP, HTTP (video tag), etc. •Standard capabilities •Transcoding, MCU, recording •Advanced capabilities •Computer vision,](https://reader030.fdocuments.in/reader030/viewer/2022041021/5ed11763bd38ed35b92cd538/html5/thumbnails/29.jpg)
28
Tutorial 1 - Hello world
![Page 30: Kurento Real Time Media Stream Processing · •WebRTC, RTP, HTTP (video tag), etc. •Standard capabilities •Transcoding, MCU, recording •Advanced capabilities •Computer vision,](https://reader030.fdocuments.in/reader030/viewer/2022041021/5ed11763bd38ed35b92cd538/html5/thumbnails/30.jpg)
Tutorial 1 - Hello world, server
29
![Page 31: Kurento Real Time Media Stream Processing · •WebRTC, RTP, HTTP (video tag), etc. •Standard capabilities •Transcoding, MCU, recording •Advanced capabilities •Computer vision,](https://reader030.fdocuments.in/reader030/viewer/2022041021/5ed11763bd38ed35b92cd538/html5/thumbnails/31.jpg)
Tutorial 1 - Hello world, client
30
![Page 32: Kurento Real Time Media Stream Processing · •WebRTC, RTP, HTTP (video tag), etc. •Standard capabilities •Transcoding, MCU, recording •Advanced capabilities •Computer vision,](https://reader030.fdocuments.in/reader030/viewer/2022041021/5ed11763bd38ed35b92cd538/html5/thumbnails/32.jpg)
31
Tutorial 1 - Hello world (Node.js)
![Page 33: Kurento Real Time Media Stream Processing · •WebRTC, RTP, HTTP (video tag), etc. •Standard capabilities •Transcoding, MCU, recording •Advanced capabilities •Computer vision,](https://reader030.fdocuments.in/reader030/viewer/2022041021/5ed11763bd38ed35b92cd538/html5/thumbnails/33.jpg)
Tutorial 1 - Hello world (Node.js)
32
![Page 34: Kurento Real Time Media Stream Processing · •WebRTC, RTP, HTTP (video tag), etc. •Standard capabilities •Transcoding, MCU, recording •Advanced capabilities •Computer vision,](https://reader030.fdocuments.in/reader030/viewer/2022041021/5ed11763bd38ed35b92cd538/html5/thumbnails/34.jpg)
Tutorial 2 - WebRTC magic mirror
33
![Page 35: Kurento Real Time Media Stream Processing · •WebRTC, RTP, HTTP (video tag), etc. •Standard capabilities •Transcoding, MCU, recording •Advanced capabilities •Computer vision,](https://reader030.fdocuments.in/reader030/viewer/2022041021/5ed11763bd38ed35b92cd538/html5/thumbnails/35.jpg)
Tutorial 2 - WebRTC magic mirror
34
![Page 36: Kurento Real Time Media Stream Processing · •WebRTC, RTP, HTTP (video tag), etc. •Standard capabilities •Transcoding, MCU, recording •Advanced capabilities •Computer vision,](https://reader030.fdocuments.in/reader030/viewer/2022041021/5ed11763bd38ed35b92cd538/html5/thumbnails/36.jpg)
35
Tutorial 2 - WebRTC magic mirror
![Page 37: Kurento Real Time Media Stream Processing · •WebRTC, RTP, HTTP (video tag), etc. •Standard capabilities •Transcoding, MCU, recording •Advanced capabilities •Computer vision,](https://reader030.fdocuments.in/reader030/viewer/2022041021/5ed11763bd38ed35b92cd538/html5/thumbnails/37.jpg)
Tutorial 3 - WebRTC one-to-one video call with
recording and filtering
36
![Page 38: Kurento Real Time Media Stream Processing · •WebRTC, RTP, HTTP (video tag), etc. •Standard capabilities •Transcoding, MCU, recording •Advanced capabilities •Computer vision,](https://reader030.fdocuments.in/reader030/viewer/2022041021/5ed11763bd38ed35b92cd538/html5/thumbnails/38.jpg)
Tutorial 3 - WebRTC one-to-one video call with
recording and filtering
37
![Page 39: Kurento Real Time Media Stream Processing · •WebRTC, RTP, HTTP (video tag), etc. •Standard capabilities •Transcoding, MCU, recording •Advanced capabilities •Computer vision,](https://reader030.fdocuments.in/reader030/viewer/2022041021/5ed11763bd38ed35b92cd538/html5/thumbnails/39.jpg)
Tutorial 3 - WebRTC one-to-one video call with
recording and filtering
38
![Page 40: Kurento Real Time Media Stream Processing · •WebRTC, RTP, HTTP (video tag), etc. •Standard capabilities •Transcoding, MCU, recording •Advanced capabilities •Computer vision,](https://reader030.fdocuments.in/reader030/viewer/2022041021/5ed11763bd38ed35b92cd538/html5/thumbnails/40.jpg)
Kurento and the FIWARE-LAB
39
• FIWARE-LAB
– Working instance of FI-WARE enabling free experimentation with
technology
– http://lab.fiware.org
• Creating a Kurento instance from an image
– Use latest version of Kurento images.
• Creating a Kurento instance using recipes
– Use Ubuntu 14.04 LTS clear image
– Use latest version of Kurento recipes
![Page 41: Kurento Real Time Media Stream Processing · •WebRTC, RTP, HTTP (video tag), etc. •Standard capabilities •Transcoding, MCU, recording •Advanced capabilities •Computer vision,](https://reader030.fdocuments.in/reader030/viewer/2022041021/5ed11763bd38ed35b92cd538/html5/thumbnails/41.jpg)
Kurento and the FIWARE-LAB
40
![Page 42: Kurento Real Time Media Stream Processing · •WebRTC, RTP, HTTP (video tag), etc. •Standard capabilities •Transcoding, MCU, recording •Advanced capabilities •Computer vision,](https://reader030.fdocuments.in/reader030/viewer/2022041021/5ed11763bd38ed35b92cd538/html5/thumbnails/42.jpg)
To learn more…
41
![Page 43: Kurento Real Time Media Stream Processing · •WebRTC, RTP, HTTP (video tag), etc. •Standard capabilities •Transcoding, MCU, recording •Advanced capabilities •Computer vision,](https://reader030.fdocuments.in/reader030/viewer/2022041021/5ed11763bd38ed35b92cd538/html5/thumbnails/43.jpg)
Thank you!
http://fiware.org
Follow @FIWARE on Twitter