Developing rich multimedia applications with Kurento: a tutorial for JavaScript Developers
-
Upload
luis-lopez -
Category
Technology
-
view
2.885 -
download
7
description
Transcript of Developing rich multimedia applications with Kurento: a tutorial for JavaScript Developers
www.kurento.org
Developing rich multimedia applications with Kurento
Developing rich multimedia applications with Kurento: a tutorial for JavaScript developers
[email protected]://www.kurento.org
http://www.nubomedia.eu
http://www.fi-ware.org
http://ec.europa.eu
www.kurento.org 2
WebRTC technologies from the perspective of a developer
Developing the client side
Developing the infrastructure
side
Before WebRTCFirst wave of
WebRTC technologiesSecond wave of
WebRTC technologies
Begin End Begin End
• APIs• Standards• FOSS
• Coherent APIs• Standards• FOSS• Multiplatform
Begin End• APIs• Standards• FOSS• Platforms• Out of the box
capabilities
2012 2014
www.kurento.org 3
What’s a WebRTC infrastructurePeer-to-Peer WebRTC Application (without media infrastructure)
WebRTC video stream
WebRTC Application with media infrastructure
Media infrastructure
www.kurento.org 4
Some WebRTC media infrastructure providers
www.kurento.org
Media infrastructure common capabilities
5
Transcoding media infrastructure
VP8 H.264
MCU media infrastructures
Recording media infrastructure
www.kurento.org
Can’t we go beyond?Media is
here Media gotthere
Media gotthere
AnalyzeTransform
Store
Transcoding
MCURecording
EnrichAugment
Adapt
SensorsContext
EventsMedia is
here
What WebRTC infrastructures are• Transcoding• MCU• Recording
What WebRTC infrastructures should be• Flexible processing• Augmented reality• Blending• Mixing• Analyzing• Etc.
6
www.kurento.org
• Interoperable media exchange (multiplatform/multiprotocol)• WebRTC, RTP, HTTP (video tag), etc.
• Standard capabilities Capabilities• Transcoding, MCU, recording
• Advanced capabilities• Computer vision, augmented reality, mixing, blending, etc.
Multimedia infrastructure
• REST API• JavaScript API• Java API
APIs
• LGPL 2.1
Is distributed through a flexible FOSS license
What’s Kurento
7
www.kurento.org
Future Internet
Multimedia Infrastructure
Simple Development
APIs
Kurento: the equation
8
www.kurento.org
Kurento Media Server (KMS): the nucleus of Kurento
• KMS is a middleware for media streams– Receives the stream– Process the stream– Issues the stream
SendReceiveAnalyze
AugmentEnrich
TransformTranscode
RecordProcess
Replicate
MediaSource
MediaSink
KMS
9
www.kurento.org
The Media API: The API for accessing KMS capabilities
SendReceiveAnalyze
AugmentEnrich
TransformTranscode
RecordProcess
Replicate
MediaSource
MediaSink
KMS
JavaMedia API
JavaScriptMedia API
RESTAPI
Applications define the processingof streams getting through KMS
10
www.kurento.org
Media API: Media Elements and Media Pipelines
Media Element
• Provides a specific media functionality
› Send/receive media
› Process media
› Transform media
• Exchange media through
› Sources
› Sinks
Media pipeline
• Chain of media elements implementing the desired media logic.
• The Media API provides the capability of creating media pipelines by joining media elements of the toolbox
Media Element
Sink
SRC
11
www.kurento.org
The Lego Game: Developers create applications connecting media elements
Application 2
Application 3
Application 1
Toolbox of media elements
12
www.kurento.org
Media API: trivial example (JavaScript)kwsMedia.create("MediaPipeline", function(error, pipeline){
pipeline.create("HttpGetEndpoint", function(error, httpGetEndpoint){pipeline.create("PlayerEndpoint", {uri : file_uri}, function(error, playerEndpoint){
playerEndpoint.connect(httpGetEndpoint, function(error){httpGetEndpoint.getUrl(function(error, url){videoInput.src = url;});playerEndpoint.play(function(error){});
});});
});});
Media Pipeline (inside KMS)
HttpGetEndpoint
Media fromfile or URI
HTTP mediastreaming Si
nk
SRC
PlayerEndpoint
13
www.kurento.org
Kurento Architecture: putting it all together
Kurento Media Server (KMS)
ReceiveVideo
AugmentedReality
SendVideo
Computer Vision
Video Playingand Recording
Kurento WS JSON-RPC API
Med
ia
Med
ia
Kurento Media Connector (KMC)
14
SIP Servlet
SOAP
HTTPServlet REST
Browser
Developer code with specific application
logic (pipelines)
Media capabilities of the browser
Kurento Media API
Specificlogic
www.kurento.org
Application execution flowClientCode
Media Connector(KMC)
MediaServer (KMS)
Crate pipeline/element…(JSON-RPC)
Commands requestingthe creation of a pipeline
Reference to media object …(JSON-RPC)
Pipelinecereation
phase
Media exchange
phase
1
2
Specific logic(e.g. AAA)
Mediapipelinecreation
Media exchange between client and server
15
www.kurento.org 16
What about Node.js?
Kurento Media Server (KMS)
ReceiveVideo
AugmentedReality
SendVideo
Computer Vision
Video Playingand Recording
Kurento WS JSON-RPC API
Med
ia
Med
ia
Kurento Media Connector (KMC)
SIP Servlet
SOAP
HTTPServlet REST
Node.js Pipeline creation +
AAA
Kurento Media API
BrowserBrowser-side code
www.kurento.org
Let’s develop with Kurento• What you need
– A Kurento instance• You can install your own Kurento instance
– http://www.kurento.org/docs/current/
• You can launch a Kurento instance at the FI-LAB (FI-WARE project)– http://lab.fi-ware.org – FI-LAB is a cloud infrastructure making possible to execute instances from images or blueprints
• kws-media-api – https://github.com/kurento/kws-media-api– http://github.com/kurento/kws-utils
» Node.js• npm install kws-media-api• npm install kws-utils
» Browser• bower install kws-media-api• bower install kws-utils
• Getting help– Kurento web site
• http://www.kurento.org
– Kurento mailing list• https://groups.google.com/forum/#!forum/kurento
– Twitter• @kurentoms
17
www.kurento.org
Kurento Hello World: Playing a file with an HTML5 client
Media Pipeline
HttpGetEndpoint
Media fromfile or URI
HTTP mediastreaming Si
nk
SRC
PlayerEndpoint
Kurento WS JSON-RPC API
Create / excec …
Ref / answer …
18
Kurento Media Connector (KMC)
www.kurento.org
Playing a file with an HTML5 client: JavaScript code
const ws_uri = “ws://your.kurento.server:8888/thrift/ws/websocket”; //change thisconst file_uri = “http://uri.to.video.file/”; //change this
var videoInput = document.getElementById("videoInput"); //video tag element
KwsMedia(ws_uri, function(kwsMedia){ //instantiate kws-media-api managerkwsMedia.create("MediaPipeline", function(error, pipeline){ //instantiate pipeline
pipeline.create("HttpGetEndpoint", function(error, httpGetEndpoint){ pipeline.create("PlayerEndpoint", {uri : file_uri}, function(error, playerEndpoint){
playerEndpoint.connect(httpGetEndpoint, function(error){httpGetEndpoint.getUrl(function(error, url){
videoInput.src = url;});
playerEndpoint.play();
});
});});
});});
Source: https://github.com/Kurento/kws-tutorial/tree/develop/VideoTagPlayer
19
www.kurento.org
Playing a file with an HTML5 client: HTML code
<!DOCTYPE html><html><head><meta charset="US-ASCII"><title>Insert title here</title>
<script type="text/javascript" src="../api/kws-media-api.js"></script><script type="text/javascript" src="demo.js"></script>
</head><body>
<button id="playButton">Click to start playing</button><br><video id="videoInput" autoplay></video>
</body></html>
Source: https://github.com/Kurento/kws-tutorial/tree/develop/VideoTagPlayer
20
www.kurento.org
Playing a file with an HTML5 client: See the example working
https://www.youtube.com/watch?v=Mp6UEL9U-v4
21
www.kurento.org
WebRTC loopback
Kurento Media Server
WebRTCStreaming
Sink
SRC
22
WebRtcEndpoint
www.kurento.org
WebRTC loopback: JavaScript code
23
Source: https://github.com/Kurento/kws-tutorial/tree/develop/WebRtcLoopback
const ws_uri = “ws://your.kurento.server:8888/thrift/ws/websocket”; //change this
var videoInput = document.getElementById("videoInput");var videoOutput = document.getElementById("videoOutput");
var webRtcPeer = kwsUtils.WebRtcPeer.startSendRecv(videoInput, videoOutput, onOffer, onError);
function onOffer(offer){KwsMedia(ws_uri, function(kwsMedia){
kwsMedia.create("MediaPipeline", function(error, pipeline){
pipeline.create("WebRtcEndpoint", function(error, webRtc){
webRtc.processOffer(offer, function(error, answer){
webRtcPeer.processSdpAnswer(answer);
});
webRtc.connect(webRtc);
});});
});};
23
www.kurento.org
WebRTC loopback: HTML code
24
<!DOCTYPE html><html><head><meta charset="US-ASCII"><title>Insert title here</title>
<script type="text/javascript" src="../api/adapter.js"></script><script type="text/javascript" src="../api/kws-media-api.js"></script><script type="text/javascript" src="../api/kws-utils.js"></script><script type="text/javascript" src="demo.js"></script>
</head><body>
<button id="launchButton">Click me ...</button><br><div style="float: left; border: 2px solid black; margin: 5px;">
<h1>Local stream</h1><video id="videoInput" autoplay></video>
</div><div style="float:left; border: 2px solid black; margin: 5px;">
<h1>Remote stream</h1><video id="videoOutput" autoplay></video>
</div></body></html>
Source: https://github.com/Kurento/kws-tutorial/tree/develop/WebRtcLoopback
24
www.kurento.org
WebRTC loopback: see the example working
25
https://www.youtube.com/watch?v=hUChVutQ2og
25
www.kurento.org
WebRTC with FaceOverlay
Media Pipeline
WebRTCStreaming
Sink
SRC
26
Sink
SRC
WebRtcEndpoint
FaceOverlayFilter
www.kurento.org
WebRTC with FaceOverlay: JavaScript code
27
Source: https://github.com/Kurento/kws-tutorial/tree/develop/FaceOverlay
var webRtcPeer = kwsUtils.WebRtcPeer.startSendRecv(videoInput, videoOutput , onOffer, onError);
function onOffer(offer) { KwsMedia(ws_uri, function(kwsMedia) { kwsMedia.create("MediaPipeline", function(error, pipeline) { pipeline.create("WebRtcEndpoint", function(error, webRtc) { pipeline.create("FaceOverlayFilter”, function(error, filter) {
//hat_uri is the uri pointing to the overlay image to draw on top of detected faces //These percent variables need to be defined to adjust overlay image to face box
filter.setOverlayedImage(hat_uri, offsetXPercent, offsetYPercent, widthPercent, heightPercent); webRtc.connect(filter, function(error) { filter.connect(webRtc, function(error) { webRtc.processOffer(offer, function(error, answer) {
webRtcPeer.processSdpAnswer(answer);});
…//many }); omited here for simplicity});
27
www.kurento.org
WebRTC recorder
2828
Media Pipeline
WebRTCStreaming
Sink
SRC
Sink
Mediato
file or URI
Media Pipeline
WebRTCStreaming
Sink
SRC
Mediafrom
file or URI
SRC
WebRtcEndpoint
WebRtcEndpoint
RecorderEndpoint
PlayerEndpoint
www.kurento.org
WebRTC Recorder: JavaScript codewebRtcPeer = kwsUtils.WebRtcPeer.startSendRecv(videoInput, videoOutput, onOffer, onError);
function onOffer(offer) { KwsMedia(ws_uri, function(kwsMedia) {
kwsMedia.create('MediaPipeline', function(error, pipeline) { pipeline.create('RecorderEndpoint', {uri : file_uri}, function(error, recorder) {pipeline.create('WebRtcEndpoint', function(error, webRtv) {
webRtc.connect(recorder, function(error) {recorder.record();webRtc.connect(webRtc, function(error) { webRtc.processOffer(offer, function(error, answer) { webRtcPeer.processSdpAnswer(answer); });
... }); //many }); omited here for simplicity}
29 29
Source: https://github.com/Kurento/kws-tutorial/tree/develop/Recorder
www.kurento.org
WebRTC with Recorder: see the example working
30
https://www.youtube.com/watch?v=wp5A4NZ-kd4
30
www.kurento.org
But … is this code beautiful?
3131
Developers love spaghetti for eating, but not for code
From that code structureto this code structure
www.kurento.org
Avoiding the callback hell• The hidden secret of the content API
– Every call to a kws-media-api returns a promise• var p1= kwsMedia.create(“MediaPipeline”);• p1.then(function(pipeline){…})
– Promises• A proxy to a value not knows when the proxy is created• Allows associating handlers to
– Value successfully obtained– Error
• https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise
• http://www.html5rocks.com/en/tutorials/es6/promises/
– Example
32 32
www.kurento.org
Are promises really useful for our objectives?
• Developing with promises– WebRTC loopack with FaceOverlay
• Source– https://github.com/Kurento/kws-media-api/tree/develop/example/WebRtcEndpoint-
FaceOverlayFilter_promises_2
• Does structure improve?– Not much– Problems
• Nesting avoiding, but chaining isn’t much beautiful• Nesting eliminated => closures eliminated
– Scope of variables (pipelines, elements, etc.) lost.– Need to pass variables along on each call
• What’s the point of using promises then?– Continue reading if you are the kind of guy of like living on the edge …
33 33
www.kurento.org 34
Generators+co: the callback hell killer
• Generators– Cooperative (non preemtive) simulation of synchronous
behavior on iterator functions by maintaining execution state• https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/
Iterators_and_Generators
– Allow multiple entry points for suspending and resuming execution at certain locations.
– Available on EC6 (Harmony)• Co
– Combines promises and generators– Uses promises to “connect” generator exit and reentrance– https://github.com/visionmedia/co
www.kurento.org 35
Revisiting the WebRTC with loopback example
var webRtcPeer = kwsUtils.WebRtcPeer.startSendRecv(videoInput, videoOutput, onOffer, onError);
function onOffer(offer){co(function*(){
var kwsMedia = yield KwsMedia(ws_uri);var pipeline = yield kwsMedia.create("MediaPipeline");
var webRtc = yield pipeline.create("WebRtcEndpoint");var filter = yield pipeline.create("FaceOverlayFilter");
//you need to define the hat_uri and percents variablesyield filter.setOverlayedImage(hat_uri, offsetXPercent, offsetYPercent, widthPercent, heightPercent);
var answer = yield webRtc.processOffer(offer);webRtcPeer.processSdpAnswer(answer);
yield webRtc.connect(filter);yield filter.connect(webRtc);
})();}
Source code of previous examples using generators:https://github.com/Kurento/kws-tutorial/tree/develop/WebRtcLoopback https://github.com/Kurento/kws-tutorial/tree/develop/FaceOverlayGeneratorhttps://github.com/Kurento/kws-tutorial/tree/develop/RecorderGenerator
www.kurento.org 36
Isn’t it beautiful ?
www.kurento.org
Let’s do some more complex stuff: a WebRTC game
Media Pipeline
Sink
SRC
Sink
SRC
Sink
SRC
37
Sink
SRC
Sink
WebRtcEndpoint
MirrorFilter PointerDetectorFilter
FaceOverlayFilter
RecorderEndpoint
www.kurento.org
WebRTC game: code
38
Source: https://github.com/Kurento/kws-tutorial/tree/develop/RecorderFilterGenerator
38
www.kurento.org
WebRTC game: see the example working
39
https://www.youtube.com/watch?v=5eJRnwKxgbY
39
www.kurento.org 40
Summary and conclusions• Kurento
– Open Source Softare• LGPL v2.1
– Multimedia infrastructure for HTML5• WebRTC• HTML5
– Advanced media processing capabilities• Recording, MCUs, Transcoding, Computer Vision, Augmented Reality, Mixing, Blending.
• kws-media-api– Access Kurento capabilities from JavaScript
• Media elements and media pipelines• Like a “Lego game”
– Beautiful code• Promises• Generators• CO
www.kurento.org
Thank you
41
Suggestions, comments and complains:[email protected]
http://www.kurento.orghttp://www.github.com/kurentoTwitter: @Kurentoms
http://www.nubomedia.eu
http://www.fi-ware.org
http://ec.europa.eu