The future of multimedia communications and services: Kurento and it's role
-
Upload
luis-lopez -
Category
Internet
-
view
1.616 -
download
8
description
Transcript of The future of multimedia communications and services: Kurento and it's role
www.kurento.org
Presentation for the GSMA WebRTC interest group on April 7th, 2014
The future of multimedia communications and services:
Kurento and it’s role
Luis [email protected]
http://www.kurento.org
www.kurento.org 2
Who am I?
• Luis Lopez– Associate Professor at ETSIT/URJC
• http://www.urjc.es• http://www.etsit.urjc.es
– Director of the FUN-LAB research group
• Future Networks Laboratory• http://www.gsyc.es
– Leader of the Kurento.org initiative• http://www.kurento.org
– Software engineer, software developer and Open Source Software Advocate
www.kurento.org 3
What does Kurento mean?
www.kurento.org 4
Kurento’s History
2004•Kurento is born as a tool for transferring research results to industry
2012•Kurento goes open source LGPL v2.1. Professional services provided by Naevatec (startup)
June’13 •1M€ of investment for providing multimedia capabilities to FI-WARE
Dec13•Kurento Featured at WebRTC Expo (Paris)
Feb’14•3.3M€ of investment for making Kurento an elastic cloud platform (NUBOMEDIA)
April’14•Kurento featured at IMS World Conference (Barcelone)
July’14•Kurento featured at DevCon5 (New York)
Sep’14•500K€ of investment for maintenance of Kurento FI-WARE APIs
www.kurento.org 5
Kurento team• 18 people: 15 developers, 1 sysadmin, 2 coordination
www.kurento.org 6
Real-time communications: our visionN
umbe
r of u
sers
General purposeRTC multimedia Services
Specific purpose RTC multimedia services
Commodity, no business opportunities here
Who cares about the immense minority?Many business opportunities here
www.kurento.org 7
The opportunity: from services to platforms and APIs
www.kurento.org 8
Who shares this vision: The WebRTC movement
Num
ber o
f use
rs
General purposeRTC multimedia Services
Specific purpose RTC multimedia services
Commodity, no business opportunities here
Who cares about the immense minority?Many business opportunities here
Disruption directionof WebRTC
www.kurento.org 9
Before and after WebRTC: the perspective of the application developer
Developing the client side
Before WebRTC After WebRTC
Begin End
• Unified APIs• Standards• FOSS• Multiplatform
www.kurento.org 10
WWW TECHNOLOGY & STANDARDS
Why developers are important for the WebRTC movement?
The effect of WWW developers on different technological areas
BankingCRM TV
CommerceDocument
Skype?Traditional phone service?
www.kurento.org 11
Should telcos take the hint?
WebRTC
WWW development community RCS/IMS development community
WebRTC makes multimedia RTC to be just another feature of your WWW application
www.kurento.org 12
What’s the relationship of WebRTC with Kurento?
Developing the client side
Developing the infrastructure
side
Before WebRTC After WebRTC Next natural step…
Begin End Begin End
Begin End
• Unified APIs• Standards• FOSS• Multiplatform
• Unified APIs• Standards• FOSS• Multiplatform
• Unified APIs• Standards• FOSS• Multiplatform
www.kurento.org 13
But is the infrastructure important?What RTC applications will be demanded by the market on the next few years?
www.kurento.org 14
Kurento vision for Future RTC multimedia services
Media ishere Media got
there
Media gotthere
AnalyzeTransform
Store
Transport
EnrichAugment
Adapt
SensorsContext
EventsMedia is
here
Traditional infrastructures
Future Internet infrastructures
www.kurento.org 15
Enrich, analyze, adapt, augment: What’s the problem?
Complexity
www.kurento.org 16
Future Internet
Multimedia Infrastructure
Simple Development
APIs
The Kurento equation for Future Internet infrastructures
www.kurento.org 17
What’s Kurento?
• Kurento.org– Developer platform
• Infrastructures + APIs• Kurento is not a service
– Future Internet• Generalized multimedia
– Audio/Video/Sensor
• Advanced media processing– Computer Vision, Augmented reality
• Interoperability– WebRTC, RTP, HTTP, all codecs, etc.
– Open Source Software• LGPL v2.1
www.kurento.org 18
Kurento Media Server: The nucleus of Kurento
• KMS is a middleware for media streams– Receives the stream– Process the stream– Issues the stream
SendReceiveAnalyze
AugmentEnrich
TransformTranscode
RecordProcess
Replicate
MediaSource
MediaSink
KMS
www.kurento.org 19
Media API: The API for accessing KMS capabilities
SendReceiveAnalyze
AugmentEnrich
TransformTranscode
RecordProcess
Replicate
MediaSource
MediaSink
KMS
JavaMedia API
JavaScriptMedia API
Otherlanguages
Applications define the processingof streams getting through KMS
www.kurento.org 20
Media API: media elements and media pipelines
Media Element
• Provides a specific media functionality
› Send/receive media
› Process media
› Transform media
• Exchange media through
› Sources
› Sinks
Media pipeline
• Chain of media elements implementing the desired media logic.
• The Media API provides the capability of creating media pipelines by joining media elements of the toolbox
Media Element
Sink
SRC
www.kurento.org 21
The Lego Game: Developers create applications connecting media elements
Application 2
Application 3
Application 1
Toolbox of media elements
www.kurento.org 22
Media API trivial exampleMediaPipeline mp = contentSession.getMediaPipelineFactory().create();
PlayerEndpoint playerEndpoint = mp.newPlayerEndpoint(file:///myFile.webm).build();
HttpGetEndpoint httpEndpoint = mp.newHttpGetEndpoint().terminateOnEOS().build();
playerEndpoint.connect(httpEndpoint);
httpEndpoint.getUrl(); //URL where the media is made available
Media Pipeline
HttpGetEndpoint
Media fromfile or URI
HTTP mediastreaming Si
nk
SRC
PlayerEndpoint
www.kurento.org 23
The signaling: adapting the WWW model to the multimedia services
Process WWW request- DDBB access- Authentication- XML processing- Etc.
HTTP request: I want this resource
HTTP response:The resource
Process media request- Media API- DDBB access- Authentication- XML processing- Etc.
Signaling request: I want this media
Signaling response:The media is here
This is how you create WWW applications
(Servlets, ASP, PHP, Rails, etc.)
This is how you createKurento enabled
applications:Multimedia RTC is just
another feature of your application
Developerscreate this
www.kurento.org 24
The Content Handler: reacting to signaling events
KMS
Sink
SRC
Sink
SRC
Sink SRC
Sink
Media API
REST API (Open API protocol) The Content HandlerEquivalent to a Servlet/ASP/PHP scriptExposes APIs for specifying:- when receiving “this request”…- … execute “this logic”Developer can use the media API
Code building the media pipeline and executing the application logic the
developer wants
www.kurento.org 25
Kurento Application Server, the container of Handlers
• Is an extension of Java EE technologies.• Compatible with all Java EE Servlet
containers• Hold Handlers
– Your specific application code• Receives signaling requests:
– I want “this media” in this way …• Dispatches request to the appropriate
handler– @annotations based mapping
• Generate an answer showing “where and how to find the media”– URL where media is located– Negotiated SDP
Java EE compatible container
HTTP Servlet
SIP Servlet
Web service
s
Kurento REST API
Specific handler implementations
Media API DD.BB.
Kurento Application Server (KAS)
Other java APIs.
www.kurento.org 26
Putting it altogether: the Architecture
Kurento Media Server (KMS)
ReceiveVideo
AugmentedReality
SendVideo
Computer Vision
Video Playingand Recording
Java EE compatible container
HTTP Servlet
SIP Servlet
Web service
s
Kurento REST API
Specific handler implementations
Signaling and WWW traffic
Med
ia
Med
ia
Signaling and WWW traffic
Media API DD.BB.
Kurento Application Server (KAS)
Other java APIs.
www.kurento.org 27
Application execution flowClientCode
ApplicationServer (KAS)
MediaServer (KMS)
I want this media in this way …(JSON-RPC request)
Commands requestingthe creation of a pipeline
What you want is here …(JSON-RPC answer)
Media negotiation
phase
Media exchange
phase
1
2
Specific applicationlogic at the server-side
(Content Handler)
Mediapipelinecreation
Media exchange between client and server
www.kurento.org 28
Kurento in the context of IMSKMS = MRF
(Media Resource Function)
KAS = AS(Application Server)
S-CSCF(Serving- Call Session
Control Function
MediaControl
MediaEvents
Context andother information
services
UE(User Equipment)
Application logic & orchestration
Media capabilities (pipeline)
Media elements can generate events through deep analysis of media• IVR• Sentiment analysis• Face recognition• Object tracking• Etc.
Application logic can use context and other external information services for controlling media elements or dynamically modifying media pipelines
www.kurento.org 29
Kurento use case: WebRTC video conference with “secret” participant
• The “connection” primitive acts from source to destination– You can connect how many sinks as you to a source– Not-connected sources or sinks don’t break anything.
• In this example– User 1 and 2 maintain a video conference
• webRtcEndpoint1.connect(webRtcEndpoint2), webRtcEndpoint2.connect(webRtcEndpoint1)
– User 3 can “see” that video conference, but he cannot participate• webRtcEndpoint1.connect(webRtcEndpoint3), webRtcEndpoint2.connect(webRtcEndpoint3’)
Media Pipeline
WebRTC audiostreaming
Sink
SRC
WebRTC videostreaming
SRC
Sink
SRC
Sink
WebRTC audiostreaming
WebRtcEndpoint 1
WebRtcEndpoint 3
WebRtcEndpoint 2SR
CSi
nk
WebRTC audiostreaming
WebRtcEndpoint 3’
User 1 User 2
User 3
www.kurento.org 30
Kurento use case: WebRTC bridge/SBC between WebRTC and RTP domains
• Kurento transparently adapts codecs and formats. Just connect:– webRtcEndpoint.connect(rtpEndpoint)– rtpEndpoint.connect(webRtcEndpoint)
• Support for the most common audio and video codecs– VP8, H.264, H.263, OPUS, AMR, Speex, G.711, etc.
Media Pipeline
WebRTC audiostreaming
Sink
SRC
SRC
Sink
Media Pipeline
Sink
SRC
SRC
Sink
WebRTC videostreaming
RTP audiostreaming
RTP videostreaming
www.kurento.org 31
Sink
SRC
SRC Sink
SinkSRC
Kurento use case: interoperable MCU
SRC
Sink
• Group communications– Dispatcher
• Media router• N to M
– Mixer• Composite
mixed media• N to 1
www.kurento.org 32
Kurento use case: Video conference recorded and distributed (CDN)
• Kurento makes possible to interconnect different types of endpoints in your application
– RecorderEndpoint• Records media in the desired format
– HttpEndpoint• Exposes the media flow through HTTP pseudostreaming (HTML5 <video> tag)
Sink
SRC
Media Pipeline
Sink
SRC
SinkSi
nkSR
C
Sink
Mediato
file or repository
CDN or Web
browser
www.kurento.org 33
Kurento use case: video-conference with augmented reality
• The infrastructure exposes advances processing capabilities• You can connect them following the same scheme
– playerEndpoint.connect(jackVaderFilter)– jackVaderFilter.connect(httpEndpoint)
• Use RTC endpoints if you want real-time media processing– WebRtcEndpoint, RtpEndpoint
Media Pipeline
Mediafrom
file or URI
Sink
SRC
SRC
Sink
JackVaderFilter
Result: https://www.youtube.com/watch?v=yJAQs23eoXw
www.kurento.org 34
Kurento use case: WebRTC video game
Result: https://www.youtube.com/watch?v=PYCw9-4oWWA
Sink
SRC
Sink
Sink
SRC
Sink
SRC
Sink
SRC
Sink
SRC
WebRtcEndpoint
MirrorFilter PointerDetectorFilter
ChromaFilter FaceOverlayFilter
Media Pipeline
• Beyond video-conferencing: receiving events from the media stream– Immersive and personalized advertising– Games and entertainment
Events (buttons)
Game or advertisement logic
Change faceChange background
www.kurento.org 35
Media PipelineMedia Pipeline
IoT and context aware applications: hot IVR plugin
• Pipelines are dynamic, you can hot modify them using context or sensor information– Ex. when device is moving faster than 40 Kh, add IVR system
• “Record this call”, “Redirect to my secretary”, “Play file one”, Etc.• Voice commands can modify the pipeline themselves
– “Record this call” add a RecorderEndpoint– “Play file 1” add a player
– Ex. When security sensor fires, activate face-recognition/person-detection, etc
Sink
SRC
SRC
Sink
Sink
SRC
Sink
SRC
SRC
Sink
IvrFilter
VoiceEventsDynamic pipeline modification Depending on context or other events
www.kurento.org 36
Media Pipeline
Smart-cities applications• Advanced computer vision filters for
– Crowd detection– Car plate recognition– Intrusion detection– Traffic control– Etc. CrowdEvents
Sink
SRC
Sink
SRC
CrowdDetector
RtspPlayer
www.kurento.org 37
Other scenarios• There are opportunities on many verticals
– eHealth– Logistics– Security– Social networks– Jounalisms
• Creation of additional Media Elements is simple– Integrate your own IVRs– Integrate your own Computer Vision capabilities– Integrate your own repositories– Etc.
• Potential applications are unlimited– Just combine the media elements
www.kurento.org 38
Thank you
Suggestions, comments and complains:[email protected]
http://www.kurento.orghttp://www.github.com/kurentoTwitter: @Kurentoms