Introduction to WebRTC

WEBRTC IN JAVASCRIPT

REAL-TIME COMMUNICATION IN JAVASCRIPT

Introduction to WebRTC

1


IETF (RTCWEB SPECIFICATION)

WebRTC consists of a protocol specification and a Javascript API specification that, if all are

implemented, will allow communication using audio, video, and data sent along the most direct possible path between participants.

2


History and Overview

• Been around since 2011, still in draft status, but it’s used in production

• Google acquired two companies (On2… video codecs and GIPS… VoIP calling)

• Wrote a Javascript API for submission to W3C and called it WebRTC (“Web Real-time Communication”)

3


Why we chose WebRTC…

• The web is moving towards real-time functionality

• People want something native to their browser

• Video and audio calls on the web !== Flash or ActiveX

4

(AND WHY YOU SHOULD TOO)


1. Have media to send (webcam, microphone, etc.)

2. Figure out best way for peers to communicate

3. Determine your media’s format (SDP)

4. Send communication and media information

5. Repeat steps 1 - 4 for other peers

5







6


getUserMedia()

• Handles all transport of video and audio information from the hardware to the web

• Each MediaStream contains multiple “tracks”

7

navigator.getUserMedia()


getUserMedia()

var constraints = { video: true };

function successCallback(stream) { var video = document.querySelector('video'); video.src = window.URL.createObjectURL(stream);}

function errorCallback(error) { console.log('navigator.getUserMedia error: ', error);}

navigator.getUserMedia(constraints, successCallback, errorCallback);

8


getUserMedia() constraints• Intended to control the media stream via a

configuration object

• Controls media type, resolution, frame rate, etc.

• Able to specify specs as either “mandatory” or “optional”

• Constraints only affect local view of your stream

9


getUserMedia() audio analysis• Only works for Chrome local audio streams, you can’t

currently analyze remote audio streams

10

// Success callback when requesting audio input streamfunction gotStream(stream) { var audioContext = new webkitAudioContext();

// Create an AudioNode from the stream var mediaStreamSource = audioContext.createMediaStreamSource(stream);

// Connect it to the destination or any other node for processing! mediaStreamSource.connect(audioContext.destination);}

navigator.webkitGetUserMedia({ audio: true }, gotStream);







11


TCP & UDP

• UDP (User Datagram Protocol)Unreliable… fast, and with a low overhead

• TCP (Transmission Control Protocol)Reliable… slow, and with a high overhead

• HTTP uses TCP for connections, vast majority of web services use TCP

12


All about dat UDP

• UDP is a better choice for streaming media…

• It’s fast

• You don’t usually need every single packet

• Caveat? You need to know where to send packets.

13


Expectations14


Reality15


… ICE BABY? TOO COLD?

But then there’s ICE

16


ICE, ICE Connections…

• ICE is a priority-based protocol for testing connections between two computers

• ICE calls these possible connection points “candidates”

• Computers send them to each other, test them out, and choose the best fit

17


STUN18


TURN19


ICE Connectionsvar pc;

pc = new RTCPeerConnection({ 'iceServers': [ { 'url': 'stun:stun.services.mozilla.com' }, { 'url': 'stun:stun.l.google.com:19302' } ]});

pc.onicecandidate = function(event) { if(event.candidate !== null) { mySocket.send(JSON.stringify({ 'ice': event.candidate })); }}

20


Gotchas• In addition to your signaling system server, you’ll also

need to run your own STUN/TURN server

• ICE checks for local network connections first, so it will almost always work over a local network

• This doesn’t mean it works over the Internet

• Particularly embarrassing when demoing for a client…

21

WEBRTC IN JAVASCRIPT 22

WAT.







23


SDP• Two peers on ICE communicate via a spec called SDP

(“Session Description Protocol”) which tell what each peer is interested in

• Do you want audio? Sure.

• Do you want video? Definitely.

• Breakfast in bed? Every Sunday.

24


SDP (continued)

• SDP also handles codec specifications

• Generally no need to modify an SDP string

• First peer sends an offer, second peer sends an answer

• Both ICE and SDP require a signaling server to set up the direct connection between peers

25


Let’s analyze an SDP string…type: offer, sdp: v=0 o=- 581856173060444966 2 IN IP4 127.0.0.1 s=- t=0 0 a=group:BUNDLE audio video a=msid-semantic: WMS 5BgnHND1rh5IsIJXBGN0EnzD0xoGWD8Hd901 m=audio 9 RTP/SAVPF 111 103 104 9 0 8 126 c=IN IP4 0.0.0.0 a=rtcp:9 IN IP4 0.0.0.0 a=ice-ufrag:JI6Xwt9wq/v5HuY7 a=ice-pwd:eNXo/5Sx1JtHCAciusSzmZzO a=fingerprint:sha-256 A0:DE:39:65:10:33:02:44:C8:14:57:62:28:89:D9:45:69:DA:B6:58:44:A8:D8:2A:BC:46:D3:E0:2B:D1:0D:E4 a=setup:actpass a=mid:audio a=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level a=extmap:3 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time a=sendrecv a=rtcp-mux a=rtpmap:111 opus/48000/2 a=fmtp:111 minptime=10; useinbandfec=1 a=rtpmap:103 ISAC/16000 a=rtpmap:104 ISAC/32000 a=rtpmap:9 G722/8000 a=rtpmap:0 PCMU/8000 a=rtpmap:8 PCMA/8000 a=rtpmap:126 telephone-event/8000 a=maxptime:60 a=ssrc:2786781455 cname:Raffm+GJz1SzeA5+ a=ssrc:2786781455 msid:5BgnHND1rh5IsIJXBGN0EnzD0xoGWD8Hd901 2c9da2ce-219e-4e89-9ad5-1eaada3d79c8 a=ssrc:2786781455 mslabel:5BgnHND1rh5IsIJXBGN0EnzD0xoGWD8Hd901 a=ssrc:2786781455 label:2c9da2ce-219e-4e89-9ad5-1eaada3d79c8 m=video 9 RTP/SAVPF 100 116 117 96

26

c=IN IP4 0.0.0.0 a=rtcp:9 IN IP4 0.0.0.0 a=ice-ufrag:JI6Xwt9wq/v5HuY7 a=ice-pwd:eNXo/5Sx1JtHCAciusSzmZzO a=fingerprint:sha-256 A0:DE:39:65:10:33:02:44:C8:14:57:62:28:89:D9:45:69:DA:B6:58:44:A8:D8:2A:BC:46:D3:E0:2B:D1:0D:E4 a=setup:actpass a=mid:video a=extmap:2 urn:ietf:params:rtp-hdrext:toffset a=extmap:3 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time a=extmap:4 urn:3gpp:video-orientation a=sendrecv a=rtcp-mux a=rtpmap:100 VP8/90000 a=rtcp-fb:100 ccm fir a=rtcp-fb:100 nack a=rtcp-fb:100 nack pli a=rtcp-fb:100 goog-remb a=rtpmap:116 red/90000 a=rtpmap:117 ulpfec/90000 a=rtpmap:96 rtx/90000 a=fmtp:96 apt=100 a=ssrc-group:FID 1802804178 3581727714 a=ssrc:1802804178 cname:Raffm+GJz1SzeA5+ a=ssrc:1802804178 msid:5BgnHND1rh5IsIJXBGN0EnzD0xoGWD8Hd901 168833a1-b0f1-439a-be29-5687f1d8072e a=ssrc:1802804178 mslabel:5BgnHND1rh5IsIJXBGN0EnzD0xoGWD8Hd901 a=ssrc:1802804178 label:168833a1-b0f1-439a-be29-5687f1d8072e a=ssrc:3581727714 cname:Raffm+GJz1SzeA5+ a=ssrc:3581727714 msid:5BgnHND1rh5IsIJXBGN0EnzD0xoGWD8Hd901 168833a1-b0f1-439a-be29-5687f1d8072e a=ssrc:3581727714 mslabel:5BgnHND1rh5IsIJXBGN0EnzD0xoGWD8Hd901 a=ssrc:3581727714 label:168833a1-b0f1-439a-be29-5687f1d8072e

WEBRTC IN JAVASCRIPT 27

WAT.


RTCPeerConnectionpc = new RTCPeerConnection(null);pc.addStream(localStream);pc.createOffer(sendingOffer);

function sendingOffer(desc) { pc.setLocalDescription(desc); sendAnswer(desc);}

// When answer received over signalingfunction gotAnswer(desc) { pc.setRemoteDescription(desc);}

28


Security on WebRTC29







30


HTTP://PASTEBIN.COM/RK9IZITX

Walkthrough

31

http://pastebin.com/RK9iziTX


Platform Support• Google Chrome (mostly)

• Chrome for Android (mostly)

• Firefox (kinda)

• Opera (not really)

• Safari & IE (srsly bro?)

• Native Java and Objective-C bindings (about the same as Chrome)

32


Nerdy Details• Video codec support: VP8, H264

• Audio codec support: G.711, iLBC, iSAC, OPUS

• Media transport protocol: RTP/RTCP

• Security protocol: SRTP

• NAT traversal: STUN/TURN/ICE

• Signaling: Peer-to-peer, but requires a signaling component (agnostic) to notify peers of status changes and new messages

33


adapter.js & Temasys

• adapter.js (maintained by Google) is a Javascript shim made to normalize browser differences

• Minimizes the effects of spec churn from subtleties

• Temasys also makes their own version of adapter.js which adds a Flash fallback for a poor man’s browsers

34


WebRTC Internals

• God’s gift to WebRTC developers

• Available at:chrome://webrtc-internals

• Or programmatically:myPeerConnection.getStats()

35


Resources• Getting Started with WebRTC

• Capturing Audio & Video in HTML5

• WebRTC in the Real World

• Google I/O Presentation Video

• HTML5 WebRTC Explained Video

• WebRTC @ MDN

• WebRTC Book

36

http://www.html5rocks.com/en/tutorials/webrtc/basics/

http://www.html5rocks.com/en/tutorials/getusermedia/intro/

http://www.html5rocks.com/en/tutorials/webrtc/infrastructure/

https://www.youtube.com/watch?v=E8C8ouiXHHk

https://vimeo.com/47682405

https://developer.mozilla.org/en-US/docs/Web/Guide/API/WebRTC

http://www.webrtcbook.com/


TELEMEDICINE PLATFORM USING WEBRTC

About Octovis

37


IOS? API’S? WEARABLES?

We’re Hiring!

38


OCTOTALK.IO

Check out a working demo!

39

http://octotalk.io


[email protected]

Interested in getting Octotalk technology in your company?

40

mailto:[email protected]


Thanks.

41

PATRICK CASON Front-end Engineer & Designer

KENNY HOUSE Software Engineer

Introduction to WebRTC

Technology

Transcript of Introduction to WebRTC