Now Hear This! Putting Voice, Video, and Text into Ruby on Rails

Putting Real-Time Voice, Video and Text into Rails

Now Hear This!

CAN YOU SPEAK MAGIC?

2

Ben Klang

A city well known for our fine airport, and that sacred elixir we call Coca Cola


3

The web today is a lot like outer space


4

On the web, no one can hear you scream


5

The scenario: you’re busy at work, working from home in your corp web application, and you decide you need to speak to a co-worker to answer a question.

Lose context: have to explain what you are looking at and limited ability to invite additional participantsRecord of communication not made a part of the app


6

Web RTC


WHAT IS WEBRTC?

7


WHAT IS WEBRTC?

•Use Camera & Microphone without plugins

7


WHAT IS WEBRTC?


•Setup peer-to-peer connectivity, 2+ people

7


WHAT IS WEBRTC?


•Setup peer-to-peer connectivity, 2+ people

•Common codecs for transmitting high-definition media in real-time

7


WHAT IS WEBRTC?

8


WHAT IS WEBRTC?

•JavaScript Browser APIfor real-time communication (voice & video)

8


WHAT IS WEBRTC?


•Standards for interoperability

8


WHAT IS WEBRTC?


•Standards for interoperability•Opus, G.711, H.264/VP8

8


WHAT IS WEBRTC?



•SDP

8


WHAT IS WEBRTC?



•SDP

•ICE/STUN/TURN

8


WHAT IS WEBRTC?



•SDP

•ICE/STUN/TURN

•DTLS-SRTP

8

CAN YOU SPEAK MAGIC?CAN YOU SPEAK MAGIC?CAN YOU SPEAK MAGIC?

WHAT IS WEBRTC TO ME?

9

Telephones in Web Browsers!


WHAT IS WEBRTC TO ME?

9

Telephones in Web Browsers!

XCommunication


10


11

Communication Topology


12

Alice Bob

The Trapezoid

This gives phone companies a lot of control: Your identity (phone number)Features available had to be common across carriers (stifled innovation)Provides a point of intercept (NSA)Communications are completely divorced from the function they serve


13

Alice Bob

The Triangle

Identity is betterFeature set is betterStill a common point of interceptStill not very contextual


14

The WebRTC TriangleAlice Bob

Signaling

Media


15

http://

When people say that WebRTC does not mandate signaling, this is what they mean: the session setup is up to the application


15

http://

Alice



15

http://

Alice Bob



15

http://

Hav

e Bob

call

me h

ere!

SDP:

v=0

o=al

ice 2

0518 0

IN IP

4 0.0

.0.0

s=-

t=0 0

m

=audio

54609 R

TP/SAVPF 1

09

Alice Bob



15

http://

Hav

e Bob

call

me h

ere!

SDP:

v=0

o=al

ice 2

0518 0

IN IP

4 0.0

.0.0

s=-

t=0 0

m

=audio

54609 R

TP/SAVPF 1

09

Bob is at:

SDP:v=0 o=bob 19915 0 IN IP4 0.0.0.0

s=- t=0 0 m=audio 61001 RTP/SAVPF 109

Alice Bob



15

http://

Hav

e Bob

call

me h

ere!

SDP:

v=0

o=al

ice 2

0518 0

IN IP

4 0.0

.0.0

s=-

t=0 0

m

=audio

54609 R

TP/SAVPF 1

09

Bob is at:

SDP:v=0 o=bob 19915 0 IN IP4 0.0.0.0


Alice Bob

SRTP

SRTP



15

http://

Hav

e Bob

call

me h

ere!

SDP:

v=0

o=al

ice 2

0518 0

IN IP

4 0.0

.0.0

s=-

t=0 0

m

=audio

54609 R

TP/SAVPF 1

09

Bob is at:

SDP:v=0 o=bob 19915 0 IN IP4 0.0.0.0


Alice Bob

SRTP

SRTP

X



15

Hav

e Bob

call

me h

ere!

SDP:

v=0

o=al

ice 2

0518 0

IN IP

4 0.0

.0.0

s=-

t=0 0

m

=audio

54609 R

TP/SAVPF 1

09

Bob is at:

SDP:v=0 o=bob 19915 0 IN IP4 0.0.0.0


Alice Bob

SRTP

SRTP

X



16

Designing Communications Apps

That’s enough about the plumbing


THE FIVE TENETS

17

Adaptive: Can use multiple channels; Still useful if any of the channels are missingFluid: Can add or remove channels on demandReference-able: Conversations can be stored and shared later via URL, with appropriate securityContextual: Communicating isn’t what you’re doing; it’s what you’re doing while you’re doing something else.


THE FIVE TENETS

17

A Modern Voice Application should be…



THE FIVE TENETS

•Adaptive

17




THE FIVE TENETS

•Adaptive

•Fluid

17




THE FIVE TENETS

•Adaptive

•Fluid

•Contextual

17




THE FIVE TENETS

•Adaptive

•Fluid

•Contextual

•Trustworthy

17




THE FIVE TENETS

•Adaptive

•Fluid

•Contextual

•Trustworthy

•Reference-able17




18

Adaptive


BEING ADAPTIVE

19

Alice joins via browser: gets full experience, chat, wideband voice, hi-def video, visualsBob joins via tablet: gets same as above thanks to mobile web, but can’t send video because his browser doesn’t support WebRTC yet??Charlie joins via smartphone: gets same as above, except no video at all because he has reduced bandwidth or a low batteryDouglas joins via landline, because he’s backward and he really like his desk phone - maybe even get audio hints of text chat?Edward also joins via browser, but has a sleeping baby, so he only listens to the audio and talks via chat


BEING ADAPTIVE

19

/

Alice joins via browser: gets full experience, chat, wideband voice, hi-def video, visualsBob joins via tablet: gets same as above thanks to mobile web, but can’t send video because his browser doesn’t support WebRTC yet??Charlie joins via smartphone: gets same as above, except no video at all because he has reduced bandwidth or a low batteryDouglas joins via landline, because he’s backward and he really like his desk phone - maybe even get audio hints of text chat?Edward also joins via browser, but has a sleeping baby, so he only listens to the audio and talks via chat


20

Fluid


BEING FLUID

21

Visual: text -> audio -> + people -> video -> text


•Start with ChatBEING FLUID

21



•Start with Chat•Add audio

BEING FLUID

21



•Start with Chat•Add audio•Invite more participants

BEING FLUID

21



•Start with Chat•Add audio•Invite more participants•Upgrade to video

BEING FLUID

21



•Start with Chat•Add audio•Invite more participants•Upgrade to video•Downgrade back to Chat

BEING FLUID

21



•Start with Chat•Add audio•Invite more participants•Upgrade to video•Downgrade back to Chat•Switch devices and continue

BEING FLUID

21



22

Contextual


“Communicating isn’t going to be what you’re doing - it’s what you’ll be doing

while you’re doing something else”

- Geoff Hollingworth Ericsson Head of AT&T Foundry

23


24How many sales reps are still in homes?

How much have we sold so far this month?

How many callers are in the queue right now?

Add my manager to this call

When is the next available appointment?

BEING CONTEXTUAL


25


26


27

Trustworthy


TRUSTWORTHY

28


TRUSTWORTHY•Don't surprise the user!

28


TRUSTWORTHY•Don't surprise the user!•Don't share the data with unexpected recipients

28


TRUSTWORTHY•Don't surprise the user!•Don't share the data with unexpected recipients•Help users make smart choices where required (with sane defaults)

28


TRUSTWORTHY•Don't surprise the user!•Don't share the data with unexpected recipients•Help users make smart choices where required (with sane defaults)•If identity is asserted, it should be user-centric and strong

28


29

Reference-able


REFERENCEABILITY

30

All participants are invited to the chat via unique URLThis URL always contains the latest state of the chat, whether in-progress, idle, or archivedMay provide enhanced features, such as downloadable copies of the text, audio or video, a summary of links or media shared, or even transcriptions of the conversationIs searchable!


REFERENCEABILITY•Make Interactions Shareable

30



REFERENCEABILITY•Make Interactions Shareable•Every conversation has a URL that:

30



REFERENCEABILITY•Make Interactions Shareable•Every conversation has a URL that:•Is permanent & unique

30



REFERENCEABILITY•Make Interactions Shareable•Every conversation has a URL that:•Is permanent & unique•Represents the latest state of the communication or request

30



REFERENCEABILITY•Make Interactions Shareable•Every conversation has a URL that:•Is permanent & unique•Represents the latest state of the communication or request•May contain content alternatives: recording + transcription + annotation

30



REFERENCEABILITY•Make Interactions Shareable•Every conversation has a URL that:•Is permanent & unique•Represents the latest state of the communication or request•May contain content alternatives: recording + transcription + annotation•Is searchable & downloadable

30



REFERENCEABILITY•Make Interactions Shareable•Every conversation has a URL that:•Is permanent & unique•Represents the latest state of the communication or request•May contain content alternatives: recording + transcription + annotation•Is searchable & downloadable•Can be shared (respecting privacy limits)

30



31

What Should We Build?


31

What Should We Build?3 Ideas


1. Live, Anonymous Matchmaking

32


33

http://funnydate.com/have-a-chat

FunnyDate

FunnyDate

SHErlock22FemaleMaleOakdale

Books, Moustaches, Puzzles

Captain Beard 23

MaleFemale

Springfield

Books, Moustaches, Puzzles

USERNAMEAGE

GENDERSEEKINGCITY

MATCHINGINTERESTS

CAN YOU SPEAK MAGIC?CAN YOU SPEAK MAGIC?

LIVE, ANONYMOUS DATING

34

FunnyDate


LIVE, ANONYMOUS DATING•Safe Introductions

34

FunnyDate


LIVE, ANONYMOUS DATING•Safe Introductions•Strict anonymity

34

FunnyDate


LIVE, ANONYMOUS DATING•Safe Introductions•Strict anonymity•No need to exchangephone numbers

34

FunnyDate


LIVE, ANONYMOUS DATING•Safe Introductions•Strict anonymity•No need to exchangephone numbers•Low friction interfaceenables users to connect in real-time

34

FunnyDate


LIVE, ANONYMOUS DATING•Safe Introductions•Strict anonymity•No need to exchangephone numbers•Low friction interfaceenables users to connect in real-time•Silly video tricks used to break the ice

34

FunnyDate


LIVE, ANONYMOUS DATING•Safe Introductions•Strict anonymity•No need to exchangephone numbers•Low friction interfaceenables users to connect in real-time•Silly video tricks used to break the ice•...or even add more anonymity

34

FunnyDate


LIVE, ANONYMOUS DATING•Safe Introductions•Strict anonymity•No need to exchangephone numbers•Low friction interfaceenables users to connect in real-time•Silly video tricks used to break the ice•...or even add more anonymity

•Upsell: Live language translation

34

FunnyDate


2. Incident Response App

35


36


INCIDENT RESPONSE

37


INCIDENT RESPONSE•Timely, Contextual Information

37


INCIDENT RESPONSE•Timely, Contextual Information•Adapt for mobile vs. desktop

37


INCIDENT RESPONSE•Timely, Contextual Information•Adapt for mobile vs. desktop•Group-based communication

37


INCIDENT RESPONSE•Timely, Contextual Information•Adapt for mobile vs. desktop•Group-based communication•Inherit from existing organization groups

37


INCIDENT RESPONSE•Timely, Contextual Information•Adapt for mobile vs. desktop•Group-based communication•Inherit from existing organization groups•Allow ad-hoc participants (guests)

37


INCIDENT RESPONSE•Timely, Contextual Information•Adapt for mobile vs. desktop•Group-based communication•Inherit from existing organization groups•Allow ad-hoc participants (guests)•Federate with external services

37



•Incident recording/logging

37



•Incident recording/logging•“Lessons learned” and process improvement

37



•Incident recording/logging•“Lessons learned” and process improvement•Links from/to issue tracking systems

37


3. Medical Records Patient Services

38


39


MEDICAL ADVICE/HISTORY

40


MEDICAL ADVICE/HISTORY•Secure Caller Authentication

40


MEDICAL ADVICE/HISTORY•Secure Caller Authentication•Reuse primary auth via website

40


MEDICAL ADVICE/HISTORY•Secure Caller Authentication•Reuse primary auth via website•Verify with voice biometrics

40


MEDICAL ADVICE/HISTORY•Secure Caller Authentication•Reuse primary auth via website•Verify with voice biometrics•Cross-check against caller location

40


MEDICAL ADVICE/HISTORY•Secure Caller Authentication•Reuse primary auth via website•Verify with voice biometrics•Cross-check against caller location•Automate Medical Claims

40


MEDICAL ADVICE/HISTORY•Secure Caller Authentication•Reuse primary auth via website•Verify with voice biometrics•Cross-check against caller location•Automate Medical Claims•Call recording/transcription

40


MEDICAL ADVICE/HISTORY•Secure Caller Authentication•Reuse primary auth via website•Verify with voice biometrics•Cross-check against caller location•Automate Medical Claims•Call recording/transcription•Medical advice given to patient automatically added to patient file

40


MEDICAL ADVICE/HISTORY•Secure Caller Authentication•Reuse primary auth via website•Verify with voice biometrics•Cross-check against caller location•Automate Medical Claims•Call recording/transcription•Medical advice given to patient automatically added to patient file•Easy Auditing/Service Quality Assurance

40


41

Demo Time


RESOURCES•WebRTC •http://webrtc.github.io/samples/ •http://www.webrtc.org/ •http://www.webrtcchallenge.com/

•Voice in Ruby •http://www.adhearsion.com/ •https://github.com/benlangfeld/ruby_speech

•Ben Klang, Mojo Lingo •@bklang •[email protected]

42

http://webrtc.github.io/samples/

http://www.webrtc.org/

http://www.webrtcchallenge.com/

http://www.adhearsion.com/

https://github.com/benlangfeld/ruby_speech

mailto:[email protected]

Now Hear This! Putting Voice, Video, and Text into Ruby on Rails

Technology

Transcript of Now Hear This! Putting Voice, Video, and Text into Ruby on Rails