SECOND SCREEN, XMPP

Steffen Larsen slarsen@braintrust.dk twitter: @zooldk

IoT Day, April 2014, Stockholm

<STREAM> ABOUT ME • Developing XMPP since 2007, MSc. Copenhagen Uni. - developed

pidgin VOIP/Video plugin, XMPP for signalling

• Member of the XMPP Standard Foundation (XSF) since 2009 - now also on the editorial team

• Developed software since 1998 for various companies: telcos, streaming companies etc.

• Have my own consulting company (BrainTrust) - Developing software and doing some startups!

RELATED PROJECTS• I have been working with:

• Open source: Tigase server, components, Strophe.js plugins etc.

• Various chat applications, for video playback commentator, support tools etc.

• XMPP over satellites - standardised maritime satellite project (MSDS) for maritime safety service (advanced group chat)

• XMPP in gaming and betting industry

• XMPP for TV presence for various telcos and cable operators in scandinavia and Europe (first one in 2008!)

I’ve done

AGENDA

• Intro to Second Screen

• Brief intro to XMPP

• Second Screen in a XMPP environment (no demo - sorry!)

• Beyond Second Screen

SECOND SCREEN?Second screen, sometimes also referred to as “companion device” (or “companion apps” when referring to software applications), is a term that refers to an additional electronic device (e.g. tablet, smartphone) that allows a content consumer to interact with the content they are consuming, such as TV shows, movies, music, or video games. Extra data is displayed on a portable device synchronised with the content being viewed on television.

-wikipedia

EVES C-CAST

control multiple view

and cameras

Do replays etc.

Do betting

APPLE AIRPLAY

AIRPLAY• Proprietary protocol (formerly known as AirTunes)

• Allows wireless streaming of audio, video and photos

• Troublesome license.. Audio part can be licensed

• Screen casting

• Discover Airplay devices on same network

• Request device control such as trick play (play, pause, ff, rev, stop etc.)

• Streams can only be decrypted by using apple keys (no other DRM systems)

CHROMECAST

CHROMECAST• An implementation of DIAL (netflix)

• DIscover And Launch - Simple! - Multicast for discover, HTTP/REST to open and launch apps (urls).

• Not “locked” to any device, but can discovered on the same network

• Gives you playback of Netflix, Youtube etc. on a simple cheap stick

• Supports screen casting from chrome etc.

CHROMECAST• Libraries for IOS and Android only (no windows)

• Browser to browser support, JS (Chrome only)

• Supports Play Ready and Encrypted Media Extensions (DRM)

• Uses Smooth Streaming and MPEG-Dash - No HLS? WTF!

• Uses Google DNS.. :-(

• White listings of AppIds

SECOND SCREEN? • An offer to interact through a secondary device than the one you are

using as primary screen

• Some call it: “Multi screen”, “Companion devices”, “Beaming”

• The secondary / companion device: smart phone, tablets and laptops etc

• Primary: TV, set-top box, xbox, play station etc.

• Second screen for implemented: kiosk systems, remote controlling, multimedia trickplay, “flick-to-screen”.. which I am going to talk about

WHY SECOND SCREEN?

TRENDS!

TV IS DEAD, LONG LIVE TV! “The ways we access television content is changing and evolving, and the multitude of devices on which we can view TV is ever growing; with TV everywhere on your tablet, mobile or PC, set top boxes and connected TV’s. How do broadcasters and content providers target audiences appropriately, and where should they be focussing their attentions?”

• Usage is different now than before the smart devices, like iPhone, Android and tablets - the want to reach the new audience! !

• second screen for better experience. Some “Smart TVs” and STB are horrible UI and slow on rendering. !

• Make the TV/STB device a stream device, make the UI on second screen

MULTIPLE DEVICES THEY NEED A WAY TO COMMUNICATE

KNOWN PROTOCOLS• DIAL - DIscovery And Launch protocol (Google chromecast) - 2012

• Apple AirPlay

• MQTT - weak security (why not use SASL?), pub/sub only in application layer

• COAP (HTTP / REST based)

• APIs and REST? (many IoT companies does this!)

• Others

REQUIREMENTS• Support for many devices!

• Routable over the Internet

• Secure with authorisation and authentication (because it over the Internet!)

• Transport agnostic (diverse devices, software stacks etc)

• Do not want to reinvent the wheel!

REQUIREMENTS• Easy Discovery & pairing - how do they find each other?

• Roles, authorisation and authentication (diff. devices and users)

• Single user or multi - let others do remote stuff.

• Control over the infrastructure (no dependencies to silos)

• Streaming media or signalling? - most want just signalling

• A protocol instead of an API

API’S VS PROTOCOLS

•Many IoT vendors only focus on API

!•Via an open standard protocol we would be able to communicate a cross vendors and devices.

DISCOVERY• Scan QR code, click download, wait, launch app =

tedious!

• Open web browser, type in URL

• OpenWRT, DDWrt and others: connect to SSID etc

• Have a protocol doing discovery and signalling for you

• Roles - user login / auth or simple devices with no login?

• provisioning / re-provisioning devices.

• Single user / Multiuser

• Do you owe all of the device stack???.. login vs. kiosk system (no login, presentation/buying etc)

ROLES VS PROVISIONING

TO CAST OR TO MIRROR?

• Mirror: to mirror your display and to transcode it on the second screen device

• Cast: to send play and trick play “commands” from the second screen to first screen and let 1’st screen decode etc.

• Stream vs. signalling. Content owners have a hard time accepting streaming (airplay mirroring.. transcending on the sending device etc)

SECOND SCREEN ENVIRONMENTS AND LIMITATIONS

• To be used on a TV or many TVs!..

• Many people in a household!

• Some native, some are browser based! (only a small limited numbers are native of STB).. a diversity of clients and hardware. need a protocol!

• DRM

• Easy to use.. people normally have a phone and have a special application installed where they can watch TV and se VOD (video on demand) - no need for discovery of the app!

• Pairing should only be done if another customer wants to take control of your TV experience!

• An Open Standard! yay!

• Secure & proven over many years! (many implementations, open source / commercial)

• Many routing mechanisms

• Federated! (setup your own)

• Flexible - beyond Second Screen!

• Transport agnostic

• TCP, BOSH - XMPP over HTTP, Websockets

• Diff. backends (many servers support native MySQL, REST or own plugins) for login and authentication

• Easy to plugin your own business logic and extensions

• Many extensions! - for routing possibilities, and others!

• Started as a chat protocol - Google, Facebook, WhatsApp etc.

• Also used in gaming (e.g chess, Bingo), synchronisation software, white boarding and collaboration software, taxi hailing etc.

• Now used in many other places!

WHY?• Can route over the internet (you do not need to be home

to use your second-screen)

• Can be used locally (link local - bonjour) as well (instead of client/server model)

• Versatile protocol - and not just yet another API!

• Open and can federate your data. Move it if you like to your own domain and server

XMPP ARCHITECTURE•Clients (persistent connection) •Servers •Plugins •Components & Gateways

• Normal XMPP architecture involves a server and a lot of clients.

• A user can login with several clients

• Servers can be federated pr. domain

• Server less mode - Link Local (Bonjour)

XMPP ARCHITECTURE

XMPP CONNECTION LIFECYCLE

• Connect with your user and cred. or anonymous login (authorisation)

• Get resource from server (or given from client)

• Send initial presence and capabilities

• Get presence from your other clients if any

• Get roster

• Communicate with communication primitives (persistent connection)

XMPP ADDRESSING AND ROUTING

• Domain routing like mail (SMTP)

• Many diff. mechanisms that can affect a route

• bare JID vs. full JID

• me@example.com vs me@example.com/Home

• extensions like: Pub/Sub, MUC, etc.

COMMUNICATION PRIMITIVES

• 3 Stanzas, thats it!

• <presence/>

• <message/>

• <iq/> (information / query)

<STANZAS>• You can define the payload it self.

• Make an extension!

• Define your own <tag> with your own name space.

• Embed and use JSON containers or other as payload. easier for browsers to understand

  <message from=‘me@example.com’ to='francisco@denmark.lit' id=‘foo'>"" " " <body/>"         <json xmlns="urn:xmpp:json:0">"           { "name": "romeo", "age": "421", "status": "single" }"         </json>         "   </message>

<STANZAS>

EXTENSIONS

RELEVANT EXTENSIONS• Entity Capabilities - XEP-0115

• Service Discovery - XEP-0030

• BOSH / XMPP over HTTP - XEP-0124

• MUC (Multi User Chat) - XEP-0045

• User Location - Geo/GPS - XEP-0080

• Stream Compression - XEP-0138

• PEP - Personal Eventing Protocol - XEP-0163

RELEVANT EXTENSIONS• Jingle - VOIP/Video - XEP-0166

• Server less Messaging (Bonjour/mDNS) - XEP-0174

• Stream Management (acknowledgement / stream resumption) - XEP-0198

• Message Carbons - XEP-0280

• Remote controlling agents - XEP-146

• MAM - Message Archiving Management - XEP-313

• Message Delivery Receipts - XEP-0184

RELEVANT EXTENSIONS• Event Logging over XMPP - XEP-0337

• IoT: Sensor Data

• IoT: Provisioning

• IoT: Control

• IoT: Concentrators

USING XMPP IN SECOND SCREEN

• Almost all OTT (over-the-top) devices like smart TVs and set-top boxes (STB) are browser based.

• We have STBs that support HTTP/1.0!

• Devices that connect natively TCP (like IOS, Android, Microsoft etc)

• Customers can use the website to watch TV/Video as well.. that is also connected (via. websockets)

customer scenario

• You connect with. second screen (IOS):

• me@example.com/IOS

• Turn on (smart)TV or STB and it connects auto.

• me@example.com/TV1

• me@example.com/TV2

CONNECTION AND DISCOVERY

COMMUNICATE!

  <message from=‘me@example.com/IOS’ to=‘me@example.com/TV1’>"" " " <body/>"           <json xmlns="urn:xmpp:json:0">"           { “play-url": “http://something.dk/nicemovieUrl.ts“ }"           </json>         "   </message>

USAGES• Second Screen stuff (trick play etc) - sending play/stop,ff commands!

• Capabilities and service discovery pr. device

• Use presence for watching the number of concurrent clients and throttle it (DRM purposes)

• Use presence for watching user interaction (channel changes, VOD, follow me etc)

• Push notifications via. full JID, bare JID, Pub/Sub (groups) and broadcasts to all

• Remote logging / control the device (customer support)

• Let your devices talk together in many topologies - one-to-one, many-to-one , many-to-many

• Remote recordings on local storage (PVR)

• Exchange bandwidth between devices

BEYOND SECOND SCREEN

BEYOND SECOND SCREEN• Instant Context Switching

• Instant statistics for online devices (second screen as well as TV etc)

• Interconnected TV with your IoT home! (GW for MQTT, zigbee etc)

• Connect with your alarm, turn off the TV etc.

• The living room devices gives away a lot of data for recommendation

XMPP AND IOT• We can connect a lot of diverse devices. No matter transport

• Create your own gateways if you want to connect to other APIs or protocols

• I’ve created a Pub/Sub to MQTT gateway to connect to MQTT connected devices

• WebRTC - federation and discovery

• UPnP cloud - uses XMPP standard for interconnecting devices

• Usage stats! - delivery of a huge amount of real-time data == BIG DATA!

UPNP CLOUD

“Cloud support in the UPnP architecture will enable IoT and internet-wide usage by allowing IP-based devices and servers to discover each other and communicate independent of the network they are connected with.”

QUESTIONS? </STREAM>

Email / XMPP: slarsen@braintrust.dk

Twitter : @zooldk thanks to t.christensen from nordija for inspiration to the slides.