Building a Streaming Apple TV app

Chris Adamson (@invalidname) CocoaConf DC • September, 2016

Slides will be available at slideshare.net/invalidname Code will be available at github.com/invalidstream

“The future of TV is apps.”

“The future of apps is TV.”

Video (how do you even?)

Digital Media

• Codecs — How we encode (represent) audio or video in a digital form

• AAC, Apple Lossless, H.264, ProRes

• Containers — How we deliver encoded media

• MP3, .aac, Shoutcast, .mov, .mp4

Codecs

• From “coder / decoder”

• Simplest: PCM (audio), M-JPEG (video)

• Modern: AAC, H.264/AVC

• Next gen: H.265 (HEVC), VP9

Inter-frame compression

Containers: QuickTime

• Dates back to the early 1990s

• Originally used Mac dual-fork files

• Content agnostic: nearly any media content can be contained in a QuickTime file

• MPEG-4 adopted QuickTime as basis of its file format (1998)

QuickTime: Great for Editing

QuickTime: OK for playback

• Resolve all references into a single file (flatten) or export (possibly re-encoding) to a file

• Dual-fork approach abandoned to support QuickTime for Windows (and later, Mac OS X)

QuickTime: Bad for Streaming

• Metadata at end of file meant you needed the whole file

• First solution: “Quick-Start”, which put metadata at beginning of file

• Later: QuickTime Streaming Server

• Also-ran to RealPlayer, Windows ASF, Flash

Streaming

• Different from progressive download in that the client does only wants the media for the current playback time

• Live sources are possible

Bad old streaming

• Real, ASF, Flash, QTSS – socket-based connections

• Hard to scale

• Bad for mobile

• Not on port 80; frequently blocked

Adaptive streaming

• Send “stream” as a series of small files over HTTP, expect client to reassemble them

• Easy to scale

• Good for mobile / inconsistent networks

• Runs on port 80; never blocked

HTTP Live Streaming

• Apple’s de facto standard for adaptive streaming

• Supported on iOS (iPhone, Apple TV) from day one

• Also: QuickTime, Safari, Microsoft Edge, Flash, Android, Roku, etc

• Standards-based competitor: MPEG-DASH

Demo

Streaming playback @IBAction func handlePlayButtonTapped(_ sender: AnyObject) { guard let demoItem = demoItem else { return }

self.playerVC?.player?.pause() let playerVC = AVPlayerViewController() playerVC.showsPlaybackControls = true let player = AVPlayer(url: demoItem.url) playerVC.player = player present(playerVC, animated: true) player.play() self.playerVC = playerVC }

This is identical to local file playback

HLS, how do you even?

• Source media is split into short (~10 second) segments, as MPEG-2 transport stream (.ts) files

• iOS 10 / macOS Sierra support “Fragmented MP4”, which is potentially MPEG-DASH compatible

• An .m3u8 playlist provides a manifest of files to be played back

#EXTM3U#EXT-X-TARGETDURATION:10#EXT-X-VERSION:3#EXT-X-MEDIA-SEQUENCE:0#EXT-X-PLAYLIST-TYPE:VOD#EXTINF:10.00000,fileSequence0.ts#EXTINF:10.00000,fileSequence1.ts#EXTINF:10.00000,fileSequence2.ts#EXTINF:10.00000,fileSequence3.ts#EXTINF:10.00000,fileSequence4.ts#EXTINF:10.00000,fileSequence5.ts#EXTINF:10.00000,fileSequence6.ts#EXTINF:10.00000,fileSequence7.ts#EXTINF:10.00000,fileSequence8.ts#EXTINF:10.00000,fileSequence9.ts#EXTINF:7.40000,fileSequence10.ts#EXT-X-ENDLIST

Streaming Tools

mediafilesegmenter

• Takes a source MP4 file and splits it into segment files and an .m3u8 playlist

• Resulting directory can be hosted on any http server to provide stream

Demo

Why is this better than a flat file?

• Easier to skip around

• Can be encrypted / DRM’ed

• Also, about that bitrate…

Variant Playlists

• Allow you to encode at multiple bitrates

• Client determines at runtime if it is keeping up, and whether to request a different variant for the next 10 sec segment

• Instead of “buffering” stalls, viewers just see degraded image/sound quality

variantplaylistcreator#EXTM3U#EXT-X-STREAM-INF:AVERAGE-BANDWIDTH=1269452,BANDWIDTH=1304420,CODECS="mp4a.40.2, avc1.4d4016",RESOLUTION=640x360,FRAME-RATE=15.000,CLOSED-CAPTIONS=NONEwifi-high/prog_index.m3u8#EXT-X-STREAM-INF:AVERAGE-BANDWIDTH=5172486,BANDWIDTH=5330176,CODECS="mp4a.40.2, avc1.4d401f",RESOLUTION=960x540,FRAME-RATE=15.000,CLOSED-CAPTIONS=NONEbroadband-high/prog_index.m3u8#EXT-X-STREAM-INF:AVERAGE-BANDWIDTH=2599260,BANDWIDTH=2677271,CODECS="mp4a.40.2, avc1.4d4016",RESOLUTION=640x360,FRAME-RATE=15.000,CLOSED-CAPTIONS=NONEbroadband-low/prog_index.m3u8#EXT-X-STREAM-INF:AVERAGE-BANDWIDTH=757070,BANDWIDTH=782983,CODECS="mp4a.40.2, avc1.42e016",RESOLUTION=640x360,FRAME-RATE=15.000,CLOSED-CAPTIONS=NONEwifi-low/prog_index.m3u8#EXT-X-STREAM-INF:AVERAGE-BANDWIDTH=474503,BANDWIDTH=507149,CODECS="mp4a.40.2, avc1.42e015",RESOLUTION=428x240,FRAME-RATE=15.000,CLOSED-CAPTIONS=NONEcellular-high/prog_index.m3u8#EXT-X-STREAM-INF:AVERAGE-BANDWIDTH=224956,BANDWIDTH=243197,CODECS="mp4a.40.2, avc1.42e015",RESOLUTION=428x240,FRAME-RATE=15.000,CLOSED-CAPTIONS=NONEcellular-low/prog_index.m3u8#EXT-X-STREAM-INF:AVERAGE-BANDWIDTH=33400,BANDWIDTH=33547,CODECS="mp4a.40.2",CLOSED-CAPTIONS=NONEaudio-only/prog_index.m3u8

Demo

Compressor’s HLS bitratesTable 1

Broadband high 5000Broadband low 2500Wi-fi high 1250Wi-fi low 735Cellular high 450Cellular low 225Audio only 32

TN 2224

https://developer.apple.com/library/ios/technotes/tn2224/_index.html

TN 2224 Bitrates

Livestreaming!

HLS livestreams• Exactly the same as the VOD streams we’ve

seen before

• Support variant playlists and encryption

• Difference: the playlist doesn’t have a #EXT-X-ENDLIST

• Client knows to refresh the playlist periodically to fetch new segment files

mediastreamsegmenter

• Works just like mediafilesegmenter to create the segment files and a playlist

• Takes as input an MPEG-2 transport stream

• Nothing in Apple’s developer toolchain can actually create such a stream

Streaming servers

• Services that create the stream files from an input stream you provide

• Can also transcode to lower bitrates and provide a variant playlist

• Can also transmux to non-HLS formats

• Generally you send them RTMP

Open Broadcast Software

https://obsproject.com

Demo

Wowza

http://wowza.com

Wowza

• Top streaming service provider

• Wowza Streaming Engine — runs on your server (real or virtual)

• Wowza Streaming Cloud — streaming as a service

Wirecast

http://telestream.net/wirecast/

Wirecast

• Proprietary streaming production software

• Mixes shots, captures from cameras, iOS devices, or screen, highly customizable

• Prices start at $500

• Watch for a 1/3 off sale on Black Friday

Demo

Securing Streams

• With -k option, mediafilesegmenter and mediastreamsegmenter will AES-128 encrypt segments, using keys from the specified location

• FairPlay Streaming for HLS provides severe DRM.

• See WWDC 2015 session “Content Protection for HTTP Live Streaming”

Takeaways• Video streaming is the best thing the Apple TV

does

• HLS streams are just flat files on a web server

• Create them with mediafilesegmenter or a streaming server

• For lifestreams RTMP to the streaming server with OBS, Wirecast, etc.

Building a Streaming Apple TV app

Chris Adamson (@invalidname) CocoaConf DC • September, 2016

Slides will be available at slideshare.net/invalidname Code will be available at github.com/invalidstream