audio streaming platform Building an Open-Source based · Building an Open-Source based audio...
Transcript of audio streaming platform Building an Open-Source based · Building an Open-Source based audio...
![Page 1: audio streaming platform Building an Open-Source based · Building an Open-Source based audio streaming platform Maxime Bugeia 2 February 2020. 2 We stream audio! 3 French public](https://reader035.fdocuments.in/reader035/viewer/2022070807/5f05e5967e708231d4154394/html5/thumbnails/1.jpg)
Building an Open-Source based audio streaming platform
Maxime Bugeia
2 February 2020
![Page 2: audio streaming platform Building an Open-Source based · Building an Open-Source based audio streaming platform Maxime Bugeia 2 February 2020. 2 We stream audio! 3 French public](https://reader035.fdocuments.in/reader035/viewer/2022070807/5f05e5967e708231d4154394/html5/thumbnails/2.jpg)
2
We stream audio!
![Page 3: audio streaming platform Building an Open-Source based · Building an Open-Source based audio streaming platform Maxime Bugeia 2 February 2020. 2 We stream audio! 3 French public](https://reader035.fdocuments.in/reader035/viewer/2022070807/5f05e5967e708231d4154394/html5/thumbnails/3.jpg)
3
French public broadcasting service
6 national channels47 local channels23 music webradios
2 Million listeners per day200k simultaneous listeners
![Page 4: audio streaming platform Building an Open-Source based · Building an Open-Source based audio streaming platform Maxime Bugeia 2 February 2020. 2 We stream audio! 3 French public](https://reader035.fdocuments.in/reader035/viewer/2022070807/5f05e5967e708231d4154394/html5/thumbnails/4.jpg)
4
Before
Black Box3rd party Icecast
![Page 5: audio streaming platform Building an Open-Source based · Building an Open-Source based audio streaming platform Maxime Bugeia 2 February 2020. 2 We stream audio! 3 French public](https://reader035.fdocuments.in/reader035/viewer/2022070807/5f05e5967e708231d4154394/html5/thumbnails/5.jpg)
5
After
Icecast
HLSOur own cloud and open source basedinfrastructure
![Page 6: audio streaming platform Building an Open-Source based · Building an Open-Source based audio streaming platform Maxime Bugeia 2 February 2020. 2 We stream audio! 3 French public](https://reader035.fdocuments.in/reader035/viewer/2022070807/5f05e5967e708231d4154394/html5/thumbnails/6.jpg)
6
Audio streaming
ICECAST
Streaming server
Master/Relay Architecture
Single bitrate
No cache
HLS
Segmented audio/video files
Sliding Playlist (m3u8)
Adaptive bitrate
Better mobile experience
Cacheable content
Just “static” filesLong persistent TCP connections
![Page 7: audio streaming platform Building an Open-Source based · Building an Open-Source based audio streaming platform Maxime Bugeia 2 February 2020. 2 We stream audio! 3 French public](https://reader035.fdocuments.in/reader035/viewer/2022070807/5f05e5967e708231d4154394/html5/thumbnails/7.jpg)
Getting audio to the cloud
7
Audio to listeners
![Page 8: audio streaming platform Building an Open-Source based · Building an Open-Source based audio streaming platform Maxime Bugeia 2 February 2020. 2 We stream audio! 3 French public](https://reader035.fdocuments.in/reader035/viewer/2022070807/5f05e5967e708231d4154394/html5/thumbnails/8.jpg)
Getting audio to the cloud
8
Audio to listeners
Producing stream formats
Icecast HLS
![Page 9: audio streaming platform Building an Open-Source based · Building an Open-Source based audio streaming platform Maxime Bugeia 2 February 2020. 2 We stream audio! 3 French public](https://reader035.fdocuments.in/reader035/viewer/2022070807/5f05e5967e708231d4154394/html5/thumbnails/9.jpg)
Getting audio to the cloud
Deliver content to listeners
9
Audio to listeners
Producing stream formats
CDNIcecast HLS
![Page 10: audio streaming platform Building an Open-Source based · Building an Open-Source based audio streaming platform Maxime Bugeia 2 February 2020. 2 We stream audio! 3 French public](https://reader035.fdocuments.in/reader035/viewer/2022070807/5f05e5967e708231d4154394/html5/thumbnails/10.jpg)
Getting audio to the cloud
Deliver content to listeners
10
Audio to listeners
Producing stream formats
CDN
Operate the platform
Icecast HLS
![Page 11: audio streaming platform Building an Open-Source based · Building an Open-Source based audio streaming platform Maxime Bugeia 2 February 2020. 2 We stream audio! 3 French public](https://reader035.fdocuments.in/reader035/viewer/2022070807/5f05e5967e708231d4154394/html5/thumbnails/11.jpg)
11
Getting audio to the cloud!
![Page 12: audio streaming platform Building an Open-Source based · Building an Open-Source based audio streaming platform Maxime Bugeia 2 February 2020. 2 We stream audio! 3 French public](https://reader035.fdocuments.in/reader035/viewer/2022070807/5f05e5967e708231d4154394/html5/thumbnails/12.jpg)
LOCAL CHANNELS
WEBRADIOS
NATIONAL CHANNELS
WAN
12
MULTICAST
Transport: multicast problem
DIRECT CONNECT
![Page 13: audio streaming platform Building an Open-Source based · Building an Open-Source based audio streaming platform Maxime Bugeia 2 February 2020. 2 We stream audio! 3 French public](https://reader035.fdocuments.in/reader035/viewer/2022070807/5f05e5967e708231d4154394/html5/thumbnails/13.jpg)
LOCAL CHANNELS
WEBRADIOS
NATIONAL CHANNELS
WAN
13
MULTICAST
Transport: multicast problem
? DIRECT CONNECT
![Page 14: audio streaming platform Building an Open-Source based · Building an Open-Source based audio streaming platform Maxime Bugeia 2 February 2020. 2 We stream audio! 3 French public](https://reader035.fdocuments.in/reader035/viewer/2022070807/5f05e5967e708231d4154394/html5/thumbnails/14.jpg)
LOCAL CHANNELS
WEBRADIOS
NATIONAL CHANNELS
WAN
14
MULTICAST
Transport: multicast problem
?No multicast in AWS!
DIRECT CONNECT
![Page 15: audio streaming platform Building an Open-Source based · Building an Open-Source based audio streaming platform Maxime Bugeia 2 February 2020. 2 We stream audio! 3 French public](https://reader035.fdocuments.in/reader035/viewer/2022070807/5f05e5967e708231d4154394/html5/thumbnails/15.jpg)
15
Transport: SRT to the rescue
![Page 16: audio streaming platform Building an Open-Source based · Building an Open-Source based audio streaming platform Maxime Bugeia 2 February 2020. 2 We stream audio! 3 French public](https://reader035.fdocuments.in/reader035/viewer/2022070807/5f05e5967e708231d4154394/html5/thumbnails/16.jpg)
LOCAL CHANNELS
WEBRADIOS
NATIONAL CHANNELS
WAN
16
MULTICAST
Transport: to the cloud!
DIRECT CONNECT
![Page 17: audio streaming platform Building an Open-Source based · Building an Open-Source based audio streaming platform Maxime Bugeia 2 February 2020. 2 We stream audio! 3 French public](https://reader035.fdocuments.in/reader035/viewer/2022070807/5f05e5967e708231d4154394/html5/thumbnails/17.jpg)
17
SRT CALLER 1
SRT CALLER 2
SRT LISTENER 1
SRT LISTENER 2
mainbackup
mainbackup
5 inputs per channel
Transport: resiliencybackup sat
backup sat
![Page 18: audio streaming platform Building an Open-Source based · Building an Open-Source based audio streaming platform Maxime Bugeia 2 February 2020. 2 We stream audio! 3 French public](https://reader035.fdocuments.in/reader035/viewer/2022070807/5f05e5967e708231d4154394/html5/thumbnails/18.jpg)
18
Audio to cloud
DIRECT CONNECT
SAT BACKUP
![Page 19: audio streaming platform Building an Open-Source based · Building an Open-Source based audio streaming platform Maxime Bugeia 2 February 2020. 2 We stream audio! 3 French public](https://reader035.fdocuments.in/reader035/viewer/2022070807/5f05e5967e708231d4154394/html5/thumbnails/19.jpg)
19
Producing stream formats!
![Page 20: audio streaming platform Building an Open-Source based · Building an Open-Source based audio streaming platform Maxime Bugeia 2 February 2020. 2 We stream audio! 3 French public](https://reader035.fdocuments.in/reader035/viewer/2022070807/5f05e5967e708231d4154394/html5/thumbnails/20.jpg)
20
Liquidsoap
Open Source Audio & Video Streaming Language
Collaboration with Radio France
One program to rule them all
Receive Transcode Control
SRT listener Output AACOutput MP3Produce HLSProduce Icecast
Switch between sourcesFallback logicExpose metrics
![Page 21: audio streaming platform Building an Open-Source based · Building an Open-Source based audio streaming platform Maxime Bugeia 2 February 2020. 2 We stream audio! 3 French public](https://reader035.fdocuments.in/reader035/viewer/2022070807/5f05e5967e708231d4154394/html5/thumbnails/21.jpg)
21
Producing the streams
DIRECT CONNECT
SAT BACKUP
IcecastTranscoderLiquidsoap
HLS
![Page 22: audio streaming platform Building an Open-Source based · Building an Open-Source based audio streaming platform Maxime Bugeia 2 February 2020. 2 We stream audio! 3 French public](https://reader035.fdocuments.in/reader035/viewer/2022070807/5f05e5967e708231d4154394/html5/thumbnails/22.jpg)
22
LiquidsoapIcecastMaster
Nginx
Icecast stream
HLS stream
Icecast stream
Transcoder Architecture
HLS files
![Page 23: audio streaming platform Building an Open-Source based · Building an Open-Source based audio streaming platform Maxime Bugeia 2 February 2020. 2 We stream audio! 3 French public](https://reader035.fdocuments.in/reader035/viewer/2022070807/5f05e5967e708231d4154394/html5/thumbnails/23.jpg)
Radio as Code
23
Define inputs
![Page 24: audio streaming platform Building an Open-Source based · Building an Open-Source based audio streaming platform Maxime Bugeia 2 February 2020. 2 We stream audio! 3 French public](https://reader035.fdocuments.in/reader035/viewer/2022070807/5f05e5967e708231d4154394/html5/thumbnails/24.jpg)
Radio as Code
24
Live: what we want to play
![Page 25: audio streaming platform Building an Open-Source based · Building an Open-Source based audio streaming platform Maxime Bugeia 2 February 2020. 2 We stream audio! 3 French public](https://reader035.fdocuments.in/reader035/viewer/2022070807/5f05e5967e708231d4154394/html5/thumbnails/25.jpg)
Radio as Code
25
Fallback logic
![Page 26: audio streaming platform Building an Open-Source based · Building an Open-Source based audio streaming platform Maxime Bugeia 2 February 2020. 2 We stream audio! 3 French public](https://reader035.fdocuments.in/reader035/viewer/2022070807/5f05e5967e708231d4154394/html5/thumbnails/26.jpg)
Radio as Code
26
Fallback logic
Never fail!
![Page 27: audio streaming platform Building an Open-Source based · Building an Open-Source based audio streaming platform Maxime Bugeia 2 February 2020. 2 We stream audio! 3 French public](https://reader035.fdocuments.in/reader035/viewer/2022070807/5f05e5967e708231d4154394/html5/thumbnails/27.jpg)
27
Output HLS
Radio as Code
Source that never fails
![Page 28: audio streaming platform Building an Open-Source based · Building an Open-Source based audio streaming platform Maxime Bugeia 2 February 2020. 2 We stream audio! 3 French public](https://reader035.fdocuments.in/reader035/viewer/2022070807/5f05e5967e708231d4154394/html5/thumbnails/28.jpg)
28
Output HLS
Radio as Code
![Page 29: audio streaming platform Building an Open-Source based · Building an Open-Source based audio streaming platform Maxime Bugeia 2 February 2020. 2 We stream audio! 3 French public](https://reader035.fdocuments.in/reader035/viewer/2022070807/5f05e5967e708231d4154394/html5/thumbnails/29.jpg)
29
Output HLS
Radio as Code
![Page 30: audio streaming platform Building an Open-Source based · Building an Open-Source based audio streaming platform Maxime Bugeia 2 February 2020. 2 We stream audio! 3 French public](https://reader035.fdocuments.in/reader035/viewer/2022070807/5f05e5967e708231d4154394/html5/thumbnails/30.jpg)
30
Output Icecast
Radio as Code
Source that never fails
![Page 31: audio streaming platform Building an Open-Source based · Building an Open-Source based audio streaming platform Maxime Bugeia 2 February 2020. 2 We stream audio! 3 French public](https://reader035.fdocuments.in/reader035/viewer/2022070807/5f05e5967e708231d4154394/html5/thumbnails/31.jpg)
31
LiquidsoapIcecastMaster
HLS files
live.m3u8live_hifi.m3u8segment1.tssegmentxxx.ts
Transcoders Architecture
HLS streamNginx
Icecast stream
Icecast stream
![Page 32: audio streaming platform Building an Open-Source based · Building an Open-Source based audio streaming platform Maxime Bugeia 2 February 2020. 2 We stream audio! 3 French public](https://reader035.fdocuments.in/reader035/viewer/2022070807/5f05e5967e708231d4154394/html5/thumbnails/32.jpg)
32
Scaling to 200k listeners!
![Page 33: audio streaming platform Building an Open-Source based · Building an Open-Source based audio streaming platform Maxime Bugeia 2 February 2020. 2 We stream audio! 3 French public](https://reader035.fdocuments.in/reader035/viewer/2022070807/5f05e5967e708231d4154394/html5/thumbnails/33.jpg)
33
https://icecast.radiofrance.fr/fip-hifi.aac
Icecast Master
HTTP :80HTTPS :443
Scale: Icecast
Transcoder 1
Transcoder 2
Icecast Relay 1
Icecast Relay 2
Icecast Relay X
AWS NLB
Icecast Relay
![Page 34: audio streaming platform Building an Open-Source based · Building an Open-Source based audio streaming platform Maxime Bugeia 2 February 2020. 2 We stream audio! 3 French public](https://reader035.fdocuments.in/reader035/viewer/2022070807/5f05e5967e708231d4154394/html5/thumbnails/34.jpg)
34
https://icecast.radiofrance.fr/fip-hifi.aac
Icecast Master
HTTP :80HTTPS :443
Scale: Icecast
Transcoder 1
Transcoder 2
Icecast Relay 1
Icecast Relay 2
Icecast Relay X
AWS NLB
Icecast Relay
No cacheNo autoscalingIcecast Relay automatic fallback
![Page 35: audio streaming platform Building an Open-Source based · Building an Open-Source based audio streaming platform Maxime Bugeia 2 February 2020. 2 We stream audio! 3 French public](https://reader035.fdocuments.in/reader035/viewer/2022070807/5f05e5967e708231d4154394/html5/thumbnails/35.jpg)
35
HLS base stream
Scale: HLS
CDN
Cache layer
Transcoder 1
Transcoder 2
Varnish 1
Varnish 2
Varnish X
AWS NLB
https://stream.radiofrance.fr/fip/fip.m3u8https://stream.radiofrance.fr/fip/fip_hifi.m3u8
fip_aac_hifi_2_301960_1579533023.ts
Segments
Playlists
![Page 36: audio streaming platform Building an Open-Source based · Building an Open-Source based audio streaming platform Maxime Bugeia 2 February 2020. 2 We stream audio! 3 French public](https://reader035.fdocuments.in/reader035/viewer/2022070807/5f05e5967e708231d4154394/html5/thumbnails/36.jpg)
36
HLS base stream
Scale: HLS
CDN
Cache layer
Transcoder 1
Transcoder 2
Varnish 1
Varnish 2
Varnish X
AWS NLB
https://stream.radiofrance.fr/fip/fip.m3u8https://stream.radiofrance.fr/fip/fip_hifi.m3u8
fip_aac_hifi_2_301960_1579533023.ts
Segments
Playlists
Autoscaling at cache layerCDN level fallback possible
![Page 37: audio streaming platform Building an Open-Source based · Building an Open-Source based audio streaming platform Maxime Bugeia 2 February 2020. 2 We stream audio! 3 French public](https://reader035.fdocuments.in/reader035/viewer/2022070807/5f05e5967e708231d4154394/html5/thumbnails/37.jpg)
37
HLS base stream
Scale: HLS
CDN
Cache layer
Transcoder 1
Transcoder 2
Varnish 1
Varnish 2
Varnish X
AWS NLB
https://stream.radiofrance.fr/fip/fip.m3u8https://stream.radiofrance.fr/fip/fip_hifi.m3u8
fip_aac_hifi_2_301960_1579533023.ts
Segments
Playlists
Just like web!
![Page 38: audio streaming platform Building an Open-Source based · Building an Open-Source based audio streaming platform Maxime Bugeia 2 February 2020. 2 We stream audio! 3 French public](https://reader035.fdocuments.in/reader035/viewer/2022070807/5f05e5967e708231d4154394/html5/thumbnails/38.jpg)
38
Operate
![Page 39: audio streaming platform Building an Open-Source based · Building an Open-Source based audio streaming platform Maxime Bugeia 2 February 2020. 2 We stream audio! 3 French public](https://reader035.fdocuments.in/reader035/viewer/2022070807/5f05e5967e708231d4154394/html5/thumbnails/39.jpg)
39
LiquidsoapIcecastMaster
Nginx
Icecast stream
HLS stream
Icecast stream
Transcoder Architecture
HLS files
![Page 40: audio streaming platform Building an Open-Source based · Building an Open-Source based audio streaming platform Maxime Bugeia 2 February 2020. 2 We stream audio! 3 French public](https://reader035.fdocuments.in/reader035/viewer/2022070807/5f05e5967e708231d4154394/html5/thumbnails/40.jpg)
40
LiquidsoapIcecastMaster
Nginx
Liquidsoap xxxx
Icecast stream
HLS stream
Icecast stream
Transcoder Architecture
One Liquidsoap per channel
HLS files
Liquidsoap xxxxLiquidsoap xxxx
Liquidsoap xxxxLiquidsoap xxxx
Liquidsoap xxxx
![Page 41: audio streaming platform Building an Open-Source based · Building an Open-Source based audio streaming platform Maxime Bugeia 2 February 2020. 2 We stream audio! 3 French public](https://reader035.fdocuments.in/reader035/viewer/2022070807/5f05e5967e708231d4154394/html5/thumbnails/41.jpg)
Automate
41
Declarative everything
Generating:
● SRT callers● Liquidsoap scripts● Icecast master & relay conf
![Page 42: audio streaming platform Building an Open-Source based · Building an Open-Source based audio streaming platform Maxime Bugeia 2 February 2020. 2 We stream audio! 3 French public](https://reader035.fdocuments.in/reader035/viewer/2022070807/5f05e5967e708231d4154394/html5/thumbnails/42.jpg)
42
Monitor
VisualizeCollectExport metrics
Node Exporter
Liquidsoap
Icecast exporter
![Page 43: audio streaming platform Building an Open-Source based · Building an Open-Source based audio streaming platform Maxime Bugeia 2 February 2020. 2 We stream audio! 3 French public](https://reader035.fdocuments.in/reader035/viewer/2022070807/5f05e5967e708231d4154394/html5/thumbnails/43.jpg)
43
Monitor
![Page 44: audio streaming platform Building an Open-Source based · Building an Open-Source based audio streaming platform Maxime Bugeia 2 February 2020. 2 We stream audio! 3 French public](https://reader035.fdocuments.in/reader035/viewer/2022070807/5f05e5967e708231d4154394/html5/thumbnails/44.jpg)
44
HLS access logs
HTTP POST Json
Custom logs Ingester
HLS
Icecast
Monitor: real time listeners
Icecast prometheus Exporter
![Page 45: audio streaming platform Building an Open-Source based · Building an Open-Source based audio streaming platform Maxime Bugeia 2 February 2020. 2 We stream audio! 3 French public](https://reader035.fdocuments.in/reader035/viewer/2022070807/5f05e5967e708231d4154394/html5/thumbnails/45.jpg)
45
Monitor
![Page 46: audio streaming platform Building an Open-Source based · Building an Open-Source based audio streaming platform Maxime Bugeia 2 February 2020. 2 We stream audio! 3 French public](https://reader035.fdocuments.in/reader035/viewer/2022070807/5f05e5967e708231d4154394/html5/thumbnails/46.jpg)
46
Learn more
https://github.com/mbugeia/srt2hls
Receive SRT streamProduce HLSPrometheus MetricsHTTP APICommand and fallback logic
https://github.com/Haivision/srt
https://www.liquidsoap.info/
![Page 47: audio streaming platform Building an Open-Source based · Building an Open-Source based audio streaming platform Maxime Bugeia 2 February 2020. 2 We stream audio! 3 French public](https://reader035.fdocuments.in/reader035/viewer/2022070807/5f05e5967e708231d4154394/html5/thumbnails/47.jpg)
47
Special thanks to the Liquidsoap team!
https://www.liquidsoap.info/
Learn more
![Page 48: audio streaming platform Building an Open-Source based · Building an Open-Source based audio streaming platform Maxime Bugeia 2 February 2020. 2 We stream audio! 3 French public](https://reader035.fdocuments.in/reader035/viewer/2022070807/5f05e5967e708231d4154394/html5/thumbnails/48.jpg)
For internal use only / prepared with love
Questions?
![Page 49: audio streaming platform Building an Open-Source based · Building an Open-Source based audio streaming platform Maxime Bugeia 2 February 2020. 2 We stream audio! 3 French public](https://reader035.fdocuments.in/reader035/viewer/2022070807/5f05e5967e708231d4154394/html5/thumbnails/49.jpg)
For internal use only / prepared with love
![Page 50: audio streaming platform Building an Open-Source based · Building an Open-Source based audio streaming platform Maxime Bugeia 2 February 2020. 2 We stream audio! 3 French public](https://reader035.fdocuments.in/reader035/viewer/2022070807/5f05e5967e708231d4154394/html5/thumbnails/50.jpg)
50
HLS / DASH Compatibility
![Page 51: audio streaming platform Building an Open-Source based · Building an Open-Source based audio streaming platform Maxime Bugeia 2 February 2020. 2 We stream audio! 3 French public](https://reader035.fdocuments.in/reader035/viewer/2022070807/5f05e5967e708231d4154394/html5/thumbnails/51.jpg)
51
mp3 / aac Compatibility
![Page 52: audio streaming platform Building an Open-Source based · Building an Open-Source based audio streaming platform Maxime Bugeia 2 February 2020. 2 We stream audio! 3 French public](https://reader035.fdocuments.in/reader035/viewer/2022070807/5f05e5967e708231d4154394/html5/thumbnails/52.jpg)
52
ogg / flac Compatibility
![Page 53: audio streaming platform Building an Open-Source based · Building an Open-Source based audio streaming platform Maxime Bugeia 2 February 2020. 2 We stream audio! 3 French public](https://reader035.fdocuments.in/reader035/viewer/2022070807/5f05e5967e708231d4154394/html5/thumbnails/53.jpg)
53
NLB
Transcoders
Segments HLSHTTP POST
Segment Forwarder
Storekeeper
PGMPostgreSQL
BusRabbitMQ
Cluster Kubernetes
Segment .ts
Playlists .m3u8
Player
Real HLS diffusion