Architecture The Evolution of Spotify Home - QCon.ai · The Evolution of Spotify Home Architecture....
Transcript of Architecture The Evolution of Spotify Home - QCon.ai · The Evolution of Spotify Home Architecture....
![Page 1: Architecture The Evolution of Spotify Home - QCon.ai · The Evolution of Spotify Home Architecture. Emily Staff Engineer Anil Data Engineer @emilymsa @anilmuppallar. Our mission is](https://reader035.fdocuments.in/reader035/viewer/2022070723/5f020d297e708231d4025595/html5/thumbnails/1.jpg)
The Evolution of Spotify Home Architecture
![Page 2: Architecture The Evolution of Spotify Home - QCon.ai · The Evolution of Spotify Home Architecture. Emily Staff Engineer Anil Data Engineer @emilymsa @anilmuppallar. Our mission is](https://reader035.fdocuments.in/reader035/viewer/2022070723/5f020d297e708231d4025595/html5/thumbnails/2.jpg)
Emily
Staff Engineer
Anil
Data Engineer
@anilmuppallar@emilymsa
![Page 3: Architecture The Evolution of Spotify Home - QCon.ai · The Evolution of Spotify Home Architecture. Emily Staff Engineer Anil Data Engineer @emilymsa @anilmuppallar. Our mission is](https://reader035.fdocuments.in/reader035/viewer/2022070723/5f020d297e708231d4025595/html5/thumbnails/3.jpg)
Our mission is to unlock the potential of human creativity — by giving a million creative artists the opportunity to live off their art and billions of fans the
opportunity to enjoy and be inspired by it.
![Page 4: Architecture The Evolution of Spotify Home - QCon.ai · The Evolution of Spotify Home Architecture. Emily Staff Engineer Anil Data Engineer @emilymsa @anilmuppallar. Our mission is](https://reader035.fdocuments.in/reader035/viewer/2022070723/5f020d297e708231d4025595/html5/thumbnails/4.jpg)
![Page 5: Architecture The Evolution of Spotify Home - QCon.ai · The Evolution of Spotify Home Architecture. Emily Staff Engineer Anil Data Engineer @emilymsa @anilmuppallar. Our mission is](https://reader035.fdocuments.in/reader035/viewer/2022070723/5f020d297e708231d4025595/html5/thumbnails/5.jpg)
![Page 6: Architecture The Evolution of Spotify Home - QCon.ai · The Evolution of Spotify Home Architecture. Emily Staff Engineer Anil Data Engineer @emilymsa @anilmuppallar. Our mission is](https://reader035.fdocuments.in/reader035/viewer/2022070723/5f020d297e708231d4025595/html5/thumbnails/6.jpg)
shelf
shelf name
card
![Page 7: Architecture The Evolution of Spotify Home - QCon.ai · The Evolution of Spotify Home Architecture. Emily Staff Engineer Anil Data Engineer @emilymsa @anilmuppallar. Our mission is](https://reader035.fdocuments.in/reader035/viewer/2022070723/5f020d297e708231d4025595/html5/thumbnails/7.jpg)
Overview
● Started with a Batch architecture
● Used services to hide complexity and be more reactive
● Leveraged GCP and added streaming pipelines to build a product based on user activity
![Page 8: Architecture The Evolution of Spotify Home - QCon.ai · The Evolution of Spotify Home Architecture. Emily Staff Engineer Anil Data Engineer @emilymsa @anilmuppallar. Our mission is](https://reader035.fdocuments.in/reader035/viewer/2022070723/5f020d297e708231d4025595/html5/thumbnails/8.jpg)
Batch2016
![Page 9: Architecture The Evolution of Spotify Home - QCon.ai · The Evolution of Spotify Home Architecture. Emily Staff Engineer Anil Data Engineer @emilymsa @anilmuppallar. Our mission is](https://reader035.fdocuments.in/reader035/viewer/2022070723/5f020d297e708231d4025595/html5/thumbnails/9.jpg)
Batch
Songs Played Logs
Word2Vec
![Page 10: Architecture The Evolution of Spotify Home - QCon.ai · The Evolution of Spotify Home Architecture. Emily Staff Engineer Anil Data Engineer @emilymsa @anilmuppallar. Our mission is](https://reader035.fdocuments.in/reader035/viewer/2022070723/5f020d297e708231d4025595/html5/thumbnails/10.jpg)
word2vecA natural language processing model to learn vector representations of words (“embeddings”) from text.
https://www.tensorflow.org/tutorials/word2vec
![Page 11: Architecture The Evolution of Spotify Home - QCon.ai · The Evolution of Spotify Home Architecture. Emily Staff Engineer Anil Data Engineer @emilymsa @anilmuppallar. Our mission is](https://reader035.fdocuments.in/reader035/viewer/2022070723/5f020d297e708231d4025595/html5/thumbnails/11.jpg)
word2vec
Input:Playlists
Output: Vector representation of tracks
![Page 12: Architecture The Evolution of Spotify Home - QCon.ai · The Evolution of Spotify Home Architecture. Emily Staff Engineer Anil Data Engineer @emilymsa @anilmuppallar. Our mission is](https://reader035.fdocuments.in/reader035/viewer/2022070723/5f020d297e708231d4025595/html5/thumbnails/12.jpg)
word2vec
Input:Playlists
Output: Vector representation of tracks
2Pac
Bach
Mozart
![Page 13: Architecture The Evolution of Spotify Home - QCon.ai · The Evolution of Spotify Home Architecture. Emily Staff Engineer Anil Data Engineer @emilymsa @anilmuppallar. Our mission is](https://reader035.fdocuments.in/reader035/viewer/2022070723/5f020d297e708231d4025595/html5/thumbnails/13.jpg)
Batch
Songs Played Logs
Word2Vec
![Page 14: Architecture The Evolution of Spotify Home - QCon.ai · The Evolution of Spotify Home Architecture. Emily Staff Engineer Anil Data Engineer @emilymsa @anilmuppallar. Our mission is](https://reader035.fdocuments.in/reader035/viewer/2022070723/5f020d297e708231d4025595/html5/thumbnails/14.jpg)
Batch
Songs Played Logs
Hadoop Jobs
Word2Vec
![Page 15: Architecture The Evolution of Spotify Home - QCon.ai · The Evolution of Spotify Home Architecture. Emily Staff Engineer Anil Data Engineer @emilymsa @anilmuppallar. Our mission is](https://reader035.fdocuments.in/reader035/viewer/2022070723/5f020d297e708231d4025595/html5/thumbnails/15.jpg)
Batch
Songs Played Logs
Hadoop Jobs Cassandra
Word2Vec
![Page 16: Architecture The Evolution of Spotify Home - QCon.ai · The Evolution of Spotify Home Architecture. Emily Staff Engineer Anil Data Engineer @emilymsa @anilmuppallar. Our mission is](https://reader035.fdocuments.in/reader035/viewer/2022070723/5f020d297e708231d4025595/html5/thumbnails/16.jpg)
Batch
Songs Played Logs
Hadoop Jobs Cassandra
Word2Vec
![Page 17: Architecture The Evolution of Spotify Home - QCon.ai · The Evolution of Spotify Home Architecture. Emily Staff Engineer Anil Data Engineer @emilymsa @anilmuppallar. Our mission is](https://reader035.fdocuments.in/reader035/viewer/2022070723/5f020d297e708231d4025595/html5/thumbnails/17.jpg)
Batch
Songs Played Logs
Hadoop Jobs Cassandra
Word2Vec
CMS
![Page 18: Architecture The Evolution of Spotify Home - QCon.ai · The Evolution of Spotify Home Architecture. Emily Staff Engineer Anil Data Engineer @emilymsa @anilmuppallar. Our mission is](https://reader035.fdocuments.in/reader035/viewer/2022070723/5f020d297e708231d4025595/html5/thumbnails/18.jpg)
Batch
Songs Played Logs
Hadoop Jobs
Fetch Shelf for HomeCassandra
Word2Vec
CMS
![Page 19: Architecture The Evolution of Spotify Home - QCon.ai · The Evolution of Spotify Home Architecture. Emily Staff Engineer Anil Data Engineer @emilymsa @anilmuppallar. Our mission is](https://reader035.fdocuments.in/reader035/viewer/2022070723/5f020d297e708231d4025595/html5/thumbnails/19.jpg)
Pros & Cons
+ Low latency to load Home
+ Fallback to old data if it fails to generate recommendations
- Recommendations updated once every 24 hours
- Calculate recommendations for every user, even if they aren’t active
- Experimentation can be difficult
- Operational overhead to maintain Cassandra and Hadoop
![Page 20: Architecture The Evolution of Spotify Home - QCon.ai · The Evolution of Spotify Home Architecture. Emily Staff Engineer Anil Data Engineer @emilymsa @anilmuppallar. Our mission is](https://reader035.fdocuments.in/reader035/viewer/2022070723/5f020d297e708231d4025595/html5/thumbnails/20.jpg)
Batch
Songs Played Logs
Hadoop Jobs
Fetch Shelf for HomeCassandra
Word2Vec
CMS
![Page 21: Architecture The Evolution of Spotify Home - QCon.ai · The Evolution of Spotify Home Architecture. Emily Staff Engineer Anil Data Engineer @emilymsa @anilmuppallar. Our mission is](https://reader035.fdocuments.in/reader035/viewer/2022070723/5f020d297e708231d4025595/html5/thumbnails/21.jpg)
Batch
Songs Played Logs
Hadoop Jobs
Fetch Shelf for HomeCassandra
Word2Vec
CMS
![Page 22: Architecture The Evolution of Spotify Home - QCon.ai · The Evolution of Spotify Home Architecture. Emily Staff Engineer Anil Data Engineer @emilymsa @anilmuppallar. Our mission is](https://reader035.fdocuments.in/reader035/viewer/2022070723/5f020d297e708231d4025595/html5/thumbnails/22.jpg)
Services2017
![Page 23: Architecture The Evolution of Spotify Home - QCon.ai · The Evolution of Spotify Home Architecture. Emily Staff Engineer Anil Data Engineer @emilymsa @anilmuppallar. Our mission is](https://reader035.fdocuments.in/reader035/viewer/2022070723/5f020d297e708231d4025595/html5/thumbnails/23.jpg)
Services
Songs Played Service
Word2VecService
![Page 24: Architecture The Evolution of Spotify Home - QCon.ai · The Evolution of Spotify Home Architecture. Emily Staff Engineer Anil Data Engineer @emilymsa @anilmuppallar. Our mission is](https://reader035.fdocuments.in/reader035/viewer/2022070723/5f020d297e708231d4025595/html5/thumbnails/24.jpg)
Services
Songs Played Service
CMS
Word2VecService
![Page 25: Architecture The Evolution of Spotify Home - QCon.ai · The Evolution of Spotify Home Architecture. Emily Staff Engineer Anil Data Engineer @emilymsa @anilmuppallar. Our mission is](https://reader035.fdocuments.in/reader035/viewer/2022070723/5f020d297e708231d4025595/html5/thumbnails/25.jpg)
Services
Create Shelf for Home CMS
Songs Played Service
Word2VecService
![Page 26: Architecture The Evolution of Spotify Home - QCon.ai · The Evolution of Spotify Home Architecture. Emily Staff Engineer Anil Data Engineer @emilymsa @anilmuppallar. Our mission is](https://reader035.fdocuments.in/reader035/viewer/2022070723/5f020d297e708231d4025595/html5/thumbnails/26.jpg)
Services
CMS
Songs Played Service
Word2VecService
Create Shelf for Home
![Page 27: Architecture The Evolution of Spotify Home - QCon.ai · The Evolution of Spotify Home Architecture. Emily Staff Engineer Anil Data Engineer @emilymsa @anilmuppallar. Our mission is](https://reader035.fdocuments.in/reader035/viewer/2022070723/5f020d297e708231d4025595/html5/thumbnails/27.jpg)
Services
CMS
Songs Played Service
Word2VecService
Create Shelf for HomeCreate Shelf
for Home
![Page 28: Architecture The Evolution of Spotify Home - QCon.ai · The Evolution of Spotify Home Architecture. Emily Staff Engineer Anil Data Engineer @emilymsa @anilmuppallar. Our mission is](https://reader035.fdocuments.in/reader035/viewer/2022070723/5f020d297e708231d4025595/html5/thumbnails/28.jpg)
Services
CMS
Songs Played Service
Word2VecService
Create Shelf for HomeCreate Shelf
for HomeCreate Shelf for Home
![Page 29: Architecture The Evolution of Spotify Home - QCon.ai · The Evolution of Spotify Home Architecture. Emily Staff Engineer Anil Data Engineer @emilymsa @anilmuppallar. Our mission is](https://reader035.fdocuments.in/reader035/viewer/2022070723/5f020d297e708231d4025595/html5/thumbnails/29.jpg)
Pros & Cons
+ Updates recommendations at request time
+ Calculate recommendations for Home users only
+ Simplified stack
+ Easier to Experiment
+ Google managed infrastructure
- High latency to load Home
- No fallback if request fails
![Page 30: Architecture The Evolution of Spotify Home - QCon.ai · The Evolution of Spotify Home Architecture. Emily Staff Engineer Anil Data Engineer @emilymsa @anilmuppallar. Our mission is](https://reader035.fdocuments.in/reader035/viewer/2022070723/5f020d297e708231d4025595/html5/thumbnails/30.jpg)
Streaming ++ Services2018 - Present
![Page 31: Architecture The Evolution of Spotify Home - QCon.ai · The Evolution of Spotify Home Architecture. Emily Staff Engineer Anil Data Engineer @emilymsa @anilmuppallar. Our mission is](https://reader035.fdocuments.in/reader035/viewer/2022070723/5f020d297e708231d4025595/html5/thumbnails/31.jpg)
Streaming Pipelines
● Google Dataflow pipelines using Spotify Scio - scala wrapper on Apache Beam
● Real time data - Unbounded stream of user events
○ All user events are available as Google Pubsub topics
● Perform aggregation operations using time based windows
○ groupBy, countBy, join...
● Store the results
○ Pubsub, BigQuery, GCS, Bigtable
![Page 32: Architecture The Evolution of Spotify Home - QCon.ai · The Evolution of Spotify Home Architecture. Emily Staff Engineer Anil Data Engineer @emilymsa @anilmuppallar. Our mission is](https://reader035.fdocuments.in/reader035/viewer/2022070723/5f020d297e708231d4025595/html5/thumbnails/32.jpg)
follow
Real time Signals
![Page 33: Architecture The Evolution of Spotify Home - QCon.ai · The Evolution of Spotify Home Architecture. Emily Staff Engineer Anil Data Engineer @emilymsa @anilmuppallar. Our mission is](https://reader035.fdocuments.in/reader035/viewer/2022070723/5f020d297e708231d4025595/html5/thumbnails/33.jpg)
Real time Signals
follow
pubsub
pubsub
pubsub
![Page 34: Architecture The Evolution of Spotify Home - QCon.ai · The Evolution of Spotify Home Architecture. Emily Staff Engineer Anil Data Engineer @emilymsa @anilmuppallar. Our mission is](https://reader035.fdocuments.in/reader035/viewer/2022070723/5f020d297e708231d4025595/html5/thumbnails/34.jpg)
Streaming Pipeline
Real time Signals
follow
pubsub
pubsub
pubsub
![Page 35: Architecture The Evolution of Spotify Home - QCon.ai · The Evolution of Spotify Home Architecture. Emily Staff Engineer Anil Data Engineer @emilymsa @anilmuppallar. Our mission is](https://reader035.fdocuments.in/reader035/viewer/2022070723/5f020d297e708231d4025595/html5/thumbnails/35.jpg)
Streaming Pipeline
Real time Signals
follow pubsub
pubsub
pubsub
pubsub
![Page 36: Architecture The Evolution of Spotify Home - QCon.ai · The Evolution of Spotify Home Architecture. Emily Staff Engineer Anil Data Engineer @emilymsa @anilmuppallar. Our mission is](https://reader035.fdocuments.in/reader035/viewer/2022070723/5f020d297e708231d4025595/html5/thumbnails/36.jpg)
pubsubStreaming Pipeline
Real time Signals
follow Create Shelves
![Page 37: Architecture The Evolution of Spotify Home - QCon.ai · The Evolution of Spotify Home Architecture. Emily Staff Engineer Anil Data Engineer @emilymsa @anilmuppallar. Our mission is](https://reader035.fdocuments.in/reader035/viewer/2022070723/5f020d297e708231d4025595/html5/thumbnails/37.jpg)
pubsubStreaming Pipeline
Real time Signals
follow Create Shelves
![Page 38: Architecture The Evolution of Spotify Home - QCon.ai · The Evolution of Spotify Home Architecture. Emily Staff Engineer Anil Data Engineer @emilymsa @anilmuppallar. Our mission is](https://reader035.fdocuments.in/reader035/viewer/2022070723/5f020d297e708231d4025595/html5/thumbnails/38.jpg)
pubsubStreaming Pipeline
Real time Signals
follow
Songs Played Service
Word2VecService
Create Shelves
![Page 39: Architecture The Evolution of Spotify Home - QCon.ai · The Evolution of Spotify Home Architecture. Emily Staff Engineer Anil Data Engineer @emilymsa @anilmuppallar. Our mission is](https://reader035.fdocuments.in/reader035/viewer/2022070723/5f020d297e708231d4025595/html5/thumbnails/39.jpg)
BT
pubsubStreaming Pipeline
BT
Write Write Shelf
Real time Signals
follow
Fetch Shelf
Songs Played Service
Word2VecService
Create Shelves
![Page 40: Architecture The Evolution of Spotify Home - QCon.ai · The Evolution of Spotify Home Architecture. Emily Staff Engineer Anil Data Engineer @emilymsa @anilmuppallar. Our mission is](https://reader035.fdocuments.in/reader035/viewer/2022070723/5f020d297e708231d4025595/html5/thumbnails/40.jpg)
pubsubStreaming Pipeline
CMS
Real time Signals
follow
BTBT
Write Write Shelf
Fetch Shelf
Songs Played Service
Word2VecService
Create Shelves
![Page 41: Architecture The Evolution of Spotify Home - QCon.ai · The Evolution of Spotify Home Architecture. Emily Staff Engineer Anil Data Engineer @emilymsa @anilmuppallar. Our mission is](https://reader035.fdocuments.in/reader035/viewer/2022070723/5f020d297e708231d4025595/html5/thumbnails/41.jpg)
Pros & Cons
+ Updates recommendations based on user events
+ Computing recommendations out of request path
+ Fresher content, driven by user sessions
+ Fallback to previously generated recommendations
+ Easy to experiment
- More complex stack
- More tuning in the system
- Event spikes
+ Guardrails
- Debugging is more complicated
![Page 42: Architecture The Evolution of Spotify Home - QCon.ai · The Evolution of Spotify Home Architecture. Emily Staff Engineer Anil Data Engineer @emilymsa @anilmuppallar. Our mission is](https://reader035.fdocuments.in/reader035/viewer/2022070723/5f020d297e708231d4025595/html5/thumbnails/42.jpg)
Lessons Learned
Batch
+ Fallback to old recommendations
+ Low latency to load Home
- Updates are slow
Services
+ Updates are fast
- High Latency to load Home
- No fallback if request fails
Streaming ++ Services
+ Updates are frequent/fast
+ Low latency to load Home
+ Fallback to old recommendations
- Balance computation frequency and downstream system load
![Page 43: Architecture The Evolution of Spotify Home - QCon.ai · The Evolution of Spotify Home Architecture. Emily Staff Engineer Anil Data Engineer @emilymsa @anilmuppallar. Our mission is](https://reader035.fdocuments.in/reader035/viewer/2022070723/5f020d297e708231d4025595/html5/thumbnails/43.jpg)
Lessons Learned
Batch
+ Fallback to old recommendations
+ Low latency to load Home
- Updates are slow
Services
+ Updates are fast
- High Latency to load Home
- No fallback if request fails
Streaming ++ Services
+ Updates are frequent/fast
+ Low latency to load Home
+ Fallback to old recommendations
- Balance computation frequency and downstream system load
![Page 44: Architecture The Evolution of Spotify Home - QCon.ai · The Evolution of Spotify Home Architecture. Emily Staff Engineer Anil Data Engineer @emilymsa @anilmuppallar. Our mission is](https://reader035.fdocuments.in/reader035/viewer/2022070723/5f020d297e708231d4025595/html5/thumbnails/44.jpg)
Takeaways
● Less overhead with managed infrastructure. Focus more on product
● If you care about timeliness, then adopt streaming pipelines
○ Beware of event spikes
● Optimize for developer productivity and ease of experimentation
○ Creating a new shelf is as simple as writing a new function.
![Page 45: Architecture The Evolution of Spotify Home - QCon.ai · The Evolution of Spotify Home Architecture. Emily Staff Engineer Anil Data Engineer @emilymsa @anilmuppallar. Our mission is](https://reader035.fdocuments.in/reader035/viewer/2022070723/5f020d297e708231d4025595/html5/thumbnails/45.jpg)
Hi! I’m Luna, Any questions?