Spotify Lessons- Learning to Let Go of Machines · Spotify is hiring! spotifyjobs.com IO Tribe....
Transcript of Spotify Lessons- Learning to Let Go of Machines · Spotify is hiring! spotifyjobs.com IO Tribe....
![Page 1: Spotify Lessons- Learning to Let Go of Machines · Spotify is hiring! spotifyjobs.com IO Tribe. Title: Spotify Lessons- Learning to Let Go of Machines Created Date: 6/27/2017 8:00:20](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d4c5e581cf2aa161c83e/html5/thumbnails/1.jpg)
Spotify Lessons:Learning to Let Go of Machines
IO Tribe
James Wen, Site Reliability Engineer at SpotifyALF Squad, Infrastructure & Operations Tribe
![Page 2: Spotify Lessons- Learning to Let Go of Machines · Spotify is hiring! spotifyjobs.com IO Tribe. Title: Spotify Lessons- Learning to Let Go of Machines Created Date: 6/27/2017 8:00:20](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d4c5e581cf2aa161c83e/html5/thumbnails/2.jpg)
![Page 3: Spotify Lessons- Learning to Let Go of Machines · Spotify is hiring! spotifyjobs.com IO Tribe. Title: Spotify Lessons- Learning to Let Go of Machines Created Date: 6/27/2017 8:00:20](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d4c5e581cf2aa161c83e/html5/thumbnails/3.jpg)
Let’s control how feature developers think about what their code is actually running on.
![Page 4: Spotify Lessons- Learning to Let Go of Machines · Spotify is hiring! spotifyjobs.com IO Tribe. Title: Spotify Lessons- Learning to Let Go of Machines Created Date: 6/27/2017 8:00:20](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d4c5e581cf2aa161c83e/html5/thumbnails/4.jpg)
Takeaways• Feature developers = happiest with
feature work • Find out developer machine
concerns and mitigate • Migrating to cloud or hybrid? Start
embracing ephemeral service design and infrastructure
![Page 5: Spotify Lessons- Learning to Let Go of Machines · Spotify is hiring! spotifyjobs.com IO Tribe. Title: Spotify Lessons- Learning to Let Go of Machines Created Date: 6/27/2017 8:00:20](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d4c5e581cf2aa161c83e/html5/thumbnails/5.jpg)
Agenda• Why? • Journey • Hybrid Cloud • Ops in Squads
• Future • Learnings
![Page 6: Spotify Lessons- Learning to Let Go of Machines · Spotify is hiring! spotifyjobs.com IO Tribe. Title: Spotify Lessons- Learning to Let Go of Machines Created Date: 6/27/2017 8:00:20](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d4c5e581cf2aa161c83e/html5/thumbnails/6.jpg)
Why?
Why don’t we want feature devs to care too much about infrastructure and machines?
![Page 7: Spotify Lessons- Learning to Let Go of Machines · Spotify is hiring! spotifyjobs.com IO Tribe. Title: Spotify Lessons- Learning to Let Go of Machines Created Date: 6/27/2017 8:00:20](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d4c5e581cf2aa161c83e/html5/thumbnails/7.jpg)
Why?
Time taken on infrastructure tasks = time taken away from feature work
Feature devs = focused on features
![Page 8: Spotify Lessons- Learning to Let Go of Machines · Spotify is hiring! spotifyjobs.com IO Tribe. Title: Spotify Lessons- Learning to Let Go of Machines Created Date: 6/27/2017 8:00:20](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d4c5e581cf2aa161c83e/html5/thumbnails/8.jpg)
Spotify Scale Stats- 140 Million+ Monthly Active Users
- 50 Million+ Subscribers - 30 Million+ Songs - 2 Billion+ Playlists
- Available in 60 markets
![Page 9: Spotify Lessons- Learning to Let Go of Machines · Spotify is hiring! spotifyjobs.com IO Tribe. Title: Spotify Lessons- Learning to Let Go of Machines Created Date: 6/27/2017 8:00:20](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d4c5e581cf2aa161c83e/html5/thumbnails/9.jpg)
Spotify Dev Scale Stats
~900 Devs ~100 Tech Teams ~2000 Services
![Page 10: Spotify Lessons- Learning to Let Go of Machines · Spotify is hiring! spotifyjobs.com IO Tribe. Title: Spotify Lessons- Learning to Let Go of Machines Created Date: 6/27/2017 8:00:20](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d4c5e581cf2aa161c83e/html5/thumbnails/10.jpg)
Spotify Machine Scale Stats
~10,000 Bare Metal Hosts~13,000 Hosts on GCP46 Hardware/VM Types
![Page 11: Spotify Lessons- Learning to Let Go of Machines · Spotify is hiring! spotifyjobs.com IO Tribe. Title: Spotify Lessons- Learning to Let Go of Machines Created Date: 6/27/2017 8:00:20](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d4c5e581cf2aa161c83e/html5/thumbnails/11.jpg)
Example: Capacity Planning
Avg # devs on a team Capacity Planning
![Page 12: Spotify Lessons- Learning to Let Go of Machines · Spotify is hiring! spotifyjobs.com IO Tribe. Title: Spotify Lessons- Learning to Let Go of Machines Created Date: 6/27/2017 8:00:20](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d4c5e581cf2aa161c83e/html5/thumbnails/12.jpg)
Scale doesn’t really matter-Smaller companies/teams = developer time is more valuable
-Larger companies/teams = wasted infra time scales as well
![Page 13: Spotify Lessons- Learning to Let Go of Machines · Spotify is hiring! spotifyjobs.com IO Tribe. Title: Spotify Lessons- Learning to Let Go of Machines Created Date: 6/27/2017 8:00:20](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d4c5e581cf2aa161c83e/html5/thumbnails/13.jpg)
Other Infrastructure Tasks- Machine provisioning- Failure planning - Security updates - Machine maintenance
![Page 14: Spotify Lessons- Learning to Let Go of Machines · Spotify is hiring! spotifyjobs.com IO Tribe. Title: Spotify Lessons- Learning to Let Go of Machines Created Date: 6/27/2017 8:00:20](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d4c5e581cf2aa161c83e/html5/thumbnails/14.jpg)
Dedicated Ops?
![Page 15: Spotify Lessons- Learning to Let Go of Machines · Spotify is hiring! spotifyjobs.com IO Tribe. Title: Spotify Lessons- Learning to Let Go of Machines Created Date: 6/27/2017 8:00:20](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d4c5e581cf2aa161c83e/html5/thumbnails/15.jpg)
Dedicated Ops?~2000 Services74 Infrastructure and Operations Engineers
If all IO engineers → dedicated ops27:1 service:engineer ratio
![Page 16: Spotify Lessons- Learning to Let Go of Machines · Spotify is hiring! spotifyjobs.com IO Tribe. Title: Spotify Lessons- Learning to Let Go of Machines Created Date: 6/27/2017 8:00:20](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d4c5e581cf2aa161c83e/html5/thumbnails/16.jpg)
Ops In SquadsFeature teams handle their own ops and
provisioning
Using the services and tooling the Infrastructure and Operations tribe has
written
![Page 17: Spotify Lessons- Learning to Let Go of Machines · Spotify is hiring! spotifyjobs.com IO Tribe. Title: Spotify Lessons- Learning to Let Go of Machines Created Date: 6/27/2017 8:00:20](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d4c5e581cf2aa161c83e/html5/thumbnails/17.jpg)
We control the level of context feature teams need to operate their services.
![Page 18: Spotify Lessons- Learning to Let Go of Machines · Spotify is hiring! spotifyjobs.com IO Tribe. Title: Spotify Lessons- Learning to Let Go of Machines Created Date: 6/27/2017 8:00:20](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d4c5e581cf2aa161c83e/html5/thumbnails/18.jpg)
- Developer Happiness- Developer effectiveness and context
![Page 19: Spotify Lessons- Learning to Let Go of Machines · Spotify is hiring! spotifyjobs.com IO Tribe. Title: Spotify Lessons- Learning to Let Go of Machines Created Date: 6/27/2017 8:00:20](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d4c5e581cf2aa161c83e/html5/thumbnails/19.jpg)
Journey
![Page 20: Spotify Lessons- Learning to Let Go of Machines · Spotify is hiring! spotifyjobs.com IO Tribe. Title: Spotify Lessons- Learning to Let Go of Machines Created Date: 6/27/2017 8:00:20](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d4c5e581cf2aa161c83e/html5/thumbnails/20.jpg)
- Ops in Squads
- Hybrid Cloud (Ephemerality)
![Page 21: Spotify Lessons- Learning to Let Go of Machines · Spotify is hiring! spotifyjobs.com IO Tribe. Title: Spotify Lessons- Learning to Let Go of Machines Created Date: 6/27/2017 8:00:20](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d4c5e581cf2aa161c83e/html5/thumbnails/21.jpg)
Starting Out
![Page 22: Spotify Lessons- Learning to Let Go of Machines · Spotify is hiring! spotifyjobs.com IO Tribe. Title: Spotify Lessons- Learning to Let Go of Machines Created Date: 6/27/2017 8:00:20](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d4c5e581cf2aa161c83e/html5/thumbnails/22.jpg)
StockholmSan Jose
Rack 2Rack 1
Historical: Feature Developer’s Context for Service’s Capacity
lon-1-dlon-1-b
lon-1-clon-1-akeys
updated
Rack 2
lon-1-f
lon-1-e
updated
![Page 23: Spotify Lessons- Learning to Let Go of Machines · Spotify is hiring! spotifyjobs.com IO Tribe. Title: Spotify Lessons- Learning to Let Go of Machines Created Date: 6/27/2017 8:00:20](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d4c5e581cf2aa161c83e/html5/thumbnails/23.jpg)
Machine Context- Packages - Hostname - Machine specs (CPU, RAM, disk, etc.)
- Uptime and service duration - Location - Local state (files on disk, info in memory)
Unboundv1.6.3
ash2-metadata-a.ash2.spotify.net
Openssl v1.0.0f
2 Cores 8 GB RAM
Tarred LogsIn Virginia
3 Years
![Page 24: Spotify Lessons- Learning to Let Go of Machines · Spotify is hiring! spotifyjobs.com IO Tribe. Title: Spotify Lessons- Learning to Let Go of Machines Created Date: 6/27/2017 8:00:20](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d4c5e581cf2aa161c83e/html5/thumbnails/24.jpg)
Feature Developer Concerns
How to get?
How many?Specs?
How long?
How to talk to it?
Where? Up to date?
How to track?
What tools on it?Maintenance? What to put
on it?
Available?
Service + Business
![Page 25: Spotify Lessons- Learning to Let Go of Machines · Spotify is hiring! spotifyjobs.com IO Tribe. Title: Spotify Lessons- Learning to Let Go of Machines Created Date: 6/27/2017 8:00:20](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d4c5e581cf2aa161c83e/html5/thumbnails/25.jpg)
![Page 26: Spotify Lessons- Learning to Let Go of Machines · Spotify is hiring! spotifyjobs.com IO Tribe. Title: Spotify Lessons- Learning to Let Go of Machines Created Date: 6/27/2017 8:00:20](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d4c5e581cf2aa161c83e/html5/thumbnails/26.jpg)
Feature Developer Concerns
How to get?
How many?Specs?
How long?
How to talk to it?
Where? Up to date?
How to track?
What tools on it?Maintenance? What to put
on it?
Available?
Service + Business
![Page 27: Spotify Lessons- Learning to Let Go of Machines · Spotify is hiring! spotifyjobs.com IO Tribe. Title: Spotify Lessons- Learning to Let Go of Machines Created Date: 6/27/2017 8:00:20](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d4c5e581cf2aa161c83e/html5/thumbnails/27.jpg)
ServerDB
![Page 28: Spotify Lessons- Learning to Let Go of Machines · Spotify is hiring! spotifyjobs.com IO Tribe. Title: Spotify Lessons- Learning to Let Go of Machines Created Date: 6/27/2017 8:00:20](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d4c5e581cf2aa161c83e/html5/thumbnails/28.jpg)
Feature Developer Concerns
How to get?
How long?
How to talk to it?
What tools on it?Maintenance? What to put
on it?
Available?
Service + BusinessHow to track?
Where? Up to date?
How many? Specs?
![Page 29: Spotify Lessons- Learning to Let Go of Machines · Spotify is hiring! spotifyjobs.com IO Tribe. Title: Spotify Lessons- Learning to Let Go of Machines Created Date: 6/27/2017 8:00:20](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d4c5e581cf2aa161c83e/html5/thumbnails/29.jpg)
ProvGun/ProvCannon
![Page 30: Spotify Lessons- Learning to Let Go of Machines · Spotify is hiring! spotifyjobs.com IO Tribe. Title: Spotify Lessons- Learning to Let Go of Machines Created Date: 6/27/2017 8:00:20](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d4c5e581cf2aa161c83e/html5/thumbnails/30.jpg)
Feature Developer Concerns
How long?
How to talk to it?
What tools on it?Maintenance? What to put
on it?
Available?
Service + BusinessHow to track? How to get?
Where? Up to date?
How many? Specs?
![Page 31: Spotify Lessons- Learning to Let Go of Machines · Spotify is hiring! spotifyjobs.com IO Tribe. Title: Spotify Lessons- Learning to Let Go of Machines Created Date: 6/27/2017 8:00:20](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d4c5e581cf2aa161c83e/html5/thumbnails/31.jpg)
DNS
![Page 32: Spotify Lessons- Learning to Let Go of Machines · Spotify is hiring! spotifyjobs.com IO Tribe. Title: Spotify Lessons- Learning to Let Go of Machines Created Date: 6/27/2017 8:00:20](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d4c5e581cf2aa161c83e/html5/thumbnails/32.jpg)
Feature Developer Concerns
How long?
How to talk to it?
What tools on it?Maintenance? What to put
on it?
Available?
Service + BusinessHow to track? How to get?
Where? Up to date?
How many? Specs?
![Page 33: Spotify Lessons- Learning to Let Go of Machines · Spotify is hiring! spotifyjobs.com IO Tribe. Title: Spotify Lessons- Learning to Let Go of Machines Created Date: 6/27/2017 8:00:20](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d4c5e581cf2aa161c83e/html5/thumbnails/33.jpg)
Nameless
![Page 34: Spotify Lessons- Learning to Let Go of Machines · Spotify is hiring! spotifyjobs.com IO Tribe. Title: Spotify Lessons- Learning to Let Go of Machines Created Date: 6/27/2017 8:00:20](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d4c5e581cf2aa161c83e/html5/thumbnails/34.jpg)
Feature Developer Concerns
How long?
How to talk to it?
What tools on it?Maintenance? What to put
on it?
Available?
Service + BusinessHow to track? How to get?
Where? Up to date?
How many? Specs?
![Page 35: Spotify Lessons- Learning to Let Go of Machines · Spotify is hiring! spotifyjobs.com IO Tribe. Title: Spotify Lessons- Learning to Let Go of Machines Created Date: 6/27/2017 8:00:20](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d4c5e581cf2aa161c83e/html5/thumbnails/35.jpg)
Cortana
![Page 36: Spotify Lessons- Learning to Let Go of Machines · Spotify is hiring! spotifyjobs.com IO Tribe. Title: Spotify Lessons- Learning to Let Go of Machines Created Date: 6/27/2017 8:00:20](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d4c5e581cf2aa161c83e/html5/thumbnails/36.jpg)
Cortana
![Page 37: Spotify Lessons- Learning to Let Go of Machines · Spotify is hiring! spotifyjobs.com IO Tribe. Title: Spotify Lessons- Learning to Let Go of Machines Created Date: 6/27/2017 8:00:20](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d4c5e581cf2aa161c83e/html5/thumbnails/37.jpg)
Feature Developer Concerns
How long?
How to talk to it?
What tools on it?Maintenance? What to put
on it?
Available?
Service + BusinessHow to track? How to get?
Where? Up to date?
How many? Specs?
![Page 38: Spotify Lessons- Learning to Let Go of Machines · Spotify is hiring! spotifyjobs.com IO Tribe. Title: Spotify Lessons- Learning to Let Go of Machines Created Date: 6/27/2017 8:00:20](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d4c5e581cf2aa161c83e/html5/thumbnails/38.jpg)
Helios and Containers
![Page 39: Spotify Lessons- Learning to Let Go of Machines · Spotify is hiring! spotifyjobs.com IO Tribe. Title: Spotify Lessons- Learning to Let Go of Machines Created Date: 6/27/2017 8:00:20](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d4c5e581cf2aa161c83e/html5/thumbnails/39.jpg)
Feature Developer Concerns
How long?
How to talk to it?
What tools on it?Maintenance? What to put
on it?
Available?
Service + Business
Where? Up to date?
How many? Specs?
How to track? How to get?
![Page 40: Spotify Lessons- Learning to Let Go of Machines · Spotify is hiring! spotifyjobs.com IO Tribe. Title: Spotify Lessons- Learning to Let Go of Machines Created Date: 6/27/2017 8:00:20](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d4c5e581cf2aa161c83e/html5/thumbnails/40.jpg)
Google Compute Platform
![Page 41: Spotify Lessons- Learning to Let Go of Machines · Spotify is hiring! spotifyjobs.com IO Tribe. Title: Spotify Lessons- Learning to Let Go of Machines Created Date: 6/27/2017 8:00:20](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d4c5e581cf2aa161c83e/html5/thumbnails/41.jpg)
ash2-cortana-a1.ash2Zone Service Group Sequential #
gew1-cortana-a-l33t.gew1Zone Service Pool Random 4 Chars
![Page 42: Spotify Lessons- Learning to Let Go of Machines · Spotify is hiring! spotifyjobs.com IO Tribe. Title: Spotify Lessons- Learning to Let Go of Machines Created Date: 6/27/2017 8:00:20](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d4c5e581cf2aa161c83e/html5/thumbnails/42.jpg)
Cortana Pool Manager
![Page 43: Spotify Lessons- Learning to Let Go of Machines · Spotify is hiring! spotifyjobs.com IO Tribe. Title: Spotify Lessons- Learning to Let Go of Machines Created Date: 6/27/2017 8:00:20](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d4c5e581cf2aa161c83e/html5/thumbnails/43.jpg)
Feature Developer Concerns
How long?
How to talk to it?
What tools on it?Maintenance? What to put
on it?
Available?
Service + BusinessHow to track? How to get?
Where? Up to date?
How many? Specs?
![Page 44: Spotify Lessons- Learning to Let Go of Machines · Spotify is hiring! spotifyjobs.com IO Tribe. Title: Spotify Lessons- Learning to Let Go of Machines Created Date: 6/27/2017 8:00:20](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d4c5e581cf2aa161c83e/html5/thumbnails/44.jpg)
Regional Managed Instance Groups
![Page 45: Spotify Lessons- Learning to Let Go of Machines · Spotify is hiring! spotifyjobs.com IO Tribe. Title: Spotify Lessons- Learning to Let Go of Machines Created Date: 6/27/2017 8:00:20](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d4c5e581cf2aa161c83e/html5/thumbnails/45.jpg)
Feature Developer Concerns
How long?
How to talk to it?
What tools on it?Maintenance? What to put
on it?
Available?
Service + BusinessHow to track? How to get?
Where? Up to date?
How many? Specs?
![Page 46: Spotify Lessons- Learning to Let Go of Machines · Spotify is hiring! spotifyjobs.com IO Tribe. Title: Spotify Lessons- Learning to Let Go of Machines Created Date: 6/27/2017 8:00:20](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d4c5e581cf2aa161c83e/html5/thumbnails/46.jpg)
MBMI: Minimal Base Machine Image
![Page 47: Spotify Lessons- Learning to Let Go of Machines · Spotify is hiring! spotifyjobs.com IO Tribe. Title: Spotify Lessons- Learning to Let Go of Machines Created Date: 6/27/2017 8:00:20](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d4c5e581cf2aa161c83e/html5/thumbnails/47.jpg)
Feature Developer Concerns
How to talk to it?
What tools on it?Maintenance? What to put
on it?
Available?
Service + BusinessHow to track? How to get?
Where? How long?Up to date?
How many? Specs?
![Page 48: Spotify Lessons- Learning to Let Go of Machines · Spotify is hiring! spotifyjobs.com IO Tribe. Title: Spotify Lessons- Learning to Let Go of Machines Created Date: 6/27/2017 8:00:20](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d4c5e581cf2aa161c83e/html5/thumbnails/48.jpg)
Phoenix
![Page 49: Spotify Lessons- Learning to Let Go of Machines · Spotify is hiring! spotifyjobs.com IO Tribe. Title: Spotify Lessons- Learning to Let Go of Machines Created Date: 6/27/2017 8:00:20](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d4c5e581cf2aa161c83e/html5/thumbnails/49.jpg)
Feature Developer Concerns
How long?
How to talk to it?
Maintenance?
Available?
Service + BusinessHow to track? How to get?
Where? Up to date?
What tools on it?
What to put on it?
How many? Specs?
![Page 50: Spotify Lessons- Learning to Let Go of Machines · Spotify is hiring! spotifyjobs.com IO Tribe. Title: Spotify Lessons- Learning to Let Go of Machines Created Date: 6/27/2017 8:00:20](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d4c5e581cf2aa161c83e/html5/thumbnails/50.jpg)
Current: Feature Developer’s Context for Service’s Capacity
GCP - europe-west-1
Pool:2 instances x (n1-standard-32)
Stockholm
Pool:4 instances x (High Mem)
![Page 51: Spotify Lessons- Learning to Let Go of Machines · Spotify is hiring! spotifyjobs.com IO Tribe. Title: Spotify Lessons- Learning to Let Go of Machines Created Date: 6/27/2017 8:00:20](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d4c5e581cf2aa161c83e/html5/thumbnails/51.jpg)
Feature Developer Concerns
How long?
How to talk to it?
Maintenance?
Available?
Service + BusinessHow to track? How to get?
Where? Up to date?
What tools on it?
What to put on it?
How many? Specs?
![Page 52: Spotify Lessons- Learning to Let Go of Machines · Spotify is hiring! spotifyjobs.com IO Tribe. Title: Spotify Lessons- Learning to Let Go of Machines Created Date: 6/27/2017 8:00:20](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d4c5e581cf2aa161c83e/html5/thumbnails/52.jpg)
Future
![Page 53: Spotify Lessons- Learning to Let Go of Machines · Spotify is hiring! spotifyjobs.com IO Tribe. Title: Spotify Lessons- Learning to Let Go of Machines Created Date: 6/27/2017 8:00:20](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d4c5e581cf2aa161c83e/html5/thumbnails/53.jpg)
Gordon (Cloud DNS)
![Page 54: Spotify Lessons- Learning to Let Go of Machines · Spotify is hiring! spotifyjobs.com IO Tribe. Title: Spotify Lessons- Learning to Let Go of Machines Created Date: 6/27/2017 8:00:20](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d4c5e581cf2aa161c83e/html5/thumbnails/54.jpg)
Feature Developer Concerns
How long?
How to talk to it?
Maintenance?
Available?
Service + BusinessHow to track? How to get?
Where? Up to date?
What tools on it?
What to put on it?
How many? Specs?
![Page 55: Spotify Lessons- Learning to Let Go of Machines · Spotify is hiring! spotifyjobs.com IO Tribe. Title: Spotify Lessons- Learning to Let Go of Machines Created Date: 6/27/2017 8:00:20](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d4c5e581cf2aa161c83e/html5/thumbnails/55.jpg)
Autoscaling
![Page 56: Spotify Lessons- Learning to Let Go of Machines · Spotify is hiring! spotifyjobs.com IO Tribe. Title: Spotify Lessons- Learning to Let Go of Machines Created Date: 6/27/2017 8:00:20](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d4c5e581cf2aa161c83e/html5/thumbnails/56.jpg)
Feature Developer Concerns
How long?
How to talk to it?
Maintenance?
Available?
Service + BusinessHow to track? How to get?
Where? Up to date?
What tools on it?
What to put on it?
How many? Specs?
![Page 57: Spotify Lessons- Learning to Let Go of Machines · Spotify is hiring! spotifyjobs.com IO Tribe. Title: Spotify Lessons- Learning to Let Go of Machines Created Date: 6/27/2017 8:00:20](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d4c5e581cf2aa161c83e/html5/thumbnails/57.jpg)
Right Sizing
![Page 58: Spotify Lessons- Learning to Let Go of Machines · Spotify is hiring! spotifyjobs.com IO Tribe. Title: Spotify Lessons- Learning to Let Go of Machines Created Date: 6/27/2017 8:00:20](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d4c5e581cf2aa161c83e/html5/thumbnails/58.jpg)
Feature Developer Concerns
How long?
How to talk to it?
Maintenance?
Available?
Service + BusinessHow to track? How to get?
Where? Up to date?
What tools on it?
What to put on it?
How many? Specs?
![Page 59: Spotify Lessons- Learning to Let Go of Machines · Spotify is hiring! spotifyjobs.com IO Tribe. Title: Spotify Lessons- Learning to Let Go of Machines Created Date: 6/27/2017 8:00:20](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d4c5e581cf2aa161c83e/html5/thumbnails/59.jpg)
Future Feature Developer’s Context for Service’s Capacity
GCP - asia-east-1
Service Pool
GCP - europe-west-1
Service Pool
GCP - us-central-1
Service Pool
![Page 60: Spotify Lessons- Learning to Let Go of Machines · Spotify is hiring! spotifyjobs.com IO Tribe. Title: Spotify Lessons- Learning to Let Go of Machines Created Date: 6/27/2017 8:00:20](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d4c5e581cf2aa161c83e/html5/thumbnails/60.jpg)
Feature Developer Concerns
How long?
How to talk to it?
Maintenance?
Available?
Service + BusinessHow to track? How to get?
Where? Up to date?
What tools on it?
What to put on it?
How many? Specs?
![Page 61: Spotify Lessons- Learning to Let Go of Machines · Spotify is hiring! spotifyjobs.com IO Tribe. Title: Spotify Lessons- Learning to Let Go of Machines Created Date: 6/27/2017 8:00:20](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d4c5e581cf2aa161c83e/html5/thumbnails/61.jpg)
Learnings
![Page 62: Spotify Lessons- Learning to Let Go of Machines · Spotify is hiring! spotifyjobs.com IO Tribe. Title: Spotify Lessons- Learning to Let Go of Machines Created Date: 6/27/2017 8:00:20](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d4c5e581cf2aa161c83e/html5/thumbnails/62.jpg)
Why Pets to Cattle was Difficult:- Manual/tedious setup- Wait times for machine becoming ready (packages, DNS)- Non-automatic security updates - A fixed, reliable hostname - SSH Access - Always up/present unless team tears down
![Page 63: Spotify Lessons- Learning to Let Go of Machines · Spotify is hiring! spotifyjobs.com IO Tribe. Title: Spotify Lessons- Learning to Let Go of Machines Created Date: 6/27/2017 8:00:20](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d4c5e581cf2aa161c83e/html5/thumbnails/63.jpg)
- Monitoring- Logging- Service Design - Incidents
Ephemerality Learnings
![Page 64: Spotify Lessons- Learning to Let Go of Machines · Spotify is hiring! spotifyjobs.com IO Tribe. Title: Spotify Lessons- Learning to Let Go of Machines Created Date: 6/27/2017 8:00:20](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d4c5e581cf2aa161c83e/html5/thumbnails/64.jpg)
- Replicate bare metal functionality, then iterate - When in doubt, devs provision up and many - Migration = great time to influence dev paradigms - Don’t need to DIY
Hybrid Learnings
![Page 65: Spotify Lessons- Learning to Let Go of Machines · Spotify is hiring! spotifyjobs.com IO Tribe. Title: Spotify Lessons- Learning to Let Go of Machines Created Date: 6/27/2017 8:00:20](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d4c5e581cf2aa161c83e/html5/thumbnails/65.jpg)
- Feature devs need carrots, sledgehammers, and/or limos to change - Edge Cases: REST API + CLI = provide enough for feature teams to handle the edge cases
DevEx Learnings
![Page 66: Spotify Lessons- Learning to Let Go of Machines · Spotify is hiring! spotifyjobs.com IO Tribe. Title: Spotify Lessons- Learning to Let Go of Machines Created Date: 6/27/2017 8:00:20](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d4c5e581cf2aa161c83e/html5/thumbnails/66.jpg)
- Decrease necessary infrastructure context - Increase reliability - Save $$$ - Increase dev happiness and productivity
Recap
![Page 67: Spotify Lessons- Learning to Let Go of Machines · Spotify is hiring! spotifyjobs.com IO Tribe. Title: Spotify Lessons- Learning to Let Go of Machines Created Date: 6/27/2017 8:00:20](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d4c5e581cf2aa161c83e/html5/thumbnails/67.jpg)
Let’s strategically control and limit how feature developers think about infrastructure.
![Page 68: Spotify Lessons- Learning to Let Go of Machines · Spotify is hiring! spotifyjobs.com IO Tribe. Title: Spotify Lessons- Learning to Let Go of Machines Created Date: 6/27/2017 8:00:20](https://reader033.fdocuments.in/reader033/viewer/2022042307/5ed3d4c5e581cf2aa161c83e/html5/thumbnails/68.jpg)
James WenEmail: [email protected]/Github: @rochesterinnyc LinkedIn: jamesrwenSpotify is hiring! spotifyjobs.com
IO Tribe