No free lunch, indeed:Three years

Phil CalçadoSoundCloud

of microservicesat SoundCloud

No free lunch, indeed:Three years

Phil CalçadoSoundCloud

of microservicesat SoundCloud

> 11 hours of audio uploaded every minute

~ 300 million people every month

heaps have been written about microservices in

the past year-ish

tl;dr

• Rapid provisioning• Basic Monitoring• Rapid application deployment

These make sense

• Rapid provisioning• Basic Monitoring• Rapid application deployment

this makes sense

why does it make me so nervous?

the SoundCloud story you might know

we moved to microservices

because $reasons

http://bit.ly/how_we_ended_up_with_microservices

the pre-history

SoundCloud, circa 2011

let’s prepare for the “microservices

explosion"

#1 provisioning

what was cool in 2010-11

what was cool in 2010-11

doozer lxc 12factor.net

much better than anything else at the

time

a problem

no resource limits (i.e. cgroups)+

naïve scheduling=

loud neighbour in your own datacentre

a problemmade for

most of our services migrated to

the problem

time

start workon v1

v1 100% deployed

start work on v2

attempt #1 - before we go sophisticated, let’s simplify what we have

warmed up pool

machineintake

just a bit of glue code

just a bit of glue codea LOT

machineintake

#2 telemetry

state of telemetry tools circa 2011-12 wasn’t

great

StatsD

let’s build our own!

but that’s not what broke…

obvious with a monolith

monolith

not so much now

?

?

?

standardise dashboards

standardise operations

https://twitter.github.io/twitter-server/Features.html#http-admin-interface

visualise

#3 deployment

> git

SquashFS

> make

unit tests integration tests

acceptance tests perf tests

> make

/dev/null

we ended up with 7 different deployment

scripts

> make > gitunit tests integration tests

unit tests integration tests

acceptance tests perf tests

containers let you spawn your mini-

SoundCloud

but why was I so nervous?

because wemessed up

there are simple and incremental ways to address these

• Rapid provisioning• Basic Monitoring• Rapid application deployment

“wtf? do you think Netflix got it right the first time?”

— @adrianco

some of the good things

Q&A