CTO at Numberly Gentoo Linux developer · Designing a scalable and distributed application...
Transcript of CTO at Numberly Gentoo Linux developer · Designing a scalable and distributed application...
Gentoo Linux developerCTO at Numberly
@ultrabug
© Eric Fischer / Flickr
Designing a scalable and distributed application
EuroPython 2015
© Eric Fischer / Flickr
USA ⇄ EUROPE
Geo distributed page hit counter web application
© Highways England / Flickr
spans over multiple and distant datacenters
#1Multi Datacenter
© Catherine Holmes
sum of the page hits from all the datacenters
#2A unique page hit count
© Catherine Holmes
no manual application (re)configuration
#3Automatic resizing
© Catherine Holmes
immediately available to all web services
#4Distributed configuration
© Catherine Holmes
...that’s some kind of contract !
Hell...
© C. K. Koay / Flickr
be lazy !
Rely on your stack
© Tripp / Flickr
Zen of pythonif the technology is hard to explain,
it's a bad idea
© HumanOS
UNIX tools philosophykeep it small and simple
© Jason Scott / Flickr
Isolated componentsbreaking it down to pieces
© Blake Thornberry / Flickr
distributed architecture style
Microservices
© Martin Grandjean / wikipedia
micromanagement = high operational cost
Microservices
© Martin Grandjean / wikipedia
Communication reliabilitynetwork latency and failure
© Steven Shorrock / Flickr
Locality vs consistencydata access lag or outage
© R2 HOX / Flickr
feature rich application stack
nginx + uWSGI
© Terabass / wikipedia
⇨ gets HTTP requests ⇨ produces jobs
⇦ returns the total hit count
collector
consumes jobs ⇨ increments a counter
processor
simple and reliable job queue server
beanstalkd
© Mooms
automated scaling and fault tolerance
Service discovery
© Cambridge University / wikipedia
multi datacenter + key / value storage
Consul
seamless service registration / health check
uWSGI consul plugin
consul-register = url=http://localhost:8500,name=my_service
to victory
Let’s build this up
© Brian Rincker / Flickr
(hopefully)
Live demo !
© shoutmeloud / Pinterest
source code : github.com/ultrabug/ep2015@ultrabug
Thanks
© Eric Fischer / Flickr