BUILD

BAKE

DEPLOY

Continuous Delivery at Netflix: From Code to the Monkeys

Dianne MarshQCon San Francisco, November 2013http://www.linkedin.com/in/diannemarsh

BUILD

BAKE

DEPLOY

Netflix Goals

High Availability

But … Move fast

Tools encourage Best Practices

But … Freedom to do the right thing

Teams Deploy Their Own Code

Run What You Wrote Rapid Innovation Rapid Detection Rapid Response

= Freedom + Responsibility

http://www.slideshare.net/garethbowles/self-servicebuilddeploymentagile2013

BUILD

Jenkins Job DSLConfiguration as CodeGroovy Script Scripts go in Version Control

http://www.slideshare.net/quidryan/configuration-as-code

BUILD BAKE

BAKE

Aminator• Create AMI from Base AMI• Image contains service and

everything needed to run it• Unit of Deployment for Test and

Prod• Abstracts Cloud Details

http://techblog.netflix.com/2013/03/ami-creation-with-aminator.html

BUILD BAKE DEPLOY

DEPLOY

Asgard: AWS Deployment ToolDeploys Netflix to the CloudRed/Black push

http://www.infoq.com/presentations/asgard

CANARY ANALYSIS

Test, Int, ProdChoose where to deployRun canary analysisScale up new instancesTurn on traffic to new ASGTurn off traffic to old ASGWait … analyze … continue

Asgard Developer Portal

GLISTEN

Extending Asgard’s WorkflowAutomated Red/Black Push

Test, Int, Prod stacksRun canary/analysisScale up new instancesTurn on traffic to new ASGRun more testsTurn off traffic to old ASGWait … analyze … continue

http://techblog.netflix.com/2013/09/glisten-groovy-way-to-use-amazons.html

BUILD

BAKE

DEPLOY

Simian Army• Chaos Monkey• Latency Monkey• Janitor Monkey• Conformity Monkey• (and more!)

Test resiliency at runtime

http://www.infoq.com/presentations/netflix-resiliency-failure-cloud

One Button Deployment?

Regional Isolation

Limit Impact of Human Error Stagger deployments Canary testing per

region

Multi-Region Consistency

Build Tooling to: Schedule

Deployments Prefer off peak Choose next available

region automatically Provide high visibility

per region

Send in the Conformity Monkey

http://techblog.netflix.com/2013/05/conformity-monkey-keeping-your-cloud.html

Have deployments diverged? Balance regional consistency

with regional isolation Provide meaningful

thresholds Build best practices into

tooling and reporting

Clean up with Janitor Monkey

Disassociate unused EIPs

Delete unassociated Amazon EBS volumes

Delete older Amazon EBS snapshots

Leverage Amazon S3 Object Expiration

https://github.com/Netflix/SimianArmy/wiki/Janitor-Home

Key Elements for Netflix

Value Self-service

Test Everywhere

Build Awareness of Multiple Regions

Avoid peak times

Roll back quickly and easily

Be Cloud Native

Put NetflixOSS to Work for You

Netflix Platform

AMINATOR

Glisten

** And 30+ more projects at http://netflix.github.io/

Keep the Conversation Going

Continuous Delivery Open Space

Ballroom B/C (here!)

1:35-2:25, immediately following lunch

Thanks!

We’re always hiring!Dianne Marsh (@dmarsh)

dmarsh@netflix.com