CyansibleBlending Blue/Green Deployments

@Betterment

Techcrunch DisruptMay 2010

90,000 customers, more every minute.

Fastest growing automated investing service

$2B+

Who are we?

betterment :

investing::

devops : engineering

it’s 2012.let’s ship some

code.

Betterment@2012

A Better Migration: From Snowflakes to StormtroopersWednesday, July 22, 6:30p - 8:00p @ AWS Pop-up Loft | NYC

Shameless Plug:

it’s 2015.

let’s ship new code…without interrupting

production

●Predictable●Repeatable●Minimal Human

Interaction●Zero User Interruption●Contained Failure

Dream Delivery

Blue/Green Deployments

http://martinfowler.com/bliki/BlueGreenDeployment.html

DNS

ELB

Pre-flight Checklist

❏Stateless Servers❏Server Health Check❏Duplicate Full Stack

including RDS Replica

Wait. Two Databases?“There's still the issue of dealing with missed transactions while the green environment was live, but depending on your design you may be able to...● feed transactions to both environments in such a

way as to keep the blue environment as a backup when the green is live. Or you may be able to...

● put the application in read-only mode before cut-over, run it for a while in read-only mode, and then switch it to read-write mode.”

http://martinfowler.com/bliki/BlueGreenDeployment.html

“Code𝓲 always works on Schema +1𝓲 ”

(A.K.A. old code works on new schema)

Publish, Migrate, Deploy

Jenkins’ Job

1.Build2.Test3.Package4.Publish5.Run Migrations6.Invoke Ansible7.Cull Zombies

Ansible’s Job

1.Check for S3 deliverables2.Spin up new EC2 Instance(s)3.Apply role(s) to instance(s)4.Find instance(s) in ELB5.Add new instance(s) to ELB &

tago status: in-use

6.Remove & tag instanceso status: zombie

Bootstrapping Ansible

./exec/ directorycontains Jenkins entry

points

Ansible code lives in the repo it provisions. 

Publish it like an app.

./exec/brochure-deploy.sh

brochure-deploy.yml

you are here.

00:00:10.888

provision-new-ec2-instances

brochure-deploy >

identify-elb-AZs

brochure-deploy > provision-new-ec2-instances >

ec2-deploy

brochure-deploy > provision-new-ec2-instances >

AMI Hierarchy

brochure-deploy.yml

you are here.

00:01:18.415

configure-instancesbrochure-deploy >

roles/deploy/brochure/tasks/main.yml

brochure-deploy > configure-instances

brochure-deploy.yml

you are here.

00:04:49.414

brochure-deploy.yml

you are here.

00:04:50.188

find-instances-in-elb

brochure-deploy >

brochure-deploy.yml

you are here.

00:04:50.911

add-instances-to-elb

brochure-deploy >

brochure-deploy.yml

you are here.

00:05:06.295

decommission-instances

brochure-deploy >

brochure-deploy.yml

you are here.

00:05:21.226

EC2 INSTANCE

●Predictable●Repeatable●Minimal Human

Interaction●Zero User Interruption●Contained Failure

Dream Delivery Achieved

The Future

● Long Running Instances + Dockero Huge speed improvement

● Post Monolith, Abandon Jenkins?o Travis CI for Build/Testo Tower for Deployment Orchestration

● Ansible Galaxy?

Questions?

alan@betterment.com@nonrational

github.com/nonrational

careers@betterment.com

All code snippets & diagrams contained in this presentation are property of Betterment, but please learn from them.

All photographs / GIFs used in this presentation are someone else’s.Street Fighter, Back To The Future, Indiana Jones, Futurama, and Arrested Development are someone

else’s property too.