Continuous Deployment @ AWS Re:Invent

119
© 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc. Continuous Integration and Deployment Best Practices on AWS - ARC307 Leo Zhadanovsky, Senior Solutions Architect, AWS @leozh JP Schneider, DevOps / Internet Jedi, Mozilla Foundation @jdotp November 13 th , 2013

description

Leo Z and I gave a talk this year at AWS Re:Invent on Continuous Integration at OFA, Mozilla Foundation, and other companies.

Transcript of Continuous Deployment @ AWS Re:Invent

Page 1: Continuous Deployment @ AWS Re:Invent

© 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.

Continuous Integration and Deployment Best Practices on AWS - ARC307 Leo Zhadanovsky, Senior Solutions Architect, AWS @leozh JP Schneider, DevOps / Internet Jedi, Mozilla Foundation @jdotp November 13th, 2013

Page 2: Continuous Deployment @ AWS Re:Invent

Who Am I?

I work for AWS I worked for the DNC 2009-2012

I was embedded in the DevOps Team @ OFA

AWS does not endorse

political candidates

Page 3: Continuous Deployment @ AWS Re:Invent

Who Am I?

JP, DevOps for Mozilla Foundation Previous gig DevOps at OFA 2012

Before that, Ops at Threadless @jdotp

Mozilla Foundation does

endorse animated cats

Page 4: Continuous Deployment @ AWS Re:Invent

CONTINUOUS INTEGRATION

Page 5: Continuous Deployment @ AWS Re:Invent

What is Continuous Integration?

Changes to code automatically deployed to mainline branch • After passing unit and mock tests

Makes changes to code and deployments iterative, not monolithic Bugs are detected quickly Allows rapid development Helps automate deployments

Page 6: Continuous Deployment @ AWS Re:Invent

DEVELOPER

Page 7: Continuous Deployment @ AWS Re:Invent

SOURCE CODE REPOSITORY

Page 8: Continuous Deployment @ AWS Re:Invent

SOURCE CODE REPOSITORY

PROJECT MANAGEMENT SERVER

Page 9: Continuous Deployment @ AWS Re:Invent

SOURCE CODE REPOSITORY

PROJECT MANAGEMENT SERVER

CONTINUOUS INTEGRATION SERVER

Page 10: Continuous Deployment @ AWS Re:Invent

SOURCE CODE REPOSITORY

PROJECT MANAGEMENT SERVER

CONTINUOUS INTEGRATION SERVER

PICK TASKS

Page 11: Continuous Deployment @ AWS Re:Invent

SOURCE CODE REPOSITORY

PROJECT MANAGEMENT SERVER

CONTINUOUS INTEGRATION SERVER

SUBMIT CODE

Page 12: Continuous Deployment @ AWS Re:Invent

SOURCE CODE REPOSITORY

PROJECT MANAGEMENT SERVER

CONTINUOUS INTEGRATION SERVER

SCHEDULE BUILD

Page 13: Continuous Deployment @ AWS Re:Invent

SOURCE CODE REPOSITORY

PROJECT MANAGEMENT SERVER

CONTINUOUS INTEGRATION SERVER

RECURRENT BUILDS

Page 14: Continuous Deployment @ AWS Re:Invent

SOURCE CODE REPOSITORY

PROJECT MANAGEMENT SERVER

CONTINUOUS INTEGRATION SERVER

CODE FETCH

Page 15: Continuous Deployment @ AWS Re:Invent

SOURCE CODE REPOSITORY

PROJECT MANAGEMENT SERVER

CONTINUOUS INTEGRATION SERVER

CODE QUALITY TESTS

TEST RESULTS

Page 16: Continuous Deployment @ AWS Re:Invent

SOURCE CODE REPOSITORY

PROJECT MANAGEMENT SERVER

CONTINUOUS INTEGRATION SERVER

BUILD OUTPUT

Page 17: Continuous Deployment @ AWS Re:Invent

SOURCE CODE REPOSITORY

PROJECT MANAGEMENT SERVER

CONTINUOUS INTEGRATION SERVER

DOCS

BINARIES & PACKAGES

DEV FACING NOTIFICATIONS

Page 18: Continuous Deployment @ AWS Re:Invent
Page 19: Continuous Deployment @ AWS Re:Invent

SOURCE CODE REPOSITORY

DNS

CONTINUOUS INTEGRATION SERVER

PROJECT MANAGEMENT SERVER

BUILDS

Page 20: Continuous Deployment @ AWS Re:Invent

PAIN POINTS • UNIT TESTS INCOMPLETE • MOCK TESTS MAINTENANCE • EXPENSIVE TEST ENVIRONMENT • TEST ENVIRONMENT ≠ PRODUCTION • DEPLOYMENT CYCLES

Page 21: Continuous Deployment @ AWS Re:Invent

ON-DEMAND

PAY AS YOU GO

ELASTIC

Page 22: Continuous Deployment @ AWS Re:Invent

= PROGRAMMABLE

PLATFORM

Page 23: Continuous Deployment @ AWS Re:Invent
Page 24: Continuous Deployment @ AWS Re:Invent
Page 25: Continuous Deployment @ AWS Re:Invent
Page 26: Continuous Deployment @ AWS Re:Invent

IF YOU CAN PROGRAM IT YOU CAN AUTOMATE IT

Page 27: Continuous Deployment @ AWS Re:Invent

A lot of options…

Configuration Management Systems • Puppet • Chef • Saltstack

Deployment Frameworks • AWS Elastic Beanstalk • AWS OpsWorks • Ansible • Fabric • Capistrano

Infrastructure Management • AWS CloudFormation

Page 28: Continuous Deployment @ AWS Re:Invent

Bake an AMI Configure dynamically

Time consuming configuration (startup time)

Static configurations (less change management)

Bootstrapping

Page 29: Continuous Deployment @ AWS Re:Invent

Bake an AMI Configure dynamically

Continuous deployment (latest code)

Environment specific (dev-test-prod)

Bootstrapping

Page 30: Continuous Deployment @ AWS Re:Invent

Obama for America

awsofa.info

Page 31: Continuous Deployment @ AWS Re:Invent

Web-Scale Applications

Page 32: Continuous Deployment @ AWS Re:Invent

500k+ IOPS DB Systems

Page 33: Continuous Deployment @ AWS Re:Invent

Services API

Page 34: Continuous Deployment @ AWS Re:Invent
Page 35: Continuous Deployment @ AWS Re:Invent

Typical Charts

Page 36: Continuous Deployment @ AWS Re:Invent
Page 37: Continuous Deployment @ AWS Re:Invent

How?

Page 38: Continuous Deployment @ AWS Re:Invent

Ingredients Ubuntu nginx boundary Unity jQuery SQLServer hbase NewRelic EC2 node.js Cybersource hive ElasticSearch Ruby Twilio EE S3 ELB boto Magento PHP EMR SES Route53 SimpleDB Campfire nagios Paypal CentOS CloudSearch levelDB mongoDB python securitygroups Usahidhi PostgresSQL Github apache bootstrap SNS OpsView Jekyll RoR EBS FPS VPC Mashery Vertica RDS Optimizely MySQL puppet tsunamiUDP R asgard cloudwatch ElastiCache cloudopt SQS cloudinit DirectConnect BSD rsync STS Objective-C DynamoDB

Page 39: Continuous Deployment @ AWS Re:Invent

Infrastructure, Configuration Management & Monitoring

Ubuntu nginx boundary Unity jQuery SQLServer hbase NewRelic EC2 node.js Cybersource hive ElasticSearch Ruby Twilio EE S3 ELB boto Magento PHP EMR SES Route53 SimpleDB Campfire nagios Paypal CentOS CloudSearch levelDB mongoDB python securitygroups Usahidhi PostgresSQL Github apache bootstrap SNS OpsView Jekyll RoR EBS FPS VPC Mashery Vertica RDS Optimizely MySQL puppet tsunamiUDP R asgard cloudwatch ElastiCache cloudopt SQS cloudinit DirectConnect BSD rsync STS Objective-C DynamoDB

Page 40: Continuous Deployment @ AWS Re:Invent

Configuration Management: Puppet

In mid-2011, we looked at options for configuration management and chose Puppet We needed to make it scale, and to get it to work with state-less, horizontally scalable infrastructure How did we do this?

Page 41: Continuous Deployment @ AWS Re:Invent

Bootstrapping Puppet with CloudInit

CloudInit is built into Ubuntu and Amazon Linux • Allows you to

pass bootstrap parameters in Amazon EC2 user-data field, in YAML format

Page 42: Continuous Deployment @ AWS Re:Invent

Bootstrapping Puppet with CloudInit

Don’t store creds in puppet manifests, store them in private Amazon S3 buckets Either pass Amazon S3 creds through CloudInit: Even better – avoid this by using AWS Identity and Access Management (IAM) roles and AWS Unified CLI’s S3 client

Page 43: Continuous Deployment @ AWS Re:Invent

Bootstrapping Puppet with CloudInit Built-in puppet support

Use certname with %i for instance id to name the node Puppetmaster must have auto sign turned on • Use security groups and/or NACLs for network-level security

In nodes.pp, use regex to match node names

Page 44: Continuous Deployment @ AWS Re:Invent

Puppet Tips

Use a base class to define your standard install

Page 45: Continuous Deployment @ AWS Re:Invent

Use runstages Don’t store credentials in puppet, store them in private Amazon S3 buckets • Use AWS IAM to secure the credentials bucket/folders within that bucket

Puppet Tips

Page 46: Continuous Deployment @ AWS Re:Invent

Puppet Tips

Use puppet only for configuration files and what makes your apps unique For undifferentiated parts of apps, use Amazon S3 backed RPM/Debian repositories • Can be either public or private repos, depending on your needs

• Amazon S3 Private RPM Repos: http://git.io/YAcsbg • Amazon S3 Private Debian Repos: http://git.io/ecCjWQ

Page 47: Continuous Deployment @ AWS Re:Invent

Puppet Tips

By using packages for applications deploys, you can set ensure => latest, and just bump the package in the repo to update Log everything with rsyslog/graylog/loggly/NewRelic/splunk

Page 48: Continuous Deployment @ AWS Re:Invent

Scaling the Puppet Masters

Use an Auto Scaling group for puppet masters • Min size => 2, use multiple Availability Zones

Either have them build themselves off of existing puppet masters in the group or off packages stored in Amazon S3 and bootstrapped through user-data Auto-sign must be on

Page 49: Continuous Deployment @ AWS Re:Invent

One thing that is difficult to prepare for…

Page 50: Continuous Deployment @ AWS Re:Invent

They had this built for the previous 3 months, a on the East Coast.

Page 51: Continuous Deployment @ AWS Re:Invent

They had this built for the previous 3 months, a on the East Coast.

We built this part in 9 hours to be safe.

AWS + Puppet +

Netflix Asgard + WAN Optimization Software +

DevOps =

Cross-Continent Fault-Tolerance On-Demand

Page 52: Continuous Deployment @ AWS Re:Invent

ARC205 Thursday @ 3:00PM Lando 4303 Deploying the ‘League of Legends’ Data

Pipeline with Chef

Page 53: Continuous Deployment @ AWS Re:Invent

Mozilla Foundation

Page 54: Continuous Deployment @ AWS Re:Invent

Webmaker.org circa 2012 • Included Apps, non-SOA: Thimble, Popcorn, Goggles • ~20 pushes of new software in 2012 • Operations and Development interacted mostly through bugzilla

tickets for deploys • Hosting in physical datacenter at Mozilla

Webmaker.org circa early 2013 • Deciding to go 12-factor, SOA in app layer • Weekly pushes of Popcorn on train model • Operations and Development interacted mostly through bugzilla

tickets for deploys

Page 55: Continuous Deployment @ AWS Re:Invent

April 2013

Webmaker begins rebuilding entire platform SOA, 12-factor in node.js exclusively Moving apps into AWS and DevOps / CI

Page 56: Continuous Deployment @ AWS Re:Invent

Since April 2013…. Openbadges, Webmaker combine for: 1339 Pushes

Pushes Per Day to Staging / Prod

Pushes Per Day (Staging and Prod)

Page 57: Continuous Deployment @ AWS Re:Invent

Who? • ~30 Paid Developers • Hundreds of Students • Thousands of Contributors • One DevOps / Internet Jedi • Multiple Teams

How? • Puppet, Jenkins, Fabric • Tight feedback loops:

Newrelic, Opsview • Culture Shift

• Staging Envs • Brave devs iterate,

keeping work in-context • Visible Ops • Cross-train developers in

operations

Page 58: Continuous Deployment @ AWS Re:Invent

What changed?

Page 59: Continuous Deployment @ AWS Re:Invent

1) Know How You Were Doing Before

2) Know What Changed When

3) Know How You Are Now Doing

= The confidence to try more things and try them faster, with minimum viable planning.

Page 60: Continuous Deployment @ AWS Re:Invent

Deployment

Pipeline

Page 61: Continuous Deployment @ AWS Re:Invent

ARC312 Friday @ 1:30PM Lando 4206 SmugMug’s Zero-Downtime Migration to

AWS

Page 62: Continuous Deployment @ AWS Re:Invent

AWS CLOUDFORMATION

STACK-BASED DEPLOYMENT SERVICE

Page 63: Continuous Deployment @ AWS Re:Invent

AWS CLOUDFORMATION TEMPLATE

Page 64: Continuous Deployment @ AWS Re:Invent
Page 65: Continuous Deployment @ AWS Re:Invent

{ "Description" : "Create RDS with username and password", "Resources" : { "MyDB" : { "Type" : "AWS::RDS::DBInstance", "Properties" : { "AllocatedStorage" : "500", "DBInstanceClass" : "db.m1.small", "Engine" : "MySQL", "EngineVersion" : "5.5", "MasterUsername" : "MyName", "MasterUserPassword" : "MyPassword" } } } }

Page 66: Continuous Deployment @ AWS Re:Invent

"AWS::CloudFormation::Init" : { "config" : { "packages" : { "yum" : { "mysql" : [], "mysql-server" : [], "httpd" : [], "php" : [], "php-mysql" : [] } }, "sources" : { "/var/www/html" : "https://s3.amazonaws.com/my-builds/build-v4.zip" } }

Page 67: Continuous Deployment @ AWS Re:Invent

{ "Parameters" : { "KeyName" : { "Description" : "Name of an existing EC2 KeyPair to enable SSH access to the instance", "Type" : "String" } }, }

Page 68: Continuous Deployment @ AWS Re:Invent

CLOUDFORMATION TEMPLATE

PROCEDURAL DEFINITION

Create it programmatically

KNOWN CONFIGURATION Store stack configuration in source control

PARAMETER DRIVEN

Dynamic and user-driven templates

COLLABORATION Share templates with ease as just files

Page 69: Continuous Deployment @ AWS Re:Invent
Page 70: Continuous Deployment @ AWS Re:Invent

APPLICATION VERSIONS

+ INFRASTRUCTURE

VERSIONS

Page 71: Continuous Deployment @ AWS Re:Invent

AWS CLOUDFORMATION

TEMPLATE

Page 72: Continuous Deployment @ AWS Re:Invent

ARC203 Wednesday @ 4:15 Lando 4303 How Adobe Deploys

Refreshing the Entire Stack Every Time

Page 73: Continuous Deployment @ AWS Re:Invent
Page 74: Continuous Deployment @ AWS Re:Invent

Mars Rover Landing by the #’s

NASA TV = HD stream, 1080p, ~1 mb/s per viewer Expecting peak of ~1m viewers All playback devices (iOS, Android, Flash, HTML5, blah) Once in a lifetime moment in history (no crashy crashy)

FUN FUN FUN

Page 75: Continuous Deployment @ AWS Re:Invent

Mars Rover Landing by the #’s

NASA TV = HD stream, 1080p, ~1 mb/s per viewer Expecting peak of ~1m viewers All playback devices (iOS, Android, Flash, HTML5, blah) Once in a lifetime moment in history (no crashy crashy) NASA says we can’t use their live stream setup It’s 6 days before the landing It’s the same week as the Olympics Available technical resources from JPL: Brett and Khawaja

NO

T FUN

NO

T FUN

NO

T FUN

Page 76: Continuous Deployment @ AWS Re:Invent

The sticky wicket:

HD Video stream size = 4mb never changes easy to cache

manifest.f4m size = 4kb New every 4 sec caching difficult

Page 77: Continuous Deployment @ AWS Re:Invent

The plan

Design a solution around our limits • Max connections to origin = 6 • Max streams per cache node = 20 • Local Latency = critical • US-WEST-1 capacity reserved for S3 static images

Test the snot out of it Hang on!

Page 78: Continuous Deployment @ AWS Re:Invent
Page 79: Continuous Deployment @ AWS Re:Invent

ok, so one of them is a rocket scientist..

Page 80: Continuous Deployment @ AWS Re:Invent

“The 42 pack”

Page 81: Continuous Deployment @ AWS Re:Invent

LOAD TESTING

Page 82: Continuous Deployment @ AWS Re:Invent
Page 83: Continuous Deployment @ AWS Re:Invent
Page 84: Continuous Deployment @ AWS Re:Invent

Benchmarking

Page 85: Continuous Deployment @ AWS Re:Invent

US-East Cache Node Performance 25.3 Gbps

Page 86: Continuous Deployment @ AWS Re:Invent

Only ~42Mbps Impact on US-East FMS Origin Servers

Page 87: Continuous Deployment @ AWS Re:Invent
Page 88: Continuous Deployment @ AWS Re:Invent

ARC303 Friday @ 1:30PM Delfino 4003 Unmeltable Infrastructure at Scale:

Using Apache Kafka, Twitter Storm and Elastic Search on AWS

Page 89: Continuous Deployment @ AWS Re:Invent

CONTINUOUS DEPLOYMENT

SMALL, FREQUENT CHANGES CONSTANTLY INTEGRATING INTO

PRODUCTION

Page 90: Continuous Deployment @ AWS Re:Invent

KEY = ITERATION

Page 91: Continuous Deployment @ AWS Re:Invent

ITERATION =

MODIFY THE SYSTEM TO BETTER MEET THE EXPECTATIONS OF

YOUR USERS

Page 92: Continuous Deployment @ AWS Re:Invent

11.6s

Mean time between

deployments (weekday)

1,079

Max number of deployments in a

single hour

10,000

Mean number of hosts

simultaneously receiving a deployment

30,000

Max number of hosts

simultaneously receiving a deployment

DEPLOYMENTS AT AMAZON.COM

Page 93: Continuous Deployment @ AWS Re:Invent

SOFTWARE DEPLOY

≠ PRODUCT LAUNCH

Page 94: Continuous Deployment @ AWS Re:Invent
Page 95: Continuous Deployment @ AWS Re:Invent

1.5 BILLION PAGE VIEWS

OCTOBER 2012

$83 MILLION IN TRANSACTIONS 4.2 MILLION ITEMS SOLD

Page 96: Continuous Deployment @ AWS Re:Invent

30 DEPLOYS PER DAY 1 DEPLOY EVERY 20 MINUTES

Page 97: Continuous Deployment @ AWS Re:Invent

"Production is truly the only place you can validate your code."

Page 98: Continuous Deployment @ AWS Re:Invent
Page 99: Continuous Deployment @ AWS Re:Invent
Page 100: Continuous Deployment @ AWS Re:Invent
Page 101: Continuous Deployment @ AWS Re:Invent
Page 102: Continuous Deployment @ AWS Re:Invent

AWS OPSWORKS INTEGRATED APPLICATION

MANAGEMENT

DMG 304 Thursday @ 3:00PM Murano 3206 AWS OPSWORKS UNDER THE HOOD

Page 103: Continuous Deployment @ AWS Re:Invent
Page 104: Continuous Deployment @ AWS Re:Invent

14 BILLION REQUESTS/MONTH 50 000 DATABASE UPDATES / SEC

NO CACHE

Page 105: Continuous Deployment @ AWS Re:Invent

DATA-DRIVEN ARCHITECTURES

Page 106: Continuous Deployment @ AWS Re:Invent

METRICS @ETSY

Page 107: Continuous Deployment @ AWS Re:Invent

METRICS @OBAMA FOR AMERICA

Page 108: Continuous Deployment @ AWS Re:Invent

COST-ORIENTED ARCHITECTURES

Page 109: Continuous Deployment @ AWS Re:Invent

PHP+APACHE+VARNISH

NGINX+NODEJS

Page 110: Continuous Deployment @ AWS Re:Invent
Page 111: Continuous Deployment @ AWS Re:Invent

CONTINUOUS INTEGRATION

CONTINUOUS DEPLOYMENT

Page 112: Continuous Deployment @ AWS Re:Invent

CONTINUOUS DEPLOYMENT

= CONTINUOUS

EXPERIMENTATION

Page 113: Continuous Deployment @ AWS Re:Invent

CONTINUOUS DEPLOYMENT

= CONTINUOUS

IMPROVEMENT

Page 114: Continuous Deployment @ AWS Re:Invent

INNOVATE

Page 115: Continuous Deployment @ AWS Re:Invent

« Want to increase innovation? Lower the cost of failure »

Joi Ito

Page 116: Continuous Deployment @ AWS Re:Invent

SPEED AND AGILITY

Experiment Often

Fail quickly at

a low cost

More Innovation

Experiment Infrequently

Failure is expensive

Less

Innovation

“ON-PREMISES”

Page 117: Continuous Deployment @ AWS Re:Invent

AWS re:Invent Pub Crawl

Join the AWS Startup Team this evening at the AWS Pub Crawl When: Wednesday November 13, 5:30pm - 7:30pm Where: Canaletto at The Venetian, 2nd Floor Who Will Be There: Startups, The AWS Startup Team, Startup Launch Companies and AWS re:Invent Hackathon winners

Page 118: Continuous Deployment @ AWS Re:Invent

Startup Spotlight Sessions with Dr. Werner Vogels Thurs. Nov 14, Marcello Room 4406

SPOT 203 - Fireside Chats – Startup Founders, 1:30-2:30pm – Eliot Horowitz, CTO of MongoDB – Jeff Lawson, CEO of Twilio – Valentino Volonghi, Chief Architect of AdRoll

SPOT 204 - Fireside Chats – Startup Influencers, 3:00-4:00pm – Albert Wegner, Managing Partner at Union Square Ventures – David Cohen, Founder and CEO of TechStars

SPOT 101 - Startup Launches, 4:15-5:15pm – 5 companies powered by AWS launching at AWS re:Invent 2013

Page 119: Continuous Deployment @ AWS Re:Invent

Please give us your feedback on this presentation

As a thank you, we will select prize winners daily for completed surveys!

ARC 307