Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013

© 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.

Continuous Integration and Deployment Best Practices on AWS - ARC307 Leo Zhadanovsky, Senior Solutions Architect, AWS @leozh JP Schneider, DevOps / Internet Jedi, Mozilla Foundation @jdotp November 13th, 2013

Who Am I?

I work for AWS I worked for the DNC 2009-2012

I was embedded in the DevOps Team @ OFA

AWS does not endorse

political candidates

Who Am I?

JP, DevOps for Mozilla Foundation Previous gig DevOps at OFA 2012

Before that, Ops at Threadless @jdotp

Mozilla Foundation does

endorse animated cats

CONTINUOUS INTEGRATION

What is Continuous Integration?

Changes to code automatically deployed to mainline branch • After passing unit and mock tests

Makes changes to code and deployments iterative, not monolithic Bugs are detected quickly Allows rapid development Helps automate deployments

DEVELOPER

SOURCE CODE REPOSITORY


PROJECT MANAGEMENT SERVER



CONTINUOUS INTEGRATION SERVER




PICK TASKS




SUBMIT CODE




SCHEDULE BUILD




RECURRENT BUILDS




CODE FETCH




CODE QUALITY TESTS

TEST RESULTS




BUILD OUTPUT




DOCS

BINARIES & PACKAGES

DEV FACING NOTIFICATIONS


DNS



BUILDS

PAIN POINTS • UNIT TESTS INCOMPLETE • MOCK TESTS MAINTENANCE • EXPENSIVE TEST ENVIRONMENT • TEST ENVIRONMENT ≠ PRODUCTION • DEPLOYMENT CYCLES

ON-DEMAND

PAY AS YOU GO

ELASTIC

= PROGRAMMABLE

PLATFORM

IF YOU CAN PROGRAM IT YOU CAN AUTOMATE IT

A lot of options…

Configuration Management Systems • Puppet • Chef • Saltstack

Deployment Frameworks • AWS Elastic Beanstalk • AWS OpsWorks • Ansible • Fabric • Capistrano

Infrastructure Management • AWS CloudFormation

Bake an AMI Configure dynamically

Time consuming configuration (startup time)

Static configurations (less change management)

Bootstrapping

Bake an AMI Configure dynamically

Continuous deployment (latest code)

Environment specific (dev-test-prod)

Bootstrapping

Obama for America

awsofa.info

Web-Scale Applications

500k+ IOPS DB Systems

Services API

Typical Charts

Ingredients Ubuntu nginx boundary Unity jQuery SQLServer hbase NewRelic EC2 node.js Cybersource hive ElasticSearch Ruby Twilio EE S3 ELB boto Magento PHP EMR SES Route53 SimpleDB Campfire nagios Paypal CentOS CloudSearch levelDB mongoDB python securitygroups Usahidhi PostgresSQL Github apache bootstrap SNS OpsView Jekyll RoR EBS FPS VPC Mashery Vertica RDS Optimizely MySQL puppet tsunamiUDP R asgard cloudwatch ElastiCache cloudopt SQS cloudinit DirectConnect BSD rsync STS Objective-C DynamoDB

Infrastructure, Configuration Management & Monitoring

Ubuntu nginx boundary Unity jQuery SQLServer hbase NewRelic EC2 node.js Cybersource hive ElasticSearch Ruby Twilio EE S3 ELB boto Magento PHP EMR SES Route53 SimpleDB Campfire nagios Paypal CentOS CloudSearch levelDB mongoDB python securitygroups Usahidhi PostgresSQL Github apache bootstrap SNS OpsView Jekyll RoR EBS FPS VPC Mashery Vertica RDS Optimizely MySQL puppet tsunamiUDP R asgard cloudwatch ElastiCache cloudopt SQS cloudinit DirectConnect BSD rsync STS Objective-C DynamoDB

Configuration Management: Puppet

In mid-2011, we looked at options for configuration management and chose Puppet We needed to make it scale, and to get it to work with state-less, horizontally scalable infrastructure How did we do this?

Bootstrapping Puppet with CloudInit

CloudInit is built into Ubuntu and Amazon Linux • Allows you to

pass bootstrap parameters in Amazon EC2 user-data field, in YAML format

Bootstrapping Puppet with CloudInit

Don’t store creds in puppet manifests, store them in private Amazon S3 buckets Either pass Amazon S3 creds through CloudInit: Even better – avoid this by using AWS Identity and Access Management (IAM) roles and AWS Unified CLI’s S3 client

Bootstrapping Puppet with CloudInit Built-in puppet support

Use certname with %i for instance id to name the node Puppetmaster must have auto sign turned on • Use security groups and/or NACLs for network-level security

In nodes.pp, use regex to match node names

Puppet Tips

Use a base class to define your standard install

Use runstages Don’t store credentials in puppet, store them in private Amazon S3 buckets • Use AWS IAM to secure the credentials bucket/folders within that bucket

Puppet Tips

Puppet Tips

Use puppet only for configuration files and what makes your apps unique For undifferentiated parts of apps, use Amazon S3 backed RPM/Debian repositories • Can be either public or private repos, depending on your needs

• Amazon S3 Private RPM Repos: http://git.io/YAcsbg • Amazon S3 Private Debian Repos: http://git.io/ecCjWQ

http://git.io/YAcsbg

http://git.io/ecCjWQ

Puppet Tips

By using packages for applications deploys, you can set ensure => latest, and just bump the package in the repo to update Log everything with rsyslog/graylog/loggly/NewRelic/splunk

Scaling the Puppet Masters

Use an Auto Scaling group for puppet masters • Min size => 2, use multiple Availability Zones

Either have them build themselves off of existing puppet masters in the group or off packages stored in Amazon S3 and bootstrapped through user-data Auto-sign must be on

One thing that is difficult to prepare for…

They had this built for the previous 3 months, a on the East Coast.

They had this built for the previous 3 months, a on the East Coast.

We built this part in 9 hours to be safe.

AWS + Puppet +

Netflix Asgard + WAN Optimization Software +

DevOps =

Cross-Continent Fault-Tolerance On-Demand

ARC205 Thursday @ 3:00PM Lando 4303 Deploying the ‘League of Legends’ Data

Pipeline with Chef

Mozilla Foundation

Webmaker.org circa 2012 • Included Apps, non-SOA: Thimble, Popcorn, Goggles • ~20 pushes of new software in 2012 • Operations and Development interacted mostly through bugzilla

tickets for deploys • Hosting in physical datacenter at Mozilla

Webmaker.org circa early 2013 • Deciding to go 12-factor, SOA in app layer • Weekly pushes of Popcorn on train model • Operations and Development interacted mostly through bugzilla

tickets for deploys

April 2013

Webmaker begins rebuilding entire platform SOA, 12-factor in node.js exclusively Moving apps into AWS and DevOps / CI

Since April 2013…. Openbadges, Webmaker combine for: 1339 Pushes

Pushes Per Day to Staging / Prod

Pushes Per Day (Staging and Prod)

Who? • ~30 Paid Developers • Hundreds of Students • Thousands of Contributors • One DevOps / Internet Jedi • Multiple Teams

How? • Puppet, Jenkins, Fabric • Tight feedback loops:

Newrelic, Opsview • Culture Shift

• Staging Envs • Brave devs iterate,

keeping work in-context • Visible Ops • Cross-train developers in

operations

What changed?

1) Know How You Were Doing Before

2) Know What Changed When

3) Know How You Are Now Doing

= The confidence to try more things and try them faster, with minimum viable planning.

Deployment

Pipeline

ARC312 Friday @ 1:30PM Lando 4206 SmugMug’s Zero-Downtime Migration to

AWS

AWS CLOUDFORMATION

STACK-BASED DEPLOYMENT SERVICE

AWS CLOUDFORMATION TEMPLATE

{ "Description" : "Create RDS with username and password", "Resources" : { "MyDB" : { "Type" : "AWS::RDS::DBInstance", "Properties" : { "AllocatedStorage" : "500", "DBInstanceClass" : "db.m1.small", "Engine" : "MySQL", "EngineVersion" : "5.5", "MasterUsername" : "MyName", "MasterUserPassword" : "MyPassword" } } } }

"AWS::CloudFormation::Init" : { "config" : { "packages" : { "yum" : { "mysql" : [], "mysql-server" : [], "httpd" : [], "php" : [], "php-mysql" : [] } }, "sources" : { "/var/www/html" : "https://s3.amazonaws.com/my-builds/build-v4.zip" } }

{ "Parameters" : { "KeyName" : { "Description" : "Name of an existing EC2 KeyPair to enable SSH access to the instance", "Type" : "String" } }, }

CLOUDFORMATION TEMPLATE

PROCEDURAL DEFINITION

Create it programmatically

KNOWN CONFIGURATION Store stack configuration in source control

PARAMETER DRIVEN

Dynamic and user-driven templates

COLLABORATION Share templates with ease as just files

APPLICATION VERSIONS

+ INFRASTRUCTURE

VERSIONS

AWS CLOUDFORMATION

TEMPLATE

ARC203 Wednesday @ 4:15 Lando 4303 How Adobe Deploys

Refreshing the Entire Stack Every Time

Mars Rover Landing by the #’s

NASA TV = HD stream, 1080p, ~1 mb/s per viewer Expecting peak of ~1m viewers All playback devices (iOS, Android, Flash, HTML5, blah) Once in a lifetime moment in history (no crashy crashy)

FUN FUN FUN

Mars Rover Landing by the #’s

NASA TV = HD stream, 1080p, ~1 mb/s per viewer Expecting peak of ~1m viewers All playback devices (iOS, Android, Flash, HTML5, blah) Once in a lifetime moment in history (no crashy crashy) NASA says we can’t use their live stream setup It’s 6 days before the landing It’s the same week as the Olympics Available technical resources from JPL: Brett and Khawaja

NO

T FUN

NO

T FUN

NO

T FUN

The sticky wicket:

HD Video stream size = 4mb never changes easy to cache

manifest.f4m size = 4kb New every 4 sec caching difficult

The plan

Design a solution around our limits • Max connections to origin = 6 • Max streams per cache node = 20 • Local Latency = critical • US-WEST-1 capacity reserved for S3 static images

Test the snot out of it Hang on!

ok, so one of them is a rocket scientist..

“The 42 pack”

LOAD TESTING

Benchmarking

US-East Cache Node Performance 25.3 Gbps

Only ~42Mbps Impact on US-East FMS Origin Servers

ARC303 Friday @ 1:30PM Delfino 4003 Unmeltable Infrastructure at Scale:

Using Apache Kafka, Twitter Storm and Elastic Search on AWS

CONTINUOUS DEPLOYMENT

SMALL, FREQUENT CHANGES CONSTANTLY INTEGRATING INTO

PRODUCTION

KEY = ITERATION

ITERATION =

MODIFY THE SYSTEM TO BETTER MEET THE EXPECTATIONS OF

YOUR USERS

11.6s

Mean time between

deployments (weekday)

1,079

Max number of deployments in a

single hour

10,000

Mean number of hosts

simultaneously receiving a deployment

30,000

Max number of hosts

simultaneously receiving a deployment

DEPLOYMENTS AT AMAZON.COM

SOFTWARE DEPLOY

≠ PRODUCT LAUNCH

1.5 BILLION PAGE VIEWS

OCTOBER 2012

$83 MILLION IN TRANSACTIONS 4.2 MILLION ITEMS SOLD

30 DEPLOYS PER DAY 1 DEPLOY EVERY 20 MINUTES

"Production is truly the only place you can validate your code."

AWS OPSWORKS INTEGRATED APPLICATION

MANAGEMENT

DMG 304 Thursday @ 3:00PM Murano 3206 AWS OPSWORKS UNDER THE HOOD

14 BILLION REQUESTS/MONTH 50 000 DATABASE UPDATES / SEC

NO CACHE

DATA-DRIVEN ARCHITECTURES

METRICS @ETSY

METRICS @OBAMA FOR AMERICA

COST-ORIENTED ARCHITECTURES

PHP+APACHE+VARNISH

NGINX+NODEJS

CONTINUOUS INTEGRATION



= CONTINUOUS

EXPERIMENTATION


= CONTINUOUS

IMPROVEMENT

INNOVATE

« Want to increase innovation? Lower the cost of failure »

Joi Ito

SPEED AND AGILITY

Experiment Often

Fail quickly at

a low cost

More Innovation

Experiment Infrequently

Failure is expensive

Less

Innovation

“ON-PREMISES”

AWS re:Invent Pub Crawl

Join the AWS Startup Team this evening at the AWS Pub Crawl When: Wednesday November 13, 5:30pm - 7:30pm Where: Canaletto at The Venetian, 2nd Floor Who Will Be There: Startups, The AWS Startup Team, Startup Launch Companies and AWS re:Invent Hackathon winners

Startup Spotlight Sessions with Dr. Werner Vogels Thurs. Nov 14, Marcello Room 4406

SPOT 203 - Fireside Chats – Startup Founders, 1:30-2:30pm – Eliot Horowitz, CTO of MongoDB – Jeff Lawson, CEO of Twilio – Valentino Volonghi, Chief Architect of AdRoll

SPOT 204 - Fireside Chats – Startup Influencers, 3:00-4:00pm – Albert Wegner, Managing Partner at Union Square Ventures – David Cohen, Founder and CEO of TechStars

SPOT 101 - Startup Launches, 4:15-5:15pm – 5 companies powered by AWS launching at AWS re:Invent 2013

Please give us your feedback on this presentation

As a thank you, we will select prize winners daily for completed surveys!

ARC 307

Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013

Technology

Transcript of Continuous Integration and Deployment Best Practices on AWS (ARC307) | AWS re:Invent 2013