Scaling DevOps

26
Scaling DevOps Jeffrey Hulten Whitepages, Inc. Wednesday, December 4, 13

description

Scaling DevOps from the Seattle Scalability Meetup, December 2013

Transcript of Scaling DevOps

Page 1: Scaling DevOps

Scaling DevOpsJeffrey HultenWhitepages, Inc.

Wednesday, December 4, 13

Page 2: Scaling DevOps

Who Am I?

Jeffrey Hulten - @jhulten

Software Engineer, Architecture Improvement at WhitePages, Inc.

Organizer of Seattle DevOps MeetUp

Too Ops for Dev, Too Dev for Ops

Wednesday, December 4, 13

Page 3: Scaling DevOps

Scaling DevOps

Your Tools Aren’t Enough

Your First 100 Machines

Watching the Horizon

Wednesday, December 4, 13

Page 4: Scaling DevOps

Your Tools Aren’t Enough

Culture

Deploy a Package

Manage the Metal

Minimize the Surface Area

Growing Pains

Wednesday, December 4, 13

Page 5: Scaling DevOps

CultureNot Just for Bacteria Anymore

Starts with one person

Infect your friends

Try changing things

Mutation must allow for failure

Requirement for survival

Wednesday, December 4, 13

Page 6: Scaling DevOps

Deploy a Package

TGZ

OS Package (.deb / .rpm)

Machine Image (AMI)

LXC Image (Docker)

Wednesday, December 4, 13

Page 7: Scaling DevOps

OS Packages Made Easier

FPM

Package Config Separately

Package Repository

Wednesday, December 4, 13

Page 8: Scaling DevOps

FPM: Effing Package Management

https://github.com/jordansissel/fpm

$ gem install fpm

Make RPM, DEB, and Solaris packages and Puppet module

Create from directories, RPM, DEB, Gem, PyPi, Tarball and more

Wednesday, December 4, 13

Page 9: Scaling DevOps

Example: Jenkins

NAME=jenkinsVERSION=1.396

.PHONY: packagepackage: rm -f jenkins.war wget http://ftp.osuosl.org/pub/hudson/war/$(VERSION)/jenkins.war fpm -s dir -t deb -n $(NAME) -v $(VERSION) \ --prefix /opt/jenkins jenkins.war

Wednesday, December 4, 13

Page 10: Scaling DevOps

Machine Images & Containers

Machine Image

Entire OS

All Services Enabled

Containers

Minimal OS Image

Run One Process (Replace `init`)

Wednesday, December 4, 13

Page 11: Scaling DevOps

Manage the Metal

Application as Inventory...

Wednesday, December 4, 13

Page 12: Scaling DevOps

Minimize the Surface Area

What is Surface Area?

Libraries?

Versions?

Components?

Data Stores?

Wednesday, December 4, 13

Page 13: Scaling DevOps

Minimize the Surface Area

What is Surface Area?

Libraries?

Versions?

Components?

Data Stores?

Deployables...

Wednesday, December 4, 13

Page 14: Scaling DevOps

Growing Pains

Don’t bottle knowledge

There are no specialists

Think about the smallest solution

Delay your decisions

Wednesday, December 4, 13

Page 15: Scaling DevOps

Your First 20 Machines

DEMO TIME!

Source at: https://github.com/whitepages/scaledemo-20-machines

AWS CloudFormation, cloud-init...

Wednesday, December 4, 13

Page 16: Scaling DevOps

Application: Random Numbers as a Service

Wednesday, December 4, 13

Page 17: Scaling DevOps

CloudFormation

Deploy Stacks

Templates in JSON

Parameters for Reuse

Wednesday, December 4, 13

Page 18: Scaling DevOps

Template: Parameters"Parameters": { "OperatorEmail": { "Type": "String", "Description": "Email address to notify if there are any scaling operations" }, "InstanceType": { "Description": "WebServer EC2 instance type", "Type": "String", "AllowedValues": [ "t1.micro", "m1.small", ...], "Default": "m1.small", "ConstraintDescription": "must be a valid EC2 instance type." },

... "SSHLocation": { "Default": "0.0.0.0/0", "Type": "String", "MaxLength": "18", "MinLength": "9", "AllowedPattern": "(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})/(\\d{1,2})", "Description": "The IP address range that can be used to SSH to the EC2 instances", "ConstraintDescription": "must be a valid IP CIDR range of the form x.x.x.x/x." } },

Wednesday, December 4, 13

Page 19: Scaling DevOps

Template: Parameters

"SSHLocation": { "Default": "0.0.0.0/0", "Type": "String", "MaxLength": "18", "MinLength": "9", "AllowedPattern": "(\\d{1,3})\\.(\\d{1,3}).../(\\d{1,2})", "Description": "...", "ConstraintDescription": "..."}

Wednesday, December 4, 13

Page 20: Scaling DevOps

Template: Mappings"Mappings": { "AWSRegionArch2AMI": { "us-east-1": { "64": "ami-955b79fc" }, "us-west-1": { "64": "ami-6ca89929" }, "us-west-2": { "64": "ami-a8395c98" } }, ...},

Wednesday, December 4, 13

Page 21: Scaling DevOps

Template: Resources"Resources": { "NotificationTopic": { "Type": "AWS::SNS::Topic", "Properties": { "Subscription": [ { "Protocol": "email", "Endpoint": { "Ref": "OperatorEmail" } } ] } }, ...}

Wednesday, December 4, 13

Page 22: Scaling DevOps

AutoScaling Group"RandServerGroup": { "Type": "AWS::AutoScaling::AutoScalingGroup", "Properties": { "AvailabilityZones": { "Fn::GetAZs": "" }, "NotificationConfiguration": { "NotificationTypes": [ "autoscaling:EC2_INSTANCE_LAUNCH", "autoscaling:EC2_INSTANCE_LAUNCH_ERROR", "autoscaling:EC2_INSTANCE_TERMINATE", "autoscaling:EC2_INSTANCE_TERMINATE_ERROR" ], "TopicARN": { "Ref": "NotificationTopic" } }, "MinSize": "1", "MaxSize": "100", "LaunchConfigurationName": { "Ref": “..." }}}

Wednesday, December 4, 13

Page 23: Scaling DevOps

Launch Config"RandLaunchConfig": { "Type": "AWS::AutoScaling::LaunchConfiguration", "Properties": { "SecurityGroups": [{ "Ref": InstanceSecurityGroup" }], "InstanceType": { "Ref": "InstanceType" }, "KeyName": { "Ref": "KeyName" }, "UserData": { "Fn::Base64": { "Fn::Join" : ["",[ "#include\n", "https://.../cloud-config.txt\n" ]]}}, "ImageId": {"Fn::FindInMap": [...] } }},

Wednesday, December 4, 13

Page 24: Scaling DevOps

Cloud Init

#cloud-configpackages: - ruby1.9.3 - git - build-essential

runcmd: - /usr/bin/gem install bundler - /usr/bin/git clone "https://github.com/whitepages/scaledemo-20-machines.git" /opt/rand - cd /opt/rand && /usr/local/bin/bundle install - cd /opt/rand &&/usr/local/bin/bundle exec randserver &

Wednesday, December 4, 13

Page 25: Scaling DevOps

Watching the Horizon

Docker / LXC

SmartStack

Akka Cluster / Riak Core

Wednesday, December 4, 13

Page 26: Scaling DevOps

[email protected]

Twitter: @jhulten

Github: jhulten

whitepages

whitepages.com/careers

Wednesday, December 4, 13