Scaling Rugged DevOps to Thousands of Applications - Panel Discussion
Scaling DevOps
-
Upload
jeffrey-hulten -
Category
Technology
-
view
404 -
download
0
description
Transcript of Scaling DevOps
Scaling DevOpsJeffrey HultenWhitepages, Inc.
Wednesday, December 4, 13
Who Am I?
Jeffrey Hulten - @jhulten
Software Engineer, Architecture Improvement at WhitePages, Inc.
Organizer of Seattle DevOps MeetUp
Too Ops for Dev, Too Dev for Ops
Wednesday, December 4, 13
Scaling DevOps
Your Tools Aren’t Enough
Your First 100 Machines
Watching the Horizon
Wednesday, December 4, 13
Your Tools Aren’t Enough
Culture
Deploy a Package
Manage the Metal
Minimize the Surface Area
Growing Pains
Wednesday, December 4, 13
CultureNot Just for Bacteria Anymore
Starts with one person
Infect your friends
Try changing things
Mutation must allow for failure
Requirement for survival
Wednesday, December 4, 13
Deploy a Package
TGZ
OS Package (.deb / .rpm)
Machine Image (AMI)
LXC Image (Docker)
Wednesday, December 4, 13
OS Packages Made Easier
FPM
Package Config Separately
Package Repository
Wednesday, December 4, 13
FPM: Effing Package Management
https://github.com/jordansissel/fpm
$ gem install fpm
Make RPM, DEB, and Solaris packages and Puppet module
Create from directories, RPM, DEB, Gem, PyPi, Tarball and more
Wednesday, December 4, 13
Example: Jenkins
NAME=jenkinsVERSION=1.396
.PHONY: packagepackage: rm -f jenkins.war wget http://ftp.osuosl.org/pub/hudson/war/$(VERSION)/jenkins.war fpm -s dir -t deb -n $(NAME) -v $(VERSION) \ --prefix /opt/jenkins jenkins.war
Wednesday, December 4, 13
Machine Images & Containers
Machine Image
Entire OS
All Services Enabled
Containers
Minimal OS Image
Run One Process (Replace `init`)
Wednesday, December 4, 13
Manage the Metal
Application as Inventory...
Wednesday, December 4, 13
Minimize the Surface Area
What is Surface Area?
Libraries?
Versions?
Components?
Data Stores?
Wednesday, December 4, 13
Minimize the Surface Area
What is Surface Area?
Libraries?
Versions?
Components?
Data Stores?
Deployables...
Wednesday, December 4, 13
Growing Pains
Don’t bottle knowledge
There are no specialists
Think about the smallest solution
Delay your decisions
Wednesday, December 4, 13
Your First 20 Machines
DEMO TIME!
Source at: https://github.com/whitepages/scaledemo-20-machines
AWS CloudFormation, cloud-init...
Wednesday, December 4, 13
Application: Random Numbers as a Service
Wednesday, December 4, 13
CloudFormation
Deploy Stacks
Templates in JSON
Parameters for Reuse
Wednesday, December 4, 13
Template: Parameters"Parameters": { "OperatorEmail": { "Type": "String", "Description": "Email address to notify if there are any scaling operations" }, "InstanceType": { "Description": "WebServer EC2 instance type", "Type": "String", "AllowedValues": [ "t1.micro", "m1.small", ...], "Default": "m1.small", "ConstraintDescription": "must be a valid EC2 instance type." },
... "SSHLocation": { "Default": "0.0.0.0/0", "Type": "String", "MaxLength": "18", "MinLength": "9", "AllowedPattern": "(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})/(\\d{1,2})", "Description": "The IP address range that can be used to SSH to the EC2 instances", "ConstraintDescription": "must be a valid IP CIDR range of the form x.x.x.x/x." } },
Wednesday, December 4, 13
Template: Parameters
"SSHLocation": { "Default": "0.0.0.0/0", "Type": "String", "MaxLength": "18", "MinLength": "9", "AllowedPattern": "(\\d{1,3})\\.(\\d{1,3}).../(\\d{1,2})", "Description": "...", "ConstraintDescription": "..."}
Wednesday, December 4, 13
Template: Mappings"Mappings": { "AWSRegionArch2AMI": { "us-east-1": { "64": "ami-955b79fc" }, "us-west-1": { "64": "ami-6ca89929" }, "us-west-2": { "64": "ami-a8395c98" } }, ...},
Wednesday, December 4, 13
Template: Resources"Resources": { "NotificationTopic": { "Type": "AWS::SNS::Topic", "Properties": { "Subscription": [ { "Protocol": "email", "Endpoint": { "Ref": "OperatorEmail" } } ] } }, ...}
Wednesday, December 4, 13
AutoScaling Group"RandServerGroup": { "Type": "AWS::AutoScaling::AutoScalingGroup", "Properties": { "AvailabilityZones": { "Fn::GetAZs": "" }, "NotificationConfiguration": { "NotificationTypes": [ "autoscaling:EC2_INSTANCE_LAUNCH", "autoscaling:EC2_INSTANCE_LAUNCH_ERROR", "autoscaling:EC2_INSTANCE_TERMINATE", "autoscaling:EC2_INSTANCE_TERMINATE_ERROR" ], "TopicARN": { "Ref": "NotificationTopic" } }, "MinSize": "1", "MaxSize": "100", "LaunchConfigurationName": { "Ref": “..." }}}
Wednesday, December 4, 13
Launch Config"RandLaunchConfig": { "Type": "AWS::AutoScaling::LaunchConfiguration", "Properties": { "SecurityGroups": [{ "Ref": InstanceSecurityGroup" }], "InstanceType": { "Ref": "InstanceType" }, "KeyName": { "Ref": "KeyName" }, "UserData": { "Fn::Base64": { "Fn::Join" : ["",[ "#include\n", "https://.../cloud-config.txt\n" ]]}}, "ImageId": {"Fn::FindInMap": [...] } }},
Wednesday, December 4, 13
Cloud Init
#cloud-configpackages: - ruby1.9.3 - git - build-essential
runcmd: - /usr/bin/gem install bundler - /usr/bin/git clone "https://github.com/whitepages/scaledemo-20-machines.git" /opt/rand - cd /opt/rand && /usr/local/bin/bundle install - cd /opt/rand &&/usr/local/bin/bundle exec randserver &
Wednesday, December 4, 13
Watching the Horizon
Docker / LXC
SmartStack
Akka Cluster / Riak Core
Wednesday, December 4, 13
Twitter: @jhulten
Github: jhulten
whitepages
whitepages.com/careers
Wednesday, December 4, 13