Loft Your Web Platform into the Clouds with Immutable Infrastructure
-
Upload
phase2 -
Category
Technology
-
view
271 -
download
1
Transcript of Loft Your Web Platform into the Clouds with Immutable Infrastructure
4© 2016 Phase2
TOPICS COVEREDImmutability in software development
Immutable Infrastructure
Building immutable artifacts
Using immutable artifacts with AWS autoscaling
Using immutable artifacts with containers
Real-world examples
7© 2016 Phase2
IMMUTABILITY IN SOFTWARE DEVELOPMENTFirst-class immutable variables in language runtimes
No need to synchronize read access in parallel systems
Immutable data stores
Elm’s time-travel debugging
Mercurial’s immutable history / write-ahead log
Immutable data structures in React.js
9© 2016 Phase2
IMMUTABLE INFRASTRUCTURE? WHAT?Continuation of infrastructure as code
Core tenet: build immutable artifacts and promote them
As in development, try to limit mutation of deployed infra
Unchanging artifacts (AMIs, containers) eliminate drift
10© 2016 Phase2
AN EXAMPLEA client had autoscaling troubles
Provisioned from scratch from a plain Ubuntu AMI
Problems:
Drift
Time to provision / autoscaling timeouts
Hard dependency on configuration management server
11© 2016 Phase2
COUNTERPOINTSMost servers may not have read-only disks
Container systems can allow read-only rootfs
Will you really disable SSH?
One theory: contaminate instances if you do
12© 2016 Phase2
WHY?Immutable infra practices enable autoscaling
Rallying cry: have cattle, not pets
It can allow for easier rollback
Possibility for canary or blue/green deployment
From here on out, we’ll discuss autoscaling readiness
14© 2016 Phase2
BUILD THEORYA modern web app likely has several build steps
Pull dependencies down
Source compilation
CSS/JS compilation and assembly
Phase2 has grunt-drupal-tasks to automate this
Build from drush make, npm theme tasks
Has a package command to make a folder or tarball
15© 2016 Phase2
VENDORINGConsider vendoring your dependencies
Great discussion about this in “Who Owns Your Availability?” episode of Arrested DevOps
16© 2016 Phase2
BUILD TARGETSDeployment repository
Tarball
Machine image
Amazon AMI / Google Private images
Rkt / Docker tagged container image
18© 2016 Phase2
BUILD PROCESS EVOLUTIONAwful
Build in-place on each of n web heads
Better
Build once in workspace, rsync to docroots
Good
Build once, store artifact, capistrano-style deploy
Goal
Build artifact, build image around artifact
19© 2016 Phase2
BUILDING IMAGESAutoscaling-ready image builds have 2 components:
Tagged code releases
A combination of Linux packages on a filesystem
Linux vulnerabilities happen fairly frequently
Be ready to build for new code releases and package CVEs
21© 2016 Phase2
WHY AUTOSCALE?Reduces stress
Automatic healing
Cost benefits
Performance benefits
Automatic scaling based on metrics
Manual adjustment up and down
23© 2016 Phase2
DRUPAL SITE COMPONENTSBig 3 components of a running Drupal site
Codebase on an app server
Relational database
Files directory
Optional
Object cache (?)
Varnish / CDN
24© 2016 Phase2
AUTOSCALING GROUPS IN AWSAn autoscaling group is attached to one or more ELBs
ELB health checks and timeouts control ASG actions
You use CloudWatch alarms to trigger autoscaling
Avg CPU > 70% for 5 min, avg CPU < 40% for 30 min
A LaunchConfig ties together an AMI and instance size
e.g. ami-beefbeef on c4.xlarge w/ 100 GB disks
One ASG is set to one LaunchConfig at a time
25© 2016 Phase2
AWS RECIPE FOR DRUPALJenkins instance building tarballs + AMIs
Autoscaling group with public Drupal instances
RDS for MySQL
S3 bucket for files directory via amazons3.module
ElastiCache for memcache via memcache.module
CloudFront for page caching / distribution
SES for email via smtp.module
26© 2016 Phase2
EXAMPLE AMI AUTOSCALING SETUPHave one build process to create a tarball / build artifact
Use Packer to start from a base AMI
File provisioner copies code up
Run Ansible to install packages
Create new LaunchConfig
Flip instances over one at a time
27© 2016 Phase2
ELIDED PACKER EXAMPLE{ "variables": { "aws_access_key" : "xxxxxxxxxxxxxxxxxxxx", "aws_secret_key" : "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", "aws_instance_type" : "t2.micro", "east_aws_ami" : "ami-00000000", "west_aws_ami" : "ami-11111111", "aws_ssh_username" : "ec2-user", “east_aws_subnet" : "subnet-00000000”, “west_aws_subnet" : "subnet-11111111", "east_aws_vpc": "vpc-00000000", "west_aws_vpc": "vpc-11111111" },
28© 2016 Phase2
ELIDED PACKER EXAMPLE "provisioners": [{ "type": "file", "source": "/path/to/my/build.tgz", "destination": "/opt/build.tgz" }, { "type" : "shell", "inline": [ "/usr/bin/ansible-playbook -ilocal /opt/playbook.yml” ]}], ],
29© 2016 Phase2
ELIDED PACKER EXAMPLE "builders": [{ "type": "amazon-ebs", "name": "east-ami", "access_key": "{{user `aws_access_key`}}", "secret_key": "{{user `aws_secret_key`}}", "region": "us-east-1", "source_ami": "{{user `east_aws_ami`}}", "instance_type": "{{user `aws_instance_type`}}", "ssh_username": "{{user `aws_ssh_username`}}", "security_group_ids" : ["sg-00000000"], "subnet_id" : "{{ user `east_aws_subnet` }}", "vpc_id": "{{user `east_aws_vpc`}}", "ami_name": "autoscale_{{user `build_tag`}}_east" },
30© 2016 Phase2
ELIDED PACKER EXAMPLE { "type": "amazon-ebs", "name": "west-ami", "access_key": "{{user `aws_access_key`}}", "secret_key": "{{user `aws_secret_key`}}", "region": "us-west-2", "source_ami": "{{user `west_aws_ami`}}", "instance_type": "{{user `aws_instance_type`}}", "ssh_username": "{{user `aws_ssh_username`}}", "security_group_ids" : ["sg-11111111"], "subnet_id" : "{{ user `west_aws_subnet` }}", "vpc_id": "{{user `west_aws_vpc`}}", "ami_name": "autoscale_{{user `build_tag`}}_west" }]}
31© 2016 Phase2
S3 SHARP EDGESS3 files with amazons3/s3fs.module use memory_limit
You may be tempted to try the s3fs daemon
Instead, s3fs_cors/amazons3_cors modules
32© 2016 Phase2
AUTOSCALING QUIRKSSchema updates
Whitescreens on new/old as schema changes flip
Add double capacity, let ELB health checks do the work
Maintenance mode
Separate editorial instance
Build one image and switch behavior on ENV
34© 2016 Phase2
MLS DIGITAL - MLSSOCCER.COM AND CLUBS21-site Drupal 7 platform
League and all club sites
Game day scaling patterns - huge traffic spikes
No autoscaling, separate docroots
Moved to multisite in prep
Build tarballs from 500 MB to 40 MB
Total opcache / APC savings of 1.6 GB
35© 2016 Phase2
MOVING MLS TO AUTOSCALINGOld setup provisioned from a base Ubuntu with Salt
Timeout problems
Moved to Packer AMI builds with all software & code
An AMI can come up if everything else is down
Systems of record for Drupal state are elsewhere
MySQL database - Amazon RDS
Memcache - Amazon ElastiCache
Files - Amazon S3 via s3fs.module
36© 2016 Phase2
SUCCESSFrom 4 pets to 2-5 cattle
Major hosting cost reduction
Later Drupal optimizations lowered instance size
41© 2016 Phase2
POTENTIAL DOCKER BENEFITSIncreased density
Ability to run heterogenous workloads
Native CPU and disk performance
Resource limits
42© 2016 Phase2
DOCKER IN PRODUCTIONDocker on single hosts is easy
Docker in production requires a lot of coordination
Overlay networking / network security
Service discovery
Intelligent scheduling and capacity management
Routing / load balancing
Secrets management
Monitoring and performance management
43© 2016 Phase2
DOCKER BUILDSBuild considerations
Dockerfile vs configuration management
Dynamic templating
Build tools
docker build Packer
Configuration management system
44© 2016 Phase2
DOCKERDocker daemon allows for tagging
Ideally, use tags from canonical repo and as branches
p2/site:v1.0.0, p2/site:v1.1.0 p2/site:dev, p2/site:stage, p2/site:prod
Build script
docker build -t p2/site:v1.1.0 . docker tag p2/site:v1.1.0 p2/site:dev docker push p2/site:v1.1.0; docker push p2/site:dev
45© 2016 Phase2
KUBERNETESOpen-source container scheduler from Google
“Manage a cluster of Linux containers as a single system to accelerate Dev and simplify Ops.”
Based on Borg design principles
Large pool of contributors and upstream/downstream users
Red Hat (OpenShift 3)
Deis (Helm)
CoreOS (Tectonic)
46© 2016 Phase2
KUBERNETESRequires you to bring overlay networking
Features
Service discovery, load balancing, cluster DNS
Pod scaling and autoscaling, node draining
Resource-aware scheduling and capacity limits
REST API and CLI clients
Secrets management
YAML or JSON definition of k8s API objects
47© 2016 Phase2
KUBERNETES OBJECTSNodes
Labels
Pods
DaemonSets
Replication controllers
Services
Deployments
48© 2016 Phase2
KUBERNETES POD YAMLapiVersion: v1 kind: Pod metadata: name: lamp app: lamp
spec: containers: - name: web image: phase2/apache24php55 resources: requests: memory: "512Mi" cpu: "1000m" limits: memory: "1024Mi" cpu: "2000m”
49© 2016 Phase2
KUBERNETES RC YAMLapiVersion: v1 kind: ReplicationController metadata: name: lamp spec: replicas: 5 selector: app: lamp template: metadata: labels: app: lamp spec: containers:
- name: lamp image: phase2/apache24php55 resources: requests: memory: "512Mi" cpu: "1000m" limits: memory: "1024Mi" cpu: “2000m” ports: - containerPort: 8080 name: lamp protocol: TCP
50© 2016 Phase2
KUBERNETES SERVICE YAMLapiVersion: v1 kind: Service metadata: name: lamp spec: selector: app: lamp ports: - name: lamp port: 8080
52© 2016 Phase2
IMMUTABLE INFRASTRUCTUREBuild immutable artifacts for deployment
Limit configuration drift in your infrastructure
Be ready to build new images for code releases or CVEs
Autoscaling can help with cost, stress, and self-healing
So How Was It? - Tell Us What You ThinkEvaluate th is session - ht tps: / /events.drupal .org/neworleans2016/schedule
Thanks!