scaling Drupal horizontally and in the

cloud

about me

name: Vladimir Ilic

email: burger.boy.daddy@gmail.com

twitter: @burgerboydaddy

http://burgerboydaddy.com

agenda

why all of this?

step 1: test locally -> from one server to the server farm,

step 2: multiple web and database servers,

step 3: how to join all that together and make site deployed on Amazon Cloud and inside Virtual Private Cloud

Amazon term

benefits

single availability zone

multiple availability zones.

why?

if you want to increase site speed

if you want your site to be responsive and to work under heavy stress

if you want to be in control what goes on your server

get it divided / decouple

Easy to do inside local development/hosting environment

Just separate web, database and cache servers

Problems

we can increase resources only vertically

Not all resources are used same way (web server will probably die before cache or MySQL)

Multiple “single points of failure”

multiple web servers – one db

Apache load balancer in front of

2-3 web servers; each server with integrated APC cache

Multiple cache servers

Powerful MySQL server

In real life you can use some other LB solution (this one is great for proof of concept moments).

Without dedicated file server; used bi-directional rsync replication

configuring Apache load balancer

Apache web server ships a load balancer module called mod_proxy_balancer (since version 2.2).

All you need to do is to enable this module and the modules mod_proxy and mod_proxy_http. Please note that without mod_proxy_http, balancer just won't work.

LoadModule proxy_module mod_proxy.so

LoadModule proxy_http_module mod_proxy_http.so

LoadModule proxy_balancer_module mod_proxy_balancer.so

many to many

In this case each web server will have it's own db server.

Reason for this:

Higher site availability; if one db server is down, second one can continue to serve customers.

Amazon AWS

Why Amazon (business point of view)

Most complete cloud solution on the market.

Almost zero upfront infrastructure investment

Just-in-time infrastructure

Pay as you go – pay what you use

Constant price drop

Easy to deploy and scale

….

why Amazon (technical benefits)

Automation – “Scriptable infrastructure”: You can create repeatable build and deployment systems by leveraging programmable (API-driven) infrastructure.

Auto-scaling: You can scale your applications up and down to match your unexpected demand without any human intervention.

Proactive Scaling: Scale your application up and down to meet your anticipated demand; Elasticity

why Amazon (technical benefits)

More Efficient Development lifecycle: Production systems may be easily cloned for use as development and test environments.

Improved Testability: Never run out of hardware for testing. Inject and automate testing at every stage during the development process.

Disaster Recovery and Business Continuity: The cloud provides a lower cost option for maintaining a fleet of DR servers and data storage.

understanding elasticity

key Amazon terms – #1

AWS – Amazon Web Services

Amazon Web Services (AWS) is a collection of remote computing services (also called web services) that together make up a cloud computing platform.

EC2     - Elastic Compute Cloud

EC2 allows users to rent virtual computers on which to run their own computer applications. EC2 allows scalable deployment of applications by providing a Web service through which a user can boot an Amazon Machine Image to create a virtual machine.

A user can create, launch, and terminate server instances as needed, paying by the hour for active servers, hence the term "elastic".

S3       - Simple Storage Service

Amazon S3 (Simple Storage Service) is an online storage web service offered by AWS.

AMI     - Amazon Machine Images

An Amazon Machine Image (AMI) is a special type of virtual appliance which is used to instantiate (create) a virtual machine within the Amazon Elastic Compute Cloud ("EC2").

key Amazon terms - #2EBS     - Elastic Block Storage

Amazon Elastic Block Storage (EBS) provides raw block devices that can be attached to Amazon EC2 instances.

Can be used like any raw block device. In a typical use case, this would include formatting the device with a filesystem and mounting said filesystem.

VPC     - Virtual Private Cloud

Amazon Virtual Private Cloud (VPC) is a commercial cloud computing service that provides a virtual private cloud.

Unlike traditional EC2 instances which are allocated internal and external IP numbers by Amazon, the customer can assign IP numbers of their choosing from one or more subnets.

VPC provides much more granular control over security.

ELB     - Elastic Load Balancing

AZ     - Amazon Availability Zones (Data Centers)

key Amazon terms - #3RDS     - Amazon Relational Database Service

Amazon RDS is a distributed relational database service by Amazon.com.

It is a web service running "in the cloud" and provides a relational database for use in applications.

Supporting

MySQL databases

Oracle databases

Microsoft SQL Server

ECU     - Elastic Computational Unit

One EC2 Compute Unit provides the equivalent CPU capacity of a 1.0-1.2 GHz 2007 Opteron or 2007 Xeon processor.

SQS     - Simple Queue Service

list of services goes on…

humor

“We will launch site on EC2 with EBS behind ELB with domain registered on Route 53

Your images will come from CloudFront, backup will go to S3

and your DB on RDS with Multi-AZ availability”

first step first – create account

Go to aws.amazon.com and just use your amazon.com account for start

After login go to IAM (Identity Access Management) to add multi-factor authentication; not to your root account, but create new account, assign privileges to it and add MFA. After that use only new account to login to your AWS (with given alias)

easy one – use CloudFormation

Fastest way to get Drupal on AWS is using predefined templates inside CloudFormation service.

In this moment you can find 4 (Drupal specific) templates

Drupal_Simple.template

Drupal_Single_Instance.template

Drupal_Single_Instance_With_RDS.template

Drupal_Multi_AZ.template

You can use any other template as starting point and customize it to your needs.

steps afterCreate KeyPair

Add your home/corporate IP to be only allowed to access server over port 22 (SSH).

Create AMI from existing machine

Drop original machine

Create new EC2 instance using just created AMI and your key-pair

Add Elastic IP and associate to your instance

Connect to instance

Add DNS CNAME record using given Amazon DNS name:

ec2-54-225-110-202.compute-1.amazonaws.com

demo

Mandatory clouds

image :-)

Amazon VPC - ultimate goal

We can install complete infrastructure required for Drupal using public set of servers

ELB (load balancer)

AMI (servers images)

RDS (Amazon relational database service)

Elastic Cache...

BUT

Amazon VPC is a way to setup an isolated partition of AWS and control the network topology.

Services

Dynamodb, ElastiCache, SQS, SES, and CloudSearch are not yet available in VPC (things change on daily basis)

RDS instances launched in VPC cannot be accessed over the internet (through the end point). You will need bastion server to access it

EC2 / VPC instances

EC2 - NAT Instance

VPC subnetsIP Ranges - When setting up a VPC you are essentially fixing the network of the VPC.

Public and Private Subnets - The VPC network can be divided further in to smaller network segments called as Subnets. Any VPC will have at least one Subnet

You can setup a Public Subnet which will have internet connectivity. Instances launched within a Public Subnet will have both outbound and inbound (through EIP) internet connectivity through the Internet Gateway attached to the Public Subnet

Private Subnets are completely locked down. They do not have internet connectivity by default

Create number of Public and Private Subnets depending upon your architecture.

VPC security groups

AMI images

EBS volumes

autoscalingholy grail

Key to Elasticity is

in autoscaling

how to autoscale

Install AWS Command Line Tools from Amazon Downloads

Download from: http://aws.amazon.com/developertools/2535

Note: AWS Auto scaling needs Amazon CloudWatch monitoring service to function. Amazon CloudWatch is billed on usage basis.

step 1

Configuring AWS Auto Scaling with AWS ELBelb-create-lb my-load-balancer --headers --listener "lb-port=80,instance-port=8080, protocol=HTTP" --availability-zones us-west-2c

lb-port -- load balancer port

instance-port -- app server port to which request needs to be forwarded

my-load-balancer -- name for my load balancer

step 2

Create a launch configurationas-create-launch-config my-lconfig --image-id ami-e38823c8a --instance-type m1.small --key my-key-pair --group my-security-group

my-lconfig -- name for launch configuration

ami-e38823c8a -- name for Amazon Machine Image (AMI) to be launched during scaling

m1.small -- Amazon EC2 instance size

my-key-pair -- Key pair / security group settings for the Amazon EC2 instances

my-security-group -- security group for instance

step 3Create an AWS Auto Scale Groupas-create-auto-scaling-group my-as-group --availability-zones us-west-2c --launch-configuration my-lconfig --max-size 11 --min-size 3 --cooldown 180 --desired-capacity 2 --load-balancers my-load-balancer

my-load-balancer -- LB name in which the new Amazon EC2 instances launched will be attached

my-as-group -- Name Auto Scale group

us-west-2c -- availability zone in which the auto scaled amazon EC2 instances will be launched

11/3 -- Maximum/Minimum number of Amazon EC2 instances maintained by Auto Scale

Desired capacity is an important component of the as-create-auto-scaling-group command. Although it is an optional parameter, desired capacity tells Auto Scaling the number of instances you want to run initially.

To adjust the number of instances you want running in your Auto Scaling group, you change the value of --desired-capacity. If you don't specify --desired-capacity, its value is the same as minimum group size

step 4this step is not available in Auto Scaling

API

Configure the Auto scaling Triggers / Alarmsas-create-or-update-trigger my-as-trigger --auto-scaling-group my-as-group --namespace "AWS/EC2" --measure CPUUtilization --statistic Average --dimensions "AutoScalingGroupName=my-as-group" --period 60 --lower-threshold 20 --upper-treshold 80 --load-breach-increment"=-2" --upper-breach-increment 4 --breach-duration 180

Measure the average CPU of the Auto Scale Group

Scale out by 4 Amazon EC2 instances. Scale down by 2 Amazon EC instances

Lower CPU Limit is 20% and Upper CPU Limit is 80%

shutdown auto scaling group

Shutdown auto-scaling group - require 3 commandsas-update-auto-scaling-group bbd4me-as-group --min-size 0 --max-size 0 --region us-west-2

as-describe-auto-scaling-groups bbd4me-as-group --headers --region us-west-2

as-delete-auto-scaling-group bbd4me-as-group --force-delete --region us-west-2

Thank you for your attention.

Questions?