SCALABLE DRUPAL SITES

USING AMAZON WEB SERVICES

Amazon Web Services commonly referred to as AWS.

INTROSHANE DAVIS

Service Delivery Manager, Lead Architect ANZ

AWS Advanced consulting partnerMember of CKAN (Open Data platform) & Drupal associationsAWS certified

INTROLINK DIGITAL CLIENTS

Australians At War Film Archive

Federal and State Govt. clientsAll Australia’s major Open Data Portals; data.gov.au, data.vic.gov.au data.nsw.gov.au, data.sa.gov.au, data.wa.gov.auSuncorp Internal BI platformNCI Supercomputing facilityUNSW Australians at War Film Archive - Scalable Drupal implementation with 11,000 hours of video.

PET MENTALITYPRECIOUS, DELICATE, IRREPLACABLE

When I started in IT, this was the state of the art PC. Everything was hand coded and time consuming to set up.People have DR plans because they’re scared of losing their one off masterpieces. Their pets.

CLOUD COMPUTING

WHAT’S CHANGED?

• Desktop PCs

• Server in a cupboard

• Virtualisation

• Virtual Dedicated Servers

• Cloud Computing

Over the last 30 years we’ve moved from desktops to the cloud in progressive steps.Server in the cupboard during the early 2000’s, VMWare virtualisation from 2005 - 2010/11Cloud Computing 2011 onwards.

CLOUD BENEFITS

• Low Cost

• Agile

• Easily Managed

Cents per hour to run

Pay for what you use

Scales up and down easily

Point and Click: • Provisioning • Monitoring • Management

40 core server on spot market as little as 08c/hour. 80c per hour for a 10 node computing cluster with 400 cores.Cloud allows environments to scale up and down within a few short minutes. The time it takes to boot a linux server.True single glass of pane management over all your computing and network services. In traditional hosting this is usually very complicated to implement, and even more costly. It comes free with Amazon Web Services. Even a novice can point and click to provision a datacenter.

PREREQUISITES FOR SCALING

CATTLE VS PETS

• Standard server images

• Independent App Layers

• Dynamic IPs and DNS

• Stand alone data and files

In order to scale, we need to get away from pet mentality. We need:Standardised server images that can be brought online quicklyIndependent App Layers so web layers can be scaled independently of DB LayersDynamic IPs and DNS so our environment is flexibleStand alone data so servers can come and go without data loss

SCALING ARCHITECTUREWHAT A SCALABLE BUILD LOOKS LIKE

ELB + AMI IN ASG = SCALING

The typical scalable build has:Dynamic DNS feeding traffic to an Elastic Load BalancerAuto scaling web instances with our web app stored in an Amazon Machine Image (AMI)An HA Database backend like Amazon Relational Database Service (RDS)

SCALING DRUPALFOUR KEY PIECES

Drupal Core

Database

/sites/all - Modules and Themes

/sites/…/files - Content

There are 4 key pieces to a Drupal site that all need separate consideration.Drupal core which needs to be updated from time to timeThe database - should be Highly Available (HA)Your custom modules and themes which may have ongoing developmentThe content and files uploaded by you and users

SCALING EACH PIECECHOOSING HORSES FOR COURSES

Drupal Core• Changes Infrequently • Usually Security Updates • Almost Static Content

Bake into the base image

AMIs provide a machine image new instances are started from with NGINX, PHP and config all baked into them. AMI based instances start within a minute or so.

SCALING EACH PIECECHOOSING HORSES FOR COURSES

Database• Should be Highly Available • Needs to be a separate layer • Use RDS Multi-AZ

RDS can be scaled vertically if necessary. For high load sites, Read Replicas may be added which is easy to do with RDS.Typically the DB sees very little load. If 4 web nodes are around 80% CPU load, the DB on a medium sized instance will be no more than 10-15% CPU load.

SCALING EACH PIECECHOOSING HORSES FOR COURSES

Themes and Modules• May Change Regularly • Version Controlled • Ideally centrally located

NFS/GlusterFS/Elastic File Store

There are many options here, all with pros and cons. The best is Elastic File Store.NFS is simple, but typically has a single point of failure and is difficult to sync across a clustered setup.Baking rapidly changing themes and modules into the AMI may mean a lot of updates to the image. Manageable but could be annoying unless scripted. (https://github.com/Datashades/updateasg) provides a solution.GlusterFS is a little complicated to set up, but robust.Elastic File Store is the best choice but limited to 3 regions at time of writing.

SCALING EACH PIECECHOOSING HORSES FOR COURSES

User Content• Changes Often • Must be consistent • Can’t be lost!

Amazon S3 or NFS

S3 is the best choice with 11 9’s of reliability. AMSA solved their MH-370 issues with S3 in less than 30 minutes. By offloading video media releases to S3, they eliminated all heavy traffic from their internal web servers.NFS is a simple solution but prone to single point of failure or replication issuesEFS is a good option when available but more expensive than S3. On a cost basis alone, S3 is the better choice.

AUTO SCALINGAUTO IS THE KEY

Scheduled Policies Load Based Policies

Scaling Health Checks

ASG allows your site to scale up and down by setting a maximum and minimum number of instances.ASG ensures healthy instances are kept online. It provides Auto recovery even for single instances inside an Auto scale group.Scheduled policies control cost by reducing the minimum instances running outside peak hours.Load based policies manage sudden demand on your site. You can monitor CPU, but site latency gives a more accurate indicator that your site needs to scale up.

AWS FOR THE WIN!COOL THINGS ARE POSSIBLE

Application Load Balancer

• Admin Traffic • User traffic • Public traffic

The great thing about AWS is they’re always delivering new services.

The Application ELB allows URLs to be directed to different groups of servers, making scaling even more flexible.A good use case for this is sending admin users to a separate group of instances with independent scaling rules. Authenticated users to another scaling group, and general users to your main pool of instances. This ensures everybody gets the best experience, and your costs are even better controlled by not having to scale out your entire build because of a single user type creating one off load.

OPSWORKSINFRACODERS ORCHESTRATION TALK

13th September Geoscience Australia

• Benefits of Orchestration • Deep dive into OpsWorks • Implementing CI

https://au.linkedin.com/in/kiwicloudninja

http://linkdigital.com.au

This talk will do a deeper technical dive into the benefits of automated provisioning and system orchestration using AWS OpsWorks to create HA scalable sites. It will explain how this allows you to implement continuous integration.Connect with me on LinkedIn: https://au.linkedin.com/in/kiwicloudninjaCompany website: http://linkdigital.com.au