Scaling Drupal on

Amazon Web Services

Tristan Roddis DrupalCamp Brighton, January 2015

What we created: www.qdl.qa

2

3

4

5

• Several hundred thousand pages

• Tens of terabytes of data

• Requirement for high throughput

• Requirement for high resilience

6

What is Drupal?

7

Let’s scale it!

8

Let’s not scale it like this!

9

Centralised database

10

Centralised filesystem

11

Avoid single point of failure

12

AWS TLAs

• EC2 – Elastic Compute Cloud = virtual servers

• RDS – Relational Database Service = database

• ELB – Elastic Load Balancer = distribute requests

13

Therefore…

14

RDS has easy redundancy

15

So now we have

16

Clustered filesystem: GlusterFS

17

GlusterFS components

• EBS – Elastic Block Storage

= file storage up to 1Tb

• LVM – Logical Volume Manager

= grow and shrink volumes

• Gluster has storage ‘bricks’

• Cluster appears as single mount point

(NFS, Gluster protocol etc.)

18

Distribute servers

• Amazon has different Regions and

Availability Zones

19

Use different Availability Zones

20

I lied…

21

Adding Solr and IIPImage…

22

Additional security

• VPC – Virtual Private Cloud = can create your

own network architecture

• Security Groups = firewall rules

• IAM = Identity and Access Management =

restrict AWS roles

23

Final setup

24

Woah – that’s complicated

25

Server provisioning

• Automate everything!

• Use Ansible (or Chef, Puppet, etc.)

26

“If you have to SSH into your servers,

then your automation has failed.”

— Rich Adams

27

Pets vs cattle

28

“When one of them gets sick, you shoot

'em in the head and replace 'em with a

new one.”

—Randy Bias, CEO of Cloudscaling

Deployment I: Drupal tools

• Drush

• Drush master

• Features

• Strongarm

29

Deployment II

• Use Fabric (or Ansible, Capistrano, etc.)

• @runs_once decorator for e.g. drush commands.

30

Load Test

• Jmeter

• Blitz.io or similar

31

Tune performance

• Tools: Apache Benchmark (ab), Yslow, Google

PageSpeed

• xhprof – remove bottlenecks

• Fix warnings and notices

• Disable database logging

• OpCode cache (Zend Optimizer, APC)

• Minify CSS/JS

• Lazy Loading via AJAX

32

33

Add caching

• Varnish

• Redis/memcached for cache tables (can use

Amazon Elasticache)

• CDN (e.g. Amazon Cloudfront)

• Re-run load tests

34

[auto]scaling

• AMIs = Amazon Machine Image

• Autoscaling policies

• “pets versus cattle” again

• Problem: servers must have latest Drupal code

• Conflict with Fabric 'push‘ technique

• Interim solution: create an AMI after

each deployment

35

Backup

• AMIs - Amazon Machine Image = snapshot

server images

• ELB snapshots

• S3 – Simple Storage Service = REST storage

• Glacier = long-term storage

• We used:

– Snapshots with RDS

– S3 with versioning, plus Glacier for Drupal

assets and image assets

36

Monitoring I: Cloudwatch alerts

37

Monitoring II: log analysis

• SumoLogic to aggregate logs (or Loggly,

Papertrail, etc.)

38

Monitoring III: notifications

• OpsGenie (or PagerDuty or similar)

39

Gotchas

• GlusterFS for PHP files – don’t!

• Custom Drupal cache – careful of

uncontrolled growth!

40

Tips

• Use “termination protection”

• Enable ELB logging

• “Maintenance 200” module

• Create a “down for maintenance” server

41

Conclusion

• 21 instances

• 6 load balancers

• 67 ELB volumes

• Lot of work to set up, but quick to alter

• Easily coped with launch traffic:

– 21k sessions/day

– peak of 350 concurrent users

42

Further Reading

• Justin Slattery, "Multiple Region Autoscaling Drupal in Amazon Web Services" http://dev.mlsdigital.net/posts/Cloud-Native-Drupal/

• Rich Adams, "AWS Tips I Wish I'd Known Before I Started" https://wblinks.com/notes/aws-tips-i-wish-id-known-before-i-started/

• Laura Hamilton, "Are your servers pets or cattle?" http://www.lauradhamilton.com/servers-pets-versus-cattle

43

www.cogapp.com/careers

Questions?

• tristanr@cogapp.com

45