Angrybirds Magento Cloud Deployment

50
Tuesday, April 24, 2012

description

Apr 24, 2012 Deployment Pipeline for Magento Enterprise in the Cloud. The Talk covers the Amazon Cloud Infrastructure; Scaling and Autoscaling in the Cloud, the Deployment Pipeline used to do continuous deployments...

Transcript of Angrybirds Magento Cloud Deployment

Page 1: Angrybirds Magento Cloud Deployment

Tuesday, April 24, 2012

Page 2: Angrybirds Magento Cloud Deployment

Tuesday, April 24, 2012

Page 3: Angrybirds Magento Cloud Deployment

Tuesday, April 24, 2012

Page 4: Angrybirds Magento Cloud Deployment

Tuesday, April 24, 2012

Page 5: Angrybirds Magento Cloud Deployment

Angry Birds Shop Cloud Deployment

Presented byKian T. Gould and Daniel Pötzinger

Tuesday, April 24, 2012

Page 6: Angrybirds Magento Cloud Deployment

Tuesday, April 24, 2012

Page 7: Angrybirds Magento Cloud Deployment

Tuesday, April 24, 2012

Page 8: Angrybirds Magento Cloud Deployment

Irregular Traffic Behaviour

Tuesday, April 24, 2012

Page 9: Angrybirds Magento Cloud Deployment

The Angry Birds Magento Universe DRM Server

Mail ServiceProducts + Warehouse Inventory

Storefronts Reporting

Ordermgmt

Manage Warehouse Fullfillment

Giftcards

Braintree

Magento Store

Shipwire

Tuesday, April 24, 2012

Page 10: Angrybirds Magento Cloud Deployment

Angrybirds Magento Universe DRM Server

Mail ServiceProducts + Warehouse Invetory

Storefronts Reporting

Ordermgmt

Manage Warehouse Fullfillment

Giftcards

Braintree

Magento Store

Shipwire

Shipwire

Find best way for

order Fullfillment

Manages global

warehouses &

inventory

Tuesday, April 24, 2012

Page 11: Angrybirds Magento Cloud Deployment

Why Amazon Cloud ?

Tuesday, April 24, 2012

Page 12: Angrybirds Magento Cloud Deployment

✓Scales quickly on demand

✓Pay only what you need

Tuesday, April 24, 2012

Page 13: Angrybirds Magento Cloud Deployment

Powerfull AWS Services

Rightscale Services

Amazon Webservices & Amazon Console

Cloud Front

Elastic Cache

S3EC2 RDS ELB R53

Tuesday, April 24, 2012

Page 14: Angrybirds Magento Cloud Deployment

Cloud Drawbacks?

Complexity Plan for EC2 Failover

Plan Security Need to adjust Application

Lack of Service

Tuesday, April 24, 2012

Page 15: Angrybirds Magento Cloud Deployment

Cloud Drawbacks?

Complexity Plan for EC2 Failover

Plan Security Need to adjust Application

Lack of Service

Architecture Requirements

High availability (Multiple Failover

and Loadbalancing)

Magento Admin Backend

independent from Shop-Frontend

Efficient and independent

background task management

Intensive Caching

Asset management in the cloud

Tuesday, April 24, 2012

Page 16: Angrybirds Magento Cloud Deployment

Tuesday, April 24, 2012

Page 17: Angrybirds Magento Cloud Deployment

Our Cloud Architecture

Tuesday, April 24, 2012

Page 18: Angrybirds Magento Cloud Deployment

Tuesday, April 24, 2012

Page 19: Angrybirds Magento Cloud Deployment

AutoscalingVarnish Array

Frontend ArrayAutoscaling

Backend Array Worker Array

ArchitectureCloud Front

CDN for SkinELB R53

DNS

RDSmySQL

S3Deployment

Elastic CacheCache Backend

Cloud FrontCDN for Assets

S3Assets Storage

FE FE FE

Tuesday, April 24, 2012

Page 20: Angrybirds Magento Cloud Deployment

Scaling is about finding the Bottleneck

Tuesday, April 24, 2012

Page 21: Angrybirds Magento Cloud Deployment

Scaling and Tuning

Frontend ArrayAutoscaling

RDSmySQL

CloudFront

Elastic CacheCache Backend

VarnishArray

BottleneckCPU

Load Balancing

FE FE FE FE FE FE

BottleneckDatabase

Varnish

FE

BottleneckBandwidth

Varnish

BottleneckCPU

FE FE FE FE FE FE

FE FE FE FE FE FE

BottleneckDatabase

Tuesday, April 24, 2012

Page 22: Angrybirds Magento Cloud Deployment

Cloud FrontCDN for Assets

Frontend

Image Caching

Frontend Backend

S3Assets Storage

Image resized?

Read and save resized images

Upload new original images to S3

Tuesday, April 24, 2012

Page 23: Angrybirds Magento Cloud Deployment

The 6 secrets of caching

1. achieve a high hit rate 2. don‘t flood your cache3. choose your cache backends wisely4. never loose all cache at once5. clean up old cache entries6. be able to clean/refresh on demand

Tuesday, April 24, 2012

Page 24: Angrybirds Magento Cloud Deployment

Magento Performance Pointers

1. Asyncronous Cache Clearing Queue (Aoe_AsyncCache)2. Take care of Cache Cleaning (Aoe_CacheCleaner)3. Disable Magento Logs4. Make page cacheable in Varnish (Aoe_Static)5. Prefix bundeled JS and CSS with timestamp

6. and... Profile, Profile, Profile:•xdebug + kcachegrind•Newrelic•xhprof

Tuesday, April 24, 2012

Page 25: Angrybirds Magento Cloud Deployment

Magento Cloud-Adjustments

1. Compress Javascript + CSS on deploy time2. Use Deployment Name as Cache prefix3. Cache Warmup Script as part of Deployment4. Health Check for Loadbalancer5. Automatic Varnish Purging6. Use E-Mail Service7. Establish Asset handling

Tuesday, April 24, 2012

Page 26: Angrybirds Magento Cloud Deployment

Tuesday, April 24, 2012

Page 27: Angrybirds Magento Cloud Deployment

Lessons learned

1. 404s are too expensive => cache or redirect

2. Cloudfront with S3 backend cannot handle gzip => deploy time bundeling/compression

3. Don‘t hit „Reports“ in the backend :-) 4.You cannot forsee everything => But you can be prepared to fix things quickly

=> continuous deployment

Tuesday, April 24, 2012

Page 28: Angrybirds Magento Cloud Deployment

Tuesday, April 24, 2012

Page 29: Angrybirds Magento Cloud Deployment

Continuous Deployment

Tuesday, April 24, 2012

Page 30: Angrybirds Magento Cloud Deployment

Tuesday, April 24, 2012

Page 31: Angrybirds Magento Cloud Deployment

Don‘t be scared of releasing to production

Tuesday, April 24, 2012

Page 32: Angrybirds Magento Cloud Deployment

Definition of Continuous Deployment

We want on-demand deploymentof tested features and fixesin a quick and reliable way

Tuesday, April 24, 2012

Page 33: Angrybirds Magento Cloud Deployment

Automating everything

You can achieve this by

BuildTesting

Deployment

Tuesday, April 24, 2012

Page 34: Angrybirds Magento Cloud Deployment

doing it frequently

..and

Tuesday, April 24, 2012

Page 35: Angrybirds Magento Cloud Deployment

Tuesday, April 24, 2012

Page 36: Angrybirds Magento Cloud Deployment

The Deployment Pipeline

Tuesday, April 24, 2012

Page 37: Angrybirds Magento Cloud Deployment

Tuesday, April 24, 2012

Page 38: Angrybirds Magento Cloud Deployment

Selenium Acceptance Tests

Commitand Build

Build Downstream and Test Strategy

SVN Backup Storage

Unit TestsInstall on „latest“

Performance Tests

Angry Birds Deployment Pipeline

Feedback to Developers

Tuesday, April 24, 2012

Page 39: Angrybirds Magento Cloud Deployment

‣ Filesystem‣ Database‣ Install Binaries‣ settings.php

Magento Installation Package

‣ htdocs [Magento Source]

‣ media

‣ .modman [Custom Packages]

‣ changelog.txt

1.set maintenance mode

2.Delta Package prepare

3.apply Database

4.sync files

5.run modman deploy-all

6.adjust Settings

7.Post install:fix p. / cache clean / ...

Tuesday, April 24, 2012

Page 40: Angrybirds Magento Cloud Deployment

Angry Birds Deployment Pipeline

Install on Deployment

SystemCopy to S3

Cloud Deployment

Staging

Integration Tests

AOE + Rovio

Cloud deployment production

Tuesday, April 24, 2012

Page 41: Angrybirds Magento Cloud Deployment

Deployment „Version 89“

Varnish Array

AutoscalingFrontend Array

Backend Array

Worker Array

Cloud Deployment

ELB

Deployment „Version 88“

Start new release by✓ cloning old deployment✓ adjusting release number

RDSmySQL S3

Varnish Array

AutoscalingFrontend Array

Backend Array

Worker Array

ELB

WFE FE FE BE BE BE

V V VV V V

WFE FE FE BE BE BE

R53DNS

Tuesday, April 24, 2012

Page 42: Angrybirds Magento Cloud Deployment

Tuesday, April 24, 2012

Page 43: Angrybirds Magento Cloud Deployment

The real life

Tuesday, April 24, 2012

Page 44: Angrybirds Magento Cloud Deployment

Tuesday, April 24, 2012

Page 45: Angrybirds Magento Cloud Deployment

Day 1 - Release 1

Tuesday, April 24, 2012

Page 46: Angrybirds Magento Cloud Deployment

Day 2 - Release 2 - „get ready for Space“

Tuesday, April 24, 2012

Page 47: Angrybirds Magento Cloud Deployment

New release deployed

Space game launched

Array scales up

USA wakes up

Day 3 - „People want it so badly“

Tuesday, April 24, 2012

Page 48: Angrybirds Magento Cloud Deployment

40% average CPU usage.

1-2 FE Servers are enough

Day 4 - Release 3 „Performance Improvement“

Tuesday, April 24, 2012

Page 49: Angrybirds Magento Cloud Deployment

Tuesday, April 24, 2012

Page 50: Angrybirds Magento Cloud Deployment

Thank you

Follow us on Twitter:

AOE: @aoemediaDaniel P: @typo3media

Tuesday, April 24, 2012