Putting a Red Nose on the Cloud · About Comic Relief o Comic Relief is a major charity based in...

33
Putting a Red Nose on the Cloud

Transcript of Putting a Red Nose on the Cloud · About Comic Relief o Comic Relief is a major charity based in...

Page 1: Putting a Red Nose on the Cloud · About Comic Relief o Comic Relief is a major charity based in the UK which strives to create a just world free from poverty o Since we first set

Putting a Red Nose on the Cloud

Page 2: Putting a Red Nose on the Cloud · About Comic Relief o Comic Relief is a major charity based in the UK which strives to create a just world free from poverty o Since we first set

About Comic Relief o  Comic Relief is a major charity based in the UK which

strives to create a just world free from poverty o  Since we first set up shop in 1985, we’ve been doing

three main things: o  We raise millions of pounds through two big

fundraising campaigns – Red Nose Day and Sport Relief.

o  We spend that money in the best possible way to tackle the root causes of poverty and social injustice.

o  We use the power of our brand to raise awareness of the issues that we care most about.

Page 3: Putting a Red Nose on the Cloud · About Comic Relief o Comic Relief is a major charity based in the UK which strives to create a just world free from poverty o Since we first set

o  Every two years, we encourage thousands of people to do something funny for money.

o  A year of planning o  6 week media campaign o  7 hours of TV on the 15th March

Page 4: Putting a Red Nose on the Cloud · About Comic Relief o Comic Relief is a major charity based in the UK which strives to create a just world free from poverty o Since we first set

What we had

o  8 year old Java application o  Deployed and scaled with the help of 12 partners o  Took months to achieve this, run through user testing,

penetration testing and authentication o  Changes were kept to an absolute minimum between

years for stability and to reduce risk

Page 5: Putting a Red Nose on the Cloud · About Comic Relief o Comic Relief is a major charity based in the UK which strives to create a just world free from poverty o Since we first set

Key Aims of New Platform

o  Unlimited by technology o  Minimise PCI exposure o  Remove reliance on any single third party supplier o  Cost-effective

o  All the money raised by the public is spent by Comic Relief to help poor and disadvantaged people in the UK and the world's poorest countries.

Page 6: Putting a Red Nose on the Cloud · About Comic Relief o Comic Relief is a major charity based in the UK which strives to create a just world free from poverty o Since we first set

What we have now Reminder: QCon Session Code : 9221 Over to you Tim...

Page 7: Putting a Red Nose on the Cloud · About Comic Relief o Comic Relief is a major charity based in the UK which strives to create a just world free from poverty o Since we first set

Thanks Zenon... This talk is a case study that intends to:

o  Give you an insight into the solution we have delivered over the last 9 months

o  Discuss the patterns we have applied and how we (and as a consequence, Comic Relief) have benefitted

Page 8: Putting a Red Nose on the Cloud · About Comic Relief o Comic Relief is a major charity based in the UK which strives to create a just world free from poverty o Since we first set

Platform Requirements o  The platform is required to:

o  serve a donation page for the public o  manage a lightweight call centre interface o  process in the region of 600,000 transactions in 7

hours o  handle in excess of 10,000 call centre operators o  handle a peak of 300 donations completing per

second o  be out of scope for PCI

Page 9: Putting a Red Nose on the Cloud · About Comic Relief o Comic Relief is a major charity based in the UK which strives to create a just world free from poverty o Since we first set

What does that look like?

Page 10: Putting a Red Nose on the Cloud · About Comic Relief o Comic Relief is a major charity based in the UK which strives to create a just world free from poverty o Since we first set

Donations Per Minute

Page 11: Putting a Red Nose on the Cloud · About Comic Relief o Comic Relief is a major charity based in the UK which strives to create a just world free from poverty o Since we first set

Donations Per Second

Page 12: Putting a Red Nose on the Cloud · About Comic Relief o Comic Relief is a major charity based in the UK which strives to create a just world free from poverty o Since we first set

Challenges o  We don't get a second chance o  Its only used once a year for 7 hours

Page 13: Putting a Red Nose on the Cloud · About Comic Relief o Comic Relief is a major charity based in the UK which strives to create a just world free from poverty o Since we first set

Previous Issues o  Testing, Integration and deployment problems o  Lack of consistency o  Single Points of Failure

o  Infrastructure provider o  Platform & Networking o  Bandwidth

o  Multiple provider relationships

o  1 year feedback cycle

Page 14: Putting a Red Nose on the Cloud · About Comic Relief o Comic Relief is a major charity based in the UK which strives to create a just world free from poverty o Since we first set

Solution Patterns o  Distributed architecture o  Multiple Infrastructure as a Service (IaaS) o  Multiple Platform as a Service (PaaS) o  Stateless pattern o  Eventually consistent data o  Minimum Time to Recovery

Page 15: Putting a Red Nose on the Cloud · About Comic Relief o Comic Relief is a major charity based in the UK which strives to create a just world free from poverty o Since we first set

Solution Patterns Stateless/Eventual Consistency

o  No High Availability datastore o  Message Queue architecture o  Enables a distributed architecture

Page 16: Putting a Red Nose on the Cloud · About Comic Relief o Comic Relief is a major charity based in the UK which strives to create a just world free from poverty o Since we first set

Solution Patterns PaaS & IaaS

o  PaaS o  Homogenised platform o  Enables multi Iaas

o  Multi IaaS o  Costs benefits for Comic Relief o  Prevents vendor lock in for Comic Relief o  Enabled rapid rollout of supporting applications

Page 17: Putting a Red Nose on the Cloud · About Comic Relief o Comic Relief is a major charity based in the UK which strives to create a just world free from poverty o Since we first set

Solution Patterns Minimum time to recovery

o  History o  Build for failure o  Reduce time to recovery

Page 18: Putting a Red Nose on the Cloud · About Comic Relief o Comic Relief is a major charity based in the UK which strives to create a just world free from poverty o Since we first set

Commoditise Dependencies o  Dependency on 3rd parties o  Usage commoditised o  IAAS

o  We can easily deploy across multiple service providers

o  Info provided by OpenCloudBrokers o  Payment Service Providers

o  We load balance across multiple providers, allowing us to ensure that our service is continuous, and able to cope with projected loads.

Page 19: Putting a Red Nose on the Cloud · About Comic Relief o Comic Relief is a major charity based in the UK which strives to create a just world free from poverty o Since we first set

Insight Layer

What does the platform look like? Internet

DNS

PaaS 1 - AWS US Cloud Foundry

(BOSH)

PaaS 1 - AWS EU Cloud Foundry

(BOSH)

Service Layer

Workers

View API

Insight

Presentation Layer

Service Layer

Workers

MGMT

PaaS 1 - Cz Cloud Foundry

(BOSH)

Presentation Layer

Service Layer

Workers

Shared Services

Logging

Metrics

Alerting

=

+ PLUS

Presentation Layer

Page 20: Putting a Red Nose on the Cloud · About Comic Relief o Comic Relief is a major charity based in the UK which strives to create a just world free from poverty o Since we first set

Pipelines Continuous Deployment to Production

o  2 pipelines integrated o  Infrastructure o  Applications

o  Converging on multiple test platforms o  Development team managing services

Page 21: Putting a Red Nose on the Cloud · About Comic Relief o Comic Relief is a major charity based in the UK which strives to create a just world free from poverty o Since we first set

Local changes to deployed platform

Pipeline - Infrastructure

$$$

Page 22: Putting a Red Nose on the Cloud · About Comic Relief o Comic Relief is a major charity based in the UK which strives to create a just world free from poverty o Since we first set

Pipeline - Applications Local changes to deployed platform

$$$

Page 23: Putting a Red Nose on the Cloud · About Comic Relief o Comic Relief is a major charity based in the UK which strives to create a just world free from poverty o Since we first set

Continuous Integration Testing The value in our pipeline comes from the testing that gives

us confidence in the consistency of our solution

o  RSpec - unit tests o  Cucumber - feature/integration tests o  ZAProxy - security tests o  Grinder - benchmarking load tests

Page 24: Putting a Red Nose on the Cloud · About Comic Relief o Comic Relief is a major charity based in the UK which strives to create a just world free from poverty o Since we first set

Other Testing Load Testing o  In addition to small scale load testing as part of our CI

deployments o  Grinder, using chef to deploy o  20 minutes lead time, up to 120 nodes used, 60,000

concurrent users (zero wait times) o  Global capability

Page 25: Putting a Red Nose on the Cloud · About Comic Relief o Comic Relief is a major charity based in the UK which strives to create a just world free from poverty o Since we first set

Failure Tolerant o  DNS round robin across multiple shards o  Scripted DNS enabling a measure of load balancing o  "Failure wagons" standing in in case of shard failure and

handing off to alternate shards

Page 26: Putting a Red Nose on the Cloud · About Comic Relief o Comic Relief is a major charity based in the UK which strives to create a just world free from poverty o Since we first set

Failure Tolerant o  Minimum time to recovery vs high availability (HA)

o  Eventual consistency o  Stateless requests o  Message queue architecture

o  Expecting failure

Page 27: Putting a Red Nose on the Cloud · About Comic Relief o Comic Relief is a major charity based in the UK which strives to create a just world free from poverty o Since we first set

Solution Challenges

o  Reliance on inflexible third-party providers o  Multiple payment providers, we are able to ensure that

we have the redundancy we need. o  Managing and automating complexity

Page 28: Putting a Red Nose on the Cloud · About Comic Relief o Comic Relief is a major charity based in the UK which strives to create a just world free from poverty o Since we first set

Flexibility - Load testing Performance confidence - results (TPS)

0

100

200

300

400

500

Red

is C

onfig

Add

ed D

EA

s

Add

ed D

EA

s

Incr

ease

d lo

ad te

st

thre

ads

Mov

ed lo

ad te

st

plat

form

to E

U

Add

ed H

A p

roxy

&

3 N

ginx

nod

es

Incr

ease

d lo

ad te

st

thre

ads

8 N

ginx

nod

es

501

Monday 10/9 Tuesday 11/9

Tran

sact

ions

per

sec

ond

Page 29: Putting a Red Nose on the Cloud · About Comic Relief o Comic Relief is a major charity based in the UK which strives to create a just world free from poverty o Since we first set

Flexibility - Supporting Platforms

Whilst building the main platform, we have also built a range of supporting platforms, including:

o  Payment provider mocks (>= 500 Donations/sec) o  An email service mock o  A data api mock o  Globally-distributed load test platform

(zero to hero in 20 minutes)

Page 30: Putting a Red Nose on the Cloud · About Comic Relief o Comic Relief is a major charity based in the UK which strives to create a just world free from poverty o Since we first set

Flexibility - Payment Service Providers

o  We have performed implementations with 11 different payment providers/interfaces, (several of which are not being used.)

o  These 3rd party integrations are key to the delivery of our service, and so this enabled us to really understand how they worked, what performance issues we might encounter.

Page 31: Putting a Red Nose on the Cloud · About Comic Relief o Comic Relief is a major charity based in the UK which strives to create a just world free from poverty o Since we first set

The part that's missing!

o  no actual data/results o  please watch this space o  only 9 days to go

o  The last 9 months have been tough but fun o  The pipelines, once created, have been the driving force

of this project o  3rd party service commoditisation has allowed Comic

Relief to stay in control of the risk

o  Thank you

Page 32: Putting a Red Nose on the Cloud · About Comic Relief o Comic Relief is a major charity based in the UK which strives to create a just world free from poverty o Since we first set

In Conclusion o  QCon is two weeks too soon o  By using the cloud we have put ourselves in a strong

position o  New Platform will only be proven on 15th March o  We have a back-up platform built by BT

Don't forget to use the engage feature on the QCon app to rate the talk and ask questions

Page 33: Putting a Red Nose on the Cloud · About Comic Relief o Comic Relief is a major charity based in the UK which strives to create a just world free from poverty o Since we first set

[email protected] @zenonhannick

[email protected] @timjsavage