Image: xkcd.com

Dependable Cloud Architecture

@mikewo

Mike Wood

http://mvwood.com

“Failure is alwaysan option.”

Image: Discovery Channel, Fair Use

Protection From:

What are we looking for?

Check out: http://bit.ly/wazbizcontImages: Office ClipArt & Godzilla Releasing Corp (Fair Use)

Hardware Failure Data Corruption Network Failure Loss of Facilities

Image: FOX, Fair Use

Human Error

What we’re trying to achieve

1. Monitoring2. Resilient Solutions

Image: Office ClipArt

Cost vs Risk

99.999% $1, … ,000.00

To get more 9’s here add more 0’s here.

Image: NASA

Monitoring

Functional Transparency

Image: Office ClipArt

Logging Messages

Hardware Health

Dependent Services Health

Telemetry

Image: NASA

Analyze your Data

Resilience

Remember: Failure is always an option.

Common Points of Failure• Machine\application crashes• Throttling (exceeding capacity)• Connectivity\Network• External service dependencies

Focus less on the uptime of hardware and more about how the solution handles it WHEN

something fails!

Try/catch != Resilient

Image: Michael Wood

Decompose your system…

Request bufferingRetry Policies

• Wait and try again• Queue until available

Queuing Enables• Asynchronous workloads• Temporal Decoupling• Load Levelling

Check out: http://bit.ly/wazrequestbuffer

Capacity BufferingContent Delivery Networks (CDN’s)

Distributed Application Cache

Local Content Cache

Enables recovery during outages or

spikes in load

Dynamic Addressing & Configuration

Dept. of Redundancy Dept.

Have a backup, somewhere elseMore than one? Cost to benefit Ratio?

Ready StateHot = full capacityWarm = scaled down, but ready to growCold = mothballed, starts from zero

Image: Mr. White

Redundancy - Its about probability

95% uptime 95% uptime 95% uptime 95% uptime

1 box : 5% downtime or 438hrs per year

2 boxes : 5/100 * 5/100 = 25/10,000 = 0.25% downtime or 22hrs per year

4 boxes : 5/100 * 5/100 * 5/100 * 5/100 = 625/100,000,0000.000625% downtime or 3.285 MINUTES per year

(that’s 18 ½ days!)

Always carry a spare75% Capacity, half of our load 75% Capacity, half of our load

50% more capacity then needed• Can absorb of temporary spikes• Time to react if need to add capacity

100% of load, 150% Capacity0% Capacity, redirect all load

Over allocated, but still functioning• Degrade, but don’t fail

SYSTEM FAILURE!!!

Accessible vs. Available

Image: Twitter, Fair Use

Availability via Degradation

Image: Michael Wood

Total Outage duration =

Time to Detect+ Time to Diagnose+ Time to Decide+ Time to ActImage: Office ClipArt

Images: Gizmodo

Virtualization and Automation

Images: Orion Pictures owns Terminator Franchise

The “HI” Point

Check out: http://bit.ly/wazinternals

Image: NASA

“Don't be too proud of this technological terror you've constructed…”

ADMIT:• Your Solution WILL fail at some point• You can learn from others just as

well as yourself

DO:• Root cause analysis• Read other root cause analysis

DON’T:• Get cocky• Stick your head in the sand

Questions

@mikewo

Mike Wood

http://mvwood.com

http://bit.ly/CloudFailSafe