Dependable Cloud Architecture - Cloud Develop Edition
29
Image: xkcd.com Dependable Cloud Architecture @mikewo Mike Wood http:// mvwood.com
-
Upload
michael-wood -
Category
Technology
-
view
587 -
download
0
description
This is a talk I gave at Cloud Develop 2013. It was an adapted from a workshop session that Brent Stineman and I did in Jan of 2013 for CodeMash.
Transcript of Dependable Cloud Architecture - Cloud Develop Edition
Image: xkcd.com
Dependable Cloud Architecture
@mikewo
Mike Wood
http://mvwood.com
“Failure is alwaysan option.”
Image: Discovery Channel, Fair Use
Protection From:
What are we looking for?
Check out: http://bit.ly/wazbizcontImages: Office ClipArt & Godzilla Releasing Corp (Fair Use)
Hardware Failure Data Corruption Network Failure Loss of Facilities
Image: FOX, Fair Use
Human Error
What we’re trying to achieve
1. Monitoring2. Resilient Solutions
Image: Office ClipArt
Cost vs Risk
99.999% $1, … ,000.00
To get more 9’s here add more 0’s here.
Image: NASA
Monitoring
Functional Transparency
Image: Office ClipArt
Logging Messages
Hardware Health
Dependent Services Health
Telemetry
Image: NASA
Analyze your Data
Resilience
Remember: Failure is always an option.
Common Points of Failure• Machine\application crashes• Throttling (exceeding capacity)• Connectivity\Network• External service dependencies
Focus less on the uptime of hardware and more about how the solution handles it WHEN
something fails!
Try/catch != Resilient
Image: Michael Wood
Decompose your system…
Request bufferingRetry Policies
• Wait and try again• Queue until available
Queuing Enables• Asynchronous workloads• Temporal Decoupling• Load Levelling
Check out: http://bit.ly/wazrequestbuffer
Capacity BufferingContent Delivery Networks (CDN’s)
Distributed Application Cache
Local Content Cache
Enables recovery during outages or
spikes in load
Dynamic Addressing & Configuration
Dept. of Redundancy Dept.
Have a backup, somewhere elseMore than one? Cost to benefit Ratio?
Ready StateHot = full capacityWarm = scaled down, but ready to growCold = mothballed, starts from zero
Image: Mr. White
Redundancy - Its about probability
95% uptime 95% uptime 95% uptime 95% uptime
1 box : 5% downtime or 438hrs per year
2 boxes : 5/100 * 5/100 = 25/10,000 = 0.25% downtime or 22hrs per year
4 boxes : 5/100 * 5/100 * 5/100 * 5/100 = 625/100,000,0000.000625% downtime or 3.285 MINUTES per year
(that’s 18 ½ days!)
Always carry a spare75% Capacity, half of our load 75% Capacity, half of our load
50% more capacity then needed• Can absorb of temporary spikes• Time to react if need to add capacity
100% of load, 150% Capacity0% Capacity, redirect all load
Over allocated, but still functioning• Degrade, but don’t fail
SYSTEM FAILURE!!!
Accessible vs. Available
Image: Twitter, Fair Use
Availability via Degradation
Image: Michael Wood
Total Outage duration =
Time to Detect+ Time to Diagnose+ Time to Decide+ Time to ActImage: Office ClipArt
Images: Gizmodo
Virtualization and Automation
Images: Orion Pictures owns Terminator Franchise
The “HI” Point
Check out: http://bit.ly/wazinternals
Image: NASA
“Don't be too proud of this technological terror you've constructed…”
ADMIT:• Your Solution WILL fail at some point• You can learn from others just as
well as yourself
DO:• Root cause analysis• Read other root cause analysis
DON’T:• Get cocky• Stick your head in the sand
Questions
@mikewo
Mike Wood
http://mvwood.com
http://bit.ly/CloudFailSafe
