Fire Drills: an Analog for Verifying Your Solution's Emergency Response Plans

Post on 18-Jul-2015

316 views 1 download

Transcript of Fire Drills: an Analog for Verifying Your Solution's Emergency Response Plans

Drew Cox on twitter @DrewCoxSA

Fire Drills: Do Them UNLESS YOU’RE INTO TRUSTING YOUR FUTURE TO FATE.

#FireDrill

Who am I?❖ Father of two !

❖ Husband of one!

❖ Born & Raised Texan!

❖ San Antonian for 9 years!

❖ Big Picture Person!

❖ Very Contextual!

❖ Passions: Exploration, Community, Flavor, Soccer, Urbanity & Libations

#FireDrill

Professional Background❖ 4 Years @ Rackspace, all in the cloud!

❖ Weekly show focused on best practice!

❖ Worked with hobbyists & Fortune 500!

❖ Passion for startups !

❖ Major award shows!

❖ Shark Tank!

❖ ‘Reddit’ AKA ‘Oprah Effect’!

❖ Super Bowl Ads!

❖ Strengths: Ideation, Maximizer, Arranger, Connectedness, & Intellection

#FireDrill

Goal: I hope to instill in each of you a healthy fear

& a healthier curiosity

#FireDrill

Goal: I hope to instill in each of you a healthy fear

& a healthier curiosity

#FireDrill

Analogies are like…

#FireDrill

What if…

#FireDrill

Or what if…

a server fails?

#FireDrill

Or what if…

a server fails?

you get hacked?

#FireDrill

Or what if…

a server fails?

you get hacked?

you hit the front page of reddit?!?

#FireDrill

Or what if…

a server fails?

you get hacked?

you hit the front page of reddit?!?

you miss an alert because you phone was off?

#FireDrill

Or what if…

a server fails?

you get hacked?

you hit the front page of reddit?!?

auto scale = auto fail?

you miss an alert because you phone was off?

#FireDrill

Or what if…

a server fails?

you get hacked?

you hit the front page of reddit?!?

you get DDoS’d?

auto scale = auto fail?

you miss an alert because you phone was off?

#FireDrill

Or what if…

a server fails?

you get hacked?

you hit the front page of reddit?!?

your domain expires?

you get DDoS’d?

auto scale = auto fail?

you miss an alert because you phone was off?

#FireDrill

Or what if…

a server fails?

you get hacked?

you hit the front page of reddit?!?

your domain expires?

you get DDoS’d?

auto scale = auto fail?

you miss an alert because you phone was off?

my developer goes rogue?

#FireDrill

Or what if…

a server fails?

you get hacked?

you hit the front page of reddit?!?

backups are corrupted?

your domain expires?

you get DDoS’d?

auto scale = auto fail?

you miss an alert because you phone was off?

my developer goes rogue?

#FireDrill

What if…

#FireDrill

“Safety Rules”

#FireDrill

Elementary School

#FireDrill

#FireDrill

Middle School

#FireDrill

#FireDrill

High School

#FireDrill

#FireDrill

College

#FireDrill

#FireDrill

#FireDrill

“Trust, But Verify” -Ronald Reagan

#FireDrill

“Trust, But Verify” -Ronald Reagan

#FireDrill

“Trust, But Verify” -Ronald Reagan’s !

Translation of a Russian Proverb

#FireDrill

#FireDrill

#FireDrill

Draw a map of your home. Show all doors and

windows.Know your site

#FireDrill

Visit each room. Find two ways out.

Review Each Component &

Have a Fall Back#FireDrill

All windows and doors should open easily. You should be able

to use them to get outside.Identify Choke

Points

#FireDrill

Make sure your home has smoke alarms. Push the test button to

make sure each alarm is working.Verify Monitoring

Solution

#FireDrill

Pick a meeting place outside. It should be in front of your home.

Everyone will meet at the meeting place.

Agree on a Collaboration

Platform#FireDrill

Make sure your house or building number can be

seen from the street.Brand a !

“Sorry Page”

#FireDrill

Talk about your plan with everyone in your home.

Your team needs to know the plan

#FireDrill

Learn the emergency phone number for your fire

department. Know Your

Escalation Path

#FireDrill

Practice your home fire escape drill!

I think you’ve got this one…

#FireDrill

❖ Monitoring!

❖ Recovery Time (reboot)!

❖ Fail Over!

❖ Backup!

❖ Scale Up!

❖ Scale Down!

❖ Traffic Spike!

❖ DDoS Attack !

!

!

What Needs Verification?❖ Domain Expiration!

❖ SSL Expiration!

❖ Payment Gateway is down!

❖ Higher Employee!

❖ Fire Employee!

❖ Compromise !

❖ Re-Seller Conflict!

❖ Etcetera

#FireDrill

❖ Monitoring!

❖ Recovery Time (reboot)!

❖ Fail Over!

❖ Backup!

❖ Scale Up!

❖ Scale Down!

❖ Traffic Spike!

❖ DDoS Attack !

!

!

What Needs Verification?❖ Domain Expiration!

❖ SSL Expiration!

❖ Payment Gateway is down!

❖ Higher Employee!

❖ Fire Employee!

❖ Compromise !

❖ Re-Seller Conflict!

❖ Etcetera

#FireDrill

Fire Drill Process1. Make a plan!

2. Document your plan!

3. Schedule the fire drill (off hours)!

4. Run the fire drill!

5. Identify room for improvement!

6. Implement changes!

7. Repeat

#FireDrill

Tips & TricksHistorical issues are great places to start!

!

!

!

!

!

#FireDrill

Tips & Tricks

#FireDrill

Tips & TricksTest run all patches & upgrades pre-prod!

!

!

!

!

!

#FireDrill

Tips & TricksIdentify high value/risk portions of your site!

!

!

!

!

!

#FireDrill

Tips & TricksModular solutions can be tested in isolation

#FireDrill

Tips & TricksLoad test to mitigate for high traffic events

#FireDrill

Fire Drills: Do Them!

#FireDrill

#FireDrill