DevOps Incident Handling - Making friends not enemies.
-
Upload
server-density -
Category
Technology
-
view
353 -
download
0
description
Transcript of DevOps Incident Handling - Making friends not enemies.
![Page 1: DevOps Incident Handling - Making friends not enemies.](https://reader033.fdocuments.in/reader033/viewer/2022052907/558e7fdb1a28ab930b8b4656/html5/thumbnails/1.jpg)
How to win friends when handling outages and downtime
David MyttonLondon DevOps - Oct 2014
blog.serverdensity.com
![Page 2: DevOps Incident Handling - Making friends not enemies.](https://reader033.fdocuments.in/reader033/viewer/2022052907/558e7fdb1a28ab930b8b4656/html5/thumbnails/2.jpg)
David Mytton
![Page 3: DevOps Incident Handling - Making friends not enemies.](https://reader033.fdocuments.in/reader033/viewer/2022052907/558e7fdb1a28ab930b8b4656/html5/thumbnails/3.jpg)
Server monitoring, cloud management, dashboards and alerting
serverdensity.com
![Page 4: DevOps Incident Handling - Making friends not enemies.](https://reader033.fdocuments.in/reader033/viewer/2022052907/558e7fdb1a28ab930b8b4656/html5/thumbnails/4.jpg)
Slides: twitter.com/davidmytton
![Page 5: DevOps Incident Handling - Making friends not enemies.](https://reader033.fdocuments.in/reader033/viewer/2022052907/558e7fdb1a28ab930b8b4656/html5/thumbnails/5.jpg)
Let’s talk about downtime
![Page 6: DevOps Incident Handling - Making friends not enemies.](https://reader033.fdocuments.in/reader033/viewer/2022052907/558e7fdb1a28ab930b8b4656/html5/thumbnails/6.jpg)
2013 Spend: ~$5bn
![Page 7: DevOps Incident Handling - Making friends not enemies.](https://reader033.fdocuments.in/reader033/viewer/2022052907/558e7fdb1a28ab930b8b4656/html5/thumbnails/7.jpg)
2013 Spend: ~$6bn
![Page 8: DevOps Incident Handling - Making friends not enemies.](https://reader033.fdocuments.in/reader033/viewer/2022052907/558e7fdb1a28ab930b8b4656/html5/thumbnails/8.jpg)
2013 Spend: ~$4bn
![Page 9: DevOps Incident Handling - Making friends not enemies.](https://reader033.fdocuments.in/reader033/viewer/2022052907/558e7fdb1a28ab930b8b4656/html5/thumbnails/9.jpg)
You will have downtime
How much do you spend?
![Page 10: DevOps Incident Handling - Making friends not enemies.](https://reader033.fdocuments.in/reader033/viewer/2022052907/558e7fdb1a28ab930b8b4656/html5/thumbnails/10.jpg)
![Page 11: DevOps Incident Handling - Making friends not enemies.](https://reader033.fdocuments.in/reader033/viewer/2022052907/558e7fdb1a28ab930b8b4656/html5/thumbnails/11.jpg)
Preparation
![Page 12: DevOps Incident Handling - Making friends not enemies.](https://reader033.fdocuments.in/reader033/viewer/2022052907/558e7fdb1a28ab930b8b4656/html5/thumbnails/12.jpg)
Preparation - On Call
● Primary?
![Page 13: DevOps Incident Handling - Making friends not enemies.](https://reader033.fdocuments.in/reader033/viewer/2022052907/558e7fdb1a28ab930b8b4656/html5/thumbnails/13.jpg)
Preparation - On Call
● Primary?
● Secondary?
![Page 14: DevOps Incident Handling - Making friends not enemies.](https://reader033.fdocuments.in/reader033/viewer/2022052907/558e7fdb1a28ab930b8b4656/html5/thumbnails/14.jpg)
Preparation - On Call
● Primary?
● Secondary?
● Reachability - Tube, 3G/4G (edge?!), Do Not Disturb mode, at the gym, family emergency, system updates
![Page 15: DevOps Incident Handling - Making friends not enemies.](https://reader033.fdocuments.in/reader033/viewer/2022052907/558e7fdb1a28ab930b8b4656/html5/thumbnails/15.jpg)
Preparation - On Call
● Off call
![Page 16: DevOps Incident Handling - Making friends not enemies.](https://reader033.fdocuments.in/reader033/viewer/2022052907/558e7fdb1a28ab930b8b4656/html5/thumbnails/16.jpg)
Preparation - On Call
● Off call
● Rotations
![Page 17: DevOps Incident Handling - Making friends not enemies.](https://reader033.fdocuments.in/reader033/viewer/2022052907/558e7fdb1a28ab930b8b4656/html5/thumbnails/17.jpg)
Preparation - On Call
● Off call
● Rotations
● Illness
![Page 18: DevOps Incident Handling - Making friends not enemies.](https://reader033.fdocuments.in/reader033/viewer/2022052907/558e7fdb1a28ab930b8b4656/html5/thumbnails/18.jpg)
Preparation - On Call
● Off call
● Rotations
● Illness
● Work the next day?
![Page 19: DevOps Incident Handling - Making friends not enemies.](https://reader033.fdocuments.in/reader033/viewer/2022052907/558e7fdb1a28ab930b8b4656/html5/thumbnails/19.jpg)
Preparation - Documentation
![Page 20: DevOps Incident Handling - Making friends not enemies.](https://reader033.fdocuments.in/reader033/viewer/2022052907/558e7fdb1a28ab930b8b4656/html5/thumbnails/20.jpg)
Preparation - Documentation
● Searchable
![Page 21: DevOps Incident Handling - Making friends not enemies.](https://reader033.fdocuments.in/reader033/viewer/2022052907/558e7fdb1a28ab930b8b4656/html5/thumbnails/21.jpg)
Preparation - Documentation
● Searchable
● Easy to edit
![Page 22: DevOps Incident Handling - Making friends not enemies.](https://reader033.fdocuments.in/reader033/viewer/2022052907/558e7fdb1a28ab930b8b4656/html5/thumbnails/22.jpg)
Preparation - Documentation
● Searchable
● Easy to edit
● Independent of your infrastructure
![Page 23: DevOps Incident Handling - Making friends not enemies.](https://reader033.fdocuments.in/reader033/viewer/2022052907/558e7fdb1a28ab930b8b4656/html5/thumbnails/23.jpg)
Preparation - Documentation
● Searchable
● Easy to edit
● Independent of your infrastructure
● Up to date
![Page 24: DevOps Incident Handling - Making friends not enemies.](https://reader033.fdocuments.in/reader033/viewer/2022052907/558e7fdb1a28ab930b8b4656/html5/thumbnails/24.jpg)
![Page 25: DevOps Incident Handling - Making friends not enemies.](https://reader033.fdocuments.in/reader033/viewer/2022052907/558e7fdb1a28ab930b8b4656/html5/thumbnails/25.jpg)
Preparation - Key Info
![Page 26: DevOps Incident Handling - Making friends not enemies.](https://reader033.fdocuments.in/reader033/viewer/2022052907/558e7fdb1a28ab930b8b4656/html5/thumbnails/26.jpg)
Preparation - Key Info
● Team contacts
![Page 27: DevOps Incident Handling - Making friends not enemies.](https://reader033.fdocuments.in/reader033/viewer/2022052907/558e7fdb1a28ab930b8b4656/html5/thumbnails/27.jpg)
Preparation - Key Info
● Team contacts
● Key vendor contacts
![Page 28: DevOps Incident Handling - Making friends not enemies.](https://reader033.fdocuments.in/reader033/viewer/2022052907/558e7fdb1a28ab930b8b4656/html5/thumbnails/28.jpg)
Preparation - Key Info
● Team contacts
● Key vendor contacts
● Credentials to key systems
![Page 29: DevOps Incident Handling - Making friends not enemies.](https://reader033.fdocuments.in/reader033/viewer/2022052907/558e7fdb1a28ab930b8b4656/html5/thumbnails/29.jpg)
Unexpected failures
![Page 30: DevOps Incident Handling - Making friends not enemies.](https://reader033.fdocuments.in/reader033/viewer/2022052907/558e7fdb1a28ab930b8b4656/html5/thumbnails/30.jpg)
Unexpected failures
● Communication systems
![Page 31: DevOps Incident Handling - Making friends not enemies.](https://reader033.fdocuments.in/reader033/viewer/2022052907/558e7fdb1a28ab930b8b4656/html5/thumbnails/31.jpg)
Unexpected failures
● Communication systems
● Network connectivity
![Page 32: DevOps Incident Handling - Making friends not enemies.](https://reader033.fdocuments.in/reader033/viewer/2022052907/558e7fdb1a28ab930b8b4656/html5/thumbnails/32.jpg)
Unexpected failures
● Communication systems
● Network connectivity
● Access to support
![Page 33: DevOps Incident Handling - Making friends not enemies.](https://reader033.fdocuments.in/reader033/viewer/2022052907/558e7fdb1a28ab930b8b4656/html5/thumbnails/33.jpg)
ALERT!
![Page 34: DevOps Incident Handling - Making friends not enemies.](https://reader033.fdocuments.in/reader033/viewer/2022052907/558e7fdb1a28ab930b8b4656/html5/thumbnails/34.jpg)
ALERT!
1. Load up incident response checklist
![Page 35: DevOps Incident Handling - Making friends not enemies.](https://reader033.fdocuments.in/reader033/viewer/2022052907/558e7fdb1a28ab930b8b4656/html5/thumbnails/35.jpg)
ALERT!
1. Load up incident response checklist
2. Log incident in JIRA
![Page 36: DevOps Incident Handling - Making friends not enemies.](https://reader033.fdocuments.in/reader033/viewer/2022052907/558e7fdb1a28ab930b8b4656/html5/thumbnails/36.jpg)
ALERT!
1. Load up incident response checklist
2. Log incident in JIRA
3. Log into Ops War Room
![Page 37: DevOps Incident Handling - Making friends not enemies.](https://reader033.fdocuments.in/reader033/viewer/2022052907/558e7fdb1a28ab930b8b4656/html5/thumbnails/37.jpg)
ALERT!
1. Load up incident response checklist
2. Log incident in JIRA
4. Public status post
3. Log into Ops War Room
![Page 38: DevOps Incident Handling - Making friends not enemies.](https://reader033.fdocuments.in/reader033/viewer/2022052907/558e7fdb1a28ab930b8b4656/html5/thumbnails/38.jpg)
ALERT!
1. Load up incident response checklist
2. Log incident in JIRA
4. Public status post
5. Initial investigation
3. Log into Ops War Room
![Page 39: DevOps Incident Handling - Making friends not enemies.](https://reader033.fdocuments.in/reader033/viewer/2022052907/558e7fdb1a28ab930b8b4656/html5/thumbnails/39.jpg)
Key response principles
![Page 40: DevOps Incident Handling - Making friends not enemies.](https://reader033.fdocuments.in/reader033/viewer/2022052907/558e7fdb1a28ab930b8b4656/html5/thumbnails/40.jpg)
Key response principles
● Log everything
![Page 41: DevOps Incident Handling - Making friends not enemies.](https://reader033.fdocuments.in/reader033/viewer/2022052907/558e7fdb1a28ab930b8b4656/html5/thumbnails/41.jpg)
Key response principles
● Log everything
● Frequent public status updates
![Page 42: DevOps Incident Handling - Making friends not enemies.](https://reader033.fdocuments.in/reader033/viewer/2022052907/558e7fdb1a28ab930b8b4656/html5/thumbnails/42.jpg)
Key response principles
● Log everything
● Frequent public status updates
● Gather the team
![Page 43: DevOps Incident Handling - Making friends not enemies.](https://reader033.fdocuments.in/reader033/viewer/2022052907/558e7fdb1a28ab930b8b4656/html5/thumbnails/43.jpg)
Key response principles
● Log everything
● Frequent public status updates
● Gather the team
● Escalate!
![Page 44: DevOps Incident Handling - Making friends not enemies.](https://reader033.fdocuments.in/reader033/viewer/2022052907/558e7fdb1a28ab930b8b4656/html5/thumbnails/44.jpg)
Postmortem
![Page 45: DevOps Incident Handling - Making friends not enemies.](https://reader033.fdocuments.in/reader033/viewer/2022052907/558e7fdb1a28ab930b8b4656/html5/thumbnails/45.jpg)
Postmortem
● Within a few days
![Page 46: DevOps Incident Handling - Making friends not enemies.](https://reader033.fdocuments.in/reader033/viewer/2022052907/558e7fdb1a28ab930b8b4656/html5/thumbnails/46.jpg)
Postmortem
● Within a few days
● Tell the story
![Page 47: DevOps Incident Handling - Making friends not enemies.](https://reader033.fdocuments.in/reader033/viewer/2022052907/558e7fdb1a28ab930b8b4656/html5/thumbnails/47.jpg)
Postmortem
● Within a few days
● Tell the story
● Provide technical detail
![Page 48: DevOps Incident Handling - Making friends not enemies.](https://reader033.fdocuments.in/reader033/viewer/2022052907/558e7fdb1a28ab930b8b4656/html5/thumbnails/48.jpg)
Postmortem
● Within a few days
● Tell the story
● Provide technical detail
● Explain what failed and why
![Page 49: DevOps Incident Handling - Making friends not enemies.](https://reader033.fdocuments.in/reader033/viewer/2022052907/558e7fdb1a28ab930b8b4656/html5/thumbnails/49.jpg)
Postmortem
● How it’s going to be fixed
![Page 50: DevOps Incident Handling - Making friends not enemies.](https://reader033.fdocuments.in/reader033/viewer/2022052907/558e7fdb1a28ab930b8b4656/html5/thumbnails/50.jpg)
stspg.io/ZDC
![Page 51: DevOps Incident Handling - Making friends not enemies.](https://reader033.fdocuments.in/reader033/viewer/2022052907/558e7fdb1a28ab930b8b4656/html5/thumbnails/51.jpg)
Summary
● Preparation
● Communication
● Checklists
● Documentation
● Postmortem
![Page 52: DevOps Incident Handling - Making friends not enemies.](https://reader033.fdocuments.in/reader033/viewer/2022052907/558e7fdb1a28ab930b8b4656/html5/thumbnails/52.jpg)
どもありがとうございます
@davidmytton
blog.serverdensity.com
www.serverdensity.com