Getting Good Things Out of Bad Failures Joy Scharmen · The Virtuous Cycle Getting Good Things Out...
Transcript of Getting Good Things Out of Bad Failures Joy Scharmen · The Virtuous Cycle Getting Good Things Out...
![Page 1: Getting Good Things Out of Bad Failures Joy Scharmen · The Virtuous Cycle Getting Good Things Out of Bad Failures Joy Scharmen](https://reader033.fdocuments.in/reader033/viewer/2022050606/5fad3d540017532c176d6278/html5/thumbnails/1.jpg)
The Virtuous CycleGetting Good Things Out of Bad Failures
Joy Scharmen
![Page 2: Getting Good Things Out of Bad Failures Joy Scharmen · The Virtuous Cycle Getting Good Things Out of Bad Failures Joy Scharmen](https://reader033.fdocuments.in/reader033/viewer/2022050606/5fad3d540017532c176d6278/html5/thumbnails/2.jpg)
I’m here to talk about failure.
![Page 3: Getting Good Things Out of Bad Failures Joy Scharmen · The Virtuous Cycle Getting Good Things Out of Bad Failures Joy Scharmen](https://reader033.fdocuments.in/reader033/viewer/2022050606/5fad3d540017532c176d6278/html5/thumbnails/3.jpg)
Why talk about failure?
Failure is amazing.
Failure is our best teacher.
![Page 4: Getting Good Things Out of Bad Failures Joy Scharmen · The Virtuous Cycle Getting Good Things Out of Bad Failures Joy Scharmen](https://reader033.fdocuments.in/reader033/viewer/2022050606/5fad3d540017532c176d6278/html5/thumbnails/4.jpg)
How do we learn from failure?
![Page 5: Getting Good Things Out of Bad Failures Joy Scharmen · The Virtuous Cycle Getting Good Things Out of Bad Failures Joy Scharmen](https://reader033.fdocuments.in/reader033/viewer/2022050606/5fad3d540017532c176d6278/html5/thumbnails/5.jpg)
Have you ever run out of integers
in an auto-incrementing primary
key column in a database?
![Page 6: Getting Good Things Out of Bad Failures Joy Scharmen · The Virtuous Cycle Getting Good Things Out of Bad Failures Joy Scharmen](https://reader033.fdocuments.in/reader033/viewer/2022050606/5fad3d540017532c176d6278/html5/thumbnails/6.jpg)
![Page 7: Getting Good Things Out of Bad Failures Joy Scharmen · The Virtuous Cycle Getting Good Things Out of Bad Failures Joy Scharmen](https://reader033.fdocuments.in/reader033/viewer/2022050606/5fad3d540017532c176d6278/html5/thumbnails/7.jpg)
Looking at you, ActiveRecord.
Frameworks that default to INT
![Page 8: Getting Good Things Out of Bad Failures Joy Scharmen · The Virtuous Cycle Getting Good Things Out of Bad Failures Joy Scharmen](https://reader033.fdocuments.in/reader033/viewer/2022050606/5fad3d540017532c176d6278/html5/thumbnails/8.jpg)
Assumptions about the size of your database.
(before it hits production)
![Page 9: Getting Good Things Out of Bad Failures Joy Scharmen · The Virtuous Cycle Getting Good Things Out of Bad Failures Joy Scharmen](https://reader033.fdocuments.in/reader033/viewer/2022050606/5fad3d540017532c176d6278/html5/thumbnails/9.jpg)
Just not thinking about itI’m just happy that it works at all.
![Page 10: Getting Good Things Out of Bad Failures Joy Scharmen · The Virtuous Cycle Getting Good Things Out of Bad Failures Joy Scharmen](https://reader033.fdocuments.in/reader033/viewer/2022050606/5fad3d540017532c176d6278/html5/thumbnails/10.jpg)
Oops, I did it again.
We had this happen to us twice.
...then it happened a third time.
![Page 11: Getting Good Things Out of Bad Failures Joy Scharmen · The Virtuous Cycle Getting Good Things Out of Bad Failures Joy Scharmen](https://reader033.fdocuments.in/reader033/viewer/2022050606/5fad3d540017532c176d6278/html5/thumbnails/11.jpg)
And we’re an operations company.
┬─┬ ノ( ゜-゜ノ)
![Page 12: Getting Good Things Out of Bad Failures Joy Scharmen · The Virtuous Cycle Getting Good Things Out of Bad Failures Joy Scharmen](https://reader033.fdocuments.in/reader033/viewer/2022050606/5fad3d540017532c176d6278/html5/thumbnails/12.jpg)
First, consider it more deeply.
Σ(-᷅_-᷄๑)
![Page 13: Getting Good Things Out of Bad Failures Joy Scharmen · The Virtuous Cycle Getting Good Things Out of Bad Failures Joy Scharmen](https://reader033.fdocuments.in/reader033/viewer/2022050606/5fad3d540017532c176d6278/html5/thumbnails/13.jpg)
We fixed one occurrence. It was simple.
![Page 14: Getting Good Things Out of Bad Failures Joy Scharmen · The Virtuous Cycle Getting Good Things Out of Bad Failures Joy Scharmen](https://reader033.fdocuments.in/reader033/viewer/2022050606/5fad3d540017532c176d6278/html5/thumbnails/14.jpg)
It happened again. Same fix.
![Page 15: Getting Good Things Out of Bad Failures Joy Scharmen · The Virtuous Cycle Getting Good Things Out of Bad Failures Joy Scharmen](https://reader033.fdocuments.in/reader033/viewer/2022050606/5fad3d540017532c176d6278/html5/thumbnails/15.jpg)
Then it happened again.Obviously what we are doing here isn’t working.
![Page 16: Getting Good Things Out of Bad Failures Joy Scharmen · The Virtuous Cycle Getting Good Things Out of Bad Failures Joy Scharmen](https://reader033.fdocuments.in/reader033/viewer/2022050606/5fad3d540017532c176d6278/html5/thumbnails/16.jpg)
Who here is familiar with retrospectives?
ret·ro·spec·tiveˌretrəˈspektiv/adjective
1. looking back on or dealing with past events or situations.
![Page 17: Getting Good Things Out of Bad Failures Joy Scharmen · The Virtuous Cycle Getting Good Things Out of Bad Failures Joy Scharmen](https://reader033.fdocuments.in/reader033/viewer/2022050606/5fad3d540017532c176d6278/html5/thumbnails/17.jpg)
Who has been to a boring retrospective?
I have. I’ve run them. Sorry.
![Page 18: Getting Good Things Out of Bad Failures Joy Scharmen · The Virtuous Cycle Getting Good Things Out of Bad Failures Joy Scharmen](https://reader033.fdocuments.in/reader033/viewer/2022050606/5fad3d540017532c176d6278/html5/thumbnails/18.jpg)
A retrospective is the pivot point between failure and learning.
If it’s boring, no one is learning.
![Page 19: Getting Good Things Out of Bad Failures Joy Scharmen · The Virtuous Cycle Getting Good Things Out of Bad Failures Joy Scharmen](https://reader033.fdocuments.in/reader033/viewer/2022050606/5fad3d540017532c176d6278/html5/thumbnails/19.jpg)
How do we have non-boring retrospectives?
![Page 20: Getting Good Things Out of Bad Failures Joy Scharmen · The Virtuous Cycle Getting Good Things Out of Bad Failures Joy Scharmen](https://reader033.fdocuments.in/reader033/viewer/2022050606/5fad3d540017532c176d6278/html5/thumbnails/20.jpg)
Create engagement. Prepare!
Don’t force people to watch the sausage being made.
![Page 21: Getting Good Things Out of Bad Failures Joy Scharmen · The Virtuous Cycle Getting Good Things Out of Bad Failures Joy Scharmen](https://reader033.fdocuments.in/reader033/viewer/2022050606/5fad3d540017532c176d6278/html5/thumbnails/21.jpg)
Before the retrospective:
Choose a facilitator
They should know who was involved and why.
![Page 22: Getting Good Things Out of Bad Failures Joy Scharmen · The Virtuous Cycle Getting Good Things Out of Bad Failures Joy Scharmen](https://reader033.fdocuments.in/reader033/viewer/2022050606/5fad3d540017532c176d6278/html5/thumbnails/22.jpg)
Before the retrospective:
Build a timeline
Gather your facts.
![Page 23: Getting Good Things Out of Bad Failures Joy Scharmen · The Virtuous Cycle Getting Good Things Out of Bad Failures Joy Scharmen](https://reader033.fdocuments.in/reader033/viewer/2022050606/5fad3d540017532c176d6278/html5/thumbnails/23.jpg)
Use your tools wisely
“We become what we behold. We shape our tools, and thereafter our tools shape us.”
― Marshall McLuhan
![Page 24: Getting Good Things Out of Bad Failures Joy Scharmen · The Virtuous Cycle Getting Good Things Out of Bad Failures Joy Scharmen](https://reader033.fdocuments.in/reader033/viewer/2022050606/5fad3d540017532c176d6278/html5/thumbnails/24.jpg)
Retrospective PrepIncident Management
ChatOps
Bot Tools
SitReps
Time
Outreach
Organization
My Tools For
![Page 25: Getting Good Things Out of Bad Failures Joy Scharmen · The Virtuous Cycle Getting Good Things Out of Bad Failures Joy Scharmen](https://reader033.fdocuments.in/reader033/viewer/2022050606/5fad3d540017532c176d6278/html5/thumbnails/25.jpg)
My personal incident management tool belt:
ChatOps
![Page 26: Getting Good Things Out of Bad Failures Joy Scharmen · The Virtuous Cycle Getting Good Things Out of Bad Failures Joy Scharmen](https://reader033.fdocuments.in/reader033/viewer/2022050606/5fad3d540017532c176d6278/html5/thumbnails/26.jpg)
My personal incident management tool belt:
Bot Tools
![Page 27: Getting Good Things Out of Bad Failures Joy Scharmen · The Virtuous Cycle Getting Good Things Out of Bad Failures Joy Scharmen](https://reader033.fdocuments.in/reader033/viewer/2022050606/5fad3d540017532c176d6278/html5/thumbnails/27.jpg)
My personal incident management tool belt:
SitReps
![Page 28: Getting Good Things Out of Bad Failures Joy Scharmen · The Virtuous Cycle Getting Good Things Out of Bad Failures Joy Scharmen](https://reader033.fdocuments.in/reader033/viewer/2022050606/5fad3d540017532c176d6278/html5/thumbnails/28.jpg)
My personal retrospective toolbelt:
TimeBlock out time.
![Page 29: Getting Good Things Out of Bad Failures Joy Scharmen · The Virtuous Cycle Getting Good Things Out of Bad Failures Joy Scharmen](https://reader033.fdocuments.in/reader033/viewer/2022050606/5fad3d540017532c176d6278/html5/thumbnails/29.jpg)
My personal retrospective toolbelt:
OutreachHave roles defined.
![Page 30: Getting Good Things Out of Bad Failures Joy Scharmen · The Virtuous Cycle Getting Good Things Out of Bad Failures Joy Scharmen](https://reader033.fdocuments.in/reader033/viewer/2022050606/5fad3d540017532c176d6278/html5/thumbnails/30.jpg)
My personal retrospective toolbelt:
OrganizationSend out the agenda, including the timeline, the day before the retrospective.
![Page 31: Getting Good Things Out of Bad Failures Joy Scharmen · The Virtuous Cycle Getting Good Things Out of Bad Failures Joy Scharmen](https://reader033.fdocuments.in/reader033/viewer/2022050606/5fad3d540017532c176d6278/html5/thumbnails/31.jpg)
People should show up to a retrospective with context to begin a discussion.
![Page 32: Getting Good Things Out of Bad Failures Joy Scharmen · The Virtuous Cycle Getting Good Things Out of Bad Failures Joy Scharmen](https://reader033.fdocuments.in/reader033/viewer/2022050606/5fad3d540017532c176d6278/html5/thumbnails/32.jpg)
Everyone is in the retrospective. The timeline is done. How do we start?
![Page 33: Getting Good Things Out of Bad Failures Joy Scharmen · The Virtuous Cycle Getting Good Things Out of Bad Failures Joy Scharmen](https://reader033.fdocuments.in/reader033/viewer/2022050606/5fad3d540017532c176d6278/html5/thumbnails/33.jpg)
Have the most involved engineer give a brief summary of what happened.
![Page 34: Getting Good Things Out of Bad Failures Joy Scharmen · The Virtuous Cycle Getting Good Things Out of Bad Failures Joy Scharmen](https://reader033.fdocuments.in/reader033/viewer/2022050606/5fad3d540017532c176d6278/html5/thumbnails/34.jpg)
Make sure everyone is engaged.
Read the room.
![Page 35: Getting Good Things Out of Bad Failures Joy Scharmen · The Virtuous Cycle Getting Good Things Out of Bad Failures Joy Scharmen](https://reader033.fdocuments.in/reader033/viewer/2022050606/5fad3d540017532c176d6278/html5/thumbnails/35.jpg)
Be compassionate for your customers.
Talk about customer impact.
![Page 36: Getting Good Things Out of Bad Failures Joy Scharmen · The Virtuous Cycle Getting Good Things Out of Bad Failures Joy Scharmen](https://reader033.fdocuments.in/reader033/viewer/2022050606/5fad3d540017532c176d6278/html5/thumbnails/36.jpg)
Take note.
![Page 37: Getting Good Things Out of Bad Failures Joy Scharmen · The Virtuous Cycle Getting Good Things Out of Bad Failures Joy Scharmen](https://reader033.fdocuments.in/reader033/viewer/2022050606/5fad3d540017532c176d6278/html5/thumbnails/37.jpg)
Pick the point you want to start from and dive in.
![Page 38: Getting Good Things Out of Bad Failures Joy Scharmen · The Virtuous Cycle Getting Good Things Out of Bad Failures Joy Scharmen](https://reader033.fdocuments.in/reader033/viewer/2022050606/5fad3d540017532c176d6278/html5/thumbnails/38.jpg)
If you ever get to “human error”, keep digging.
![Page 39: Getting Good Things Out of Bad Failures Joy Scharmen · The Virtuous Cycle Getting Good Things Out of Bad Failures Joy Scharmen](https://reader033.fdocuments.in/reader033/viewer/2022050606/5fad3d540017532c176d6278/html5/thumbnails/39.jpg)
No, really.
If you ever get to “human error”, keep digging.
![Page 40: Getting Good Things Out of Bad Failures Joy Scharmen · The Virtuous Cycle Getting Good Things Out of Bad Failures Joy Scharmen](https://reader033.fdocuments.in/reader033/viewer/2022050606/5fad3d540017532c176d6278/html5/thumbnails/40.jpg)
Most Important:Always Assume Good Intent
![Page 41: Getting Good Things Out of Bad Failures Joy Scharmen · The Virtuous Cycle Getting Good Things Out of Bad Failures Joy Scharmen](https://reader033.fdocuments.in/reader033/viewer/2022050606/5fad3d540017532c176d6278/html5/thumbnails/41.jpg)
Defensiveness kills retrospection.
![Page 42: Getting Good Things Out of Bad Failures Joy Scharmen · The Virtuous Cycle Getting Good Things Out of Bad Failures Joy Scharmen](https://reader033.fdocuments.in/reader033/viewer/2022050606/5fad3d540017532c176d6278/html5/thumbnails/42.jpg)
One way you can tell a retrospective is good:
you have a ridiculous list of remediation items.
![Page 43: Getting Good Things Out of Bad Failures Joy Scharmen · The Virtuous Cycle Getting Good Things Out of Bad Failures Joy Scharmen](https://reader033.fdocuments.in/reader033/viewer/2022050606/5fad3d540017532c176d6278/html5/thumbnails/43.jpg)
“re-architect the whole platform”
Remediations can be anything from:
“fix typo on line 5”.
“make the speed of light go faster”.
to
to
![Page 44: Getting Good Things Out of Bad Failures Joy Scharmen · The Virtuous Cycle Getting Good Things Out of Bad Failures Joy Scharmen](https://reader033.fdocuments.in/reader033/viewer/2022050606/5fad3d540017532c176d6278/html5/thumbnails/44.jpg)
Don’t do every remediation.
Don’t discount big projects!
and
![Page 45: Getting Good Things Out of Bad Failures Joy Scharmen · The Virtuous Cycle Getting Good Things Out of Bad Failures Joy Scharmen](https://reader033.fdocuments.in/reader033/viewer/2022050606/5fad3d540017532c176d6278/html5/thumbnails/45.jpg)
What do you do with all of these remediations?
Bring them to product as well as engineering!
![Page 46: Getting Good Things Out of Bad Failures Joy Scharmen · The Virtuous Cycle Getting Good Things Out of Bad Failures Joy Scharmen](https://reader033.fdocuments.in/reader033/viewer/2022050606/5fad3d540017532c176d6278/html5/thumbnails/46.jpg)
Product can be your best friend.
Do you have a need? Your customers do too.
Product is great at getting needs in front of customers.
![Page 47: Getting Good Things Out of Bad Failures Joy Scharmen · The Virtuous Cycle Getting Good Things Out of Bad Failures Joy Scharmen](https://reader033.fdocuments.in/reader033/viewer/2022050606/5fad3d540017532c176d6278/html5/thumbnails/47.jpg)
Heroku PipelinesPipelines is a product that came out of an engineering need.
![Page 48: Getting Good Things Out of Bad Failures Joy Scharmen · The Virtuous Cycle Getting Good Things Out of Bad Failures Joy Scharmen](https://reader033.fdocuments.in/reader033/viewer/2022050606/5fad3d540017532c176d6278/html5/thumbnails/48.jpg)
Is your fix a small thing you can add to existing customer tools?
Engineering should be able to do this with minimal product sign off.
![Page 49: Getting Good Things Out of Bad Failures Joy Scharmen · The Virtuous Cycle Getting Good Things Out of Bad Failures Joy Scharmen](https://reader033.fdocuments.in/reader033/viewer/2022050606/5fad3d540017532c176d6278/html5/thumbnails/49.jpg)
You can improve your customers’ experience.
Your customers, your fellow engineers, and your community can benefit from your own
needs and hard won experience.
![Page 50: Getting Good Things Out of Bad Failures Joy Scharmen · The Virtuous Cycle Getting Good Things Out of Bad Failures Joy Scharmen](https://reader033.fdocuments.in/reader033/viewer/2022050606/5fad3d540017532c176d6278/html5/thumbnails/50.jpg)
Back to the story.
![Page 51: Getting Good Things Out of Bad Failures Joy Scharmen · The Virtuous Cycle Getting Good Things Out of Bad Failures Joy Scharmen](https://reader033.fdocuments.in/reader033/viewer/2022050606/5fad3d540017532c176d6278/html5/thumbnails/51.jpg)
Done: Tooling
![Page 52: Getting Good Things Out of Bad Failures Joy Scharmen · The Virtuous Cycle Getting Good Things Out of Bad Failures Joy Scharmen](https://reader033.fdocuments.in/reader033/viewer/2022050606/5fad3d540017532c176d6278/html5/thumbnails/52.jpg)
Done: Process
![Page 53: Getting Good Things Out of Bad Failures Joy Scharmen · The Virtuous Cycle Getting Good Things Out of Bad Failures Joy Scharmen](https://reader033.fdocuments.in/reader033/viewer/2022050606/5fad3d540017532c176d6278/html5/thumbnails/53.jpg)
Next: Automation
![Page 54: Getting Good Things Out of Bad Failures Joy Scharmen · The Virtuous Cycle Getting Good Things Out of Bad Failures Joy Scharmen](https://reader033.fdocuments.in/reader033/viewer/2022050606/5fad3d540017532c176d6278/html5/thumbnails/54.jpg)
Next: Fix inputs
* https://github.com/rails/rails/pull/24962
![Page 55: Getting Good Things Out of Bad Failures Joy Scharmen · The Virtuous Cycle Getting Good Things Out of Bad Failures Joy Scharmen](https://reader033.fdocuments.in/reader033/viewer/2022050606/5fad3d540017532c176d6278/html5/thumbnails/55.jpg)
Every failure is a chance to learn.
Make those chances count.
Thank you.
![Page 56: Getting Good Things Out of Bad Failures Joy Scharmen · The Virtuous Cycle Getting Good Things Out of Bad Failures Joy Scharmen](https://reader033.fdocuments.in/reader033/viewer/2022050606/5fad3d540017532c176d6278/html5/thumbnails/56.jpg)
Joy Scharmen / @peculiaire / [email protected]
Retrospective Resource Wiki:
http://retrospectivewiki.org
https://www.oreilly.com/ideas/the-infinite-hows
Infinite Hows:
https://devcenter.heroku.com
Heroku Dev Center:
https://github.com/peculiaire/incident-lifecycle/blob/master/retrotemplate.md
Retrospective Template: