Why don'we just say "No" - Devopsdays Silicon Valley Jun 2014
-
Upload
dominica-degrandis -
Category
Technology
-
view
1.039 -
download
3
description
Transcript of Why don'we just say "No" - Devopsdays Silicon Valley Jun 2014
Why don’t We just say no? !
!
!
Ge$ng!things!done!doesn't!always!
start!with!"Yes". @dominicad!www.ddegrandis.com!
@dominicad!www.ddegrandis.com!
For a number of reasons, it’s hard to say no.!
@dominicad!www.ddegrandis.com!
#1.!!We!do!stuff!for!people!we!like.!!!
!
@dominicad!www.ddegrandis.com!
Ask!your!neighbor!this!quesCon….!
!!
“Why!do!you!take!on!more!work!than!
you!have!capacity!to!do?”!!!
@dominicad!www.ddegrandis.com!
Reason!#!2!–!We!are!team!players!
@dominicad!www.ddegrandis.com!
Reason!#!3!–!Fear!from!those!in!posiCon!of!power!
@dominicad!www.ddegrandis.com!
Reason!#!4!–!People!pleaser!
@dominicad!www.ddegrandis.com!
Why do Development teams seem to be able to say no
more than Ops teams? !
Stuff Ops deals with that Dev traditionally doesn’t: • Live prod issues • DDOS attacks • 100% disk utilization • access requests at 2:00 am • opening up firewalls
Over the last 11 years, a lot of Dev teams have been given permission to say,
“We’ve already done the planning for
this week. Your next opportunity to get
your request considered is next week.”
Saying “No” has been institutionalized by using timeframes.
This doesn’t work for Ops very well… “Sorry - that live production fix will have to wait until next week” What if instead of using timeframes to
say “No”, we used limits to say “No”?
A!look!at!one!team’s!evoluCon!
40 Ops Engineers (SysAdmin, DBA, Network, Mon, Sec)
Tasked to build out/retrofit 6 data-centers across 6 diff countries.
AND…. • keep the lights on four existing data centers • build out a new platform architecture • support live issues (on-call) • roll out a new configuration management tool • deploy a crap load of new features • deal with 3 reorgs over a 6 month period !!
!
!
!!Problems:
• Conflicting priorities • risky dependencies • over-commitments • interrupt-driven context switching
resulting in missed commitments. • Confusion around the new org structure !!
!
!
How!we!got!started!on!road!to!
improving!–!looked!at!3!data!pts!
!!
!
open closed
#1 Can we keep up with the demand?
0!
10!
20!
30!
40!
50!
60!
1!
2!
3!
4!
5!
6!
7!
8!
9!
10!
20!
30!
40!
50!
60!
70!
80!
90!
100!
110!
120!
130!
140!
150!
160!
170!
180!
190!
200!
250!
300!
More!
#"#ckets"
# days it took for ticket to go from created to closed
#2 Lead time – how long does it take to get work done?
Validate Design Deliver Done Implement
doing
Maint"
Projects
Implement
done !!
96
#3!Where!is!work!stuck?!!
Customer!Mtg!Cme!
!!
Invited!customers!to!a!meeCng!and!showed!them!the!data.!!– showed!them!the!demand!and!what!was!ge$ng!done!!!
– showed!them!the!Lead!Cme!!
– showed!!where!work!got!stuck!!!
Customers!appreciated!the!visibility!into!Ops!
!We!took!advantage!of!that!by!humbly!asking!for!their!help.!
!beginning!with!all!of!those!Cckets!si$ng!in!the!validate!state.!!
Next!steps!
The!introducCon!of!a!work`in`progress!(wip)!
limit.!!Some!of!these!guys!had!20!–!40!Cckets!in!
their!queue.!!We!asked!them,!“Does!this!seem!
reasonable?”!
!
How!about!10?!!!
Let’s!head!in!that!direcCon!and!see!what!
happens.!!!
!
and!then!there!were!reorgs!
The!1st!!org!restructure!change!created!the!A!Team!!`!to!focus!on!compleCng!projects!close!to!being!done,!but!sCll!hanging!on.!
!
• This!team!didn’t!have!to!respond!to!one!off!requests!and!wasn’t!supposed!to!be!on`call.!
!!
• 2nd!org!structure!change!split!Ops!into!3!teams!(live!Ops,!Build,!architecture)!
• Live`Ops!with!25%!of!team!and!60%!of!the!work!!
!
!
• access requests for systems, non-Zabbix monitor • hardware investigation/verification/fixes - vlan/port changes - data retrieval (i.e. logs, network stats, etc) • configuration triage - firewalls, load balancers,
OS settings • (small) capacity expansion • verification of configs/services across shards • database development consultation • security compliance mitigation
Live Ops tasks
Live!Ops!SRE!Changes!
• Socialized wip limit idea over 6 months and
gradually lowered it from 10 to 7 – out of 18 guys, average is 5-7.
• Hired 4 more people, although 2 got stolen by another team
• Closed out all tickets with no activity > 90 days
• Started saying “No” to last minute requests. !
Hi D.C., Team SRE has a very large number of changes scheduled for today already, and an even larger number of requests in our backlog that this request will displace if moved to the front of the queue. It would not be fair to other teams if we jumped on this immediately while planned work is pushed off. Monitoring should be a requirement for a service to go live, not a last minute addition. For us to fully support a live service, please implement monitoring before going live. For future requests, please give us as much notice as possible, and make sure to create a ticket (xxx.com) so we can prioritize and schedule the changes as necessary. Here's the ticket for this work…. Respectfully, A.H !
Live!Ops!SRE!changes!con’t!
• Took time during standups to focus on kaizen improvements.
• Reduced validate state from 7 to 5 to 3 days.
• Found creative way to deal with walkups, and work done via personal relationships
• 15 min daily sync up at 3pm instead of interrupting.
• 5 min videos to present Ops review data to customers
!
!
A few simple rules
Clear definitions of done between queues
kaikzen/��!noun, continuous improvement A business philosophy or system based on making positive changes on a regular basis.
“Disney thinks
we can’t do it.
Let’s prove
them wrong.” -Steve Jobs
“Asking this much of people,
even when they wanted to give
it, was not acceptable.”
!
!! ! ! ! !!`!Ed!Catmull!
“WIP it down
to the personal safety level”!
“No” is an honorable reply to
someone asking too much
from you.
“The!takeaway!for!leaders!is!
realizing!the!power!you!have!
over!other!people!when!you!!
ask!something!of!them.”!
!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!`!Vaness!Bohns
!!
!
!
But,!what!about!when!the!show!must!
go!on?!
!
World!Cup!finals!`!!Sunday,!July!13!
!!
fifa regulations ensure they have capacity. “pg 8 e) ensuring the presence of a sufficient number of
ground staff and security stewards to guarantee safety.”
Let’s not expect day shift workers to also cover the night shift.
@dominicad!www.ddegrandis.com!
Honey-do list rules for saying no to the spouse…
@dominicad!www.ddegrandis.com!
Credits!!
hqp://www.psychologytoday.com/blog/caveman`logic/201403/why`so`
many`people`just`cant`say`no!
!
The!Book!of!NO:!250!Ways!to!Say!It!—!and!Mean!It!—!And!Stop!People`
Pleasing!Forever.
hqp://www.amazon.com/The`Book`No`People`pleasing`Forever/dp/
0071460780!
!
Vanessa!Bohns!!`!!social!psychologist!and!professor!of!management!
sciences!at!the!University!of!Waterloo!in!Ontario!
hqp://hereandnow.wbur.org/2014/03/31/saying`no`psychology!
!
eBay!arCcle!!hqp://www.uie.com/arCcles/death_of_relaunch/!
!
Fifa!regulaCons!
http://resources.fifa.com/mm/document/tournament/competition/01/47/38/17/regulationsfwcbrazil2014_update_e_neutral.pdf !