@jezhumble CTO, //gotocon.com/dl/goto-cph-2016/slides/Jez...“Firms with high-performing IT...
Transcript of @jezhumble CTO, //gotocon.com/dl/goto-cph-2016/slides/Jez...“Firms with high-performing IT...
![Page 1: @jezhumble CTO, //gotocon.com/dl/goto-cph-2016/slides/Jez...“Firms with high-performing IT organizations were twice ... When automated tests pass, I’m confident the software is](https://reader033.fdocuments.in/reader033/viewer/2022050205/5f58628fd762d654485a16db/html5/thumbnails/1.jpg)
@jezhumble CTO, https://devops-research.com/
secrets of high performing teams: science edition
![Page 2: @jezhumble CTO, //gotocon.com/dl/goto-cph-2016/slides/Jez...“Firms with high-performing IT organizations were twice ... When automated tests pass, I’m confident the software is](https://reader033.fdocuments.in/reader033/viewer/2022050205/5f58628fd762d654485a16db/html5/thumbnails/2.jpg)
![Page 3: @jezhumble CTO, //gotocon.com/dl/goto-cph-2016/slides/Jez...“Firms with high-performing IT organizations were twice ... When automated tests pass, I’m confident the software is](https://reader033.fdocuments.in/reader033/viewer/2022050205/5f58628fd762d654485a16db/html5/thumbnails/3.jpg)
Outline
How to make your data suck less
Writing good survey questions
Making sure the survey questions are good - with SCIENCE
What we found… that we did (AND didn’t) expect
Things about Continuous Delivery
Things about Management
![Page 4: @jezhumble CTO, //gotocon.com/dl/goto-cph-2016/slides/Jez...“Firms with high-performing IT organizations were twice ... When automated tests pass, I’m confident the software is](https://reader033.fdocuments.in/reader033/viewer/2022050205/5f58628fd762d654485a16db/html5/thumbnails/4.jpg)
Statistics
Dr Nicole Forsgren | @nicolefv
![Page 5: @jezhumble CTO, //gotocon.com/dl/goto-cph-2016/slides/Jez...“Firms with high-performing IT organizations were twice ... When automated tests pass, I’m confident the software is](https://reader033.fdocuments.in/reader033/viewer/2022050205/5f58628fd762d654485a16db/html5/thumbnails/5.jpg)
Not all data is created equal
Who here thinks surveys are sh*t?
![Page 6: @jezhumble CTO, //gotocon.com/dl/goto-cph-2016/slides/Jez...“Firms with high-performing IT organizations were twice ... When automated tests pass, I’m confident the software is](https://reader033.fdocuments.in/reader033/viewer/2022050205/5f58628fd762d654485a16db/html5/thumbnails/6.jpg)
Not all data is created equal
Who here thinks surveys are sh*t?
Who here LOVES the data from their log files?
And who has seen sh*t data in a log file?
![Page 7: @jezhumble CTO, //gotocon.com/dl/goto-cph-2016/slides/Jez...“Firms with high-performing IT organizations were twice ... When automated tests pass, I’m confident the software is](https://reader033.fdocuments.in/reader033/viewer/2022050205/5f58628fd762d654485a16db/html5/thumbnails/7.jpg)
What is a Latent Construct?
![Page 8: @jezhumble CTO, //gotocon.com/dl/goto-cph-2016/slides/Jez...“Firms with high-performing IT organizations were twice ... When automated tests pass, I’m confident the software is](https://reader033.fdocuments.in/reader033/viewer/2022050205/5f58628fd762d654485a16db/html5/thumbnails/8.jpg)
We use
PSYCHOMETRICS
to make our survey data good*
*or give us a reasonable assurance that it’s telling us what we think it’s telling us (& some of this can also apply to your log data)
![Page 9: @jezhumble CTO, //gotocon.com/dl/goto-cph-2016/slides/Jez...“Firms with high-performing IT organizations were twice ... When automated tests pass, I’m confident the software is](https://reader033.fdocuments.in/reader033/viewer/2022050205/5f58628fd762d654485a16db/html5/thumbnails/9.jpg)
Psychometrics Includes:
Construct creation (manual)
When possible: use previously validated constructs
Based on definitions and theory, carefully and precisely worded, card sorting task, pilot tested
Construct evaluation (statistics)
Establishing Validity: discriminant and convergent
Establishing Reliability
![Page 10: @jezhumble CTO, //gotocon.com/dl/goto-cph-2016/slides/Jez...“Firms with high-performing IT organizations were twice ... When automated tests pass, I’m confident the software is](https://reader033.fdocuments.in/reader033/viewer/2022050205/5f58628fd762d654485a16db/html5/thumbnails/10.jpg)
Analysis Methods
A note about prediction: one of three conditions must be met:
• Longitudinal (no, this is cross-sectional) • Randomized, experimental design (no, this is a non-
experiment) • Theory-based design
When this condition was not met, only correlations were tested and reported.
![Page 11: @jezhumble CTO, //gotocon.com/dl/goto-cph-2016/slides/Jez...“Firms with high-performing IT organizations were twice ... When automated tests pass, I’m confident the software is](https://reader033.fdocuments.in/reader033/viewer/2022050205/5f58628fd762d654485a16db/html5/thumbnails/11.jpg)
Psychometrics Writing Example: Culture
Does it matter to our study?
More than just intuition?
What KIND of culture?
National identity and norms
Adaptive culture
Value learning (2014 study)
Value information flow and trust (2014 and 2015 studies -- Westrum culture)
![Page 12: @jezhumble CTO, //gotocon.com/dl/goto-cph-2016/slides/Jez...“Firms with high-performing IT organizations were twice ... When automated tests pass, I’m confident the software is](https://reader033.fdocuments.in/reader033/viewer/2022050205/5f58628fd762d654485a16db/html5/thumbnails/12.jpg)
Psychometrics Writing Example: Culture
Does it matter to our study?
More than just intuition?
What KIND of culture?
National identity and norms
Adaptive culture
Value learning (2014 study)
Value information flow and trust (2014 and 2015 studies -- Westrum culture)
![Page 13: @jezhumble CTO, //gotocon.com/dl/goto-cph-2016/slides/Jez...“Firms with high-performing IT organizations were twice ... When automated tests pass, I’m confident the software is](https://reader033.fdocuments.in/reader033/viewer/2022050205/5f58628fd762d654485a16db/html5/thumbnails/13.jpg)
Westrum typology
Pathological Power-oriented
Bureaucratic Rule-oriented
Generative Performance-oriented
Low cooperation Modest cooperation High cooperation
Messengers shot Messengers neglected Messengers trained
Responsibilities shirked Narrow responsibilities Risks are shared
Bridging discouraged Bridging tolerated Bridging encouraged
Failure leads to scapegoating Failure leads to justice Failure leads to inquiry
Novelty crushed Novelty leads to problems Novelty implemented
Try writing items
yourself!
Use strong statements with clear language.
Westrum, “A Typology of Organizational Cultures” | http://bmj.co/1BRGh5q
![Page 14: @jezhumble CTO, //gotocon.com/dl/goto-cph-2016/slides/Jez...“Firms with high-performing IT organizations were twice ... When automated tests pass, I’m confident the software is](https://reader033.fdocuments.in/reader033/viewer/2022050205/5f58628fd762d654485a16db/html5/thumbnails/14.jpg)
Westrum Culture Items
On my team, information is actively sought. On my team, failures are learning opportunities, and
messengers of them are not punished. On my team, responsibilities are shared. On my team, cross-functional collaboration is
encouraged and rewarded. On my team, failure causes inquiry. On my team, new ideas are welcomed.
Found to be valid
& reliable
Predictive of IT Performance & Organizational Performance
![Page 15: @jezhumble CTO, //gotocon.com/dl/goto-cph-2016/slides/Jez...“Firms with high-performing IT organizations were twice ... When automated tests pass, I’m confident the software is](https://reader033.fdocuments.in/reader033/viewer/2022050205/5f58628fd762d654485a16db/html5/thumbnails/15.jpg)
Likert-type scale
![Page 16: @jezhumble CTO, //gotocon.com/dl/goto-cph-2016/slides/Jez...“Firms with high-performing IT organizations were twice ... When automated tests pass, I’m confident the software is](https://reader033.fdocuments.in/reader033/viewer/2022050205/5f58628fd762d654485a16db/html5/thumbnails/16.jpg)
Psychometrics Analysis Example: Notification of Failure
At my organization… We are primarily notified of failures by reports from
customers. We are primarily notified of failures by the NOC. We get failure alerts from logging and monitoring systems. We monitor system health based on threshold warnings (ex.
CPU exceeds 100%). We monitor system health based on rate-of-change warnings
(ex. CPU usage has increased by 25% over the last 10 minutes).
Original in 2014, but
there was a surprise. Can you spot it?
![Page 17: @jezhumble CTO, //gotocon.com/dl/goto-cph-2016/slides/Jez...“Firms with high-performing IT organizations were twice ... When automated tests pass, I’m confident the software is](https://reader033.fdocuments.in/reader033/viewer/2022050205/5f58628fd762d654485a16db/html5/thumbnails/17.jpg)
Psychometrics Analysis Example: Notification of Failure
At my organization… We are primarily notified of failures by reports from
customers. We are primarily notified of failures by the NOC. We get failure alerts from logging and monitoring systems. We monitor system health based on threshold warnings (ex.
CPU exceeds 100%). We monitor system health based on rate-of-change warnings
(ex. CPU usage has increased by 25% over the last 10 minutes).
Notification from INSIDE
Notification from
OUTSIDE
![Page 18: @jezhumble CTO, //gotocon.com/dl/goto-cph-2016/slides/Jez...“Firms with high-performing IT organizations were twice ... When automated tests pass, I’m confident the software is](https://reader033.fdocuments.in/reader033/viewer/2022050205/5f58628fd762d654485a16db/html5/thumbnails/18.jpg)
More data tests!
Plus, we test to make sure the survey doesn’t have other problems.
Common method variance (CMV) (aka CMB for Bias)
Early vs. late responders
Survey drop-off rates and bias
![Page 19: @jezhumble CTO, //gotocon.com/dl/goto-cph-2016/slides/Jez...“Firms with high-performing IT organizations were twice ... When automated tests pass, I’m confident the software is](https://reader033.fdocuments.in/reader033/viewer/2022050205/5f58628fd762d654485a16db/html5/thumbnails/19.jpg)
Okay NOW we can look at the data
![Page 20: @jezhumble CTO, //gotocon.com/dl/goto-cph-2016/slides/Jez...“Firms with high-performing IT organizations were twice ... When automated tests pass, I’m confident the software is](https://reader033.fdocuments.in/reader033/viewer/2022050205/5f58628fd762d654485a16db/html5/thumbnails/20.jpg)
IT performance matters!
“Firms with high-performing IT organizations were twice as likely to exceed their profitability, market share and productivity goals.”
http://bit.ly/2014-devops-report/
http://bit.ly/2015-devops-report/
The State of DevOps Report is a joint production of Puppet and DevOps Research and Assessments LLC. Authors: Nicole Forsgren, Jez Humble, Gene Kim, Alanna Brown, Nigel Kersten.
![Page 21: @jezhumble CTO, //gotocon.com/dl/goto-cph-2016/slides/Jez...“Firms with high-performing IT organizations were twice ... When automated tests pass, I’m confident the software is](https://reader033.fdocuments.in/reader033/viewer/2022050205/5f58628fd762d654485a16db/html5/thumbnails/21.jpg)
IT performance
lead time for changes
release frequency
time to restore service
change fail rate
![Page 22: @jezhumble CTO, //gotocon.com/dl/goto-cph-2016/slides/Jez...“Firms with high-performing IT organizations were twice ... When automated tests pass, I’m confident the software is](https://reader033.fdocuments.in/reader033/viewer/2022050205/5f58628fd762d654485a16db/html5/thumbnails/22.jpg)
![Page 23: @jezhumble CTO, //gotocon.com/dl/goto-cph-2016/slides/Jez...“Firms with high-performing IT organizations were twice ... When automated tests pass, I’m confident the software is](https://reader033.fdocuments.in/reader033/viewer/2022050205/5f58628fd762d654485a16db/html5/thumbnails/23.jpg)
Highest Correlation with IT performance
“Our code, app configurations and system configurations are in a version control system”
“We get failure alerts from logging and monitoring systems”
“Developers merge their code into trunk daily”
“When development and operations teams interact, the outcome is generally win/win.”
“Developers break up large features into small, incremental changes.”
![Page 24: @jezhumble CTO, //gotocon.com/dl/goto-cph-2016/slides/Jez...“Firms with high-performing IT organizations were twice ... When automated tests pass, I’m confident the software is](https://reader033.fdocuments.in/reader033/viewer/2022050205/5f58628fd762d654485a16db/html5/thumbnails/24.jpg)
Top Predictors of IT Performance
peer-reviewed change approval process
version control everything
proactive monitoring
high trust organizational culture
win-win relationship between dev and ops
![Page 25: @jezhumble CTO, //gotocon.com/dl/goto-cph-2016/slides/Jez...“Firms with high-performing IT organizations were twice ... When automated tests pass, I’m confident the software is](https://reader033.fdocuments.in/reader033/viewer/2022050205/5f58628fd762d654485a16db/html5/thumbnails/25.jpg)
Westrum typology - information flowPathologicalPower-oriented
BureaucraticRule-oriented
GenerativePerformance-oriented
Low cooperation Modest cooperation High cooperation
Messengers shot Messengers neglected Messengers trained
Responsibilities shirked Narrow responsibilities Risks are shared
Bridging discouraged Bridging tolerated Bridging encouraged
Failure leads to scapegoating
Failure leads to justice Failure leads to inquiry
Novelty crushed Novelty leads to problems Novelty implemented
![Page 26: @jezhumble CTO, //gotocon.com/dl/goto-cph-2016/slides/Jez...“Firms with high-performing IT organizations were twice ... When automated tests pass, I’m confident the software is](https://reader033.fdocuments.in/reader033/viewer/2022050205/5f58628fd762d654485a16db/html5/thumbnails/26.jpg)
https://rework.withgoogle.com/blog/five-keys-to-a-successful-google-team/
![Page 27: @jezhumble CTO, //gotocon.com/dl/goto-cph-2016/slides/Jez...“Firms with high-performing IT organizations were twice ... When automated tests pass, I’m confident the software is](https://reader033.fdocuments.in/reader033/viewer/2022050205/5f58628fd762d654485a16db/html5/thumbnails/27.jpg)
2015 study (with maths)
Forsgren, N., J. Humble (2016). "The Role of Continuous Delivery in IT and Organizational Performance." In the Proceedings of the Western Decision Sciences Institute (WDSI) 2016, Las Vegas, NV. Available at SSRN: http://ssrn.com/abstract=2681909
![Page 28: @jezhumble CTO, //gotocon.com/dl/goto-cph-2016/slides/Jez...“Firms with high-performing IT organizations were twice ... When automated tests pass, I’m confident the software is](https://reader033.fdocuments.in/reader033/viewer/2022050205/5f58628fd762d654485a16db/html5/thumbnails/28.jpg)
2016 results
![Page 29: @jezhumble CTO, //gotocon.com/dl/goto-cph-2016/slides/Jez...“Firms with high-performing IT organizations were twice ... When automated tests pass, I’m confident the software is](https://reader033.fdocuments.in/reader033/viewer/2022050205/5f58628fd762d654485a16db/html5/thumbnails/29.jpg)
High performers spend 29% more time on new work than low performers, and 22% less time on unplanned work and rework.
![Page 30: @jezhumble CTO, //gotocon.com/dl/goto-cph-2016/slides/Jez...“Firms with high-performing IT organizations were twice ... When automated tests pass, I’m confident the software is](https://reader033.fdocuments.in/reader033/viewer/2022050205/5f58628fd762d654485a16db/html5/thumbnails/30.jpg)
some surprises
![Page 31: @jezhumble CTO, //gotocon.com/dl/goto-cph-2016/slides/Jez...“Firms with high-performing IT organizations were twice ... When automated tests pass, I’m confident the software is](https://reader033.fdocuments.in/reader033/viewer/2022050205/5f58628fd762d654485a16db/html5/thumbnails/31.jpg)
Which of these measure effective test practices?
Developers primarily create & maintain acceptance tests
QA primarily create & maintain acceptance tests
Primarily created & maintained by outsourced party
When automated tests pass, I’m confident the software is releasable
Test failures are likely to indicate a real defect
It’s easy for developers to fix acceptance tests
Developers share a common pool of test servers to reproduce failures
Developers create on demand test environments
Developers use their own dev environments to reproduce failures
![Page 32: @jezhumble CTO, //gotocon.com/dl/goto-cph-2016/slides/Jez...“Firms with high-performing IT organizations were twice ... When automated tests pass, I’m confident the software is](https://reader033.fdocuments.in/reader033/viewer/2022050205/5f58628fd762d654485a16db/html5/thumbnails/32.jpg)
Which of these measure effective test practices?
Developers primarily create & maintain acceptance tests
QA primarily create & maintain acceptance tests
Primarily created & maintained by outsourced party
When automated tests pass, I’m confident the software is releasable
Test failures are likely to indicate a real defect
It’s easy for developers to fix acceptance tests
Developers share a common pool of test servers to reproduce failures
Developers create on demand test environments
Developers use their own dev environments to reproduce failures
![Page 33: @jezhumble CTO, //gotocon.com/dl/goto-cph-2016/slides/Jez...“Firms with high-performing IT organizations were twice ... When automated tests pass, I’m confident the software is](https://reader033.fdocuments.in/reader033/viewer/2022050205/5f58628fd762d654485a16db/html5/thumbnails/33.jpg)
Change management
All production changes must be approved by an external body (e.g. change approval board, manager, etc.) before deployment or implementation (R)
Only high-risk changes, such as database changes, require approval
We have no change approval process
We rely on peer review to manage changes
![Page 34: @jezhumble CTO, //gotocon.com/dl/goto-cph-2016/slides/Jez...“Firms with high-performing IT organizations were twice ... When automated tests pass, I’m confident the software is](https://reader033.fdocuments.in/reader033/viewer/2022050205/5f58628fd762d654485a16db/html5/thumbnails/34.jpg)
Now for management stuff
We all know managing WIP is important, right?
Correlation between WIP and ITPerf is negligible
![Page 35: @jezhumble CTO, //gotocon.com/dl/goto-cph-2016/slides/Jez...“Firms with high-performing IT organizations were twice ... When automated tests pass, I’m confident the software is](https://reader033.fdocuments.in/reader033/viewer/2022050205/5f58628fd762d654485a16db/html5/thumbnails/35.jpg)
Now for management stuff
We all know managing WIP is important, right?
Correlation between WIP and ITPerf is negligible
What’s going on?
![Page 36: @jezhumble CTO, //gotocon.com/dl/goto-cph-2016/slides/Jez...“Firms with high-performing IT organizations were twice ... When automated tests pass, I’m confident the software is](https://reader033.fdocuments.in/reader033/viewer/2022050205/5f58628fd762d654485a16db/html5/thumbnails/36.jpg)
Lean management SEM
![Page 37: @jezhumble CTO, //gotocon.com/dl/goto-cph-2016/slides/Jez...“Firms with high-performing IT organizations were twice ... When automated tests pass, I’m confident the software is](https://reader033.fdocuments.in/reader033/viewer/2022050205/5f58628fd762d654485a16db/html5/thumbnails/37.jpg)
Lean product management SEM
![Page 38: @jezhumble CTO, //gotocon.com/dl/goto-cph-2016/slides/Jez...“Firms with high-performing IT organizations were twice ... When automated tests pass, I’m confident the software is](https://reader033.fdocuments.in/reader033/viewer/2022050205/5f58628fd762d654485a16db/html5/thumbnails/38.jpg)
Conclusions
Even if you think it’s obvious, TEST WITH DATA.
(if the results don’t surprise you, you’re doing it wrong)
(if you don’t also confirm some things you expected, you’re doing it wrong)
We CAN have it all, or at least throughput AND stability.
DevOps culture & practices have a measurable impact on IT & org perf
![Page 39: @jezhumble CTO, //gotocon.com/dl/goto-cph-2016/slides/Jez...“Firms with high-performing IT organizations were twice ... When automated tests pass, I’m confident the software is](https://reader033.fdocuments.in/reader033/viewer/2022050205/5f58628fd762d654485a16db/html5/thumbnails/39.jpg)
thank you!
© 2016 DevOps Research and Assessment LLChttps://devops-research.com/
To receive the following: • A copy of this presentation • A 100 page excerpt from Lean Enterprise • An excerpt from the DevOps Handbook • A 20m preview of my Continuous Delivery video workshop • Discount code for CD video + interviews with Eric Ries & more
Just pick up your phone and send an email To: [email protected] Subject: devops continuous delivery sounds great