Achieving High Performance DevOps · Confidential –Oracle Internal/Restricted/Highly Restricted...

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

Achieving High Performance DevOpsA Roadmap for Managers and Decision Makers

Vlado BarunReal-World PerformanceServer Technologies

Confidential – Oracle Internal/Restricted/Highly Restricted


Safe Harbor Statement

The following is not intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.

Confidential – Oracle Internal/Restricted/Highly Restricted 3


What this presentation is NOT about

• Something you can buy

– The latest/coolest tools

– Services (i.e. consulting)

• Case Studies*

Thus, feel free to leave if you expected any of the above …

* ie. DevOps Enterprise Summit videos - events.itrevolution.com/us/

events.itrevolution.com/us/


What this presentation IS about

• Understanding the root cause of the problem

• First order principles common to various approaches to solve the problem

• Common pitfalls

• Roadmap for achieving high-performance DevOps


Introductions

• 21 years in IT

• Various roles

– Database Engineer

– DBA

– Database Architect

–Manager of Database Dev & Ops

– Performance Engineer

–Organizational Change Engineer

Vlado Barun, M.Sc., OCE, OCA, MCP


Why are you here?



The Challenges we face

• Long Time-to-Market

– Time from Business approval to successful production deployment

• Agile methods (i.e. Scrum) re-introduce feedback loops

Business Development QA Operations

Time-to-Market


The Challenges we face

• It takes too many resources

– Dev and Ops are already stretched thin

– Not enough (powerful) hardware

• Error prone– 70% of outages due to human errors*

– If we fix the error and redeploy, we introduce more delays and contention for resources

– If we fix it in-place, risk increases

* “Database Administration: The Complete Guide to Practices and Procedures” by Craig Mullins


Is Automation the solution?

• Pros

– Reduces human errors

– Reduces time and resources needed for deployments• Sure, but by how much?

Event Duration

------------------------ -------------------------

Waiting for Information hours/days 95.77%

Preparation minutes/hours 4.16%

Execution seconds/minutes 0.07%


Is Automation the solution?

• Cons

– automation of a broken process can itself take a long time and be resource intensive

–Maintaining automation of a broken process can be cost prohibitive

• First fix the process, then automate it


Root cause analysis

• Why does it take so long?

• Why does it take so many resources?

• Why is it so error prone?


Time-to-Market vs Stability

• Frequent deployments result in

– Faster time-to-market => increased business value

– Higher probability of new production bugs => decreased stability

• Common response to stability issue

– Enforce bureaucratic process => reduced frequency of deployments => slower time-to-market


The Solution ?


How?



The Mission

Transition

from

Time-to-Market vs Stability Mindset

to

Time-to-Market & Stability Mindset


The Vision

havehave


DevOps executes the Mission

Time-to-Market

vs

Stability

Mindset

&



“Most successful DevOps teams have a strong rapport between groups and believe that everyone is part of the solution, not the problem.“

- Kellyn Pot’Vin-Gorman


*Source: DBA and DevOps - The Last Frontier

https://www.delphix.com/ebook/dba-devops-last-frontier



“More investors don't copy our model because our model is too simple“* – Charlie Munger

“When basic beats flashy”* - Morgan Housel


*Source: Take a Simple Idea and Take It Seriously

https://www.fool.com/investing/2016/07/06/take-a-simple-idea-and-take-it-seriously.aspx


What is DevOps?


• A set of practices

– Leadership and Organizational Culture

–Management• Based on Lean and Agile principles

– Technology• Based on Continuous Delivery


Where do we start?



Metrics



Benefits of metrics

• Provides a specific definition

– Reduces confusion, allows us to talk about the “same thing”

• Allows us to set goals

• Progress can me measured

• Helps with building a case with the “Business” side


4 Key Metrics to start with

• Time-to-Market

– Change lead time• from code committed to code deployed AND successfully executing in production

– Deployment frequency

• Stability– Change Failure Rate• % of changes that result in service degradation or requires corrective action

–Mean-Time-To-Recover (MTBR)• Time to restore service after an incident occurs


State of DevOps - Performance by Cluster 1


1Source: 2017 State of DevOps Report - bit.ly/2017-devops-report

http://bit.ly/2017-devops-report

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Confidential – Oracle Internal/Restricted/Highly Restricted 27


State of DevOps – Time to Market1




State of DevOps – Stability 1




State of DevOps1



What do we do now?



Leadership



Leadership

• By 2020, 50% of the CIO’s that have not transformed their capabilities will be displaced from the digital leadership team2

• Transformational Leadership3

– Vision

– Inspirational communication

– Intellectual stimulation

– Personal recognition

– Supportive leadership


2Source: Gartner Predicts3Source: Rafferty & Griffin: Dimensions of transformational leadership: Conceptual and empirical extensions

https://www.gartner.com/binaries/content/assets/events/keywords/infrastructure-operations-management/iome5/gartner-predicts-for-it-infrastructure-and-operations.pdf

https://www.sciencedirect.com/science/article/pii/S1048984304000207#%21

http://psycnet.apa.org/record/2004-14786-002


Culture



Culture


4Source: Westrum, R. A typology of organisational cultures

How organisations process information4

Pathological(Power oriented)

Bureaucratic(Rule oriented)

Generative(Performance oriented)

Low cooperation Modest cooperation High cooperation

Messengers shot Messengers neglected Messengers trained

Responsibilities shirked Narrow responsibilities Risks are shared

Bridging discouraged Bridging tolerated Bridging encouraged

Failure→scapegoating Failure→ justice Failure→ inquiry

Novelty crushed Novelty→ problems Novelty implemented

qualitysafety.bmj.com/content/13/suppl_2/ii22


How do you change your Organizational Culture?



Lean Management Practices



Lean Management Practices• Product Management

– Experimentation• Small Batch size

– i.e. Earliest Testable Product

• Frequent Customer Feedback

– i.e. A/B testing

– Entrust your team

– “…failure and innovation are inseparable twins” – Jeff Bezos

• Team Structure

1. Highly Collaborative Teams

2. Dedicated DevOps

3. Interdisciplinary teams



Incentives

• “Perhaps the most important rule in management is to get the incentives right.” – Charlie Munger

• Law of unintended consequences



Continuous Delivery



Continuous Delivery

• Version Control

• Continuous Integration and trunk-based development

• Left Shift on Security

• Test Automation

• Deployment Automation



Conway’s Law5

"Any organization that designs a system … will inevitably produce a design whose structure is a copy of the organization's communication structure.“

- Melvin Conway


5Source: How Do Committees Invent?

http://www.melconway.com/research/committees.html


Roadmap for Achieving High Performance DevOps


Leadership Culture

Metrics

Continuous Delivery

Lean Management

Practices

IT PerformanceOrganization Performance


What can I do today?• Time-to-Market & Stability Mindset

• Metrics

– Find where most of the time is spend then address the top consumers• i.e. change approval -> streamline approval process

• i.e. deployment scrip creation/execution –> automation

– Find the root cause for failures then fix them• i.e. wrong version/config -> version control/configuration management

• i.e. bug count -> Test automation + Formal Inspections

• Transformational Leadership Dimensions

• Generative Culture

• Lean Management Practices– Experimentation



Resources

• Lean Enterprise

• Leading the Transformation

• The Phoenix Project

• The DevOps Handbook

• Continuous Delivery

• 2017 State of DevOps Report

• How to Build a High-Performing IT Team


https://www.amazon.com/Lean-Enterprise-Performance-Organizations-Innovate/dp/1449368425/

https://www.amazon.com/Leading-Transformation-Applying-DevOps-Principles/dp/1942788010

https://www.amazon.com/Phoenix-Project-DevOps-Helping-Business/dp/0988262509/

https://www.amazon.com/DevOps-Handbook-World-Class-Reliability-Organizations/dp/1942788002/ref=pd_bxgy_14_2?_encoding=UTF8&pd_rd_i=1942788002&pd_rd_r=X06KH50NB8J5VGETZ08K&pd_rd_w=KbJni&pd_rd_wg=YOSIC&psc=1&refRID=X06KH50NB8J5VGETZ08K

https://www.amazon.com/Continuous-Delivery-Deployment-Automation-Addison-Wesley/dp/0321601912/ref=pd_bxgy_14_3?_encoding=UTF8&pd_rd_i=0321601912&pd_rd_r=ESKZH0HSSS55Y5V81240&pd_rd_w=dVDLs&pd_rd_wg=DGsJ6&psc=1&refRID=ESKZH0HSSS55Y5V81240

https://www.amazon.com/Leading-Transformation-Applying-DevOps-Principles/dp/1942788010/

https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&cad=rja&uact=8&ved=0ahUKEwidsbPHn7HZAhVB9mMKHVD5Bc8QFggsMAA&url=https://puppet.com/resources/whitepaper/how-build-high-performing-it-team&usg=AOvVaw2Dj5QMala45lZj4gTvhDjd


Resources


• DBA and DevOps - The Last Frontier

• Jez Humble on Continuous Delivery

• Toyota Kata

• How to F Up “Respect for People” & Really Piss Off your Employees

• The Agile Mindset - And Beyond

• Making sense of MVP

• NUUMI 2015 – This American Life

https://www.delphix.com/ebook/dba-devops-last-frontier

https://continuousdelivery.com/blog

http://www-personal.umich.edu/~mrother/Homepage.html

https://vimeo.com/127084504

https://www.youtube.com/watch?v=C13JC_YP2Q8

http://blog.crisp.se/2016/01/25/henrikkniberg/making-sense-of-mvp

https://www.thisamericanlife.org/561/nummi-2015


Safe Harbor Statement

The preceding is not intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.


Achieving High Performance DevOps · Confidential –Oracle Internal/Restricted/Highly Restricted...

Documents

Transcript of Achieving High Performance DevOps · Confidential –Oracle Internal/Restricted/Highly Restricted...