Freedom and Responsibility

Freedom and ResponsibilityContinuous Delivery and DevOps

Mike Ruangutai

Head of Platform Engineering

@

Previous lives

Kareo

Expansion stage start-up providing SaaS clinical and practice management software.

$15 billion in patient claim transactions annually

18,000 doctors and healthcare providers

39 million patient records including ePrescriptions Drug-to-drug interaction Patient-drug allergies Lab orders and results processing

Weekly software releases, with road map to daily releases.

IGN | GameSpy

Platform-as-a-Service for triple-A and independent game developers and publishers.

300 million users across all major gaming platforms.

70 million monthly views on IGN.com

1100 web, application, and database servers

Peak traffic: 3 billion requests/day 25,000 data transactions/second

Release on commit, aka Continuous Deployment

What I learned along the way Understand why you want to change

Get early wins; this journey will likely take years

Do Continuous Integration first

Early software starts monolithic and will require architecture changes

Culture of trust and continuous improvement

Why do we want to do continuous delivery?

How many believe we are paid to write code?

We are paid to make decisions.

Decisions lead tooutcomes.

Positive

Negative

Outcome

How do we make decisions quickly and adjust to enable the business?

Adrian CockcroftChief Architect, Netflix

• Originally coined by military Air Force strategist John Boyd

• Predominant philosophy in many Bay Area companies

OODA Loop

http://en.wikipedia.org/wiki/OODA_loop

Ideas

Build

Code

Measure

Data

Learn

Build, Measure, LearnEric RiesCTO, IMVUFounder Lean Startup Movement

Goal: To get around this loop as fast as possible.In the same time duration:

Opponent: 3 decisionsYou: 6 decisions

You have the advantage.


What does this have to do with product engineering?

Maslow’s Hierarchy of NeedsAs product engineers, our goal is to get as high up this hierarchy as possible.

And getting higher requires the ability to make more decisions, use more data, and take more actions than your opponent.

At minimum get here

Goal is to get here

Continuous Delivery


Enables product engineering teams to get around this loop quickly by delivering software up the value stream frequently and consistently.

Who is doing continuous delivery?

Steven Greene - VP, Program Management

Mike Saha – Director, Release Engineering• 900+ deployments annually

• 8 releases/day• 25 core products• Up to 120,000

automated tests• 120+ Scrum teams• Multi year transformation

FutureSmart Printer Software

Embedded Software Systems powering HP printer, scanner, and multi-function devices.

Gary Gruver - VP, Quality Engineering

• $2 billion LOB• 400 engineers across four

continents• Re-architected from ground

up• 10 million lines of code

• 75,000 lines of code churn/day

• 15,000 testing hours/day• 3 year transformation, 2008

- 2011




Cost Driver Improvements




Cycle Time Improvements

What is continuous delivery?

“Reduce the cost, time, and risk of delivering incremental changes to users.” – Jez Humble, author Continuous Delivery , VP @ Opscode

1. Software is deployable throughout its lifecycle.

2. The team prioritizes keeping the software deployable over working on new features.

3. Anybody can get fast, automated feedback on the production readiness of their systems anytime somebody makes a change to them.

4. You can perform push button deployment of any version of the software to any environment on-demand.

“…a business sponsor could request that the current development version of the software be deployed to production at a moment’s notice – and nobody would bat an eyelid, let alone panic.” - Martin Fowler

Pillars of Continuous Delivery

Continuous Delive

ry

Continuous

Integration

Automated

Testing

DevOps

To become a continuous delivery organization, DevOps, Automation Testing, and Continuous Integration are integral parts of the process.

Continuous IntegrationWhat does it mean to be doing continuous integration?

1. Software is deployable throughout its lifecycle.

2. The team prioritizes keeping the software deployable over working on new features.

3. Anybody can get fast, automated feedback on the production readiness of their systems anytime somebody makes a change to them.

4. You can perform push button deployment of any version of the software to any environment on-demand.

1. Engineers are checking in code several times a day to trunk; feature branches are frowned upon.

2. Builds are generated and automated tests are executed to validate the integrity of the software.

3. Builds and test times are kept fast to ensure fast feedback cycles.

4. Testing is done continuously in a production mirrored environment.

Delivery

team

• Checks in

Version

control

• Triggers

Build & Unit Tests

• Triggers

Automated

Acceptance Test

• Approval

User Acceptance Test

• Approval

Production

Release

Continuous Delivery Pipeline

Continuous Integration pipline

Automated TestingWho is responsible for quality?When does quality become important?

“Cease dependence on mass inspection to achieve quality. Improve the process and build quality into the product in the first place.”

- W. Edwards Deming

W. Edwards Deming

Created Total Quality Movement after World War II in Japan.

Core to Japan’s dominance of US car manufacturers

Two implications to software development:

1. Testing is not something we do after development is complete. Testing should happen all the time.

2. Testers are not responsible for quality. Everyone is responsible for quality.

W. Edwards Deming

Created Total Quality Movement after World War II in Japan.

Core to Japan’s dominance of US car manufacturers

Functional AcceptanceUsability

ExploratoryShowcase

Non-functional acceptance

(performance, scaling)

UnitIntegration

Automated Manual

Automated Automated | Manual

Technology Facing

Development Support Product feedback

Business Facing

Cheapest way to fix a bug is to not check it in in the first place.That is why automated tests are important.

DevOpsBuilding a Culture of Trust

Where we are today…

Organizational gatekeepers to control risk and increase trust.

Diametric opposition Product paid to ideate changes. Developers are paid to make changes. QA are paid to question changes. Operations are paid to prevent changes. Often times because they end up having to support those changes.

*Change = Risk

Goal: To get around this loop as fast as possible.If this is our goal, then work center silos detract from this goal.

Continuous Delivery will ultimately fail.


Agility without SafetyIs easy.

Safety without AgilityIs easy.

Safety with AgilityIs hard.

DevOps is…

A culture shift that encourages collaboration and communication to foster building better-quality software more quickly and more reliably.

An environment where developers, QA, and operations draw on each others’ expertise to eliminate waste and bottlenecks in the development lifecycle.

DevOpsandOODA

Metrics:• Performance • Availability• Usability• Cohort• Traffic

Product enablement:• Feature toggles• Canary testing• A/B testing

Using Product and Technology metrics to move higher in Maslow’s Hierarchy of Needs

Development:• CI Server• Infrastructure Automation• Test Automation• Agile architecture, i.e. micro services

DevOps

DevOps is about Freedom and Responsibility.

Summing it all up.


Continuous Delive

ry

Continuous

Integration

Automated

Testing

DevOps

All three are fundamental.


Continuous Delive

ry

Process

Technology

People

Alternatively you can look at it this way.

Achieving Continuous Delivery is hard. But it is fundamental.It will likely take us a while to get there. But the most critical requirement are…

People.People are the root of the creative process. And getting to continuous deliver will be a creative endeavor.

Among other things, we should have:

1. The Culture of Trust which is created by giving responsible people the freedom to make decisions (and experiment).

2. The tools and technology that provide responsible people the data they need to make decisions.

3. A process that gives responsible people the fast cycle times needed to make quick decisions - OODA.

How do you know you’ve done it?

“…a business sponsor could request that the current development version of the software be deployed to production at a moment’s notice – and nobody would bat an eyelid, let alone panic.” - Martin Fowler

Thanks!

Freedom and Responsibility

Documents

Transcript of Freedom and Responsibility