Metrics to Power DevOps
-
Upload
collabnet -
Category
Technology
-
view
524 -
download
2
Transcript of Metrics to Power DevOps
1 Copyright ©2014 CollabNet, Inc. All Rights Reserved.
Metrics to Power DevOpsToward making decisions based on data – not opinion
Sujoy Sen
Sr. Director, Field EngineeringCollabNet, Inc.
Kevin Hancock
Vice PresidentPractice Head – IT Transformation Consulting
2 Copyright ©2014 CollabNet, Inc. All Rights Reserved.
Kevin Hancock, Senior Director Worldwide Field Operations
In his ten plus year at CollabNet, Kevin has led successful implementations of CollabNet products and services at some of the world’s largest enterprises.
Kevin works with organizations to transform their globally distributed application development teams into modern, agile organizations, helping to save millions of dollars while accelerating software delivery. Kevin holds a Bachelor of Science degree from Carnegie Mellon University, and a Masters degree from the University of Pittsburgh.
Sujoy Sen, Practice Head – IT Transformation Consulting
Sujoy is part of the leadership team in the Tech Advisory group of TechMahindra. He is heading a practice which helps customers improve the success rate of their critical IT projects. Sujoy is a Certified Six Sigma Black Belt & Manager of Organizational Excellence from American Society for Quality, a PMP, a CISA and an Agile Trainer & Coach.
Sujoy has more than 19 years of experience across various domains and functions. He has been part of Tech Mahindra’s IT & Quality consulting group for the last 10 years.
Today’s Presenters
3 Copyright ©2014 CollabNet, Inc. All Rights Reserved.
• Where are we now?
• What’s the problem?
• What do we need?
• How do we get there?
Agenda
4 Copyright ©2014 CollabNet, Inc. All Rights Reserved.4 Copyright ©2014 CollabNet, Inc. All Rights Reserved.
Where are we now?
5 Copyright ©2014 CollabNet, Inc. All Rights Reserved.
IT Priorities are changing
McKinsey Global Survey, 2014
• IT is expected to improve business process effectiveness
• IT spending on innovation is expected to increase by 50% and
spending on transaction activities is expected to go down by
15%
• Further current spending on next gen infrastructure is at it
highest in 5 years
• To address the concerns and make effectives use of additional
infrastructure spending, IT should be more agile and responsive
resulting in more innovation
• To enable innovation, a culture of experimentation where IT is
allowed to test new innovations quickly with customer, capture
feedback and continuously improve
• This requires the ability of continuous delivery resulting in better,
faster and cheaper innovation
The focus has shifted from reducing cost to improving the effectiveness of business processes
6 Copyright ©2014 CollabNet, Inc. All Rights Reserved.
Agile Upstream
32%*
Source: February 5, 2014, “How Can You Scale Your Agile Adoption?” Forrester report
*Do standup, burndown/burnup, and
short iterations; product owner; and
constant feedback.
Agile
upstream • Agile Upstream are those practices usually associated with planning, managing and executing a Software project.
• Scrum, Kanban, XP are all used upstream in the process.
• But what about Downstream?
7 Copyright ©2014 CollabNet, Inc. All Rights Reserved.
13%†
Source: February 5, 2014, “How Can You Scale Your Agile Adoption?” Forrester report
†Do upstream plus continuous
integration, continuous delivery, and
test-driven development.
Agile
downstream• Agile Downstream refers to Test
automation (Test-driven Development). Continuous Integration, Continuous Delivery.
• Test automation is a key part of any Continuous Integration process
• Successful Continuous Integration requires Build Management
• Successful Continuous Delivery requires Release Management
8 Copyright ©2014 CollabNet, Inc. All Rights Reserved.
• Agile Software Development
• Continuous Integration
• Continuous Delivery
• Visible and accessible Metrics
• Access for ALL Stakeholders (RBAC permitting)
We need an Agile System
9 Copyright ©2014 CollabNet, Inc. All Rights Reserved.
Few Companies Have Achieved Enterprise Agility
Plan (Upstream) Deploy (Downstream)
Wo
rkgr
ou
pEn
terp
rise
Perform Upstream Agile: standup, burndown/burnup, and short iterations; product
owner; and constant feedback
13%
<20% A few %
32%Perform Upstream + Downstream
Agile: Perform upstream plus continuous integration, continuous
delivery, and test-driven development
Have Scaled Agile across their organization
Have achieved
Enterprise Agility
10 Copyright ©2014 CollabNet, Inc. All Rights Reserved.10 Copyright ©2014 CollabNet, Inc. All Rights Reserved.
What’s the Problem?
11 Copyright ©2014 CollabNet, Inc. All Rights Reserved.
• Highly optimized silos or work centers make sense for Waterfall
– Trust each work center to build the most “perfect product” to pass to the next work center
– We know this rarely works
• Agile Enterprises optimize for the System
– Identify and fix bottlenecks
– Only improve a work center if it improves the system
Silos of Data and Automation
12 Copyright ©2014 CollabNet, Inc. All Rights Reserved.
• The first way:
– Systems Thinking
• The second way:
– Amplify feedback loops
• The third way:
– Culture of continual experimentation and learning
A model for DevOps Empowerment
13 Copyright ©2014 CollabNet, Inc. All Rights Reserved.
The First Way: Systems Thinking
• Focus on value streams
• Think globally
• Understand the system as a whole
• Stop problems early and often
Operations
(customer)
Development
(business)
• Centralised source code and environment configuration.
• Consistent and automated release processes.
• Decreased cycle times, faster release cadence.
14 Copyright ©2014 CollabNet, Inc. All Rights Reserved.
• Protect the integrity of the entire system of work, versus completion of tasks
• Expose visual data so everyone can see how their decisions affect the entire system
The second way: Amplify Feedback Loops
Operations
(customer)
Development
(business)
• Find and fix defects, performance and availability issues faster.
• Enable specialised functions such as Ops and InfoSec to collaborate more effectively.
• Increase communication across the entire business.
• Get more work done
15 Copyright ©2014 CollabNet, Inc. All Rights Reserved.
• Continual experimentation
• Tasking risks and learning from failure
• Repetition and practice - pre-requisites to mastery
• Create rituals that reward risk taking
• Introduce faults to increase resilience
The Third way: Culture of continual experimentation and learning
Operations
(customer)
Development
(business)
16 Copyright ©2014 CollabNet, Inc. All Rights Reserved.16 Copyright ©2014 CollabNet, Inc. All Rights Reserved.
What Do We Need?
17 Copyright ©2014 CollabNet, Inc. All Rights Reserved.
Key Metrics that Power DevOps
1. Change Volume2. Lead Time (from dev to deploy)3. Deployment Frequency4. Percentage of Failed Deployments5. Build Count By Day6. File Churn7. Build duration8. Cycle time
Traditional agile metrics have focused here
Enterprise DevOps Metrics require consistency across projects, enterprise-wide
visibility, and the ability to drill to project data and people
Open ALMPlatform
18 Copyright ©2014 CollabNet, Inc. All Rights Reserved.
• As an executive, I need to see across projects to determine the relative success of different teams.
Portfolio Dashboards
Any metric is more valuable when it is put into context, and compared with other teams.
19 Copyright ©2014 CollabNet, Inc. All Rights Reserved.
• How many user stories and new lines of code are being deployed?
Change Volume
The key is to understand the risk of a given deployment – more user stories and more lines of code adds risk.
20 Copyright ©2014 CollabNet, Inc. All Rights Reserved.
• How long does it take for a new requirement/story/task tp go from the time when it hits development to when it successfully gets deployed into production.
Average Age (from development to deployment)
Average age is an important indicator of efficiency in the process – when tracked using value stream mapping, it can help the team to visualize areas in the process which need improvement. Cycle time should reduce as the team gets a better hold of the lifecycle.
21 Copyright ©2014 CollabNet, Inc. All Rights Reserved.
• How often is the team deploying new code?
Team Deployments
For an Agile team delivering code in 2 or 4-week sprints, this metric should become predictable, and correlate closely, if not exactly to those sprints.
22 Copyright ©2014 CollabNet, Inc. All Rights Reserved.
• On average, how long are builds taking to complete?
Build Duration
This metrics should remain stable over time. Spikes in build duration should be red-flagged and investigated.
23 Copyright ©2014 CollabNet, Inc. All Rights Reserved.
• How is the team trending in terms of cycle time, over the lifecycle of a project?
Cycle Time
Technical debt will add to cycle time, but so will general complexity. Most projects start out strong, but over time degrade. Choose the level that is acceptable and seek stability at that level.
24 Copyright ©2014 CollabNet, Inc. All Rights Reserved.
Velocity and Estimation
• How are teams doing? Will they finish what is promised?
Velocity will let us know how much work a team can get done, if they are consistent in their estimation and improving; while the burnup will let us know if we are finishing work quicker than it is being added to the queue
25 Copyright ©2014 CollabNet, Inc. All Rights Reserved.25 Copyright ©2014 CollabNet, Inc. All Rights Reserved.
How Do We Get There?The Cloud Framework to Power DevOps
26 Copyright ©2014 CollabNet, Inc. All Rights Reserved.
Connect Teams, Processes, and Tools
27 Copyright ©2014 CollabNet, Inc. All Rights Reserved.
TechM ADOPT Platform with TeamForge
Your data center
Public clouds
Dashboards
Control center for delivery pipelines, with social activity streams and open pluggable API’s
GitEye
Code Search
Code Review
Git + Gerrit
Jenkins
Hudson
Tracker Discussions Wiki Documents Reports File Releases
JIRA
Selenium
PPM
One Platform, Multiple Possibilities
QTP
Platform 4
28 Copyright ©2014 CollabNet, Inc. All Rights Reserved.
Tool Snapshot
Project
Dashboard
Agile BacklogsConfiguration
Management
Unit Test / Code
quality tool
integration
Jenkins
integration
Collaboration
Tools
Code Review
Tool
Customized
Reports
Release
Planning
Defect
Tracker
Defect
Tracker/Impediment
Tracker
Infrastructure
Automation
Traceability for
application lifecycle
activities
Task
Tracking /
Managing
Sprint
Platform 4
29 Copyright ©2014 CollabNet, Inc. All Rights Reserved.
Scrum
Team 2
Test (T) QA
(Q)
Scrum
Team 1
Dev (D)
T A Prod
Prod (P)
D
Sprint
Build?
Build?
Unit tests
Unit tests
Unit tests
Build?
Continuous Integration
Continuous Testing
Automate
d Tests
Automate
d Tests
Unit tests
Continuous Deployment
Continuous Release & Monitoring
Co
nti
nu
ou
s P
lan
nin
g
Continuous Delivery
Build (B)
Daily DailyDaily/Release
point
• Devops should create a
flow that results in faster
through put time
• Our approach ensures that
we improve the overall
throughput by bringing in
process, tools, automation
and training interventions
where the flow gets stuck
• TechM proposes a holistic
approach to enable Devops
in an organization by
slowing eliminating legacy
practices which don’t
promote collaboration
Focus on Cadence and Orchestration
30 Copyright ©2014 CollabNet, Inc. All Rights Reserved.
Simplified process for continuous delivery
Task
to
Package
Idea
to
task
Package
to Deployment
Business
vision
Products
Identification
Product
Roadmap
Product Epics (product
backlog)
Release PlansSprint Plans
(Sprint backlog)Tasks
Design CodeAutomated
Unit test
Automated Code
quality check
Trigger
Integration
Automated
BuildPackage
Check-in to
repository
Deploy to
QA Env*
Manual/ Automated
SIT Testing
Deploy to
UAT Env*
Manual / Automated UAT
Testing
Deploy to Production
Env*
S
E
S EStart End
Automated monitoring in
production
I2T
T2P
P2D
Processes
1
Processes
1
31 Copyright ©2014 CollabNet, Inc. All Rights Reserved.
Phased Approach
Isolated Teams
SME Shared
Teams
Continuous
Planning
Continuous
Release and
Monitoring
Continuous
Integration
Continuous
Testing
Continuous
Delivery
Product Teams
Multi Layered
Teams
Multidisciplinary
Project Teams
Demand pipeline
established; Release plans
are developed and
maintained for individual
applications; Projects are
able to establish
traceability across
requirements, design,
development, testing and
defect management is
maintained through out
the lifecycle
Projects are able
demonstrate the
behavior of continuous
feedback; as opposed
to a periodic schedule,
builds are triggered by
every commit to the
single code repository
Project is able to align
itself with enterprise
release cadence;
environments are
provisioned ‘on
demand’; releases
deployed across
multiple environments
including data
migration
Continuous regression
reduces the number of
defects; most
integrated bugs are
fixed regularly to avoid
last minute changes;
test cases reused;
‘non-functional’ code
quality measured and
technical debt
improved
Dev and ops team work
together seamlessly
delivering both projects
and managing
operations. Metrics are
captured to monitor
process performance
and continuous
improvement.
Level 1
Level 2
Level 3
Level 4
Level 5
VALUE
CO
LLA
BO
RA
TIO
N
Capability Model
2
• Our assessment framework is built
on this staged approach. Staged
approach does necessarily mean
that it is a sequence
• TechM would evaluate organization
as-is status against these stages
• As-is assessment would cover
people, process, tools, policies, KPIs
and flow
• As an outcome of the assessment
we would provide the gaps and the
transformation roadmap to cover
these gaps
• The Devops consultant/ Program
manager would work towards
implementing this transformation
roadmap
32 Copyright ©2014 CollabNet, Inc. All Rights Reserved.
Transformation Phases
33 Copyright ©2014 CollabNet, Inc. All Rights Reserved.
iSPEED tm Methodology
Study Plan Enable and Expand Diffuse
• Review historical data,
documents, tools and processes
• Understand the organization
structure
• Interview key stakeholders
using diagnostic tool
• Prepare business case
• Identify multiple solutions
blueprints
• Prepare transformation
roadmap
• Identify Enterprise Change
Enablement team
• Develop workflows as per
selected blueprint
• Review the workflows with all
stakeholders
• Tune up workflows
• Develop detailed process,
templates, checklists, guidelines
• Modify tool workflows
• Facilitate deployment planning
workshops, prioritize process
activities , prepare deployment
plans
• Enable teams by training and
support
• Identify projects and deploy
• Aggregate data
• Enable Continuous
Improvement
• Quantify benefits and Showcase
projects
• Collate and analyze
performance data
• Benchmark and improve
• Repot
• Establish improvement log
• Continue deployment support
Interview• Interview key
customer
stakeholders
• Understand customer
context
• Abstract key business
drivers
• Enlist the top
priorities
2 to 3 WKS 4 to 6 WKS 8 to 12 WKS OngoingMax 4hrs
Feasibility Decision
• Business case
• Solution blueprint
• Transformation roadmap
• Process Workflows
• Processes, templates, checklist
• Trainings documents
• Project data
• Audit & Management reports
• Management reports
• Improvement log
• Fixed Price
• Fixed Price
• Time and Hire
• Outcome based pricing
• Fixed Price
• Time and Hire
• Outcome based pricing
• Time and Hire
Deliverables
Engagement model
34 Copyright ©2014 CollabNet, Inc. All Rights Reserved.
DevOps Transformation Assessment (DOT)
ADCM provides a structured and tested
approach for any organizations Devops
strategy. Outcome of the assessment
provides a roadmap that can be used for
Devops strategy and rollout
35 Copyright ©2014 CollabNet, Inc. All Rights Reserved.
DOT Analytics
0
0.5
1
1.5
2
2.5
3
3.5
4
Business ITAlignment
Portfolio planning
Release planning
Capability planning
Capacity planning
Infrastructureplanning
Agile deliveryplanning
Task planning
Value addition
Collaboration
Institutionalization
• Each stage has multiple focus areas.
The depiction here is for continuous
planning
• To arrive at the score, TechM
consultants would speak to various
stakeholders and take them through a
series of questions
• Each focus area is evaluates for
• the value addition that it is
currently bringing,
• The collaboration it facilitates
• Level of implementation that
has been achieved
• Focus areas where today nothing exists
is given a score of zero
• Further for each area a detailed
improvement plan is prepared
Continuous planning
36 Copyright ©2014 CollabNet, Inc. All Rights Reserved.
DOT Roadmap Sample
Continuous planning W1 W2 W3 W4 W5 W6 W7
Business IT
alignment
People
Process
Tooling
Flow & integration
Performance & outcomes
Deliverables
Portfolio
Planning
People
Process
Tooling
Flow & integration
Performance & outcomes
Deliverables
Release
Planning
People
Process
Tooling
Flow & integration
Performance & outcomes
Deliverables
Capability
Planning
People
Process
Tooling
Flow & integration
Performance & outcomes
Deliverables
Capacity
Planning
People
Process
Tooling
Flow & integration
Performance & outcomes
Deliverables
Infrastructure planning
People
Process
Tooling
Flow & integration
Performance & outcomes
Deliverables
Agile delivery planning
People
Process
Tooling
Flow & integration
Performance & outcomes
Deliverables
Task
planning
People
Process
Tooling
Flow & integration
Performance & outcomes
Deliverables
37 Copyright ©2014 CollabNet, Inc. All Rights Reserved.37 Copyright ©2014 CollabNet, Inc. All Rights Reserved.
Thank You
Contact Details:
Kevin Hancock - [email protected]
Sujoy Sen – Tech [email protected]
38 Copyright ©2014 CollabNet, Inc. All Rights Reserved.38 Copyright ©2014 CollabNet, Inc. All Rights Reserved.
© 2014 CollabNet, Inc., All rights reserved. CollabNet is a trademark or registered trademark of CollabNet Inc., in the US and other countries. All other trademarks, brand names, or product names belong to their respective holders.
CollabNet, Inc.8000 Marina Blvd., Suite 600Brisbane, CA 94005
www.collab.net
+1-650-228-2500+1-888-778-9793blogs.collab.nettwitter.com/collabnetwww.facebook.com/collabnetwww.linkedin.com/company/collabnet-inc