Post on 06-Dec-2014
description
Agile Metrics
Craig RudmanVP Services, Agile Coach
Purpose
• Greater alignment of strategy to execution• Improve the flow of work• Improve product quality• Support continuous improvement efforts• Improve the quality of life• Increase trust
Discuss ways to go about measuring work and workflow
Roadmap
Context
• What can we measure?
• What should we measure?
• What principles inform agile metrics?
Systems Thinking
• Who is your customer?
• The nature of demand
• Value creation
Agile at Scale
• Portfolio• Program• Team
Why Metrics?
• Increase visibility of work• Increase alignment of strategy to
execution• Improve trust• Improved product quality• Improved workflow• Improved performance• Identify and solve problems
Metrics give us insight into the health of our organization, our people, processes, and tools.
We use metrics to help us adapt and improve
We can quantify those things we can seeWe can qualify the things we can't see
What can we measure?
Quantify• Effort• Time• System Components• Tests• Events• Data
Qualify• Quality• Perceptions• Feelings• Attitudes
We should measure the things we value
What should we measure?
If your goal is…• Improving visibility of work
• Better alignment• Increasing trust• Improving quality• Improving process• Solving problems
then you should measure…→ work-in-process→ context→ commitments→ breakage and rework→ flow→ root causes
The Manifesto for Agile Software Development
We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:
• Individuals and interactions over processes and tools• Working software over comprehensive documentation• Customer collaboration over contract negotiation• Responding to change over following a plan
That is, while there is value in the items on the right, we value the items on the left more.
The Twelve Principles of Agile Software Development • Our highest priority is to satisfy the customer through early and continuous delivery of
valuable software.• Welcome changing requirements, even late in development. Agile processes harness change
for the customer's competitive advantage.• Deliver working software frequently, from a couple of weeks to a couple of months, with a
preference to the shorter timescale.• Business people and developers must work together daily throughout the project.• Build projects around motivated individuals. Give them the environment and support they
need, and trust them to get the job done.• The most efficient and effective method of conveying information to and within a
development team is face-to-face conversation.• Working software is the primary measure of progress.• Agile processes promote sustainable development. The sponsors, developers, and users
should be able to maintain a constant pace indefinitely.• Continuous attention to technical excellence and good design enhances agility.• Simplicity--the art of maximizing the amount of work not done--is essential.• The best architectures, requirements, and designs emerge from self-organizing teams.• At regular intervals, the team reflects on how to become more effective, then tunes and
adjusts its behavior accordingly.
Agile Metrics Support Agile PrinciplesIf motivated by these principles… Measure for these outcomes…
Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.
Customer satisfaction Cycle time and throughput Product value realized
Welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage.
Changeability of requirements Value of change
Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.
Frequency of delivery Product stability
Business people and developers must work together daily throughout the project. Frequency/quality of collaboration Delays
Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.
Alignment of work to motivated people Quality of team support Release predictability
The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.
Efficiency of communication Effectiveness of communication
Working software is the primary measure of progress. Rate of product adoption Frequency of product usage Failure demand Breakage
Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.
Burnout, attrition, turnover, morale Stability of cadence and velocity Stability of team organization
Continuous attention to technical excellence and good design enhances agility. Skills growth Increasing rates of reuse Design/code consistency and quality Test coverage/automation
Simplicity--the art of maximizing the amount of work not done--is essential. Low complexity Short work queues
The best architectures, requirements, and designs emerge from self-organizing teams. Internal locus of control High feature/story acceptance rates
At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.
Consistency of cadence Effectiveness of retrospectives Progress towards improvement goals
Exercise: Five Votes
• Read through the agile principles• Think about where you would like to see
improvement• Cast your five votes accordingly
– You can put them all on one item– You can distribute them across up to five items
• Working as a group, we will discuss a set of metrics to support the highest-priority item
Customers Suppliers
Assets/ExpensesProfit/LossCash Flow
Market ShareCustomer SatisfactionEmployee SatisfactionShareholder Benefit
Society BenefitKnowledge Creation
Product/Service QualityTime to Market
Demand
Revenue
Resources
Take a Systems View
Demand
Value
Revenue
Metrics help us understand and manage these things
The primary metric for agile is whether or not value is being created. This is determined empirically, by demonstration, at the end of every single iteration.
Your business is a value-creation system
The Big Blue Company
Information Services
Data Management
21
Application Development218
Infrastructure/Ops77
Services94
Claims (206)
Prov. & Gov't Relations (162)
Sales & Marketing (146)
Finance (106)
All Others (155)
Customer Service (227)
Membership (129)
Who is your customer?
Demand
Value
Revenue
Mandates
Preven
tive M
aintenance
How much work can you manage?
Initiatives
Production Support
Enhancement Requests
Defects
Rework
Stuff and Nonsense
74%
18%
8%
How much work can you manage?
Production Support
Enhancement Requests
Defects
Rework
Stuff and Nonsense
Mandates
Preven
tive M
aintenance
Initiatives
209,320
49,78822,150
Total capacity = 281,258 hrs
Types of Demand
• What is the nature of customer demand?– How much of it is failure demand?– How much of it is mandates?– How much of it is value demand?
• If demand exceeds capacity, what are your options?
“All time spent handling failure demand is waste.”*
*Mary Poppendieck
Portfolio Management
• Failure Demand (from 74% to 50%)– Defects– Rework– Missing Features
• Value Demand (from 8% to 20%)– Preventive Maintenance– New Features or Services– New Capabilities
• Mandates (from 18% to 30%)– Regulatory compliance
50%
30%
20%
Failure Demand Mandates
Value Demand
Constrain effort to align with strategy
It’s All About Choices• Failure Demand
– Take a long-term view– Perform root-cause analysis to find the source of greatest demand– Fix it– Repeat the process
• Value Demand– Rank initiatives according to highest value– Estimate the size of the effort– Find the highest value with the least effort– Build that thing with the least delay– Repeat the process
• Mandates– Fix the due date– Prioritize features using a MoSCoW model (Must, Should, Could, Won't)– Evaluate design options and favor the simplest approach, even if sub-optimal– Optimize workflow to maximize throughput
How we build itWhat we build
Things We Can Measure
Performance• Throughput• Cycle Time• Effort• Schedule• Inventory
– Work Not Started– Work In Progress
Value• ROI• Revenue• Cost• Cash Flow• Market Share• Growth• Quality
Value Goals
• Increase revenue• Increase market share• Increase customer satisfaction• Reduce cost• Manage risk
Features Deliver Value…• Improve customer
satisfaction for both members and providers
• Increase the number of claims processed per employee
• Reduce administrative costs as a percentage of revenue
Add claim status inquiry to web
Add additional conditions to claims adjudication so that
fewer claims "PEND"
Reduce dependencies between systems by introducing service
layer to architecture
… But Only When In Production
• How long will you have to wait before the feature makes it into production?
• How long will the feature have to be in use before it pays for development and begins to add value?
Ideation
Design
Planning
Coding
Testing
Staging
Release
Effort incurs development costs
Time waiting for new features represents lost opportunity costs
Cost/Benefit Analysis of Features
• Begin by establishing the value proposition• Evaluate your solution options• Calculate development effort and cost of each• Calculate the cost of delay• Track adoption and use of the feature to
measure value
Example: Claim Status on the Web
Every day, Customer Service handles some number of phone calls with certain percentage of those calls are inquiring into claim status. That leads us to the following analysis:
• Assume 2400 phone calls each day on average• Assume 20% of them on average (480) are claim status inquiries• Assume 10 minutes per call on average• Assume $50 per hour is the average fully-loaded cost of a CSR• 4800 minutes or 80 hours per day on claim status inquiry equates
to $4,000 per day• If there are 253 work days in a year, that equals $1,012,000 spent
responding to claim status inquiry each yearNote: the example shown is purely fictional and for illustration purposes only.
Evaluating Options
Three solutions are considered:1. Do nothing2. Develop a feature with fully-committed
teams in four months at a cost of $255,0003. Develop the same feature with partially-
committed teams in sixteen months at a cost of $338,000
Jan-13
Feb-13
Mar-13
Apr-13
May-13
Jun-13Jul-1
3
Aug-13
Sep-13
Oct-13
Nov-13
Dec-13Jan
-14
Feb-14
Mar-14
Apr-14
May-14
Jun-14Jul-1
4
Aug-14
Sep-14
Oct-14
Nov-14
Dec-14Jan
-15
Feb-15
Mar-15
Apr-15
May-15
Jun-15Jul-1
5
Aug-15
Sep-15
Oct-15
Nov-15
Dec-15
$-
$20,000
$40,000
$60,000
$80,000
$100,000
$120,000
$140,000
$160,000
$180,000
Option 1Option 2Option 3
Cost of DelayTotal 3-year cost:1. Do Nothing: $3,040,0002. Dedicated team: $977,0003. Shared team: $2,070,000
$1,093,400
Performance Goals
• Increase throughput• Reduce cycle time• Improve quality• Eliminate waste• Improve predictability
Better Processes Improve Performance
• Increase throughput• Decrease schedule
variance• Shorten feature cycle
time
Increase frequency of releases
Negotiate scope
Reduce WIP to move smaller batches through development
Goal: Improve Throughput
• Features per quarter• Stories per sprint• Issues per day• Value per month
Throughput is the rate at which a system achieves its goal in units of output over time
High Variability Reduces Flow
Symptoms– Flow becomes
unpredictable– Bottlenecks occur– Cycle time increases– Throughput goes down
Response– Reduce WIP– Increase slack– Swarm around bottlenecks
Knowledge work inherently has a high degree of variability
Flow
Design Capacity: the maximum amount of work that a system is capable of completing over time
Cycle Time: the amount of time that work spends in the system
Utilization: the amount of capacity being used at a given time (%)
Effective Capacity: the actual amount of work that a system is capable of completing over time
WIP: the amount of work in that is in flight at a given time
Slack: the amount of capacity not being used at a given time (%)
Throughput: the average rate at which completed work exits the system
Constraint: anything that limits throughput
Collecting Flow Data
• Model value stream• Record transitions• Calculate cycle time• Calculate WIP
Feature Ready Dev/Test Add'l Test DeployCycle Time
(days)1 2/1/13 2/2/13 2/2/13 2/3/13 22 2/1/13 2/2/13 2/3/13 2/3/13 23 2/1/13 2/3/13 2/3/13 2/5/13 44 2/1/13 2/3/13 2/4/13 2/5/13 45 2/1/13 2/3/13 2/6/13 2/7/13 66 2/1/13 2/4/13 2/9/13 2/9/13 87 2/1/13 2/4/13 2/9/13 2/9/13 88 2/1/13 2/6/13 2/9/13 2/10/13 9
Date Ready Dev/Test Add'l Test Deployed Total WIP2/1/13 10 0 0 0 102/2/13 8 1 1 0 102/3/13 5 2 1 2 82/4/13 7 3 2 2 122/5/13 7 3 0 4 102/6/13 6 3 1 4 102/7/13 8 3 0 5 112/8/13 9 4 0 5 132/9/13 10 2 2 7 14
2/10/13 9 1 3 8 13
Cumulative Flow Diagram
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 2402468
10121416182022242628303234363840
ReadyDevTestDeployed
WIP (9 items)
Throughput (1.25 items/day)
Cycle Time (10 days)
Agile Teams
• Team: a group of people linked together with common purpose
• Backlog: Work identified, but not yet started• Sprint: a time-boxed interval of time during which work is
completed• Story: a narrative about the feature to be developed with clear
completion criteria• Story Points: an estimate of the size of a story compared to
other stories• Velocity: team throughput, as measured by story points per
sprint
Velocity Measures
• Burndown: total work remaining
• Burnup: total work completed
• Scope Change: additions or subtractions of work from the backlog
• Glidepath: estimated time remaining until the backlog is depleted based on available data
SprintPts
Completed Burnup BurndownScope
Changes Glidepath0 750 9001 40 40 790 80 9152 90 130 700 8503 10 140 790 100 7844 75 215 765 50 7195 120 335 565 -80 6546 32 367 533 5897 60 427 473 5238 95 522 378 4589 393
10 32811 26212 19713 13214 6715 116 0
1 2 3 4 5 6 7 80
100
200
300
400
500
600
700
800
900
1000
BurnupBurndownGlidepath
Velocity Reporting
Velocity = 65 pts/sprint
Velocity Pros and ConsUsed Well• Helps teams make commitments
that they can keep• Helps estimate the depth of a
backlog• Helps product owners know how
much runway to prepare• Helps teams reflect on their
efficiency and identify ways to improve
• Accurately reflects the sustainable pace of a team
• Emphasizes the importance of “done”
Used Poorly• Misunderstood as a measure of
team effectiveness• Misused as a measure of
individual performance • Invites meaningless or damaging
comparisons between teams• Leads to Inflated Velocity
Syndrome (IVS), a misguided attempt to improve throughput by focusing on utilization rather than outcomes.
Commitment
• Scope is negotiable• Teams are self-governing• Deadlines matter• Done means done
Agility is rooted in commitment
Commitment Variance
1 2 3 4 5 6 7 80
20
40
60
80
100
120
140 PlanActualRange (-20%)Range (+20%)
A Good Metric
• Supports your goals• Fits your process• Is fact-based• Difficult to
manipulate
Inspect and AdaptPortfolio• Inspect: Demand (planned and unplanned)• Adapt: Control allocation of effort to demand categories
Program• Inspect: Workflow; feature acceptance rates; value attainment• Adapt: Reduce WIP; negotiate feature scope and priority; remove
bottlenecks; increase slack; smaller, more frequent releases
Team• Inspect: Team throughput, commitment variance, quality• Adapt: Negotiate story scope and priority, remove impediments,
regular retrospectives, continuous improvement