Practical Uses of Software Measurement for Process Improvement
January 10, 2007 - V1.0
Larry Dribin, Ph.D.
847-807-7390
01/10/2007CSPIN f v1.0 - © Pearl Street Group 2007 2
Agenda
Software Measurement Programs IT Measurement for Many Companies Today Software Process Improvement Programs Tailoring a IT Measurement Program to Support Process
Improvement
The goal of this presentation is to discuss how software measurement programs can be designed to better support process improvement.
The goal of this presentation is to discuss how software measurement programs can be designed to better support process improvement.
Lord Kelvin Comment:‘When you can measure what you are speaking about and express it in numbers, you know something about it; but when you cannot measure, when you cannot express it in numbers, your knowledge is of a meager and unsatisfactory kind…”
Lord Kelvin, Popular Lectures and Addresses, 1889
Lord Kelvin Comment:‘When you can measure what you are speaking about and express it in numbers, you know something about it; but when you cannot measure, when you cannot express it in numbers, your knowledge is of a meager and unsatisfactory kind…”
Lord Kelvin, Popular Lectures and Addresses, 1889
01/10/2007CSPIN f v1.0 - © Pearl Street Group 2007 3
The Measurement Cycle
Why Measure:
To Manage– Quality– Cycle Time– Productivity– Risk
To Communicate– Management– Peers
To Improve Performance(Process Improvement)
“What gets measured gets done!”“What gets measured gets done!”
Measures
Decisions
Facts
Actions
Results
01/10/2007CSPIN f v1.0 - © Pearl Street Group 2007 4
Software Measurement Programs
Analysis &Decision Making
Processes
MeasurementStorage Process
MeasurementCollectionProcesses
DashboardProduction Process
And Reporting
Get Value!
Make
Decisions
OriginalProcesses &Organization
RefineProcesses, Activities And/or Organizations
01/10/2007CSPIN f v1.0 - © Pearl Street Group 2007 5
IT Measurement for Many Companies Today
Gather and report on numerous measures (25 to over 100) Typically focus is on financial and operational performance
measures
Example Financial Measures:– IT Budget – Plan vs. Actual– Hours spent on discretionary development vs. “maintenance”
Example Operational Measures– Number of calls to the Help Desk– Number of calls answered by 1st level support
01/10/2007CSPIN f v1.0 - © Pearl Street Group 2007 6
Example Financial Measure
IT Budget vs Actual Expense - Total IT
AnyCo Common Costs (I.e help desk), which accrued for the entire year, were allocated to September resulting in the large variance.
$-
$500.0
$1,000.0
$1,500.0
$2,000.0
$2,500.0
Jan Feb Mar Apr May Jun Jul Aug Sep Oct
Thousands
Actual Plan
01/10/2007CSPIN f v1.0 - © Pearl Street Group 2007 7
Example Operational Measure
Trouble Ticket Run Rate and Escalation to Level 2
Customer Care Center Service Objective for Problems Escalated to Level 2 is <60%
2174
2699 2505
2963 2863
3261
2839
1283 1538
1247 1378 1519 1386 1467
1338
2268
57% 55%
51% 48%
45% 47%
59% 55%
0
500
1000
1500
2000
2500
3000
3500
Mar Apr May Jun Jul Aug Sep Oct 0%
10%
20%
30%
40%
50%
60%
Tkts Escal to Lev 2 % Escal to Lev 2
01/10/2007CSPIN f v1.0 - © Pearl Street Group 2007 8
Software Process Improvement Programs
There are many types of process improvement initiatives that IT Organizations are pursuing today:– CMMI– ITIL– CoBiT / SOX– TQM– QA Improvement– Testing Improvement– Agile Development
All of these initiatives require process measurement to be effective
Strive for “Best Practices”, but require “GSP – Good Software Practices”.
Similar to FDA requirement that Pharmaceutical companies follow “GMP –Good Manufacturing Practices”.
Strive for “Best Practices”, but require “GSP – Good Software Practices”.
Similar to FDA requirement that Pharmaceutical companies follow “GMP –Good Manufacturing Practices”.
01/10/2007CSPIN f v1.0 - © Pearl Street Group 2007 9
Process Improvement Measures
OriginalProcesses
OriginalProcesses
Process Progress Measures measure how well the process is being
implemented:
ValueTomorrow
ValueToday
RestructuredProcesses
RestructuredProcesses
Outcome Measures measure the result of
the improvement:(Financial, Operational,
&/or Customer)
Process improvement measures consist of both Outcome and Progress Metrics
Process improvement measures consist of both Outcome and Progress Metrics
01/10/2007CSPIN f v1.0 - © Pearl Street Group 2007 10
IT Measurement and Software Process Improvement
The IT financial and operational measures described above are typically designed to assist IT management explain what IT has been doing for the business
This is important, because if business managers do not feel that they are getting value for their IT dollars, they:– Reduce IT Budgets – Reduce IT Headcount– Outsource IT
As a result of this focus, many IT Measurement Programs are not designed to support process improvement.
As a result of this focus, many IT Measurement Programs are not designed to support process improvement.
01/10/2007CSPIN f v1.0 - © Pearl Street Group 2007 11
CMMI Measurement Focus Measurement & Analysis is a Key Process Area for the CMMI Focus on is developing Measurement Programs:
Collecting, storing, reporting and analyzing measurement results Measurement is embedded in all Key Process Areas (KPA) CMM Measurement Requirements for Requirement
Management KPAMeasurements are made and used to determine the status of the activities for managing requirements.
Examples of these measures include:– Status of each requirement– Change activity for each requirement– Cumulative number of changes to the requirements, including
total number of changes proposed, open, approved and incorporated into the system baseline.
These practices provide high level guidance when developing specific measures for process improvement
These practices provide high level guidance when developing specific measures for process improvement
01/10/2007CSPIN f v1.0 - © Pearl Street Group 2007 12
Designing Measurement Programs
Follow a similar approach to application development:– Gather Measurement Requirements– Analyze and Prioritize Measurement Requirements– Design a Measurement Solution– Develop the Measurement Solution
• What to collect• How to collect• Where to store• When and How-Too Present and Report Measurements Results
– Deploy Measurement Solution– Harvest the Benefits
Useful techniques:– Goal->Question->Metric to identify key metrics– Dashboards to organize the metrics for specific audiences– Balanced Score Cards to develop balanced dashboards linked to
organizational goals
01/10/2007CSPIN f v1.0 - © Pearl Street Group 2007 13
Goal->Question->Metric Technique
Well known technique developed by Victor Basili to identify the measures needed for process improvement
1. Define Goals using the Balanced Score Card or other technique Example Goal: Let’s improve programmer productivity.
2. Ask Questions about goals to define the process improvement initiative in more detail
Questions: How much does an average developer develop?
3. Questions lead to defining specific measures such as: Number of developers working in a month Time each developer spends working in the month Total number of Lines of Code (LOC) (or Function Points – FP)
Produced by each developer per month Productivity = (#LOC or FP) / (# Hours Worked)
-> Average Lines Of Code (LOC) or Function Points (FP) produced per developer per month
01/10/2007CSPIN f v1.0 - © Pearl Street Group 2007 14
Example – Requirements Management Effectiveness
Goal:– Improve Requirements Management
Questions:– How many Requirement do we have?– Are these the right requirements for our problem?– Has each Requirement been approved by all key stakeholders?– Has the Requirement changed?
Potential Metrics:– Process Progress Measure:
• Percent of Project Teams following the new/improved process– Outcome Measures:
• Number of Requirements • Status of each Requirement• Number of changes to the requirements• Number of Requirements Defects found during Requirements
Validation• Number of Requirements Defects found during Coding• Number of Requirements Defects found during Testing
01/10/2007CSPIN f v1.0 - © Pearl Street Group 2007 15
Example – Production Defects Goal:
– Reduce Application Production Defects Questions:
– What is a Defect?– What is an Application?– What Application did the Defect occur in?
Potential Measures– Process Progress Measures:
• The percent of Project Teams tracking Defects using standard defect classifications
– Outcome Measures: • Production Defects – defined as defects logged to the Help
Desk by Application by Defect Type using standard defect classifications
01/10/2007CSPIN f v1.0 - © Pearl Street Group 2007 16
Detailed Pareto of Trouble Tickets within Category
Trouble Ticket by Category - PC Software
Data for October 20xx
180 164
100
70 66 65
57 55 48
37 29
25 22 16%
31% 41%
47% 53%
59% 64%
69% 74% 77% 80% 82% 84%
0 20 40 60 80
100 120 140 160 180
BSCS
EXCHANGE
OTHER
.INSTALL NEW SOF
TWAR
WINDOWS 95
SAP MS-WORD
DIVERCTI
INTERNET EXPLORER
MS-EXCEL
FAX SR. NT CAR
S REMEDY
01/10/2007CSPIN f v1.0 - © Pearl Street Group 2007 17
Example – Testing Metrics Goal:
– Measure Regression Test progress and when completed Questions:
– What is Regression Test Progress?– How do we know that we are done?
Potential Measures– Process Progress Measures:
• The percent of Project Teams tracking Regression Test Progress– Outcome Measures:
• Number of Regression Tests Planned• Number of Regression Tests Completed• Percent of Regression Tests that Completed Successfully• Newly Opened Defects per Week• Newly Closed Defects per Week• Total Defects Open
01/10/2007CSPIN f v1.0 - © Pearl Street Group 2007 18
Example – Regression Test Progress
0%10%20%30%40%50%60%70%80%90%
100%
Planned % Executed % Success Rate %
Date: 07/08/05 07/15/05 07/22/05 07/29/05 08/05/05 08/12/05 08/19/05 08/26/05Total: 7536 7536 7536 7536 7536 7536 7536 7536
Planned: 563 1611 2610 3727 4905 6199 7536 7536Planned % 7% 21% 34% 49% 65% 82% 100% 100%
Executed % 10% 30% 50% 63% 71%Success Rate % 99.2% 99.0% 99.3% 99.3% 99.3%
01/10/2007CSPIN f v1.0 - © Pearl Street Group 2007 19
Example - Regression Test Completed
Regression and residual System Test Defects Run Rate
0
5
10
15
20
25
30
35
40
45
7/8 7/15 7/22 7/29 8/5 8/12 8/19 8/26
No
of D
efec
ts
New Closed Total Open Forecasted Total
Defect RateWeek Ending 7/8 7/15 7/22 7/29 8/5 8/12 8/19 8/26New 6 14 11 12 5Closed 0 10 13 3 9Total Open 27 31 29 38 34Forecasted Total 30 35 40 42 40 35 27 19
Complete when all Regression Tests have been executed, No. of Open Minor Defects < 20 and No
Critical or Major Defects
01/10/2007CSPIN f v1.0 - © Pearl Street Group 2007 20
Dashboards Provide a holistic graphical view of multiple related
measures displayed to enable stakeholders to see measurement interdependencies and take appropriate action
May have the same measure occur on different dashboards Different Dashboards for different stakeholders
– Directors of Development
– Test Managers
– Project Managers
Project Management Dashboard
Project
Cost
Work Completed
# of Defects
Director Management Dashboard
Project
Cost
Work Completed
# of Defects
% TestCompleted
Test Management Dashboard
Project
Cost
% Test Completed
# of Defects
01/10/2007CSPIN f v1.0 - © Pearl Street Group 2007 21
Example of What a Dashboard Can Show
Project starts out understaffed Effort & Overtime - Staff
200
250
300
350
400
450
T1 T2 T3 T4 T5 T6
Ho
urs
Plan Actual Actual @ 40 Hrs
Simulated Data
Earned Value - Cost
$0
$500
$1,000
$1,500
$2,000
T1 T2 T3 T4 T5 T6
Cumulative Budgeted Effort of Work PerformedCumulative Actual Effort of Work Performed
Simulated Data
IT Earned Value - Schedule
0
50
100
150
200
250
300
T1 T2 T3 T4 T5 T6
Val
ue
Ad
ded
Planned Tasks Completed Tasks
Simulated Data
PM is catching up via overtime and hiring additional resources Project is getting back to
plan But is starting to run over budget because of the overtime
01/10/2007CSPIN f v1.0 - © Pearl Street Group 2007 22
The Balanced Score Card and Process Improvement
IT Measurement Programs should link to multiple organizational goals such as: Customer and Process Improvement views as well as the Financial and Operational reporting needs of IT Management
How do the business owners
see us?
Financial View
What must we excel at?
OperationalView
How do customers see
us?
CustomerView
How do we improve, add value & stay
competitive?
Process Improvement View
Organization
The Balanced Score Card provides 4 key views and is an excellent technique for designing a Measurement Program that links to organizational goals.
The Balanced Score Card provides 4 key views and is an excellent technique for designing a Measurement Program that links to organizational goals.
Source: Kaplan, R. & Norton, D., ”The Balanced Scorecard - Measures that Drive Performance”, HBR, January-February, 1992.
Source: Kaplan, R. & Norton, D., ”The Balanced Scorecard - Measures that Drive Performance”, HBR, January-February, 1992.
01/10/2007CSPIN f v1.0 - © Pearl Street Group 2007 23
Sample Balanced Score Card for Project Manager
Financial: – Effort & Overtime– Earned Value - Effort
Customer:– Defect Rate– Earned Value – Schedule– Defect Priority
Operational:– Defects found by hours– Defects by cause– Turnover
Improvement:– Build Defects
Effort & Overtime
200
250
300
350
400
T1 T2 T3 T4 T5 T6
Ho
urs
Plan Actual Actual @ 35 Hrs
Simulated Data
Earned Value - Effort
0
500
1,000
1,500
2,000
T1 T2 T3 T4 T5 T6Budgeted Effort of Work PeformedActual Effort of Work Performed
Simulated Data
Change/Defect Rate - In Project
0
5
10
15
20
T1 T2 T3 T4 T5 T6
Ch
ang
es
New Closed Open
Simulated Data
Earned Value - Schedule
0
50
100
150
200
T1 T2 T3 T4 T5 T6
Task
s C
om
ple
ted
Planned Tasks Completed Tasks
Simulated Data
Defect Priority - In Project
0
2
4
6
8
10
12
T1New
T1Closed
T2 New
T2 Closed
T3New
T3Closed
Critical High Low
Simulated Data
Defects Found/Appraisal Hr Cumulative
0.00
1.00
2.00
3.00
4.00
5.00
T1 T2 T3 T4 T5 T6Actual Plan
Simulated Data
Changes/Defects by Cause
0
1
2
3
4
5
6
7
Use
Req
Dsg Mig
Tran
Cod
e
Inst
all
Doc
Trai
n
Per
f
Impl
T1 T2 T3
Simulated Data
Build Defects
0
2
4
6
8
10
12
14
T1 T2 T3 T4 T5 T6
No
. of
Def
ects
Build Process Release Mgmt Other Total
Simulated Data
01/10/2007CSPIN f v1.0 - © Pearl Street Group 2007 24
Transactional System 1 Time
Reporting
How do the business owners
see us?
Financial View
What must we excel at?
OperationalView
How do customers see
us?
CustomerView
How do we improve, add value & stay
competitive?
Process Improvement View
Organization
How do the business owners
see us?
Financial View
How do the business owners
see us?
Financial View
What must we excel at?
OperationalView
What must we excel at?
OperationalView
How do customers see
us?
CustomerView
How do customers see
us?
CustomerView
How do we improve, add value & stay
competitive?
Process Improvement View
How do we improve, add value & stay
competitive?
Process Improvement View
Organization
Transactional System 3
Project Mgmt
Metrics Program Architecture
Transactional System 2
Defect Tracking
Data Repository
(Data Warehouse
or Data Mart)
Data Model
Data Dictionary
Metrics Catalog
Test Management Dashboard
ProjectCost
% Test Completed
# of Defects
Test Management Dashboard
ProjectCost
% Test Completed
# of Defects
Project Management Dashboard
ProjectCost
Work Completed
# of Defects
Project Management Dashboard
ProjectCost
Work Completed
# of Defects
Director Management Dashboard
ProjectCost
Work Completed
# of Defects
% TestCompleted
Director Management Dashboard
ProjectCost
Work Completed
# of Defects
% TestCompleted
Company Balanced Score
Card
DepartmentalDashboards
Company Strategy & Goals
01/10/2007CSPIN f v1.0 - © Pearl Street Group 2007 25
Summary
Design your Software Measurement Program to collect measures that can be used for Process Improvement
The result will be to increase the effectiveness of the measurement program itself and increase the effectiveness of software process improvement initiatives.
The result will be to increase the effectiveness of the measurement program itself and increase the effectiveness of software process improvement initiatives.
Analysis &Decision Making
Processes
MeasurementStorage Process
MeasurementCollectionProcesses
DashboardProduction Process
And Reporting
Get Value!
MakeDecisions
OriginalProcesses &Organization
RefineProcesses, Activities And/or Organizations
Analysis &Decision Making
Processes
MeasurementStorage Process
MeasurementCollectionProcesses
DashboardProduction Process
And Reporting
Get Value!
MakeDecisions
MakeDecisions
OriginalProcesses &Organization
RefineProcesses, Activities And/or Organizations
01/10/2007CSPIN f v1.0 - © Pearl Street Group 2007 26
Questions and Discussion
Larry Dribin, Ph.D.
847-807-7390
Top Related