Practical Uses of Software Measurement for Process Improvement January 10, 2007 - V1.0 Larry Dribin,...

Practical Uses of Software Measurement for Process Improvement

January 10, 2007 - V1.0

Larry Dribin, Ph.D.

[email protected]

847-807-7390

mailto:[email protected]

01/10/2007CSPIN f v1.0 - © Pearl Street Group 2007 2

Agenda

Software Measurement Programs IT Measurement for Many Companies Today Software Process Improvement Programs Tailoring a IT Measurement Program to Support Process

Improvement

The goal of this presentation is to discuss how software measurement programs can be designed to better support process improvement.

The goal of this presentation is to discuss how software measurement programs can be designed to better support process improvement.

Lord Kelvin Comment:‘When you can measure what you are speaking about and express it in numbers, you know something about it; but when you cannot measure, when you cannot express it in numbers, your knowledge is of a meager and unsatisfactory kind…”

Lord Kelvin, Popular Lectures and Addresses, 1889

Lord Kelvin Comment:‘When you can measure what you are speaking about and express it in numbers, you know something about it; but when you cannot measure, when you cannot express it in numbers, your knowledge is of a meager and unsatisfactory kind…”

Lord Kelvin, Popular Lectures and Addresses, 1889


The Measurement Cycle

Why Measure:

To Manage– Quality– Cycle Time– Productivity– Risk

To Communicate– Management– Peers

To Improve Performance(Process Improvement)

“What gets measured gets done!”“What gets measured gets done!”

Measures

Decisions

Facts

Actions

Results


Software Measurement Programs

Analysis &Decision Making

Processes

MeasurementStorage Process

MeasurementCollectionProcesses

DashboardProduction Process

And Reporting

Get Value!

Make

Decisions

OriginalProcesses &Organization

RefineProcesses, Activities And/or Organizations


IT Measurement for Many Companies Today

Gather and report on numerous measures (25 to over 100) Typically focus is on financial and operational performance

measures

Example Financial Measures:– IT Budget – Plan vs. Actual– Hours spent on discretionary development vs. “maintenance”

Example Operational Measures– Number of calls to the Help Desk– Number of calls answered by 1st level support


Example Financial Measure

IT Budget vs Actual Expense - Total IT

AnyCo Common Costs (I.e help desk), which accrued for the entire year, were allocated to September resulting in the large variance.

$-

$500.0

$1,000.0

$1,500.0

$2,000.0

$2,500.0

Jan Feb Mar Apr May Jun Jul Aug Sep Oct

Thousands

Actual Plan


Example Operational Measure

Trouble Ticket Run Rate and Escalation to Level 2

Customer Care Center Service Objective for Problems Escalated to Level 2 is <60%

2174

2699 2505

2963 2863

3261

2839

1283 1538

1247 1378 1519 1386 1467

1338

2268

57% 55%

51% 48%

45% 47%

59% 55%

0

500

1000

1500

2000

2500

3000

3500

Mar Apr May Jun Jul Aug Sep Oct 0%

10%

20%

30%

40%

50%

60%

Tkts Escal to Lev 2 % Escal to Lev 2


Software Process Improvement Programs

There are many types of process improvement initiatives that IT Organizations are pursuing today:– CMMI– ITIL– CoBiT / SOX– TQM– QA Improvement– Testing Improvement– Agile Development

All of these initiatives require process measurement to be effective

Strive for “Best Practices”, but require “GSP – Good Software Practices”.

Similar to FDA requirement that Pharmaceutical companies follow “GMP –Good Manufacturing Practices”.

Strive for “Best Practices”, but require “GSP – Good Software Practices”.

Similar to FDA requirement that Pharmaceutical companies follow “GMP –Good Manufacturing Practices”.


Process Improvement Measures

OriginalProcesses

OriginalProcesses

Process Progress Measures measure how well the process is being

implemented:

ValueTomorrow

ValueToday

RestructuredProcesses

RestructuredProcesses

Outcome Measures measure the result of

the improvement:(Financial, Operational,

&/or Customer)

Process improvement measures consist of both Outcome and Progress Metrics

Process improvement measures consist of both Outcome and Progress Metrics


IT Measurement and Software Process Improvement

The IT financial and operational measures described above are typically designed to assist IT management explain what IT has been doing for the business

This is important, because if business managers do not feel that they are getting value for their IT dollars, they:– Reduce IT Budgets – Reduce IT Headcount– Outsource IT

As a result of this focus, many IT Measurement Programs are not designed to support process improvement.

As a result of this focus, many IT Measurement Programs are not designed to support process improvement.


CMMI Measurement Focus Measurement & Analysis is a Key Process Area for the CMMI Focus on is developing Measurement Programs:

Collecting, storing, reporting and analyzing measurement results Measurement is embedded in all Key Process Areas (KPA) CMM Measurement Requirements for Requirement

Management KPAMeasurements are made and used to determine the status of the activities for managing requirements.

Examples of these measures include:– Status of each requirement– Change activity for each requirement– Cumulative number of changes to the requirements, including

total number of changes proposed, open, approved and incorporated into the system baseline.

These practices provide high level guidance when developing specific measures for process improvement

These practices provide high level guidance when developing specific measures for process improvement


Designing Measurement Programs

Follow a similar approach to application development:– Gather Measurement Requirements– Analyze and Prioritize Measurement Requirements– Design a Measurement Solution– Develop the Measurement Solution

• What to collect• How to collect• Where to store• When and How-Too Present and Report Measurements Results

– Deploy Measurement Solution– Harvest the Benefits

Useful techniques:– Goal->Question->Metric to identify key metrics– Dashboards to organize the metrics for specific audiences– Balanced Score Cards to develop balanced dashboards linked to

organizational goals


Goal->Question->Metric Technique

Well known technique developed by Victor Basili to identify the measures needed for process improvement

1. Define Goals using the Balanced Score Card or other technique Example Goal: Let’s improve programmer productivity.

2. Ask Questions about goals to define the process improvement initiative in more detail

Questions: How much does an average developer develop?

3. Questions lead to defining specific measures such as: Number of developers working in a month Time each developer spends working in the month Total number of Lines of Code (LOC) (or Function Points – FP)

Produced by each developer per month Productivity = (#LOC or FP) / (# Hours Worked)

-> Average Lines Of Code (LOC) or Function Points (FP) produced per developer per month


Example – Requirements Management Effectiveness

Goal:– Improve Requirements Management

Questions:– How many Requirement do we have?– Are these the right requirements for our problem?– Has each Requirement been approved by all key stakeholders?– Has the Requirement changed?

Potential Metrics:– Process Progress Measure:

• Percent of Project Teams following the new/improved process– Outcome Measures:

• Number of Requirements • Status of each Requirement• Number of changes to the requirements• Number of Requirements Defects found during Requirements

Validation• Number of Requirements Defects found during Coding• Number of Requirements Defects found during Testing


Example – Production Defects Goal:

– Reduce Application Production Defects Questions:

– What is a Defect?– What is an Application?– What Application did the Defect occur in?

Potential Measures– Process Progress Measures:

• The percent of Project Teams tracking Defects using standard defect classifications

– Outcome Measures: • Production Defects – defined as defects logged to the Help

Desk by Application by Defect Type using standard defect classifications


Detailed Pareto of Trouble Tickets within Category

Trouble Ticket by Category - PC Software

Data for October 20xx

180 164

100

70 66 65

57 55 48

37 29

25 22 16%

31% 41%

47% 53%

59% 64%

69% 74% 77% 80% 82% 84%

0 20 40 60 80

100 120 140 160 180

BSCS

EXCHANGE

OTHER

.INSTALL NEW SOF

TWAR

WINDOWS 95

SAP MS-WORD

DIVERCTI

INTERNET EXPLORER

MS-EXCEL

FAX SR. NT CAR

S REMEDY


Example – Testing Metrics Goal:

– Measure Regression Test progress and when completed Questions:

– What is Regression Test Progress?– How do we know that we are done?

Potential Measures– Process Progress Measures:

• The percent of Project Teams tracking Regression Test Progress– Outcome Measures:

• Number of Regression Tests Planned• Number of Regression Tests Completed• Percent of Regression Tests that Completed Successfully• Newly Opened Defects per Week• Newly Closed Defects per Week• Total Defects Open


Example – Regression Test Progress

0%10%20%30%40%50%60%70%80%90%

100%

Planned % Executed % Success Rate %

Date: 07/08/05 07/15/05 07/22/05 07/29/05 08/05/05 08/12/05 08/19/05 08/26/05Total: 7536 7536 7536 7536 7536 7536 7536 7536

Planned: 563 1611 2610 3727 4905 6199 7536 7536Planned % 7% 21% 34% 49% 65% 82% 100% 100%

Executed % 10% 30% 50% 63% 71%Success Rate % 99.2% 99.0% 99.3% 99.3% 99.3%


Example - Regression Test Completed

Regression and residual System Test Defects Run Rate

0

5

10

15

20

25

30

35

40

45

7/8 7/15 7/22 7/29 8/5 8/12 8/19 8/26

No

of D

efec

ts

New Closed Total Open Forecasted Total

Defect RateWeek Ending 7/8 7/15 7/22 7/29 8/5 8/12 8/19 8/26New 6 14 11 12 5Closed 0 10 13 3 9Total Open 27 31 29 38 34Forecasted Total 30 35 40 42 40 35 27 19

Complete when all Regression Tests have been executed, No. of Open Minor Defects < 20 and No

Critical or Major Defects


Dashboards Provide a holistic graphical view of multiple related

measures displayed to enable stakeholders to see measurement interdependencies and take appropriate action

May have the same measure occur on different dashboards Different Dashboards for different stakeholders

– Directors of Development

– Test Managers

– Project Managers

Project Management Dashboard

Project

Cost

Work Completed

# of Defects

Director Management Dashboard

Project

Cost

Work Completed

# of Defects

% TestCompleted

Test Management Dashboard

Project

Cost

% Test Completed

# of Defects


Example of What a Dashboard Can Show

Project starts out understaffed Effort & Overtime - Staff

200

250

300

350

400

450

T1 T2 T3 T4 T5 T6

Ho

urs

Plan Actual Actual @ 40 Hrs

Simulated Data

Earned Value - Cost

$0

$500

$1,000

$1,500

$2,000

T1 T2 T3 T4 T5 T6

Cumulative Budgeted Effort of Work PerformedCumulative Actual Effort of Work Performed

Simulated Data

IT Earned Value - Schedule

0

50

100

150

200

250

300

T1 T2 T3 T4 T5 T6

Val

ue

Ad

ded

Planned Tasks Completed Tasks

Simulated Data

PM is catching up via overtime and hiring additional resources Project is getting back to

plan But is starting to run over budget because of the overtime


The Balanced Score Card and Process Improvement

IT Measurement Programs should link to multiple organizational goals such as: Customer and Process Improvement views as well as the Financial and Operational reporting needs of IT Management

How do the business owners

see us?

Financial View

What must we excel at?

OperationalView

How do customers see

us?

CustomerView

How do we improve, add value & stay

competitive?

Process Improvement View

Organization

The Balanced Score Card provides 4 key views and is an excellent technique for designing a Measurement Program that links to organizational goals.

The Balanced Score Card provides 4 key views and is an excellent technique for designing a Measurement Program that links to organizational goals.

Source: Kaplan, R. & Norton, D., ”The Balanced Scorecard - Measures that Drive Performance”, HBR, January-February, 1992.

Source: Kaplan, R. & Norton, D., ”The Balanced Scorecard - Measures that Drive Performance”, HBR, January-February, 1992.


Sample Balanced Score Card for Project Manager

Financial: – Effort & Overtime– Earned Value - Effort

Customer:– Defect Rate– Earned Value – Schedule– Defect Priority

Operational:– Defects found by hours– Defects by cause– Turnover

Improvement:– Build Defects

Effort & Overtime

200

250

300

350

400

T1 T2 T3 T4 T5 T6

Ho

urs

Plan Actual Actual @ 35 Hrs

Simulated Data

Earned Value - Effort

0

500

1,000

1,500

2,000

T1 T2 T3 T4 T5 T6Budgeted Effort of Work PeformedActual Effort of Work Performed

Simulated Data

Change/Defect Rate - In Project

0

5

10

15

20

T1 T2 T3 T4 T5 T6

Ch

ang

es

New Closed Open

Simulated Data

Earned Value - Schedule

0

50

100

150

200

T1 T2 T3 T4 T5 T6

Task

s C

om

ple

ted

Planned Tasks Completed Tasks

Simulated Data

Defect Priority - In Project

0

2

4

6

8

10

12

T1New

T1Closed

T2 New

T2 Closed

T3New

T3Closed

Critical High Low

Simulated Data

Defects Found/Appraisal Hr Cumulative

0.00

1.00

2.00

3.00

4.00

5.00

T1 T2 T3 T4 T5 T6Actual Plan

Simulated Data

Changes/Defects by Cause

0

1

2

3

4

5

6

7

Use

Req

Dsg Mig

Tran

Cod

e

Inst

all

Doc

Trai

n

Per

f

Impl

T1 T2 T3

Simulated Data

Build Defects

0

2

4

6

8

10

12

14

T1 T2 T3 T4 T5 T6

No

. of

Def

ects

Build Process Release Mgmt Other Total

Simulated Data


Transactional System 1 Time

Reporting


see us?

Financial View


OperationalView


us?

CustomerView


competitive?


Organization


see us?

Financial View


see us?

Financial View


OperationalView


OperationalView


us?

CustomerView


us?

CustomerView


competitive?



competitive?


Organization

Transactional System 3

Project Mgmt

Metrics Program Architecture

Transactional System 2

Defect Tracking

Data Repository

(Data Warehouse

or Data Mart)

Data Model

Data Dictionary

Metrics Catalog


ProjectCost

% Test Completed

# of Defects


ProjectCost

% Test Completed

# of Defects


ProjectCost

Work Completed

# of Defects


ProjectCost

Work Completed

# of Defects


ProjectCost

Work Completed

# of Defects

% TestCompleted


ProjectCost

Work Completed

# of Defects

% TestCompleted

Company Balanced Score

Card

DepartmentalDashboards

Company Strategy & Goals


Summary

Design your Software Measurement Program to collect measures that can be used for Process Improvement

The result will be to increase the effectiveness of the measurement program itself and increase the effectiveness of software process improvement initiatives.

The result will be to increase the effectiveness of the measurement program itself and increase the effectiveness of software process improvement initiatives.


Processes




And Reporting

Get Value!

MakeDecisions




Processes




And Reporting

Get Value!

MakeDecisions

MakeDecisions




Questions and Discussion

Larry Dribin, Ph.D.

[email protected]

847-807-7390

mailto:[email protected]

Practical Uses of Software Measurement for Process Improvement January 10, 2007 - V1.0 Larry Dribin,...

Documents

Transcript of Practical Uses of Software Measurement for Process Improvement January 10, 2007 - V1.0 Larry Dribin,...