Jean-Marc Desharnais LMAGL / SELAM # How to Implement a Measurement Program 1997 1998 1999 2000 2001...

Jean-Marc Desharnais LMAGL / SELAM #

How to Implement a Measurement Program

19971998

19992000

2001

2002

Jean-Marc Desharnais 2

To improve decision-making

To monitor improvement

Allow the synergy of the people

Why is Measurement so Important?


Two Elements of a Measurement Program

Static perspective - the outcome

A structured set of measures related to goals and quantifiable objectives

Dynamic perspective - how the outcome is achieved

Activities that must be planned, scheduled and controlled, and managed by experts with clearly defined responsibilities


Application

The measurement program is also:– An application– In constant evolution– To be maintained by competent personnel


How to Cope with Roadblocks

Measurement programs not related to objectives are useless and not worth the data collection and data storage expenses involved. A measurement program is a tool, and not itself an objective;

The measurement program should also take in consideration the organizational and individual dimensions, and discriminate between public and private data. One of the major obstacles to data collection is the confidential nature of the data and of their use (e.g. effort management);

Measurement programs must be nurtured in a favorable environment that permits timely, valid and complete measures. The measurement results must also be delivered in a timely manner, otherwise they are useless for decision-making; furthermore, an incomplete or invalid measure may lead to incorrect conclusions and decisions.


Parts of a Measurement Program

Analysis sequence

Goals and Objectives

Standards Reports

Measures Key Ratios

Tools used to gather measurements

Resource Process Product

Maturity Level - Key Processes

Mod

el

Model

feedback feedback

1

2

3

4

5

7

6

3


DefinedDefined

ManagedManaged

OptimizingOptimizing

Initial /Ad Hoc

Repeatable

1

2

3

4

5

ISO 9000

Our organization

Short term Target

Software Engineering Institute’s (S E I)Capability Maturity Model (C M M)


Level characteristics Problems to work

Optimized Improving process Automation

ManagedQuantitative : measuredprocesses

DefinedQualitative : depends on individuals

Process measurement and analysisQuality management

RepeatableIntuitive : depends on individuals

Peer reviewsIntergroup coordinationSoftware product engineeringIntegrated software managementTraining programOrganization process definitionOrganization process focus

Initial Ad hoc: control and plans non effective

Process change managementTechnology innovationDefect prevention

Configuration managementQuality assuranceSubcontract managementProject tracking and oversightProject planningRequirements management

Levels

1 - Selection of ProcessesSelect Processes at Your Level of Maturity


2 - Define Goals and Objectives (Example 1)

Key process: Application Maintenance

Goal: Improve the value delivered by application maintenance

Objective: The percent of total dollars spent on Application Corrections compared to percent of total dollars spent on Preventive /Perfective/ Adaptive Maintenance will be reduced by 75% by the end of 1998.


2 - Define Goals and Objectives (Example 2)

Key process: Configuration management

Goal: Reduce failures due to configuration errors

Objective: The number of failures attributable to configuration management shall be less than 1 in 100 by the end of 1998.

Objectives must be specific in terms of measurement / key ratio to be achieved and the timescale for achievement.


Use “standards” (define your own, if necessary)

• Define what to measure• Define how to measure• Define when to measure

Need a mechanism or procedure

3, 4, & 5 - Define Measures


General term– what is a project ?– what is an application ?– what is maintenance ?– etc..

Do we have standard ? Do we want to use tools ?

– which one ?– why ?– when ?

Questions


Measures (Examples)

Resources

ToolsMethodsExperience

Process

EffortDuration# of defectsMaintenance Category

Product

LOCFunction PointUnit Process


6 & 7 - Design the Measurement Program

Key process, goals

Questions(how)

Objectives EntitiesAttributesMetrics

Tools

Definitions

DocumentationReferences

Datamodel

ProcessReports


There are two ways to see the data validation:– a priori– a posteriori

Data Validation


A Priori

Do we have a written procedure ?

Is it practical for the organization ?

Do we know the procedure ?

Do we have clear responsibilities ?

Can we follow and control the data ?


A Posteriori

What are the validation criteria ?

Can we explain marginal cases ?

How do we know we have the correct data ?

Do we have a lot on non elucidated cases ?


For IS Group it is related to:– the level of maturity– the capacity to set objectives– the acceptance of standards by all the group

For the measurement group, it is related to:– the capacity to manage the measurement program

as a project– the capacity to deliver the measurement program as

an application

Success Factors


Measures Relating to Maintenance Objective

unit of measure

integer

integer

integer

integer

hours

hours

hours

FPC

FPC

frequency ofcollection

monthly

monthly

monthly

start of stage

stage end

stage end

stage end

stage end

stage end

recordedby

MT

MT

MT

MT

MT

MT

MT

FPCrepres.FPC

repres.

storedon

FRL

PMW

PMW

PMW

FRL

FRL

FPCdatabase

FPCdatabase

FRL

data item

number of Faults Reports raised

effort expended on corrective maintenance

effort expended on adaptive maintenance

elapsed time to clear Fault Reports

number of faults fixed

number of faults outstanding

size of system at start of maintenance stage

size of system at end of maintenance stage

number of Faults Reports that were not faults

Key: MT maintenance team FPC function point count

FRL fault report logPMW project management workbench


Effort by Maintenance Category (Example)

Category

Adap.

Corr.

Perf.

Prev.

SupportSum

Hou

rs p

er 1

00 F

P

300

200

100

0

Hours per 100 FP

System X

Quarter

199601199504

199503199502

199501199404

199403199402

199401199304

199303199302

199301


Implementation of a Measurement Program

– Management commitment build-up – Staff commitment build-up – Selection of the key processes to be

improved – Identification of the goals and

objectives related to the key processes

Define

– Design of the measurement program – Description of the Information System to

be put in place – Deployment of the measurement

program

Realize


Step 1 - Management Commitment Build-up

This first step consists of:Finding the necessary information that will help management make a decision on the relevance of implementing a measurement program within the organization.


Step 1 - Issues

Middle managers do not readily see the relevance of initiating measurement programs in software engineering since :

– They perceive them to be expensive and bureaucratic

– They mention significant delays prior obtaining expected results

– The impact of these measurements is limited– They often get contradictory advice from experts


Step 2 - Staff Commitment Build-up

This step consists of:

Finding necessary arguments that will lead the staff involved in data collection process:– To accept the measurement program– To commit to support the measurement program


Step 2 - Issues

It appears that there is almost always major staff reluctance to accept a measurement program.

– Main objections are related to: Measures relevance Validity of measure units Effort required to complete measures Effective use of measures

– Solutions are related to the: Choice of a pilot project Documentation Responsibilities Training Data availability


Step 3 - Selection of the Key Processes to be Improved

This step consists in:– Evaluating the maturity level of the software

development organisation– Selecting the priorities to be given to the key

processes targeted for improvement programs.


Step 3 - Issues

It is possible that management sees this step as:– Not significant– Too expensive– Not useful and/or too long

Solutions are related to the:– Stick on the maturity level (part 1 of a

measurement program)– Options confirmation or information


The purpose of this step is:– To determine goals and objectives of the

measurement program – Related to the key processes

(eg: improve estimating for development project)

Step 4 - Identification of the Goals and Objectives


Step 4 - Issues

The goals are sometimes:– Ambiguous– Redundant– In terms of action instead of intention– Not realistic

We must pay attention to:– The capability to achieve them– Their priority

GoalsGoals

Goals


Step 5 - Design of the Measurement Program

This step consist of :– Designing a measurement program to

See if the objectives have been reached Understand why they have not been reached


Step 5 - Issues

The issues are related to :– The organisations expectations about tools– The use of ill-defined concepts– The definition of objectives only related to the

available measures– Weak or non-existent documentation


Roles and Responsibilities

Level of responsibilities to be considered:– Strategic level

Identify who are responsible to deliver or define objectives

– Tactical level Responsibilities related to the measurement support The choice of methods, definitions, tools and standards

– Operational level Who provides the database on the standards with or without

specific tools It is also at this level that we can find the database and

reports


Type of Personnel

• Sponsor• Champion• Expert• Consultant• Delivery personnel


Step 6 - Description of the Information System to be Put in Place

This step is related to :– The modeling of all the measures to be collected to

meet the objectives This data is based directly on the unit of measure

and indirectly on the standards


Step 7 - Deployment of the Measurement Program

This step consists of deploying the MP through:– Selection of a pilot site– Personnel training– Assigning responsibilities and tasks– Setting-up the measurement group


The main problems are :– Having to resolve to many problems in a short period of time– Training is not really addressed– Roles and responsibilities are not clear– The measurement group is in a situation of conflict– The delivery people are too busy– The measurement personnel is inexperienced

Possible solutions are related to :– The selection of a pilot site– The training– The roles and responsibilities– The choice of the personnel for the measurement group

Step 7 - Issues

Jean-Marc Desharnais LMAGL / SELAM # How to Implement a Measurement Program 1997 1998 1999 2000 2001...

Documents

Transcript of Jean-Marc Desharnais LMAGL / SELAM # How to Implement a Measurement Program 1997 1998 1999 2000 2001...