Post on 29-Dec-2015
Jean-Marc Desharnais LMAGL / SELAM #
How to Implement a Measurement Program
19971998
19992000
2001
2002
Jean-Marc Desharnais 2
To improve decision-making
To monitor improvement
Allow the synergy of the people
Why is Measurement so Important?
Jean-Marc Desharnais 3
Two Elements of a Measurement Program
Static perspective - the outcome
A structured set of measures related to goals and quantifiable objectives
Dynamic perspective - how the outcome is achieved
Activities that must be planned, scheduled and controlled, and managed by experts with clearly defined responsibilities
Jean-Marc Desharnais 4
Application
The measurement program is also:– An application– In constant evolution– To be maintained by competent personnel
Jean-Marc Desharnais 5
How to Cope with Roadblocks
Measurement programs not related to objectives are useless and not worth the data collection and data storage expenses involved. A measurement program is a tool, and not itself an objective;
The measurement program should also take in consideration the organizational and individual dimensions, and discriminate between public and private data. One of the major obstacles to data collection is the confidential nature of the data and of their use (e.g. effort management);
Measurement programs must be nurtured in a favorable environment that permits timely, valid and complete measures. The measurement results must also be delivered in a timely manner, otherwise they are useless for decision-making; furthermore, an incomplete or invalid measure may lead to incorrect conclusions and decisions.
Jean-Marc Desharnais 6
Parts of a Measurement Program
Analysis sequence
Goals and Objectives
Standards Reports
Measures Key Ratios
Tools used to gather measurements
Resource Process Product
Maturity Level - Key Processes
Mod
el
Model
feedback feedback
1
2
3
4
5
7
6
3
Jean-Marc Desharnais 7
DefinedDefined
ManagedManaged
OptimizingOptimizing
Initial /Ad Hoc
Repeatable
1
2
3
4
5
ISO 9000
Our organization
Short term Target
Software Engineering Institute’s (S E I)Capability Maturity Model (C M M)
Jean-Marc Desharnais 8
Level characteristics Problems to work
Optimized Improving process Automation
ManagedQuantitative : measuredprocesses
DefinedQualitative : depends on individuals
Process measurement and analysisQuality management
RepeatableIntuitive : depends on individuals
Peer reviewsIntergroup coordinationSoftware product engineeringIntegrated software managementTraining programOrganization process definitionOrganization process focus
Initial Ad hoc: control and plans non effective
Process change managementTechnology innovationDefect prevention
Configuration managementQuality assuranceSubcontract managementProject tracking and oversightProject planningRequirements management
Levels
1 - Selection of ProcessesSelect Processes at Your Level of Maturity
Jean-Marc Desharnais 9
2 - Define Goals and Objectives (Example 1)
Key process: Application Maintenance
Goal: Improve the value delivered by application maintenance
Objective: The percent of total dollars spent on Application Corrections compared to percent of total dollars spent on Preventive /Perfective/ Adaptive Maintenance will be reduced by 75% by the end of 1998.
Jean-Marc Desharnais 10
2 - Define Goals and Objectives (Example 2)
Key process: Configuration management
Goal: Reduce failures due to configuration errors
Objective: The number of failures attributable to configuration management shall be less than 1 in 100 by the end of 1998.
Objectives must be specific in terms of measurement / key ratio to be achieved and the timescale for achievement.
Jean-Marc Desharnais 11
Use “standards” (define your own, if necessary)
• Define what to measure• Define how to measure• Define when to measure
Need a mechanism or procedure
3, 4, & 5 - Define Measures
Jean-Marc Desharnais 12
General term– what is a project ?– what is an application ?– what is maintenance ?– etc..
Do we have standard ? Do we want to use tools ?
– which one ?– why ?– when ?
Questions
Jean-Marc Desharnais 13
Measures (Examples)
Resources
ToolsMethodsExperience
Process
EffortDuration# of defectsMaintenance Category
Product
LOCFunction PointUnit Process
Jean-Marc Desharnais 14
6 & 7 - Design the Measurement Program
Key process, goals
Questions(how)
Objectives EntitiesAttributesMetrics
Tools
Definitions
DocumentationReferences
Datamodel
ProcessReports
Jean-Marc Desharnais 15
There are two ways to see the data validation:– a priori– a posteriori
Data Validation
Jean-Marc Desharnais 16
A Priori
Do we have a written procedure ?
Is it practical for the organization ?
Do we know the procedure ?
Do we have clear responsibilities ?
Can we follow and control the data ?
Jean-Marc Desharnais 17
A Posteriori
What are the validation criteria ?
Can we explain marginal cases ?
How do we know we have the correct data ?
Do we have a lot on non elucidated cases ?
Jean-Marc Desharnais 18
For IS Group it is related to:– the level of maturity– the capacity to set objectives– the acceptance of standards by all the group
For the measurement group, it is related to:– the capacity to manage the measurement program
as a project– the capacity to deliver the measurement program as
an application
Success Factors
Jean-Marc Desharnais 19
Measures Relating to Maintenance Objective
unit of measure
integer
integer
integer
integer
hours
hours
hours
FPC
FPC
frequency ofcollection
monthly
monthly
monthly
start of stage
stage end
stage end
stage end
stage end
stage end
recordedby
MT
MT
MT
MT
MT
MT
MT
FPCrepres.FPC
repres.
storedon
FRL
PMW
PMW
PMW
FRL
FRL
FPCdatabase
FPCdatabase
FRL
data item
number of Faults Reports raised
effort expended on corrective maintenance
effort expended on adaptive maintenance
elapsed time to clear Fault Reports
number of faults fixed
number of faults outstanding
size of system at start of maintenance stage
size of system at end of maintenance stage
number of Faults Reports that were not faults
Key: MT maintenance team FPC function point count
FRL fault report logPMW project management workbench
Jean-Marc Desharnais 20
Effort by Maintenance Category (Example)
Category
Adap.
Corr.
Perf.
Prev.
SupportSum
Hou
rs p
er 1
00 F
P
300
200
100
0
Hours per 100 FP
System X
Quarter
199601199504
199503199502
199501199404
199403199402
199401199304
199303199302
199301
Jean-Marc Desharnais 21
Implementation of a Measurement Program
– Management commitment build-up – Staff commitment build-up – Selection of the key processes to be
improved – Identification of the goals and
objectives related to the key processes
Define
– Design of the measurement program – Description of the Information System to
be put in place – Deployment of the measurement
program
Realize
Jean-Marc Desharnais 22
Step 1 - Management Commitment Build-up
This first step consists of:Finding the necessary information that will help management make a decision on the relevance of implementing a measurement program within the organization.
Jean-Marc Desharnais 23
Step 1 - Issues
Middle managers do not readily see the relevance of initiating measurement programs in software engineering since :
– They perceive them to be expensive and bureaucratic
– They mention significant delays prior obtaining expected results
– The impact of these measurements is limited– They often get contradictory advice from experts
Jean-Marc Desharnais 24
Step 2 - Staff Commitment Build-up
This step consists of:
Finding necessary arguments that will lead the staff involved in data collection process:– To accept the measurement program– To commit to support the measurement program
Jean-Marc Desharnais 25
Step 2 - Issues
It appears that there is almost always major staff reluctance to accept a measurement program.
– Main objections are related to: Measures relevance Validity of measure units Effort required to complete measures Effective use of measures
– Solutions are related to the: Choice of a pilot project Documentation Responsibilities Training Data availability
Jean-Marc Desharnais 26
Step 3 - Selection of the Key Processes to be Improved
This step consists in:– Evaluating the maturity level of the software
development organisation– Selecting the priorities to be given to the key
processes targeted for improvement programs.
Jean-Marc Desharnais 27
Step 3 - Issues
It is possible that management sees this step as:– Not significant– Too expensive– Not useful and/or too long
Solutions are related to the:– Stick on the maturity level (part 1 of a
measurement program)– Options confirmation or information
Jean-Marc Desharnais 28
The purpose of this step is:– To determine goals and objectives of the
measurement program – Related to the key processes
(eg: improve estimating for development project)
Step 4 - Identification of the Goals and Objectives
Jean-Marc Desharnais 29
Step 4 - Issues
The goals are sometimes:– Ambiguous– Redundant– In terms of action instead of intention– Not realistic
We must pay attention to:– The capability to achieve them– Their priority
GoalsGoals
Goals
Jean-Marc Desharnais 30
Step 5 - Design of the Measurement Program
This step consist of :– Designing a measurement program to
See if the objectives have been reached Understand why they have not been reached
Jean-Marc Desharnais 31
Step 5 - Issues
The issues are related to :– The organisations expectations about tools– The use of ill-defined concepts– The definition of objectives only related to the
available measures– Weak or non-existent documentation
Jean-Marc Desharnais 32
Roles and Responsibilities
Level of responsibilities to be considered:– Strategic level
Identify who are responsible to deliver or define objectives
– Tactical level Responsibilities related to the measurement support The choice of methods, definitions, tools and standards
– Operational level Who provides the database on the standards with or without
specific tools It is also at this level that we can find the database and
reports
Jean-Marc Desharnais 33
Type of Personnel
• Sponsor• Champion• Expert• Consultant• Delivery personnel
Jean-Marc Desharnais 34
Step 6 - Description of the Information System to be Put in Place
This step is related to :– The modeling of all the measures to be collected to
meet the objectives This data is based directly on the unit of measure
and indirectly on the standards
Jean-Marc Desharnais 35
Step 7 - Deployment of the Measurement Program
This step consists of deploying the MP through:– Selection of a pilot site– Personnel training– Assigning responsibilities and tasks– Setting-up the measurement group
Jean-Marc Desharnais 36
The main problems are :– Having to resolve to many problems in a short period of time– Training is not really addressed– Roles and responsibilities are not clear– The measurement group is in a situation of conflict– The delivery people are too busy– The measurement personnel is inexperienced
Possible solutions are related to :– The selection of a pilot site– The training– The roles and responsibilities– The choice of the personnel for the measurement group
Step 7 - Issues