Alain Abran - iceaaonline.com · HOW GOOD AN ESTIMATION PROCESS? ICEAA International Training Week...

HOW GOOD AN ESTIMATION PROCESS?

ICEAA International Training Week October 17-20, 2016, Bristol (UK)

Alain Abran Ecole de technologie supérieure – University of Québec (Canada)

1

Presented at the 2016 International Training Symposium: www.iceaaonline.com/bristol2016

Alain Abran

20 years 20 years

+ 35 PhD

Development

Maintenance

Process Improvement

ISO: 19761,

9216, 25000,

15939, 14143,

19759

© Copyrights Abran 2016


Agenda

1. Estimation: Craft or Engineering?

2. Estimation Phases – Roles & Responsibilities

3. Conditions for credible estimation models

4. COSMIC Estimation models with data from industry

5. Conclusions

3



Mathematical Models

5

Built with completed projects with almost no uncertainty in the inputs!


Inputs at Estimation Time…

6 Imprecise Inputs at Feasibility Analysis – Much Greater Error Range


Software Estimation Tools: Availability & Costs

11



Software Estimation Tools

12


Quality?


Consumers & Quality!

13



Estimation Tools & Quality?

14



Or?

(Software) Estimation



Agenda


2. Estimation Phases – Roles & Responsibilities

3. Conditions for credible estimation models


5. Conclusions

16



Mathematical Models

19

Built with completed projects with almost no uncertainty in the inputs!


Inputs at Estimation Time…

20 Imprecise Inputs at Feasibility Analysis – Much Greater Error Range


25

Estimator role: Provide information

about uncertainty range



Manager role:

Pick a number & Manage Risk



Agenda


2. Estimation phases & Roles & responsibilites

3. Conditions for credible estimation models: A. Measurements of the Inputs

B. Distribution of Data

C. Quality of the Models


5. Conclusions

27



Issues with software metrics

1st generation software size metrics Systematic errors: step function with min & max Invalid maths! No measurement unit Designed for unaccountability! Examples: - Usecase Points - Function Points - Story Points

29



Usecase Points & Similar Points-based Metrics

30

Actors Dev. team

Usecases

Specs

Programming

language



Usecase Points & similar Points-based metrics

31

Table 1: Entities, Attributes, and Measurement Rules

Entity Attribute Measurement rule Actor Complexity (of

actor)

The type of complexity (simple, average, or complex) of the

interaction between the actor and the system

Use case Complexity (of use

case)

The type of complexity (simple, average, or complex)

measured in the number of transactions

Specification of

requirements

Relevance of the

technical quality

requirements

The level of relevance (from 0 to 5) of each of the 13 known

non-functional qualities

Stability of the

requirements

The level of stability (from 0 to 5) of the functional and non-

functional requirements

Development

team

Familiarity with

the methodology

The level (from 0 to 5) of skills and knowledge of the

development methodology in use for the project.

Part-time status The level (from 0 to 5) of part-time staff on the team

Analysis capability The level (from 0 to 5) of analysis capabilities of the

development team with respect to project needs

Application

experience

The level (from 0 to 5) of team experience with the

application domain of the system

Object-oriented

experience

The level (from 0 to 5) of team experience with object-

oriented design

Motivation The level (from 0 to 5) of team motivation

Programming

language

Difficulty The level (from 0 to 5) of programming difficulty




32

x

x

x

x

Apples Chairs

Cars

Books

Houses

= ……?




33

x

x

x

x

Apples Chairs

Cars

Books

Houses

= ……?

Primary School = Fail ! © Copyrights Abran 2016


1st Generation of Function Points: Step Functions!

© Copyrights 2016: COSMIC and authors

Function Points (FP)

3 FP

4 FP

6 FP

3-step size range for the IFPUG External Input Transactions

Key limitations:

- Only 3 values

- Limited ranges (min,max)

- No single measurement unit of 1 FP!

ICEAA Bristol (UK), Oct. 2016

34


2nd Generation with COSMIC


COSMIC

Function

Points

(CFP)

No abitrary max

A single CFP exists

& is well defined1

2

43

65

87

109

11


35


1st and 2nd Generations of FSM


Function Points (FP)

3 FP

4 FP

6 FP

COSMIC Function Points - CFP

1CFP

2

3 4

5 6

7

8 9

10

11 12

….

No abitrary max

A single CFP exists &

well defined


36


COSMIC sizes are measured on a true ratio scale

There is no upper limit to the size of a functional process

Largest observed functional processes?

In avionics >100 CFP

The size of the smallest change to an existing functional process is 1 CFP

Open, freely available (via www.cosmic-sizing.org )

© Copyrights 2016: COSMIC and authors ICEAA Bristol (UK), Oct. 2016

37


http://www.cosmic-sizing.org/



1st & 2nd generation of Function Points Methods

1980 1985 1990 1995 2000

COSMIC FFP

v. 2.0

Allan

Albrecht

FPA IFPUG 4.0

IFPUG 4.1

MkII FPA

MkII FPA

v.1.3

Full FP’s v.1

3-D FP’s

Feature

Points

ISO ‘FSM’

Standard

14143

COSMIC v.

4.0.1

2016

IFPUG 4.3

1st generation

2nd generation


38


A sound Measurement Foundation

2nd Generation Functional Size: No upper size limit

No ‘weigths’

No unsound mathematical operations

Based on a concept common to all types of software of any ‘size’: a data movement

A measurement unit: 1 data movement of a single data group

A measurement symbol: 1 CFP

COSMIC – ISO 19761 (Common Software Measurement International Consortium)

39



Agenda


2. The estimation phases - Roles & responsibilites





5. Conclusions

40



Another Key Issue:

The quality of the inputs to the

estimation models


Graph of the dataset

© Copyright 2016 Alain Abran

42


Outliers: Values of the data that are significantly far from the average of the population of the dataset.

Candidate outliers:

Typically at least 1 or 2 orders of magnitude larger than a data point closer to it.

Can be identified from a graphical representation.

The Grubbs test, also referred to as the ESD method (Extreme Studentized Deviate).

The studentized values measure how many standard deviations each value is from the sample mean.

When the P-value for the Grubbs test is less than 0.05, that value is a significant outlier at the 5.0% significance level.

Values with a modified Z score greater than 3.5 in absolute value may well be outliers.

http://www.graphpad.com/quickcalcs/Grubbs1.cfm

Copyright © 2015 Alain

Abran

45

Identification of outliers












Frequency distribution

Copyright © 2016 Alain Abran

46

Size (Independent variable );

N=212

Effort (Dependent variable ); N=21


Example of wrong decisions dues to outliers

47

A New Software Metric to Complement the

IFPUG Function Points:

The Software Non-functional

Assessment Process: the SNAP Function Points


48

Author Very strong relationship of SNAP with Effort

R2 = 0 ,89 (R2 max = 1,0)


Author’s assertion on Figure 4:

R2 = .89

Spearman test for rank correlation of .85, with an associated confidence of statistical significance of greater than 99% (p-value <.0001).

49


50


51

Invalidity Range


52

What it really looks like

for the range for which there is

enough data points

Approxmimatively:

An R2 = 0.3

Not R2 = 0.89 (R2 max = 1,0)


53

What it really looks like for the

range for which there is enough

data points

Approxmimatively:

An R2 = 0.3

Not R2 = 0.89 (R2 max = 1,0)

Conclusion:

the claim that SNAP is useful for effort estimation is invalid


Used when variables are not normally distributed

Weak support for linear regression mathematical transformations

The log transform is often used to obtain normal distributions for either the size or the effort variable, or both.

Copyright © 2015 Alain

Abran

54

Log transformations


Scatter plot (n=312)

© Copyrights 2016 Alain Abran

55


Agenda


2. The estimation phases - Roles & responsibilites





5. Conclusions

56



Dataset with sparsely populated size intervals

Copyrights © 2016 Alain Abran

57



Copyrights © 2016 Alain Abran

58

Sparsely populated interval




59


Representativeness & Population sample


60

= Outside

of range!


Agenda


2. Estimation phases & Roles & responsibilites





5. Conclusions

61



COSMIC data from Industry

Practical experimentations with the

COSMIC method in Automotive

embedded software field

By: Sophie Stern

Renault

63



Renault – 2012

64

© Copyrights Renault 2016


Renault – 2012

65



Renault – 2012 66



Renault: Estimation & Negociations

67



Renault COSMIC context & usage

Automated measurements Matlab Simulink 99% accuracy

Estimation of CPU memory space based on COSMIC function points Planning of Requirements Specifications workload based on COSMIC functions points

68



Industry data with COSMIC

Productivity Analysis & benchmarking of projects from:

Financial governmental organization (Canada)

69



Which estimation model to use in which contexts?




International benchmarking with ISBSG


Organization A: Effort = 31hrs/CFP x CFP + 2,411 hours

ISBSG benchmark: Effort = 10hrs/CFP x CFP + 2,138 hours



Agenda


2. The phases in estimation

3. Economics concepts for estimation models

4. Estimation models with COSMIC: data from industry

5. Conclusions 1. Uncertainties – range of candidate estimates

2. Conclusions

78



The inputs to productivity models have little uncertainty = Known Facts

79



Models Built with completed projects


Imprecise Inputs at Feasibility Analysis – Much Greater Error Range

81


Recent Guidelines for Practitioners

A Guideline describing a

range of Approximate

Sizing methods Size/Cost estimates are usually

needed before the FUR have

been defined in detail

A Guideline on ‘Assuring the

accuracy of COSMIC

measurements’

The COSMIC Functional Size Measurement Method

Version 4.0.1

GGuuiiddeelliinnee ffoorr EEaarrllyy oorr RRaappiidd

CCOOSSMMIICC FFuunnccttiioonnaall SSiizzee

MMeeaassuurreemmeenntt bbyy uussiinngg aapppprrooxxiimmaattiioonn aapppprrooaacchheess

July 2015


Version 3.0.1

GGuuiiddeelliinnee ffoorr aassssuurriinngg tthhee aaccccuurraaccyy ooff

mmeeaassuurreemmeennttss

VERSION 0.93

February 2011


82


Imprecise Inputs at Feasibility Analysis – Much Greater Error Range

83


Version 3.0.1

GGuuiiddeelliinnee ffoorr aassssuurriinngg tthhee aaccccuurraaccyy ooff

mmeeaassuurreemmeennttss

VERSION 0.93

February 2011


Version 4.0.1

GGuuiiddeelliinnee ffoorr EEaarrllyy oorr RRaappiidd

CCOOSSMMIICC FFuunnccttiioonnaall SSiizzee

MMeeaassuurreemmeenntt bbyy uussiinngg aapppprrooxxiimmaattiioonn aapppprrooaacchheess

July 2015


85


86


87


88


The ‘Feel Good’

ProjectCodeMeter assertions: http://www.projectcodemeter.com/cost_estimation/kop3.html

‘A professional software tool to measure & estimate Time, Cost, Complexity, Quality & Maintainability of software projects, Development Team Productivity by analyzing their source code’.

‘Using a modern software sizing algorithm called Weighted Micro Function Points (WMFP) a successor to solid ancestor scientific methods as COCOMO, COSYSMO, Maintainability Index, Cyclomatic Complexity, & Halstead Complexity’.

‘More accurate results than traditional software sizing tools, while being faster & simpler to configure’.

93



Or?

(Software) Estimation



Building ‘good’ estimation process & good estimation models

Sound Measurement Units

Sound Maths!

Recognition of uncertainties:

how to recognize this & how to deal with it

The estimator has to provide information, not a single estimate

The manager has to select a single budget number

& manage risks through contingency planning.

Discipline, rigor, commitments & $$$

95



96

Want to know about good & bad practices in software estimation?



97

[email protected]



Alain Abran - iceaaonline.com · HOW GOOD AN ESTIMATION PROCESS? ICEAA International Training Week...

Documents

Transcript of Alain Abran - iceaaonline.com · HOW GOOD AN ESTIMATION PROCESS? ICEAA International Training Week...