Iwsm2014 measurement process - improving the iso 15939 standard (luigi buglione & alain abran)

www.eng.it

Measurement Process: Improving the ISO 15939 Standard

24° International Workshop on Software Measurement (IWSM) and 9th International Conference on Software

Process and Product Measurement (MENSURA)

Rotterdam (Netherlands) - October 6-8, 2014

Luigi BuglioneBuglione

Alain AbranAbran

www.eng.it2 IWSM-MENSURA 2014 – October 6-8, 2014© 2014 Luigi Buglione & Alain Abran

Measurement Process Goals of the presentation

1. Discuss the value arising from a Measurement Plan 2. Introduce suggestions for a refined Measurement Information Model (MIM) 3. Propose a recommended content for a Measurement Plan


ETS - GELOG At a glance

gelog.etsmtl.ca

gelog.etsmtl.ca


Engineering At a glance

www.eng.itwww.eng.it


Agenda

• Introduction– Three steps back…– Some basic questions…

• ISO 15939 – Measurement Process– Context– What is included in the current version (2007)– What could be included in a next version

• Improving the MIM– Refining and Improving the MIM

• Measurement Plan– The ‘5Ws + 2H’ proposal– The BMP technique

• Summary & Next Steps

• Q&&A

Measurement Process


Introduction Three Steps back...

You cannot You cannot controlcontrol what you what you cannot cannot measure measure but...but...

...You cannot...You cannot measuremeasure what what youyou cannotcannot define define but...but...

...You cannot...You cannot definedefine what what you don’tyou don’t knowknow......


Introduction Some basic questions....

When measuring in your organization, do you have a ‘measurement plan’ or not?

Do you properly define your measures with needed details for a precise measurement or are they simply listed? E.g. What is a LOC?

When establishing your set of measures, do you start from your own goal or from a predefined list (e.g. PSM) or are you goal-oriented?

Do you balance your measures by multiple criteria or are you mostly focusing on time and costs? If yes, why not also other perspectives (e.g. quality, risk, etc.)?


Agenda



• Improving the MIM– Refining and Improving the MIM



• Q&&A

Measurement Process


ISO/IEC 15939 Context

• Specific ISO standard on Measurement 1° Ed: 1999, 2° Ed: 2007

• ‘Explodes’ the ones in ISO 12207, 15288 or 15504 process models

• Adopt the VIM (Int. Vocabulary of Metrologogy) terms and glossary


ISO/IEC 15939 What is included in the current version

• Information Needs & Products Introduced a MIM (Measurement

Information Model) MIM structures a series of

measurement-related concepts Several levels: from the ‘information

needs’ till measurable attributes• Metrology Alignment

Partial alignment with language/concepts proposed in Metrology

Breakout in the Software Engineering domain

E.g. VIM (International Vocabulary of Metrology)

Measurement Experience Base (MEB) A good estimation could be done

using your own historical data MEB is the organizational database

data from activities Present from the first 1999 edition of

the standard Fundamental for re-establishing and

tuning threshold values from data

Some strengths...Some strengths...


ISO/IEC 15939 What could be included in a next version

Some improvement points...Some improvement points...

• MIM structure Now: The measurement concepts included in the ISO 15939 standard are

not fully included in the current MIM graphical representation Tomorrow: MIM picture could be split into ‘phases’ and specify better

roles and moments in time during the life cycle for detecting issues for improvements

• Clause 5.2 - ‘Plan the measurement process’ Now: single key measurement tasks may be collected into a

measurement plan Tomorrow: could refer to such coordination but using proper criteria for

obtaining a valuable measurement plan as the combination of multiple viewpoints/perspectives (stakeholders) introduce formally the ‘measurement plan’ definition and concept in the normative part (currently ‘weak’ §5.2.6.1 + Annex F)

Clause 4.2.3 – ‘Select measures’ Now: asks for criteria to select measures, but with no suggestions about

‘how’ Tomorrow: insert explicitly possible criteria (e.g. balancing

criteria/number of measures by perspectives, overall cost, information value, etc.)


Agenda



• Improving the MIM– Refining the MIM– Improving the MIM



• Q&&A

Measurement Process


Improving the MIM Refining the MIM – Elements & Structure

• Phases Three main phases: Data Analysis, Data

Preparation, Data Collection Each phase will include some of the

activities of the current MIM • Use of measurement results

Analysis of relationships across different measurement results

Assessment against reference contexts for evaluation and/or decision-making

‘Hidden’ details to be elicited and made visible Decision criteria Assumptions Expected relationships Estimates or evaluation Numerical thresholds or targets Statistical confidence limits ...


Improving the MIM Improving the MIM – Additional concepts

• Standard Reference Model Generic/Accepted model of Info Needs It’d include:

Set of individual relationships, jointly with measures to be considered (e.g. evaluation and/or decision criteria)

Algorithm combining measures in implied criteria

• Organizational Reference Context Aligned with the Standard Reference

Model It’d include:

Decision criteria (w/target values, evaluation scales)

Reference values according to contexts (benchmarking domains, application types, etc. – www.isbsg.org)

Data Analysis As currently defined but...

It’d be better feeded by data/information coming from the Standard Model and the Organizational Reference Context


Agenda






• Q&&A

Measurement Process


Measurement Plan The ‘5Ws+2H’ proposal

• The original ‘5W+H’ journalist rule Typically a good paper should contain:

Who, What, Why, When, Where, How Who: not well expressed now RACI matrix for

describing who should collect, prepare, and analyze data

• The 2° H – a possible addition ‘How Much’ threshold(s) value(s) As in Control Charts (TQM), possibly two (2) thresholds:

UCL (Upper Control Level) LCL (Lower Control Level)

Threshold(s) should be periodically (re)evaluated from MEB data

• Utility and applicability Same suggestion for a good (whatever) plan It could be a possible refinement for the current

Appendix F (‘Example elements of measurement planning’)

In such way it could be possible to check easily with a structured manner what has/hasn’t included in the measurement plan

Not necessarily the ‘measurement plan’ should be part of a project/quality plan, it could/should be a separate plan for including a proper level of detail


Balancing Multiple Perspectives (BMP)Measurement Plan

• Not too few, not too many... A common-sense rule, always valid in the ‘real life’, less respected

in the business world ‘sustainability’ Not forgetting that ‘you cannot control what you cannot measure’ ... what is the right number of measures? ...and what is the right number of viewpoints to consider? Goal: optimize the TCO, based on a positive ROI/VOI

• How much does it cost to measure? Measurement IS NOT part of Project Management

E.g. CMMI MEA, ISO/IEC 15504-2 MAN.5, etc... Did you forecast the proper budget on Measurement?

• How many viewpoints to consider at the same time? Stakeholders’ analysis and involvement is fundamental to success Are we looking to the right stakeholders? Are we achieving ‘value’ from our product/services? ...we need to properly mix all needed elements in our ‘Measurement Plan’ in

order to be successful but ‘balanced’ Don’t forget to analyze ‘counter-effects’ (pros & cons)


Measurement Plan

• Measurement procedure: Determine the dimensions of interest

in the project.

Determine the list of the most representative measures associated with each dimension.

For each of the measures selected, identify which other control variables might be impacted negatively.

Figure out the best combination of indicators and the causal relations between them in order to build a measurement plan for the project.

BMP: Balancing Multiple Perspectives

• Balancing Multiple Perspectives (BMP) Multi-perspective approach proposed in 2005 www.semq.eu/leng/modtechbmp.htm


BMP: Causal Relationships

• Not sufficient to perform steps#1 and #2 (design a measurement plan)• Next (required) step: establish coherent and proper relationships among

goals through measures (i.e. the BSC strategic map) for achieving both single perspective goals, as well as overall organizational ones

NBNB: also explicitly required by ISO 9001:2008 (§4.1 lett.c)

Measurement Plan


Agenda






• Q&&A

Measurement Process


Summary && Next Steps

• Measurement in the practice Measurement is one of the most needed and relevant processes, often under-

evaluated It helps to manage a project but is separated from the Project Management process Measuring helps in determining the ‘how much’ we need to produce for better

estimates time (effort/duration) and costs/prices, reducing more and more subjective views and errors

• ISO/IEC 15939 – up today ISO standard on the Measurement Process Aligned with Metrology rules and standards (e.g. VIM, GUM) First edition (1999), second (current) edition: 2007 A third edition is on the way...some possible improvements suggested

• ISO/IEC 15939 – suggestions for tomorrow MIM – refining the structure (phases), introducing roles (RACI) and thresholds (from

MEB) Measurement plan – introduce a template, applying the 5W’s+2H rules and BMP

technique Next Steps

Refine this proposal into more operative terms for possible introduction into next ISO 15939 revision process

...let’s start measure introducing these suggestions!

All models are wrong. Some models are useful.

(George Box, Mathematician, 1919-2013)

All models are wrong. Some models are useful.

(George Box, Mathematician, 1919-2013)

Measur. Framework


Lessons Learned...Measurement Process


Q && A

Bedankt voor uw aandachtBedankt voor uw aandacht!!Thanks for your attentionThanks for your attention!!

Measurement Process


Our Contact Data

Luigi Buglione

Engineering Ingegneria Informatica/ETS

[email protected]

AlainAbran

ETS Montré[email protected]

a

Measurement Process

Iwsm2014 measurement process - improving the iso 15939 standard (luigi buglione & alain abran)

Software

Transcript of Iwsm2014 measurement process - improving the iso 15939 standard (luigi buglione & alain abran)