CMMI Conference Denver, Colorado 2004€¦ · Evolving a Measurement Program for Systems and...

Evolving a MeasurementEvolving a MeasurementProgram for Systems andProgram for Systems and

Software Engineering ProcessSoftware Engineering ProcessImprovementImprovement

CMMI ConferenceCMMI ConferenceDenver, ColoradoDenver, Colorado

20042004

Techniques for Meas Pgm - 2version CMMI Conf 2004© 2004 Kasse Initiatives, LLC

WelcomeWelcome

Wilkommen

Bienvenido

WelKom

Bienvenue

BienvenutoVälkom

men

Tervetuloa Witamy

Huan Yín

ЌАΛΟΣΟΡΙΣΑΤ

Ε

��


MeasurementMeasurementIs It ReallyIs It ReallyNecessary?Necessary?


At Project Start, DoAt Project Start, DoYou Know…?You Know…?

�Can it be done?�How long will it take?�How much will it cost?�How many people will it take?�What is the risk?�What are the tradeoffs?�How many errors will there be?


What Do You KnowWhat Do You KnowNow?Now?

�How much does your current developmentprocess cost?

�How much value does each piece of theprocess add?

�What would the impact be of deleting,modifying, adding a procedure to the process?

�What activities contribute the most to the finalproduct cost?

�Have you tried to improve the currentdevelopment process?�What changes in cost/value resulted from that

improvement effort?


What Do You KnowWhat Do You KnowNow?Now? -- 22

�What processes represent the greatestpotential for return on improvementinvestment?

�How would you quantify the value of theprocess improvement investment?

�Do you really want to know where the moneyis going in your software developmentprojects?

�What value do you think you are delivering toyour customers? Do they agree?

�How much is the knowledge of your costs andthe value delivered worth to you?


MeasurementMeasurementand Analysisand Analysis


Measurement andMeasurement andAnalysis OverviewAnalysis Overview

�A measurement initiative involves thefollowing:�Specifying the objectives of measurement and

analysis such that they are aligned with establishedinformation needs and business objectives

�Defining the measures to be used, the datacollection process, the storage mechanisms, theanalysis processes, the reporting processes, andthe feedback processes

�Implementing the collection, storage, analysis, andpresentation of the data

�Providing objective results that can be used inmaking business judgments and taking appropriatecorrective actions


EstablishEstablishMeasurementMeasurementObjectivesObjectives

�Measurement objectives�Document the purposes for which measurement

and analysis is done�Specify the kinds of actions that may be taken

based on the results of the data analyses�Continually ask the question – what value will this

measurement be to those people who will be askedto supply the raw measurement data and who willreceive the analyzed results

�Involve the end users of the measurement andanalysis results in setting measurement objectiveswhenever possible

�Maintain traceability of the proposed measurementobjectives to the information needs and businessobjectives


Specify MeasuresSpecify Measures

�Identify candidate measures based ondocumented objectives and refine intoprecisely defined, quantifiable measures

�Define how data can and will be derived fromother measures

�Examples of Commonly Used DerivedMeasures�Defect density�Peer review coverage�Reliability measures


Specify Data CollectionSpecify Data Collectionand Storage Proceduresand Storage Procedures

�Specify how to collect and store the data foreach required measure�Make explicit specifications of how, where, and

when the data will be collected

�Develop procedures for ensuring that the datacollected is valid data

�Ensure that the data is stored such that it is easilyaccessed, retrieved, and restored as needed


Specify AnalysisSpecify AnalysisProceduresProcedures

�Define the analysis procedures in advance�Ensure that the results that will be fed back

are understandable and easily interpretable�Collecting data for the sake of showing an

assessor the data is worthless�Showing how it can be used to manage and control

the project is what counts


CollectCollectMeasurement DataMeasurement Data

�Collect the measurement data as defined, atthe points in the process that were agreed to,according to the time scale established

�Generate data for derived measures

�Perform integrity checks as close to thesource of the data as possible


Analyze theAnalyze theMeasurement DataMeasurement Data

�Conduct the initial analyses�Interpret the results and make preliminary

conclusions from explicitly stated criteria�Conduct additional measurement and

analyses passes as necessary to gainconfidence in the results

�Review the initial results with all stakeholders�Prevents misunderstandings and rework

�Improve measurement definitions, datacollection procedures, analyses techniques asneeded to ensure meaningful results thatsupport business objectives


Store theStore theMeasurement DataMeasurement Dataand Analyses Resultsand Analyses Results

�The stored information should contain orreference the information needed to:�Understand the measures�Assess them for reasonableness and applicability

�The stored information should also:�Enable the timely and cost effective future use of

the historical data and results�Provide sufficient context for interpretation of the

data, measurement criteria, and analyses results


Communicate theCommunicate theMeasurement ResultsMeasurement Results

�Keep the relevant stakeholders up-to-dateabout measurement results on a timely basis

�Follow up with those who need to know theresults�Increases the likelihood that the reports will be

used�Assist the relevant stakeholders in

understanding and interpreting themeasurement results


Measurement andMeasurement andAnalysis GroupAnalysis Group

�Consider creating a measurement group thatis responsible for supporting the Measurementand Analysis activities of multiple projects


Measurement andMeasurement andAnalysis ToolsAnalysis Tools

�Incorporate tools used in performingMeasurement and Analysis activities such as:�Statistical packages�Database packages�Spreadsheet programs�Graphical packages�Packages that support data collection over

networks and the internet


Measurement andMeasurement andAnalysis TrainingAnalysis Training

�Provide training to all people who will performor support the Measurement and Analysisprocess�Data collection, analyses, and reporting processes�Measurement tools�Goal-Question-Metric Paradigm�How to establish measures

� how to determine efficiency and effectiveness�Quality factors measures (e.g., maintainability,

expandability)�Basic and advanced statistical techniques


Basic MeasuresBasic Measures


Basic MeasuresBasic Measures

�Estimate Size and/or Complexity - a relativelevel of difficulty or complexity should beassigned for each size attribute

�Examples of attributes to estimate for SystemsEngineering include:�Number of logic gates�Number of interfaces

�Examples of size measurements for SoftwareEngineering include:�Function Points�Lines of Code�Number of requirements


Basic MeasuresBasic Measures -- 22

�Determine effort and cost – use historicaldata or models

�Establish the project’s schedule based on thesize and complexity estimations

�Include, or at least consider, infrastructureneeds such as critical computer resources

�Identify risks associated with the cost,resources, schedule, and technical aspects ofthe project

�Control data (various forms of documentation)required to support a project in all of its areas.


Basic MeasuresBasic Measures –– 33

�Identify the knowledge and skills needed toperform the project according to the estimates

�Select and implement methods for providingthe necessary knowledge and skills�Training (Internal and External)�Mentoring�Coaching�On-the-job application of learned skills

�Monitor staffing needs – based on effortrequired and the necessary knowledge andskills to achieve the defined tasks



�Involve relevant stakeholders throughout theproduct lifecycle

�Track technical performance (Completionof activities and milestones against theschedule Example:�Product components designed, constructed, unit

tested and integrated�Compare actual milestones completed vs.

established commitments�Monitor commitments and critical

dependencies against those documented inthe project plan



�Track quality – Problems/Defects(open/closed by product/activity)�Problems and defects are direct contributors to

the amount of rework that must be performed—asignificant cost factor in development andmaintenance


Effectiveness ofEffectiveness ofProcessesProcesses


Effectiveness ofEffectiveness ofProcessesProcesses

�In addition to defining the processes that wewish to follow on our project, we need toensure we are following them and we shouldbe able to determine if the processes areworking for us the way we expected them to�How well are the processes working?


RequirementsRequirementsManagementManagement ––Effectiveness ExampleEffectiveness Example

�Number of change requests per monthcompared with the original number ofrequirements for the project�Critical change requests�Intermediate change requests�Nice to have change requests

�Time spent on change requests up until a Y/Ndecision is given from the Senior Contractgroup

�Number and size of critical change requeststhat arise after the requirements phase hasbeen completed


Set of StandardSet of StandardOrganizationalOrganizational

ProcessesProcesses


Importance of anImportance of anOrganizational ViewOrganizational Viewof Processesof Processes

�Builds a common vocabulary�Allows others to anticipate behavior and be

more proactive in their interactions�Allows the organization to measure a

controlled set of processes to gain economy ofscale

�Trends can be seen and predictability can beachieved

�Process performance baselines can bedeveloped to support quantitativemanagement later


OrganizationalOrganizationalMeasurementMeasurementRepositoryRepository

�Develop an organization measurementrepository - include:�Product and process measures that are related to

the organization’s set of standard processes

�The related information needed to understandand interpret the measurement data and asses itfor reasonableness and applicability

�Develop operational definitions for themeasures to specify the point in the processwhere the data will be collected and for theprocedures for collecting valid data


Slightly MoreSlightly MoreAdvanced MeasuresAdvanced Measures


Peer ReviewsPeer Reviews


Peer ReviewPeer ReviewMeasuresMeasures

�Optimum Checking Rate (e.g., Number ofpages to be checked per hour)

�Logging Rate (e.g., Number of major anddefects logged per hous)

�Number of Major and Minor Defects�Effectiveness - Number of Major Defects

found in this stage compared to the totalnumber of defects found so far


TestingTesting


Defects DiscoveredDefects DiscoveredDuring TestingDuring Testing

�Effectiveness - Number of Major defects foundin a particular testing phase or instantiation ofa testing phase compared to the total numberof defects found during testing

�Number of defects projected to escape fromthe current testing phase


Test CoverageTest Coverage

�Statement coverage measures whether eachexecutable statement is encountered

�Block coverage is the same as statementcoverage except that the unit of codemeasured is each sequence of non-branchingstatements

�Decision coverage measures whether booleanexpressions tested in control structures (suchas if-statements or while-statements)evaluated to both true and false

�Condition coverage measures the true or falseoutcome of each boolean sub-expression


Test CoverageTest Coverage -- 22

�Function Coverage�Measures whether you invoked each function or

procedure�Call Coverage

�Measures whether you executed each function call�Based on the idea that faults commonly occur in

interfaces between modules�Loop Coverage

�Measures whether you executed each loop bodyzero times, exactly once, and more than once

�Relational Operator Coverage�Measures whether boundary situations occur with

relational operators (=)


Risk ManagementRisk Management

Techniques for Meas Pgm - 40version CMMI Conf 2004© 2004 Kasse Initiatives, LLCImpact

Prob

abili

ty

Time

Prioritizing RisksPrioritizing Risks

�Priorities are established by merging theinformation available into a three dimensionaltable


PrioritizingPrioritizing -- 22

Catastrophic

Critical

Marginal

Negligible

Probable Possible Unlikely

Top

High

Serious

Medium

High

Serious

Medium

Low

Serious

Medium

Low

Bottom

Probability:

Impact:


PrioritizingPrioritizing -- 33

Immediate Short Medium

1

2

3

4

2

3

4

5

3

4

5

6

5 6 7

6 7 8

Top

High

Serious

Medium

Low

Bottom

Long

4

5

6

7

8

9

TimeFrame:


Quantitative ProjectQuantitative ProjectManagementManagement


QuantitativeQuantitativeManagementManagementConceptsConcepts

�Quantitative Management is tied to theorganization’s strategic goals for productquality, service quality, and processperformance

�When higher degrees of quality andperformance are demanded, the organizationand projects must determine if they have theability to improve the necessary processes tosatisfy the increased demands


QuantitativeQuantitativeManagementManagementConceptsConcepts -- 22

�Achieving the necessary quality and processperformance objectives requires stabilizing theprocesses that contribute most to theachievement of the objectives

�Assuming the technical requirements can bemet, the next decision is to determine if it iscost effective

�Reducing process variation is an importantaspect to quantitative management


Establish the Organization’sEstablish the Organization’sProcess PerformanceProcess PerformanceBaselineBaseline

�Process performance is a measure of theactual results achieved by following a process

�Measure the organization’s processperformance for the organization’s set ofstandard processes at various levelsincluding:�Individual processes (e.g., test case inspection

element)�Sequence of connected processes�Processes for the complete lifecycle�Processes for developing individual work products


Establish the Organization’sEstablish the Organization’sProcess PerformanceProcess PerformanceBaselineBaseline -- 22

�There may be several process performancebaselines to characterize performance forsubgroups of the organization – Examplesinclude:�Product Line�Application domain�Complexity�Team size�Work product size


Steps to Establish Project’sSteps to Establish Project’sQuality and ProcessQuality and ProcessPerformance ObjectivesPerformance Objectives

�Identify the quality and process performanceneeds and priorities of the customer, endusers, and other relevant stakeholders�Reliability�Maintainability and Expandability�Usability�Development cycle time

�Determine how quality and processperformance will be measured


Steps to Establish Project’sSteps to Establish Project’sQuality and ProcessQuality and ProcessPerformance ObjectivesPerformance Objectives -- 22

�Define and document measurable quality andprocess performance objectives for the project

�Examples of Quality Objectives�Mean time between failures�Critical resource utilization

�Examples of Process Performance Objectives�Percentage of defects removed by type of

verification activity�Defect escape rates�Number and density of defects (by severity) found

during the first year following product delivery�Rework time as a percentage of total project life-

cycle time


Steps to Establish Project’sSteps to Establish Project’sQuality and ProcessQuality and ProcessPerformance ObjectivesPerformance Objectives -- 33

�Establish objectives for each life-cycle stageto track progress and predict achievement ofproject’s objectives

�Establish criteria to identify whichsubprocesses are the main contributors toachieving the identified quality and processperformance objectives and for whichpredictable performance is important


MeasuresMeasuresand Analyticand AnalyticTechniquesTechniques


Select Measures andSelect Measures andAnalytic TechniquesAnalytic Techniques

�Identify the measures that are appropriate forstatistical management

�Examples of subprocess control measuresinclude:�Ratios of estimated to measured values of the

planning parameters�Coverage and efficiency of work product

inspections�Test coverage and efficiency�Reliability, Maintainability, and Expandability�Percentage of the total defects inserted or found in

the different stages of the lifecycle


�There are a number of quantitative tools consideredto be applicable to statistical process or quality control:�Quantifying and Predicting Process Performance

� Control Charts� Histograms

�Cause and Effect Relationships� Scatter diagrams� Run charts� Cause-and-effect (fishbone) diagrams� Bar charts� Pareto charts� Interrelationship Diagraph� Kiviat Diagram

Quantitative ToolsQuantitative Tools


Control ChartsControl Charts

�Control charts – techniques for quantifyingprocess behavior�Classical control charts have a centerline and

control limits on both sides of the centerline�Both the centerline and the limits represent

estimates that are calculated from a set ofobservations collected while the process is running

�The centerline and control limits cannot beassigned arbitrarily as they are intended to showwhat the process can actually do

Florac, W.A. & Carleton, A.D. Measuring the Software ProcessAddison-Wesley, 1999

Techniques for Meas Pgm - 55version CMMI Conf 2004© 2004 Kasse Initiatives, LLCNumerical data taken

in time sequence

UpperControlLimit(UCL)

LowerControlLimit(LCL)

Upper andLower

Control Limitsrepresent the

natural variationIn the process

METRIC:PROCESS CONTROL CHART TYPE:

A point above or below thecontrol lines suggests that the

measurement has a specialpreventable or removable cause

Plotted points are eitherindividual measurements or the

means of small groups ofmeasurements

The chart is used for continuousand time control of the process

and prevention of causes

The chart is analyzed usingstandard Rules to define the

control status of the process

Datarelating to

the process

Center Line (CL)(Mean of data used toset up the chart)

Statistical Methods for Software QualityAdrian Burr – Mal Owen, 1996


HistogramsHistograms

�Histograms�Allows a project team to take measurement data

and display the distribution of the observed values�Show the frequencies of events that have occurred

in ways that make it easy to compare distributionsand see central tendencies

�Illustrates quickly the underlying distribution of thedata

�Helps indicate if there has been a change in theprocess



HistogramsHistograms -- 22

�Histograms - continued�Provides useful information for predicting future

performance of the process�Helps answer the question “Is the process capable

of meeting my customers requirements?”�Histograms are created by grouping the results into

“cells” and then counting the number in each cell



Num

ber o

f Day

s

Product – Service Staff Hours

20

56

181614121086420

32 5452504846444240383634

HistogramsHistograms


Applying StatisticalApplying StatisticalMethods toMethods toUnderstandUnderstand

VariationVariation


Understanding VariationUnderstanding Variation

�Understanding variation is achieved bycollecting and analyzing process and productmeasures so that special causes of variationcan be identified and addressed to achievepredictable performance

�All characteristics of processes and productsdisplay variation when measured over time

�Variation may be due to�Special or “assignable” causes of variation�Natural or common causes


SummarySummary

�Techniques for Establishing a MeasurementProgram include:�Clearly define the need for a measurement

program�Establish a measurement initiative with objectives

that are aligned with established information needsand business objectives

�Ensure basic measures are included for planning,tracking, and taking corrective action as necessary

�Incorporate process effectiveness measures�Establish organizational standard processes


SummarySummary -- 22

�Establish and utilize measures such as peer reviewmeasures, testing measures, and risk managementmeasures

�Evolve into project management based on aquantitative understanding of the organization’sand project’s defined processes


Tim KasseTim KasseKasse Initiatives LLCKasse Initiatives LLC

PMB 293PMB 2931900 Preston Road, Suite 2671900 Preston Road, Suite 267

Plano, Texas 75093Plano, Texas 75093U.S.A.U.S.A.

Tel: +1 972 987 7606Tel: +1 972 987 7606

E-mail: [email protected]

Web Site: www.kasseinitiatives.com

Evolving a Measurement Program f...WelcomeMeasurement�Is It Really Necessa...At Project Start, Do You Know…?Can it be done?How long will it take?How much will it cost?How many people will it take?What is the risk?What are the tradeoffs?How many errors will there be?

What Do You Know Now?How much does your current devel...How much value does each piece o...What would the impact be of dele...What activities contribute the m...Have you tried to improve the cu...What changes in cost/value resul...

What Do You Know Now? - 2What processes represent the gre...How would you quantify the value...Do you really want to know where...What value do you think you are ...How much is the knowledge of you...

Measurement �and AnalysisMeasurement and Analysis Overvie...A measurement initiative involve...Specifying the objectives of mea...Defining the measures to be used...Implementing the collection, sto...Providing objective results that...

Establish Measurement ObjectivesMeasurement objectivesDocument the purposes for which ...Specify the kinds of actions tha...Continually ask the question – w...Involve the end users of the mea...Maintain traceability of the pro...

Specify MeasuresIdentify candidate measures base...Define how data can and will be ...Examples of Commonly Used Derive...Defect densityPeer review coverageReliability measures

Specify Data Collection and Stor...Specify how to collect and store...Make explicit specifications of ...Develop procedures for ensuring ...Ensure that the data is stored s...

Specify Analysis ProceduresDefine the analysis procedures i...Ensure that the results that wil...Collecting data for the sake of ...Showing how it can be used to ma...

Collect �Measurement DataCollect the measurement data as ...Generate data for derived measur...Perform integrity checks as clos...

Analyze the Measurement DataConduct the initial analysesInterpret the results and make p...Conduct additional measurement a...Review the initial results with ...Prevents misunderstandings and r...Improve measurement definitions,...

Store the Measurement Data and A...The stored information should co...Understand the measuresAssess them for reasonableness a...The stored information should al...Enable the timely and cost effec...Provide sufficient context for i...

Communicate the Measurement Resu...Keep the relevant stakeholders u...Follow up with those who need to...Increases the likelihood that th...Assist the relevant stakeholders...

Measurement and Analysis GroupConsider creating a measurement ...

Measurement and Analysis ToolsIncorporate tools used in perfor...Statistical packagesDatabase packagesSpreadsheet programsGraphical packagesPackages that support data colle...

Measurement and Analysis Trainin...Provide training to all people w...Data collection, analyses, and r...Measurement toolsGoal-Question-Metric ParadigmHow to establish measureshow to determine efficiency and ...Quality factors measures (e.g., ...Basic and advanced statistical t...

Basic MeasuresBasic MeasuresEstimate Size and/or Complexity ...Examples of attributes to estima...Number of logic gatesNumber of interfacesExamples of size measurements fo...Function PointsLines of CodeNumber of requirements

Basic Measures - 2Determine effort and cost – use ...Establish the project’s schedule...Include, or at least consider, i...Identify risks associated with t...Control data (various forms of d...

Basic Measures – 3Identify the knowledge and skill...Select and implement methods for...Training (Internal and External)MentoringCoachingOn-the-job application of learne...Monitor staffing needs – based o...

Basic Measures - 4Involve relevant stakeholders th...Track technical performance (Com...Product components designed, con...Compare actual milestones comple...Monitor commitments and critical...

Basic Measures - 5Track quality – Problems/Defects...Problems and defects are direct ...

Effectiveness of ProcessesEffectiveness of ProcessesIn addition to defining the proc...How well are the processes worki...

Requirements Management – Effect...Number of change requests per mo...Critical change requestsIntermediate change requestsNice to have change requestsTime spent on change requests up...Number and size of critical chan...

Set of Standard Organizational P...Importance of an Organizational ...Builds a common vocabularyAllows others to anticipate beha...Allows the organization to measu...Trends can be seen and predictab...Process performance baselines ca...

Organizational Measurement Repos...Develop an organization measurem...Product and process measures tha...The related information needed t...Develop operational definitions ...

Slightly More Advanced MeasuresPeer ReviewsPeer Review�MeasuresOptimum Checking Rate (e.g., Num...Logging Rate (e.g., Number of ma...Number of Major and Minor Defect...Effectiveness - Number of Major...

TestingDefects Discovered During Testin...Effectiveness - Number of Major ...Number of defects projected to e...

Test CoverageStatement coverage measures whet...Block coverage is the same as st...Decision coverage measures wheth...Condition coverage measures the ...

Test Coverage - 2Function CoverageMeasures whether you invoked eac...Call CoverageMeasures whether you executed ea...Based on the idea that faults co...Loop CoverageMeasures whether you executed ea...Relational Operator CoverageMeasures whether boundary situat...

Risk ManagementPrioritizing RisksPriorities are established by me...

Prioritizing - 2Prioritizing - 3Quantitative Project ManagementQuantitative Management ConceptsQuantitative Management is tied ...When higher degrees of quality a...

Quantitative Management Concepts...Achieving the necessary quality ...Assuming the technical requireme...Reducing process variation is an...

Establish the Organization’s Pro...Process performance is a measure...Measure the organization’s proce...Individual processes (e.g., test...Sequence of connected processesProcesses for the complete lifec...Processes for developing individ...

Establish the Organization’s Pro...There may be several process per...Product LineApplication domainComplexityTeam sizeWork product size

Steps to Establish Project’s Qua...Identify the quality and process...ReliabilityMaintainability and Expandabilit...UsabilityDevelopment cycle timeDetermine how quality and proces...

Steps to Establish Project’s Qua...Define and document measurable q...Examples of Quality ObjectivesMean time between failuresCritical resource utilizationExamples of Process Performance ...Percentage of defects removed by...Defect escape ratesNumber and density of defects (b...Rework time as a percentage of t...

Steps to Establish Project’s Qua...Establish objectives for each li...Establish criteria to identify w...

Measures �and Analytic Technique...Select Measures and Analytic Tec...Identify the measures that are a...Examples of subprocess control m...Ratios of estimated to measured ...Coverage and efficiency of work ...Test coverage and efficiencyReliability, Maintainability, an...Percentage of the total defects ...

Quantitative ToolsThere are a number of quantitati...Quantifying and Predicting Proce...Control ChartsHistogramsCause and Effect RelationshipsScatter diagramsRun chartsCause-and-effect (fishbone) diag...Bar chartsPareto chartsInterrelationship DiagraphKiviat Diagram

Control ChartsControl charts – techniques for ...Classical control charts have a ...Both the centerline and the limi...The centerline and control limit...

Slide 55HistogramsHistogramsAllows a project team to take me...Show the frequencies of events t...Illustrates quickly the underlyi...Helps indicate if there has been...

Histograms - 2Histograms - continuedProvides useful information for ...Helps answer the question “Is th...Histograms are created by groupi...

HistogramsApplying Statistical Methods to ...Understanding VariationUnderstanding variation is achie...All characteristics of processes...Variation may be due toSpecial or “assignable” causes o...Natural or common causes

SummaryTechniques for Establishing a Me...Clearly define the need for a me...Establish a measurement initiati...Ensure basic measures are includ...Incorporate process effectivenes...Establish organizational standar...

Summary - 2Establish and utilize measures s...Evolve into project management b...

Tim Kasse�Kasse Initiatives LLC�...

CMMI Conference Denver, Colorado 2004€¦ · Evolving a Measurement Program for Systems and...

Documents

Transcript of CMMI Conference Denver, Colorado 2004€¦ · Evolving a Measurement Program for Systems and...