Controlling Project Performance Using the Project Defect Model 1 March 18, 2005 Ben Linders

Controlling Project Performance Using the Project Defect Model

PSQT 2005 Conference,Las Vegas, May 3

Ben LindersOperational Development & Quality

Ericsson R&D, The Netherlandsben.linders@ericsson.com, +31 161 24 9885

Controlling Project Performance Using the Project Defect Model 2 March 18, 2005 Ben Linders

Overview

• Why a defect model?• How does it work?• Experiences from projects• Conclusions

Measurements for product qualityand process effectiveness

Controlling Project Performance Using the Project Defect Model 3 March 18, 2005 Ben Linders

Ericsson, The Netherlands

• Market Unit Northern Europe & Main R&D Design Center • R&D: Intelligent Networks

– Strategic Product Management– Product marketing & technical sales support– Provisioning & total project management– Development & maintenance– Customization– Supply & support

• 1300 employees, of which 350 in R&D

Projects: Quality next to Lead-time and Costs

Controlling Project Performance Using the Project Defect Model 4 March 18, 2005 Ben Linders

Purpose Project Defect Model

Why?– to control quality of the product during development– and improve development/inspection/test processes

Business Benefit:➨ Better planning & tracking➨ Early risks signals➨ Save time and costs➨ Happy customers!

Controlling Project Performance Using the Project Defect Model 5 March 18, 2005 Ben Linders

History of the Model

• 2001– Defined, introduced in first project

• 2002– Used in 2 projects, improved along the way– First release predictions

• 2003– Industrialize model/tool– Used in all (5) major projects

• 2004– Management decisions based on model– New applications: Solution/Total projects, defect flows

• 2005– Extension with Cost of Quality

Controlling Project Performance Using the Project Defect Model 6 March 18, 2005 Ben Linders

Modeling Defect Flow

Insertion: Where are defects made? How to prevent?Detection: Where are defects found? Early/economic removal?

Controlling Project Performance Using the Project Defect Model 7 March 18, 2005 Ben Linders

Process View

ProcessInputs and outputsInfluencing factorsMeasurement

Defects Inserted (documentation,

code)

Defects Detected (Inspection, test)

(Un)happy customers

Design ProcessCompetence, skillsTools, environment

Test ProcessCompetence, skills

Test CapacityTools, environment

Resident Defects in Delivered Product

Resident Defects in Design Base

Detection Rate

Defect Density

Fault Slip Through

Defect Level

Defect Classification

Controlling Project Performance Using the Project Defect Model 8 March 18, 2005 Ben Linders

Planning & Tracking of Quality• Plan Quality Up Front

– Documents/code (# defects made)– Inspection & Test effectiveness (% detection rate)

Quality consequence of project approach

• Track Quality during project– Actual # defects found (inspection/test)– Estimate remaining defects: to be found / delivered

Quality view of design/test, quicker escalation

• Decide based upon Quality Status– Toll Gates (go/no go) and Product Release

Product Quality figures

Controlling Project Performance Using the Project Defect Model 9 March 18, 2005 Ben Linders

Implementation• Tool: Excel based defect data base & estimation• Frequent estimation & analysis/feedback sessions • Weekly tracking & reporting of product quality• Includes proven techniques: ODC, requirement coverage, test matrices

Tailored per project, flexible, result orientedOverall data based on all projects: Planning constants

Quality data, additional to time & costs!

Controlling Project Performance Using the Project Defect Model 10 March 18, 2005 Ben Linders

Results

• Data from the projects• Feedback sessions• Conclusions

11 projects, of which 2 ongoingIncremental development, team basedDifferent size/length: size factor used.RUP based process

Controlling Project Performance Using the Project Defect Model 11 March 18, 2005 Ben Linders

Detection rates projectsProject detection rates Q1 2005 (PSQT Conference)

Proj A Proj B Proj C Proj D Proj E Proj F* Proj G Proj H * Proj J Proj K Proj L AverageRate 95% 95% 90% 59% 97% 86% 93% 88% 91% 94% 93% 91%Size 1 4 1 1 5 3 1 4 1 2 3

* Project still ongoing at time of measurement

• Limited variance– Project D different: Integration of products (no design)– Range (excluding D) from 86%-97% (projects F and H still ongoing)

• Average detection rate in line with industry figures:– DACS: Typical software projects 15% slip though (85% detection)– Jones: Average 85%, most efficient 95%

Analyze/track projects that go below the target performance of 90%

Controlling Project Performance Using the Project Defect Model 12 March 18, 2005 Ben Linders

Injection rates phasesPhase injection rates, Q1 2005 (PSQT Conference)

R e quire m e n A rc hite c tureD e s ig n C o de D o c wa re

Rate 7% 18% 12% 58% 4%

• Very elaborated architecture (feasibility phase). Many defects made, but most of them are found in the architecture reviews.

• Lean design, few defects made.• Most defects made during coding

“Normal” defect pattern, sufficient focus on defect prevention.

Controlling Project Performance Using the Project Defect Model 13 March 18, 2005 Ben Linders

Detection rates phases, averages

Phase detection rates, Q1 2005 (PSQT Conference)R e quire m e n A rc hite c tureD e s ig n C o de D o c wa re F unc t io n Te S ys te m Te s N e two rk Te A v e ra g e

Det. Rate 56% 64% 51% 36% 70% 56% 49% 23% 51%

• High requirements/architecture/design: effective inspections, good architecture skills

• Low code detection: improvement program ongoing• Function & system test: Acceptable rates• Network test, low rate, but defects that are found would give significant

problems to customers: Good cost/benefit of the test phase

Focus on inspection improvement, capture defects earlier

Controlling Project Performance Using the Project Defect Model 14 March 18, 2005 Ben Linders

Detection rates phases, variance

0%10%20%30%40%50%60%70%80%90%

100%

Requir

emen

tsArch

itectu

re

Design

Code

Docware

Functi

on Tes

tSys

tem Tes

tNetw

ork Tes

t

Total

Det. Rate

• Large variance, except for docware & total (excl proj D)

Process alignment, standardize & re-use best practices

Controlling Project Performance Using the Project Defect Model 15 March 18, 2005 Ben Linders

Feedback sessions• Frequent, short• At the workplace• All data available (Excel)• Design/test leaders

Show data ask questions form conclusions take needed actions

Feedback sessions enabled earlier conclusions, better acceptance of results, and quick and focused corrective/preventive actions.

Feedback: Collected data delivered to the people that have been doing the work, in orderto support their understanding of the situation athand and help them to take needed actions

Controlling Project Performance Using the Project Defect Model 16 March 18, 2005 Ben Linders

Conclusions Project Defect Model helps projects to:

– Estimate/track defects: Improve product release quality, save time/cost– Design/test progress: Better planning, risk management, decisions

Benefits for R&D– Project portfolio: Dimension project teams/maintenance teams– Product quality: Less maintenance, satisfied customers– Employees: More involved, empowered, motivated

Controlling Project Performance Using the Project Defect Model 17 March 18, 2005 Ben Linders

Further reading References

– Managing the software process. Watts Humphrey. – Metrics and models in Software Quality Engineering. Stephen H. Kan.

Papers (see also PSQT conference paper!)– Controlling Product Quality During Development with a Defect Model, in

Proceedings ESEPG 2003– Make what’s counted count, in Better Software magazine march 2004– Measuring Defects to Control product Quality, in Measure! Knowledge!

Action! The NESMA anniversary book. Oct 2004. ISBN: 90-76258-18-X– A Proactive Attitude Towards Quality: The Project Defect Model,

Software Quality Professional Dec 2004 (with Hans Sassenburg)

Ben Linders, Ericsson R&D, The Netherlandsben.linders@ericsson.com, +31 161 24 9885