Using Metrics to Reduce Cost of Re-work

Dwight LamppertSenior Test Manager

Franklin Templeton

2

Opening Question

“What is the job of testing?”

3

Potential Response

“The job of testing is to find defects.”

4

Myth Exposed

Myth: The Job of Testing Is to Find Defects1

1The Seven Principles of Lean Software Development by Mary Poppendieck

5

“Quality is everyone’s responsibility”

Myth: The Job of Testing Is to Find Defects

“The job of tests, and the people that develop and run tests, is to prevent defects, not find them. A quality assurance organization should champion processes that build quality into the code from the start rather than test quality in later.”1

1The Seven Principles of Lean Software Development by Mary Poppendieck

6

At a recent software seminar

“The common attributes of successful software projects are

Heroes Chaos Re-work Untracked OT”

7

Today’s Discussion

Case Study – 4 projects2 DefinitionsTesting Metrics (what to collect &

analyze)Metrics from first 2 projects Process Improvement StepsSuccess Story – Projects 3 & 4Some trends & conclusions

8

Project Background

Legacy mature system (mainframe) Industry regulatory enhancements Time-boxed: 3 to 4 months Waterfall SDLC

First 2 Projects Project 1 – Dec09 – 1X Project 2 – June10 – 2.3X

9

Definition: Re-work

Re-work – AD time to fix and testing time to report and re-test System Testing and Production defects

10

Definition: Static Testing

Static Testing - Testing of a component or system at specification or implementation level without execution of that software, e.g., reviews or static code analysis.

11

Static vs. Dynamic Testing

From System Testing with an Attitude, p.14, by Nathan Petschenik

12

Testing Defect Analysis

When was the defect introduced (SDLC phase)

When was the defect discovered (testing phase)

13

Points of Origin for Defects2

1. Requirements2. Design3. Code4. User documentsUser documents5. Bad fixes (new defects due to repairs of

older defects)

2Software Engineering Best Practices: Lessons from Successful Projects in the Top Companies by Capers Jones

14

Defect Discovery Phases

1. Static Testing Document walk-throughs & reviews Pre-testing Test Plan review All before Build is complete

2. System Testing

3. Production

15

Tracking the Cost of Re-work

1. Defect Fix Time

2. Re-test Time

COR (Cost of Re-work) – calculated by multiplying time x rate – cost per defect

16

Testing Rework Workflow

Re-work time during Test Execution is tracked as 2 Components

1. Time to Fix a Defect

2. Time to Re-test Defect Fixes

17

Project 1 - Defect Fix Cost

Defect Fix Cost

$0 $21 $49

$406

$1,735

$0

$500

$1,000

$1,500

$2,000

Requirements Design Build System Test Transition

SDLC Phase of Defect Discovery

18

Issues from first 2 projects

1. High % of project = Cost of Re-work

19

COR = 23 to 29% Project

Cost of Rework - % of Total Project

29%

23%

0%5%

10%15%20%25%30%35%

1 2 3 4

Project

% o

f R

ewo

rk

COR - % of Project

20

Issues from first 2 projects

1. High % of project = Cost of Re-work

2. High % of Requirements & Design (R&D) defects found later

21

20-30% R&D Defects found later

% R & D Defects found later

29%

21%

0%

5%

10%

15%

20%

25%

30%

35%

1 2 3 4

Project

%

% R & D Defectsfound later

22

Issues from first 2 projects

1. High % of project = Cost of Re-work

2. High % of Requirements & Design (R&D) defects found later

3. DDP (Defect Discovery %) – slightly better than average

23

Defect Discovery Percentage (DDP)

DDP = % of Total Defects discovered before implementation

Provides a metric to evaluate the overall effectiveness of testing.

90% - Good95% - Very Good

24

DDP = average +

Defect Detection %

89.288.2

80

85

90

95

100

1 2 3 4

Project

% DDP%

25

Process Improvement Steps

1. Developed framework in Quality Center to track COR

2. Presentations to business partners & executives

3. Lively discussion during presentations4. AD appointed a new AD Lead5. AD improved their internal code review6. More emphasis on and participation

during reviews, walk-throughs – Static Testing

26

Process Improvement Steps

7. More collaboration between AD and testing teams

8. Increased emphasis on unit testing

9. Risk-based testing approach

10. Improved regression work

11. Staff increased in domain knowledge

27

Projects 3 & 4 – Process Improved

1. Project 1 – Dec09 – 1x

2. Project 2 – June10 – 2.3x

3. Project 3 – May11 – 3.4x

4. Project 4 – June11 – 2.3x

28

Static Testing emphasized

Defects Found During Static Testing

10

92

339

0

50

100

1 2 3 4

Project

# of

Def

ects

Static Testing Defects

29

System Testing Defects

Defects Found During System Testing

12

23

3

8

05

1015

2025

1 2 3 4

Project

# of

Def

ects

System TestingDefects

30

% R&D Defects found later

% R & D Defects found later

7%

29%

21%

2%

0%

5%

10%

15%

20%

25%

30%

35%

1 2 3 4

Project

%

% R & D Defectsfound later

31

DDP (Defect Discovery %)

Defect Detection %

89.2%88.2%

98.1%

95.5%

80%

85%

90%

95%

100%

1 2 3 4

Project

% DDP%

32

COR as % Project Cost

Cost of Rework - % of Total Project

12%

29%

23%

6%0%5%

10%15%20%25%30%35%

1 2 3 4

Project

% o

f R

ewo

rk

COR - % of Project

33

Defect Discovery Rate

DDR = % of Total Defects discovered during an SDLC or Test Phase

Static Testing System Testing TransitionGoal - Trying to find out if we are

finding defects earlier.

34

DDR Trends

17.6

70.5

11.7

27

62.1

10.8

89.3

7.8

1.9

88.6

6.84.5

0

10

20

30

40

50

60

70

80

90

100

Static System Production

----------------C

od

ing

-----------------

As DDR for Static Testing

increases, DDR for System Testing and Production

Defects decreases

35

When is Unit Testing complete?

“A necessary criterion for completion of developer testing is that the developers believe that the software is ready to be delivered to the users on the day it is delivered to the system-test team.”

From System Testing with an Attitude, p. 33, by Nathan Petschenik

36

Today’s Discussion

Defect metrics – powerful storyStatic Testing worksIt is a collaborative effortThe positive effect ripples into Unit

Testing, System Testing, UAT, and implementation

No shortcuts here – influence your project to do it right

37

Some Final Thoughts

Training needed for our staff (testing, AD, business, PM community) - how to be more effective in document reviews and walk-throughs.

Perspectives Tester – static testing > a paradise (a

new maturity level)

38

Questions & Answers

???