TP Half-day Tutorials

5/6/2014 1:00:00 PM

Root Cause Analysis for

Software Testers

Presented by:

Alon Linetzki

Best-Testing

Brought to you by:

340 Corporate Way, Suite 300, Orange Park, FL 32073

888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com

Alon Linetzki Best-Testing

Alon Linetzki, founder and managing director of Best-Testing, has been a coach and consultant in development, testing, and quality assurance for twenty-eight years. Alon helps organizations enhance the test engineer’s professional and personal skills, training test managers in optimization and test process improvement, and optimizing their test operations. His main areas of expertise are in agile testing and transitioning to agile, exploratory testing, test process Improvement, risk-based testing, and test automation. Alon is a member of the ISTQB® authors and review team for the ISTQB® Agile Tester Add-on certification, cofounder of ISTQB® in Israel, leader of the ISTQB® Partner Program, and founder of the SIGiST Israel conference.

Root Cause Analysis in Testing

Partners in this course are:

Course Partners

March 2014

2

March 20143

The course materials have been prepared by:

Best – Testing,

Copyright: © Best – Testing, 2008

All materials contained in this Handbook were prepared by Best-Testing. All rights of this material are

reserved solely for Best-Testing. The material is intended for personal, noncommercial use. All

materials published in this material are protected by copyright, and owned or controlled by Best-

Testing, or the party credited as the provider of the content. You may not modify, publish, transmit,

participate in the transfer of sale of, reproduce, create new works from, distribute, perform, store on

any magnetic device or any other device, display, on in any way exploit, any of the content in whole

or in part. You may not alter or remove any trademark, copyright or other notice from copies of the

content. You may no use the material in this material for the purpose of training of any kind, internal

or for customers, without beforehand a written approval of Best-Testing.

The Use of this material

The material in this material is designed to assist the student during the course. It does not include all

of the information that will be referred to during the course and should not be regarded as a

replacement for reference manuals or other instructions.

Copyrights

March 20144

Limit of responsibility

Best-Testing invests significant effort in updating this material, however, Best-Testing is not responsible of

any errors or material which may not meet specific requirements. The user alone is responsible for

decisions based on the information contained in this material.

Protected Trademark

In this material, protected trademarks appear that are under copyright. All rights to the trademarks in this

material are reserved to the authors.

Best–Testing wishes you success in the course!

Copyrights

Introduction

March 2014

5

• No bullets…should be fun…

Chapter 1

6

Alon Linetzki

30 years in IT, 20+ years as a test engineer and a test manager

Certified Scrum Master, Scrum Alliance, 2008

ISQTB® Partner Program leader, ISTQB® Agile Tester Certification Leader

Specializes in Agile testing test strategy & optimizationtest process improvement test management test design risk based testing test automation Building Smart Teams

A degree in ‘testing’ – B.Sc. statistics and criminology,

International Speaker worldwide, since 1995

Co-founder of the Israeli Testing Certification Board (www.itcb.org.il, 2004)

Founder of the SIGiST Israel (testing forum) in Israel (www.sigist.org.il, 2000)

We shall cover…

Introduction

Presenting participants and trainer

Workshop expectations

What is Root Cause Analysis?

How to use RCA for Defect analysis?

7

March 2014

We shall cover…

How to Use RCA for analyzing critical problems?

Cause Effect graphing, amended

Case Study #0

Summary

How to use RCA for process improvement?

Case Study #1

Case Study #2

Retrospective

Workshop retrospective

8

March 2014

(*) Bonus topic – only if there is enough time left…

Professional Background

Years experience in testing/development/both/other

Responsibilities in current role

Have you implemented RCA? Can you share?

Workshop expectations?

Introduce participants…

9

March 2014

Interactive sessions

Discussions, skepticism, curiosity

Examples, demonstration

Thinking, communicating

Keeping an open mind for ideas and concepts

Contribute to the class team

Learn…discuss…review…

Give quick feedback on everything you may think of

Time boxed sessions…breaks…exercises…discussions

Introducing workshop dynamics10

March 2014

There are FAR more slides than we can

cover…

Say if you have something to say… do not

wait till the end of the workshop!

Introducing workshop dynamics11

March 2014

Start : 10:15

Lunch: 11:45 – 12:45

Finish line: 14:15

But… we may have more

breaks if we feel like it…

Logistics…

March 2014

12

What Root Cause Analysis?

March 2014

13Chapter 2

Root Cause Analysis definition(My interpretation)

From wiki:

Root cause analysis (RCA) is a class of problem solving

methods aimed at identifying the root causes of problems

or events.

The practice of RCA is predicated on the belief that

problems are best solved by attempting to correct or

eliminate root causes, as opposed to merely addressing

the immediately obvious symptoms.

By directing corrective measures at root causes, it is

hoped that the likelihood of problem recurrence will be

minimized.

14

Root Cause Analysis definition

From Bill Willson’s website:

Root cause analysis (RCA) is a methodology for

finding and correcting the most important reasons

for performance problems.

It differs from troubleshooting and problem-solving

in that these disciplines typically seek solutions to specific difficulties, whereas RCA is directed at

underlying issues

15

Root Cause Analysis - definition

Root cause analysis (RCA) is a methodology for finding

and correcting the most important reasons for

performance problems.

It differs from troubleshooting and problem-solving in

that these disciplines typically seek solutions to

specific difficulties, whereas RCA is directed at

underlying issues.

16

Root Cause Analysis definition -summary

As a business process improvement tool, RCA seeks

out unnecessary constraints as well as inadequate

controls.

In safety and risk management, it looks for both

unrecognized hazards and broken or missing barriers.

It helps target CAPA (corrective action and preventive

action) efforts at the points of most leverage.

RCA is an essential ingredient in pointing

organizational change efforts in the right direction.

Finally, it is probably the only way to find the core

issues contributing to your toughest problems.

17

Root Cause Analysis(My interpretation)

A problem solving method/process designed to search for the root causes of a problem using a predefined structural thinking process, identifying the underlying issues, with the expectation that dealing with these issues will dramatically reduce the likelihood of the problem to occur.

The process involves data collection, cause charting, root cause identification and recommendation generation and implementation

18

“Cows falling on a road from a mountain”

– is it a problem or a symptom?

Should we eliminate all cows on that area?

Should we dig-out the mountain?

Should we rotate the sign?

Should we divert the road elsewhere?

It seems that sometimes eliminating the causes is not an easy task, and finding the problems is even harder!

19

Root causes are underlying causes

The investigator’s goal should be to identify

specific underlying causes

The more specific the investigator can be about

why an event occurred, the easier it will be to

arrive at recommendations that will prevent

recurrence.

Practical RCA – definition guidelines

March 2014

20

Root causes are those that can reasonably be

identified

Occurrence investigations must be cost beneficial

It is not practical to keep valuable manpower

occupied indefinitely searching for the root

causes of occurrences…

Structured RCA helps analysts get the most out of

the time they have invested in the investigation.

Practical RCA – definition guidelines

March 2014

21

Root causes are those over which management has control

Analysts should avoid using general cause classifications such as “operator error”, “equipment failure” or “external factor”

Such causes are not specific enough to allow management to make effective changes

Management needs to know exactly why a failure occurred before action can be taken to prevent recurrence

We must also identify a root cause that management can influence:

Identifying “severe weather” as the root cause of parts not being delivered on time to customers is not appropriate

Severe weather is not controlled by management.

Practical RCA – definition guidelines

March 2014

22

Root causes are those for which effective recommendations can

be generated (if cannot be done, keep on searching!)

Recommendations should directly address the root causes

identified during the investigation

If the analysts arrive at vague recommendations such as,

“Improve adherence to written policies and procedures,” then

they probably have not found a basic and specific enough

cause and need to expend more effort in the analysis process.

Practical RCA – definition guidelines

March 2014

23

Root causes are underlying causes

Root causes are those that can reasonably be

identified

Root causes are those over which

management has control

Root causes are those for which effective

recommendations can be generated (if

cannot be done, keep on searching!)

Practical RCA – definition guidelines

March 2014

24

Time for discussion…25

March 2014

Root Cause Analysis for Beginners - by James J. Rooney and Lee N. Vanden Heuvel

References – what is Root Cause Analysis?26

March 2014

How to use RCA in Defect Analysis?

March 2014

27

Escaping defects occur all the time

We try to contain them, from going to production,

but much efforts should be done in preventing

them going from Design to Code

A set of questions may help out leading us to Data

Collection on those Defects for performing the

RCA – next page…

Defects RCA introduction

March 2014

28

Defects RCA guided questions

Data Collection

March 2014

29

Question Who

asks?

Describe circumstances and consequence of the

Problem?

Test Team

Where was the problem introduced? Dev Team

Describe the root cause of the problem Dev Team

What reviews were held for the phase? Dev Team

How and why did the problem escape testing? Test Team

What could be done to prevent this type of error in the

future?

Both

What could be done to find this type of problem in the

future?

Both

John Ruberto – Root Cause for Problems Escape

Defect Analysis is the process of analyzing a defect

to determine its root cause.

Defect Prevention is the process of addressing

root causes of defects to prevent their future

occurrence.

RCA – few definitions

March 2014

30

Defect Source – Finding defects in the stage they

were introduced and as early in the lifecycle as

possible – Eliminating escaping defects

RCA – few definitions

March 2014

31

Defects Root Cause Analysis 32

Canceled Defects Root Cause Analysis

Cancelled defects are not real defects of the

system-under-test

They can be the result of: environment problem

(non product), out of scope, test design or test

execution problem, un-reproducible defect, not

understanding the specifications, etc.

33

Canceled Defects Root Cause Analysis34

Code Inspection - Example

March 2014

35

Background:

180 people on the project

3,700 modules interconnecting with each other

Complex infrastructure

25 testers

Using Test Director/Quality Center as reporting

platform

Strategic customer

Code Inspection - Example

March 2014

36

Code Inspection - Example

March 2014

37

Analysis:

Main concern – Complience with design (10%

Medium+)

Investigated RCA:

A few specs were written by same designer

A few modules were written by 3 programmers

Initiated training for designer and programmers

Code Inspection - Example

March 2014

38

Background:

Another project,

450 people

working, 55 testers,

4,300 modules,

strategic customer

Focus on:

Performance

Maintainability

Proper algorithm

usage

Code Inspection - Example

March 2014

39

Analysis:

Investigated RCA:

Performance defects were on resource usage, memory

leaks

Implementing improper algorithm was mainly due to

specification documents written unclear + implementation

complexity chosen by a few programmers

Initiated training for architects and programmers

Adding defect template fields:

Injected, Detected, Removed

Version [build] + Date per field

[Updating the defect lifecycle process (status)]

Changing field CRUD permissions – Dev, Test, Managers

Defining reports

Updating the Defect Management process

Training the teams

Implementing action plan…

Implementation guidelines –

defects RCA

March 2014

40

RCA process defined

March 2014

41

Time for discussion…42

March 2014

John Ruberto – Root Cause for Problems Escape,

http://blog.ruberto.com/2008/04/another-post/

References

March 2014

43

How to Use RCA for analyzing critical

problems?

March 2014

44

Challenges the current method could

not solve – using 5whys

45

‘5ys’ or ‘5 whys’ technique, and the cause-effect diagram.

Presenting a problem,

Asking “why?” it happens, finding the effect that caused it

(1 effect),

Presenting the effect on the diagram,

Asking “why?” it happens… [back to previous step, unless

we ask it for 5 times already]

Done.

Presenting the 5whys Technique46

‘5ys’ or ‘5 whys’ technique, and the cause-effect

diagram.

Presenting the RCA Technique47

CauseCause Cause Cause

Cause

Problem

Why

#1

Why

#2Why

#3Why

#4

Why

#5

Thinking path…

‘5ys’ or ‘5 whys’ technique, and the cause-effect diagram.

1. There is the assumption that a single cause, at each level

of "why", is sufficient to explain the effect in question.

2. What if one of the ‘Why’ is answered wrongly? Maybe

our answer is possible, but what if the actual cause is

something else entirely?

3. When we have found the problem, and draw the route,

how ‘strong’ is this solution? Maybe we should prefer one

over the other?

Challenges: what the method can

not solve48

Enhancing the method – case study49

Short structured interview with rep’s of management, development, release, system, testing, product teams.

Step 1: Draw a cause-effect diagram & exercise the 5whys

Step 2: Investigate the arrows/causes for:

What evidence you have that the cause exist? (relevancy)

What evidence you have that the cause leads to the effect? (strength)

Is anything else needed together with the cause for the effect to occur? (strength)

Is there evidence that the cause is contributing to the problem I’m looking at? How much it contributes? (Impact: direct/indirect)

Enhancing the Method:

Example project50

Enhancing the Method:

Example project51

Type Question Cause-Effect

Relevancy What evidence you have that the cause exist? H/M/L

Strength

(S or W)

What evidence you have that the cause leads to

the effect?

H/M/L

Strength

(S or W)

Is anything else needed, together with the

cause, for the effect to occur?

Yes/No

Impact

(D or I)

Is there a evidence that the cause is contributing

to the problem I’m looking at?

Yes / No

Impact

(D or I)

How much this cause is contributing to a

possible resolution?

Direct /

Indirect

Mark

Enhancing the Method:

Example project52

Type Question Cause-Effect

Relevancy What evidence you have that the cause exist? High (3)

Strength

(S or W)

What evidence you have that the cause leads to

the effect?

Medium (2)

Strength

(S or W)

Is anything else needed, together with the

cause, for the effect to occur?

Yes (1)

Impact

(D or I)

Is there a evidence that the cause is contributing

to the problem I’m looking at?

Yes (1)

Impact How much this cause is contributing to a

possible resolution?

Direct (2)

Mark 9

You should mark each arrow using this table.

Step 3: Identify the routes leading to the

problem/s,

Step 4: Identify the strength and direction (impact)

they have (calculating the mark for each arrow),

Step 5: Choose the best route to focus on,

[Improve it, and go to the next one].

Enhancing the Method:

Example project53

Case Study - implementation54

Background:

company was using a very advanced technology, and

a complex product line,

Complex product, uses mechanics, electronics,

hardware, software, devices, cooling device, has water

resistant, has heating resistant, accurate up to

1:1,000,000 cm,

In the last 0.5 year, 50% of released machines

returned from the floor (clients) for fixing,

Example project – Hi-Tech Company55

SQA manager was at a course I gave, and liked

one of the tools,

He thought automation can solve many of his

problems, because:

A lot more tests running,

Identifying more defects before the clients do,

Less products coming back,

Clients are happy!

Example project – Hi-Tech Company56

I investigated their automation

needs,

Followed the steps of the

enhanced method,

Found out their problems might

be elsewhere…

Example project – Hi-Tech Company57

Lets see the drawing board from

that meeting…

1st drawing – RCA meeting

58

Our way of thinking12

The RCA meeting (company exec’s and directors):

At first, the belief was that the primary problem

was:

Partial Test Planning (less tests are executed)

Example project – Hi-Tech Company59

Lets see an illustration diagram …

1st drawing – RCA meeting60

Many clients

ask for

different Sw

of the

product

Many

versions

open in

parallel

Complexity of

version control

management is

very high

Defining req’

not good

enough by

client

Spec Lvl 0

No specs

in lvl 1

Spec Lvl

1 not

complete

or does

not fit

Spec Lvl

2 not

written

Good

definition of

Spec Lvl 0

Spec Lvl

1 fits

Spec Lvl

2 fit

Spec Lvl 2

does not

fit/complete

Code

written

with low

match to

client req’

Only

Partial Test

planning

and not full

coverage

Partial test

case planning

and coverage

Partial test

execution

and low

coverage

Our way of thinking1

2

1st drawing – RCA meeting61

Many clients

ask for

different Sw

of the

product

Many

versions

open in

parallel

Complexity of

version control

management is

very high

Defining req’

not good

enough by

client

Spec Lvl 0

No specs

in lvl 1

Spec Lvl

1 not

complete

or does

not fit

Spec Lvl

2 not

written

Good

definition of

Spec Lvl 0

Spec Lvl

1 fits

Spec Lvl

2 fit

Spec Lvl 2

does not

fit/complete

Code

written

with low

match to

client req’

Only

Partial Test

planning

and not full

coverage

Partial test

case planning

and coverage

Partial test

execution

and low

coverage

Our way of thinking1

2

1st drawing – RCA meeting62

Many clients

ask for

different Sw

of the

product

Many

versions

open in

parallel

Complexity of

version control

management is

very high

Defining req’

not good

enough by

client

Spec Lvl 0

No specs

in lvl 1

Spec Lvl

1 not

complete

or does

not fit

Spec Lvl

2 not

written

Good

definition of

Spec Lvl 0

Spec Lvl

1 fits

Spec Lvl

2 fit

Spec Lvl 2

does not

fit/complete

Code

written

with low

match to

client req’

Only

Partial Test

planning

and not full

coverage

Partial test

case planning

and coverage

Partial test

execution

and low

coverage

Our way of thinking1

2

2nd drawing – RCA meeting63

Many clients

ask for

different Sw

of the

product

Many

versions

open in

parallel

Complexity of

version control

management is

very high

Defining req’

not good

enough by

client

Spec Lvl 0

No specs

in lvl 1

Spec Lvl

1 not

complete

or does

not fit

Spec Lvl

2 not

written

Good

definition of

Spec Lvl 0

Spec Lvl

1 fits

Spec Lvl

2 fit

Spec Lvl 2

does not

fit/complete

Code

written

with low

match to

client req’

Only

Partial Test

planning

and not full

coverage

Partial test

case planning

and coverage

Partial test

execution

and low

coverage

Our way of thinking1 2

After a while, we shifted the focus and agreed that

the real problem was actually:

Poor Product Quality

Because that was the reason the clients returned

their product.

And we started RCA from there.

After a while, we started to see the light – real

problems started to crystallize, problems that

involved people and processes

Example project – Hi-Tech Company64

3rd drawing – RCA meeting

65

Our way of thinking

3rd drawing – RCA meeting66

Many clients

ask for

different Sw

of the

product

Many

versions

open in

parallel

SCM - Complexity

of version control

management is very

high

Defining req’ not

good enough by

client

Spec Lvl 0

No

specs in

lvl 1

Spec Lvl 1

not complete

or does not

fit

Spec Lvl

2 not

written

Good

definition

of Spec

Lvl 0

Spec Lvl

1 fits

Spec Lvl

2 fit

Spec Lvl 2

does not

fit/complete

Code written

with low

match to

client req’

Only

Partial Test

planning

and not full

coverage

Partial test

case

planning and

coverage

Partial test

execution

and low

coverage

1 2

Tight

schedule

projectPrioritization

and

compromise

on scope to

clients

Low

Quality

Product

Req’

managemen

t not good

enoughLack of methods

and techniques

in testing

Low lvl of test

identification

We then defined the relevancy, strength and

impact of each arrow (cause),

And calculated the grades for the arrows (which

are not seen here),

Example project – Hi-Tech Company67

Back to the board…

4th drawing – RCA meeting

68

5th drawing – RCA meeting69

Many

clients ask

for

different

Sw of the

product

Many

versions

open in

parallel

SCM - Complexity of version control

management is very high

Defining

req’ not

good

enough by

client

Spec Lvl 0

No specs in

lvl 1

Spec Lvl 1

not

complete

or does not

fit

Spec Lvl 2

not written

Good

definition

of Spec Lvl

0

Spec Lvl 1

fits

Spec Lvl 2

fit

Spec Lvl 2

does not

fit/complete

Code

written

with low

match to

client req’

Only

Partial Test

planning

and not

full

coverage

Partial test

case

planning

and

coverage

Partial test

execution

and low

coverage

Our way of thinking12

Tight

schedule

project

Prioritization

and

compromise on

scope to clients

Low

Quality

Product

Req’

management

not good

enoughLack of methods and

techniques in testing

Low lvl of test

identification

S/D

W/D

W/I

S/I

We went back to double check the RCA of the

routes leading to the primary problem, marking

the arrows with their grades (from the table,

remember?)

We ended up circling the main causes, that have

initiated the strongest routes that are directly

impacting our problem,

Example project – Hi-Tech Company70

6th drawing – RCA meeting

71

Last drawing – RCA meeting72

Many

clients ask

for

different

Sw of the

product

Many

versions

open in

parallel

SCM - Complexity of version control

management is very high

Defining

req’ not

good

enough by

client

Spec Lvl 0

No specs in

lvl 1

Spec Lvl 1

not

complete

or does not

fit

Spec Lvl 2

not written

Good

definition

of Spec Lvl

0

Spec Lvl 1

fits

Spec Lvl 2

fit

Spec Lvl 2

does not

fit/complete

Code

written

with low

match to

client req’

Only

Partial Test

planning

and not

full

coverage

Partial test

case

planning

and

coverage

Partial test

execution

and low

coverage

Our way of thinking1

2

Tight

schedule

project

Prioritization

and

compromise on

scope to clients

Low

Quality

Product

Req’

manageme

nt not

good

enoughLack of methods and

techniques in testing

Low lvl of test

identification

S/D

W/D

W/I

S/I

Last drawing – RCA meeting73

Many

clients ask

for

different

Sw of the

product

Many

versions

open in

parallel

SCM - Complexity of version control

management is very high

Defining

req’ not

good

enough by

client

Spec Lvl 0

No specs in

lvl 1

Spec Lvl 1

not

complete

or does not

fit

Spec Lvl 2

not written

Good

definition

of Spec Lvl

0

Spec Lvl 1

fits

Spec Lvl 2

fit

Spec Lvl 2

does not

fit/complete

Code

written

with low

match to

client req’

Only

Partial Test

planning

and not

full

coverage

Partial test

case

planning

and

coverage

Partial test

execution

and low

coverage

Our way of thinking1

2

Tight

schedule

project

Prioritization

and

compromise on

scope to clients

Low

Quality

Product

Req’

manageme

nt not

good

enoughLack of methods and

techniques in testing

Low lvl of test

identification

S/D

W/D

W/I

S/I

74

4/6

Last drawing – RCA meetingLets see the routes…

3/3

5 major Root Topics were Identified, explained and prioritized:

1. Produce requirements from client definitions

2. Requirements management

3. Either ‘No Spec Level 1’, or ‘Spec level 1 not matching requirements’

4. Lack of methods and techniques in testing for development and testing teams

5. Allot of clients define slightly different requirement for the SW – allot of specials

We defined a pragmatic corrective actions plan, with priority items.

Example project – Hi-Tech Company75

Major Areas of Concern identified and prioritized:

1. Requirements Management

2. Configuration Management

3. Design Documentation and Flow

4. Testing Methodologies, techniques and tools

Not discussed:

- Release Management

- Risk Management + Risk Based Testing

- Requirements Definition

- Project Management

- Professional Development

Example project – Hi-Tech

Company76

Organization Language!

Differentiating problems from

symptoms

77

If we follow the questioning and grading process, than problems shall be the ‘leafs’ of the cause-effect diagram, with:

A positive answer to the questions:

Relevancy – ‘yes’

Highest strength mark (direct/Indirect), and

Direct impact on the undesired outcome (initial hypothesis)

Highest grade per route guideline,

Translate into company language (areas of concern)

Enables an Immediate vs. long term action plan

Differentiating Problems from

Symptoms78

Just what our office needs..

Solving problems rather than

covering symptoms

79

Cause-effect & RCA may eliminate the problems of the model

when answering the questions and marking them on the diagram

for the best route(s),

(but) We must deal with symptoms some time, in the short term,

We should estimate effort needed (or money required) for each

route, and integrate that into our decision (i.e. 60/40 weight)

The questions (answers) make sure that our analysis is with

minimal deviations, and that the route we take is a ‘strongest’ one,

It is a constructive process that leads to Understanding, Clarity,

Focus and the right Priority setting

Summary 80

Further enhancing the mode, we must think of the following:

What about the junctions points (inbound and outbound):

direct impact of routes with those? Indirect? Impact on speed of

performance (bottle-necks)?

What is the ROI of this method within context?

Can we validate a route? Can we tie it to be a successful

problem eliminator?

How much the method is context dependant?

Can we hook it to Test Process Improvement methods or other

Key Performance/Area Indicators?

Other?

Food for Thought…81

Time for discussion…82

March 2014

RCA and Process Improvement?

March 2014

83

Introduction

Case Study #1

Case Study #2

Process improvement should use data and RCA

outputs

CI example… - process was changed, training was

done to relevant people

Defects example… - lessons learned, training was done

to relevant testers, discussions with development

made it clear which information is required more on

the defect template

RCA and Process Improvement?

March 2014

84

Analyzing test environments

utilization downtime

March 2014

85

Analysis:

Small, yet impacting, breaks in work, caused huge

impact on the release (big team of testers)

Breaks were 15-45 min each

Areas of impact were: performance, network, DB

Analyzing test environments

utilization downtime

March 2014

86

Improved process:

Defined measurements for monitoring downtime

of test environments

Identify trend of areas of impact on infrastructure

Used it weekly to establish Status of Downtime, to

decrease technical impacts (reached -25% waste

on next release).

Analyzing test environments

utilization downtime

March 2014

87

Analyzing development effort

with defects found

March 2014

88

15

26

11

120

65

43

32

187

12

38

210

294

588

2184

504

210

126

1344

336378

80%

99%

21%

62%

145%

230%

286%

156%

40%

113%

0%

50%

100%

150%

200%

250%

300%

1

50

SC MAF FBF CSM E-Care E-Support Rater AR Reports BL Interface

Investment vs Defects

Defects Dev Effort Def % vs Dev Eff %

~28 years development.

~150 people in development

team.

Analysis:

Areas of defects clustering mainly in: Rater, E-Support, E-Care, AR

Relative % of defects are much higher than % of development effort (days)

Investigated UT and Staging (integration test) phases, found missing testing types done, UT & INT regression testing, performance testing, interface testing

Further considerations: Complexity, New/Legacy code, # people touching code

Analyzing development effort

with defects found

March 2014

89

Improved process:

Defined Unit test and Integration test process and

strategy guidelines and work instructions

Trained programmers in UT and INT techniques and

new processes

Aligned effort estimation and duration with project

management and product management

Defined coverage criteria for UT and INT

Initiated test automation project on UT and INT levels

Analyzing development effort

with defects found

March 2014

90

Time for discussion…91

March 2014

Root Cause Analysis techniques can help us in

finding the underlying problems (root causes), and

get to deal with real problems

Structured RCA methods support productive

thinking, identifying problems and identify more

symptoms

Cause Effect Graphing (amended), Defects RCA

RCA and process improvement

Summary

Retrospective

March 2014

92

“It is not the strongest of the species that survives, nor the most intelligent but the one that is most responsive to change”

Charles Darwin

A changing world…93

Or perhaps . . .

94

. . . the one who had anticipated all possible

requirements !

Root Cause Analysis in Testing