8/20/2009 © Endero Oyj 2005 1

Applying TPI – An

Unusual Case

Kari Kakkonen

Conformiq Software

© Endero Oyj 2005

2

THE INTELLIGENT ART OF INTEGRATION.

Applying TPI - an unusual case

Kari Kakkonen

Director, Testing and Consulting,

Endero Oyj

Dec 1st, 2005 at EuroSTAR

© Endero Oyj 2005

3

BACKGROUND ON PRESENTATION

Based on consulting work done during 2002-2005 and

consequent research paper finished on Feb 2005

Work done at Quality Assurance Consulting Services Unit of

Conformiq Software Ltd

The unit sold to Endero Ltd in June 2005, leaving Conformiq to

focus on model-based testing tools

Presenter personally part of all the assessments leading to

conclusions in this presentation, although in not all the

assessment part of Conformiq / Endero benchmark

© Endero Oyj 2005

4

KEY LEARNING POINTS

How to apply Test Process Improvement (TPI) to unusual

concepts without sacrificing the comparability to other TPI-scores

What kind of improvements to find in small companies (cases)

Multiple additions to TPI questionnaires for gaining more

information

© Endero Oyj 2005

5

ABSTRACT

TPI is often used with plan-driven processes in big companies.

TPI can also be used meaningfully in more agile companies, as

well as in maintenance testing.

One needs to be able to assess current testing capability in any

environment, identify the improvement needs, compare capability

and progress to own company and other companies, and get this

in concise form to get value out of money.

The session will describe three cases focusing on problematic

areas and their consequent improvements.

A comparison example to other companies will be shown to

pinpoint more opportunities of improvement.

© Endero Oyj 2005

6

NEEDS FOR ANY TEST PROCESS IMPROVEMENT

Objective is to identify

– ’ok’ –areas

– Critical improvement needs

– Improvement needs with biggest gain

– Smaller opportunities

Prerequisites

– Improvement needs to be continuous

– Related to actual work

– Benefit the overall process and business

Assumption

– Individual parts of testing process can be described and developed independently

– Dependencies can be named and also developed independently

© Endero Oyj 2005

7

LEVEL OF EFFORT

Time

Effort

Basic level,

continuous

improvement

Improvement

projects

Tailoring

© Endero Oyj 2005

8

TARGET LEVEL AND TIMETABLE

Time

Quality

Present level

Target level = ?

?Now

© Endero Oyj 2005

9

TPI IN SHORT

Test Process Improvement (TPI®) is a model for improving various testing processes

Developed and documented as a book by Tim Koomen and Martin Pol at Sogeti, the Netherlands

Systematic way to

– Assess software testing processes

– Find improvement opportunities

– Develop software testing step by step

20 key areas get a score based on fulfilling a number of checkpoints

Improvements relate to the score

Conformiq’s QA Consulting Services Unit (now part of Endero) cooperates with Tim Koomen

© Endero Oyj 2005

10

TPI MATURITY MATRIX EXAMPLE

A project

scores 0 to D

on all 20 key

areas

Key areas

with leftmost A

should be

developed

first

© Endero Oyj 2005

11

ASSUMPTIONS IN TPI

TPI-model seems to make some underlying assumptions, which

one can see by

– using TPI with an objective state of mind

– scrutinizing the checkpoints, dependencies and the book

These assumptions manifest themselves in

– Order and content of checkpoints

– Dependencies of key areas

– Improvement suggestions

© Endero Oyj 2005

12

ASSUMPTIONS

The assumptions seem to be that

– A certain order of improvements is always appropriate

– A software development organization always

– Is from medium to large size

– Has a separate, independent testing group

– Is very plan-driven

– Has fixed process phasing

– Has testing personnel with great level of autonomy

© Endero Oyj 2005

13

COMPARABILITY ISSUES

We find that the model can be applied and adjusted in many situations

One must take care of the comparability to other TPI-assessments

– The flexible points in the TPI model are when

– It’s only more or less information

– It doesn’t really matter

– Things that are needed to overcome the assumptions, but don’t alter the score, in the ”spirit of TPI”

– The corner stones that must be honored are especially

– Development order

– Big underlying assumptions such as independent test group

Comparability is needed to

– Utilize TPI fully – use it as intended as much as possible

– Be able to benchmark with other TPI-assessments

© Endero Oyj 2005

14

ASSESSMENT GROUND RULES

Scoring in key areas is only a secondary objective

Finding improvement needs is essential

Improvement goal has to be kept fresh in mind

Information is needed regardless of the way one gets it – ask any

questions necessary

© Endero Oyj 2005

15

SUGGESTIONS ABOUT POSSIBILITIES OF TPI

Take elements or ideas from other assessment models to get fuller picture of the test situation– E.g. if the object is to reduce the risk, one might also ask questions

inspired by Test Improvement Model (TIM)

Apply the context-driven analysis – adjust thinking from single checkpoint to thinking in terms of key area

level– based on the size of the assessed project and the process choices

Complete the big picture of the assessor by additional questions– Assessor experience– Other models, checklists, literature

Recognize both TPI-based and other realistic improvement suggestions

Prioritize the improvement suggestions and the key areas to be improved– By TPI– By additional perspectives

© Endero Oyj 2005

16

APPLYING MODELS

Choice of

models

Project

contextAssessor

expertise

Assessment

objectives

Assessment

with model

Model

suggestions

Applying

assessment

Applying

suggestions

Prioritized

suggestions

© Endero Oyj 2005

17

ADDITIONAL QUESTIONS TO BE USED WITH TPI

– What is included in your normal week of testing work?

– What are your roles in testing?

– How does testing relate to coding

– What is a typical project like, in terms of duration, size, type and resources

– How do you know what to test and when?

– What is the role and meaning of testing?

– How do you plan and prepare testing?

– What kind of build cycle do you have?

– Describe a typical test case?

– Describe skills and shortcomings of testing

© Endero Oyj 2005

18

GENERAL ABOUT CASES

Decision to use (also) something else than plan-driven approach

Small to medium sized companies or projects

– (quite typical in Finland)

Testing is (semi)integrated into the development team

Represent both software applications and embedded software

© Endero Oyj 2005

19

CASE ONE

Agile processes: Scrum

Small company

Embedded and application software

Plan and risk driven testing

Mostly developer with testing responsibilities, some full-time testers on project basis

High level of personal responsibility

(Data partially altered to anonymize)

© Endero Oyj 2005

20

SOME TPI ADJUSTMENTS MADE

Full-time requirement accepted for non-full-timers when no

conflicts of interests reported, enough time when testing

Generally approved daily build cycle replaces test manager

authorization for test environment installations

Meeting cycle is accepted as is, carefully assessing the objectives

of meetings against TPI checkpoints

© Endero Oyj 2005

21

SOME IMPROVEMENT SUGGESTIONS

Score: Half A, some B, Rest on initialized level

– (didn’t get any higher due to adjustments)

Reduce dependability on individuals by increasing (light)

documentation

Reserve (consultant) resources to automate routine testing

Include reviews into process officially (strategy, plans, checklists)

From verbal progress reporting into concise reports

Maintain structured approach on environment although decision

cycle during project is faster

© Endero Oyj 2005

22

CASE TWO

Small company

Embedded, critical software

Non-documented testing techniques

High personnel expertise

Developer has tester duties

Almost zero defects in production

(Data partially altered to anonymize)

© Endero Oyj 2005

23

SOME TPI ADJUSTMENTS MADE

Well-done just-in-time test case design accepted

Email and verbal reporting accepted, the level is essential, not the

means

Not suggesting to document everything

Trying to find out how to become more sustainable and

repeatable, adjusting heavily the order of TPI-based improvement

suggestions

© Endero Oyj 2005

24

SOME IMPROVEMENT SUGGESTIONS

Score: mostly initial level

– (didn’t get any higher due to adjustments)

Create a test strategy, prioritized

Document existing practices with simple checklists

Create a learning capability for new employees

© Endero Oyj 2005

25

CASE THREE

Several small projects in a fairly large company

– Average size 2-3 persons development effort in 1 month

– Separate acceptance testers part timing from their normal duties

Application software with large user base

Quick development cycle requires lighter organization and planning but

quite demanding level of software quality

TPI was appropriate to find improvements although some checkpoints

would automatically fail

Focus on acceptance testing

(Data partially altered to anonymize)

© Endero Oyj 2005

26

SOME TPI ADJUSTMENTS MADE

Full time test manager and tester requirements approved with

principle ”practically full time during project”

Any well-thought written record approved as document

– Level of planning assessed rigorously

Level and quality of communication assessed, not the means

Level of low-level testing assessed based on second-hand

information

© Endero Oyj 2005

27

SOME IMPROVEMENT SUGGESTIONS

Score: level A to B

– (didn’t get any higher due to adjustments)

– not possible to increase some key areas without violating TPI

– Given both prioritized suggestions and ”needed for TPI” suggestions

Testing environment needs a responsible support person

Test automation usage could increase from the level of using test management tools, as there is a centralized expert team in the company

Email communicated decisions should be stored to a central location

Checklists to be used with reviews

Keep the present moderate level of written planning

Test strategy thinking should be encouraged, not to be done only as a part of process

© Endero Oyj 2005

28

SCOPE OF IMPROVEMENT

The w

hole

com

pan

y

Time

Improvement

brainstorm

Ideas

proving

Piloting

ideas in a

project

Prepar

ation

Implementation

in project …x

Implementation

in project d…

Implementation

in project c

Implementation

in project b

Implementation

in project a

Piloting

ideas in

2nd project

Piloting

ideas in 3rd

projectAssessment:

1 project

represents

the company

© Endero Oyj 2005

29

BENCHMARK EXAMPLE

Project

should be

compared to

average

Key Areas Controlled Efficient Optimizing

0 1 2 3 4 5 6 7 8 9 10 11 12 13

Test strategy A B C D

Life-cycle model A B

Moment of involvement A B C D

Estimating and planning A B

Test specification techniques A B

Static test techniques A B

Metrics A B C D

Test automation A B C

Test environment A B C

Office environment A

Commitment and motivation A B C

Test functions and training A B C

Scope of methodology A B C

Communications A B C

Reporting A B C D

Defect management A B C

Testware management A B C D

Test process management A B C

Evaluation A B

Low-level testing A B C

Min, max

Project

Average

© Endero Oyj 2005

30

BENCHMARK VALIDATION EXAMPLE

Key Areas Controlled Efficient Optimizing

0 1 2 3 4 5 6 7 8 9 10 11 12 13

Test strategy A B C D

Life-cycle model A B

Moment of involvement A B C D

Estimating and planning A B

Test specification techniques A B

Static test techniques A B

Metrics A B C D

Test automation A B C

Test environment A B C

Office environment A

Commitment and motivation A B C

Test functions and training A B C

Scope of methodology A B C

Communications A B C

Reporting A B C D

Defect management A B C

Testware management A B C D

Test process management A B C

Evaluation A B

Low-level testing A B C

Min

Int’l avg

Conformiq avg

Int’l max

Conformiq max

Int’l avg

from Sogeti

survey

Conformiq/

Endero avg

from 30+

assess-

ments

© Endero Oyj 2005

31

CONCLUSION

Improvement suggestions can be found, regardless the size or

type of project or company

– TPI-score might not be superb – but that is beside the point

Improvements in the small projects tend to be

– in the same areas as in large projects, but to be achieved with

different means or level of intensity

– prioritized differently

TPI-assessments can be enhanced with

– Context-driven touch to the company

– Ideas from other models

– Additional questions and checklists

© Endero Oyj 2005

32

CONTACT INFORMATION

kari.kakkonen@endero.com

www.endero.com

Endero Ltd

Kiviaidankatu 2 F

00210 Helsinki

Finland

© Endero Oyj 2005

33

ENDERO IN SHORTThe intelligent art of integration.

Software Services Partner for IT-departments and software

companies

Services include software specialist and testing services

Primary market: Finland

Founded: 1994

Traded at Helsinki Stock Exchange since 1999

59 persons (in the end of 2004)

Turnover 5,5 M€ (2004)

Offices: Helsinki