Risk-Based Testing for Agile Projects

© 2011 Improve Quality Services BV PRISMA tutorial- 1

1

Risk-Based Testing for Agile Projects

Erik van Veenendaal

www.erikvanveendaal.nl

PRISMA®

Erik van VeenendaalErik van Veenendaal

� Founder Improve Quality Services BV

� In testing since 1989 working for many

different clients and in many different roles

� Author “TMap”, “ISTQB Foundation” and

“TMMi” and many other books and papers

� Former Vice-President International Software

Testing Qualifications Board (ISTQB)

� Supporting member IREB board

� Keynote speaker, e.g., EuroSTAR, STAR

� Winner of the European Testing Excellence

Award

Improve IT Services BV 2

www. erikvanveenendaal.nl


Testing ChallengesTesting Challenges

� Increasing importance and size of

software in society as a whole− e.g., amount of software in consumer product

doubles every 24 months,

− number of defects is almost constant

� High Competition & Outsourcing− Time-To-Market, Product Quality, Price Levels are essential for

business success

� Testing often takes 30 - 40% of project costs

�Risk-based testing deals with both effectiveness

and efficiency


AgileAgile ≠ ≠ SilverSilver BullitBullit


1

10

100

1000

10000

10 100 1000

Project Size (KLOC)

Defe

cts

“despite continuous growth in adoption of agile, many organizations are still facing the

the same challenges as in previous years”, World Quality Report, 2014



What is Risk?What is Risk?

� “A factor that could result in a future negative

consequence; usually expressed as impact and

likelihood” (ISTQB Glossary)

�Testers ‘only’ have the responsibility to identify the

risks and provide information on their status

� “to dare to undertake”

− management attitude and styleDD..

Testing = Risk ManagementTesting = Risk Management

� Objective: most feasible coverage

− effective usage of limited resources

− optimize test effort

� Limited resources

�staffing

�infrastructure

�time !

�..

� The right level and type of coverage on the right

parts at the right time

� Always applicable, whether it is V-model, Agile or ....



Our Challenge,.Our Challenge,.


if if only we only we

knew !!knew !!

RiskRisk--Based TestingBased Testing

�Risk identification looks at ways of

establishing what the risks are and where

they are

�Risk analysis looks into the critical, complex

and potential error prone areas

�Then we build tests to mitigate the risk

�Subsequently we monitor and report

regarding the risks



Based on Practical ExperiencesBased on Practical Experiences


9

Risk IdentificationRisk Identification

� Split up test basis in functional or technical items

� V-model: based on requirements

� Agile: backlog items / user stories for next sprint

� Can also be based on a brainstorm session

� Maximum number of appr. 35 risk items


Risk item 1 Functionality

Risk item 2 Security

Risk item 3 Functionality

Risk item 4 Interoperability


Risk AnalysesRisk Analyses

� Risk = impact x likelihood

− What is the impact for the business ?

− What is the likelihood that there are defects ?

� Determine factors based on previous projects, e.g.,

defect patterns


Impact – business risk

Lik

eli

ho

od

tec

hn

ica

l ri

sk

You already know this !

Exercise: Risk Factors


Factors From PracticeFactors From Practice

� Likelihood

− complexity

− new development

(level of re-uses)

− interfacing

− size

− technology

− geographical spread

− inexperience

(of development team)

� Impact

− business importance

(“selling item”)

− financial (or other)

damage (e.g. safety)

− usage intensity

− external visibility

− cost of rework

Customization

required

defect patterns / history


Risk Assessment in a Nutshell (1)Risk Assessment in a Nutshell (1)

1) Stakeholder analysis: Assigning Factors

2) Individual scoring

Business

importance

Usage

intensity

Safety

Item 1

Item 2

Item 3

Item 4

Item 5

5

5

4

5

4

5

4

5

2

1

they shall

make

choices


3) Consensus meeting

4) Position the risk items in a Product Risk Matrix


Likelihood Impact

Complex

ity

New

develo

pment

Interfacin

g

Tech

nology

Experien

ce level

Busin

ess import.

Usag

e inten

sity

Safety

Item 1 5 3 2 1 5 16 5 4 1 10

Item 2 2 1 2 1 2 8 3 3 1 7

Item n

Risk Assessment in a Nutshell (2)Risk Assessment in a Nutshell (2)


Product Risk MatrixProduct Risk Matrix


IIII

IVIV

II

IIIIII

L

H

M

L HM

xx

x

x

x

Likelihood

Impact

Must Test

Should Test

Could Test

“Won’t Test”

focus of

unit testing

focus of

acceptance

testing

MoSCoW priorities

AgileAgile -- Play the Card Game: Risk PokerPlay the Card Game: Risk Poker

� Poker Planning / Wide Band Delphi based




Wide Band Delphi / Poker Planning (1)Wide Band Delphi / Poker Planning (1)

� Bottom-up estimation techniques- wide variance in estimates

- commitment of people involved

- accurate estimation we use the collective wisdom of the team

� Each estimator has cards with valid estimates

� A user story (product backlog item) is discussed

� Each estimator selects an estimate

� Cards are turned over

� Discuss differences (especially outliers)

� Repeat until consensus

17


User Story

Explain

Estimates

Collect

Share info

OK?

Final

estimate

Moderator Estimator

OK?

Wide Band Delphi / Poker Planning (2)Wide Band Delphi / Poker Planning (2)

18



Risk Poker the same, but,Risk Poker the same, but,

� Scoring separately for likelihood and impact

� Discussion shall be factor based

� Limited number of cards for each value (color)

� Agreement, Product owner (impact) and lead-

developer (likelihood) act as tie breaker

19

20

Example User Story

As as bank account holder, I want a yearly financial

overview of my accounts, so that I can submit my taxes

Impact ?

Likelihood ?

Let’s PlayLet’s Play

Impact:

Improve IT Services BV

Likelihood:



AgileAgile Product Product RiskRisk MatrixMatrix

example

User Story

See any “issues” with this?


Differentiated Test Approach !!Differentiated Test Approach !!

� Test design

techniques

� Reviews & inspection

� Test design start-up

meetings

� Reviews of test design

� Level of detail of test

cases

� Exit (DoD) criteria,

e.g., coverage

� Level of independence

� More time & effort

� Most experienced

person

� Priority setting

� Regression testing

� Re-testing

without this a risk assessment

doesn’t make much sense !!



RealReal--LifeLife AgileAgile ExampleExample

IIII

IVIV

II

IIIIII

LL

HH

MM

LL HMM

xx

x

x

x

Likelihood

Impact

Must Test

Should Test

Could Test

“Won’t Test”

Test design techniquesReview designSupport module testsReview module tests

Test design techniquesReview design

Exploratory TestingSupport module testsReview module tests

Exploratory Testing

AgileAgile ““OneOne Page Test Plan”Page Test Plan”


Must Test

D.. Test Approach D..

Should test

DD Test Approach D..

Could Test


Would Test


Shall include

a definition of “done”

A 1-2 hour session

to achieve a one page

Sprint Test Plan


CommunicationCommunication


Product Risk Matrix Product Risk Matrix –– InitialInitial

US

58

US

31US

49

US

27

Impact - Business risk

Likelihood-TechnicalRisk

IVIII

III

26Improve IT Services BV


Product Risk Matrix Product Risk Matrix –– ProgressProgress

US

58

US

31US

49

US

27

Impact - Business risk

Likelihood-TechnicalRisk

IVIII

III

27Improve IT Services BV


Key learning pointsKey learning points

��A structured and A structured and practical approachpractical approach for for

riskrisk--based based testing is testing is availableavailable

��Make it Make it AgileAgile, but use the formal rules, but use the formal rules

��Efficient, Effective and AttractiveEfficient, Effective and Attractive

��Define Define a a riskrisk--based based differentiated test differentiated test

approachapproach

��Practice riskPractice risk--based based trackingtracking & “& “reporting”reporting”


DownloadsDownloads,,

Go to www.erikvanveenendaal.nl for:

� A full PRISMA white paper (e-book)

� Paper: 10 Recommendations on

Risk-Based Testing

� Paper: Product Risk Assessments in Agile Projects

(Risk-Poker)

� To acquire the PRISMA freeware tool sent a

request to [email protected]

� Book “Practical Risk-Based Testing; The PRISMA

Approach”, ISBN9-78490-986070 (www.utn.nl)Improve IT Services BV 29


www.erikvanveenendaal.nl

[email protected]

@ErikvVeenendaal


,,


Risk-Based Testing for Agile Projects

Software

Transcript of Risk-Based Testing for Agile Projects