Risk-based Testing: Not for the Fainthearted
14
Risk-based Testing: Not for the Fainthearted George Wilkinson Grove Consultants [email protected] www.grove.co.uk
-
Upload
techwellpresentations -
Category
Technology
-
view
191 -
download
0
description
If you’ve tried to make testing really count, you know that “risk” plays a fundamental part in deciding where to direct your testing efforts and how much testing is enough. Unfortunately, project managers often do not understand or fully appreciate the test team’s view of risk—until it is too late. Is it their problem or is it ours? After spending a year on a challenging project that was set up as purely a risk mitigation exercise, George Wilkinson saw first-hand how risk management can play a vital role in providing focus for our testing activities, and how sometimes we as testers need to improve our communication of those risks to the project stakeholders. George provides a foundation for anyone who is serious about understanding risk and employing risk-based testing on projects. He describes actions and behaviors we should demonstrate to ensure the risks are understood, thus allowing us to be more effective during testing.
Transcript of Risk-based Testing: Not for the Fainthearted
Risk-based Testing: Not for the Fainthearted
George Wilkinson Grove Consultants
[email protected] www.grove.co.uk
Definition: ISO 31000: (2009) Risk Management standard definition, defines risk as:
Risk…a definition
“the effect of uncertainty on objectives whether positive or negative”
A more common project definition of ‘risk’ is:
“the probability of an event, hazard, threat or situation occurring resulting in undesirable consequences”
Testing…the challenges
Test: a definition
Our objectives: • Find defects• Raise confidence• Have conformance to regulation • Combinations of the above
risks are present due to:a) Time pressuresb) System sophistication c) Budget restrictionsd) Changing requirementse) Resource challenges …and on…and on…and on…
In the act of ‘testing’ we need
FOCUS!!!
“a particular process or method for trying or assessing”
The TiNA project
London Civil and Military Airspace
System
Existing FDP system
Replacement FDP system
Why is your system important?
flight numbers are continually growing world-wide
in 2010: over the entire UK 2.1 million flights were handled in controlled airspace
over 200 million passengers flew
do you have a reason why your system is important?
…if not find one!
“The safety of the passengers rely upon
this system”
Understanding Risk
Probability
100% 5
75% 4
50% 3
25% 2
1% 1
Consequence
Negligible
Low
Moderate
ExcessiveSerious
1 2 3 54
Risk
Risk 1 Risk Exposure = 2*2 = 4Risk 2Risk Exposure = 4*2 = 8Risk 3 Risk Exposure = 2*4 = 8Risk 4 Risk Exposure = 4*4 = 16
Risk Exposure = Probability * Consequence
Risk Management
risk identificatio
n
ENGAGE THE STAKEHOLDERS!
• risk workshops• expert interviews• heuristics• lessons learned• checklists
Try to…calculate
Risk Exposure
Investigate
Ignore Project
Mitigation
Test Contin
gency plan
risk analysis
risk mitigation (control)
Risk monitoring
Risk-based Testing…points to note
The main initiative being…
to provide detailed information to help ‘management’ make a decision on whether a system is ready for release
can be hard to commit to…can help answer:
how much testing is enoughis a risky business in itselfdoes not mean test all known risks
Is it them or is it us…?
Number Software Risk item Frequency Frequency1 Misunderstanding of requirements 5
1 Lack of top management support 5
3 Lack of adequate user involvement 4
4 Failure to gain user commitment 3
5 Failure to manage end user expectations 3
6 Changes to requirements 3
7 Lack of an effective project management methodology 3
Source: Top Ten Lists of Software Project Risks: Evidence from Literature Survey by Tharwon Arnuphaptrairong
Project Management is project focussed
Test Management is product focussed
Behaviours in Risk Based Testing #1
build the relationship with Project Managementtest need to work close with Project Managementappreciate Project Management objectivespush Project Management for risk based decisions
learn to use ‘risk language’when verbalising riskswhen writing risks
all ‘risk management’ activities should be efficientact boldly, time is limitedensure risk workshops are facilitated well learn the lessons from past mistakes
Behaviours in Risk Based Testing #2
Chanceof failure
RiskManagement
PlanFor Test
Test and report
retain the ‘risk based approach’in strategy and planningin execution on the next release
….when you are tired
Risk-based reporting
Progress through the test plan
today end date
residual risks of
releasing TODAY
Res
idua
l Ris
ks
start
Source: Risk Based E-Business Testing – Paul Gerrard & Neil Thompson
all risks ‘open’ at the start
Defect data provides a message…choose well
Day 1 Day 2 Day 3 Day 4 Day 5 Day 6 Day 7 Day 8 Day 9 Day 10
0
20
40
60
80
100
120
140
Found
Fixed
Funct
ion 1
Funct
ion 2
Funct
ion 3
Funct
ion 4
Funct
ion 5
Funct
ion 6
Funct
ion 7
Funct
ion 8
Funct
ion 9
Funct
ion 1
00
10
20
30
40
50
60
70
defect density
defects found versus fixed
defect density analysis
Summary
risks in modern systems are prevalent and testing requires focus
risk based testing requires:a good relationship with project managementan efficient risk management approach good communication and reportingdiscipline
success comes with taking risks
