Why Do We Bother With Test Strategies
-
Upload
anurag-khode -
Category
Documents
-
view
217 -
download
0
Transcript of Why Do We Bother With Test Strategies
-
8/9/2019 Why Do We Bother With Test Strategies
1/34
Listen | Challenge | Understand | Interpret | Create
Private & Confidential Experimentus Ltd 85 Tottenham Court Road London W1T 4TQ T: +44 (0)870 770 6099 www.experimentus.com
Why do we bother with Test Strategies?
Presented by:
Geoff ThompsonSIGiST9th December 2008
-
8/9/2019 Why Do We Bother With Test Strategies
2/34
Agenda
Drivers for change?
What had to change?
What we did first
The problem
How we made it work
Experimentus Ltd 2008
2
-
8/9/2019 Why Do We Bother With Test Strategies
3/34
Drivers for change
Experimentus Ltd 2008
3
-
8/9/2019 Why Do We Bother With Test Strategies
4/34
4
-
8/9/2019 Why Do We Bother With Test Strategies
5/34
5
Experimentus Ltd 2008
The Business was demanding a higher
standard to remain competitive
The Business wanted clearer
identification of Risk to quality and
Risk Mitigation
The IT department was under pressure
to do more for less
There was a continual drive to
increase in productivity
- Cost Reduction
- Risk Management
- Improving Quality
- Revenue Enhancement
-
8/9/2019 Why Do We Bother With Test Strategies
6/34
6 Experimentus Ltd 2008
So they were looking for change to avoid the
wrong solution going live late!
-
8/9/2019 Why Do We Bother With Test Strategies
7/34
They wanted software that just worked!
7 Experimentus Ltd 2008
-
8/9/2019 Why Do We Bother With Test Strategies
8/34
-
8/9/2019 Why Do We Bother With Test Strategies
9/34
How did we find out?
We reviewed many assessment methods We looked at TPI and TMM
Selected TMM because:
It was a research project developed by Illinois Inst. of Technology
It describes the processes, which ensure a well planned and controlled test
projectHas very clear 5 levels of test maturity
Is based on the evolutionary testing model
Has been empirically validated
Inspired by the CMM (complementary)
9
Experimentus Ltd 2008
-
8/9/2019 Why Do We Bother With Test Strategies
10/34
TMM
10 Experimentus Ltd 2008
The 5 levels of TMMi
-
8/9/2019 Why Do We Bother With Test Strategies
11/34
11
How does it work?
Experimentus Ltd 2008
Maturitylevel
Indicates Testing Processcapability
ContainsKey process
areas
Achieves (Sub) Goals ContainsKey practices
Describe Implementation
-
8/9/2019 Why Do We Bother With Test Strategies
12/34
For more information see
www.tmmifoundation.org
12
Experimentus Ltd 2008
-
8/9/2019 Why Do We Bother With Test Strategies
13/34
The recommended improvements
Needed a Test Policy Needed a Testing Strategy/Framework
Needed a test process definition
Needed a defined Risk Based Approach to Testing
Needed a career path for testers
Needed to complete the roll out of TestDirector
Needed to implement reviews and static analysis
13
So what happened next?
Experimentus Ltd 2008
-
8/9/2019 Why Do We Bother With Test Strategies
14/34
What we did first
14 Experimentus Ltd 2008
-
8/9/2019 Why Do We Bother With Test Strategies
15/34
TESTING POLICY TESTING DEFINITIONRisk Based Testing is the process by which we explore and understand the status of the business benefits and the business risks associated with a release of software or of new or upgraded hardware, by:-1. Making a prioritised list of risks,2. Performing tests that explore each risk,3. As risks evaporate and new ones emerge, adjust the test effort to stay focused on the current crop.
TESTING SYSTEMDevelopment and execution of a test plan in accordance with The Legal & General Test Strategy:-
i. Testing will be organised in line with the Testing V model,ii. The different levels (phases) of testing will be clearly defined and understood,iii. Testing will be an integral part of the Project Lifecycle,iv. Formal test techniques will be used at all levels (phases) of testing, v. Appropriate training in testing will be provided,vi. Defined roles and responsibilities and Career paths will be available, vii. Appropriate Entry and Exit criteria will be defined, communicated and enforced for all test levels (phases), viii. The Test environment requirements,ix. Test tools will be used where appropriate,x. All testware will be maintained under Configuration Management, to enable audit and reuse,xi. Metrics and measures will be recorded and used to enable test improvement and better estimating for test activity, xi As well as the process for Incident Management , the definition of fault priority and severity rating will be defined.
MEASUREMENT OF TESTING EFFECTIVENESSTest Effectiveness will be measured by comparing the volume of Priority 1, 2 and 3 faults found in the first 6 months of live runningversus the volume of priority 1, 2 and 3 faults found during testing - DEFAULT DETECTION RATE (DDR).MEASUREMENT OF TEST EFFICIENCYTest Efficiency will be measured by determining the total cost of testing (both Business and BIS time both planning and executionbut excluding fix costs) as a ratio of the total design and build costs (including Project Management) TEST EFFICIENCY RATE(TER).
TESTING STANDARDSDDR of less than XXXXTER of XXXX or less TEST PROCESS IMPROVEMENTQuality reviews will take place at key stages in the test projects lifecycle. Specific testing focused Post Project reviews will be held after every release of software or of new or upgraded hardware.
------------------------------------------- ---- -------------------------------------------We agree to the general principals outlined above, and changes or additions made will be subject to change control.
Manager )________________
Dated 1st
Experimentus Ltd 2008
15
-
8/9/2019 Why Do We Bother With Test Strategies
16/34
Test Tools
Reviews
GenericTest Process
Measures andMetrics
HR
Implementationand
Communications
Risk BasedTesting
CorporateTest Strategy
Experimentus Ltd 2008
16
-
8/9/2019 Why Do We Bother With Test Strategies
17/34
CorporateTest
Strategy
GenericTest
Process
Test
Tools HRReviews
Implementationsteps
Implementationsteps
Implementationsteps
Implementationsteps
Pros- Quicker delivery- Early buy-in
- Earlier benefit realisation
Cons- Rework- Harder to manage Strategic vision- Delivery costs are larger
- Loss of confidence due to rework- Support costs are higher
Delivery choices - Vertical
Implementationsteps
Experimentus Ltd 2008
17
-
8/9/2019 Why Do We Bother With Test Strategies
18/34
CorporateTestStrategy
GenericTestProcess
TestTools HR
Reviews
Implementation steps
Implementation steps
Implementation steps
Cons- Slower delivery- Needs commitment and trust
Pros- Minimal rework- Clear strategic view- Clear understanding of impacts between
deliveries- Greater acceptance as full picture understood- Quick wins giving early buy-in
Delivery choices - Horizontal
Experimentus Ltd 2008
18
-
8/9/2019 Why Do We Bother With Test Strategies
19/34
The problem
19 Experimentus Ltd 2008
-
8/9/2019 Why Do We Bother With Test Strategies
20/34
Up to this point we had tremendous support Particularly from the Project and Programme Management Community
BUT
The process and templates just didn't get used to provide the greatest benefit sowere in danger of being bypassed by everyone
WHY?
20 Experimentus Ltd 2008
-
8/9/2019 Why Do We Bother With Test Strategies
21/34
There were two problems in theimplementation of the process improvements:
Testers
Project and Programme Managers
21 Experimentus Ltd 2008
-
8/9/2019 Why Do We Bother With Test Strategies
22/34
The Testers
Couldnt see past their noses!
22
Experimentus Ltd 2008
-
8/9/2019 Why Do We Bother With Test Strategies
23/34
They couldn't plan!
Common issues
Requirements
Estimating
Activity plans
Resource requests
Budgeting
Planning was traditionally successful but not effective, when undertakentwo weeks before test execution was due to start, when everything wasknown!
23 Experimentus Ltd 2008
-
8/9/2019 Why Do We Bother With Test Strategies
24/34
Lessons learnt
Test resources were traditionally put in place long after the project has started,
and often just before test execution starts when...All requirements are defined and almost built
Budget has been given rather than asked for
Resources are set
Timescales are set, and probably squeezed
Basically everything is in place to enable accurate plans to be drawn up!
Development Managers provided mentoring to Test Managers using theirexperience of planning when all that was available was a high level requirement
Test Managers were taught
High level estimating
Early Project Planning
24 Experimentus Ltd 2008
-
8/9/2019 Why Do We Bother With Test Strategies
25/34
The Project and Programme Managers
Couldn't understand the deliverables from testing
Test Strategy?
Test Plans?
They knew they needed a Test Strategy but didn't know why we always have
one
They were not clear when each deliverable should be produced
They still didn't invite testing in at the start of a project
25 Experimentus Ltd 2008
-
8/9/2019 Why Do We Bother With Test Strategies
26/34
Lessons learnt
Align Test deliverables with Project Deliverables
We reintroduced the section for an overview of testing that had been removedfrom the Prince2 template Quality Plan
Test Policy remained Test Strategy abolished and replaced with a documentedTest Process, and a Master Test Plan
Where appropriate rename Test Deliverables to match Project Deliverable
So lets see what we did to achieve this
26 Experimentus Ltd 2008
-
8/9/2019 Why Do We Bother With Test Strategies
27/34
Early Test Planning the inputs and outputs
Customers Quality
Expectations
AcceptanceCriteria
Quality Approachto Project
Deliverables
Quality PlanSummary
ApprovedTesting Budget
Project Plan
Project Controls
Project RisksInputstoTestPlanning
Testing Budget
Test Stages forProject Plan
Project Controls
Testing Risks
Original idea
and estimateproduced
High LevelTerms ofReference
Project
Initiation
Document
Quality Plan
ApprovedBusinessCase
Business
Requirements
Testing Estimate
InputfromT
esting
Test Policy
Test Managers
Roles andResponsibilities
EstimatingMetrics
27
Experimentus Ltd 2008
-
8/9/2019 Why Do We Bother With Test Strategies
28/34
Master Test Planning - the inputs and outputs
MTP or PID
PID
Master TestPlan
AcceptanceTest Work
Package
Team Plan
OperationalAcceptanceTest WorkPackage
SystemIntegrationTest WorkPackage
System TestWork Package
ComponentIntegrationTestWork Package
Component
Test Work
Package
Team Plan
Team Plan
Team Plan
Team Plan
Team Plan
Risk BasedTesting
Assessment
Customers Quality
Expectations
AcceptanceCriteria
Quality Approachto Project
Deliverables
Quality PlanSummary
Approved
Testing Budget
Project Plan
Project Controls
Project RisksInputstoTestPlan
ning
28
Experimentus Ltd 2008
Test Framework
-
8/9/2019 Why Do We Bother With Test Strategies
29/34
Testing Deliverable Breakdown
29 Experimentus Ltd 2008
Master Test Plan
Component Test /Component
Integration Test
Workpackage
Plan
System
Integration Test
System Test /Business
Acceptance Test
Operational
Acceptance Test
Co-Existence
Test
Production
Acceptance Test
Test
Specification
Tested/Integrated
Components
Actual Results
Workpackage
Plan
Test
Specification
Integrated
Systems
Actual Results
Test Closure
Report
Test Incidents
Workpackage
Plan
Test
Specification
Tested Solution
Actual Results
Test Closure
Report
Daily Incident
Report
Daily Progress
Report
Test Incidents
Team Plan
Test
Specification
Operationally
Tested Solution
Actual Results
Test Closure
Report
Daily Incident
Report
Daily Progress
Report
Test Incidents
Team Plan
Test
Specification
Co-Existence
Tested Solution
Actual Results
Test Closure
Report
Daily Incident
Report
Daily Progress
Report
Test Incidents
Team Plan
Test
Specification
Tested
Implementation
Actual Results
Test Closure
Report
Test Incidents
Handover Report
Weekly Test
Specification
Status Report
-
8/9/2019 Why Do We Bother With Test Strategies
30/34
Learning styles
Having a process isn't the full answer though!
Humans will initially learn the process verbatim, and will expect significant helpto be able to adapt the processes to their environment
We first have to learn, before we can be agile with our use of the process
This takes time, which is why the process improvement experts suggest an 18month timeline before process and culture come together to be at their most
efficient
30
Experimentus Ltd 2008
-
8/9/2019 Why Do We Bother With Test Strategies
31/34
Reflective
ObservationUnderstanding
ConcreteExperience
Following the rules
Active
ExperimentationDoing
AbstractConceptualisation
Thinking
Learning styles
This is where no realevaluation of whether the
process is right or notoccurs
At this point theprocess usage is
second nature, withall users being ableto evaluate what is
right and wrong.
31
Experimentus Ltd 2008
-
8/9/2019 Why Do We Bother With Test Strategies
32/34
No matter how adaptable to change people are, there will always be cultural
issues which if not managed will inhibit the benefits of the change being realised
Remember...
Experimentus Ltd 2008
That to make any process change work, agility in its delivery is the keyits no good enforcing change that no one can understand or is able to use
En
thusiasm Maximum
Hype
InitialInvention
ActualUse
Dis -illusion-
ment
PracticalBenefit
RIP
32
-
8/9/2019 Why Do We Bother With Test Strategies
33/34
While computers are not likely to make mistakes, people are. And people control
the computers.
Fred Joseph Gruenberger, 1975
Computers programmers are among the great innovators of our time. Unhappily,
amongst their most enduring accomplishments are several new techniques forwasting time.
Robert Jourdain, 1986
To the future but remember....
Experimentus Ltd 2008
33
-
8/9/2019 Why Do We Bother With Test Strategies
34/34
Thank You
Any questions?
Experimentus Ltd 2008
34