1 TIME BOXED TESTING BCS SIGIST 13 th July 1998 Graham Thomas - OSI Group.
-
Upload
sharyl-robbins -
Category
Documents
-
view
219 -
download
0
Transcript of 1 TIME BOXED TESTING BCS SIGIST 13 th July 1998 Graham Thomas - OSI Group.
![Page 1: 1 TIME BOXED TESTING BCS SIGIST 13 th July 1998 Graham Thomas - OSI Group.](https://reader035.fdocuments.in/reader035/viewer/2022062309/56649ea25503460f94ba6552/html5/thumbnails/1.jpg)
1
TIME BOXED TESTING
BCS SIGIST
13th July 1998
Graham Thomas - OSI Group
![Page 2: 1 TIME BOXED TESTING BCS SIGIST 13 th July 1998 Graham Thomas - OSI Group.](https://reader035.fdocuments.in/reader035/viewer/2022062309/56649ea25503460f94ba6552/html5/thumbnails/2.jpg)
2
Abstract
There is great pressure upon developers today to improve productivity and effectiveness. To achieve this there is a move away from the traditional structured methodologies towards more dynamic, iterative and RAD approaches.
This is being combined with Object and Component based techniques, and delivered with a new generation of IDE’s, to produce thin client, web based, voice and data products.
So how do we test it? Propose Time Boxed Testing Outline an approach Discuss the problems Give a flavour of the fun ?
![Page 3: 1 TIME BOXED TESTING BCS SIGIST 13 th July 1998 Graham Thomas - OSI Group.](https://reader035.fdocuments.in/reader035/viewer/2022062309/56649ea25503460f94ba6552/html5/thumbnails/3.jpg)
3
Agenda
Development Lifecycle Testing Lifecycle Testing Helix Testing Stage by Stage Traceability Testing Process How to sell time boxed testing Lessons Learnt
![Page 4: 1 TIME BOXED TESTING BCS SIGIST 13 th July 1998 Graham Thomas - OSI Group.](https://reader035.fdocuments.in/reader035/viewer/2022062309/56649ea25503460f94ba6552/html5/thumbnails/4.jpg)
4
The Project
Telephony based project 2 types of proprietary hardware Client Server running Windows NT Interfaces to legacy systems Rational Rose case tool (using UML) Objectory process Visual Studio Was considering Java & thin clients Incremental/Iterative development
![Page 5: 1 TIME BOXED TESTING BCS SIGIST 13 th July 1998 Graham Thomas - OSI Group.](https://reader035.fdocuments.in/reader035/viewer/2022062309/56649ea25503460f94ba6552/html5/thumbnails/5.jpg)
5
Incompatible Models?
Sys Test
Classic V Model
Rqmts.
Analysis
Design
Build Unit Test
Int Test
UAT
Breadth
Depth
Iterative (RAD) Model
![Page 6: 1 TIME BOXED TESTING BCS SIGIST 13 th July 1998 Graham Thomas - OSI Group.](https://reader035.fdocuments.in/reader035/viewer/2022062309/56649ea25503460f94ba6552/html5/thumbnails/6.jpg)
6
Difficult to Represent Test Stages
Breadth
Depth
unit
unit
unit
ComponentIntegration
ComponentIntegration
System
Acceptance
ComponentIntegration
![Page 7: 1 TIME BOXED TESTING BCS SIGIST 13 th July 1998 Graham Thomas - OSI Group.](https://reader035.fdocuments.in/reader035/viewer/2022062309/56649ea25503460f94ba6552/html5/thumbnails/7.jpg)
7
Project Lifecycle
. . .
![Page 8: 1 TIME BOXED TESTING BCS SIGIST 13 th July 1998 Graham Thomas - OSI Group.](https://reader035.fdocuments.in/reader035/viewer/2022062309/56649ea25503460f94ba6552/html5/thumbnails/8.jpg)
8
Testing Lifecycle
. . .
![Page 9: 1 TIME BOXED TESTING BCS SIGIST 13 th July 1998 Graham Thomas - OSI Group.](https://reader035.fdocuments.in/reader035/viewer/2022062309/56649ea25503460f94ba6552/html5/thumbnails/9.jpg)
9
P
DC
E
A dual spiral life cycle ?
PlanDesignConstructExecute
RqmtsAnalysisDesignBuild
. . .R
AD
B
![Page 10: 1 TIME BOXED TESTING BCS SIGIST 13 th July 1998 Graham Thomas - OSI Group.](https://reader035.fdocuments.in/reader035/viewer/2022062309/56649ea25503460f94ba6552/html5/thumbnails/10.jpg)
10
Testing Lifecycle
Unit Testing Component Integration Testing System Testing Acceptance Testing Year 2000 Millenium Compliance Testing Processes and Activities Tools & Traceability Resources Planning
![Page 11: 1 TIME BOXED TESTING BCS SIGIST 13 th July 1998 Graham Thomas - OSI Group.](https://reader035.fdocuments.in/reader035/viewer/2022062309/56649ea25503460f94ba6552/html5/thumbnails/11.jpg)
11
Scope of Test Phases
ComponentIntegrationTesting
UnitTesting System
TestingUserAcceptanceTesting
ConicalModel
Iterative Release
![Page 12: 1 TIME BOXED TESTING BCS SIGIST 13 th July 1998 Graham Thomas - OSI Group.](https://reader035.fdocuments.in/reader035/viewer/2022062309/56649ea25503460f94ba6552/html5/thumbnails/12.jpg)
12
Tuned Test Lifecycle
System Testing
- Non functional Requirements- Performance Analysis
ComponentIntegrationTesting
UnitTesting
Iterative Release
User Acceptance
Testing- Scenario Testing- Acceptance Criteria
Testing Activities Brought Forward
![Page 13: 1 TIME BOXED TESTING BCS SIGIST 13 th July 1998 Graham Thomas - OSI Group.](https://reader035.fdocuments.in/reader035/viewer/2022062309/56649ea25503460f94ba6552/html5/thumbnails/13.jpg)
13
Unit Testing
Unit Testing
StaticAnalysis
DynamicAnalysis
UnitTesting
Iterative Time-Boxed Activity
Specific to delivered components
‘White Box’ testing techniques employed
External Interfaces tested
Use of tools for Static & Dynamic analysis Quality
Assurance
ExternalInterfaceTesting
UnitIntegration
Testing
Sub UnitTesting
![Page 14: 1 TIME BOXED TESTING BCS SIGIST 13 th July 1998 Graham Thomas - OSI Group.](https://reader035.fdocuments.in/reader035/viewer/2022062309/56649ea25503460f94ba6552/html5/thumbnails/14.jpg)
14
Unit Testing
SubUnit
Unit
Unit
Unit
Package
Unit Integration Testing
SubUnit
SubUnit
Package
PackageUnit
Unit
Unit
Unit Testing
SubUnit
SubUnit
SubUnitSub
Unit
SubUnit
Sub Unit Testing
SubUnit
Stubs, Drivers, Harnesses
Previously tested & checked-in
![Page 15: 1 TIME BOXED TESTING BCS SIGIST 13 th July 1998 Graham Thomas - OSI Group.](https://reader035.fdocuments.in/reader035/viewer/2022062309/56649ea25503460f94ba6552/html5/thumbnails/15.jpg)
15
Component Integration Testing
ComponentIntegration
Non FunctionalRequirements
FunctionalRequirements
Regression Test
ComponentIntegrationTesting
Iterative Time-Boxed Activity
Testing prioritised by Business
End to End process threads where possible
Contains elements of following test phases
Build regression test packs
ScenarioRegression
Test
PerformanceAnalysis
PackageIntegration
![Page 16: 1 TIME BOXED TESTING BCS SIGIST 13 th July 1998 Graham Thomas - OSI Group.](https://reader035.fdocuments.in/reader035/viewer/2022062309/56649ea25503460f94ba6552/html5/thumbnails/16.jpg)
16
Package
Component Integration Testing
Package Package Integration must precede Component Integration
Component
Component
Component
Package
Package
Step 1 - Package Integration
![Page 17: 1 TIME BOXED TESTING BCS SIGIST 13 th July 1998 Graham Thomas - OSI Group.](https://reader035.fdocuments.in/reader035/viewer/2022062309/56649ea25503460f94ba6552/html5/thumbnails/17.jpg)
17
Component Integration Testing
Component
Component
Component
Component
Component
Drop1
ComponentComponent
Drop2
Component
Component
Component
ComponentComponent
Component
Component
Component
. . .
Step 2 - Component Integration
![Page 18: 1 TIME BOXED TESTING BCS SIGIST 13 th July 1998 Graham Thomas - OSI Group.](https://reader035.fdocuments.in/reader035/viewer/2022062309/56649ea25503460f94ba6552/html5/thumbnails/18.jpg)
18
Component Integration Testing
Automated Regression Test Pack
Component
Component
Component
ComponentComponent
Component
ComponentComponent
Component
Component
ComponentComponent
Component
Component
. . .
Component
ComponentComponentComponent
Component
Component
Component
Component
ComponentComponent
Component
Component
ComponentComponent
ComponentComponent
Drop by Drop
![Page 19: 1 TIME BOXED TESTING BCS SIGIST 13 th July 1998 Graham Thomas - OSI Group.](https://reader035.fdocuments.in/reader035/viewer/2022062309/56649ea25503460f94ba6552/html5/thumbnails/19.jpg)
19
System Testing
FunctionalRequirements
Regression Test
Non FunctionalRequirements
PerformanceAnalysis
Design Verification
SystemTesting
One off activity Structured testing
techniques End to End process
threads Ties together the
development iterations Regression test
already constructed
![Page 20: 1 TIME BOXED TESTING BCS SIGIST 13 th July 1998 Graham Thomas - OSI Group.](https://reader035.fdocuments.in/reader035/viewer/2022062309/56649ea25503460f94ba6552/html5/thumbnails/20.jpg)
20
Acceptance Testing
AcceptanceCriteriaTesting
ScenarioRegressionTesting
BusinessProcessTesting
UsabilityTesting
AcceptanceTesting
Data Take On & Conversion
Testing
One off activity Test that the computer
system integrates with business processes
Structured testing techniques
End to End process threads
Regression test already constructed
![Page 21: 1 TIME BOXED TESTING BCS SIGIST 13 th July 1998 Graham Thomas - OSI Group.](https://reader035.fdocuments.in/reader035/viewer/2022062309/56649ea25503460f94ba6552/html5/thumbnails/21.jpg)
21
Y2000 - Millennium Compliance6 Date Scenarios
2 Business Tests
ThisCentury
AcrossCentury
Boundary
Whollynext
CenturyJan 1st Feb 29th Dec 31st
FinancialYear ?
FiscalPeriods ?
Separate test runs Carried out in
sequence Some tests can be
exercised post Y2000
![Page 22: 1 TIME BOXED TESTING BCS SIGIST 13 th July 1998 Graham Thomas - OSI Group.](https://reader035.fdocuments.in/reader035/viewer/2022062309/56649ea25503460f94ba6552/html5/thumbnails/22.jpg)
22
Testing Lifecycle TraceabilityHigh Level Function
FunctionalRequirement
Use Case
Scenario
Test Case
Test Script
Non FR
DevelopmentRepository
Test Planning
UserRequirements
![Page 23: 1 TIME BOXED TESTING BCS SIGIST 13 th July 1998 Graham Thomas - OSI Group.](https://reader035.fdocuments.in/reader035/viewer/2022062309/56649ea25503460f94ba6552/html5/thumbnails/23.jpg)
23
Testing Processes and Activities
Test Management
Test Reporting Monitor Measure Manage
ExitTest StageSign off
Test Planning Test Design Test Preparation Analysis of Change
Test Execution Test Automation Regression Test
QualityInterface
ProblemMgmt
Conf. Mgmt.Interface
Process
Entry
Activity
![Page 24: 1 TIME BOXED TESTING BCS SIGIST 13 th July 1998 Graham Thomas - OSI Group.](https://reader035.fdocuments.in/reader035/viewer/2022062309/56649ea25503460f94ba6552/html5/thumbnails/24.jpg)
24
Processes
Essential Configuration Management
Impact Analysis using a Test Planning repository
Quality Management
Good practice Fault Reporting - on the Intranet - Live
Gathering metric information
![Page 25: 1 TIME BOXED TESTING BCS SIGIST 13 th July 1998 Graham Thomas - OSI Group.](https://reader035.fdocuments.in/reader035/viewer/2022062309/56649ea25503460f94ba6552/html5/thumbnails/25.jpg)
25
Tools
Full set of testing tools is required
Static analysis
Dynamic Analysis
Interactive debugging
Test Automation/Capture Replay essential
Test Planning Repository
Performance monitoring (stress/load)
Fault Reporting system
![Page 26: 1 TIME BOXED TESTING BCS SIGIST 13 th July 1998 Graham Thomas - OSI Group.](https://reader035.fdocuments.in/reader035/viewer/2022062309/56649ea25503460f94ba6552/html5/thumbnails/26.jpg)
26
Monitoring
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
Fault
0
20
40
60
80
100
120
140
Cumulative Error Curve
![Page 27: 1 TIME BOXED TESTING BCS SIGIST 13 th July 1998 Graham Thomas - OSI Group.](https://reader035.fdocuments.in/reader035/viewer/2022062309/56649ea25503460f94ba6552/html5/thumbnails/27.jpg)
27
Monitoring
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
0
5
10
15
20
25
30
35
40
45
50
Outstanding
![Page 28: 1 TIME BOXED TESTING BCS SIGIST 13 th July 1998 Graham Thomas - OSI Group.](https://reader035.fdocuments.in/reader035/viewer/2022062309/56649ea25503460f94ba6552/html5/thumbnails/28.jpg)
28
Monitoring
1 2 3 4 5 6 7 8 9 10 11 12 13
0
20
40
60
80
100
120
140
Faults by Priority
A
B
C
![Page 29: 1 TIME BOXED TESTING BCS SIGIST 13 th July 1998 Graham Thomas - OSI Group.](https://reader035.fdocuments.in/reader035/viewer/2022062309/56649ea25503460f94ba6552/html5/thumbnails/29.jpg)
29
Monitoring
W1 W2 W3 W40
5
10
15
20
25
30
Aged Defect Analysis by Priority
A
B
C
![Page 30: 1 TIME BOXED TESTING BCS SIGIST 13 th July 1998 Graham Thomas - OSI Group.](https://reader035.fdocuments.in/reader035/viewer/2022062309/56649ea25503460f94ba6552/html5/thumbnails/30.jpg)
30
Test Resources
UnitTesting
ComponentIntegrationTesting
SystemTesting
UserAcceptanceTesting
TESTTEAM
Testers
BusinessUsersDevelopers
BusinessUsers
Developers
Secondment
![Page 31: 1 TIME BOXED TESTING BCS SIGIST 13 th July 1998 Graham Thomas - OSI Group.](https://reader035.fdocuments.in/reader035/viewer/2022062309/56649ea25503460f94ba6552/html5/thumbnails/31.jpg)
31
Test Planning
Definition
Test Strategy
Testing Process
Developm
ent
Non Functional Testing
Test drop
Test drop
Test drop
Decision
Test Approach
Delivery
System Test
Acceptance Test
Y2000 Test
Time-boxed Testing
![Page 32: 1 TIME BOXED TESTING BCS SIGIST 13 th July 1998 Graham Thomas - OSI Group.](https://reader035.fdocuments.in/reader035/viewer/2022062309/56649ea25503460f94ba6552/html5/thumbnails/32.jpg)
32
Selling the Approach
Target the following; Test Team
Systems Architect
Development team
Standards & Methods
Business Representatives
Project Sponsor
Via Animated Presentation Test Strategy document
![Page 33: 1 TIME BOXED TESTING BCS SIGIST 13 th July 1998 Graham Thomas - OSI Group.](https://reader035.fdocuments.in/reader035/viewer/2022062309/56649ea25503460f94ba6552/html5/thumbnails/33.jpg)
33
So how easy was this?
![Page 34: 1 TIME BOXED TESTING BCS SIGIST 13 th July 1998 Graham Thomas - OSI Group.](https://reader035.fdocuments.in/reader035/viewer/2022062309/56649ea25503460f94ba6552/html5/thumbnails/34.jpg)
34
So What went well?
Systems Architect liked it The presentation was well received
![Page 35: 1 TIME BOXED TESTING BCS SIGIST 13 th July 1998 Graham Thomas - OSI Group.](https://reader035.fdocuments.in/reader035/viewer/2022062309/56649ea25503460f94ba6552/html5/thumbnails/35.jpg)
35
What was difficult?
Initial cool response from other members of the test team
Development Team were uneasy Dev. Why test anyway, our code is wonderful and our test
harnesses will do all the testing? Business Representatives were unhappy about;
Anything less than 100% testing Bringing their effort forward in the development lifecycle The difference between the ‘Use Case’ modeling they had
done, and the test planning that they would need to carry out
![Page 36: 1 TIME BOXED TESTING BCS SIGIST 13 th July 1998 Graham Thomas - OSI Group.](https://reader035.fdocuments.in/reader035/viewer/2022062309/56649ea25503460f94ba6552/html5/thumbnails/36.jpg)
36
What was difficult? (2)
Standards & MethodsSaw nothing wrong with their approach, it was either;
• structured in which case their approach worked fine,
or• it was incremental, in which case their approach worked
fine.• They did not do iterative development, but if they did their
approach would still work.
This took several meetings to gain approval. Project Sponsor
Cancelled the project. (Time Boxed Testing was not cited as the reason !!!)
![Page 37: 1 TIME BOXED TESTING BCS SIGIST 13 th July 1998 Graham Thomas - OSI Group.](https://reader035.fdocuments.in/reader035/viewer/2022062309/56649ea25503460f94ba6552/html5/thumbnails/37.jpg)
37
So what fun did we have ?
![Page 38: 1 TIME BOXED TESTING BCS SIGIST 13 th July 1998 Graham Thomas - OSI Group.](https://reader035.fdocuments.in/reader035/viewer/2022062309/56649ea25503460f94ba6552/html5/thumbnails/38.jpg)
38
What did I learn from this?
Organisations change their development approach more quickly than their testing approach
There is a reluctance to do anything less than 100% testing
Testing is a key and integral part of any development lifecycle
The more you think about it, the more sense Time Boxed Testing makes
![Page 39: 1 TIME BOXED TESTING BCS SIGIST 13 th July 1998 Graham Thomas - OSI Group.](https://reader035.fdocuments.in/reader035/viewer/2022062309/56649ea25503460f94ba6552/html5/thumbnails/39.jpg)
39
Contact Information
Graham Thomas
OSIthe management support company
e-mail : [email protected]
Phone : 0171 323 3353
Copies of slides including animated gifs at
www.osi.co.uk