The Three Holy Grails of Test Development€¦ · test development test automation process oriented...
Transcript of The Three Holy Grails of Test Development€¦ · test development test automation process oriented...
![Page 1: The Three Holy Grails of Test Development€¦ · test development test automation process oriented dependencies. slide - 11 The Holy Grail • Part of the legend of king arthur •](https://reader035.fdocuments.in/reader035/viewer/2022081405/5f0788157e708231d41d7220/html5/thumbnails/1.jpg)
P R E S E N T A T I O N
International Conference On
Software Testing, Analysis & ReviewDEC 4-8, 2000 • COPENHAGEN, DENMARK
Presentation
Bio
Return to Main Menu TG1
Thursday, Dec 7, 2000
The Three Holy Grailsof Test Development
Hans Buwalda
![Page 2: The Three Holy Grails of Test Development€¦ · test development test automation process oriented dependencies. slide - 11 The Holy Grail • Part of the legend of king arthur •](https://reader035.fdocuments.in/reader035/viewer/2022081405/5f0788157e708231d41d7220/html5/thumbnails/2.jpg)
The 3 Holy Grailsof
Test DevelopmentHans Buwalda
CMG - TestFrame Research [email protected]
www.testframe.com
Hans BuwaldaCMG - TestFrame Research Centre
![Page 3: The Three Holy Grails of Test Development€¦ · test development test automation process oriented dependencies. slide - 11 The Holy Grail • Part of the legend of king arthur •](https://reader035.fdocuments.in/reader035/viewer/2022081405/5f0788157e708231d41d7220/html5/thumbnails/3.jpg)
slide - 2
The Quest
![Page 4: The Three Holy Grails of Test Development€¦ · test development test automation process oriented dependencies. slide - 11 The Holy Grail • Part of the legend of king arthur •](https://reader035.fdocuments.in/reader035/viewer/2022081405/5f0788157e708231d41d7220/html5/thumbnails/4.jpg)
slide - 3
The Challenges for a Test Process
• Testing should be fun
• Testing should be effective
• Testing should be efficient
• Testing should be under control
![Page 5: The Three Holy Grails of Test Development€¦ · test development test automation process oriented dependencies. slide - 11 The Holy Grail • Part of the legend of king arthur •](https://reader035.fdocuments.in/reader035/viewer/2022081405/5f0788157e708231d41d7220/html5/thumbnails/5.jpg)
slide - 4
Common problems with testing
• Time consuming
• Costly
• Tends to be neglected
• Experienced as boring to do (in particularthe execution)
• Hard to start in time
![Page 6: The Three Holy Grails of Test Development€¦ · test development test automation process oriented dependencies. slide - 11 The Holy Grail • Part of the legend of king arthur •](https://reader035.fdocuments.in/reader035/viewer/2022081405/5f0788157e708231d41d7220/html5/thumbnails/6.jpg)
slide - 5
Even more common problems with testing
• Difficult to manage:! What is the progress! What is the quality
• The proper resources (users, specialists)are not (made) available when needed
• Lack of test design competency
• etcetera
![Page 7: The Three Holy Grails of Test Development€¦ · test development test automation process oriented dependencies. slide - 11 The Holy Grail • Part of the legend of king arthur •](https://reader035.fdocuments.in/reader035/viewer/2022081405/5f0788157e708231d41d7220/html5/thumbnails/7.jpg)
slide - 6
Dimensions of Testing
• Organisation and management
• Test development
• Test automation
![Page 8: The Three Holy Grails of Test Development€¦ · test development test automation process oriented dependencies. slide - 11 The Holy Grail • Part of the legend of king arthur •](https://reader035.fdocuments.in/reader035/viewer/2022081405/5f0788157e708231d41d7220/html5/thumbnails/8.jpg)
slide - 7
Re-usable test products
test development
test execution
test cluster
navigation scheme
…check balanceenter customer…
A B C D. . .transfer Houston Klein 210check balance Klein 210transfer Savy Klein 150check balance Klein 360. . .
![Page 9: The Three Holy Grails of Test Development€¦ · test development test automation process oriented dependencies. slide - 11 The Holy Grail • Part of the legend of king arthur •](https://reader035.fdocuments.in/reader035/viewer/2022081405/5f0788157e708231d41d7220/html5/thumbnails/9.jpg)
slide - 8
tool
navigation script
targetsystemseparation
report
test design
• test conditions• test lines
test clusters
test plan
• actual results• comparison with
expectations• management
information
• input data• expected outcomes• documentation
management
systemdevelopment
QA/Auditors
end users
The product life cycle
![Page 10: The Three Holy Grails of Test Development€¦ · test development test automation process oriented dependencies. slide - 11 The Holy Grail • Part of the legend of king arthur •](https://reader035.fdocuments.in/reader035/viewer/2022081405/5f0788157e708231d41d7220/html5/thumbnails/10.jpg)
slide - 9
Example Test Condition
nr description...3.51 it is checked that the exit date is after the entry date...
test condition 3.51
name entry date exit dateenter employment Bill Goodfellow 1999-10-02 1999-10-01check error message The exit date must be after the entry date.
coupling with the actual test lines in the cluster
![Page 11: The Three Holy Grails of Test Development€¦ · test development test automation process oriented dependencies. slide - 11 The Holy Grail • Part of the legend of king arthur •](https://reader035.fdocuments.in/reader035/viewer/2022081405/5f0788157e708231d41d7220/html5/thumbnails/11.jpg)
slide - 10
Independence of life cycles
systemdevelopment
testdevelopment
testautomation
process oriented dependencies
![Page 12: The Three Holy Grails of Test Development€¦ · test development test automation process oriented dependencies. slide - 11 The Holy Grail • Part of the legend of king arthur •](https://reader035.fdocuments.in/reader035/viewer/2022081405/5f0788157e708231d41d7220/html5/thumbnails/12.jpg)
slide - 11
The Holy Grail
• Part of the legend of king arthur
• Most of all a symbol
• To find the holy grail one has to be theperfect knight
• To search for it is as important as finding it
![Page 13: The Three Holy Grails of Test Development€¦ · test development test automation process oriented dependencies. slide - 11 The Holy Grail • Part of the legend of king arthur •](https://reader035.fdocuments.in/reader035/viewer/2022081405/5f0788157e708231d41d7220/html5/thumbnails/13.jpg)
slide - 12
The Holy Grail as a model
• General principles, good to “strive” for, butdifficult to fully achieve
• The three principles suggested in thispresentation are for test development
• Coming close to the grails will help quite a bit
• Being far away is a good recipe for trouble
• The first “victim” is the manageability
![Page 14: The Three Holy Grails of Test Development€¦ · test development test automation process oriented dependencies. slide - 11 The Holy Grail • Part of the legend of king arthur •](https://reader035.fdocuments.in/reader035/viewer/2022081405/5f0788157e708231d41d7220/html5/thumbnails/14.jpg)
slide - 13
The three “holy grails” of Test Development
effective “clustering” of tests
the proper level of actions
choosing the right approach per cluster
![Page 15: The Three Holy Grails of Test Development€¦ · test development test automation process oriented dependencies. slide - 11 The Holy Grail • Part of the legend of king arthur •](https://reader035.fdocuments.in/reader035/viewer/2022081405/5f0788157e708231d41d7220/html5/thumbnails/15.jpg)
slide - 14
The First Grailclusteringclustering
![Page 16: The Three Holy Grails of Test Development€¦ · test development test automation process oriented dependencies. slide - 11 The Holy Grail • Part of the legend of king arthur •](https://reader035.fdocuments.in/reader035/viewer/2022081405/5f0788157e708231d41d7220/html5/thumbnails/16.jpg)
slide - 15
well known, the V Model (simplified)
requirements
logicaldesign
physicaldesign
programming module tests
subsystem tests
systemtests
acceptancetests
(inflexibel) use of the V Model can involve risks (see Brian Maricks article: www.testing.com/writings/new-models.pdf)
![Page 17: The Three Holy Grails of Test Development€¦ · test development test automation process oriented dependencies. slide - 11 The Holy Grail • Part of the legend of king arthur •](https://reader035.fdocuments.in/reader035/viewer/2022081405/5f0788157e708231d41d7220/html5/thumbnails/17.jpg)
slide - 16
The “cluster model” as an alternative
• fit to project situation as an explicit step
• V Model is “member of the class”
• further decisions are per cluster:• how -> which technique, automate or not• when -> develop, execute• who -> stake holders, tester, auditor, ...• why -> business risk, complexity, made by John, ...
![Page 18: The Three Holy Grails of Test Development€¦ · test development test automation process oriented dependencies. slide - 11 The Holy Grail • Part of the legend of king arthur •](https://reader035.fdocuments.in/reader035/viewer/2022081405/5f0788157e708231d41d7220/html5/thumbnails/18.jpg)
slide - 17
What is not likely to be good clustering
• combinations of low and high level tests
• when you have to change all of them for anew system release
• if all clusters look alike
• clusters are dependent on each other
• you can’t start developing them now
![Page 19: The Three Holy Grails of Test Development€¦ · test development test automation process oriented dependencies. slide - 11 The Holy Grail • Part of the legend of king arthur •](https://reader035.fdocuments.in/reader035/viewer/2022081405/5f0788157e708231d41d7220/html5/thumbnails/19.jpg)
slide - 18
Typical consequences / symptoms
• complaining people
• unnecessary high test maintenance (highimpact of system changes on the test)
• difficulties in running any test
• “unpleasant” test process
• no (sense of) control
![Page 20: The Three Holy Grails of Test Development€¦ · test development test automation process oriented dependencies. slide - 11 The Holy Grail • Part of the legend of king arthur •](https://reader035.fdocuments.in/reader035/viewer/2022081405/5f0788157e708231d41d7220/html5/thumbnails/20.jpg)
slide - 19
Clustering recommendations
• Logical to all concerned
• Independent from other clusters
• Well differentiated and clear in scope
• Fitting the priorities and planning of the project
• Balanced in size and amount
![Page 21: The Three Holy Grails of Test Development€¦ · test development test automation process oriented dependencies. slide - 11 The Holy Grail • Part of the legend of king arthur •](https://reader035.fdocuments.in/reader035/viewer/2022081405/5f0788157e708231d41d7220/html5/thumbnails/21.jpg)
slide - 20
Some angles for clustering• business and failure risks• planning and control• test execution• complexity of the test• stake holders and others involved• part of the system• detail level• quality aspect• technique used for the test (double role)• functional and technical requirements• . . .
![Page 22: The Three Holy Grails of Test Development€¦ · test development test automation process oriented dependencies. slide - 11 The Holy Grail • Part of the legend of king arthur •](https://reader035.fdocuments.in/reader035/viewer/2022081405/5f0788157e708231d41d7220/html5/thumbnails/22.jpg)
slide - 21
Example of a (customer friendly) businessrisks analysis at a financial institution
if thisgoes
wrong then...
must haveall customers
one customer should have
could have
nice to have
no workaround
workaround nice to have
should have
… there are financial(or worse) consequenses
for our customer
… there are non-financialconsequenses
for our customer
… there are financial(or worse) consequenses
for ourselves
… there are non-financialconsequenses
for ourselves
all customers
one customer
no workaround
workaround
should have
should have
![Page 23: The Three Holy Grails of Test Development€¦ · test development test automation process oriented dependencies. slide - 11 The Holy Grail • Part of the legend of king arthur •](https://reader035.fdocuments.in/reader035/viewer/2022081405/5f0788157e708231d41d7220/html5/thumbnails/23.jpg)
slide - 22
tests subsystem 4
tests subsystem 1
tests subsystem 2
tests subsystem 3
The clustering process (example)
integration
performance
business scenario’s
tests module 1
tests module 2
tests module 3
tests module 4
integration
performance
![Page 24: The Three Holy Grails of Test Development€¦ · test development test automation process oriented dependencies. slide - 11 The Holy Grail • Part of the legend of king arthur •](https://reader035.fdocuments.in/reader035/viewer/2022081405/5f0788157e708231d41d7220/html5/thumbnails/24.jpg)
slide - 23
The Second Grailapproach per cluster
![Page 25: The Three Holy Grails of Test Development€¦ · test development test automation process oriented dependencies. slide - 11 The Holy Grail • Part of the legend of king arthur •](https://reader035.fdocuments.in/reader035/viewer/2022081405/5f0788157e708231d41d7220/html5/thumbnails/25.jpg)
slide - 24
What is not a good approach (..)
• Complaining people
• The wrong level of techniques, like using aboundary value analysis in a high levelbusiness test
• Lots of hard labour without interestingresults (“monks work”)
![Page 26: The Three Holy Grails of Test Development€¦ · test development test automation process oriented dependencies. slide - 11 The Holy Grail • Part of the legend of king arthur •](https://reader035.fdocuments.in/reader035/viewer/2022081405/5f0788157e708231d41d7220/html5/thumbnails/26.jpg)
slide - 25
Common Design Techniques Summary
• Equivalence Partitioning
• Boundary Value Analysis
• Output Forcing
• State Models
• Error Guessing
• Reference: [Beizer, 1990] [Kit, 1996]
![Page 27: The Three Holy Grails of Test Development€¦ · test development test automation process oriented dependencies. slide - 11 The Holy Grail • Part of the legend of king arthur •](https://reader035.fdocuments.in/reader035/viewer/2022081405/5f0788157e708231d41d7220/html5/thumbnails/27.jpg)
slide - 26
Clusters
Test Conditions
Tests
Specifications Experts
• Soap Opera Testing• Boundary value analysis• Equivalence classes• Joint testware development
• Decision tables• Graphs• Data flow analysis• Entity life cycle techniques
• Joint specification development
• Joint conditiondevelopment
• Functional breakdown• Quality attributes• Architectural breakdown
Testing techniques
![Page 28: The Three Holy Grails of Test Development€¦ · test development test automation process oriented dependencies. slide - 11 The Holy Grail • Part of the legend of king arthur •](https://reader035.fdocuments.in/reader035/viewer/2022081405/5f0788157e708231d41d7220/html5/thumbnails/28.jpg)
slide - 27
Soap Operas
Ashley hears about Jack's deposit when he thought he hadto go. Victoria lectures her father about what's wrong with himand Nikki but Victor advises her that it's none of her businessOlivia learns Dru has no regrets about leaving and takes greatsatisfaction in having Lily as her companion. Dru then asks Oliviawhy she is raking Malcolm over the coals. Stopping by Gina's,Nikki spots Brad and sits with him, admitting she doesn't want tobe alone tonight. Victor stops by Mack's party at the CrimsonLights. Ashley takes a home pregnancy test. Worried about Billy,Raul makes call and J.T. claims he doesn't know where Billy is.Raul rushes over and finds Billy out cold in the snow Raul worrieswhen he can't find a pulse. . . .
![Page 29: The Three Holy Grails of Test Development€¦ · test development test automation process oriented dependencies. slide - 11 The Holy Grail • Part of the legend of king arthur •](https://reader035.fdocuments.in/reader035/viewer/2022081405/5f0788157e708231d41d7220/html5/thumbnails/29.jpg)
slide - 28
Properties of Soap Operas
• About “real life”
• But condensed
• And more extreme
![Page 30: The Three Holy Grails of Test Development€¦ · test development test automation process oriented dependencies. slide - 11 The Holy Grail • Part of the legend of king arthur •](https://reader035.fdocuments.in/reader035/viewer/2022081405/5f0788157e708231d41d7220/html5/thumbnails/30.jpg)
slide - 29
Soap Operas for testing
• Define a scope of the test to develop
• Identify with the business environment
• Which elements would make things difficult
• Draft scenario’s (typical some dozen lines)
• Write them down in clusters
![Page 31: The Three Holy Grails of Test Development€¦ · test development test automation process oriented dependencies. slide - 11 The Holy Grail • Part of the legend of king arthur •](https://reader035.fdocuments.in/reader035/viewer/2022081405/5f0788157e708231d41d7220/html5/thumbnails/31.jpg)
slide - 30
Examples of story lines when used for testing
Pension Fund
World Wide Transaction System for an international Bank
William starts as a metal worker for Industrial EntropyIncorporated in 1955. During his career he becomes ill, workspart time, marries, divorces, marries again, gets 3 children, oneof which dies, then his wife dies and he marries again and gets 2more children….
A fish trade company in Japan makes a payment to a vendor onIceland. It should have been a payment in Icelandic Kronur, butit was done in Yen instead. The error is discovered after 9 daysand the payment is revised and corrected, however, the interestcalculation (value dating)…
![Page 32: The Three Holy Grails of Test Development€¦ · test development test automation process oriented dependencies. slide - 11 The Holy Grail • Part of the legend of king arthur •](https://reader035.fdocuments.in/reader035/viewer/2022081405/5f0788157e708231d41d7220/html5/thumbnails/32.jpg)
slide - 31
Example of test lines
from to amount valuta trans nrenter payment 123421344 4124244123 120000 yen &keep tx1check value dating &tx1 $0.47wait days 9
order to reverse &tx1
from to amount valuta trans nrenter payment 123421344 4124244123 1200000000 IKr &keep tx2check value dating &tx2 $7,701.56
. . . .
![Page 33: The Three Holy Grails of Test Development€¦ · test development test automation process oriented dependencies. slide - 11 The Holy Grail • Part of the legend of king arthur •](https://reader035.fdocuments.in/reader035/viewer/2022081405/5f0788157e708231d41d7220/html5/thumbnails/33.jpg)
slide - 32
Soap Operas (in testing) are not necessarily:
• “Extreme”
• Far fetched
• Long and elaborate
• Pieces of art and creativity
![Page 34: The Three Holy Grails of Test Development€¦ · test development test automation process oriented dependencies. slide - 11 The Holy Grail • Part of the legend of king arthur •](https://reader035.fdocuments.in/reader035/viewer/2022081405/5f0788157e708231d41d7220/html5/thumbnails/34.jpg)
slide - 33
“Killer Soaps”
• More specifically aimed at finding hiddenproblems
• Run when everything else has passed
• One option: put a killer soap at the end of anormal cluster
• Ask the “specialists” for input
![Page 35: The Three Holy Grails of Test Development€¦ · test development test automation process oriented dependencies. slide - 11 The Holy Grail • Part of the legend of king arthur •](https://reader035.fdocuments.in/reader035/viewer/2022081405/5f0788157e708231d41d7220/html5/thumbnails/35.jpg)
slide - 34
Test Design Templates
• Spreadsheet based technique for designing tests
• Introduced in 1998 as an extension to testframe byedward kit
• Friendly technique and yet very strong, for examplein identifying needed situations and combinations
• For a further description please have a look at ed’sarticle in software development magazine:http://www.sdmagazine.com/breakrm/features/s992f2.shtml
![Page 36: The Three Holy Grails of Test Development€¦ · test development test automation process oriented dependencies. slide - 11 The Holy Grail • Part of the legend of king arthur •](https://reader035.fdocuments.in/reader035/viewer/2022081405/5f0788157e708231d41d7220/html5/thumbnails/36.jpg)
slide - 35
Example ofTest DesignTemplates
(applied toSDT’sReviewPropackage forsupportinginspections andreviews)
Feature Hierarchy: Default Entry Type Standard
Matrix Summary: Check that the default (non-customized) Entry Types can be selected/used
Test Design: Technique Feature Combination
Risk Analysis: Impact Likelihood
Test Case ID: DETS01 DETS02 DETS03 DETS04
Test Case Validity:
Priority:Test Condition
Log Entry Input Fields * * * *Document Reviewed =Location =Type = defect question external issue praiseSeverity =Status =Disposition Code =Summary =Attachments =
Select Default Entry *Select Entry #2 *Select Entry #3 *Select Entry #4 *Expected Results
Out
put=
Go
to R
evie
wer
's
Log
Vie
w, L
og is
in th
e R
evie
wer
's L
og F
orum
, C
orre
ct E
ntry
Typ
e is
se
lect
ed in
the
Log
Ent
ry
Out
put=
Go
to R
evie
wer
's
Log
Vie
w, L
og is
in th
e R
evie
wer
's L
og F
orum
, C
orre
ct E
ntry
Typ
e is
se
lect
ed in
the
Log
Ent
ry
Out
put=
Go
to R
evie
wer
's Lo
g Vi
ew, L
og is
in th
e Re
view
er's
Log
Foru
m,
Corr
ect E
ntry
Typ
e is
se
lect
ed in
the
Log
Entry
Out
put=
Go
to R
evie
wer
's
Log
Vie
w, L
og is
in th
e R
evie
wer
's L
og F
orum
, C
orre
ct E
ntry
Typ
e is
se
lect
ed in
the
Log
Ent
ry
Valid
High
Valid
High
Valid Valid
High High
NoEquivalence Class
LowHigh
![Page 37: The Three Holy Grails of Test Development€¦ · test development test automation process oriented dependencies. slide - 11 The Holy Grail • Part of the legend of king arthur •](https://reader035.fdocuments.in/reader035/viewer/2022081405/5f0788157e708231d41d7220/html5/thumbnails/37.jpg)
slide - 36
Usage of Test Design Templates as portal toSoap Operas
Template ID: MB des 1 MB des 2 MB des 3 MB des 4….
customer * * * *last namefirst namebalance positive too low positive positivenumber
confirmation letter yes yesautomatic numbering yes
….tested in scenario: MB01 MB01, MB02 MB02
matching
![Page 38: The Three Holy Grails of Test Development€¦ · test development test automation process oriented dependencies. slide - 11 The Holy Grail • Part of the legend of king arthur •](https://reader035.fdocuments.in/reader035/viewer/2022081405/5f0788157e708231d41d7220/html5/thumbnails/38.jpg)
slide - 37
A life cycle for test development
clustering
testconditions test design
templates testscenario’s
reviewexecutionmaintenance...
clus
ter l
evel
over
all
match(cross ref)
![Page 39: The Three Holy Grails of Test Development€¦ · test development test automation process oriented dependencies. slide - 11 The Holy Grail • Part of the legend of king arthur •](https://reader035.fdocuments.in/reader035/viewer/2022081405/5f0788157e708231d41d7220/html5/thumbnails/39.jpg)
slide - 38
What can joint sessions give you
• Test Strategy
• Acceptance Criteria
• Cluster Grouping
• Test conditions
• Evaluation of Results
• Starting up development of scenarios
![Page 40: The Three Holy Grails of Test Development€¦ · test development test automation process oriented dependencies. slide - 11 The Holy Grail • Part of the legend of king arthur •](https://reader035.fdocuments.in/reader035/viewer/2022081405/5f0788157e708231d41d7220/html5/thumbnails/40.jpg)
slide - 39
Joint Testware Development (JTD)
• Moderator / chairman
• Users
• Business specialists
• Developers
• Testers
TM
![Page 41: The Three Holy Grails of Test Development€¦ · test development test automation process oriented dependencies. slide - 11 The Holy Grail • Part of the legend of king arthur •](https://reader035.fdocuments.in/reader035/viewer/2022081405/5f0788157e708231d41d7220/html5/thumbnails/41.jpg)
slide - 40
Set-up of a joint session for a telecomprovider• 1st session
• Introduction by moderator and project manager• explanation about the JTD procedure• explanation of the functional area by a specialised
user
• 2nd session• start of production of test conditions
• 3rd session• start of production of test scenarios
• 4th session• evaluation test scenarios
![Page 42: The Three Holy Grails of Test Development€¦ · test development test automation process oriented dependencies. slide - 11 The Holy Grail • Part of the legend of king arthur •](https://reader035.fdocuments.in/reader035/viewer/2022081405/5f0788157e708231d41d7220/html5/thumbnails/42.jpg)
slide - 41
The Third Graillevel of actions
![Page 43: The Three Holy Grails of Test Development€¦ · test development test automation process oriented dependencies. slide - 11 The Holy Grail • Part of the legend of king arthur •](https://reader035.fdocuments.in/reader035/viewer/2022081405/5f0788157e708231d41d7220/html5/thumbnails/43.jpg)
slide - 42
Composition of Action Words
• Specification of an action, a check or adocumentary statement
• Communication between Navigation andTest Cases
• Consistent
• Standard
• By-product of the test analysis
![Page 44: The Three Holy Grails of Test Development€¦ · test development test automation process oriented dependencies. slide - 11 The Holy Grail • Part of the legend of king arthur •](https://reader035.fdocuments.in/reader035/viewer/2022081405/5f0788157e708231d41d7220/html5/thumbnails/44.jpg)
slide - 43
• “test all”
• low level “navigation” in a high level test:
• unclear “insiders only” language
What is probably not a good action level
type "Hans"
press key <tab>
type "Buwalda"
click button OK
...
set code Fc122x XX33
...
![Page 45: The Three Holy Grails of Test Development€¦ · test development test automation process oriented dependencies. slide - 11 The Holy Grail • Part of the legend of king arthur •](https://reader035.fdocuments.in/reader035/viewer/2022081405/5f0788157e708231d41d7220/html5/thumbnails/45.jpg)
slide - 44
Typical consequences / symptoms
• tests become quite unreadable (especiallyfor non experts)
• unpleasant work to make the tests
• hard to understand the results
• high risk of mistakes
• and, of course, heavy maintenancedependency
![Page 46: The Three Holy Grails of Test Development€¦ · test development test automation process oriented dependencies. slide - 11 The Holy Grail • Part of the legend of king arthur •](https://reader035.fdocuments.in/reader035/viewer/2022081405/5f0788157e708231d41d7220/html5/thumbnails/46.jpg)
slide - 45
Designing Action Words
• What actions does the test tool perform witha specific Action Word
• Scope of the test determines the ActionWord level
• Manage the set of Action Words
• Document information about the ActionWords
![Page 47: The Three Holy Grails of Test Development€¦ · test development test automation process oriented dependencies. slide - 11 The Holy Grail • Part of the legend of king arthur •](https://reader035.fdocuments.in/reader035/viewer/2022081405/5f0788157e708231d41d7220/html5/thumbnails/47.jpg)
slide - 46
Low-level• Low-level functions take care of the
communication with the application
• Low-level functions are being called from highlevel action word functions
• Use of low-level functions enhancesmaintainability
• Typical examples:! push button! select list box item
![Page 48: The Three Holy Grails of Test Development€¦ · test development test automation process oriented dependencies. slide - 11 The Holy Grail • Part of the legend of king arthur •](https://reader035.fdocuments.in/reader035/viewer/2022081405/5f0788157e708231d41d7220/html5/thumbnails/48.jpg)
slide - 47
Intermediate level
• Optional, use for complex navigation's
• Navigation at level of:! Window! Record! Message! ...
• Called as subroutine by high level actionwords
![Page 49: The Three Holy Grails of Test Development€¦ · test development test automation process oriented dependencies. slide - 11 The Holy Grail • Part of the legend of king arthur •](https://reader035.fdocuments.in/reader035/viewer/2022081405/5f0788157e708231d41d7220/html5/thumbnails/49.jpg)
slide - 48
High level
• Defined by testers,! (not by navigators and not in advance)
• Specification of the test, navigation:! Adds default values! Moves across windows! Takes care of unexpected situations! ...
• Uses low level and intermediate level
![Page 50: The Three Holy Grails of Test Development€¦ · test development test automation process oriented dependencies. slide - 11 The Holy Grail • Part of the legend of king arthur •](https://reader035.fdocuments.in/reader035/viewer/2022081405/5f0788157e708231d41d7220/html5/thumbnails/50.jpg)
slide - 49
The life cycle
• high level business oriented tests• production acceptance tests
• functional tests• technical tests
• low level functional tests• technical tests
specifications
design
programming
high levelactions
intermediatelevel actions
low levelactions
Test Execution
. . .
. . .
systemdevelopment
testdevelopment
navigation
![Page 51: The Three Holy Grails of Test Development€¦ · test development test automation process oriented dependencies. slide - 11 The Holy Grail • Part of the legend of king arthur •](https://reader035.fdocuments.in/reader035/viewer/2022081405/5f0788157e708231d41d7220/html5/thumbnails/51.jpg)
slide - 50
Acknowledgements
• Edward Kit
• Dorothy Graham
• Brian Marick
• Dirk van Dael
• Erik Jansen
• Dennis Janssen
![Page 52: The Three Holy Grails of Test Development€¦ · test development test automation process oriented dependencies. slide - 11 The Holy Grail • Part of the legend of king arthur •](https://reader035.fdocuments.in/reader035/viewer/2022081405/5f0788157e708231d41d7220/html5/thumbnails/52.jpg)
slide - 51
References• Beizer, Boris, Software Testing Techniques, Second Edition, Van
Nostrand Reinhold, 1990• Buwalda, Hans, Testing with Action Words, STAR May 1998• Buwalda, Hans, and Kasdorp, Maartje, Getting Automated Testing
Under Control, STQE Magazine, November 1999• Buwalda, Hans, Soap Opera Testing, STAR East May 2000• Graham, Dorothy, and Fewster, Mark, Automating Software
Testing, 1999• Kit, Edward, Software Testing in the Real World, Addison Wesley
Longman, 1996• Myers, Glenford, The Art Of Software Testing, Wiley Interscience,
1979• Kit, Edward, Integrated, Effective Test Design and Automation,
Software Development Magazine, February 1999
![Page 53: The Three Holy Grails of Test Development€¦ · test development test automation process oriented dependencies. slide - 11 The Holy Grail • Part of the legend of king arthur •](https://reader035.fdocuments.in/reader035/viewer/2022081405/5f0788157e708231d41d7220/html5/thumbnails/53.jpg)
Thursday 7 December 2000
Keynote 1 The Three Holy Grails of Test Development Hans Buwalda Hans Buwalda is project director at CMG, a leading European information technology services group. He is responsible for new developments around the TestFrame approach for testing and test automation of which he is the main architect. In 1996 he presented the main ideas for the first time to an international audience in a speech called 'Testing with Action Words, abandoning record and playback'. Since then the method is being used in an increasing number of countries and Hans has become a frequent speaker at industry conference, tutorials and workshops.