1
Software EngineeringSoftware EngineeringSoftware Testing TechniquesSoftware Testing Techniques
TestabilityTestability
OperabilityOperability——it operates cleanlyit operates cleanly
ObservabilityObservability——the results of each test case are readilythe results of each test case are readilyobservedobserved
ControllabilityControllability——the degree to which testing can be automatedthe degree to which testing can be automatedand optimizedand optimized
DecomposabilityDecomposability——testing can be targetedtesting can be targeted
SimplicitySimplicity——reduce complex architecture and logic to simplifyreduce complex architecture and logic to simplifyteststests
StabilityStability——few changes are requested during testingfew changes are requested during testing
UnderstandabilityUnderstandability——of the designof the design
What is aWhat is a ““GoodGood”” Test?Test?
A good test has a high probability of findingA good test has a high probability of findingan erroran error
A good test is not redundant.A good test is not redundant.
A good test should beA good test should be ““best of breedbest of breed””
A good test should be neither too simple norA good test should be neither too simple nortoo complextoo complex
Test Case DesignTest Case Design
"Bugs lurk in corners"Bugs lurk in cornersand congregate atand congregate atboundaries ..."boundaries ..."
BorisBoris BeizerBeizer
OBJECTIVEOBJECTIVE
CRITERIACRITERIA
CONSTRAINTCONSTRAINT
to uncover errorsto uncover errors
in a complete mannerin a complete manner
with a minimum of effort and timewith a minimum of effort and time
Exhaustive TestingExhaustive Testing
loop < 20 Xloop < 20 X
There are 10 possible paths! If we execute oneThere are 10 possible paths! If we execute onetest per millisecond, it would take 3,170 years totest per millisecond, it would take 3,170 years totest this program!!test this program!!
1414
Selective TestingSelective Testing
loop < 20 Xloop < 20 X
Selected pathSelected path
Software TestingSoftware Testing
Methods
Strategies
white-boxmethods
black-boxmethods
WhiteWhite--Box TestingBox Testing
... our goal is to ensure that all... our goal is to ensure that allstatements and conditions havestatements and conditions havebeen executed at least once ...been executed at least once ...
Why Cover?Why Cover?
logic errors and incorrect assumptionslogic errors and incorrect assumptionsare inversely proportional to a path'sare inversely proportional to a path'sexecution probabilityexecution probability
we oftenwe oftenbelievebelievethat a path is notthat a path is notlikely to be executed; in fact, reality islikely to be executed; in fact, reality isoften counter intuitiveoften counter intuitive
typographical errors are random; it'stypographical errors are random; it'slikely that untested paths will containlikely that untested paths will containsomesome
Basis Path TestingBasis Path Testing
First, we compute the cyclomaticcomplexity:
number of simple decisions + 1
or
number of enclosed areas + 1
In this case, V(G) = 4
CyclomaticCyclomatic ComplexityComplexity
A number of industry studies have indicatedA number of industry studies have indicatedthat the higher V(G), the higher the probabilitythat the higher V(G), the higher the probabilityor errors.or errors.
V(G)V(G)
modulesmodules
modules in this range aremodules in this range aremore error pronemore error prone
Basis Path TestingBasis Path TestingNext, we derive theNext, we derive theindependent paths:independent paths:
Since V(G) = 4,Since V(G) = 4,there are four pathsthere are four paths
Path 1: 1,2,3,6,7,8Path 1: 1,2,3,6,7,8Path 2: 1,2,3,5,7,8Path 2: 1,2,3,5,7,8Path 3: 1,2,4,7,8Path 3: 1,2,4,7,8Path 4: 1,2,4,7,2,4,...7,8Path 4: 1,2,4,7,2,4,...7,8
Finally, we derive testFinally, we derive testcases to exercise thesecases to exercise thesepaths.paths.
11
22
3344
55 66
77
88
Basis Path Testing NotesBasis Path Testing Notes
you don't need a flow chart,you don't need a flow chart,but the picture will help whenbut the picture will help whenyou trace program pathsyou trace program paths
count each simple logical test,count each simple logical test,compound tests count as 2 orcompound tests count as 2 ormoremore
basis path testing should bebasis path testing should beapplied to critical modulesapplied to critical modules
Graph MatricesGraph Matrices A graph matrix is a square matrix whose sizeA graph matrix is a square matrix whose size
(i.e., number of rows and columns) is equal to(i.e., number of rows and columns) is equal tothe number of nodes on a flow graphthe number of nodes on a flow graph
Each row and column corresponds to anEach row and column corresponds to anidentified node, and matrix entries correspond toidentified node, and matrix entries correspond toconnections (an edge) between nodes.connections (an edge) between nodes.
By adding aBy adding a link weightlink weight to each matrix entry, theto each matrix entry, thegraph matrix can become a powerful tool forgraph matrix can become a powerful tool forevaluating program control structure duringevaluating program control structure duringtestingtesting
Control Structure TestingControl Structure Testing
Condition testingCondition testing —— a test case design method thata test case design method thatexercises the logical conditions contained in a programexercises the logical conditions contained in a programmodulemodule
Data flow testingData flow testing —— selects test paths of a programselects test paths of a programaccording to the locations of definitions and uses ofaccording to the locations of definitions and uses ofvariables in the programvariables in the program
Loop TestingLoop Testing
NestedNestedLoopsLoops
ConcatenatedConcatenatedLoopsLoops UnstructuredUnstructured
LoopsLoops
SimpleSimplelooploop
Loop Testing: Simple LoopsLoop Testing: Simple Loops
Minimum conditionsMinimum conditions——Simple LoopsSimple Loops
1. skip the loop entirely1. skip the loop entirely
2. only one pass through the loop2. only one pass through the loop
3. two passes through the loop3. two passes through the loop
4. m passes through the loop m < n4. m passes through the loop m < n
5. (n5. (n--1), n, and (n+1) passes through1), n, and (n+1) passes throughthe loopthe loop
where n is the maximum numberwhere n is the maximum numberof allowable passesof allowable passes
Loop Testing: Nested LoopsLoop Testing: Nested Loops
Start at the innermost loop. Set all outer loops to theirStart at the innermost loop. Set all outer loops to theirminimum iteration parameter values.minimum iteration parameter values.
Test the min+1, typical, maxTest the min+1, typical, max--1 and max for the1 and max for theinnermost loop, while holding the outer loops at theirinnermost loop, while holding the outer loops at theirminimum values.minimum values.
Move out one loop and set it up as in step 2, holding allMove out one loop and set it up as in step 2, holding allother loops at typical values. Continue this step untilother loops at typical values. Continue this step untilthe outermost loop has been tested.the outermost loop has been tested.
If the loops are independent of one anotherIf the loops are independent of one anotherthen treat each as a simple loopthen treat each as a simple loopelse* treat as nested loopselse* treat as nested loops
endifendif**
for example, the final loop counter value of loop 1 isfor example, the final loop counter value of loop 1 isused to initialize loop 2.used to initialize loop 2.
Nested LoopsNested Loops
Concatenated LoopsConcatenated Loops
BlackBlack--Box TestingBox Testing
requirementsrequirements
eventseventsinputinput
outputoutput
BlackBlack--Box TestingBox Testing
How is functional validity tested?How is functional validity tested?
How is system behavior and performance tested?How is system behavior and performance tested?
What classes of input will make good test cases?What classes of input will make good test cases?
Is the system particularly sensitive to certain inputIs the system particularly sensitive to certain inputvalues?values?
How are the boundaries of a data class isolated?How are the boundaries of a data class isolated?
What data rates and data volume can the systemWhat data rates and data volume can the systemtolerate?tolerate?
What effect will specific combinations of data haveWhat effect will specific combinations of data haveon system operation?on system operation?
GraphGraph--Based MethodsBased Methods
new
file
menu select generates
(generation time 1.0 sec)
document
window
document
text
is represented as
contains
Attributes:
background color: whitetext color: default color
or preferences
(b)
object#1
Directed link
(link weight)
object#2
object
#3
Undirected link
Parallel links
Node weight(value
)
(a)
allows editingof
To understand theTo understand theobjects that areobjects that aremodeled inmodeled insoftware and thesoftware and therelationships thatrelationships thatconnect theseconnect theseobjectsobjects
In this context, weIn this context, weconsider the termconsider the term““objectsobjects”” in the broadestin the broadestpossible context. Itpossible context. Itencompasses dataencompasses dataobjects, traditionalobjects, traditionalcomponents (modules),components (modules),and objectand object--orientedorientedelements of computerelements of computersoftware.software.
Equivalence PartitioningEquivalence Partitioning
useruserqueriesqueries
mousemousepickspicks
outputoutputformatsformats
promptsprompts
FKFKinputinput
datadata
Sample EquivalenceSample EquivalenceClassesClasses
user supplied commandsuser supplied commands
responses to system promptsresponses to system prompts
file namesfile namescomputational datacomputational data
physical parametersphysical parametersbounding valuesbounding valuesinitiation valuesinitiation values
output data formattingoutput data formattingresponses to error messagesresponses to error messagesgraphical data (e.g., mouse picks)graphical data (e.g., mouse picks)
data outside bounds of the programdata outside bounds of the programphysically impossible dataphysically impossible dataproper value supplied in wrong placeproper value supplied in wrong place
Valid dataValid data
Invalid dataInvalid data
Boundary Value AnalysisBoundary Value Analysis
useruserqueriesqueries
mousemousepickspicks
outputoutputformatsformats
promptsprompts
FKFKinputinput
datadata
outputoutputdomaindomaininput domaininput domain
Comparison TestingComparison Testing
Used only in situations in which the reliability of softwareUsed only in situations in which the reliability of softwareis absolutely critical (e.g., humanis absolutely critical (e.g., human--rated systems)rated systems)
Separate software engineering teams develop independentSeparate software engineering teams develop independentversions of an application using the same specificationversions of an application using the same specification
Each version can be tested with the same test data to ensureEach version can be tested with the same test data to ensurethat all provide identical outputthat all provide identical output
Then all versions are executed in parallel with realThen all versions are executed in parallel with real--timetimecomparison of results to ensure consistencycomparison of results to ensure consistency
Orthogonal Array TestingOrthogonal Array Testing
Used when the number of input parameters is small andthe values that each of the parameters may take areclearly bounded
One input item at a time L9 orthogonal array
XY
Z
XY
Z
OOTOOT——Test Case DesignTest Case Design
BerardBerard [BER93] proposes the following approach:[BER93] proposes the following approach:
1.1. Each test case should be uniquely identified and should be expliEach test case should be uniquely identified and should be explicitlycitlyassociated with the class to be tested,associated with the class to be tested,
2.2. The purpose of the test should be stated,The purpose of the test should be stated,
3.3. A list of testing steps should be developed for each test and shA list of testing steps should be developed for each test and shouldouldcontain [BER94]:contain [BER94]:
a.a. a list of specified states for the object that is to be testeda list of specified states for the object that is to be tested
b.b. a list of messages and operations that will be exercised asa list of messages and operations that will be exercised asa consequence of the testa consequence of the test
c.c. a list of exceptions that may occur as the object is testeda list of exceptions that may occur as the object is tested
d.d. a list of external conditions (i.e., changes in the environmenta list of external conditions (i.e., changes in the environment externalexternalto the software that must exist in order to properly conduct theto the software that must exist in order to properly conduct the test)test)
e.e. supplementary information that will aid in understanding orsupplementary information that will aid in understanding orimplementing the test.implementing the test.
Testing MethodsTesting Methods
FaultFault--based testingbased testing
The tester looks for plausible faults (i.e., aspects of the impThe tester looks for plausible faults (i.e., aspects of the implementation of thelementation of thesystem that may result in defects). To determine whether these fsystem that may result in defects). To determine whether these faults exist, testaults exist, testcases are designed to exercise the design or code.cases are designed to exercise the design or code.
Class Testing and the Class HierarchyClass Testing and the Class Hierarchy
Inheritance does not obviate the need for thorough testing of alInheritance does not obviate the need for thorough testing of all derived classes.l derived classes.In fact, it can actually complicate the testing process.In fact, it can actually complicate the testing process.
ScenarioScenario--Based Test DesignBased Test Design
ScenarioScenario--based testing concentrates on what the user does, not what thebased testing concentrates on what the user does, not what theproduct does. This means capturing the tasks (via useproduct does. This means capturing the tasks (via use--cases) that the user hascases) that the user hasto perform, then applying them and their variants as tests.to perform, then applying them and their variants as tests.
OOT Methods: Random TestingOOT Methods: Random Testing
Random testingRandom testing
identify operations applicable to a classidentify operations applicable to a class
define constraints on their usedefine constraints on their use
identify aidentify a miminummiminum test sequencetest sequence
an operation sequence that defines the minimum lifean operation sequence that defines the minimum lifehistory of the class (object)history of the class (object)
generate a variety of random (but valid) test sequencesgenerate a variety of random (but valid) test sequences
exercise other (more complex) class instance lifeexercise other (more complex) class instance lifehistorieshistories
OOT Methods: Partition TestingOOT Methods: Partition Testing
Partition TestingPartition Testing reduces the number of test cases required to test a class inreduces the number of test cases required to test a class in
much the same way as equivalence partitioning formuch the same way as equivalence partitioning forconventional softwareconventional software
statestate--based partitioningbased partitioning categorize and test operations based on their ability to changecategorize and test operations based on their ability to change
the state of a classthe state of a class
attributeattribute--based partitioningbased partitioning categorize and test operations based on the attributes that theycategorize and test operations based on the attributes that they
useuse
categorycategory--based partitioningbased partitioning categorize and test operations based on the generic functioncategorize and test operations based on the generic function
each performseach performs
OOT Methods: InterOOT Methods: Inter--Class TestingClass Testing
InterInter--class testingclass testing For each client class, use the list of class operators toFor each client class, use the list of class operators to
generate a series of random test sequences. The operatorsgenerate a series of random test sequences. The operatorswill send messages to other server classes.will send messages to other server classes.
For each message that is generated, determine theFor each message that is generated, determine thecollaborator class and the corresponding operator in thecollaborator class and the corresponding operator in theserver object.server object.
For each operator in the server object (that has been invokedFor each operator in the server object (that has been invokedby messages sent from the client object), determine theby messages sent from the client object), determine themessages that it transmits.messages that it transmits.
For each of the messages, determine the next level ofFor each of the messages, determine the next level ofoperators that are invoked and incorporate these into the testoperators that are invoked and incorporate these into the testsequencesequence
OOT Methods: Behavior TestingOOT Methods: Behavior Testing
emptyacctopen setupAccnt
set upacct
deposit(initial)
workingacct
withdrawal(final)
deadacct close
nonworkingacct
deposit
withdrawbalance
creditaccntInfo
Figure 14.3 St at e diagram f or Account class (adapt ed f rom [ KIR94] )
The tests to beThe tests to bedesigned shoulddesigned shouldachieve all stateachieve all statecoverage [KIR94].coverage [KIR94].That is, theThat is, theoperationoperationsequences shouldsequences shouldcause thecause theAccount class toAccount class tomake transitionmake transitionthrough allthrough allallowable statesallowable states
Testing PatternsTesting Patterns
Pattern name:Pattern name: pair testingpair testing
Abstract:Abstract: A processA process--oriented pattern, pair testing describes a technique that isoriented pattern, pair testing describes a technique that isanalogous to pair programming (Chapter 4) in which two testers wanalogous to pair programming (Chapter 4) in which two testers work togetherork togetherto design and execute a series of tests that can be applied to uto design and execute a series of tests that can be applied to unit, integrationnit, integrationor validation testing activities.or validation testing activities.
Pattern name:Pattern name: separate test interfaceseparate test interface
Abstract:Abstract: There is a need to test every class in an objectThere is a need to test every class in an object--oriented system,oriented system,includingincluding ““internal classesinternal classes”” (i.e., classes that do not expose any interface(i.e., classes that do not expose any interfaceoutside of the component that used them). The separate test inteoutside of the component that used them). The separate test interface patternrface patterndescribes how to createdescribes how to create ““a test interface that can be used to describe specifica test interface that can be used to describe specifictests on classes that are visible only internally to a componenttests on classes that are visible only internally to a component..”” [LAN01][LAN01]
Pattern name:Pattern name: scenario testingscenario testingAbstract:Abstract: Once unit and integration tests have been conducted, there is aOnce unit and integration tests have been conducted, there is a needneedto determine whether the software will perform in a manner thatto determine whether the software will perform in a manner that satisfies users.satisfies users.The scenario testing pattern describes a technique for exercisinThe scenario testing pattern describes a technique for exercising the softwareg the softwarefrom the userfrom the user’’s point of view. A failure at this level indicates that the softs point of view. A failure at this level indicates that the softwarewarehas failed to meet a user visible requirement. [KAN01]has failed to meet a user visible requirement. [KAN01]
Top Related