1

Software Testing StrategiesSoftware Testing Strategies

based onbased onChapter 13 - Chapter 13 - Software Engineering: A Practitioner’s Software Engineering: A Practitioner’s

Approach, 6/eApproach, 6/ecopyright © 1996, 2001, 2005

R.S. Pressman & Associates, Inc.For University Use Only

May be reproduced ONLY for student use at the university levelwhen used in conjunction with Software Engineering: A Practitioner's Approach.

Any other reproduction or use is expressly prohibited.

2

Software Software TestingTestingTesting is the process of exercising a program with the Testing is the process of exercising a program with the

specific intent of finding errors prior to delivery specific intent of finding errors prior to delivery to the end user.to the end user.

errorserrors

requirements conformancerequirements conformance

performanceperformance

an indicationan indicationof qualityof quality

3

Who Tests the Who Tests the Software?Software?

developerdeveloper independent testerindependent tester

Understands the system Understands the system

but, will test "gently"but, will test "gently"

and, is driven by "and, is driven by "deliverydelivery""

Must learn about the system,Must learn about the system,but, will attempt to but, will attempt to breakbreak it it

and, is driven by and, is driven by qualityquality

4

Levels of TestingLevels of Testing Unit testingUnit testing Integration testingIntegration testing Validation testingValidation testing

Focus is on software requirementsFocus is on software requirements System testingSystem testing

Focus is on system integrationFocus is on system integration Alpha/Beta testingAlpha/Beta testing

Focus is on customer usageFocus is on customer usage Recovery testingRecovery testing

forces the software to fail in a variety of ways and verifies that recovery is properly performedforces the software to fail in a variety of ways and verifies that recovery is properly performed Security testingSecurity testing

verifies that protection mechanisms built into a system will, in fact, protect it from improper verifies that protection mechanisms built into a system will, in fact, protect it from improper penetrationpenetration

Stress testingStress testing executes a system in a manner that demands resources in abnormal quantity, frequency, or volumeexecutes a system in a manner that demands resources in abnormal quantity, frequency, or volume

Performance TestingPerformance Testing test the run-time performance of software within the context of an integrated systemtest the run-time performance of software within the context of an integrated system

5

Unit TestingUnit Testing

modulemoduleto beto betestedtested

test casestest cases

resultsresults

softwaresoftwareengineerengineer

interface interface

local data structureslocal data structures

boundary conditionsboundary conditions

independent pathsindependent paths

error handling pathserror handling paths

6

Integration Testing StrategiesIntegration Testing StrategiesOptions:Options:

•• the “big bang” approachthe “big bang” approach•• an incremental construction strategyan incremental construction strategy

• Top Down, bottom-up, sandwich Top Down, bottom-up, sandwich IntegrationIntegration

7

OOT OOT StrategyStrategy class testing is the equivalent of unit testingclass testing is the equivalent of unit testing

operationsoperations within the class are tested within the class are tested the the state behaviorstate behavior of the class is examined of the class is examined

integration applied three different integration applied three different strategies/levels of abstractionstrategies/levels of abstraction thread-basedthread-based testing—integrates the set of testing—integrates the set of

classes required to respond to classes required to respond to one input or eventone input or event use-baseduse-based testing—integrates the set of classes testing—integrates the set of classes

required to respond to required to respond to one use caseone use case clustercluster testing—integrates the set of classes testing—integrates the set of classes

required to demonstrate required to demonstrate one collaborationone collaboration

…if there is no nesting of classes

…this is pushing…

8

test casestest cases

resultsresults

DebuggingDebugging

suspectedsuspectedcausescauses

identifiedidentifiedcausescauses

correctionscorrections

regressionregressionteststests

new testnew testcasescases

Debugging: A Diagnostic ProcessDebugging: A Diagnostic Process

9

Software Testing Software Testing TechniquesTechniques

based onbased onChapter 14 - Chapter 14 - Software Engineering: A Practitioner’s Software Engineering: A Practitioner’s

Approach, 6/eApproach, 6/e copyright © 1996, 2001, 2005

R.S. Pressman & Associates, Inc.For University Use Only

May be reproduced ONLY for student use at the university levelwhen used in conjunction with Software Engineering: A Practitioner's Approach.

Any other reproduction or use is expressly prohibited.

10

What is a “Good” Test?What is a “Good” Test? a high probability of finding an errora high probability of finding an error not redundant.not redundant. neither too simple nor too complexneither too simple nor too complex

"Bugs lurk in corners "Bugs lurk in corners and congregate at and congregate at boundaries ..."boundaries ..."

Boris BeizerBoris Beizer

OBJECTIVE:OBJECTIVE:

CRITERIA:CRITERIA:

CONSTRAINT:CONSTRAINT:

to uncover errorsto uncover errors

in a complete mannerin a complete manner

with a minimum of effort and timewith a minimum of effort and time

11

Exhaustive TestingExhaustive Testing

loop < 20 Xloop < 20 X

There are approx. 10 possible paths! If we execute oneThere are approx. 10 possible paths! If we execute onetest per millisecond, it would take 3,170 years totest per millisecond, it would take 3,170 years to test this program!!test this program!!

1414

Where does 10 14 come from?

White-Box or Black-Box?White-Box or Black-Box?

12

RE in V ModelRE in V Model

systemrequireme

nts

systemintegration

softwarerequireme

ntspreliminar

ydesign

detaileddesign

code &debug

acceptancetest

softwareintegratio

ncomponen

ttest

unittest

Time

Leve

l of

ab

stra

cti

on

analyze an

d d

esign

test and in

tegrate

[Chung]

13

Software TestingSoftware Testing

White-Box TestingWhite-Box Testing

... our goal is to ensure that ... our goal is to ensure that allall statements and conditionsstatements and conditions have have

been executed at least been executed at least onceonce ... ...

Black-BoxBlack-Box Testing Testing

requirementsrequirements

eventseventsinputinput

outputoutput

14

BasisBasis Path Path TestingTesting

First, we compute the cyclomatic complexity:

number of simple decisions + 1

or

number of enclosed areas + 1

In this case, V(G) = 4

White-Box TestingWhite-Box Testing

15

BasisBasis Path Path TestingTestingNext, we derive the Next, we derive the

independent paths:independent paths:

Since V(G) = 4,Since V(G) = 4,there are four pathsthere are four paths

Path 3: 1,2,3,6,7,8Path 3: 1,2,3,6,7,8Path 2: 1,2,3,5,7,8Path 2: 1,2,3,5,7,8Path 1: 1,2,4,7,8Path 1: 1,2,4,7,8

Path 4: 1,2,4,7,2,4,...7,8Path 4: 1,2,4,7,2,4,...7,8

Finally, we derive testFinally, we derive testcases to exercise these cases to exercise these paths.paths.

11

22

3344

55 66

77

88

White-Box TestingWhite-Box Testing

A number of industry studies have indicated A number of industry studies have indicated that the higher V(G), the higher the probability that the higher V(G), the higher the probability or errors.or errors.

16

Loop TestingLoop Testing

Nested Nested LoopsLoops

ConcatenatedConcatenated Loops Loops Unstructured Unstructured

LoopsLoops

Simple Simple looploop

White-Box TestingWhite-Box Testing

Why is loop testing important?

17

Equivalence Partitioning & Equivalence Partitioning & Boundary Value AnalysisBoundary Value Analysis

If x = 5 then …

What would be the equivalence classes?

If x > -5 and x < 5 then …

White-Box TestingWhite-Box TestingBlack-Box Black-Box

TestingTesting

18

Comparison TestingComparison Testing

Used only in situations in which the reliability of Used only in situations in which the reliability of software is absolutely critical (e.g., human-rated software is absolutely critical (e.g., human-rated systems)systems) Separate software engineering Separate software engineering teamsteams develop develop

independentindependent versionsversions of an application using the of an application using the same specificationsame specification

Each version can be tested with the Each version can be tested with the same test datasame test data to to ensure that all provide identical output ensure that all provide identical output

Then all versions are Then all versions are executed in parallelexecuted in parallel with real- with real-time comparison of results to ensure consistencytime comparison of results to ensure consistency

Black-BoxBlack-Box Testing Testing

19

OOT—Test Case OOT—Test Case DesignDesignBerard [BER93] proposes the following approach:Berard [BER93] proposes the following approach:

1.1. Each test case should be uniquely identified and should be explicitly Each test case should be uniquely identified and should be explicitly associated with the associated with the classclass to be tested, to be tested,

2.2. A list of testing steps should be developed for each test and should A list of testing steps should be developed for each test and should contain [BER94]:contain [BER94]:

a.a. a list of specified a list of specified statesstates for the object that is to be tested for the object that is to be tested

b.b. a list of a list of messages and operationsmessages and operations that will be exercised as a that will be exercised as a consequence of the testconsequence of the test how can this be done?how can this be done?

c.c. a list of a list of exceptionsexceptions that may occur as the object is tested that may occur as the object is tested

d.d. a list of a list of external conditionsexternal conditions (i.e., changes in the environment external (i.e., changes in the environment external to the software that must exist in order to properly conduct the test)to the software that must exist in order to properly conduct the test)

{people, machine, time of operation, etc.}

20

OOT Methods: Behavior TestingOOT Methods: Behavior Testing

emptyacctopen setup Accnt

set upacct

deposit(initial)

workingacct

withdrawal(final)

deadacct close

nonworkingacct

deposit

withdrawbalance

creditaccntInfo

Figure 14.3 State diagram for Account class (adapted from [ KIR94])

The tests to be The tests to be designed should designed should achieve achieve all state all state coveragecoverage [KIR94]. [KIR94]. That is, the That is, the operation operation sequences should sequences should cause the Account cause the Account class to make class to make transition through transition through all allowable statesall allowable states

Is the set of initial input data enough?

21

Omitted Omitted SlidesSlides

22

TestabiliTestabilityty

OperabilityOperability—it operates cleanly—it operates cleanly ObservabilityObservability—the results of each test case are readily observed—the results of each test case are readily observed ControllabilityControllability—the degree to which testing can be automated and —the degree to which testing can be automated and

optimizedoptimized DecomposabilityDecomposability—testing can be targeted—testing can be targeted SimplicitySimplicity—reduce complex architecture and logic to simplify tests—reduce complex architecture and logic to simplify tests StabilityStability—few changes are requested during testing—few changes are requested during testing UnderstandabilityUnderstandability—of the design—of the design

23

Strategic IssuesStrategic Issues

Understand the Understand the usersusers of the software and develop a of the software and develop a profile for each user profile for each user categorycategory..

Develop a Develop a testing plantesting plan that emphasizes “rapid cycle testing.” that emphasizes “rapid cycle testing.”

Use effective Use effective formal technical reviewsformal technical reviews as a filter prior to testing as a filter prior to testing

Conduct Conduct formal technical reviewsformal technical reviews to assess the test strategy and test cases to assess the test strategy and test cases themselves. themselves.

24

Counting BugsCounting Bugs

Sometimes reliability requirements take the form:Sometimes reliability requirements take the form:"The software shall have no more than X bugs/1K LOC""The software shall have no more than X bugs/1K LOC"But how do we measure bugs at delivery time?But how do we measure bugs at delivery time?

Bebugging Process - based on a Monte Carlo technique for statistical analysis of random Bebugging Process - based on a Monte Carlo technique for statistical analysis of random events.events.1. before testing, a known number of bugs (seeded bugs) are secretly inserted.1. before testing, a known number of bugs (seeded bugs) are secretly inserted.2. estimate the number of bugs in the system2. estimate the number of bugs in the system3. remove (both known and new) bugs.3. remove (both known and new) bugs.

# of detected seeded bugs/ # of seeded bugs = # of detected bugs/ # of bugs in the system # of detected seeded bugs/ # of seeded bugs = # of detected bugs/ # of bugs in the system # of bugs in the system # of bugs in the system = = # of seeded bugs x # of detected bugs# of seeded bugs x # of detected bugs / /# of detected seeded # of detected seeded bugsbugs

Example: Example: secretely seed 10 bugssecretely seed 10 bugsan independent test team detects 120 bugs (6 for the seeded)an independent test team detects 120 bugs (6 for the seeded)# of bugs in the system # of bugs in the system = = 10 x 120/6 10 x 120/6 = = 200200# of bugs in the system after removal = 200 - 120 - 4 = 76# of bugs in the system after removal = 200 - 120 - 4 = 76

But, deadly bugs vs. insignifant ones; not all bugs are equally detectable; ( Suggestion But, deadly bugs vs. insignifant ones; not all bugs are equally detectable; ( Suggestion [Musa87]:[Musa87]:

"No more than X bugs/1K LOC may be detected during testing""No more than X bugs/1K LOC may be detected during testing""No more than X bugs/1K LOC may be remain after delivery,"No more than X bugs/1K LOC may be remain after delivery,

as calculated by the Monte Carlo seeding technique"as calculated by the Monte Carlo seeding technique"

NFRs:NFRs:ReliabilityReliability

 

[Chung, RE Lecture Notes]][Chung, RE Lecture Notes]]

25

Cyclomatic Cyclomatic ComplexityComplexityA number of industry studies have indicated A number of industry studies have indicated

that the higher V(G), the higher the probability that the higher V(G), the higher the probability or errors.or errors.

V(G)V(G)

modulesmodules

modules in this range are modules in this range are more error pronemore error prone

White-Box TestingWhite-Box Testing