Chapter 13 Software Testing Strategies - Aalborg...
Transcript of Chapter 13 Software Testing Strategies - Aalborg...
These coursew are materials are to be used in conjunction w ith Soft ware Engineering: A Pract it ioner’s Approach,6/e and are provided w ith permission by R.S. Pressman & Associates, Inc., copyright © 1996, 2001, 2005 1
Software Engineering: A PractitionerSoftware Engineering: A Practitioner’’s Approach, 6/es Approach, 6/e
Chapter 13Chapter 13Software Testing StrategiesSoftware Testing Strategies
copyright © 1996, 2001, 2005
R.S. Pressman & Associates, Inc.
For University Use OnlyMay be reproduced ONLY for student use at the university level
when used in conjunction with Software Engineering: A Practitioner's Approach.Any other reproduction or use is expressly prohibited.
These coursew are materials are to be used in conjunction w ith Soft ware Engineering: A Pract it ioner’s Approach,6/e and are provided w ith permission by R.S. Pressman & Associates, Inc., copyright © 1996, 2001, 2005 2
Software TestingSoftware Testing
Testing is the process of exercising aTesting is the process of exercising aprogram with the specific intent of findingprogram with the specific intent of findingerrors prior to delivery to the end user.errors prior to delivery to the end user.
These coursew are materials are to be used in conjunction w ith Soft ware Engineering: A Pract it ioner’s Approach,6/e and are provided w ith permission by R.S. Pressman & Associates, Inc., copyright © 1996, 2001, 2005 3
What Testing ShowsWhat Testing Shows
errorserrorsrequirements conformancerequirements conformance
performanceperformance
an indicationan indicationof qualityof quality
These coursew are materials are to be used in conjunction w ith Soft ware Engineering: A Pract it ioner’s Approach,6/e and are provided w ith permission by R.S. Pressman & Associates, Inc., copyright © 1996, 2001, 2005 4
Who Tests the Software?Who Tests the Software?
developerdeveloper independent testerindependent testerUnderstands the system Understands the system but, will test "gently"but, will test "gently"and, is driven by "delivery"and, is driven by "delivery"
Must learn about the system,Must learn about the system,but, will attempt to break itbut, will attempt to break itand, is driven by qualityand, is driven by quality
These coursew are materials are to be used in conjunction w ith Soft ware Engineering: A Pract it ioner’s Approach,6/e and are provided w ith permission by R.S. Pressman & Associates, Inc., copyright © 1996, 2001, 2005 5
Testing StrategyTesting Strategyunit testunit test integrationintegration
testtest
validationvalidationtesttest
systemsystemtesttest
V ModelV Model
These coursew are materials are to be used in conjunction w ith Soft ware Engineering: A Pract it ioner’s Approach,6/e and are provided w ith permission by R.S. Pressman & Associates, Inc., copyright © 1996, 2001, 2005 7
Testing StrategyTesting Strategy
We begin by We begin by ‘‘testing-in-the-smalltesting-in-the-small’’ and move toward and move toward‘‘testing-in-the-largetesting-in-the-large’’
For conventional softwareFor conventional software The module (component) is our initial focusThe module (component) is our initial focus Integration of modules followsIntegration of modules follows
For OO softwareFor OO software our focus when our focus when ““testing in the smalltesting in the small”” changes from an changes from an
individual module (the conventional view) to an OO class thatindividual module (the conventional view) to an OO class thatencompasses attributes and operations and impliesencompasses attributes and operations and impliescommunication and collaborationcommunication and collaboration
These coursew are materials are to be used in conjunction w ith Soft ware Engineering: A Pract it ioner’s Approach,6/e and are provided w ith permission by R.S. Pressman & Associates, Inc., copyright © 1996, 2001, 2005 8
Strategic IssuesStrategic Issues
State testing objectives explicitly.State testing objectives explicitly. Understand the users of the software and develop a profile for each user category.Understand the users of the software and develop a profile for each user category. Develop a testing plan that emphasizes Develop a testing plan that emphasizes ““rapid cycle testing.rapid cycle testing.”” Build Build ““robustrobust”” software that is designed to test itself software that is designed to test itself Use effective formal technical reviews as a filter prior to testingUse effective formal technical reviews as a filter prior to testing Conduct formal technical reviews to assess the test strategy and test casesConduct formal technical reviews to assess the test strategy and test cases
themselves.themselves. Develop a continuous improvement approach for the testing process.Develop a continuous improvement approach for the testing process.
These coursew are materials are to be used in conjunction w ith Soft ware Engineering: A Pract it ioner’s Approach,6/e and are provided w ith permission by R.S. Pressman & Associates, Inc., copyright © 1996, 2001, 2005 9
Unit TestingUnit Testing
modulemoduleto beto be
testedtested
test casestest cases
resultsresults
softwaresoftwareengineerengineer
These coursew are materials are to be used in conjunction w ith Soft ware Engineering: A Pract it ioner’s Approach,6/e and are provided w ith permission by R.S. Pressman & Associates, Inc., copyright © 1996, 2001, 2005 10
Unit TestingUnit Testing
interface interface local data structureslocal data structuresboundary conditionsboundary conditionsindependent pathsindependent pathserror handling pathserror handling paths
modulemoduleto beto be
testedtested
test casestest cases
These coursew are materials are to be used in conjunction w ith Soft ware Engineering: A Pract it ioner’s Approach,6/e and are provided w ith permission by R.S. Pressman & Associates, Inc., copyright © 1996, 2001, 2005 11
Unit Test EnvironmentUnit Test Environment
ModuleModule
stubstub stubstub
driverdriver
RESULTSRESULTS
interface interface local data structureslocal data structuresboundary conditionsboundary conditionsindependent pathsindependent pathserror handling pathserror handling paths
test casestest cases
These coursew are materials are to be used in conjunction w ith Soft ware Engineering: A Pract it ioner’s Approach,6/e and are provided w ith permission by R.S. Pressman & Associates, Inc., copyright © 1996, 2001, 2005 12
Integration Testing StrategiesIntegration Testing StrategiesOptions:Options:
•• the the ““big bangbig bang”” approach approach•• an incremental construction strategyan incremental construction strategy
These coursew are materials are to be used in conjunction w ith Soft ware Engineering: A Pract it ioner’s Approach,6/e and are provided w ith permission by R.S. Pressman & Associates, Inc., copyright © 1996, 2001, 2005 13
Top Down IntegrationTop Down Integration
top module is tested with top module is tested with stubsstubs
stubs are replaced one at stubs are replaced one at a time, "depth first"a time, "depth first"
as new modules are integrated, as new modules are integrated, some subset of tests is re-runsome subset of tests is re-run
AA
BB
CC
DD EE
FF GG
These coursew are materials are to be used in conjunction w ith Soft ware Engineering: A Pract it ioner’s Approach,6/e and are provided w ith permission by R.S. Pressman & Associates, Inc., copyright © 1996, 2001, 2005 14
Bottom-Up IntegrationBottom-Up Integration
drivers are replaced one at a drivers are replaced one at a time, "depth first"time, "depth first"
worker modules are grouped into worker modules are grouped into builds and integratedbuilds and integrated
AA
BB
CC
DD EE
FF GG
clustercluster
These coursew are materials are to be used in conjunction w ith Soft ware Engineering: A Pract it ioner’s Approach,6/e and are provided w ith permission by R.S. Pressman & Associates, Inc., copyright © 1996, 2001, 2005 15
Sandwich TestingSandwich Testing
Top modules areTop modules aretested with stubstested with stubs
Worker modules are grouped into Worker modules are grouped into builds and integratedbuilds and integrated
AA
BB
CC
DD EE
FF GG
clustercluster
These coursew are materials are to be used in conjunction w ith Soft ware Engineering: A Pract it ioner’s Approach,6/e and are provided w ith permission by R.S. Pressman & Associates, Inc., copyright © 1996, 2001, 2005 16
Object-Oriented TestingObject-Oriented Testing
begins by evaluating the correctness and consistency ofbegins by evaluating the correctness and consistency ofthe OOA and OOD modelsthe OOA and OOD models
testing strategy changestesting strategy changes the concept of the the concept of the ‘‘unitunit’’ broadens due to encapsulation broadens due to encapsulation integration focuses on classes and their execution across aintegration focuses on classes and their execution across a
‘‘threadthread’’ or in the context of a usage scenario or in the context of a usage scenario validation uses conventional black box methodsvalidation uses conventional black box methods
test case design draws on conventional methods, buttest case design draws on conventional methods, butalso encompasses special featuresalso encompasses special features
These coursew are materials are to be used in conjunction w ith Soft ware Engineering: A Pract it ioner’s Approach,6/e and are provided w ith permission by R.S. Pressman & Associates, Inc., copyright © 1996, 2001, 2005 17
Broadening the View of Broadening the View of ““TestingTesting””
It can be argued that the review of OO analysisIt can be argued that the review of OO analysisand design models is especially useful because theand design models is especially useful because thesame semantic constructs (e.g., classes, attributes,same semantic constructs (e.g., classes, attributes,operations, messages) appear at the analysis,operations, messages) appear at the analysis,design, and code level. Therefore, a problem in thedesign, and code level. Therefore, a problem in thedefinition of class attributes that is uncovereddefinition of class attributes that is uncoveredduring analysis will circumvent side effects thatduring analysis will circumvent side effects thatmight occur if the problem were not discoveredmight occur if the problem were not discovereduntil design or code (or even the next iteration ofuntil design or code (or even the next iteration ofanalysis).analysis).
These coursew are materials are to be used in conjunction w ith Soft ware Engineering: A Pract it ioner’s Approach,6/e and are provided w ith permission by R.S. Pressman & Associates, Inc., copyright © 1996, 2001, 2005 18
Testing the CRC ModelTesting the CRC Model
1. Revisit the CRC model and the object-relationship model.1. Revisit the CRC model and the object-relationship model.
2. Inspect the description of each CRC index card to determine if a2. Inspect the description of each CRC index card to determine if adelegated responsibility is part of the collaboratordelegated responsibility is part of the collaborator’’s definition.s definition.
3. Invert the connection to ensure that each collaborator that is3. Invert the connection to ensure that each collaborator that isasked for service is receiving requests from a reasonable source.asked for service is receiving requests from a reasonable source.
4. Using the inverted connections examined in step 3, determine4. Using the inverted connections examined in step 3, determinewhether other classes might be required or whetherwhether other classes might be required or whetherresponsibilities are properly grouped among the classes.responsibilities are properly grouped among the classes.
5. Determine whether widely requested responsibilities might be5. Determine whether widely requested responsibilities might becombined into a single responsibility.combined into a single responsibility.
6. Steps 1 to 5 are applied6. Steps 1 to 5 are applied iteratively iteratively to each class and through to each class and througheach evolution of the OOA model.each evolution of the OOA model.
These coursew are materials are to be used in conjunction w ith Soft ware Engineering: A Pract it ioner’s Approach,6/e and are provided w ith permission by R.S. Pressman & Associates, Inc., copyright © 1996, 2001, 2005 19
OOT StrategyOOT Strategy
class testing is the equivalent of unit testingclass testing is the equivalent of unit testing operations within the class are testedoperations within the class are tested the state behavior of the class is examinedthe state behavior of the class is examined
integration applieintegration appliess three different strategies three different strategies thread-based testingthread-based testing——integrates the set of classesintegrates the set of classes
required to respond to one input or eventrequired to respond to one input or event use-based testinguse-based testing——integrates the set of classes requiredintegrates the set of classes required
to respond to one use caseto respond to one use case cluster testingcluster testing——integrates the set of classes required tointegrates the set of classes required to
demonstrate one collaborationdemonstrate one collaboration
These coursew are materials are to be used in conjunction w ith Soft ware Engineering: A Pract it ioner’s Approach,6/e and are provided w ith permission by R.S. Pressman & Associates, Inc., copyright © 1996, 2001, 2005 20
Smoke TestingSmoke Testing A common approach for creating A common approach for creating ““daily buildsdaily builds”” for product for product
softwaresoftware Smoke testing steps:Smoke testing steps:
Software components that have been translated into code areSoftware components that have been translated into code areintegrated into a integrated into a ““build.build.””
A build includes all data files, libraries, reusable modules, andA build includes all data files, libraries, reusable modules, andengineered components that are required to implement one or moreengineered components that are required to implement one or moreproduct functions.product functions.
A series of tests is designed to expose errors that will keep the buildA series of tests is designed to expose errors that will keep the buildfrom properly performing its function.from properly performing its function.
The intent should be to uncover The intent should be to uncover ““show stoppershow stopper”” errors that have the errors that have thehighest likelihood of throwing the software project behind schedule.highest likelihood of throwing the software project behind schedule.
The build is integrated with other builds and the entire product (inThe build is integrated with other builds and the entire product (inits current form) is smoke tested daily.its current form) is smoke tested daily.
The integration approach may be top down or bottom up.The integration approach may be top down or bottom up.
These coursew are materials are to be used in conjunction w ith Soft ware Engineering: A Pract it ioner’s Approach,6/e and are provided w ith permission by R.S. Pressman & Associates, Inc., copyright © 1996, 2001, 2005 21
High Order TestingHigh Order Testing Validation testingValidation testing
Focus is on software requirementsFocus is on software requirements
System testingSystem testing Focus is on system integrationFocus is on system integration
Alpha/Beta testingAlpha/Beta testing Focus is on customer usageFocus is on customer usage
Recovery testingRecovery testing forces the software to fail in a variety of ways and verifies that recovery isforces the software to fail in a variety of ways and verifies that recovery is
properly performedproperly performed
Security testingSecurity testing verifies that protection mechanisms built into a system will, in fact, protect itverifies that protection mechanisms built into a system will, in fact, protect it
from improper penetrationfrom improper penetration
Stress testingStress testing executes a system in a manner that demands resources in abnormal quantity, executes a system in a manner that demands resources in abnormal quantity,
frequency, or volumefrequency, or volume
Performance TestingPerformance Testing test the run-time performance of software within the context of an integratedtest the run-time performance of software within the context of an integrated
systemsystem
These coursew are materials are to be used in conjunction w ith Soft ware Engineering: A Pract it ioner’s Approach,6/e and are provided w ith permission by R.S. Pressman & Associates, Inc., copyright © 1996, 2001, 2005 22
Debugging:Debugging:A Diagnostic ProcessA Diagnostic Process
These coursew are materials are to be used in conjunction w ith Soft ware Engineering: A Pract it ioner’s Approach,6/e and are provided w ith permission by R.S. Pressman & Associates, Inc., copyright © 1996, 2001, 2005 23
The Debugging ProcessThe Debugging Processtest casestest cases
resultsresults
DebuggingDebugging
suspectedsuspectedcausescauses
identifiedidentifiedcausescauses
correctionscorrections
regressionregressionteststests
new testnew testcasescases
These coursew are materials are to be used in conjunction w ith Soft ware Engineering: A Pract it ioner’s Approach,6/e and are provided w ith permission by R.S. Pressman & Associates, Inc., copyright © 1996, 2001, 2005 24
Debugging EffortDebugging Effort
time requiredtime requiredto diagnose theto diagnose thesymptom andsymptom anddetermine thedetermine thecausecause
time requiredtime requiredto correct the errorto correct the errorand conductand conductregression testsregression tests
These coursew are materials are to be used in conjunction w ith Soft ware Engineering: A Pract it ioner’s Approach,6/e and are provided w ith permission by R.S. Pressman & Associates, Inc., copyright © 1996, 2001, 2005 25
Symptoms & CausesSymptoms & Causes
symptomsymptomcausecause
symptom and cause may be symptom and cause may be geographically separated geographically separated
symptom may disappear when symptom may disappear when another problem is fixedanother problem is fixed
cause may be due to a cause may be due to a combination of non-errors combination of non-errors
cause may be due to a system cause may be due to a system or compiler erroror compiler error
cause may be due to cause may be due to assumptions that everyone assumptions that everyone believesbelieves
symptom may be intermittentsymptom may be intermittent
These coursew are materials are to be used in conjunction w ith Soft ware Engineering: A Pract it ioner’s Approach,6/e and are provided w ith permission by R.S. Pressman & Associates, Inc., copyright © 1996, 2001, 2005 26
Consequences of BugsConsequences of Bugs
damagedamage
mildmild annoyingannoyingdisturbingdisturbing
seriousseriousextremeextreme
catastrophiccatastrophic
infectiousinfectious
Bug TypeBug Type
Bug Categories:Bug Categories: function-related bugs, function-related bugs, system-related bugs, data bugs, coding bugs, system-related bugs, data bugs, coding bugs, design bugs, documentation bugs, standards design bugs, documentation bugs, standards violations, etc.violations, etc.
These coursew are materials are to be used in conjunction w ith Soft ware Engineering: A Pract it ioner’s Approach,6/e and are provided w ith permission by R.S. Pressman & Associates, Inc., copyright © 1996, 2001, 2005 27
Debugging TechniquesDebugging Techniques
brute force / testingbrute force / testing
backtrackingbacktracking
inductioninduction
deductiondeduction
These coursew are materials are to be used in conjunction w ith Soft ware Engineering: A Pract it ioner’s Approach,6/e and are provided w ith permission by R.S. Pressman & Associates, Inc., copyright © 1996, 2001, 2005 28
Debugging: Final ThoughtsDebugging: Final Thoughts
Don't run off half-cocked, Don't run off half-cocked, thinkthink about the about the symptom you're seeing.symptom you're seeing.
Use toolsUse tools (e.g., dynamic debugger) to gain (e.g., dynamic debugger) to gain more insight.more insight.
If at an impasse, If at an impasse, get helpget help from someone else. from someone else.
Be absolutely sure to Be absolutely sure to conduct regression testsconduct regression tests when you do "fix" the bug.when you do "fix" the bug.
1.1.
2.2.
3.3.
4.4.