3103131031 TESTINGTESTING1
TESTINGTESTINGTESTINGTESTING
310313310313SOFTWARE ENGINEERINGSOFTWARE ENGINEERING
310313310313SOFTWARE ENGINEERINGSOFTWARE ENGINEERING
3103131031 TESTINGTESTING2
LEARNIING OBJECTIIVES
1. Understand the dddddddddd ddddddd ddddd ddd derificat ion ddd software validat ion .
2. Understand the dddddddddd dd ddd dddddd ddddddd ddd dddddd ddddddd.
3. Understand three basic st rategies that can be uddd togener at e syst emt est cases.
4. Understand a st r at egy for perform ing different kinds of software test ing .
3103131031 TESTINGTESTING3
TESTING OUTLINETESTING OUTLINE
The Unified Process and Testing — Overview
– Testing — Life Cycle Role
– Testing Overview
– Testing — Verification & Validation
– Testing Truths
The Unified Development Process — Testing Activities– Plan Tests
– Design Tests: White Box; Black Box; Regression
– Implement Tests
– Perform Tests: Unit; Integration; System; Acceptance
– Evaluate Tests
9
3103131031 TESTINGTESTING4
Inception Elaboration Construction Transition
TESTING LIFE CYCLE ROLETESTING LIFE CYCLE ROLE
PhasesCore Workflows
Requirements
Analysis
Design
Implementation
Testing
iter.#1
iter.#2
— — — — —iter.#n-1
iter.#n
Increments
Iter
atio
n
3103131031 TESTINGTESTING5
TESTING LIFE CYCLE ROLETESTING LIFE CYCLE ROLE
the testing workflow is primarily employed when each build is integrated and system tested
Testing is primary in focus both during elaboration (when the executable architectural baseline is tested) and during construction (when the bulk of the system is implemented)
During transition, focus is on fixing defects and doing regression testing
Previously developed test cases are used later as regression test cases
The test model is maintained throughout the life cycle, but it evolves:– obsolete test cases are removed– some test cases are refined into regression test cases– new test cases are created for each new build
3103131031 TESTINGTESTING6
TESTING ARTIFACTS & WORKERSTESTING ARTIFACTS & WORKERS
responsible for
TestEngineer
ComponentEngineer
responsible for
TestCase
XTest
ModelTest
EvaluationTest
Procedure
XTestPlan
XDefect
TestComponent
IntegrationTester
SystemTester
responsible for
3103131031 TESTINGTESTING7
TESTING WORKERSTESTING WORKERS
The test engineer – responsible for the integrity of the test model;
– plans tests by decide on test and the test schedule
– selects and describes the test cases and test procedures;
– evaluates the results of integration and system tests
The component engineer is responsible for test components that automate some of the test procedures
The integration tester performs integration tests for each build and reports defects
The system tester performs the system tests for a build which are derived from the use case
3103131031 TESTINGTESTING8
TESTING ARTIFACTSTESTING ARTIFACTS
A test model is a collection of test cases, test procedures and test components that describe how executable components (e.g., builds) are tested by integration and system tests
A test case specifies one way of testing the system—what to test with which input or result and under what conditions
A test procedure specifies how to perform one or several test cases or parts of them
A test plan describes the testing strategies, resources and schedule
A test evaluation is an evaluation of the results of the testing efforts
A test component automates one or several test procedures or parts of them (sometimes called test drivers, test harnesses or test scripts)
A defect is a system anomaly (i.e., a bug)
3103131031 TESTINGTESTING9
TESTING OVERVIEWTESTING OVERVIEW
Testing tries to Testing tries to finding differencesfinding differences between thebetween the specifiedspecified (expected) and the(expected) and the observedobserved (existing)(existing) system behaviorsystem behavior..
Testing tries to Testing tries to finding differencesfinding differences between thebetween the specifiedspecified (expected) and the(expected) and the observedobserved (existing)(existing) system behaviorsystem behavior..
usually done by developers that were not involved in system implementation
to test a system effectively, must have a detailed understanding of the whole system
not a job for novices
Goal:Goal: design tests that will systematically find defects
Our aim is to break the system (make it fail)
9.1
3103131031 TESTINGTESTING10
TESTING — VERIFICATION & VALIDATIONTESTING — VERIFICATION & VALIDATION
verification is the process of making sure that we have built the product right (i.e., it meets its stated requirements) most of the testing workflow is targeted at doing verification
validation is the process of making sure that we have built the right product (i.e., it is fit for its purpose) acceptance tests deal mainly with validation
testing verifies the results of implementation by testing each build as well as final versions of the system by– planning the tests required for each iteration
– designing and implementing the tests by creating test cases that specify what to test test procedures that specify how to test test components to automate the testing, if possible
3103131031 TESTINGTESTING11
THE REALITY OF TESTING THE REALITY OF TESTING
Testing cannot show the absence of software errors;it can only show that software errors are present!
It is impossible to completely test a nontrivial system.(Most systems will have bugs in them.)
Testing is a destructive activity.(We try to make the software fail)
Thus, it is often difficult for software engineers to effectively test their own software (there no intencive to
make the software fail)
Hence, good testing has the opposite natureof good software engineering
(i.e., good software engineering software does not failgood testing makes the software fail).
3103131031 TESTINGTESTING12
UP TESTING ACTVITIESUP TESTING ACTVITIES
IntegrationTester
TestEngineer
SystemTester
PlanTests
PerformIntegration
Tests
ImplementTests
PerformSystemTests
EvaluateTests
ComponentEngineer
DesignTests
PerformUnit Test
3103131031 TESTINGTESTING13
There are 1014 possible paths!It takes 3,170 years to exhaustively test the program if each test requires 1 ms!!
WHY NOT EXHAUSTIVE CODE TESTING?WHY NOT EXHAUSTIVE CODE TESTING?
loop < 20 timesHow long would it take to How long would it take to exhaustively test this program?exhaustively test this program?
3103131031 TESTINGTESTING14
GoalGoal: design a set of tests that has thedesign a set of tests that has the highest highest likelihoodlikelihood of uncovering defects with theof uncovering defects with the minimum amountminimum amount ofof time and efforttime and effort..
GoalGoal: design a set of tests that has thedesign a set of tests that has the highest highest likelihoodlikelihood of uncovering defects with theof uncovering defects with the minimum amountminimum amount ofof time and efforttime and effort..
PLAN TESTSPLAN TESTS
WHY?WHY? Tests cost money and take time to develop, perform and evaluate (up to 40% of project effort often devoted to testing).
A Test plan specificfies
1. Testing strategy: specifies the criteria and goals of testing• what kinds of tests to perform and how to perform them
• the required level of test and code coverage
• the percentage of tests that should execute with a specific result
2. Estimate of resources required: human/system
3. Schedule for the testing: when to run which tests
9.5.1
3103131031 TESTINGTESTING15
UP TESTING ACTVITIESUP TESTING ACTVITIES
IntegrationTester
TestEngineer
SystemTester
PlanTests
PerformIntegration
Tests
ImplementTests
PerformSystemTests
EvaluateTests
ComponentEngineer
DesignTests
PerformUnit Test
3103131031 TESTINGTESTING16
DESIGN TESTS : DESIGN TESTS : TEST CASETEST CASE
A Test case A Test case is one way of testing the system; is one way of testing the system; specifies specifies whatwhat, , conditionsconditions, , howhow..
A Test case A Test case is one way of testing the system; is one way of testing the system; specifies specifies whatwhat, , conditionsconditions, , howhow..
Specifices: What to test expected result, under what condition and how to test
White Box Tests: “testing-in-the-small”Derive test cases to verify component logic based on data/control structures Availability of source code is required.
(Tests based on knowledge of internal workings of a component.)
Black Box Tests: “testing-in-the-large”Derive test cases to verify component functionality based on the inputs and outputs Availability of source code is not required!
(Tests based only on knowledge of specified functionality of a component.)
Regression Tests: selective White Box and Black Box re-testing to ensure that no new defects are introduced after a change
9.3.2
3103131031 TESTINGTESTING17
DESIGN TESTS : DESIGN TESTS : TEST CASE DOCUMENTATIONTEST CASE DOCUMENTATION
Test case name name of the test case
Description of the test what the test is designed to do
Target class/component/subsystem name name of the thing
to be tested
Target class/component/subsystem operation name of the
operation to be tested
Test type black box/white box; valid/invalid input
Test value(s) inputs to be used for the test
Verification expected result
3103131031 TESTINGTESTING18
DESIGN TESTS : DESIGN TESTS : WHITE BOX TESTINGWHITE BOX TESTING
Goal:Goal: eTo design selective test that we have eTo design selective test that we have executed/exercised allexecuted/exercised all
Goal:Goal: eTo design selective test that we have eTo design selective test that we have executed/exercised allexecuted/exercised all
Why is this important?
1.1. all independent pathsall independent paths in the codein the code at least onceat least once• Basic Path TestingBasic Path Testing
2.2. all logical decisionsall logical decisions on theiron their truetrue andand false sidesfalse sides• Condition TestCondition Test
3.3. all loops at their boundaries all loops at their boundaries and within their boundsand within their bounds• Loop TestingLoop Testing
4.4. internal data structuresinternal data structures to ensure their validityto ensure their validity• Data Flow TestingData Flow Testing
3103131031 TESTINGTESTING19
Even though the false Even though the false branch executes no branch executes no additional statements, additional statements, it needs to be tested!it needs to be tested!
WHY NOT JUST STATEMENT TESTING?WHY NOT JUST STATEMENT TESTING?
i = j + 1if (A && B {
x = y + z;}n = p + 1;
i = j + 1
A && B
x = y + z
n = p + 1
TrueFalse
suppose we modify the program to:
i = 0if (A && B {
i = 1;}x = y / i;
3103131031 TESTINGTESTING20
WHITE BOX TESTING : WHITE BOX TESTING : BASIS PATH TESTINGBASIS PATH TESTING
9.4.2
Overview
1. Using the code, draw a corresponding flow graph.
(First drawing the activity diagram for the code may help, but is not required.)
2. Determine the cyclomatic complexity of the flow graph.
3. Determine a basis set of linearly independent paths.
4. Prepare test cases that force the execution of each path in the basis set.
Goal:Goal: To exercise each To exercise each independent pathindependent path at least once. at least once.Goal:Goal: To exercise each To exercise each independent pathindependent path at least once. at least once.
3103131031 TESTINGTESTING21
1. From the code, draw a corresponding flow graph. -Each circle represent one or mor non-brabching source statement
WHITE BOX TESTING:WHITE BOX TESTING:BASIS PATH TESTINGBASIS PATH TESTING
Sequence
If-then-else
...
Case
Do-untilDo-while
9.4.2
3103131031 TESTINGTESTING22
BASIS PATH TESTING EXAMPLEBASIS PATH TESTING EXAMPLE
procedure: process records1. do while records remain2. read record;3. if record field 1 = 04. then store in buffer;5. increment counter;6. elseif record field 2 = 07. then reset counter;8. else store in file;9. endif10. endif11. enddo
end
3103131031 TESTINGTESTING25
WHITE BOX TESTING : WHITE BOX TESTING : BASIS PATH TESTINGBASIS PATH TESTING
2. Determine the cyclomatic complexity of the flow graph.
cyclomatic complexity: A quantitative measure of thelogical complexity of code
Ways to compute cyclomatic complexity V(G):
V(G) = the number of regions (areas bounded by nodes and edges—area outside the graph is also a region)
V(G) = the number of edges - the number of nodes + 2
V(G) = the number of (simple) predicate nodes + 1
provides an upper bound on the number of pathsthat need to be tested in the code
3103131031 TESTINGTESTING27
WHITE BOX TESTING : WHITE BOX TESTING : BASIS PATH TESTINGBASIS PATH TESTING
3. Determine a basis set of linearly independent paths.
A n independent path is any path that introduces at least one new set of processing statements or a new condition
A basis set is the set of independent paths through the code (the basic set is not unique.)
Test cases derived from a basis set are guaranteed to execute every statement at least one time during testing
The cycromatic complexity provides an upper bound on the number of independent paths
3103131031 TESTINGTESTING28
1-2-3-6-7-9-10-1-11
BASIS PATH TESTING — BASIS PATH TESTING — EXAMPLEEXAMPLE
1-2-3-6-8-9-10-1-11 1-2-3-4-5-10-1-11 1-11
1
91011
2
4
587
3
6
3103131031 TESTINGTESTING29
WHITE BOX TESTING : WHITE BOX TESTING : BASIS PATH TESTINGBASIS PATH TESTING
4. Prepare test cases that force the execution of each path in the basis set.
Notes:
An activity diagram is not need , but the picture will help when you trace the program paths
Count each logical test—compound tests count as the number of Boolean operators + 1 (i.e., count each simple predicate)
Basis path testing should be applied to all components, if possible, and to critical components always
Basis path testing does not test all possible combinations of all paths through the code; it just tests every path at least once.
3103131031 TESTINGTESTING30
ASU WHITE BOX TESTING : BASIS PATH TESTINGASU WHITE BOX TESTING : BASIS PATH TESTING
checkRegistration Procedure
A student can register for a course if he has taken the prerequisites. However, he may not register for a course if he has already taken a course that is an exclusion. A student who does not have the prerequisites can register for a course if he is currently registered in the prerequisite course and has the permission of the instructor. Notwithstanding the preceding, the instructor may waive the prerequisite for a student.
Procedure checkRegistration return(result)
1. If (enrollment is open) AND (student does not have exclusions) AND ((student has prerequisites) OR (student has permission))
2. Register3 return(success)4. Else5. return(failure)6. EndifEnd
3103131031 TESTINGTESTING31
ASU — BASIS PATH TESTING:ACTIVITY DIAGRAMASU — BASIS PATH TESTING:ACTIVITY DIAGRAM
3103131031 TESTINGTESTING32
ASU — BASIS PATH TESTING : FLOW GRAPHASU — BASIS PATH TESTING : FLOW GRAPH
3103131031 TESTINGTESTING33
ASU — BASIS PATH TESTING : TEST DATAASU — BASIS PATH TESTING : TEST DATA
checkRegistration Procedure
Basis set path 1: enrollment is open
student does not have exclusions
student has prerequisites
Basis set path 2: enrollment is open
student does not have exclusions
student does not have prerequisites
student has permission
Basis set path 3: enrollment is open
student does not have exclusions
student does not have prerequisites
student does not have permission
Basis set path 4: enrollment is open
student has exclusions
Basis set path 5: enrollment is closed
3103131031 TESTINGTESTING34
BASIS PATH TESTING EXAMPLE :BASIS PATH TESTING EXAMPLE : VIDEO SALESVIDEO SALES
For the video sales and rental shop example, consider the following
procedure for reserving a copy of a movie video.
Procedure: reserveVideoCopy return(result)
1. If (status = “available”) OR ((status = “rented”) AND (returnDate
requestDate))
2. status = “reserved”
3. link video copy instance to member instance
4. result = “success”
5. Else
6. result = “failure”
7. Endif
End
Draw the flow graph for the reserveVideoCopy procedure. Clearly
label the regions of the flow graph.
3103131031 TESTINGTESTING35
IS BASIS PATH TESTING ENOUGH?IS BASIS PATH TESTING ENOUGH?
int Q;. . .i = j + 1;if ( A() && B() ) {
x = y + z;}n = p + 1;. . .
Boolean A() { Boolean B() {if (Flag1) { if (Flag2) {
Q = 0; return TRUE;return TRUE; } else {
} else { int x = 10/Q;return FALSE; return FALSE;
} }} }
i = j + 1
A()&&B()
x = y + z
n = p + 1
TrueFalse
Flag1=FALSE Flag1=TRUEFlag 2=TRUE
What happens when Flag1=TRUE and Flag2=FALSE?
This isThis issufficientsufficientfor basisfor basispath testingpath testing
3103131031 TESTINGTESTING36
WHITE BOX TESTING : WHITE BOX TESTING : CONDITION TESTINGCONDITION TESTING
Goal:Goal: To further exercises the To further exercises the true and falsetrue and false value of value of each each simple logical conditionsimple logical condition in a component. in a component.
Goal:Goal: To further exercises the To further exercises the true and falsetrue and false value of value of each each simple logical conditionsimple logical condition in a component. in a component.
1. Logical conditions
simple condition: (a rel-op b) where rel-op={<, ≤, =, ≠, ≥, >} (may be negated with NOT), e.g., a≤b; NOT(a≤b)
compound condition: two or more simple conditions connected with AND, OR, e.g., (a>b) AND (c<d)
relational expression: (E1 rel-op E2) where E1 and E2 are arithmetic expressions, e.g., ((a*b+c)>(a+b+c))
Boolean expression: non relation expression(e.g., NOT A)
errors to test for include (incorrect/missing/extra):– Boolean operator – relational operator– Boolean variable – arithmetic expression– Boolean parenthesis
3103131031 TESTINGTESTING37
1. Branch testing
For a compound condition C, test true and false branches of C and every simple condition of C
e.g., for C = (a>b) AND (c<d) we test for: C TRUE and FALSE
a>b TRUE and FALSE
c<d TRUE and FALSE
WHITE BOX TESTING : WHITE BOX TESTING : CONDITION TESTINGCONDITION TESTING (cont’d) (cont’d)
2. Domain testing
For an expression E1 rel-op E2, test for E1 greater than, equal to, or less than E2
– This guarantees detection of rel-op error if E1 and E2 are correct
– To detect errors in E1 and E2, the difference between E1 and E2 for the tests E1 greater or less than E2 should be as small as possible
– for an expression with n variables, 2n tests are required
3103131031 TESTINGTESTING38
Goal:Goal: To execute To execute loopsloops at end within their bounds. at end within their bounds.Goal:Goal: To execute To execute loopsloops at end within their bounds. at end within their bounds.
2. Nested Loops1 Conduct simple loop tests for the innermost loop
while holding the outer loops at their minimumiteration
2 Work outward, conducting tests for the nextinnermost loop
3 Continue until all the loops have been tested
Tests grow geometrically as the level of nesting increases
WHITE BOX TESTING : WHITE BOX TESTING : LOOP TESTINGLOOP TESTING
1. Simple Loops (n iterations)1 skip the loop entirely2 only one pass through the loop3 two passes through the loop4 m passes through the loop where m < n5 n-1, n, n+1 passes through the loop
3103131031 TESTINGTESTING39
WHITE BOX TESTING : WHITE BOX TESTING : LOOP TESTINGLOOP TESTING (cont’d) (cont’d)
3. Concatenated Loops If loops are independent loops use
simple loop testing
If loops are dependent loops use nested loop testing
4. Unstructured Loops Redesign the code !!!
These loop test are designed to have the hightest likelihood of uncovering errors with the minimum amount of effort and overlap of tests.
3103131031 TESTINGTESTING40
WHITE BOX TESTING : WHITE BOX TESTING : DATA FLOW TESTINGDATA FLOW TESTING
Goal:Goal: To ensure that the To ensure that the value of a variablevalue of a variable is correct at is correct at certain points of execution in the code.certain points of execution in the code.
Goal:Goal: To ensure that the To ensure that the value of a variablevalue of a variable is correct at is correct at certain points of execution in the code.certain points of execution in the code.
select test paths according to the locations ofdefinitions and uses of a variable
DEF(S) = {X | S contains a definition of X} locations of definition of X
USE(S´) = {X | S´ contains a use of X} locations of use of X
Data Use (DU) Chain (X) is the set of [X, S, S´] where X is “live”(i.e., not redefined between S and S´)
One testing strategy: Every data use chain must be covered once For every variable, do a test along the path from where the
variable is defined to the statement(s) where the variable is used (can be combined with basis path testing)
3103131031 TESTINGTESTING41
DESIGN TESTS : DESIGN TESTS : BLACK BOX TESTINGBLACK BOX TESTING
Black box techniques attempt to find:– incorrect or missing functions– interface incompatibility errors– data structure or external database access errors– performance errors– initialization and termination errors
Codeinput
actual ? predictedoutput = output
Test cases should:– reduce by more than 1 the number of additional test cases required
(i.e., a test case should cover a range of input or output values)– tell us something about presence or absence of a class of errors
(e.g., all character data is correctly/incorrectly processed)
3103131031 TESTINGTESTING42
all possible input/output values
BLACK BOX TESTING : BLACK BOX TESTING : EQUIVALENCE PARTITIONINGEQUIVALENCE PARTITIONING
For each partition we design test cases which are based on valid and invalid inputs/outputs.
userqueries
numericaldata output
formatrequests
commandkey input
mousepicks onmenu
responsesto prompts
GOAL : To partitionGOAL : To partition inputs and outputs to create inputs and outputs to create thorough test coverage of a thorough test coverage of a class of errors.class of errors.
GOAL : To partitionGOAL : To partition inputs and outputs to create inputs and outputs to create thorough test coverage of a thorough test coverage of a class of errors.class of errors.
9.4.2
3103131031 TESTINGTESTING43
not member of set
EQUIVALENCE PARTITIONING : EQUIVALENCE PARTITIONING : GUIDE LINE FOR GUIDE LINE FOR SELECTING EQUIVALENCE CLASSESSELECTING EQUIVALENCE CLASSES1. If input is a range one valid and two invalid equivalence classes:
2. If input is a specific value one valid and two invalid equivalence classes:
3. If input is a set of related values one valid and one invalid class:
4. If input is Boolean one valid and one invalid class:
in range greater than rangeless than range
value greater than valueless than value
member of set
BooleannonBoolean
3103131031 TESTINGTESTING44
userqueries
EQUIVALENCE PARTITIONING : EQUIVALENCE PARTITIONING : SELECTING TEST SELECTING TEST VALUES — BOUNDARY VALUESVALUES — BOUNDARY VALUES
As well as selecting test data “inside” an equivalence class, we also select data at the “edges” of the equivalence class.
numericaldata output
formatrequests
responsesto prompts
commandkey input
mousepicks onmenu
More errors occur at the More errors occur at the boundariesboundaries than in the “ than in the “centercenter”.”.More errors occur at the More errors occur at the boundariesboundaries than in the “ than in the “centercenter”.”.
9.4.2
3103131031 TESTINGTESTING45
not member of set
EQUIVALENCE PARTITIONING : EQUIVALENCE PARTITIONING : GUIDELINES FOR GUIDELINES FOR SELECTING BOUNDARY VALUESSELECTING BOUNDARY VALUES 1. If the input is a range and is bounded by a and b, then use a, b, and
values just above and just below a and b, respectively.
2. If the input is a number of discrete values, use the minimum and the maximum of the values and values just above and just below them, respectively. (Can also be applied to a single specific input value.)
3. For inputs that are a set of values, test all values in the set (if possible) and one value outside the set.
a b
min max
member of set
3103131031 TESTINGTESTING46
EQUIVALENCE PARTITIONING : EQUIVALENCE PARTITIONING : GUIDELINES FOR GUIDELINES FOR SELECTING BOUNDARY VALUES SELECTING BOUNDARY VALUES (cont’d)(cont’d)
5. Apply guidelines 1 and 2 to create output values at the minimum and maximum expected values.
e.g., if the output is a table, create a minimum size table (1 row) and a maximum size table
6. If data structures have boundaries, test these boundary values and values just above and just below them, respectively.
e.g., for an array of bounds 1 to 10 — test array index equal to 0, 1,2, 9, 10, 11
1 100 2 9 11
4. For inputs that are Boolean, test for both Boolean values (T, F) and for a non-Boolean value.
nonBoolean Boolean
3103131031 TESTINGTESTING47
ASU — BOUNDARY VALUE SELECTIONASU — BOUNDARY VALUE SELECTION
Procesure : Student Record SearchIn the ASU System, the Maintain student information use case uses a function that searches the database for the record of a given student. The input to the search function is a student ID. The output is either a success indication and the student record containing the given student ID or a failure indication and a message indicating the nature of the failure. The valid range of student IDs is from 1000000 to 9999999.
For the following test categories:(a) typical(b) boundary(c) otherwhat specific test values and test cases would you use to testwhether the search procedure works correctly and accepts only valid student IDs?
3103131031 TESTINGTESTING48
ASU BLACK BOX TESTING : ASU BLACK BOX TESTING : SOLUTIONSOLUTION
3103131031 TESTINGTESTING49
ASU BLACK BOX TESTING : ASU BLACK BOX TESTING : SOLUTIONSOLUTION
3103131031 TESTINGTESTING50
ASU BLACK BOX TESTING : ASU BLACK BOX TESTING : SOLUTIONSOLUTION
3103131031 TESTINGTESTING51
BLACK BOX TESTING : BLACK BOX TESTING : SOLUTIONSOLUTION
3103131031 TESTINGTESTING52
BLACK BOX TESTING : BLACK BOX TESTING : VIDEO SALESVIDEO SALES
Procedure: validateRating
In the video sales and rental shop system, there is a procedure to
validate the review rating input by the user. (Recall that a review
can be given a rating, which is an integer value, from 1 to 5.)
For each of the following test categories:
(a) typical
(b) boundary
(c) other
what specific test values would you use to test whether the
validateRating procedure works correctly and accepts only valid rating values?
3103131031 TESTINGTESTING53
BLACK BOX TESTING :BLACK BOX TESTING : THREAD TESTINGTHREAD TESTING
An event-based approach where tests are based on events which trigger system actions particularly appropriate for object oriented systems
used after classes have been unit tested and integrated into subsystems
need to identify and execute each possible processing thread from use cases
may not be possible to do complete thread testing too many input/output combinations
focus on most commonly exercised threads basic paths through flow of events
3103131031 TESTINGTESTING54
BLACK BOX TESTING :BLACK BOX TESTING : THREAD TESTINGTHREAD TESTING
single thread: obj3obj2obj4
obj1
obj3
obj2
obj4
obj5
I1(obj1)
I2(obj1)
I3(obj1)
I1(obj3)
I1(obj2)
O1(obj4)
O2(obj4)
O1(obj5)
multiple thread: obj1obj2{obj5, obj4} multi-input thread: into obj1
3103131031 TESTINGTESTING55
BLACK BOX TESTING :BLACK BOX TESTING : STATE-BASED TESTINGSTATE-BASED TESTING
State based testing focuses on comparing the resulting state of a class with the expected state
Derive test cases using statechart diagram for a class
Derive a representative set of “stimuli” (events) for each transition
Check the attributes of the class after applying a “stimuli” to determine of the specified state has been reached
We first need to put the class in the desired state before applying the “stimuli”
9.4.2
3103131031 TESTINGTESTING56
Comparison Testing– when reliability is absolutely critical, write several versions of the
software — perhaps to be used as backups– run same test data on all versions; cross-check results for
consistency Caution: if the error is from a bad specification and all
versions are built from the same specification, then the same bug will be present in all copies!
Cause-Effect Graphing1. causes (input conditions) and effects (actions) are identified for a
subsystem2. a cause-effect graph is developed3. the graph is converted to a decision table4. decision table rules are converted into test cases
OTHER BLACK BOX TECHNIQUESOTHER BLACK BOX TECHNIQUES
Error Guessing– use application domain experience or knowledge to select test values
9.4.2
3103131031 TESTINGTESTING58
UP TESTING ACTVITIESUP TESTING ACTVITIES
IntegrationTester
TestEngineer
SystemTester
PlanTests
PerformIntegration
Tests
ImplementTests
PerformSystemTests
EvaluateTests
ComponentEngineer
DesignTests
PerformUnit Test
3103131031 TESTINGTESTING59
IMPLEMENT TESTSIMPLEMENT TESTS
Goal:Goal: To automate test procedures as much as possible. To automate test procedures as much as possible.Goal:Goal: To automate test procedures as much as possible. To automate test procedures as much as possible.
Running test cases can be very tedious and time consuming– many possible input values and system states to test
A test component: a program that automates one or several test procedures or parts of them
There are tools available to help write test components:– perform actions for a test case and the tool “records” the actions
– parameterize the recorded script to accept a variety of input values
Spreadsheets and/or database applications can be used to store the required input data and the results of each test
3103131031 TESTINGTESTING60
UP TESTING ACTVITIESUP TESTING ACTVITIES
IntegrationTester
TestEngineer
SystemTester
PlanTests
PerformIntegration
Tests
ImplementTests
PerformSystemTests
EvaluateTests
ComponentEngineer
DesignTests
PerformUnit Test
3103131031 TESTINGTESTING61
The time uncertainty in testing is the debug part!
PERFORM TESTSPERFORM TESTS
Software Configuration- System requirement specification- System analysis specification- System design specification- Source data
Test Configuration-Test case-Tests procedure-Testing tools-Expected result
Testresults
Expectedresults
Errors
Errorratedata
Corrections
PerformTest
Evaluate
Updatereliability& quality
model
Debug
Done
No errors
9.2
3103131031 TESTINGTESTING62
PERFORM TESTS — PERFORM TESTS — TESTING STRATEGIESTESTING STRATEGIES
We need to integrate test cases into a well-planned series of steps that are taken to test a component/subsystem test procedure– what are the steps that need to be conducted to test a component?– when are the steps planned and undertaken?– how much effort, time and resources will be required to do the steps?
A testing strategy indicates which testing techniques (white box, black box, thread) are appropriate at which point in time
We need to balance:flexibility & creativity with planning & management
Testing often occurs when deadline pressures are most severe.(Progress must be measurable and problems identified as early as possible.)
3103131031 TESTINGTESTING63
A TESTING STRATEGYA TESTING STRATEGY
Requirements capture R
U Unit testing
AAnalysis
We developWe developthe systemthe systemfrom thefrom theoutside inoutside in
We testWe testthe systemthe systemfrom thefrom theinside outinside outDDesign
CImplementation(coding)
I Integration testing
SdSystem testing(developer)
ScSystem testing(customer)
3103131031 TESTINGTESTING65
UP TESTING ACTVITIESUP TESTING ACTVITIES
IntegrationTester
TestEngineer
SystemTester
PlanTests
PerformIntegration
Tests
ImplementTests
PerformSystemTests
EvaluateTests
ComponentEngineer
DesignTests
PerformUnit Test
3103131031 TESTINGTESTING66
Test cases
UNIT TESTINGUNIT TESTING
Emphasis is on White Box techniques.
interfacelocal data structuresboundary conditionsindependent pathserror-handling paths
Component
9.4.2
3103131031 TESTINGTESTING68
Test cases
UNIT TESTING — UNIT TESTING — TESTING PROCEDURESTESTING PROCEDURES
Driver and/or stubs must be developed for each unit test
interfacelocal data structuresboundary conditionsindependent pathserror-handling paths
Driver
Stub StubResults
driver: component that calls component to be testedstub: component called by component to be tested
Componentto be tested
3103131031 TESTINGTESTING69
UNITING TESTING : UNITING TESTING : OBJECT-ORIENTED TESTINGOBJECT-ORIENTED TESTING
What to unit test?– A unit test has to be at least a class, but object state makes testing
difficult
A class must be tested in every state it can ever enter(i.e., use state-based testing)
How to deal with encapsulation ?– Encapsulation hides what is inside an object hard to know its state
We need to provide a method, for testing only, that reports all of an object’s state
How to deal with Inheritance and polymorphism ?– If a subclass overrides methods of an already tested superclass, what
needs to be tested — only the overridden methods?
No! All of a subclass’s methods need to be tested againdue to dynamic binding and substitutability
3103131031 TESTINGTESTING70
UP TESTING ACTVITIESUP TESTING ACTVITIES
IntegrationTester
TestEngineer
SystemTester
PlanTests
PerformIntegration
Tests
ImplementTests
PerformSystemTests
EvaluateTests
ComponentEngineer
DesignTests
PerformUnit Test
3103131031 TESTINGTESTING71
INTEGRATION TESTINGINTEGRATION TESTING
If components all work individually, why more testing?If components all work individually, why more testing?If components all work individually, why more testing?If components all work individually, why more testing?
Big Bang!! Incremental“builds”
Incrementalconstruction
strategy
Regressiontesting
Interaction errors cannot be uncovered by unit testing!(e.g., interface misuse, interface misunderstanding, timing errors)
Integration approaches
versus
9.4.3
3103131031 TESTINGTESTING72
S2S2
S5S4
S3
S6 S7S6
INTEGRATION TESTING : INTEGRATION TESTING : TOP DOWN TOP DOWN
pro: Early testing and error detection of user interface components; can demonstrate a complete function of the system early
con: Cannot do significant low-level processing until late in the testing; need to write and test stubs
1.The top subsystem is tested with stubs
2.The stubs are replaced one at a time“depth-first” or “breadth-first”
3. As new subsystems are integrated, some subset of previous tests is re-run (regression testing)
S1
S3
S4
Usually not appropriate for OO development.(Except to test operations of classes.)
S7
S5
3103131031 TESTINGTESTING73
D2
INTEGRATION TESTING : INTEGRATION TESTING : BOTTOM UP BOTTOM UP
pro: Interaction faults are more easily found; easier test case design and no need for stubs
con: User interface components are tested last
2. Drivers are replaced one ata time “depth-first”
1. Subsystems are groupedinto builds and integrated
Appropriate for OO development.(Test classes use their own test driver.)
S2 S6
S1
S7
S3
S4 S5
3103131031 TESTINGTESTING74
S3
D2
pro: parallel testing activities
con: need many drivers and stubs
INTEGRATION TESTING :INTEGRATION TESTING : SANDWICH SANDWICH
Lower-level subsystems are groupedinto builds and tested with drivers
Top-level subsystemsare tested with stubs
S2 S6 S7
S1
S3
S4 S5 Can significantly shortentotal testing time.
3103131031 TESTINGTESTING75
INTEGRATION TESTING — CRITICAL SUBSYSTEMSINTEGRATION TESTING — CRITICAL SUBSYSTEMS
Critical subsystems should be tested as early as possible!Critical subsystems should be tested as early as possible!Critical subsystems should be tested as early as possible!Critical subsystems should be tested as early as possible!
Critical subsystems are those that:
1. have high risk
2. address several software requirements (i.e., several use cases)
3. have a high level of control
4. are complex or error prone high cyclomatic complexity
5. have specific performance requirements
Regression testing is a must for critical subsystems!
3103131031 TESTINGTESTING76
UP — TESTING ACTVITIESUP — TESTING ACTVITIES
IntegrationTester
TestEngineer
SystemTester
PlanTests
PerformIntegration
Tests
ImplementTests
PerformSystemTests
EvaluateTests
ComponentEngineer
DesignTests
PerformUnit Test
3103131031 TESTINGTESTING77
SYSTEM TESTINGSYSTEM TESTING
System testing is testing of System testing is testing of the entire systemthe entire system to be to be sure the system functions properly when integrated.sure the system functions properly when integrated.
System testing is testing of System testing is testing of the entire systemthe entire system to be to be sure the system functions properly when integrated.sure the system functions properly when integrated.
Some specific types of system tests:
Functional The developers verify that all user functions work as specified in the system requirements specification
Performance The developers verify that the nonfunctional requirements are met
Pilot A selected group of end users verifies common functionality in the target environment
Acceptance The client/user verifies usability, functional and nonfunctional requirements against system requirements specification
Installation customer verifies usability, functional and nonfunctional requirements in real use
9.4.4
3103131031 TESTINGTESTING78
SYSTEM SYSTEM TESTINGTESTING:PERFORMANCE TESTING:PERFORMANCE TESTING
stress tests Verify that the system can continue functioning when confronted with many simultaneous requests How high can we go? Do we fail-soft or collapse?
volume tests Verify that the system can handle large amounts of data, high complexity algorithms, or high disk fragmentation
security tests Verify that access protection mechanisms work make penetration cost more than value of entry
timing tests Verify that the system meets timing constraints usually for real-time and embedded systems
recovery tests Verify that the system can recover when forced to fail in various ways
database recovery is particularly important
3103131031 TESTINGTESTING79
SYSTEM TESTING : SYSTEM TESTING : PILOT TESTINGPILOT TESTING
Alpha test: A test in a controlled environment so thatdevelopers can observe users
software
Developer site
client tests atdeveloper‘s site
Beta test:A test in a real environment so that bugs areuncovered from regular usage patterns
developerreviews bug
reportssoftware
Client site
clienttests at
client site
3103131031 TESTINGTESTING80
SYSTEM TESTING : SYSTEM TESTING : ACCEPTANCE TESTINGACCEPTANCE TESTING
Acceptance testing is the process of ensuring that the Acceptance testing is the process of ensuring that the software meets software meets the user’s the user’s reasonablereasonable expectations.expectations.
Acceptance testing is the process of ensuring that the Acceptance testing is the process of ensuring that the software meets software meets the user’s the user’s reasonablereasonable expectations.expectations.
Can be very subjective good requirements specification from the start is important!
The method of defect resolution should be negotiated before the transition phase!
Acceptance tests (black box) cover classes of tests such as:– functionality – documentation– performance – reliability, etc.
Configuration review (audit)– An audit ensures that all elements of the software delivered to the
customer (the software configuration) have been properly developed and documented
3103131031 TESTINGTESTING81
UP TESTING ACTVITIESUP TESTING ACTVITIES
IntegrationTester
TestEngineer
SystemTester
PlanTests
PerformIntegration
Tests
ImplementTests
PerformSystemTests
EvaluateTests
ComponentEngineer
DesignTests
PerformUnit Test
3103131031 TESTINGTESTING82
EVALUATE TESTSEVALUATE TESTS
The test engineer need to evaluate the results of the testing by :
– compare the results with the goals outlined in the test plan
– prepare metrics that let the test engineers determine the current quality of the software
How do we know when to stop testing?
We can consider:
1. testing completeness: the % of test cases that have been run and the % of code that has been tested
2. reliability: based on trends in testing error rate when compared to previous projects
3103131031 TESTINGTESTING83
EVALUATE TESTS : EVALUATE TESTS : TESTING ERROR RATETESTING ERROR RATE
Past history can be used to plot expected failure rate.We compare this with actual failure rate for this project.
Testing time
Fai
lure
s p
er t
est
hou
r
This project
Expected failure rate(based on previous projects)
Possible outcomes:• perform additional tests to find more defects• relax the test criteria, if set too high• deliver acceptable parts of the system;
continue testing unacceptable parts
3103131031 TESTINGTESTING84
TESTING UNDER PRESSURETESTING UNDER PRESSURE
All test are importantAll test are important, but if you need to restrict testing … , but if you need to restrict testing … All test are importantAll test are important, but if you need to restrict testing … , but if you need to restrict testing …
Testing a system’s capabilities (functionality) is more important than testing its components– Identify things that will stop users doing their jobs.
Testing old capabilities is more important than testing new capabilities– Users expect existing features to keep working!
Testing typical situations is more important than testing boundary cases– Normal usage patterns are more important than atypical ones.
3103131031 TESTINGTESTING85
Debugging Debugging [9.3.4][9.3.4]
A consequence of testing, but a separate activity –> not testing
Can be one of the most exasperating parts of development–> easy to lose focus or follow the wrong path!
time required tocorrect the error
time required todetermine the natureand location of theerror
Debugging islike trying tocure a sickperson.
We see thesymptomsand try tofind thecause.
3103131031 TESTINGTESTING86
Debugging — Symptoms & CausesDebugging — Symptoms & Causes
Symptom and cause may begeographically separated
Symptom may disappear whenanother problem is fixed
Cause may be due to a systemor compiler error
Cause may be due to humanerror that is hard to trace
Cause may be due to assumptionsthat everyone believes
Symptom may be intermittent
symptom
cause
So, what can we do?
Pressure to fix bugs often causes new bugs to be introduced
3103131031 TESTINGTESTING87
DEBUGGING TECHNIQUESDEBUGGING TECHNIQUES
1. Brute force: try to let the computer find the error– Memory dumps– Run-time traces everywhere
The hope is that somewhere in all this informationwe’ll find the bug!
2. Backtracking: start where the problem occurs and read the code backward until the problem is found
OK for small programs.
3. Cause elimination: try to create input test data sets that would prove or disprove a particular theory
Difficult, but sometimes your best bet!
3103131031 TESTINGTESTING88
DEBUGGING — FINAL THOUGHTSDEBUGGING — FINAL THOUGHTS
Debugging tools can help gain more insight into the problem
Talk to your colleagues — They can see things you’ve grown too tired to see!
Isolate and focus –> Think through the problem—some of the best debugging happens away from the computer!
Be absolutely sure to conduct regression tests when you “fix” the bug!
3103131031 TESTINGTESTING89
TESTING SUMMARYTESTING SUMMARY
Testing requires good planning. know what you are trying to test for
The use of the right test in the right situation. white box, black box, regression
Test that are systematic and thorough! Have a testing strategy( unit, integration, system, acceptance)
Clear criterion for knowing when to stop. Decide beforehand how much testing will be enough.
Top Related