1

Software Testing

  Testing can only reveal the presence of errors and not the absence of errors.

  A successful test is a one which discovers more, yet-undiscovered, errors

  An ideal testing process never ends.

Definition: Testing is a process of executing a program with data, with the sole intention of finding errors in the program.

Definition: “Quality assurance is a planned and systematic pattern of all actions necessary to provide adequate confidence that the item conforms to established requirements.”

-- IEEE

2

Objectives of software testing

  To find more errors in the software design and implementation

  To confirm that the product meets (satisfies) the requirements, including non-functional requirements

  Testing is one type of validation — Validation is a process that answers to the

question,“Are we developing the right product?”

Testing and Debugging

  Debugging requires testing efforts to find the errors first, and then retest the program after correcting the errors. (Thus, testing occurs before and after debugging.)

  Testing is usually performed manually .

  Debugging is often performed with the help of tool support.

Definitions Testing is a process of finding the presence of errors. Debugging is a process of finding and correcting errors.

3

Test Case

Example: Let y = f(x) be the function to be tested; a test

case will indicate for what range of values of (input) x the function will return (output) and for what input values the function will not return an output

Definition A test case is a description of input/output relationship of one particular testing activity - a set of “inputs” and expected “output(s)”.

Sample Test Cases

The following are some of the test cases for the factorial function “ int factorial (int x) ”

Input x Expected output Actual output

x = 0 1

x = 1 1

x >= 0 x!

x < 0 error

4

Testing principles

  All tests should be traceable to their source (requirements and/or code).

  Testing activity should be planned (test cases prepared) well before testing process is exercised.

  Exhaustive testing is not possible.

  To be most effective, testing should be conducted by a third party – not by the developers. — Quality Assurance team

Test case design methods

Both black-box and white-box testing are required for a complete testing process.

black-box testing The software is assumed to be a black box (with unknown code). Test cases are derived from problem descriptions, requirements, design documents, etc.

white-box testing Test cases are derived by examining the source code.

5

Black-box and White-box testing •  Test cases can be

derived before the program is written.

•  The objective is to ensure that the program meets (satisfies) the requirements.

•  Knowledge of the application domain is necessary but not about the program or the programming language.

•  Test cases can be derived only after the program is complete.

•  The objective is to ensure that all parts of the logic of the program are correct.

•  Knowledge of the programming language is needed but possibly not the application domain.

White-box testing

  Also called glass-box testing and structural testing

  Derivation of test cases from the program’s structure.

  What we would like to do... exercise every possible execution path   What we are able to do... exercise every possible statement at least once

path coverage

statement coverage

6

Example – computing factorial int factorial (int n) { int i, ret = 0; if (n < 0) return ret; i = n; ret = 1; while (i > 0) { ret = ret * i; i = i - 1; } return ret;}

Input n

Expected output

n < 0 Error

n==0 1

n > 0 n!

Test cases

Flow graph notation

  Also called “program graph”

  Abstractly describes the control flow of the program

  Consists of nodes and arrows •  Each node represents a contiguous segment of

code (statement(s) or expression(s)). •  Arrows represent control flow transitions.

7

if if-else case

while

sequence

repeat-until for

Flow graph for the factorial example

ret = 0;

(n < 0) return ret;

i = n; ret = 1;

(i > 0)

ret = ret * i; i = i - 1;

return ret;

int i, ret=0;

int factorial (int n) { int i, ret = 0; if (n < 0) return ret; i = n; ret = 1; while (i > 0) { ret = ret * i; i = i - 1; } return ret;}

8

For path coverage we must ensure that every node is executed by at least one test case. 1) number nodes

2) check test cases

ret = 0;

(n < 0) return ret;

i = n; ret = 1;

(i > 0)

ret = ret * i; i = i - 1;

return ret;

int i, ret=0; 1

2

3

4

5

6

7

8

Case statements n < 0 1,2,3,8 n==0 1,2,3,4,5,7 n > 0 1,2,3,4,5,6,7

Test cases

Another example

Remove an integer from an array and shift the array, adding 0 at the end.

void squeeze (int k, int[] arr) { boolean flag = false; int index = 0, j; while (index < arr.length && !flag) { if (arr[index] == k) { for (j = index; j < arr.length; j++) arr[j] = arr[j+1]; arr[N] = 0; flag = true; } else index = index + 1; }} Draw the flow graph

and generate the test cases.

9

What about this one? public char reportGrade (int[] exams) { char ch = ‘ ‘; int s,i; s = sum (exams) / exams.length; if (s >= 90) ch = 'A'; else if (s < 90 && s >= 75) ch = 'B'; else if (s < 75 && s >= 60) ch = 'C'; else if (s < 60 && s >= 50) ch = 'D'; else ch = 'F'; return ch;}

public int sum (int[] exams) { int n, total = 0; for (n = 0; n < exams.length; n++) total = total + exams[n]; return total;}

White-box Testing Issues

When is statement coverage impossible?

When is statement coverage impractical?

Does statement coverage ensure correctness?

10

Black-box testing

Black-box testing is a testing approach in which test cases are generated without knowledge of the program (source code).

  also called functional testing or behavioral testing

  Test cases can be derived even before coding begins.

Example: Automated teller machine (ATM)

Consider the functional requirement for a withdraw operation. Test cases include...

 withdraw an amount less than the balance  withdraw an amount equal to the balance  withdraw an amount more than the balance  withdraw an amount beyond a maximum  withdraw a negative amount (robustness

testing)

11

Objectives of black-box testing   Identify improper implementation of

functionalities.

  Identify missing functionalities.

  Identify errors in interfaces among the components of the software. ... including external software such as a database.

  Identify performance (and other non-functional) errors.

  Identify initialization and/or termination errors.

Equivalence partitioning

  A black-box testing method

  Partition the input domain (the set of potential input values) into mutually disjoint subsets, called equivalence classes.

  Every element of a subset shares one characteristic with all other elements in the same subset. — As a consequence, it is sufficient to select one element from

each subset, there by reduce the time for testing.

12

Input domain

Output domain

Equivalence partitioning does not impose any constraints on the output domain.

Software system

Equivalence partitioning example

Factorial example •  Three subsets of input (the parameter ‘n’) are

negative integers zero positive integers

Equivalence classes: negative integers - invalid factorial input zero - base case for recursive definition of factorial positive integers - factorial defined recursively

13

More examples A function that accepts a 2-digit positive integer as input

Calculate income tax based on net income

  All integers less than 10   Integers between 10 and 99 both inclusive   All integers greater than 99

  Income between $0 through $33000   Income between $33001 through $150000   Income above $150000

Limitations of Equivalence Partitioning

  User is responsible for ensuring the mutual exclusiveness of the partitions.

  Too difficult to determine equivalence classes

  Too many equivalence classes

14

Boundary Value Analysis   Boundary value analysis is a technique that

complements the equivalence partitioning approach.

  Boundary value analysis can help to partition the input domain when the input lies within a range(s) of values.

  Boundary value analysis works best when parameters (if more than one) are fairly independent of each other.

Guidelines for boundary value analysis

  Select the members at the boundary of each equivalence class.

  Select a “middle” value.

  Also select those values which are adjacent to the boundaries.

  Include out of range values for robustness.

Example:   Suppose an equivalence class contains values from 120 through 175.   Test case values:

119, 120, 121, 149, 174, 175, 176

15

More examples

Calculate property tax based on the value of the property.

•  Single variable function

Calculate the score of a competition based on the number of events participated and the score given for each such event.

•  Two-variable function

Xmin Xmax

Ymin

Ymax

Boundary value analysis for a two-variable function - f (x,y)

16

Boundary values for different data types Data type Boundary cases

Integer, Double (Real) Range of values given in the problem

String Empty string, String with one char, String of maximum length, String of length greater than the maximum

Array Similar to String

File Empty file, File with one record, file of max. size, file overflow

Testing Phases Unit testing

•  Test individual components in isolation. •  Often performed by the developer.

Module testing •  Test a group of components (units) that interact with

each other or a group that implements a particular functionality.

•  May be a group of procedures for the imperative paradigm; a group of methods or classes in the O-O paradigm.

•  Interfaces among these components must also be tested.

17

Testing Phases(continued) Subsystem testing

•  Test a group of modules and their interfaces. •  A subsystem may be a higher-level group of

modules in the procedural paradigm or a package in the O-O paradigm.

•  Interfaces between modules, particularly access restrictions across modules, must be tested along with functionalities

Integration testing •  This is a combination of module and subsystem

testing.

Testing Phases(continued)

Acceptance testing •  Test the entire system as one single product. •  GUI testing falls into this category. •  Also requires testing of interfaces between

external components such as a database or a device driver.

System testing •  Testing to conform that the software product

meets the standards and/or specific criteria set by the customers/users.

18

Classifications of Testing   Black-box testing and White-box testing classify

testing methods – processes to generate test cases and test data.

  Stages of testing classify the portion of the software product that must be tested at any one time.

  Combining these two... — A unit can be tested both by black-box testing and by

white-box testing. — A module may undergo black-box testing and white-box

testing.

  Conclusion: Each classification defines a different perspective of testing

Further classifications of testing

Bottom-up testing •  Test the units first, then the modules, then the

subsystems and so on until the entire product is tested

Top-down testing •  Test the entire software product as one unit, then test its

individual subsystems giving emphasis to their interfaces, then test the modules and so on until the components are broken down at the unit level.

•  May require “stubs”.

19

Stubs - example public static void main(String[] args) { int[] arr = new int[20]; read(arr); sort(arr); if (search(arr, 23)) System.out.println(“23 is in the array”); else System.out.println(“23 NOT in the array”);}

public void read(int[] array) { //STUB System.out.println (“read – to be implemented”);}

public void sort(int[] array) { //STUB System.out.println (“sort to be implemented”);}

public boolean search(int[] array, int k) { //STUB return true; }

A few more testing classifications Desk Checks

•  Developers often perform walk-throughs of their own, or colleague’s code.

Alpha testing •  Testing performed at the developer’s site. •  Generally performed by the QA team, not by the

development team. •  Purpose: to check and correct errors before the release of

the actual product

Beta testing •  Testing performed at the customer’s site. •  Generally conducted by the end users (may be the customer) •  Purpose: to report back to the developers any errors before

product release

20

Regression testing   Regression testing is the act of repeating a suite

of test cases after a significant change to a system.

  Regression testing is an important part of software maintenance.

  Documentation of test cases is essential to successful regression testing.

  Dependency graphs can help to determine which components require regression testing.

Object-Oriented Software Testing Research on O-O testing reports that

testing methods proposed for imperative approach are not adequate for O-O approach.

O-O poses additional problems due to the distinguishing characteristics such as encapsulation, inheritance and polymorphism.

21

Class testing Class testing are modules.

Methods are units.

Interactions within a class are module interfaces.

Interactions between different classes are subsystem interfaces.

Testing O-O Encapsulation Within O-O design encapsulation specifies

access restrictions to the structural and behavioral members of a class. Such encapsulation should be tested.

•  Check whether all the public members are accessible from everywhere.

•  Check whether all the private members are hidden outside the class.

•  If language supports other access mechanisms, such as protected or class-specific scope, those features should be tested for both accessibility and non-accessibility.

22

Testing Inheritance   Subclasses introduce new features (attributes

and methods) and can modify inherited behaviors.

  Modified behaviors must be tested in the context where they are used.

  Polymorphism adds complexity (e.g., co-variance or contra-variance?)

  Multiple inheritance adds complexity (What if two methods with the same signature conflict? i.e., both are inherited from different superclasses.)

  Any change to a superclass requires regression testing of all subclasses.