1

Software Testing

2

Path Testing

3

Structural Testing• Also known as glass box, structural, clear box and

white box testing. A software testing technique whereby explicit knowledge of the internal workings of the item being tested are used to select the test data.

• Unlike black box testing that uses the program specification to examine outputs, white box testing is based on specific knowledge of the source code to define the test cases and to examine outputs.

4

Structural Testing

• Structural testing methods are very amenable to:– Rigorous definitions

• Data flow, control flow, objectives, coverage criteria, relation to programming language semantics

– Mathematical analysis• Graphs, path analysis

– Precise measurement• Metrics, coverage analysis

5

Program Graph - Definition

• Also commonly known as Control Flow Graph (CFG)

• Defined as below:“Given a program written in an imperative

programming language, its Program Graph, is a directed labeled graph in which nodes are either groups of one or more statements or fragments of a statement, and edges represent flow of control”

6

Program Graph

• If i, j, are nodes in the program graph, there is an edge from node i, to node j in the program graph if an only if, the statement corresponding to node j, can be executed immediately after the last statement of the group of statement(s) that correspond to node i.

• A group of statements that makes up a node in the Program Graph is called a basic block.

• There is a straightforward algorithm to segment a code fragment into basic blocks and create the corresponding Program Graph.

7

Program Graph

• Three types of nodes:– Process block– Decision node– Junction node

8

Example: Basic Blocks

FindMean (FILE ScoreFile){ float SumOfScores = 0.0;

int NumberOfScores = 0; float Mean=0.0; float Score;Read(ScoreFile, Score);while (! EOF(ScoreFile) {

if (Score > 0.0 ) {SumOfScores = SumOfScores + Score;NumberOfScores++;}

Read(ScoreFile, Score);}/* Compute the mean and print the result */if (NumberOfScores > 0) {

Mean = SumOfScores / NumberOfScores;printf(“ The mean score is %f\n”, Mean);

} elseprintf (“No scores found in file\n”);}

1

23

4

5

7

6

8

9

9

Example: Program GraphStart

2

3

4 5

6

7

8 9

Exit

1

F

T F

T F

T

10

Example: Program Graph

11

DD-Paths • A DD-Path is a chain obtained from a program graph,

where a chain is a path in which the initial and terminal nodes are distinct, and every interior node has indegree = 1, and outdegree = 1.

• DD-Paths are used to create DD-Path Graphs.

• An example of a chain is shown below:

Initial node Internal nodes Final node

12

Test Coverage Metrics

• The motivation of using DD-paths is that they enable very precise descriptions of test coverage.

• In our quest to identify gaps and redundancy in our test cases as these are used to exercise (test) different aspects of a program we use formal models of the program structure to reason about testing effectiveness.

• Test coverage metrics are a device to measure the extent to which a set of test cases covers a program.

13

Test Coverage MetricsMetric Description of Coverage

C0 Every Statement

C1 Every DD-Path

C1P Every predicate to each outcome

C2 C1 Coverage + loop coverage

Cd C1 Coverage + every dependent pair of DD-Paths

CCC Condition coverage

CCDC Condition Decision coverage

CMCDC Modified Condition Decision coverage

CMCC Multiple condition coverage

Cik Every program path that contains up to k repetitions of a loop (usually k=2)

Cstat “Statistically significant” fraction of paths

C∞ All possible execution paths

14

Statement Coverage C0

• Statement coverage based testing aims to devise test cases that collectively exercise all statements in a program.

• Equivalent to Node Coverage of CFG

• Minimal Testing Criteria for code based testing

• All statements can be covered without covering outcomes of Decision Nodes

• Node Coverage of CFG does not guarantee Link Coverage

15

Example

16

Statement Coverage

• The following test case achieves statement coverage

• A = 2, B = 0, X = 3

• However, branch coverage is not achieved

17

Decision Coverage C1

• Decision coverage (or branch coverage, or predicate coverage) based testing aims to devise test cases that evaluate each decision of the program to True and False.

• Here the term simple “Decision” refers to either a single predicate or a compound Boolean expression that is considered as a single unit that evaluates to True or False. This amounts to traversing every edge in the DD-Path graph.

• For example in predicate coverage for the condition if(A or B) then C we could consider the test cases A=True, B= False

(true case), and A=False, B=False (false case). Note if the program was encoded as if(A) then C we would not detect any problem.

18

Decision Coverage C1

1 2

T F

T F

Here a T,T and F,F combination willsuffice to have Decisioncoverage C1

P1

P2

19

Example

20

Decision Coverage

• Decision coverage or Branch coverage is achieved by:

• A = 3, B = 0, X = 3 • A = 2, B = 1, X = 1

21

C1P Coverage

• This is the same as the C1 but now we must consider test cases that exercise all possible outcomes of the choices T,T, T,F, F,T, F,F for the predicates P1, and P2 respectively, in the DD-Path graph.

T F

T F

P1

P2

22

C1P Coverage

• The above test cases provide Decision coverage by setting the two decision outcomes to T, F and F, T values.

• C1P Coverage would two more test cases, i.e., for decision outcomes T, T and F, F.

23

Condition Coverage

• It requires that all conditions within a decision assume both T and F values

• Condition coverage is achieved by:

• A = 2, B = 0, X = 4 • A = 1, B = 1, X = 1

24

Condition-Decision Coverage

• Decision coverage does not subsume condition coverage

• Nor does condition coverage subsume decision coverage.

• For example, these test cases achieve condition coverage but not decision coverage:

• A = 1, B = 0, X = 3 • A = 2, B = 1, X = 1

25

Condition-Decision Coverage

• Requires a set of test cases that cover all decision and condition outcomes

26

Multiple Condition Coverage Testing

• Now if we consider that the predicate P1 is a compound predicate (i.e. (A or B)) then Multiple Condition Coverage Testing requires that each possible combination of inputs be tested for each decision.

• Example: “if (A or B)” requires 4 test cases:A = True, B = TrueA = True, B = FalseA = False, B = TrueA = False, B = False

• The problem: For n conditions, 2n test cases are needed, and this grows exponentially with n

27

Example

28

Multiple Condition Coverage• Requires a set of test cases that cover all

combinations of condition outcomes in each decision

• In the example, 8 combinations need to be covered:

29

Multiple Condition Coverage• These can be covered by following 4 test

cases:

30

Multiple Condition-Decision Coverage (MCDC)

• Requires that each condition within a condition determine outcome of the decision

• Stronger than both Decision coverage and Condition coverage

• Example:

(C1 or C2) and C3

31

Loop Coverage

• The simple view of loop testing coverage is that we must devise test cases that exercise the two possible outcomes of the decision of a loop condition that is one to traverse the loop and the other to exit (or not enter) the loop.

• An extension would be to consider a modified boundary value analysis approach where the loop index is given a minimum, minimum +, a nominal, a maximum -, and a maximum value or even robustness testing.

• Once a loop is tested, then the tester can collapse it into a single node to simplify the graph for the next loop tests. In the case of nested loops we start with the inner most loop and we proceed outwards.

• If loops are knotted then we must apply data flow analysis testing techniques, that we will examine later in the course.

32

Statistically Significant Path Coverage Testing

• Exhaustive testing of software is not practical because variable input values and variable sequencing of inputs result in too many possible combinations to test.

• Techniques have been developed for applying statistical methods to derive sample test cases that address how to select the best sample of test cases and provide a statistical level of confidence or probability that a program implements its functional specification correctly.

• The goal of statistically significant coverage is to develop methods for software testing based on statistical methods, which may use usage information from operational profiles to develop methods for software testing.