Ch6: Software Verification

2

Statement coverage criterion

Informally:

Formally:

Difficult to minimize the number of test cases and still ensure the execution of all statements

3

Statement coverage criterion (contd..)

Example

read (x); read (y);if x > 0 then

write ("1");else

write ("2");end if;if y > 0 then

write ("3");else

write ("4");end if; 

{<x = 2, y = 3>, <x = - 13, y = 51>, <x = 97, y = 17>, <x = - 1, y = - 1>}covers all statements

{<x = - 13, y = 51>, <x = 2, y = - 3>} is minimal

4

Statement coverage criterion

if x < 0 then x := -x;end if z = :x;

This may be rewritten as:

if x < 0 then x := -x;else null;end ifz := x;

x = -2 covers all statements, but does not coverthe case when x is positive and the then branch is not entered.

Have to test both x < 0, and x >= 0 cases to ensure completeness

5

Edge coverage criterion

Criterion:

What is a control flow graph, and how to construct it?

Finer than statement coverage criterion

6

Control graph construction rules

I/O, assignment, or procedure call

G G1 2

if-then-else

G1

if-then

G1

while loop

G1

G 2

two sequential statements

7

Simplification

A sequence of edges can be collapsed into just one edge

. . .n n nnn k-1 k1 2 3

n1 nk

8

Weakness of edge coverage criterion

found := false; counter := 1;while (not found) and counter < number_of_items loop

if table (counter) = desired_element then found := true;

end if;counter := counter + 1;

end loop;if found then

write ("the desired element is in the table");else

write ("the desired element is not in the table");end if;  test cases: (1) empty table, (2) table with 3 items, second of

which is the item to look for

9

Let us consider an algorithm which searches for an element in the table. Now, consider if we execute this program fragment with the first test case, which is an empty table. In this case, the while loop is not executed at all. In the second test case, when the table has three items, in the first pass through the loop, the element is not found, so the second pass through the loop is executed. In the second pass through the loop, the element is discovered, and the loop exits. Now, with these two test cases, both the branches of the if-then-else statement are executed. However, this will not reveal the error. This is because notice that the while loop has two clauses. The two test cases exercised both the possible values of the first part (namely, element found & not found), but only possible value of the second clause, namely, if counter < number_of_elements_ in the loop. It does not exercise the second possibility of the second clause, namely, what would happen if the counter = number_of_elements_in_the_loop. Thus, edge coverage criterion does not reveal the error. A stricter criterion which requires all the four possiblities (2 of the first clause) and 2 of the second clause would have revealed the error.

10

Condition coverage criterion

Condition coverage:

Finer than edge coverage

11

Weakness of condition coverage criterion

 if x ≠ 0 then

y := 5; else

z := z - x; end if;if z > 1 then

z := z / x; else

z := 0; end if; 

{<x = 0, z = 1>, <x = 1, z = 3>} causes the execution of all edges, but fails to expose the risk of a division by zero

12

Path-coverage criterion

Path coverage criteria:

Finer than previous kinds of coverage Issues with path coverage criteria

Path coverage criterion may be used as a guide to determining a few critical paths.

13

White-box testing (contd..)

Guidelines for testing loops:

Guidelines for IF and CASE statements:

Once a criterion has been chosen, actual input values may be chosen.

14

White-box testing (contd..)

The presence of unreachable statements may mean that 100% coverage is not achieved.

In summary, whatever coverage criterion we decide, human intervention is needed to solve problems (reachability, etc.)

15

White-box testing (contd..)

Problems with white-box testing:

16

Black-box testing

What is black-box testing:

17

Black-box testing (contd..)

The program receives as input a record describing an invoice. (A detailed description of the format of the record is given.) The invoice must be inserted into a file of invoices that is sorted by date. The invoice must be inserted in the appropriate position: If other invoices exist in the file with the same date, then the invoice should be inserted after the last one. Also, some consistency checks must be performed: The program should verify whether the customer is already in a corresponding file of customers, whether the customer’s data in the two files match, etc.

Example specification

18

Black-box testing (contd..)

Did you consider these cases?

• An invoice whose date is the current date• An invoice whose date is before the current date

(This might be even forbidden by law)This case, in turn, can be split into the two following subcases: • An invoice whose date is the same as that

some existing invoice • An invoice whose date does not exist in any

previously recorded invoice• Several incorrect invoices, checking different types of

inconsistencies

19

Systematic black-box techniques

Syntax-driven testing Decision table based testing Cause-effect graph based testing

20

Syntax driven testing

Applicability:

Role of formal specification:

21

Syntax driven testing (contd..)

Example

Consider testing an interpreter of the following language

<expression> ::= <expression> + <term>|<expression> - <term> | <term>

<term> ::= <term> * <factor> | <term> / <factor> | <factor>

<factor> ::= ident | ( <expression>)

22

Syntax driven testing (contd..)

How to complete coverage principle:

Automated generation of test cases:

Minimal test set: