White-Box Testing

Pfleeger, S. Software Engineering Theory and Practice 2nd Edition. Prentice Hall, 2001.Ghezzi, C. et al., Fundamentals of Software Engineering. Prentice Hall, 2002.Pressman, R., Software Engineering A Practitioner’s Approach, Mc Graw Hill, 2005.Hutchinson, Marnie, Software Testing Fundamentals, Wiley, 2003.

0310

Coverage

• Statement

• Branch

• Condition

• Path

• Def-use

• Others

Statement Coverage

• Every statement gets executed at least once

• Every node in the CFG gets visited at least once

Example (from Hutchinson, Software Testing Fundamentals, Wiley, 2003)

2

3

4

7

5

1

8

9

6

2

3

4

7

5

1

8

9

6

Examples: number of paths needed

for Statement Coverage? (from Hutchinson, Software Testing Fundamentals, Wiley, 2003)

2

3

4

7

5

1

8

9

6

2

3

4

7

5

1

8

9

6

Examples: number of paths needed

for Statement Coverage? (from Hutchinson, Software Testing Fundamentals, Wiley, 2003)

2

3

4

7

5

1

8

9

6

2

3

4

7

5

1

8

9

6

4 1

Branch coverage

• Every decision is made true and false• Every edge in a CFG of the program gets

traversed at least once• Also known as

– Decision coverage– All edge coverage– Basis path coverage

• Branch is finer than statement• C1 is finer than C2 if

T1C1 T2C2 T2 T1

Examples: number of paths needed

for Branch Coverage? (from Hutchinson, Software Testing Fundamentals, Wiley, 2003)

2

3

4

7

5

1

8

9

6

2

3

4

7

5

1

8

9

6

Examples: number of paths needed

for Branch Coverage? (from Hutchinson, Software Testing Fundamentals, Wiley, 2003)

2

3

4

7

5

1

8

9

6

2

3

4

7

5

1

8

9

6

5 2

Condition coverage

• Every complex condition is made true and false by every possible combination– E.G., (x and y)

• x = true, y = true• x=false, y=true• x = true, y= false• x =false, y = false

• There are lots of different types of condition coverage: Condition, multiple condition, condition/decision, modified condition/decision (MCDC), …I’m only covering the simplest.

• Condition coverage is not finer than branch coverage – There are pathological cases where you can achieve Condition and not

Branch– Under most circumstances, achieving Condition achieves Branch

Condition Coverage: CFGs• One way to determine the number of paths is to break the compound

conditional into atomic conditionals• Suppose you were writing the CFG for the assembly language

implementation of the control construct If (A AND B) then

CEndif

(short circuit eval) (no short circuit eval) LD A LD A ; in general, lots of BZ :endif LAND B ; code for A and

B LD B BZ :endif BZ :endif JSR C JSR C :endif nop:endif nop

AND Condition

1. read (a,b); 2. if (a == 0 && b == 0) then {3. c a + b }4. else c a * b

paths:1, 2A,2B,3, J1, 2A, 2B, 4, J1, 2A, 4, J

2A

2B

Join

3

4

1

OR Condition

1. read (a,b); 2. if (a == 0 || b == 0) then }3. c a + b;4. while( c < 100)5. c a + b;}

Paths:1, 2A, 3, 4, 5, 4 … J1, 2A, 3, 4, J1, 2A, 2B, 3, 4, 5, 4, … J1,2A, 2B, J

2A

3

Join

4

2B

5

1

Path

• A path is a sequence of statements

• A path is sequence of branches

• A path is a sequence of edges

Examples: number of paths needed

for Path Coverage? (from Hutchinson, Software Testing Fundamentals, Wiley, 2003)

2

3

4

7

5

1

8

9

6

2

3

4

7

5

1

8

9

6

Examples: number of paths needed

for Path Coverage? (from Hutchinson, Software Testing Fundamentals, Wiley, 2003)

2

3

4

7

5

1

8

9

6

2

3

4

7

5

1

8

9

6

5 16

Path Coverage-1• Every distinct path through code

is executed at least once

• Example1. read (x)2. read (z)3. if x 0 then begin4. y x * z;5. x z end6. else print ‘Invalid’ 7. if y > 1 then8. print y9. else print ‘Invalid’

• Test Paths:1, 2, 3, 4, 5, J1, 7, 8, J21, 2, 3, 4, 5, J1, 7, 9, J21, 2, 3, 6, J1, 7, 8, J2,1, 2, 3, 6, J1, 7, 9, J2

1,2,3

4,5

Join1

6

7

8

Join2

9

Counting Paths

• It is not feasible to calculate the total number of paths

Linearly independent paths

• It is feasible to calculate the number of linearly independent paths

• The number of linearly independent paths is the number of end-to-end paths required to touch every path segment at least once

• A linearly independent path introduces at least one new set of process statements or a new condition

Independent Paths• Consider the following encoding:

– Label each edge in the CFG with a positive integer

– Describe a path using a vector where each element of the vector is the number of times the edge at that index is traversed:

A

D

1

B C

2

43

• The path ABD can be written as

[1, 0, 1, 0]

Independent Paths• Consider the following encoding:

– Label each edge in the CFG with a positive integer

– Describe a path using a vector where each element of the vector is the number of times the edge at that index is traversed:

• Paths can be combined by adding or subtracting these path vectors

• A basis set is a set of linearly independent paths• Each path in the basis set can not be formed as a

combination of other paths in the basis set

Basis Set• A basis set:

– p13: [1,0,1,0]– p14: [1,0,0,1]– p23: [0,1,1,0]

• The path 24 can be constructed by p23+p14-p13[0,1,1,0] + [1,0,0,1] – [1,0,1,0] = [0,1,0,1]

A

D

1

B

2

43

• p13, p24 is not a basis set– p13: [1,0,1,0]

– p24: [0,1,0,1]

– (No way to construct [1,0,0,1])

• Another basis set:– p13: [1,0,1,0]

– p14: [1,0,0,1]

– p23: [0,1,0,1]

Examples: number of linearly independent

paths? (from Hutchinson, Software Testing Fundamentals, Wiley, 2003)

2

3

4

7

5

1

8

9

6

2

3

4

7

5

1

8

9

6

Examples: number of linearly independent

paths? (from Hutchinson, Software Testing Fundamentals, Wiley, 2003)

2

3

4

7

5

1

8

9

6

2

3

4

7

5

1

8

9

6

5 5

Cyclomatic Complexity• Software metric for the logical complexity of a

program.• Defines the number of independent paths in the

basis set of a program• Provides the upper bound for the number of tests

that must be conducted to ensure that all statements been have executed at least once

• For Edges (E) and Nodes (N) V(G) = E – N + 2

Independent Path Coverage• Basis set does not yield minimal

test set• Example

1. read (x)2. read (z)3. if x 0 then begin4. y x * z;5. x z end6. else print ‘Invalid’ 7. if y > 1 then8. print y9. else print ‘Invalid’

• Cyclomatic complexity: 3• Test Paths:

1, 2, 3, 4, 5, J1, 7, 8, J21, 2, 3, 4, 5, J1, 7, 9, J21, 2, 3, 6, J1, 7, 8, J2,

1,2,3

4,5

Join1

6

7

8

Join2

9

Def-Use Coverage• Def-use coverage: every path

from every definition of every variable to every use of that definition is exercised in some test.

• Example1. read (x)2. read (z)3. if x 0 then begin4. y x * z;5. x z end6. else print ‘Invalid’ 7. if y > 1 then8. print y9. else print ‘Invalid’

1,2,3

4,5

Join

6

7

8

Join

9

Def: x, zUse: x

Def: y, xUse: x, z

Use: none

Use: y

Use: yUse: none

Test Path: 1, 2, 3, 4, 5, 7, 8, J

Strength of Coverage

Statement

Branch

Def-Use

Path

Arrows point from weaker to stronger coverage.

Stronger coverage requires more test cases.

Condition

Groups:

A

B

D

B1D2D1

E

C

A is read(x, y, z)

B is if ((x>10) && (y<20))

D is if (z >10)

E is repeat until g = 5

Show me different minimal test sets that satisfy the 4 criteria

Statement:

A

B

D

B1D2D1

E

C

A is read(x, y, z)

B is if ((x>10) && (y<20))

D is if (z >10)

E is repeat until g = 5

•A,B,B1,E C: x(A) = 0; g (E) =5

•A, B,D,D1,E C: x(A)=11, y(A) = 11, z(A)=11, g(E)=5

•A, B, D, D2, E, C: x(A)=11, y(A) = 11, z(A)=9, g(E)=5

Branch:

A

B

D

B1D2D1

E

C

A is read(x, y, z)

B is if ((x>10) && (y<20))

D is if (z >10)

E is repeat until g = 5

•A,B,B1,E C: x(A) = 0; g (E) =5

•A, B,D,D1,E C: x(A)=11, y(A) = 11, z(A)=11, g(E)=5

•A, B, D, D2, E, C: x(A)=11, y(A) = 11, z(A)=9, g(E)=5

•A,B,B1,E,A …: x(A) = 0; g (E) =4

Condition:

A

B

D

B1D2D1

E

C

A is read(x, y, z)

B is if ((x>10) && (y<20))

D is if (z >10)

E is repeat until g = 5

•A,B,B1,E C: x(A) = 0; g (E) =5

•A,B,B1,E C: x(A) = 11; y(A) = 22; g (E) =5

•A,B,B1,E C: x(A) = 0; y(A) = 22; g (E) =5

•A, B,D,D1,E C: x(A)=11, y(A) = 11, z(A)=11, g(E)=5

•A, B, D, D2, E, C: x(A)=11, y(A) = 11, z(A)=9, g(E)=5

•A,B,B1,E,A …: x(A) = 0; g (E) =4

Independent Paths:

A

B

D

B1D2D1

E

C

A is read(x, y, z)

B is if ((x>10) && (y<20))

D is if (z >10)

E is repeat until g = 5

•A,B,B1,E C: x(A) = 0; g (E) =5

•A, B,D,D1,E C: x(A)=11, y(A) = 11, z(A)=11, g(E)=5

•A, B, D, D2, E, C: x(A)=11, y(A) = 11, z(A)=9, g(E)=5

•A,B,B1,E,A,B,B1,E,C: x(A) = 0; g (E) =4