Black-Box Testing Techniques III

Prepared by

Stephen M. Thebaut, Ph.D.University of Florida

Software Testing and Verification

Lecture 6

Black-Box Test Case Design Techniques Considered

• Partition testing• Combinatorial Approaches• Boundary Value Analysis• Intuition & Experience

Another Cause-Effect Example: Symbol Table Storage SpecificationThe conditions for storing an identifier in one of two symbol tables are: (a) must be from 2 to 8 characters in length; (b) first character must be a letter or “$”; (c) other characters must be a letter or digit.If the first character is a letter, the identifier will be stored in symbol table A. If the first character is “$”, it will be stored in symbol table B.If the first character is neither a letter nor “$”, or if condition (c) is not satisfied, error message J11 is output.If condition (a) is not satisfied, error message J12 is output.

What are the “Effects”?

The conditions for storing an identifier in one of two symbol tables are: (a) must be from 2 to 8 characters in length; (b) first character must be a letter or “$”; (c) other characters must be a letter or digit.If the first character is a letter, the identifier will be stored in symbol table A. If the first character is “$”, it will be stored in symbol table B.If the first character is neither a letter nor “$”, or if condition (c) is not satisfied, error message J11 is output.If condition (a) is not satisfied, error message J12 is output.

What are the “Effects”?

The conditions for storing an identifier in one of two symbol tables are: (a) must be from 2 to 8 characters in length; (b) first character must be a letter or “$”; (c) other characters must be a letter or digit.If the first character is a letter, the identifier will be stored in symbol table A. If the first character is “$”, it will be stored in symbol table B.If the first character is neither a letter nor “$”, or if condition (c) is not satisfied, error message J11 is output.If condition (a) is not satisfied, error message J12 is output.

What are the “Causes”?

The conditions for storing an identifier in one of two symbol tables are: (a) must be from 2 to 8 characters in length; (b) first character must be a letter or “$”; (c) other characters must be a letter or digit.If the first character is a letter, the identifier will be stored in symbol table A. If the first character is “$”, it will be stored in symbol table B.If the first character is neither a letter nor “$”, or if condition (c) is not satisfied, error message J11 is output.If condition (a) is not satisfied, error message J12 is output.

What are the “Causes”?

The conditions for storing an identifier in one of two symbol tables are: (a) must be from 2 to 8 characters in length; (b) first character must be a letter or “$”; (c) other characters must be a letter or digit.If the first character is a letter, the identifier will be stored in symbol table A. If the first character is “$”, it will be stored in symbol table B.If the first character is neither a letter nor “$”, or if condition (c) is not satisfied, error message J11 is output.If condition (a) is not satisfied, error message J12 is output.

Causes and Effects

Causes: Effects:(1) 2 no. chars 8 (31) store in table A(2) 1st char is letter (32) store in table B(3) 1st char is $ (33) output msg J11(4) other chars only letters/digits (34) output msg J12

only(35) output msgs J11 and J12

Boolean Graphs

(1)

(2)

(3)

(4)

(31)

(32)

[2,8]

let

$

others let/dig

−> A

−> B

E

Boolean Graphs

(1)

(2)

(3)

(4)

(31)

(32)

[2,8]

let

$

others let/dig

−> A

−> B

E

Another Cause-Effect Example: Symbol Table Storage SpecificationThe conditions for storing an identifier in one of two symbol tables are: (a) must be from 2 to 8 characters in length; (b) first character must be a letter or “$”; (c) other characters must be a letter or digit.If the first character is a letter, the identifier will be stored in symbol table A. If the first character is “$”, it will be stored in symbol table B.If the first character is neither a letter nor “$”, or if condition (c) is not satisfied, error message J11 is output.If condition (a) is not satisfied, error message J12 is output.

Boolean Graphs (cont’d)

(1)

(2)

(3)

(4)

(31)

(32)

[2,8]

let

$

others let/dig

−> A

−> B

E

Л

Boolean Graphs

(1)

(2)

(3)

(4)

(31)

(32)

[2,8]

let

$

others let/dig

−> A

−> B

E

Another Cause-Effect Example: Symbol Table Storage SpecificationThe conditions for storing an identifier in one of two symbol tables are: (a) must be from 2 to 8 characters in length; (b) first character must be a letter or “$”; (c) other characters must be a letter or digit.If the first character is a letter, the identifier will be stored in symbol table A. If the first character is “$”, it will be stored in symbol table B.If the first character is neither a letter nor “$”, or if condition (c) is not satisfied, error message J11 is output.If condition (a) is not satisfied, error message J12 is output.

Boolean Graphs (cont’d)

(1)

(2)

(3)

(4)

(31)

(32)

[2,8]

let

$

others let/dig

−> A

−> B

E

Л

Boolean Graphs (cont’d)

(1)

(2)

(3)

(4)

(31)

(32)

[2,8]

let

$

others let/dig

−> A

−> B

E

Л

Л

Boolean Graphs (cont’d)

(1)

(2)

(3)

(4)

(33)

(34)

(35)

[2,8]

let

$

others let/dig

J11 only

J12 only

J11 & J12

E

Boolean Graphs (cont’d)

(1)

(2)

(3)

(4)

(33)

(34)

(35)

[2,8]

let

$

others let/dig

J11 only

J12 only

J11 & J12

E

Another Cause-Effect Example: Symbol Table Storage SpecificationThe conditions for storing an identifier in one of two symbol tables are: (a) must be from 2 to 8 characters in length; (b) first character must be a letter or “$”; (c) other characters must be a letter or digit.If the first character is a letter, the identifier will be stored in symbol table A. If the first character is “$”, it will be stored in symbol table B.If the first character is neither a letter nor “$”, or if condition (c) is not satisfied, error message J11 is output.If condition (a) is not satisfied, error message J12 is output.

Boolean Graphs (cont’d)

(1)

(2)

(3)

(4)

(33)

(34)

(35)

[2,8]

let

$

others let/dig

J11 only

J12 only

J11 & J12

E (A)Л

Another Cause-Effect Example: Symbol Table Storage SpecificationThe conditions for storing an identifier in one of two symbol tables are: (a) must be from 2 to 8 characters in length; (b) first character must be a letter or “$”; (c) other characters must be a letter or digit.If the first character is a letter, the identifier will be stored in symbol table A. If the first character is “$”, it will be stored in symbol table B.If the first character is neither a letter nor “$”, or if condition (c) is not satisfied, error message J11 is output.If condition (a) is not satisfied, error message J12 is output.

Boolean Graphs (cont’d)

(1)

(2)

(3)

(4)

(33)

(34)

(35)

[2,8]

let

$

others let/dig

J11 only

J12 only

J11 & J12

E (A)

(B)V

Л

Another Cause-Effect Example: Symbol Table Storage SpecificationThe conditions for storing an identifier in one of two symbol tables are: (a) must be from 2 to 8 characters in length; (b) first character must be a letter or “$”; (c) other characters must be a letter or digit.If the first character is a letter, the identifier will be stored in symbol table A. If the first character is “$”, it will be stored in symbol table B.If the first character is neither a letter nor “$”, or if condition (c) is not satisfied, error message J11 is output.If condition (a) is not satisfied, error message J12 is output.

Boolean Graphs (cont’d)

(1)

(2)

(3)

(4)

(33)

(34)

(35)

[2,8]

let

$

others let/dig

J11 only

J12 only

J11 & J12

E (A)

(B)V

Л

Boolean Graphs (cont’d)

(1)

(2)

(3)

(4)

(33)

(34)

(35)

[2,8]

let

$

others let/dig

J11 only

J12 only

J11 & J12

E (A)

(B)V

Л

Another Cause-Effect Example: Symbol Table Storage SpecificationThe conditions for storing an identifier in one of two symbol tables are: (a) must be from 2 to 8 characters in length; (b) first character must be a letter or “$”; (c) other characters must be a letter or digit.If the first character is a letter, the identifier will be stored in symbol table A. If the first character is “$”, it will be stored in symbol table B.If the first character is neither a letter nor “$”, or if condition (c) is not satisfied, error message J11 is output.If condition (a) is not satisfied, error message J12 is output.

Boolean Graphs (cont’d)

(1)

(2)

(3)

(4)

(33)

(34)

(35)

[2,8]

let

$

others let/dig

J11 only

J12 only

J11 & J12

E

Л

(A)

(B)V

Л

Boolean Graphs (cont’d)

(1)

(2)

(3)

(4)

(33)

(34)

(35)

[2,8]

let

$

others let/dig

J11 only

J12 only

J11 & J12

E

Л

(A)

(B)V

Л

Another Cause-Effect Example: Symbol Table Storage SpecificationThe conditions for storing an identifier in one of two symbol tables are: (a) must be from 2 to 8 characters in length; (b) first character must be a letter or “$”; (c) other characters must be a letter or digit.If the first character is a letter, the identifier will be stored in symbol table A. If the first character is “$”, it will be stored in symbol table B.If the first character is neither a letter nor “$”, or if condition (c) is not satisfied, error message J11 is output.If condition (a) is not satisfied, error message J12 is output.

Boolean Graphs (cont’d)

(1)

(2)

(3)

(4)

(33)

(34)

(35)

[2,8]

let

$

others let/dig

J11 only

J12 only

J11 & J12

E

Л

(A)

(B)V

Л

Boolean Graphs (cont’d)

(1)

(2)

(3)

(4)

(33)

(34)

(35)

[2,8]

let

$

others let/dig

J11 only

J12 only

J11 & J12

E

Л

(A)

(B)V

Л

Boolean Graphs (cont’d)

(1)

(2)

(3)

(4)

(33)

(34)

(35)

[2,8]

let

$

others let/dig

J11 only

J12 only

J11 & J12

E

Л

Л

(A)

(B)V

Л

Boolean Graphs (cont’d)

(1)

(2)

(3)

(4)

(33)

(34)

(35)

[2,8]

let

$

others let/dig

J11 only

J12 only

J11 & J12

E

Л

Л

(A)

(B)V

Л

Boolean Graphs (cont’d)

(1)

(2)

(3)

(4)

(33)

(34)

(35)

[2,8]

let

$

others let/dig

J11 only

J12 only

J11 & J12

E

Л

Л

(A)

(B) Л

V

Л

Boolean Graphs (cont’d)

(1)

(2)

(3)

(4)

(33)

(34)

(35)

[2,8]

let

$

others let/dig

J11 only

J12 only

J11 & J12

E

Л

Л

(A)

(B) Л

V

Л

A Variation on Test Case Selection Strategy #3• Test case selection “Strategy #3”

considers ALL feasible combinations of connected Cause values that result in each Effect being True.

• For complex specifications, this can be impractical.

• We now consider a variation on this strategy which “culls” all but the combinations “of greatest interest”.

A Variation on Test Case Selection Strategy #3• Test case selection “Strategy #3”

considers ALL feasible combinations of connected Cause values that result in each Effect being True.

• For complex specifications, this can be impractical.

• We now consider a variation on this strategy which “culls” all but the combinations “of greatest interest”.

A Variation on Test Case Selection Strategy #3• Test case selection “Strategy #3”

considers ALL feasible combinations of connected Cause values that result in each Effect being True.

• For complex specifications, this can be impractical.

• We now consider a variation on this strategy which “culls” all but the combinations “of greatest interest”.

Test Case Selection Strategy #3 Plus “Culling Rules”REPEAT

Select the next (initially, the first) Effect.Tracing back through the graph (right to left), find all feasible combinations of connected Cause values that result in the Effect being True, subject to the following culling rules:1. When encountering an nth-degree

OR-node that must be True, consider only those n combinations for which exactly one incoming edge is True.

Test Case Selection Strategy #3 Plus “Culling Rules”REPEAT

Select the next (initially, the first) Effect.Tracing back through the graph (right to left), find all feasible combinations of connected Cause values that result in the Effect being True, subject to the following culling rules:1. When encountering an nth-degree

OR-node that must be True, consider only those n combinations for which exactly one incoming edge is True.

Test Case Selection Strategy #3 Plus “Culling Rules” (cont’d)

2. When encountering an nth-degree AND-node that must be False, consider only those n combinations for which exactly one incoming edge is False.

For each new such combination found:Determine values of all other Effects, andEnter values for each Cause and Effect in anew column of the test case coverage

matrix.UNTIL each Effect has been selected.

Test Case Selection Strategy #3 Plus “Culling Rules” (cont’d)

2. When encountering an nth-degree AND-node that must be False, consider only those n combinations for which exactly one incoming edge is False.

For each new such combination found:Determine values of all other Effects, andEnter values for each Cause and Effect in anew column of the test case coverage

matrix.UNTIL each Effect has been selected.

Rationale for these Culling Rules?• Number of combinations decreases by a

factor of O(2 ) to O(n) at each true OR node and each false AND node.

• Idea: cover only the minimally sufficient conditions for the desired result.

n

T

(V F

(Л

Applying Strategy #3 Plus Culling Rules

(1)

(2)

(3)

(4)

(31)

(32)

[2,8]

let

$

others let/dig

−> A

−> B

E

Л

Л

Applying Strategy #3 Plus Culling Rules

(1)

(2)

(3)

(4)

(31)

(32)

[2,8]

let

$

others let/dig

−> A

−> B

E

Л

Л

Coverage Matrix

TEST CASESCAUSES 1 2 3 4 5 6 7 8 9 10

2 no. chars 8 (1)

T

1st char is letter (2)

T

1st char is $ (3)

F

others letters/digits (4)

T

EFFECTSstore in table A (31)

T

store in table B (32)

F

output J11 only (33)

F

output J12 only (34)

F

output J11 & J12 (35)

F

Applying Strategy #3 Plus Culling Rules (cont’d)

(1)

(2)

(3)

(4)

(31)

(32)

[2,8]

let

$

others let/dig

−> A

−> B

E

Л

Л

Coverage Matrix (cont’d)

TEST CASESCAUSES 1 2 3 4 5 6 7 8 9 10

2 no. chars 8 (1)

T T

1st char is letter (2)

T F

1st char is $ (3)

F T

others letters/digits (4)

T T

EFFECTSstore in table A (31)

T F

store in table B (32)

F T

output J11 only (33)

F F

output J12 only (34)

F F

output J11 & J12 (35)

F F

Applying Strategy #3 Plus Culling Rules (cont’d)

(1)

(2)

(3)

(4)

(33)

(34)

(35)

[2,8]

let

$

others let/dig

J11 only

J12 only

J11 & J12

E

Л

Л

(A)

(B) Л

V

Л

Applying Strategy #3 Plus Culling Rules (cont’d)

(33) 1, B

Applying Strategy #3 Plus Culling Rules (cont’d)

(1)

(2)

(3)

(4)

(33)

(34)

(35)

[2,8]

let

$

others let/dig

J11 only

J12 only

J11 & J12

E

Л

Л

(A)

(B) Л

V

Л

Applying Strategy #3 Plus Culling Rules (cont’d)

(33) 1, B B (4 V A)

Applying Strategy #3 Plus Culling Rules (cont’d)

(33) 1, B B (4 V A) (4, A) V (4, A) V (4, A) (T,T) (T,F) (F,T)

Applying Strategy #3 Plus Culling Rules (cont’d)

(33) 1, B B (4 V A) (4, A) V (4, A) V (4, A) (T,T) (T,F) (F,T) culled (rule 1)

Applying Strategy #3 Plus Culling Rules (cont’d)

(33) 1,4, A 1, 4, A

Applying Strategy #3 Plus Culling Rules (cont’d)

(1)

(2)

(3)

(4)

(33)

(34)

(35)

[2,8]

let

$

others let/dig

J11 only

J12 only

J11 & J12

E

Л

Л

(A)

(B) Л

V

Л

Applying Strategy #3 Plus Culling Rules (cont’d)

(33) 1,4, A 1, 4, A

A 2, 3

Applying Strategy #3 Plus Culling Rules (cont’d)

(33) 1,4, A 1, 4, 2, 3

Applying Strategy #3 Plus Culling Rules (cont’d)

(33) 1,4, A A (2, 3)

1, 4, 2, 3

Applying Strategy #3 Plus Culling Rules (cont’d)

(33) 1,4, A A (2, 3) (2, 3) V (2, 3) V (2, 3) (F,F) (F,T) (T,F)

1, 4, 2, 3

Applying Strategy #3 Plus Culling Rules (cont’d)

(33) 1,4, A A (2, 3) (2, 3) V (2, 3) V (2, 3) (F,F) (F,T) (T,F) culled (rule 2) and infeasible

1, 4, 2, 3

Applying Strategy #3 Plus Culling Rules (cont’d)

(33) 1, 4, 2, 3 1, 4, 2, 3

1, 4, 2, 3

Applying Strategy #3 Plus Culling Rules (cont’d)

(33) 1, 4, 2, 3 1, 4, 2, 3

1, 4, 2, 3

Coverage Matrix (cont’d)

TEST CASESCAUSES 1 2 3 4 5 6 7 8 9 10

2 no. chars 8 (1)

T T T T T

1st char is letter (2)

T F T F F

1st char is $ (3)

F T F T F

others letters/digits (4)

T T F F T

EFFECTSstore in table A (31)

T F F F F

store in table B (32)

F T F F F

output J11 only (33)

F F T T T

output J12 only (34)

F F F F F

output J11 & J12 (35)

F F F F F

Applying Strategy #3 Plus Culling Rules (cont’d)

(1)

(2)

(3)

(4)

(33)

(34)

(35)

[2,8]

let

$

others let/dig

J11 only

J12 only

J11 & J12

E

Л

Л

(A)

(B) Л

V

Л

Applying Strategy #3 Plus Culling Rules (cont’d)

(34) 1, B

Applying Strategy #3 Plus Culling Rules (cont’d)

(1)

(2)

(3)

(4)

(33)

(34)

(35)

[2,8]

let

$

others let/dig

J11 only

J12 only

J11 & J12

E

Л

Л

(A)

(B) Л

V

Л

Applying Strategy #3 Plus Culling Rules (cont’d)

(34) 1, B B (4 V A)

Applying Strategy #3 Plus Culling Rules (cont’d)

(34) 1, B B (4 V A) 4, A

Applying Strategy #3 Plus Culling Rules (cont’d)

(34) 1, 4, A

Applying Strategy #3 Plus Culling Rules (cont’d)

(1)

(2)

(3)

(4)

(33)

(34)

(35)

[2,8]

let

$

others let/dig

J11 only

J12 only

J11 & J12

E

Л

Л

(A)

(B) Л

V

Л

Applying Strategy #3 Plus Culling Rules (cont’d)

(34) 1, 4, A A (2, 3)

Applying Strategy #3 Plus Culling Rules (cont’d)

(34) 1, 4, A A (2, 3) (2, 3) V (2, 3) V (2, 3) (F,F) (F,T) (T,F)

Applying Strategy #3 Plus Culling Rules (cont’d)

(34) 1, 4, A A (2, 3) (2, 3) V (2, 3) V (2, 3) (F,F) (F,T) (T,F) culled (rule 2) and infeasible

Applying Strategy #3 Plus Culling Rules (cont’d)

(34) 1, 4, 2, 3 1, 4, 2, 3

Coverage Matrix (cont’d)

TEST CASESCAUSES 1 2 3 4 5 6 7 8 9 10

2 no. chars 8 (1)

T T T T T F F

1st char is letter (2)

T F T F F T F

1st char is $ (3)

F T F T F F T

others letters/digits (4)

T T F F T T T

EFFECTSstore in table A (31)

T F F F F F F

store in table B (32)

F T F F F F F

output J11 only (33)

F F T T T F F

output J12 only (34)

F F F F F T T

output J11 & J12 (35)

F F F F F F F

Applying Strategy #3 Plus Culling Rules (cont’d)

(1)

(2)

(3)

(4)

(33)

(34)

(35)

[2,8]

let

$

others let/dig

J11 only

J12 only

J11 & J12

E

Л

Л

(A)

(B) Л

V

Л

Applying Strategy #3 Plus Culling Rules (cont’d)(35) 1, B

Which are just the conditions associated witherror messages J11 (B) and J12 (1).

Combining these conditions from (33) and(34) yields:

(35) 1, 4, 2, 3 1, 4, 2, 3

1, 4, 2, 3

Coverage Matrix (cont’d)

TEST CASESCAUSES 1 2 3 4 5 6 7 8 9 10

2 no. chars 8 (1)

T T T T T F F

1st char is letter (2)

T F T F F T F

1st char is $ (3)

F T F T F F T

others letters/digits (4)

T T F F T T T

EFFECTSstore in table A (31)

T F F F F F F F F F

store in table B (32)

F T F F F F F F F F

output J11 only (33)

F F T T T F F F F F

output J12 only (34)

F F F F F T T F F F

output J11 & J12 (35)

F F F F F F F T T T

Coverage Matrix (cont’d)

TEST CASESCAUSES 1 2 3 4 5 6 7 8 9 10

2 no. chars 8 (1)

T T T T T F F

1st char is letter (2)

T F T F F T F T F F

1st char is $ (3)

F T F T F F T F T F

others letters/digits (4)

T T F F T T T F F T

EFFECTSstore in table A (31)

T F F F F F F F F F

store in table B (32)

F T F F F F F F F F

output J11 only (33)

F F T T T F F F F F

output J12 only (34)

F F F F F T T F F F

output J11 & J12 (35)

F F F F F F F T T T

Coverage Matrix (cont’d)

TEST CASESCAUSES 1 2 3 4 5 6 7 8 9 10

2 no. chars 8 (1)

T T T T T F F

1st char is letter (2)

T F T F F T F T F F

1st char is $ (3)

F T F T F F T F T F

others letters/digits (4)

T T F F T T T F F T

EFFECTSstore in table A (31)

T F F F F F F F F F

store in table B (32)

F T F F F F F F F F

output J11 only (33)

F F T T T F F F F F

output J12 only (34)

F F F F F T T F F F

output J11 & J12 (35)

F F F F F F F T T T

Coverage Matrix (cont’d)

TEST CASESCAUSES 1 2 3 4 5 6 7 8 9 10

2 no. chars 8 (1)

T T T T T F F F F F

1st char is letter (2)

T F T F F T F T F F

1st char is $ (3)

F T F T F F T F T F

others letters/digits (4)

T T F F T T T F F T

EFFECTSstore in table A (31)

T F F F F F F F F F

store in table B (32)

F T F F F F F F F F

output J11 only (33)

F F T T T F F F F F

output J12 only (34)

F F F F F T T F F F

output J11 & J12 (35)

F F F F F F F T T T

Complete Coverage Matrix

TEST CASESCAUSES 1 2 3 4 5 6 7 8 9 10

2 no. chars 8 (1)

T T T T T F F F F F

1st char is letter (2)

T F T F F T F T F F

1st char is $ (3)

F T F T F F T F T F

others letters/digits (4)

T T F F T T T F F T

EFFECTSstore in table A (31)

T F F F F F F F F F

store in table B (32)

F T F F F F F F F F

output J11 only (33)

F F T T T F F F F F

output J12 only (34)

F F F F F T T F F F

output J11 & J12 (35)

F F F F F F F T T T

Cause-Effect Analysis: Discussion Questions & Exercises• Under what circumstances should Cause-

Effect Analysis be used for test case design?

Recall...

• Program Specification: An ordered pair of numbers, (x, y), are input and a message is output stating whether they are in ascending order, descending order, or equal. If the input is other than an ordered pair of numbers, an error message is output.

• Equivalence Classes:{ (x, y) | x<y } (V){ (x, y) | x>y } (V){ (x, y) | x=y } (V){ input is other than an ordered

pair of numbers } (I)

Valid classes

Invalid class

A More Complex Case...

Part of a More Complex Program Specification:

Three numbers, x, y, and z, are input. If x is a whole number and less than 40, and if y is non-negative, the output is z+(y/x). If x is greater than or equal to 40, or if y is positive, or if z is odd and at least as large as x, then the output is...

• (Some) Valid Equivalence Classes:– { x | x is a whole number } (V)– { x | x < 40 } (V), { x | x ≥ 40 } (V)– { y | y = 0 } (V), { y | y > 0 } (V)– { z | z is odd } (V)– { (x, z) | z x } (V)

. . .

Cause-Effect Analysis: Discussion Questions & Exercises• Under what circumstances should Cause-

Effect Analysis be used for test case design?

Cause-Effect Analysis: Discussion Questions & Exercises• Under what circumstances should Cause-

Effect Analysis be used for test case design?Whenever a systematic means is needed to identify appropriate combinations of input "Causes" resulting in output "Effects". E.g., when dealing with complex, multiple-input situations. (In the absence of a systematic means, the tendency is to select an arbitrary subset of conditions that could lead to an inferior test set.)

Cause-Effect Analysis: Discussion Questions & Exercises• Are there any other obvious benefits of Cause-

Effect Analysis?

Cause-Effect Analysis: Discussion Questions & Exercises• Are there any other obvious benefits of Cause-

Effect Analysis?A beneficial side effect is that it facilitates the discovery of specification incompleteness and ambiguity.

Cause-Effect Analysis: Discussion Questions & Exercises• What are the pros and cons of having a set of

mutually exclusive Effects?

Cause-Effect Analysis: Discussion Questions & Exercises• What are the pros and cons of having a set of

mutually exclusive Effects?Pros: Working with the model to identify test case templates is simplified since the issue of determining the truth value of "other" Effects goes away. The model is more complete in the sense that all combinations of individual output conditions/behavior have been explicitly represented in the model. This simplifies the task of using the model to ensure coverage of all such combinations.

Cause-Effect Analysis: Discussion Questions & Exercises• What are the pros and cons of having a set of

mutually exclusive Effects?

Cons: The number of such combinations may be very large. (E.g., consider multimedia applications that are rich in sensory stimuli: icons, text fields, windows, animations, colors, sounds, tactile feedback, etc.) This can result in a model that is unwieldy. Also, if the output effects are inherently independent (do not interact), they probably do not need to be tested in combination with one another.

Cause-Effect Analysis: Discussion Questions & Exercises (cont’d)• Suppose that some program Effect is

associated with integer input X being either 30 or even. What are the pros and cons of defining { X | X 30 V EVEN(X) } to be a Cause?

• Devise a scenario that illustrates some creative ideas for how a well-engineered CASE tool could effectively support Cause-Effect Analysis.

Cause-Effect Analysis: Discussion Questions & Exercises (cont’d)• Suppose that some program Effect is

associated with integer input X being either 30 or even. What are the pros and cons of defining { X | X 30 V EVEN(X) } to be a Cause?

• Devise a scenario that illustrates some creative ideas for how a well-engineered CASE tool could effectively support Cause-Effect Analysis.

C-E Analysis Process Steps

1. Identify Causes and Effects2. Deduce Logical Relationships and

Constraints3. Identify an appropriate Test Case

Selection Strategy4. Construct a Test Case Coverage Matrix

Cause-Effect Analysis: Discussion Questions & Exercises (cont’d)• Cause-Effect Analysis seems well suited

for “single-state-transition” program models in which Causes are mapped to Effects in one conceptual step. How could you apply the strategy to multi-state-transition program models?

Black-Box Test Case Design Techniques Considered

• Partition testing• Combinatorial Approaches• Boundary Value Analysis• Intuition & Experience

Boundary Value Analysis

• A technique based on identifying, and generating test cases to explore boundary conditions.

• Boundary conditions are an extremely rich source of errors.

• Natural language based specifications of boundaries are often ambiguous, as in “for input values of X between 0 and 40,...”

Boundary Value Analysis

• A technique based on identifying, and generating test cases to explore boundary conditions.

• Boundary conditions are an extremely rich source of errors.

• Natural language based specifications of boundaries are often ambiguous, as in “for input values of X between 0 and 40,...”

Boundary Value Analysis

• A technique based on identifying, and generating test cases to explore boundary conditions.

• Boundary conditions are an extremely rich source of errors.

• Natural language based specifications of boundaries are often ambiguous, as in “for input values of X between 0 and 40,...”

Boundary Value Analysis (cont’d)• May be applied to both input and output

conditions.• Also applicable to white box testing (as

will be illustrated later).

Boundary Value Analysis (cont’d)• May be applied to both input and output

conditions.• Also applicable to white box testing (as

will be illustrated later).

Guidelines for Identifying Boundary Values• “Range” guideline:

K will range in value from 0.0 to 4.0.– Identify values at the endpoints of the

range and just beyond.– Boundary values: 0.0- (I) 0.0 (V) 4.0 (V) 4.0+ (I)

Guidelines for Identifying Boundary Values• “Range” guideline:

K will range in value from 0.0 to 4.0.– Identify values at the endpoints of the

range and just beyond.– Boundary values: 0.0- (I) 0.0 (V) 4.0 (V) 4.0+ (I)

Guidelines for Identifying Boundary Values• “Range” guideline:

K will range in value from 0.0 to 4.0.– Identify values at the endpoints of the

range and just beyond.– Boundary values: 0.0- (I) 0.0 (V) 4.0 (V) 4.0+ (I)

Guidelines for Identifying Boundary Values• “Range” guideline:

K will range in value from 0.0 to 4.0.– Identify values at the endpoints of the

range and just beyond.– Boundary values: 0.0- (I) 0.0 (V) 4.0 (V) 4.0+ (I)

Guidelines for Identifying Boundary Values (cont’d)• “Number of values” guideline:

The file will contain 1-25 records.– Identify the minimum, the maximum,

and values just below the minimum and above the maximum.

– Boundary values: empty file (I), file with 1 (V), 25 (V), and 26 (I) records

Guidelines for Identifying Boundary Values (cont’d)• “Number of values” guideline:

The file will contain 1-25 records.– Identify the minimum, the maximum,

and values just below the minimum and above the maximum.

– Boundary values: empty file (I), file with 1 (V), 25 (V), and 26 (I) records

Guidelines for Identifying Boundary Values (cont’d)• “Number of values” guideline:

The file will contain 1-25 records.– Identify the minimum, the maximum,

and values just below the minimum and above the maximum.

– Boundary values: empty file (I), file with 1 (V), 25 (V), and 26 (I) records

Guidelines for Identifying Boundary Values (cont’d)• “Number of values” guideline:

The file will contain 1-25 records.– Identify the minimum, the maximum,

and values just below the minimum and above the maximum.

– Boundary values: empty file (I), file with 1 (V), 25 (V), and 26 (I) records

Boundary Value Analysis Exercise

Identify appropriate boundary values for the following program specification fragment.

City Tax Specification 1:

The first input is a yes/no response to the question “Do you reside within the city?” The second input is gross pay for the year in question.A non-resident will pay 1% of the gross pay in city tax.Residents pay on the following scale:- If gross pay is no more than $30,000, the tax is 1%.- If gross pay is more than $30,000, but no more than $50,000, the tax is 5%.- If gross pay is more than $50,000, the tax is 15%.

Black-Box Test Case Design Techniques Considered

• Partition testing• Combinatorial Approaches• Boundary Value Analysis• Intuition & Experience

Test Case Design Based on Intuition and Experience• Also known as Error Guessing, Ad Hoc

Testing, Artistic Testing, etc.• Testers utilize intuition and experience

to identify potential errors and design test cases to reveal them.

• Guidelines:– Design tests for reasonable but

incorrect assumptions that may have been made by developers.

(cont’d)

Test Case Design Based on Intuition and Experience• Also known as Error Guessing, Ad Hoc

Testing, Artistic Testing, etc.• Testers utilize intuition and experience

to identify potential errors and design test cases to reveal them.

• Guidelines:– Design tests for reasonable but

incorrect assumptions that may have been made by developers.

(cont’d)

Test Case Design Based on Intuition and Experience• Also known as Error Guessing, Ad Hoc

Testing, Artistic Testing, etc.• Testers utilize intuition and experience

to identify potential errors and design test cases to reveal them.

• Guidelines:– Design tests for reasonable but

incorrect assumptions that may have been made by developers.

(cont’d)

Test Case Design Based on Intuition and Experience• Also known as Error Guessing, Ad Hoc

Testing, Artistic Testing, etc.• Testers utilize intuition and experience

to identify potential errors and design test cases to reveal them.

• Guidelines:– Design tests for reasonable but

incorrect assumptions that may have been made by developers.

(cont’d)

Intuition and Experience (cont’d)• Guidelines: (cont’d)

– Design tests to detect errors in handling special situations or cases.

– Design tests to explore unexpected or unusual program use or environmental scenarios.

Intuition and Experience (cont’d)• Guidelines: (cont’d)

– Design tests to detect errors in handling special situations or cases.

– Design tests to explore unexpected or unusual program use or environmental scenarios.

Intuition and Experience (cont’d)• Examples of data conditions to explore:

(1)(2) Repeated instances or occurrences(3) Repeated instances or occurrences(4) Bl anks or null char acters in strings (eT c.)(-5) Negative numbers(#) Non-numeric values in numeric fields (or

vic3 versa)(6789) Inputs that are too long or too short

Intuition and Experience (cont’d)• Examples of data conditions to explore:

(1)(2) Repeated instances or occurrences(3) Repeated instances or occurrences(4) Bl anks or null char acters in strings (eT c.)(-5) Negative numbers(#) Non-numeric values in numeric fields (or

vic3 versa)(6789) Inputs that are too long or too short

Intuition and Experience (cont’d)• Examples of data conditions to explore:

(1) (incomplete or missing input)(2) Repeated instances or occurrences(3) Repeated instances or occurrences(4) Bl anks or null char acters in strings (eT c.)(-5) Negative numbers(#) Non-numeric values in numeric fields (or

vic3 versa)(6789) Inputs that are too long or too short

Intuition and Experience (cont’d)• Examples of data conditions to explore:

(1) (incomplete or missing input)(2) Repeated instances or occurrences(3) Repeated instances or occurrences(4) Bl anks or null char acters in strings (eT c.)(-5) Negative numbers(#) Non-numeric values in numeric fields (or

vic3 versa)(6789) Inputs that are too long or too short

Intuition and Experience (cont’d)• Examples of data conditions to explore:

(1) (incomplete or missing input)(2) Repeated instances or occurrences(3) Repeated instances or occurrences(4) Bl anks or null char acters in strings (eT c.)(-5) Negative numbers(#) Non-numeric values in numeric fields (or

vic3 versa)(6789) Inputs that are too long or too short

Intuition and Experience (cont’d)• Examples of data conditions to explore:

(1) (incomplete or missing input)(2) Repeated instances or occurrences(3) Repeated instances or occurrences(4) Bl anks or null char acters in strings (eT c.)(-5) Negative numbers(#) Non-numeric values in numeric fields (or

vic3 versa)(6789) Inputs that are too long or too short

Intuition and Experience (cont’d)• Examples of data conditions to explore:

(1) (incomplete or missing input)(2) Repeated instances or occurrences(3) Repeated instances or occurrences(4) Bl anks or null char acters in strings (eT c.)(-5) Negative numbers(#) Non-numeric values in numeric fields (or

vic3 versa)(6789) Inputs that are too long or too short

Intuition and Experience (cont’d)• Examples of data conditions to explore:

(1) (incomplete or missing input)(2) Repeated instances or occurrences(3) Repeated instances or occurrences(4) Bl anks or null char acters in strings (eT c.)(-5) Negative numbers(#) Non-numeric values in numeric fields (or

vic3 versa)(6789) Inputs that are too long or too short

Intuition and Experience (cont’d)• Testing based on intuition and

experience can be extremely effective.• Test plans should reflect the explicit

allocation of resources for this activity.• Consider the “Try to break our system –

lunch is on us” example…

Intuition and Experience (cont’d)• Testing based on intuition and

experience can be extremely effective.• Test plans should reflect the explicit

allocation of resources for this activity.• Consider the “Try to break our system –

lunch is on us” example…

Intuition and Experience (cont’d)• Testing based on intuition and

experience can be extremely effective.• Test plans should reflect the explicit

allocation of resources for this activity.• Consider the “Try to break our system –

lunch is on us” example…

Intuition and Experience Exercise

Using intuition and experience, identify tests you would want to design for a subroutine that is to input and sort a list of strings based on a user-specified field.

Black-Box Testing Techniques III

Prepared by

Stephen M. Thebaut, Ph.D.University of Florida

Software Testing and Verification

Lecture 6