Doron Peled,University of Warwick

Why testing?Reduce design/programming errors.Can be done during development, before

production/marketing.Practical, simple to do.Check the real thing, not a model.Scales up reasonably.Being state of the practice for decades.

Part 1: Testing of black box finite state machine

Know:Transition relationSize or bound on size

Wants to know:In what state we started?In what state we are?Transition relationConformanceSatisfaction of a temporal property

Finite automata (Mealy machines)

S - finite set of states. (size n)– set of inputs. (size d)O – set of outputs, for each transition.(s0 S - initial state). S S - transition relation. S O – output on edge.

Why deterministic machines? Otherwise no amount of experiments would

guarantee anything. If dependent on some parameter (e.g.,

temperature), we can determinize, by taking parameter as additional input.

We still can model concurrent system. It means just that the transitions are deterministic.

All kinds of equivalences are unified into language equivalence.

Also: connected machine (otherwise we may never get to the completely separate parts).

Determinism

When the black box is nondeterministic, we might never test some choices.

b/1a/1

a/1

Preliminaries: separating sequences

s1

s3

s2

a/0b/1 b/0

b/1

a/0

a/0

Start with one block containing all states {s1, s2, s3}.

A: separate to blocks of states with different output.

s1

s3

s2

a/0b/1 b/0

b/1

a/0

a/0

Two sets, separated using the string b {s1, s3}, {s2}.

Repeat B: Separate blocks based on moving to different blocks.

s1

s3

s2

a/0b/1 b/0

b/1

a/0

a/0

Separate first block using b to three singleton blocks.Separating sequences: b, bb.Max rounds: n-1, sequences: n-1, length: n-1.For each pair of states there is a separating sequence.

Want to know the state of the machine (at end). Homing sequence.Depending on output, would know in what

state we are. Algorithm: Put all the states in one block

(initially we do not know what is the state).Then repeatedly partitions blocks of states, as

long as they are not singletons, as follows: Take a non singleton block, append a

distinguishing sequence that separates at least two states.

Update all blocks to the states after executing .

Max length: (n-1)2 (Lower bound: n(n-1)/2.)

Example (homing sequence)

s1

s3

s2

a/0b/1 b/0

b/1

a/0

a/0{s1, s2, s3}

{s1, s2} {s3}{s1} {s2} {s3}

b

b1 0

011

1

On input b and output 1, still don’t know if was in s1 or s3, i.e., if currently in s2 or s1.So separate these cases with another b.

Synchronizing sequenceOne sequence takes the machine to

the same final state, regardless of the initial state or the outputs.

Not every machine has a synchronizing sequence.

Can be checked whether exists and can be found in polynomial time.

State identification: Want to know in which state the

system has started (was reset). Can be a preset distinguishing

sequence (fixed), or a tree (adaptive).

May not exist (PSPACE complete to check if preset exists, polynomial for adaptive).

Best known algorithm: exponential length for preset,polynomial for adaptive [LY].

Sometimes cannot identify initial state

b/1a/1 s1

s3

s2

a/1

b/0

b/1

a/1

Start with a:in case of being in s1 or s3 we’ll move to s1 and cannot distinguish.Start with b:In case of being in s1 or s2 we’ll move to s2 and cannot distinguish.

The kind of experiment we do affects what we can distinguish. Much like the Heisenberg principle in Physics.

Conformance testing Unknown deterministic finite state system B. Known: n states and alphabet . An abstract model C of B. C satisfies all the

properties we want from B. C has m states. Check conformance of B and C. Another version: only a bound n on the number

of states l is known.

Check conformance with a given state machine

Black box machine has no more states than specification machine (errors are mistakes in outputs, mistargeted edges).

Specification machine is reduced, connected, deterministic. Machine resets reliably to a single initial state (or use homing

sequence).

s1

s3

s2

a/1

b/0

b/1

a/1

?=

a/1

b/1

Conformance testing [Ch,V]

a/1

b/1

Cannot distinguish if reduced or not.

a/1

b/1

a/1

b/1

a/1

b/1a/1

b/1

Conformance testing (cont.)

ab b

aa

a

a b

b

b

a

Need: bound on number of states of B.

a

Preparation:Construct a spanning tree

b/1a/1 s1

s3

s2

a/1

b/0

b/1

a/1

s1

s2s3

b/1a/1

How the algorithm works?According to the spanning

tree, force a sequence of inputs to go to each state.

1. From each state, perform the distinguishing sequences.

2. From each state, make a single transition, check output, and use distinguishing sequences to check that in correct target state.

s1

s2s3

b/1a/1

Reset or hom

ing

Reset or hom

ing

Distinguishing sequences

Comments1. Checking the different distinguishing

sequences (m-1 of them) means each time resetting and returning to the state under experiment.

2. A reset can be performed to a distinguished state through a homing sequence. Then we can perform a sequence that brings us to the distinguished initial state.

3. Since there are no more than m states, and according to the experiment, no less than m states, there are m states exactly.

4. Isomorphism between the transition relation is found, hence from minimality the two automata recognize the same languages.

Combination lock automaton

Assume accepting states.Accepts only words with a specific suffix

(cdab in the example).

s1 s2 s3 s4 s5bdc a

Any other input

When only a bound on size of black box is known…Black box can “pretend” to behave

as a specification automaton for a long time, then upon using the right combination, make a mistake.

b/1a/1s1

s3

s2

a/1

b/0

b/1

a/1

b/1

Pretends to be S3

Pretends to be S1

a/1

Conformance testing algorithm [VC] The worst that can happen is a

combination lock automaton that behaves differently only in the last state. The length of it is the difference between the size n of the black box and the specification m.

Reach every state on the spanning tree and check every word of length n-m+1 or less. Check that after the combination we are at the state we are supposed to be, using the distinguishing sequences.

No need to check transitions: already included in above check.

Complexity: m2 n dn-m+1

Probabilistic complexity: Polynomial. Distinguishing sequences

s1

s2s3

b/1a/1

Words of length n-m+1

Reset or hom

ing

Reset or hom

ing

Model Checking Finite state description of a system B. LTL formula . Translate into an automaton P. Check whether L(B) L(P)=. If so, S satisfies . Otherwise, the intersection

includes a counterexample. Repeat for different properties.

Buchi automata (-automata) S - finite set of states. (B has l n states) S0 S - initial states. (P has m states) - finite alphabet. (contains p letters) S S - transition relation. F S - accepting states.Accepting run: passes a state in F infinitely

often.System automata: F=S, deterministic, one initial state.

Property automaton: not necessarily deterministic.

Example: check a

a, aa

a <>a

Example: check <>a

aa

a

a<>a

Example: check <>a

Use automatic translation algorithms, e.g., [Gerth,Peled,Vardi,Wolper 95]

aa

a, a<>a

System

c b

a

Every element in the product is a counter example for the checked property.

c b

a

aa

a

a

s1 s2

s3 q2

q1

s1,q1

s1,q2 s3,q2

s2,q1a

b

ca

Acceptance isdetermined byautomaton P.

<>a

Model Checking / Testing Given Finite state

system B. Transition relation of B

known. Property represent by

automaton P. Check if L(B) L(P)=. Graph theory or BDD

techniques. Complexity: polynomial.

Unknown Finite state system B.

Alphabet and number of states of B or upper bound known.

Specification given as an abstract system C.

Check if B C. Complexity: polynomial

if number states known. Exponential otherwise.

Black box checking [PVY]

Property represent by automaton P.

Check if L(B) L(P)=.

Graph theory techniques.

Unknown Finite state system B.

Alphabet and Upper bound on Number of states of B known.

Complexity: exponential.

Experimentsaa

bb cc

reset

a

ab

b

c

c

try b a

ab

b

c

c

try c

fail

Simpler problem: deadlock?

Nondeterministic algorithm:guess a path of length n from the initial state to a deadlock state.Linear time, logarithmic space.

Deterministic algorithm:systematically try paths of length n, one after the other (and use reset), until deadlock is reached.Exponential time, linear space.

Deadlock complexityNondeterministic algorithm:

Linear time, logarithmic space.Deterministic algorithm:

Exponential (p n-1) time, linear space.Lower bound: Exponential time (use

combination lock automata).How does this conform with what we

know about complexity theory?

Modeling black box checking

Cannot model using Turing machines: not all the information about B is given. Only certain experiments are allowed.

We learn the model as we make the experiments.

Can use the model of games of incomplete information.

Games of incomplete information Two players: player, player (here, deterministic). Finitely many configurations C. Including:

Initial Ci , Winning : W+ and W- . An equivalence relation on C (the player cannot

distinguish between equivalent states). Labels L on moves (try a, reset, success, fail). The player has the moves labeled the same from

configurations that are equivalent. Deterministic strategy for the player: will lead to a

configuration in W+ W-. Cannot distinguish between equivalent configurations.

Nondeterministic strategy: Can distinguish between equivalent configurations..

Modeling BBC as gamesEach configuration contains an automaton

and its current state (and more).Moves of the player are labeled with

try a, reset... Moves of the -player withsuccess, fail.

c1 c2 when the automata in c1 and c2 would respond in the same way to the experiments so far.

A naive strategy for BBC Learn first the structure of the black box. Then apply the intersection. Enumerate automata with n states

(without repeating isomorphic automata). For a current automata and new

automata, construct a distinguishing sequence. Only one of them survives.

Complexity: O((n+1)p (n+1)/n!)

On-the-fly strategy Systematically (as in the deadlock

case), find two sequences v1 and v2 of length <=m n.

Applying v1 to P brings us to a state t that is accepting.

Applying v2 to P brings us back to t.

Apply v1 v2 n to B. If this succeeds,

there is a cycle in the intersection labeled with v2, with t as the P (accepting) component.

Complexity: O(n2p2mnm).

v1

v2

Learning an automaton

Use Angluin’s algorithm for learning an automaton.

The learning algorithm queries whether some strings are in the automaton B.

It can also conjecture an automaton Mi and asks for a counterexample.

It then generates an automaton with more states Mi+1 and so forth.

A strategy based on learning

Start the learning algorithm.Queries are just experiments to B.For a conjectured automaton Mi ,

check if Mi P = If so, we check conformance of Mi with

B ([VC] algorithm). If nonempty, it contains some v1 v2

. We test B with v1 v2

n. If this succeeds: error, otherwise, this is a counterexample for Mi .

Complexity l - actual size of B. n - an upper bound of size of B. d - size of alphabet. Lower bound: reachability is similar to

deadlock. O(l 3 d l + l 2mn) if there is an error. O(l 3 d l + l 2 n dn-l+1+ l 2mn) if there is no error.If n is not known, check while time allows. Probabilistic complexity: polynomial.

Some experimentsBasic system written in SML (by Alex

Groce, CMU).Experiment with black box using Unix

I/O.Allows model-free model checking of C

code with inter-process communication.Compiling tested code in SML with BBC

program as one process.

Part 2: Software testing

Testing is not about showing that there are no errors in the program.

Testing cannot show that the program performs its intended goal correctly.

So, what is software testing?Testing is the process of executing the

program in order to find errors.A successful test is one that finds an error.

Some software testing stages Unit testing – the lowest level, testing

some procedures. Integration testing – different pieces of code. System testing – testing a system as a whole. Acceptance testing – performed by the

customer. Regression testing – performed after updates. Stress testing – checking the code under

extreme conditions. Mutation testing – testing the quality of the

test suite.

Some drawbacks of testing

There are never sufficiently many test cases.

Testing does not find all the errors. Testing is not trivial and requires

considerable time and effort. Testing is still a largely informal task.

Black-Box (data-driven, input-output) testing

The testing is not based on the structure of the program (which is unknown).

In order to ensure correctness, every possible input needs to be tested - this is impossible!

The goal: to maximize the number of errors found.

testingIs based on the internal structure of the

program.There are several alternative criterions

for checking “enough” paths in the program.

Even checking all paths (highly impractical) does not guarantee finding all errors (e.g., missing paths!)

Some testing principles

A programmer should not test his/her own program. One should test not only that the program does what

it is supposed to do, but that it does not do what it is not supposed to.

The goal of testing is to find errors, not to show that the program is errorless.

No amount of testing can guarantee error-free program.

Parts of programs where a lot of errors have already been found are a good place to look for more errors.

The goal is not to humiliate the programmer!

Inspections and Walkthroughs

Manual testing methods.Done by a team of people.Performed at a meeting

(brainstorming).Takes 90-120 minutes.Can find 30%-70% of errors.

Code Inspection

Team of 3-5 people. One is the moderator.

He distributes materials and records the errors.

The programmer explains the program line by line.

Questions are raised. The program is

analyzed w.r.t. a checklist of errors.

Checklist for inspectionsData declarationAll variables

declared?Default values

understood?Arrays and strings

initialized?Variables with similar

names?Correct initialization?

Control flowEach loop terminates?DO/END statements

match?

Input/outputOPEN statements

correct?Format specification

correct?End-of-file case handled?

Walkthrough

Team of 3-5 people. Moderator, as before. Secretary, records

errors. Tester, play the role

of a computer on some test suits on paper and board.

Selection of test cases (for white-box testing)

The main problem is to select a good coveragecriterion. Some options are:

Cover all paths of the program. Execute every statement at least once. Each decision has a true or false value at least

once. Each condition is taking each truth value at least

once. Check all possible combinations of conditions in

each decision.

Cover all the paths of the program

Infeasible.Consider the flow diagram

on the left.It corresponds to a loop.The loop body has 5 paths.If the loops executes 20times there are 5^20

different paths!May also be unbounded!

How to cover the executions?IF (A>1)&(B=0) THEN X=X/A;

END;IF (A=2)|(X>1) THEN X=X+1;

END;

Choose values for A,B,X. Value of X may change, depending on A,B. What do we want to cover? Paths? Statements?

Conditions?

Statement coverageExecute every statement at least onceBy choosingA=2,B=0,X=3each statement will

be chosen.The case where the

tests fail is not checked!

IF (A>1)&(B=0) THEN X=X/A; END;

IF (A=2)|(X>1) THEN X=X+1; END;

Now x=1.5

Decision coverageEach decision has a true and false outcome at least once.

Can be achieved using A=3,B=0,X=3 A=2,B=1,X=1

Problem: Does not test individual conditions. E.g., when X>1 is erroneous in second decision.

IF (A>1)&(B=0) THEN X=X/A; END;

IF (A=2)|(X>1) THEN X=X+1; END;

Decision coverage

A=3,B=0,X=3 IF (A>1)&(B=0) THEN X=X/A; END;

IF (A=2)|(X>1) THEN X=X+1; END;

Now x=1

Decision coverage

A=2,B=1,X=1

The case where A1 and the case where x>1 where not checked!

IF (A>1)&(B=0) THEN X=X/A; END;

IF (A=2)|(X>1) THEN X=X+1; END;

Condition coverageEach condition has a true and false value at least once.

For example: A=1,B=0,X=3 A=2,B=1,X=0

lets each condition be true and false once.

Problem:covers only the path where the first test fails and the second succeeds.

IF (A>1)(A>1)&(B=0) THEN X=X/A; END;

IF (A=2)|(X>1) THEN X=X+1; END;

Condition coverage

A=1,B=0,X=3 IF (A>1) (A>1) & (B=0) THEN X=X/A; END;

IF (A=2) | (X>1) THEN X=X+1; END;

Condition coverage

A=2,B=1,X=0

Did not check the first THEN part at all!!!

Can use condition+decision coverage.

IF (A>1)(A>1)&(B=0) THEN X=X/A; END;

IF (A=2)|(X>1) THEN X=X+1; END;

Multiple Condition CoverageTest all combinations of all conditions in each test.

A>1,B=0 A>1,B≠0 A1,B=0 A1,B≠0 A=2,X>1 A=2,X1 A≠2,X>1 A≠2,X1

IF (A>1)&(B=0) THEN X=X/A; END;

IF (A=2)|(X>1) THEN X=X+1; END;

A smaller number of cases: A=2,B=0,X=4 A=2,B=1,X=1 A=1,B=0,X=2 A=1,B=1,X=1Note the X=4 in the firstcase: it is due to the factthat X changes beforebeing used!

IF (A>1)&(B=0) THEN X=X/A; END;

IF (A=2)|(X>1) THEN X=X+1; END;

Further optimization: not all combinations.For C /\ D, check (C, D), (C, D), (C, D).For C \/ D, check (C, D), (C, D), (C, D).

Preliminary:Relativizing assertions

(B) : x1= y1 * x2 + y2 /\ y2 >= 0Relativize B) w.r.t. the assignment

becomes B) [Y\g(X,Y)]e(B) expressed w.r.t. variables at

A.) (B)A =x1=0 * x2 + x1 /\ x1>=0

Think about two sets of variables,before={x, y, z, …} after={x’,y’,z’…}.

Rewrite (B) using after, and the assignment as a relation between the set of variables. Then eliminate after.

Here: x1’=y1’ * x2’ + y2’ /\ y2’>=0 /\x1=x1’ /\ x2=x2’ /\ y1’=0 /\ y2’=x1now eliminate x1’, x2’, y1’, y2’.

(y1,y2)=(0,x1)

A

B

A

B

(y1,y2)=(0,x1)

Y=g(X,Y)

Verification conditions: tests

C) B)= t(X,Y) /\ C)

D) B)=t(X,Y) /\ D)

B)= D) /\ y2x2y2>=x2

B

C

D

B

C

Dt(X,Y)

FT

FT

How to find values for coverage?

•Put true at end of path.•Propagate path backwards.•On assignment, relativize expression.•On “yes” edge of decision, add decision as conjunction.•On “no” edge, add negation of decision as conjunction.•Can be more specific when calculating condition with multiple condition coverage.

A>1 & B=0

A=2 | X>1

X=X+1

X=X/Ano

no

yes

yes

true

true

How to find values for coverage?

A>1 & B=0

A=2 | X>1

X=X+1

X=X/Ano

no

yes

yes

true

true

A≠2 /\ X>1

(A≠2 /\ X/A>1) /\ (A>1 & B=0)

A≠2 /\ X/A>1Need to find a

satisfying assignment:A=3, X=6, B=0Can also calculate path condition forwards.

How to cover a flow chart? Cover all nodes, e.g., using search strategies:

DFS, BFS. Cover all paths (usually impractical). Cover each adjacent sequence of N nodes. Probabilistic testing. Using random number

generator simulation. Based on typical use. Chinese Postman: minimize edge traversal

Find minimal number of times time to travel each edge using linear programming or dataflow algorithms.Duplicate edges and find an Euler path.

Test cases based on data-flow analysis

Partition the program into pieces of code with a single entry/exit point.

For each piece find which variables are set/used/tested.

Various covering criteria: from each set to each

use/test From each set to

some use/test.

X:=3

z:=z+x

x>y

t>y

Test case design for black box testing

Equivalence partitionBoundary value analysisCause-effect graphs

Equivalence partition

Goals: Find a small number of test cases. Cover as much possibilities as you can.

Try to group together inputs for which the program is likely to behave the same.

Specificationcondition

Valid equivalenceclass

Invalid equivalenceclass

Example: A legal variable

Begins with A-Z Contains [A-Z0-9] Has 1-6 characters.

Specificationcondition

Valid equivalenceclass

Invalid equivalenceclass

Starting char

Chars

Length

Starts A-Z Starts other

[A-Z0-9] Has others

1-6 chars 0 chars, >6 chars

1 2

3 4

56 7

Equivalence partition (cont.)

Add a new test case until all valid equivalence classes have been covered. A test case can cover multiple such classes.

Add a new test case until all invalid equivalence class have been covered. Each test case can cover only one such class.

Specificationcondition

Valid equivalenceclass

Invalid equivalenceclass

Example

AB36P (1,3,5) 1XY12 (2) A17#%X (4)

Specificationcondition

Valid equivalenceclass

Invalid equivalenceclass

Starting char

Chars

Length

Starts A-Z Starts other

[A-Z0-9] Has others

1-6 chars 0 chars, >6 chars

1 2

3 4

56 7

(6) VERYLONG (7)

Boundary value analysis

In every element class, select values that are closed to the boundary. If input is within range -1.0 to +1.0,

select values -1.001, -1.0, -0.999, 0.999, 1.0, 1.001.

If needs to read N data elements, check with N-1, N, N+1. Also, check with N=0.

Test case generation based on LTL specification

Compiler ModelChecker

Path conditioncalculation

First orderinstantiator

Testmonitoring

Transitions

Path Flow

chart

LTLAut

Goals Verification of software. Compositional verification. Only a unit of code. Parametrized verification. Generating test cases.

A path found with some truth assignment satisfying the path condition. In deterministic code, this assignment guarantees to derive the execution of the path.

In nondeterministic code, this is one of the possibilities.Can transform the code to force replying the path.

Divide and Conquer Intersect property automatonproperty automaton with the

flow chartflow chart, regardless of the statements and program variables expressions.

Add assertions from the property automaton to further restrict the path condition.

Calculate path conditions for sequences found in the intersection.

Calculate path conditions on-the-fly. Backtrack when condition is false.Thus, advantage to forward calculation of path conditions (incrementally).

Spec:¬at l2U (at l2/\ ¬at l2/\(¬at l2U at l2))

¬at l2

at l2

¬at l2

at l2

l2:x:=x+z

l3:x<t

l1:…

l2:x:=x+z

l3:x<t

l2:x:=x+z

XX==

Spec: ¬at l2U (at l2/\ xy /\ (¬at l2/\(¬at l2U at l2 /\ x2y )))

¬at l2

at l2/\xy

¬at l2

at l2/\x2y

l2:x:=x+z

l3:x<t

l1:…

l2:x:=x+z

l3:x<t

l2:x:=x+z

XX==

xy

x2y

Example: GCD l1:x:=a

l5:y:=z

l4:x:=y

l3:z:=x rem y

l2:y:=b

l6:z=0? yesno

l0

l7

Example: GCD l1:x:=a

l5:x:=y

l4:y:=z

l3:z:=x rem y

l2:y:=b

l6:z=0? yesno

Oops…with an error (l4 and l5 were switched).

l0

l7

Why use Temporal specification

Temporal specification for sequential software?

Deadlock? Liveness? – No! Captures the tester’s intuitionintuition about the

location of an error:“I think a problem may occur when the program runs through the main while loop twice, then the if condition holds, while t>17.”

Example: GCD l1:x:=a

l5:x:=y

l4:y:=z

l3:z:=x rem y

l2:y:=b

l6:z=0? yesno

l0

l7

a>0/\b>0/\at l0 /\at l7

at l0/\a>0/\b>0

at l7

Example: GCD l1:x:=a

l5:x:=y

l4:y:=z

l3:z:=x rem y

l2:y:=b

l6:z=0? yesno

l0

l7

a>0/\b>0/\at l0/\at l7

Path 1: l0l1l2l3l4l5l6l7a>0/\b>0/\a rem b=0

Path 2: l0l1l2l3l4l5l6l3l4l5l6l7 a>0/\b>0/\a rem b0

Potential explosion

Bad point: potential explosionGood point: may be chopped on-the-fly

Drivers and Stubs Driver: represents the program

or procedure that called our checked unit.

Stub: represents a procedure called by our checked unit.

In our approach: replace both of them with a formula representing the effect the missing code has on the program variables.

Integrate the driver and stub specification into the calculation of the path condition.

l1:x:=a

l5:x:=y

l4:y:=z

l3:z’=x rem y/\x’=x/\y’=x

l2:y:=b

l6:z=0? yesno

l0

l7

Conclusions Black box testing: Know transition relation,

or bound on number of states, want to find initialstate, structure, conformance, temporal property.

Software testing:Unit testing, code inspection, coverage, test case generation.

Model checking and testing have a lot in common:CAV 2004+ISSTA 2004 together, in Boston, MA.