Download - Conformance Test Suites, Extensionally

Conformance Test Suites, Extensionally

Arend RensinkUniversity of Twente

Dutch Workshop on Formal Testing Techniques

University of Twente13 September 2007

DWFTT, September 2007Conformance Test Suites,

Extensionally 2

Automaton construction

Approach

Specification

SuspensionAutomaton

Test Suite

Test Automaton

ImplementationUnder Test

Formal

Physical

Non-deterministicalgorithm

Closer to actual implementation

• Finite if spec is regular

• On-the-fly construction

• Completeness criteria

• Direct reasoning & manipulation

Current

New


Extensionally 3

Background

• Specification s given by LTS– Distinguished inputs and outputs– Weakly convergent

• Stable, output-free states: quiescent– Made explicit by -labelled self-loop– Suspension language Susp(s): traces with

’s• Implementation i modelled by IOTS– Input enabled

• Conformance (ioco)i ioco s , 8 w 2 Susp(s):

Out(i after w) µ Out(s after w)


Extensionally 4

Example

• Self-closing door

• Wrong implementation:

press

openpress press

close

input: pressoutputs: open close

press

openpress

press

closepress

Error: 2 Out(i after press¢open)

explicit quiescence

input enabled


Extensionally 5

Example (continued)

• Alternative specs:

press

openpress press

close

press non-determinism

under-specification

for press

press

open

close

press


Extensionally 6

IOCO analyzed

•The following are equivalent:– 8 w2Susp(s):

Out(i after w) µ Out(s after w)– 8 w2Susp(s), x2U:

w¢x 2 Susp(i) ) w¢x 2 Susp(s)– Susp(i) Å (Susp(s)¢U) µ Susp(s)

– Susp(i) Å (Susp(s)¢U n Susp(s)) = ;

•So the following are equivalent: – i conforms to s– i has no suspension trace accepted by

Susp(s)¢Un Susp(s)


Extensionally 7

Intuition

• Susp(s): correct behaviour

• Susp(s)¢U: Just one more response…

• Susp(s)¢U n Susp(s): Forbidden paths


Extensionally 8

Example

press

openpress

close

press

press

openpress

close

openclose

close

open

press

Spec s Construct: Susp(s)¢U n Susp(s)

press

open

press

close

openpress

close

openclose

close

open press

Example error trace: press¢open¢

final state


Extensionally 9

reduction & -saturation

• Given a set of traces W µ L*

– W- removes all traces from W with multiple successive ’s (of the form v¢¢

¢w)– W+ adds v¢+¢w for all v¢¢w 2 W

• Facts: If W is a suspension language– W+ = W– (W-)+ = W

• Improvement on previous result:– i conforms to s iff – i has no suspension trace accepted by

Susp(s)-¢Un Susp(s)

remove -cycles


Extensionally 10

Test automata

• Let T be an arbitrary automaton (with ’s)T accepts i , Susp(i) Å Lang(T) = ;

• Properties defined for arbitrary T– soundness: i ioco s implies T accepts i– exhaustiveness: T accepts i implies i ioco s

• Seen previously:– Susp(s)-¢U n Susp(s) sound & exhaustive for s

• General criteria (main result, part 1):– T is sound for s if and only if

Lang(T) µ (Susp(s)¢U n Susp(s)) ¢ L*

– T is exhaustive for s if Lang(T) ¶ (Susp(s)-¯ U n Susp(s))

No output after


Extensionally 11

Main result (part 2)

• Define: T is full if– v · w 2 Lang(T) implies v 2 Lang(T)– v¢a¢w 2 Lang(T) implies v¢¢a¢w 2 Lang(T)– v¢a 2 Lang(T) implies v 2 Lang(T)– v¢U µ Lang(T) implies v 2 Lang(T)

• For all i: fill(T) accepts i iff T accepts i• Necessary criterion for exhaustiveness:– If T is sound for s, then

T is exhaustive for s only if Lang(fill(T)) ¶ (Susp(s)-¯ U n Susp(s))

fill(T) can be constructed

from T


Extensionally 12

Example

press

openpress

close

press

Spec s

Susp(s)-¯U n Susp(s)

press

openpress

close

openclose

close

open

press

press

openpress

close

openclose

close

open

press

Susp(s)¢U n Susp(s)

Straightforward automata

constructions only


Extensionally 13

Given:

Possible with extensional characterisation

Compositionality

Test Tsound/complete for

Spec s

Spec s’

Composition(Parallel composition,action refinement, …)

Test T’

CompositionDesired:


Extensionally 14


From test automata to test suites

Specification

SuspensionAutomaton

Test Suite

Test Automaton


Formal

Physical



Extensionally 15

From test automata to test suites• Equivalent testing power:– Test automaton T– Set {Ti}i with i Lang(Ti) = Lang(T)

• Tretmans test cases t as test automata:– Cycle-free– One final accepting state (= fail) – One final non-accepting state (= pass) – Non-final states either stimulus or response

• Tretmans’ algorithm generates {ti}i s.t.:– Lang(ti) µ Susp(s)¢Un Susp(s) for all ti

– 9i: w2 Lang(ti) for all w2 Susp(s)¢Un Susp(s)


Extensionally 16


Direct testing with automata

Specification

SuspensionAutomaton

Test Suite

Test Automaton


Formal

Physical



Extensionally 17

Direct testing with automata

• Given a test automaton T– Deterministic, with explicit quiescence– Corresponds to entire test suite

• Synchronisation (product) of IuT and T:– Stimulus specified by T sent to IuT– Response of IuT mimicked in T

• Test verdict– Final state of T reached: fail– Response of IuT absent in T: pass– No reachable final state in T: pass– Done testing: pass


Extensionally 18

Conclusion

• Test automata fill gap in existing theory– Yield precise conditions for completeness– Enable constructions on test suite– Work modulo Tretmans’ “test criteria”

(not covered in this presentation)

• Re-opens discussion on– Interaction between test and IuT– Input enabledness of test cases