Property-Based Test Generation

Li Tan, Oleg Sokolsky, and Insup Lee

University of Pennsylvania

Temporal Property Translator

Test Harness

LTL formulae1, …, n

Criteria not being

covered

Simulation-based randomized

test generator

Quasi linear (Lasso-shape) proof

structure Test trace generator

Traces {1,…,n}

Traces

Testing result

Interface Definition

Model Checker

Specification Model(CHARON)

InformalSystem Specification

Environ. Modeling

System Modeling

Coverage Criteria

Behavior specification

LTL formulae1, …, n

Hardware Specification/limitation

ImplementationTest result

Feature conflictdetection

The Overview of Our Approach

Goal: Using model-checking technique to make test generation more efficient, flexible, and centered on the system-specific properties (features).

Step I. Preparing specifications Properties (feature specification) as linear temporal logic formula (optional) System specification system as CHARON (for hybrid systems)

and EFSM (for discrete systems)

Step II. Test generation using model checkers. (For hybrid systems) Simulation-based test generation with the

assistance of predicate abstraction reachability analysis.

(For Discrete system) (Option A) Using the proof structures of evidence-ready model

checkers. (Option B) Reducing the test generation for LTL formula to safety check

(For temporal specification only) Functional test. Generating non-trivial test traces for temporal specification (feature

specification) Detecting conflicting in temporal specification.

Step III: Realizing test harness.

Temporal (feature) Spec.+

Model (optional)

Test suite (Finite set of finite traces)

Model-checking basedTest generator

From Property and Model to Test suite: Property-based test generation

I. From infinite length to finite: Synthesizing test suites for 9LTL property

A infinite Lasso-shaped test suite can be checked adequately by finite steps if the implementation is bounded.

Turn=1, :c1, : c2

Turn=1, c1, : c2

Turn=2, c1, : c2

Turn=2, c1, c2

Turn=1, c1, c2

Turn=1, c1, c2

A quasi-linear proof skeleton

+

Estimating the number of relevant implementation states using slicing

A finite test suite

Test Generation using Model Checkers

Option A: Modifying model checkers and retaining proofs.Option B: Using the idea of reducing LTL model checking to reachability

analysis [A. Biere etc], but enhancing the observer to retain proof

SMV model

Linear Temporal Logic Specification

+SMV model

+Extended

Observer ModelRepetition information

Extracted Proof

Generated test trace

II. From infinite numbers of traces to finite: selecting interesting traces

System properties are required to be held on all the paths, we will select only nontrivial paths, whose characteristics are caught by ELTL formula systematically deriving from the properties.

LTL => ELTL formulae a2e()={(’ ! (’))|’ Á }

F= G(req -> F(cancelÇresponse))

F Æ (: G(req ! false))= F Æ F(req)

Test the case that there is request

F Æ (: G(true ! F(cancel Ç response)))

= F Æ FG(: (cancel Ç response))Test the case that no cancel or reponses occurs after time t, (hence should not a

request occur).

F Æ (: G(req ! F(false Ç response)))= F Æ F(req Æ G(: response))

Test the case that no response follows a request

(hence a cancel must be placed)

F Æ (: G(req ! F(cancelÇ false)))= F Æ F(req Æ G(: cancel))

Test the case that no cancel follows a request

(hence a reponse must be placed)

From only Property to Test suite: Functional test generation

So, what if only behavior (feature) specification is available ……

LTL formulae

Nontrivial ELTL formulaeDerived from

=a2e()

0 2 Æ 2 1 2 Æ 2 n 2 Æ 2 …….

Buchi automaton B0 Buchi automaton B1 Buchi automaton Bn

A trace satisfies 0 A trace satisfies 1 A trace satisfies n

Check nonemptinessCheck nonemptiness

SystemModeling

CHARON(Model) Flatten hybrid

model

Implementation

Test Suite

Predicate set

Bad set

Reachability Checker

Yes w/ TraceSimulation/refinement

NO w/ more predicates

YES

No

Testing Hybrid system: simulation-based test generator with predicate-abstract reachability analysis

Concretize

CoverageCriteria

An implementation of simulation-based test generator

a. CHARON simulator with test generator

b. Progress report of test generator

c. Visual display of generated test traces.

Realizing Test Harness

Charon model for CARA

Charon modelFor patient

Closed Charon Model for CARASimulation-based

test generator

Coverage criteriae

Test trace

Variable back_EMFValue Time60.0 0.00170.0 0.002

……

Test harness as I/O Interface

CARA simulator/model-generated code

Standalone executable program

Test Result

Conclusion

1. Applying model-checking technique to traditional domain of test generation is appealing.

1. Test generation is centralized on system-specific properties

2. State-of-art model checkers may be adapted as general purpose test generator (and think properties as programs ).

3. Techniques in model checking may help find interesting test traces and provide new angle to view and think test generation.

2. Property-based test generation requires integrated efforts.

1. Test generation witness generation. 1. Proof is necessary to generate partial test suite and perform

optimization.2. Proof is also needed to extend the notion of “testable” properties.

2. Model-based code generation may help build test harness.