Property-Based Test Generation
Li Tan, Oleg Sokolsky, and Insup Lee
University of Pennsylvania
Temporal Property Translator
Test Harness
LTL formulae1, …, n
Criteria not being
covered
Simulation-based randomized
test generator
Quasi linear (Lasso-shape) proof
structure Test trace generator
Traces {1,…,n}
Traces
Testing result
Interface Definition
Model Checker
Specification Model(CHARON)
InformalSystem Specification
Environ. Modeling
System Modeling
Coverage Criteria
Behavior specification
LTL formulae1, …, n
Hardware Specification/limitation
ImplementationTest result
Feature conflictdetection
The Overview of Our Approach
Goal: Using model-checking technique to make test generation more efficient, flexible, and centered on the system-specific properties (features).
Step I. Preparing specifications Properties (feature specification) as linear temporal logic formula (optional) System specification system as CHARON (for hybrid systems)
and EFSM (for discrete systems)
Step II. Test generation using model checkers. (For hybrid systems) Simulation-based test generation with the
assistance of predicate abstraction reachability analysis.
(For Discrete system) (Option A) Using the proof structures of evidence-ready model
checkers. (Option B) Reducing the test generation for LTL formula to safety check
(For temporal specification only) Functional test. Generating non-trivial test traces for temporal specification (feature
specification) Detecting conflicting in temporal specification.
Step III: Realizing test harness.
Temporal (feature) Spec.+
Model (optional)
Test suite (Finite set of finite traces)
Model-checking basedTest generator
From Property and Model to Test suite: Property-based test generation
I. From infinite length to finite: Synthesizing test suites for 9LTL property
A infinite Lasso-shaped test suite can be checked adequately by finite steps if the implementation is bounded.
Turn=1, :c1, : c2
Turn=1, c1, : c2
Turn=2, c1, : c2
Turn=2, c1, c2
Turn=1, c1, c2
Turn=1, c1, c2
A quasi-linear proof skeleton
+
Estimating the number of relevant implementation states using slicing
A finite test suite
Test Generation using Model Checkers
Option A: Modifying model checkers and retaining proofs.Option B: Using the idea of reducing LTL model checking to reachability
analysis [A. Biere etc], but enhancing the observer to retain proof
SMV model
Linear Temporal Logic Specification
+SMV model
+Extended
Observer ModelRepetition information
Extracted Proof
Generated test trace
II. From infinite numbers of traces to finite: selecting interesting traces
System properties are required to be held on all the paths, we will select only nontrivial paths, whose characteristics are caught by ELTL formula systematically deriving from the properties.
LTL => ELTL formulae a2e()={(’ ! (’))|’ Á }
F= G(req -> F(cancelÇresponse))
F Æ (: G(req ! false))= F Æ F(req)
Test the case that there is request
F Æ (: G(true ! F(cancel Ç response)))
= F Æ FG(: (cancel Ç response))Test the case that no cancel or reponses occurs after time t, (hence should not a
request occur).
F Æ (: G(req ! F(false Ç response)))= F Æ F(req Æ G(: response))
Test the case that no response follows a request
(hence a cancel must be placed)
F Æ (: G(req ! F(cancelÇ false)))= F Æ F(req Æ G(: cancel))
Test the case that no cancel follows a request
(hence a reponse must be placed)
From only Property to Test suite: Functional test generation
So, what if only behavior (feature) specification is available ……
LTL formulae
Nontrivial ELTL formulaeDerived from
=a2e()
0 2 Æ 2 1 2 Æ 2 n 2 Æ 2 …….
Buchi automaton B0 Buchi automaton B1 Buchi automaton Bn
A trace satisfies 0 A trace satisfies 1 A trace satisfies n
Check nonemptinessCheck nonemptiness
SystemModeling
CHARON(Model) Flatten hybrid
model
Implementation
Test Suite
Predicate set
Bad set
Reachability Checker
Yes w/ TraceSimulation/refinement
NO w/ more predicates
YES
No
Testing Hybrid system: simulation-based test generator with predicate-abstract reachability analysis
Concretize
CoverageCriteria
An implementation of simulation-based test generator
a. CHARON simulator with test generator
b. Progress report of test generator
c. Visual display of generated test traces.
Realizing Test Harness
Charon model for CARA
Charon modelFor patient
Closed Charon Model for CARASimulation-based
test generator
Coverage criteriae
Test trace
Variable back_EMFValue Time60.0 0.00170.0 0.002
……
Test harness as I/O Interface
CARA simulator/model-generated code
Standalone executable program
Test Result
Conclusion
1. Applying model-checking technique to traditional domain of test generation is appealing.
1. Test generation is centralized on system-specific properties
2. State-of-art model checkers may be adapted as general purpose test generator (and think properties as programs ).
3. Techniques in model checking may help find interesting test traces and provide new angle to view and think test generation.
2. Property-based test generation requires integrated efforts.
1. Test generation witness generation. 1. Proof is necessary to generate partial test suite and perform
optimization.2. Proof is also needed to extend the notion of “testable” properties.
2. Model-based code generation may help build test harness.
Top Related