Testing with Formal Methods Ed Brinksma course 2004 A Formal Framework.
-
Upload
lauren-pearson -
Category
Documents
-
view
217 -
download
0
Transcript of Testing with Formal Methods Ed Brinksma course 2004 A Formal Framework.
Testing with Formal Methods
Ed Brinksma
course 2004
A Formal Framework
TT 2004, Formal Framework 2© Ed Brinksma/Jan Tretmans
Software Testing
Testing is: important much practiced 30% - 50% of project effort expensive time critical not constructive
(but sadistic?)
But also: ad-hoc, manual, error-prone hardly theory / research no attention in curricula not cool :
“if you’re a bad programmer you might be a tester”
Attitude is changing: more awareness more professional
Improvements possible
with formal methods !?
TT 2004, Formal Framework 3© Ed Brinksma/Jan Tretmans
Model Based Testingwith Formal Methods
Claims :
Formal specifications are more precise, more complete, unambiguous
Formal models can be formally verified
Formal models allow automatic generation of tests
TT 2004, Formal Framework 4© Ed Brinksma/Jan Tretmans
Types of Testing
unit
integration
system
performancerobustness
functional behaviour
white box
black box
Level of detail
Accessibility
Characteristics
usability
reliability
module
security
TT 2004, Formal Framework 5© Ed Brinksma/Jan Tretmans
Formal Methods
Use mathematics to model relevant parts of software
Precise, formal semantics: no room for ambiguity or misinterpretation
Allow formal validation and reasoning about systems
Amenable to tools: automation
Examples: Z Temporal Logic First order logic SDL Specification and
Description Language LOTOS Promela Labelled Transition Systems Finite state machine Process algebra ......
TT 2004, Formal Framework 6© Ed Brinksma/Jan Tretmans
Formal methods: proving properties research sound theories “clean”
Testing : trial & error practice heuristics “dirty hands”
Testing & Formal MethodsA Perfect Couple ?
“Testing is not necessary after formal verification”
“Testing can only detect the presence of errors, not their absence”
“Formal methods are toys for boys”
“Formal methods have extreme potential - but not for my project”
TT 2004, Formal Framework 7© Ed Brinksma/Jan Tretmans
Testing with Formal Methods
Testing with respect to a formal specification
Precise, formal definition of correctness :
good and unambiguous basis for testing
Formal validation of tests
Algorithmic derivation of tests :
tools for automatic test generation
Allows to define measures expressing coverage
and quality of testing
TT 2004, Formal Framework 8© Ed Brinksma/Jan Tretmans
Testing & Formal Methods
Claims :
Combining the “mathematically clean” world of formal
methods and the “dirty hands” world of testing
Testing and formal methods are both necessary in
software development
Formal methods improve the testing process
Formal testing stimulates the use of formal methods
TT 2004, Formal Framework 9© Ed Brinksma/Jan Tretmans
Goal:Testing functional behaviour
of black-box implementation
with respect to specificationin formal language
based on formal definition
of conformance
Specification BasedFunctional Testing
with Formal Methods
formal model
systemimplementatio
n
formal testing
TT 2004, Formal Framework 10© Ed Brinksma/Jan Tretmans
testtool
implementationunder test
pass
fail
Test AutomationTraditional test automation
= tools to execute and manage test cases
Why not generate tests automatically ?
specification TTCNTTCNtestcase
s
TT 2004, Formal Framework 11© Ed Brinksma/Jan Tretmans
test
executionpass / fail
Formal Testing
test
generation
test suite TS
specification S
implementation i
correctness criterion
implementation
relation
i passes Ts
i ioco s
soundexhaustiv
e
TT 2004, Formal Framework 12© Ed Brinksma/Jan Tretmans
Formal relations
conforms-to IMPS SPECS iut conforms-to spec: implemetation under test obeys
specification hard: relates formal specification and real world system
imp MODS SPECS iIUT imp spec: model of the iut obeys specification relates formal objects
test hypothesis: iut conforms-to spec iIUT imp spec
iut passes t =def v ( EXEC ( t, iut )) = pass execute a test: EXEC: TESTS IMPS P OBS verdict function: v: P OBS { pass, fail } iut passes Ts t Ts . iut passes t
TT 2004, Formal Framework 13© Ed Brinksma/Jan Tretmans
Formal Testing : Conformance
s SPECS SpecificationIUT Implementation under Test
IUT is concrete, physical object
Model the physical world
But IUT is black box ! ?
Assume that model iIUT exists
specification S
implementation IUT
correctness criterion
IUT conforms-to s
TT 2004, Formal Framework 14© Ed Brinksma/Jan Tretmans
Formal Testing : Conformance
specification S
implementation iIUT
formalcorrectness
criterioniIUT imp s
s SPECS Specification
iIUT MODS model of IUT
Test hypothesis : each concrete IUT can be modelled
by some iIUT MODS
Conformance : iIUT imp s
iIUT is not known ;
testing to learn about iIUT
TT 2004, Formal Framework 15© Ed Brinksma/Jan Tretmans
Conformance testing
conforms-to IMPS SPECS iut passes t =def v ( EXEC ( t, iut )) = pass
iut passes Ts t Ts . iut passes t
conformance: complete: iut conforms-to spec iut
passes Ts
sound: iut conforms-to spec iut passes Ts
exhaustiveness: iut conforms-to spec iut passes Ts
ioco: input output conformance
TT 2004, Formal Framework 16© Ed Brinksma/Jan Tretmans
Formal Testing
exec : TESTS IMPS (OBS)
der : SPECS (TESTS)
Ts TESTS
s SPECS
IUT IMPS
imp
iIUT MODS
obs : TESTS MODS (OBS)
t: (OBS) {fail,pas
s}
OBS
pass
fail
Proof soundness and exhaustivess:iMODS .
( tder(s) . t(obs(t,i)) = pass ) i imp s
Test hypothesis :
IUTIMPS . iIUT MODS .
tTESTS . exec(t,IUT) = obs(t,iIUT)
TT 2004, Formal Framework 17© Ed Brinksma/Jan Tretmans
Formal Testing : Test Derivation
test
generation
test suite TS
specification S
Test generation :
der : SPECS ( TESTS )
Test suite - set of test cases : T TESTSTest case : t TESTS
TT 2004, Formal Framework 18© Ed Brinksma/Jan Tretmans
Formal Testing : Test Execution
test
execution OBS
test suite T
implementation IUT
Test execution leads to a set of observations : exec : TESTS IMPS ( OBS )
Model of test execution : obs : TESTS MODS ( OBS )
i IUT
TT 2004, Formal Framework 19© Ed Brinksma/Jan Tretmans
Formal Testing : Test Hypothesis
OBS
test suite T
Observational framework : TESTS, OBS, exec, obs
Test hypothesis :
IUT IMPS . iIUT MODS .
t TESTS . exec ( t, IUT ) = obs ( t, iIUT )
obsi IUT
test
execution
IUT
exec
IUT
exec
TT 2004, Formal Framework 20© Ed Brinksma/Jan Tretmans
Formal Testing : Verdicts
Observations are interpreted : t : (OBS) { fail, pass }
test
execution OBS t
pass
fail
TT 2004, Formal Framework 21© Ed Brinksma/Jan Tretmans
Testing for Conformance
IUT passes Ts def t Ts . IUT passes t
IUT passes t def t ( exec ( t, IUT ) ) = pass
Test hypothesis :
t TESTS . exec ( t, IUT ) = obs ( t, iIUT )Proof obligation :
i MODS .
( t Ts . t ( obs ( t, i ) ) = pass ) i imp s
IUT passes Ts IUT conforms-to s?
Definition : IUT conforms-to sIUT conforms-to s
iIUT imp s
t Ts . t ( obs ( t, iIUT ) ) = pass
t Ts . t ( exec ( t, IUT ) ) = pass
t Ts . IUT passes t
IUT passes Ts
TT 2004, Formal Framework 22© Ed Brinksma/Jan Tretmans
Testing for Conformance
IUT passes Ts
Proof of completeness on models
leads to completeness for tested systems :
i conforms-to s
exhaustive
sound
Proof obligation :
i MODS .
( t Ts . t ( obs ( t, i ) ) = pass ) i imp s
Test derivation: find such a Ts, der : SPECS P TESTS
TT 2004, Formal Framework 23© Ed Brinksma/Jan Tretmans
Formal Testing
exec : TESTS IMPS (OBS)
der : SPECS (TESTS)
Ts TESTS
s SPECS
IUT IMPS
imp
iIUT MODS
obs : TESTS MODS (OBS)
t: (OBS) {fail,pas
s}
OBS
pass
fail
Proof soundness and exhaustivess:iMODS .
( tder(s) . t(obs(t,i)) = pass ) i imp s
Test hypothesis :
IUTIMPS . iIUT MODS .
tTESTS . exec(t,IUT) = obs(t,iIUT)
TT 2004, Formal Framework 24© Ed Brinksma/Jan Tretmans
Approaches to Formal Testing
Finite State Machine based approaches: language acceptance
transition tours
distinguishing sequences
Labelled Transition Systems concurrency theory
implementation relations
Abstract Data Type testing
TT 2004, Formal Framework 25© Ed Brinksma/Jan Tretmans
Labelled Transition SystemsLabelled Transition System S, L, T, s0
ConReq
ConConf
Discon Data
Discon
states
actions,labels
transitionsT S (L{}) S
initial states0 S
TT 2004, Formal Framework 26© Ed Brinksma/Jan Tretmans
Labelled Transition Systems
S0 S1dub
transition
dub coffeeS0 S3
transitioncomposition
kwart teaS0 executable
sequence
non-executablesequence
kwart soupS0
LTS(L) all transitionsystems over L
L = { dub,kwart, coffee,tea,soup}
dub
coffee
kwart
tea
S1 S2
S3
S0s
S4
S5
TT 2004, Formal Framework 27© Ed Brinksma/Jan Tretmans
Labelled Transition Systems
dub
coffee
dub
tea
S1 S2
S3 S4
S0traces (s) = { L* | s }
s after = { s’ | s
s’ }
s Sequences of observable actions:
Reachable states:
traces(s) = { , dub, dub coffee, dub tea }
s after dub = { S1, S2 }
s after dub tea = { S4 }
TT 2004, Formal Framework 28© Ed Brinksma/Jan Tretmans
Formal Testing with Transition Systems
t: (traces){fail,pass}
exec : TESTS IMPS (OBS)
traces
der : LTS (TTS)
Ts TTS
s LTS
IUT IMPS
ioco
iIUT IOTS pass
fail
obs : TTS IOTS (traces)
Proof soundness and exhaustivess:
iIOTS .
( tder(s) . t(obs(t,i)) = pass )
i ioco s
Test hypothesis :
IUTIMPS . iIUT IOTS .
tTTS . exec(t,IUT) = obs(t,iIUT)