April 20, 2006 Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands 1 Model...
-
date post
22-Dec-2015 -
Category
Documents
-
view
214 -
download
1
Transcript of April 20, 2006 Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands 1 Model...
April 20, 2006April 20, 2006 Model Program Based Black-Box Testing, LModel Program Based Black-Box Testing, Lentedagen, Vught, The Netherlandsentedagen, Vught, The Netherlands
11
Model Program Based Model Program Based Black-Box TestingBlack-Box Testing
Margus VeanesMargus Veanes
Foundations of Software Foundations of Software Engineering Microsoft Research, Engineering Microsoft Research,
RedmondRedmond
April 20, 2006 Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands
2
System-level black-box System-level black-box testingtesting
►Requires model of system behaviorRequires model of system behavior►Behavior is often Behavior is often
reactive/nondeterministicreactive/nondeterministic Implementation is multi-threaded or Implementation is multi-threaded or
distributeddistributed Thread scheduling hard to controlThread scheduling hard to control
►State space is typically infiniteState space is typically infinite Objects, unbounded value typesObjects, unbounded value types
►Traditional FSM-based testing does often Traditional FSM-based testing does often not scale not scale
April 20, 2006 Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands
3
Model based testingModel based testing
Model
Implementation
Test OracleTest Cases
Are run by Providesactual results for
PassFail
Provides expected results forGenerates
UserInfo
April 20, 2006 Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands
4
Our approachOur approach
► Behavior is described by Behavior is described by model programsmodel programs Written in Written in Spec# Spec# oror AsmL AsmL Describe reactive system behavior as Describe reactive system behavior as model model
automataautomata Use Use alternating simulation alternating simulation for conformancefor conformance
► Model exploration, validation and model-Model exploration, validation and model-based testing are provided by the based testing are provided by the Spec Spec ExplorerExplorer tool developed at MSR tool developed at MSR Supports scenario controlSupports scenario control Provides Provides offlineoffline as well as as well as onlineonline testing testing Views test cases as Views test cases as game strategiesgame strategies Checks conformance violationsChecks conformance violations
April 20, 2006 Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands
5
The big pictureThe big picture
Model program
Constrained model
Test cases
Pass/Fail
Implementation
Scenario control
Modeling
Test generation
Test execution Coding
Online testing
April 20, 2006 Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands
6
Conformance checkingConformance checkingcomponent viewcomponent view
Constrained model
Model
IUT wrapper
IUT
Refinement checker
Controllable
actions
Controllable
actions
Observable
actions
Observable
actions
Pass/Fail
d1……
c1
Object mapping
April 20, 2006 Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands
7
Example: Example: ChatChat
IUT wrapper
IUT
Chat server
Controllable actions:
Observable actions:
Login …Send
Receive
Refinement checker
Constrained chat model
b.Send(“hi”)
m.Send(“hello”)
b.Send(“bye”)
m.Rcv(“hi”)
m.Rcv(“bye”)
b.Rcv(“hello”)
b.Rcv(“hello”)
m.Rcv(“bye”)
m.Rcv(“hi”)
b.Rcv(“hello”)
BobMarym
b
Object mapping
April 20, 2006 Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands
8
Refinement relationRefinement relation
►Conformance between the model and the Conformance between the model and the system under test is system under test is alternating refinementalternating refinement
b.Send(“hi”)
m.Send(“hello”)
Spec System
b.Send(“hi”)
m.Send(“hello”)
m.Receive(b,“hi”)
b.Receive(m,“hello”)
b.Receive(m,“hello”)
m.Receive(b,“hi”)
b.Receive(m,“hello”)
April 20, 2006 Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands
9
Example: Example: Valid chat scenarioValid chat scenario
Bob Server Mary
B.Send(“hi”)M.Send(“hello”)
M.Receive(B,“hi”)
B.Receive(M,“hello”)
Tim
e
Service should guarantee Service should guarantee local consistencylocal consistency of message delivery (FIFO of message delivery (FIFO wrt sender)wrt sender)
SendSend is a is a controllable controllable actionaction
ReceiveReceive is an is an observable observable actionaction
B.Send(“bye”)
M.Receive(B,“bye”)
April 20, 2006 Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands
10
Model programs and Model programs and automataautomata
► States are mappings of variables to values States are mappings of variables to values first-order structuresfirst-order structures
► Initial state is given by initial assignment to Initial state is given by initial assignment to variables variables
► Actions are defined by method invocationsActions are defined by method invocations Actions are either Actions are either controllablecontrollable or or observableobservable
► An action An action mm((vv)) is is enabledenabled is a state is a state SS if the if the preconditionprecondition of of mm((vv)) is true in is true in SS
► An action An action mm((vv)) transitionstransitions from state from state SS11 to to statestate SS22 if if mm((vv)) is enabled in is enabled in SS11 and its and its execution in execution in SS11 yields yields SS22
April 20, 2006 Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands
11
Model program : Model program : Chat serviceChat service
State:
Actions:
class Client { bool entered = false; Map<Client, Seq<string>> unreceivedMsgs = Map{};}
class Client { void Send(string message) requires entered; { foreach (Client c in enumof(Client), c != this, c.entered) c.unreceivedMsgs[this] += Seq{message}; }
void Receive(Client sender, string message) requires sender != this && unreceivedMsgs[sender].Length > 0 && unreceivedMsgs[sender].Head == message; { unreceivedMsgs[sender] = unreceivedMsgs[sender].Tail;}…}
ControllableControllable::
Observable:Observable:
April 20, 2006 Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands
12
Online testingOnline testing
►For large reactive systems deriving an For large reactive systems deriving an exhaustive test-suite is infeasible exhaustive test-suite is infeasible
►Merge test generation and test Merge test generation and test execution into one processexecution into one process
►Online testing is a form of model-based Online testing is a form of model-based stress testingstress testing
►Purely random exploration is wastefulPurely random exploration is wasteful Use Use scenario controlscenario control to direct online testing to direct online testing
April 20, 2006 Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands
13
Exploration & Scenario Exploration & Scenario controlcontrol
► Exploration is the process of unfolding a rule-Exploration is the process of unfolding a rule-based model program into a transition system based model program into a transition system Actions move the system from state to stateActions move the system from state to state State comes from variables containing valuesState comes from variables containing values Compound data types are possible (sets, maps, Compound data types are possible (sets, maps,
sequences, etc…)sequences, etc…) Objects are identitiesObjects are identities
► Scenario control Scenario control Imposes limits on explorationImposes limits on exploration
► Using filters, state groupings, Using filters, state groupings, scenario actionsscenario actions , … , …
April 20, 2006 Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands
14
Example Demo: Example Demo: exploration of a coffee machineexploration of a coffee machine
April 20, 2006 Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands
15
Scenario controlScenario control
► In general the transition system is infinite, but we In general the transition system is infinite, but we can impose limitscan impose limits
► GoalGoal Create a state space of manageable size that satisfies a Create a state space of manageable size that satisfies a
given testing goalgiven testing goal► Two main tasksTwo main tasks
Restrict action parameter domains to interesting valuesRestrict action parameter domains to interesting values Restrict the state space to interesting statesRestrict the state space to interesting states
► NoteNote: : The two tasks are not necessarily independent!The two tasks are not necessarily independent!
April 20, 2006 Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands
16
Scenario control (cont.)Scenario control (cont.)
► The following techniques are used in Spec The following techniques are used in Spec ExplorerExplorer State filtersState filters Stopping conditionsStopping conditions State groupingsState groupings Search orderSearch order Enabling conditions on actionsEnabling conditions on actions Scenario actionsScenario actions
► Usually a combination of all (or some) of the Usually a combination of all (or some) of the techniques is needed to achieve desired effecttechniques is needed to achieve desired effect
April 20, 2006 Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands
17
Example of scenario actions: Example of scenario actions: Chat initializationChat initialization
► Use a parameterized scenario action Start.Use a parameterized scenario action Start. Invoking Start(Invoking Start(nn) produces a sequence of ) produces a sequence of nn Create()Create() invocations invocations
followed by followed by nn Enter()Enter() invocations invocations[Action(Kind=Scenario)]void Start(int nrOfMembers) requires enumof(Client) == Set{}; { Seq<Client> clients = Seq{}; // 1-- The given number of // clients are created for (int i = 0; i < nrOfMembers; i++) clients += Seq{Create()};
// 2-- all clients enter the session for (int i = 0; i < nrOfMembers; i++) clients[i].Enter();}
Start(2)
Create()/c0;Create()/c1;c0.Enter();c1.Enter();
April 20, 2006 Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands
18
Test generationTest generation► The result of exploration is a finite transition system The result of exploration is a finite transition system
with observable and controllable actions : with observable and controllable actions : a model a model automaton A. automaton A.
► Testing can be viewed as a Testing can be viewed as a gamegame between two players between two players Testing tool (TT) Testing tool (TT)
► Making Making controllablecontrollable transitions transitions Implementation under test (IUT)Implementation under test (IUT)
► Making Making observableobservable transitions transitions► The “rules of the game” are dictated by The “rules of the game” are dictated by AA► Tests are generated fromTests are generated from A. A. Test cases are Test cases are strategiesstrategies
A strategy tells what move TT should make in a state where A strategy tells what move TT should make in a state where its actions are enabledits actions are enabled
The purpose of a strategy may be to achieve transition The purpose of a strategy may be to achieve transition coverage or to reach a certain goal statecoverage or to reach a certain goal state
April 20, 2006 Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands
19
Demo: chatDemo: chat
April 20, 2006 Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands
20
ExperiencesExperiences
► Most tool features driven by demands of Most tool features driven by demands of internal users at MS (mostly testers)internal users at MS (mostly testers)
► Models help discover more bugs during Models help discover more bugs during modeling (design bugs) than testingmodeling (design bugs) than testing Testers do not get credit for finding those Testers do not get credit for finding those
bugs !!!bugs !!!
► During testing models help discover deep During testing models help discover deep system-level bugs where manual test system-level bugs where manual test scenarios don’tscenarios don’t Such bugs are hard to understand and fixSuch bugs are hard to understand and fix
April 20, 2006 Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands
21
Experiences (cont.)Experiences (cont.)
► Bugs appear in both models and Bugs appear in both models and implementations implementations (the ratio is roughly 50-50)(the ratio is roughly 50-50)
► Code coverage is a poor measure for testing Code coverage is a poor measure for testing concurrent software, often a single execution concurrent software, often a single execution thread provides the same coverage.thread provides the same coverage.
► New versions of implementation usually New versions of implementation usually require only local changes to models, require only local changes to models, whereas manual tests often need to be whereas manual tests often need to be rewritten completelyrewritten completely
► The tool is built on .NET but has been used to The tool is built on .NET but has been used to test distributed C++ applications. Most test test distributed C++ applications. Most test harnesses at MS are built in managed code.harnesses at MS are built in managed code.
April 20, 2006 Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands
22
OutlookOutlook
► User wish list:User wish list: Language independence for modelingLanguage independence for modeling
► Use C#, VB, …Use C#, VB, … Better control over explorationBetter control over exploration
► Scenario-oriented modelsScenario-oriented models Model compositionModel composition
► Model features separately and compose the modelsModel features separately and compose the models Model validationModel validation
► Validate properties on modelsValidate properties on models
Drives the focus of current research at Drives the focus of current research at FSEFSE
April 20, 2006 Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands
23
Related foundational workRelated foundational work
►Abstract state machinesAbstract state machines► Interface automataInterface automata► IInput-nput-OOutput utput COCOnformance (nformance (IOCOIOCO) )
theorytheory►MMarkov arkov DDecision ecision PProcess (rocess (MDPMDP) theory) theory
April 20, 2006 Model Program Based Black-Box Testing, Lentedagen, Vught, The Netherlands
24
Thank you!Thank you!
►FSE web site:FSE web site: http://http://research.microsoft.comresearch.microsoft.com/foundations/foundations
►Downloading Spec Explorer:Downloading Spec Explorer: http://http://
research.microsoft.com/specexplorerresearch.microsoft.com/specexplorer
Questions?Questions?