Model-based testing of executable statecharts

Model-BasedTes,ngofExecutableStatecharts

TomMens,AlexandreDecanSo-wareEngineeringLab

UniversityofMons,Belgium

TomMens–SATTOSE2016–Bergen,Norway–July2016

Many“agile”developmenttechniquesprovidelightweightapproachestofacilitatechangeandincreasereliabilityofso-ware

•  Qualityassessment(e.g.badsmellsandrefactoring)•  Defensiveprogramming(e.g.designbycontract)•  Test-drivendevelopment(e.g.unittesSngandbehavior-drivendevelopment)

•  DynamicverificaSonofbehaviouralproperSes

Weproposetoraisethesetechniquestothelevelofexecutable(statechart)models

Agileanddefensivedevelopment

2


Futurework(spoiler)

FacilitateevoluSonofbehaviouraldesignmodels– DetecSngmodelsmells– Modelrefactoring

•  E.g.spliXngupacomplexstatechartintomulSplestatecharts–  SemanScvariaSon

•  DetecSngifstatechartiscompaSblewithalternaSvesemanScs–  Variabilityanalysis

•  Considerproductfamilies(e.g.differentmicrowavevariants)andanalysecommonaliSesandvariabiliSesintheirstatechartmodels

– DesignspaceexploraSon•  AnalyseprosandconsofsyntacScallydifferent,butsemanScallysimilarstatecharts

3


• AdvancedmodeltesSng(focusofthistalk)•  Contract-drivenmodeling•  Test-drivenmodeling(unittesSngandBDDforstatecharts)

•  DynamicverificaSon(propertystatecharts)

• Futurework•  Modelqualityassessment(modelsmells)•  Modelqualityimprovement(modelrefactoring)•  Modelchecking•  Modelvariabilityanalysis•  DesignspaceexploraSon•  ModelcomposiSonandscalability•  SemanScvariaSon

Agileanddefensivemodelling

4


Runningexample

Microwaveoven<<component>>

Input

<<component>>Door

opened() closed()

<<component>>Control ler

-power : integer-timer : integer

<<component>>Lamp

switch_on() switch_off()

<<component>>Power

reset() inc() dec()

<<component>>Heating

set(power : integer) on() off()

<<component>>Turntable

start() stop()

<<component>>Timer

inc() dec() reset() tick()

<<component>>Display

clear() set(i : integer) set(s : string)

<<component>>Cooking

start() stop()

<<component>>WeightSensor

item_placed() item_removed()

beep(d : integer)

<<component>>Beeper

Visual Paradigm Standard Edition(University of Mons)

5


Runningexample

Usecasename:CookFoodSummary:Userputsfoodinoven,andovencooksfood.Assump,ons:Ovenhasbeenconfiguredwithweightsensorandturntable.Precondi,ons:Ovenisclosedandempty.Postcondi,ons:Ovenhascookedthefood.Ovenisclosedandempty.Basiccourseofac,on:1.Useropensdoor.2.Userputsfoodinovenandclosesdoor.3.UsersetscookingSmeviacontrolpanel.4.Userpressesstartbubon.5.Magnetronindicatorlightswitcheson.Magnetronstartscookingfood.6.RemainingcookingSmeisdisplayedconSnuously.7.SystemnoSfiesuserwhencookingSmehaselapsed.Magnetronindicatorlightswitchesoff.8.Useropensdoor,removesfoodfromoven,andclosesdoor.9.Systemclearsdisplayandresetsdefaultvaluesforcooking.


Runningexample

Usecasename:CookFoodAlternatecourses:1a:Userpressesstartbubonwhiledoorisopen.Systemdoesnotstartcooking.3a:Userpressesstartbubonwhilenofoodisintheoven.Systemdoesnotstartcooking.3b:UserpressesstartbubonwhilecookingSmeiszero.Systemdoesnotstartcooking.5a:Useropensdoorduringcooking.Magnetronstopsandindicatorlightturnsoff.Userremovesfood,closesdoorandpressesStop.Gotostep9.5b:Useropensdoorduringcooking.Magnetronstopsandindicatorlightturnsoff.UserclosesdoorandpressesStarttoresumecooking.Gotostep5.5c:UserpressesStopduringcooking.Magnetronstopsandindicatorlightturnsoff.UserpressesStarttoresumecooking.Gotostep5.


Runningexample

Microwaveoven


Runningexample

Ovencontrollerstatechart


So-ware-controlledsystemsaredifficulttodevelop

Controlso-warecanbeverycomplex– ConSnuousinteracSonbetweenso-wareandhardware

– ConSnuousinteracSonwithexternalworldandusers

– Mustrespectfunc8onalrequirements•  OvenshouldcookfoodplacedinovenwithspecifiedpowerandduraSon

– Mustrespectnon-func8onalrequirements•  Ovenshouldstopsendingmicrowavesifdoorsareopened

10


•  AddpreciseanddynamicallyverifiablespecificaSonstoexecutableso-warecomponents(e.g.,methods,funcSons,classes)

•  BasedonBertrandMeyer’s“DesignbyContract”•  Theso-warecompomentshouldrespectacontract,composedof–  precondi8ons–  postcondi8ons–  invariants

Contract-drivendevelopment

11


Example(takenfromwww.eiffel.com/values/design-by-contract/introducSon)

Contract-drivendevelopment

classDICTIONARY[ELEMENT]featureput(x:ELEMENT;key:STRING)isrequirecount<=capacitynotkey.emptyensurehas(x)item(key)=xcount=oldcount+1endinvariant0<=countcount<=capacityend

12


Contract-drivenmodelling

Contractsformicrowavecontroller

contextcontrollerinv:notsent(heaSng_on)oracSve(cookingmode)inv:Smer>=0inv:0<power<=MAXPOWER

contextcookingmodepre:Smer>0inv:Smer>=0inv:power==power@prepost:received(door_opened)orSmer==0

contextreadyinv:Smer>0


Tellingstories

Story(eventdoor_opened,eventitem_placed,eventdoor_closed,eventSmer_dec).tell(interpreter)

14


Exampleoffailingcontract

InvariantErrorState:controllerAsserSon:Smer>=0ConfiguraSon:[controller,doorclosed,closedwithitem,programmode,notready]Step:eventSmer_decinternaltransiSononclosedwithitem

15


SoluSontofailingcontract

AddguardstotheacSonsassociatedtotheeventsthatincrementanddecrementpowerandSmer

Smer_dec[Smer>0]/Smer-=1power_inc[power<MAXPOWER]/power+=1power_dec[power>1]/power-=1

16


Test-drivendevelopment

testnegaSve_Smer:Story(door_opened,item_placed,door_closed,Smer_dec).tell(statechart)statechart.execute()assertEqual(State(controller).Smer,0)testno_hea8ng_when_door_is_not_closed:Story(door_opened,item_placed,Smer_inc,cooking_start).tell(statechart)statechart.execute()assertFalseacSve(cookingmode)assertFalsesent(heaSng_on)

testnegaSve_Smer...FAILtestno_heaSng_when_door_is_not_closed…ok===========================================AsserSonError:-1!=0----------------------------------------------------------------------Ran2testsin0.005sFAILED(failures=1)

WithoutguardsonSmer_decevent

17


Test-drivendevelopment

testnegaSve_Smer:Story(door_opened,item_placed,door_closed,Smer_dec).tell(statechart)statechart.execute()assertEqual(State(controller).Smer,0)testno_hea8ng_when_door_is_not_closed:Story(door_opened,item_placed,Smer_inc,cooking_start).tell(statechart)statechart.execute()assertFalseacSve(cookingmode)assertFalsesent(heaSng_on)

18

testnegaSve_Smer...oktestno_heaSng_when_door_is_not_closed…ok----------------------------------------------------------------------Ran2testsin0.005sOK

WithguardsonSmer_decevent


•  IncludecustomertestpracScesintoTDD•  EncouragecollaboraSonbetweendevelopers,QA,andnon-technicalstakeholders(domainexperts,projectmanagers,users)

•  Useadomain-specific(non-technical)languagetospecifyhowthecodeshouldbehave–  BydefiningfeaturespecificaSonsandscenarios

•  Reducesthetechnicalgapbetweendevelopersandotherprojectstakeholders

Behaviour-DrivenDevelopment

19


So-warebehaviourcanbedescribedinadomain-specific(non-technical)languagesuitedtonon-developers–  usingtheGherkinlanguage–  SupportedbyCucumberframeworkinmanylanguages

Behaviour-drivendevelopment

20


Example(takenfromdocs.behat.org/en/v2.5/guides/1.gherkin.html)


Feature:ServecoffeeInordertoearnmoneycustomersshouldbeabletobuycoffeeScenario:BuylastcoffeeGiventhereis1coffeele-inthemachineAndIhavedeposited1dollarWhenIpressthecoffeebubonThenIshouldbeservedacoffee

21



Feature:Nohea8ngifdoorisopenedScenario:Nohea8ngwhennothingisdoneGivenIdonothingAndIexecutethestatechartThenstatecooking_modeshouldnotbeacSveAndeventheaSng_onshouldnotbefiredScenario:Nohea8ngwhenitemisplacedGivenIsendeventdoor_openedWhenIsendeventitem_placedTheneventheaSng_onshouldnotbefiredScenario:Nohea8ngwhendoorisnotclosedGivenIsendeventdoor_openedAndIsendeventitem_placedWhenIsendeventdoor_closedTheneventheaSng_onshouldnotbefired

Firstvariant.

SSllreferstospecificdetailsofthestatechart(stateandeventnames)

22

1featurepassed,0failed,0skipped3scenariospassed,0failed,0skipped11stepspassed,0failed,0skipped,0undefinedTook0m0.005s



Feature:Nohea8ngifdoorisopenedScenario:Nohea8ngwhennothingisdoneWhenIpowerupthemicrowaveThenheaSngshouldnotbeonScenario:Nohea8ngwhenitemisplacedGivenIopenthedoorWhenIplaceanitemThenheaSngshouldnotturnonScenario:Nohea8ngwhendoorisnotclosedGivenIopenthedoorAndIplaceanitemWhenIclosethedoorThenheaSngshouldnotturnon

Secondvariant.

Muchclosertonaturallanguage.Allstatecharts-specificconcepts

areabstractedaway.

23


Coverageanalysis

Statecoverage:81.82%Enteredstates:controller(3)|doorclosed(4)|dooropened(2)|closedwithoutitem(3)|openedwithoutitem(2)|openedwithitem(2)|closedwithitem(1)|notready(1)|programmode(1)Remainingstates:cookingmode|readyTransi,oncoverage:16.67%Processedtransi,ons:openedwithoutitem[item_placed]->openedwithitem(2)closedwithoutitem[door_opened]->openedwithoutitem(2)openedwithitem[door_closed]->closedwithitem(1)

24


Propertystatecharts

DefineandverifybehaviouralproperSesby1.  instrumenSngthestatechartinterpreter2.  intercepSngspecificacSonsofstatechartbeing

executed•  entered(<NAMEOFSTATE>)•  exited(<NAMEOFSTATE>)•  consumed(<NAMEOFEVENT>)•  sent(<NAMEOFEVENT>)•  …

3.  execuSngapropertystatechartthatverifiesadesirableorundesirableproperty

25


Propertystatecharts

<<property statechart>>Heating does not start if door is opened

door is closed

door is opened

consumed(door_closed)

consumed(door_opened)

failure

sent(heating_on)

<<property statechart>>Heating must stop when door is opened

heating is off

heating is on

sent(heating_off)

sent(heating_on)

heating is on while door is opened

failure

consumed(door_opened)

consumed(tick)

sent(heating_off)

26


Toolsupport

Sismic=SismicInteracSveStatechartModelInterpreterandChecker– PythonlibraryavailableonPythonPackageIndex(PyPI)

–  releasedunderopensourcelicenceLGPLv3– Sourcecode•  github.com/AlexandreDecan/sismic

– DocumentaSon•  sismic.readthedocs.io

27


Toolsupport

SismicsupportsallaforemenSonedconcepts– StatechartexecuSon– Designbycontract– UnittesSng– BDD– Coverageanalysis– Propertystatecharts– Andmore…

28


Sismicfileformat

RepresenSngastatechartasaYAMLfile

rootstate:name:controllercontract:-always:notsent('heaSng_on')oracSve('cookingmode')-always:Smer>=0-always:0<power<=MAXPOWERini,al:doorclosedonentry:|power=DEFAULTSmer=0transi,ons:-event:input_cooking_stopacSon:|Smer=0

29


Sismicfileformat

RepresenSngastatechartasaYAMLfile

states:-name:doorclosedini,al:closedwithoutitemstates:-name:closedwithoutitemtransi,ons:-event:door_openedtarget:openedwithoutitem-name:closedwithitemini,al:programmodeonexit:send('display_clear')transi,ons:-event:door_openedtarget:openedwithitem-event:input_Smer_incac,on:|Smer=Smer+1send('display_set',text='TIMER:%d'%Smer)…


SismicExecuSngstatecharts

Stepwiseexecu8onofstatechartbehaviourfromsismic.ioimportimport_from_yaml

fromsismic.interpreterimportInterpreterfromsismic.modelimportEvent

withopen('microwave.yaml')asf:statechart=import_from_yaml(f)

interpreter=Interpreter(statechart)

interpreter.execute_once()MacroStep(None,[],>['controller','doorclosed','closedwithoutitem'],<[])

interpreter.queue(Event(’door_opened’))interpreter.execute_once()

MacroStep(Event(door_opened),[TransiSon(closedwithoutitem,openedwithoutitem,door_opened)],>['dooropened','openedwithoutitem'],<['closedwithoutitem','doorclosed'])

31


SismicRunningstories

fromsismic.storiesimportStorystory=Story([Event('door_opened'),Event('item_placed'),Event('door_closed'),Event(’Smer_inc'),Event(’cooking_start'),Event(’Sck')])trace=story.tell(interpreter)

32

MacroStep(None,[],>['controller','doorclosed','closedwithoutitem'],<[]),MacroStep(Event(door_opened),[Transi8on(closedwithoutitem,openedwithoutitem,door_opened)],>['dooropened','openedwithoutitem'],<['closedwithoutitem','doorclosed']),MacroStep(InternalEvent(lamp_on),[],>[],<[]),MacroStep(Event(item_placed),[Transi8on(openedwithoutitem,openedwithitem,item_placed)],>['openedwithitem'],<['openedwithoutitem']),MacroStep(Event(door_closed),[Transi8on(openedwithitem,closedwithitem,door_closed)],>['doorclosed','closedwithitem','programmode','notready'],<['openedwithitem','dooropened']),…


SismicRunningstories

MacroStep(InternalEvent(lamp_off),[],>[],<[]),MacroStep(Event(8mer_inc),[Transi8on(closedwithitem,None,8mer_inc)],>[],<[]),MacroStep(None,[Transi8on(notready,ready,None)],>['ready'],<['notready']),MacroStep(InternalEvent(display_set,text=TIMER:1),[],>[],<[]),MacroStep(Event(cooking_start),[Transi8on(ready,cookingmode,cooking_start)],>['cookingmode'],<['ready','programmode']),MacroStep(InternalEvent(hea8ng_set_power,power=900),[],>[],<[]),MacroStep(InternalEvent(hea8ng_on),[],>[],<[]),MacroStep(InternalEvent(lamp_on),[],>[],<[]),MacroStep(InternalEvent(turntable_start),[],>[],<[]),MacroStep(Event(8ck),[Transi8on(cookingmode,None,8ck)],>[],<[]),MacroStep(None,[Transi8on(cookingmode,programmode,None)],>['programmode','notready'],<['cookingmode']),MacroStep(InternalEvent(display_set,text=REMAINING:0),[],>[],<[]),MacroStep(InternalEvent(hea8ng_off),[],>[],<[]),MacroStep(InternalEvent(lamp_off),[],>[],<[]),MacroStep(InternalEvent(turntable_stop),[],>[],<[]),MacroStep(InternalEvent(beep,number=3),[],>[],<[]),MacroStep(InternalEvent(display_set,text=DONE),[],>[],<[])]

33


SismicunittesSng

•  Usingpython’sbuilt-inunibestmodule$python-munibestheaSng_unibest.py–v

deftest_no_heaSng_when_nothing_is_done(self):self.interpreter.execute()self.assertFalse(self.is_heaSng())deftest_no_heaSng_when_item_is_placed(self):events=map(Event,['door_opened','item_placed'])story=Story(events)story.tell(self.interpreter)self.interpreter.execute()self.assertFalse(self.is_heaSng())

34


SismicBDD

•  UsingPython’sbehavemodule$sismic-behavemicrowave.yaml--featuresheaSng.feature

frombehaveimportgiven,when,thenfromsismic.ioimportimport_from_yamlfromsismic.interpreterimportInterpreterfromsismic.interpreter.helpersimportlog_tracefromsismic.modelimportEvent@given('Iexecutethestatechart')defexecute_statechart(context):_execute_statechart(context,force_execuSon=True)@then('state{state_name}shouldbeacSve')defstate_is_acSve(context,state_name):assertstate_nameincontext._statechart.states,'Unknownstate{}'.format(state_name)assertstate_nameincontext._interpreter.configuraSon,'State{}isnotacSve'.format(state_name) 35


SismicRegressiontesSng

Whenanerrorisencountered(e.g.duetofailingcontractorbug),story_from_tracecanreproducethescenariooftheobservedbehavior,whichcanbeusedasthebasisofaregressiontest.

36


SismicCommunicaSngstatecharts•  Statechartscancommunicatewithotherstatechartsorexternalcomponents(e.g.auserinterface)bysending/receivingevents

•  Realisedbydynamicallybindingtheirstatechartinterpreters

37


SismicCommunicaSngstatechartsExampleforsomeelevatorstatechart:Eventssentbybubonsarepropagatedtoelevatorelevator=Interpreter(import_from_yaml(open(‘elevator.yaml')))bubons=Interpreter(import_from_yaml(open(‘bubons.yaml')))bubons.bind(elevator)bubons.queue(Event(’floor_2_pushed'))bubons.execute()AwaiSngeventsinbubons:[Event(bubon_2_pushed)]AwaiSngeventsinbubons:[InternalEvent(floorSelected,floor=2)]AwaiSngeventsinelevator:[Event(floorSelected,floor=2)]elevator.execute()print('Currentfloor:',elevator.context.get('current'))Currentfloor:2

38


SismicOtherfeatures

OthersemanScvariantsofstatecharts– outer-firstinsteadofinner-firstsemanScs;– changingpriorityofevents– …

DifferentwaysofdealingwithSme– RealSmeversussimulatedSme

39


Conclusion

Wesupportvariouswaystoteststatechartmodels– Usingcontracts– Usingunittests– Usingdomain-specificfeaturesandscenarios(BDD)

– Usingpropertystatecharts

ImplementedinSismic,anopensourcePythonlibraryforinterpreSngstatecharts

40


Futurework

•  MoreadvancedtesSngtechniques– AutomaScgeneraSonofcontractsbasedonscenariospecificaSons

– AutomaScgeneraSonoftestsbasedoncontractspecificaSons

– MutaSontesSng–  SupportforconSnuousintegraSon

•  Explore/comparewith(dynamic?)modelcheckingtechniques–  Basedontemporallogics,labeledtransiSonsystems,…– UsingDwyer’sspecificaSonpaberns

•  Andmanymore…41


Futurework

FacilitatestatechartevoluSon– DetecSngmodelsmells– Modelrefactoring

E.g.spliXngupacomplexstatechartintomulSplestatecharts–  SemanScvariaSon

DetecSngifstatechartiscompaSblewithalternaSvesemanScs–  Variabilityanalysis

Considerproductfamilies(e.g.differentmicrowavevariants)andanalysecommonaliSesandvariabiliSesintheirstatechartmodels

– DesignspaceexploraSonAnalyseprosandconsofsyntacScallydifferent,butsemanScallysimilarstatecharts

42

Model-based testing of executable statecharts

Science

Transcript of Model-based testing of executable statecharts