Testing Applications ofCyber-Physical Systemsinthe Presenceof Uncertainty

MartinA.SchneiderFraunhofer FOKUS,Berlin,Germany

October10th,STVWorkshop,Berlin,Germany

Projectfacts:Totalcost:EUR3713233,75

EUcontribution:EUR3713233,75

Coordinator:OsloMedtech,Norway

Topic(s):ICT-01-2014- SmartCyber-PhysicalSystems

Fundingscheme:RIA- ResearchandInnovationaction

Overallprojectobjective:ImprovingCPSdependabilityviasystematicandautomatedtestingofUncertaintyinCPS

Theconsortium

Resultsandmethods

Keyexpectedresults:• UnderstandingUncertainty(U-Taxonomy)

• ModelingFramework• ExtensibleandConfigurable

• TestingFramework• ExtensibleandConfigurable

• ToolsimplementingTaxonomyandFrameworks

• Standards(Crosscutting)

Model-BasedTesting:• Abstraction• ManagingComplexity

• Automation

• Systematic

Search-BasedTesting• Optimization• SmartMechanisms

• Discoveringunknownuncertainties

• GeneticAlgorithms.....

4

Socrates

»I know that I know nothing«

»I know that I don’t know«

»I know that I don’t know with certainty«©PhotographbyGregO'Beirne.CroppedbyUser:Tomisti /WikimediaCommons/CC-BY-SA-3.0 /GFDL

Agenda

1. UncertaintyandCyber-PhysicalSystems2. UncertaintyTaxonomy3. UncertaintyModelling4. UncertaintyTesting

©FraunhoferFOKUS

Uncertainty

“anydeviationfromtheunachievableidealofcompletelydeterministicknowledgeoftherelevantsystem”

Walkeretal.(2003):Defininguncertainty:aconceptualbasisforuncertaintymanagementinmodel-baseddecisionsupport

„systemstateofincompleteorinconsistentknowledgesuchthatitisnotpossible[…]whichoftwoormorealternativeenvironmentalorsystemconfigurationsholdataspecificpoint”

A.J.Ramirezetal.(2012):Ataxonomyofuncertaintyfordynamicallyadaptivesystems.

©FraunhoferFOKUS

UncertaintyinCyber-PhysicalSystems

• Cyber-physical systems are connected embedded systems thatintegrate computation,networking and physical processes.

• Uncertainty arises from interaction between• elements of the CPS‘s infrastructure InfrastructureLevel• application(s)and the infrastructure of the CPS IntegrationLevel• humans and the environment with the CPS Application Level

©FraunhoferFOKUS

CPSmay be notdependable

• undesiredbehaviour ofaCPSisobservedatruntime• duetouncertaintyinthedigitalxphysical

environment

•Challenge• Howtofindfindsuchscenariosefficiently intheinfiniteandcomplexspaceofthescenarios?

• Solution• Searchalgorithms

UseCasesforUncertaintyTestingAutomatedWarehouse• automaticallystoresandunloadsgoods• manualinterventionsometimesrequired

• handlinggoods• updatingdatabase

GeoSports• automaticallytracksallkindsofmovementsduringamatch(positioningviatriangulation)

• improvingperformanceofathletes• athletewearsadevicethatconstantlycommunicateswithlocatinginfrastructure

©FraunhoferFOKUS

©ULM

AHa

ndlingSystem

s,Spain

©FutureP

osition

X,Swed

en

UncertaintyandKnowledge

©FraunhoferFOKUS

knowledgeweareawareof

things we knowthat we don‘t knowthem

things we don‘t knowthat knowledge exists

things we don‘t knowand are notaware of

knownknown

knownunknown

unknownknown

unknownunknown

certainty uncertainty

knowledge exists

awareness

UncertaintyandKnowledge

©FraunhoferFOKUS

knowledgeweareawareof

thingsweknowthatwedon‘tknowthem

thingswedon‘tknowthatknowledgeexists

thingswedon‘tknowandarenotawareof

knownknown

knownunknown

unknownunknown

certainty uncertainty

provided by use cases,observed inthe field

goal of U-Testtofindsuch

uncertainties

knowledge exists

awareness

UncertaintyandRisk

• uncertainty w.r.t.to the occurrence (likelihood)of arisk• uncertaintiesdonothaveaprobabilityassigned• uncertaintycoverspositiveandnegativeoutcomeswhileriskfocussesonnegativeoutcomes,e.g.,threats

• uncertainty as asource of risk• uncertain behavior:manifestation of anuncertainty as anbehavior of the CPSwith anegativeimpact onits dependability

©FraunhoferFOKUS

Agenda

1. UncertaintyandCyber-PhysicalSystems2. UncertaintyTaxonomy3. UncertaintyModelling4. UncertaintyTesting

©FraunhoferFOKUS

UncertaintyTaxonomy(Excerpt)

• nature• epistemic• aleatoric

• environment• cyberenvironment• physicalenvironment

• cause• humanbehavior• naturalprocess• technologicalprocess

• impact• direct• indirect• impactedelement

©FraunhoferFOKUS

Agenda

1. UncertaintyandCyber-PhysicalSystems2. UncertaintyTaxonomy3. UncertaintyModelling4. UncertaintyTesting

©FraunhoferFOKUS

UncertaintyModelling

Uncertainy Modelling Framework(UMF)

• StateMachines• describetheexpectedinput/outputbehavioroftheSUT• from the perspective of SUT

• Uncertainties• characterization of uncertainties interms of the UMF• that are related to the model

©FraunhoferFOKUS

[guard] trigger / effect

GeoSports StateMachine

©FraunhoferFOKUS

Agenda

1. UncertaintyandCyber-PhysicalSystems2. UncertaintyTaxonomy3. UncertaintyModelling4. UncertaintyTesting

©FraunhoferFOKUS

Search-basedUncertaintyTesting

• cover known uncertainties described by use case providers• by using use case descriptions (state machines)• by using information from modelled uncertaintes

• discover unknown uncertainties• by exploiting information from known uncertainties (coupling effect)• by recombining uncertainties

©FraunhoferFOKUS

Search-based Uncertainty Testing

• genetic algorithm

• individual:state machines representing use cases• mutation:applying mutation operators to state machines

• first generation:apply mutation operators solely based onuncertainty information

• further generations:increase amount of mutations notrelated tomodelled uncertainties

• crossover:combination of uncertainties

©FraunhoferFOKUS

MutationOperatorsMutationOperator Description Constraints/Comments

AddTransition Addsanewtransitionbyduplicatinganexistingoneandsettinganewsourceandtargetstate.

RemoveTransition Completelyremovesthetransition. Transitionshavinganinitialstateassourceorafinalnodeastargetmustnotberemoved.

Equivalentto‘ChangeGuard:replaceexpressionwithfalse’.

RemoveTransition(withStateMerge)

Completelyremovesthetransition.

Mergesthesourceandtargetstateiftheremovedtransitionistheonlyoneconnectingthem(optional:withthesamedirection).Thisavoidmutilatedstatemachineswhichinhibitgeneratingtestcases.

Transitionshavinganinitialstateassourceorafinalnodeastargetmustnotberemoved.

Equivalentto‘ChangeGuard:replaceexpressionwithfalse’.

ReverseTransition Swapssourceandtargetofthetransition.

Transitionshavinganinitialstateassourceorafinalnodeastargetmustnotbereversed.

Optional:Transitionsbeingtheonlyonethatconnectsourceandtargetstatemustnotberemoved(optional:withthesamedirection).Thisavoidmutilatedstatemachineswhichinhibitgeneratingtestcases.

ChangeSource/Target Movethesourceorthetargetofthetransitiontoanyotherstate.

Incasethetargetstateofthetransitionischanged,thetargetmustnotbetheinitialstate.Incasethesourcestateofthetransitionischanged,thesourcemustnotbethefinalnode.

MutationOperator Description Constraints/Comments

RemoveTrigger Transformsthetransitiontoacompletiontransition.

RemoveGuard Removestheguardofatransitioncompletely.

Equivalentto‘ChangeGuard:replaceexpressionwithtrue’

RemoveEffect Removestheeffectofatransitioncompletely.

ChangeTriggerOperation Changestheoperationtoanotheroneofthesameinterfaceoftheoriginaloperation.

ChangeGuard/ChangeEffect

- replaceexpressionwithtrue/false- negateexpression- replacesubexpressionwithtrue/false- negatesubexpression- changelogicaloperator- changerelationaloperator- changearithmeticoperator- changesetoperator- changequantifier- replaceoperand

guard/effectmutationoperators- removestatement- movestatement- fixparameter/propertyofacalledmethodorsentsignal- changecalledmethodorsentsignal- changeoperator- fixoperand(replaceitwithaliteral)- changeoperand(replacewithvariable,callparameterorsignalpropertyofthesametype)- replaceresult:replaceright-hand-side(RHS)expressionwithdefaultvalueofleft-hand-side(LHS)

GuardsandeffectsarewritteninC#.

Search-basedTestingwithaGeneticAlgorithm

22

...

startingpoint

candidatesolutions

mutation

qualityevaluation(fitnessevaluation)

40

15

70

10

crossover

85

&selection

testcasegeneration&execution

testcasegeneration&execution

=statemachines

©FraunhoferFOKUS

exploitingmodelled

uncertainties

recombineuncertainties

©FraunhoferFOKUS

Mutation OperatorRemove Trigger

Example

©FraunhoferFOKUS

mountMutation OperatorAdd Transition

Example

Search-basedUncertaintyTesting:FitnessFactors

• generic,simple,model-basedprofileforfitnessfactors

©FraunhoferFOKUS

CoverageCriteria

• TraditionalTransitionCoverage (statemachine)

• UncertaintyCoverage (model)

• MutationTransitionCoverage (statemachine)

• KnownUncertaintySpaceCoverage (allgenerationsrelatedtoasingleuncertainty)

©FraunhoferFOKUS

#𝑡𝑟𝑎𝑛𝑠𝑖𝑡𝑖𝑜𝑛𝑠)*+,-,.#𝑡𝑟𝑎𝑛𝑠𝑖𝑡𝑖𝑜𝑛𝑠/00

#𝑢𝑛𝑐𝑒𝑟𝑡𝑎𝑖𝑛𝑡𝑖𝑒𝑠)*+,-,.#𝑢𝑛𝑐𝑒𝑟𝑡𝑎𝑖𝑡𝑖𝑒𝑠4*.,00,.

#𝑚𝑢𝑡𝑎𝑡𝑖𝑜𝑛𝑠)*+,-,.#𝑚𝑢𝑡𝑎𝑡𝑖𝑜𝑛𝑠/00

#467/78*9:#:7/7,:× #:7/7,:<= ×#*>,-/78*9:×?

Afew,early numbers...

MutationOperator

#TestCases

#RemovedTestCases

#RemainingTestCasesCompletePath MutatedTransition

Coverage

ChangeTransitionTarget 51 0 0 51

ChangeTransitionSource 5 5 0 0

RemoveTransition 5 0 0 5

AddTransition 51 0 51 0

RemoveEffect 5 5 0 0RemoveGuard 252 1 0 251

RemoveTrigger 51 0 0 51

[1] Test cases generated by MS SpecExplorer based on the mutated state machines by traversing the state machines.[2] Test cases generated by MS SpecExplorer do not necessarily end in a final state. Hence, first all complete pathsstarting from an initial state and ending in a final state are selected in the first stage.

Conclusions &FutureWork

• small effort for testers• start from functional models (state machines)• add declarative uncertainty descriptions

• reduction of search space• search is guided by modelled uncertainties

• configurable and extendable• by modelled uncertainties• and model-based fitness factors

• empirical evaluation onthe case studies

Thankyouforyourattention!

@utesth2020

www.u-test.eu

U-test.eu