Claudia Szabo and Yong Meng Teo Department of Computer ...teoym/pub/10/PADS2010-slides.pdf ·...

Claudia Szabo and Yong Meng TeoDepartment of Computer ScienceNational University of SingaporeNational University of Singapore

email: [email protected]

Introduction Objective Related Work Validation of Data-driven Simulations◦ CoDES Framework Overview◦ Validation Approach◦ Validation Approach Validation of General Model Properties Formal Validation of Model Execution◦ Experimental Analysis

Conclusion

May 18, 2010 24th ACM/IEEE/SCS Workshop on Principles of Advanced and Distributed Simulation 2

“Capability to select and assemble simulation i i bi ti t ti fcomponents in various combinations to satisfy user

requirements”

Si l ti t “ bl lf t i d it Simulation component - “a reusable, self contained unit that is independently testable and usable in a variety of contexts. [It] interacts with its environment only through a well defined interface of inputs and outputs.”

Various levels of composability:

◦ technical, syntactic (engineering), semantic, pragmatic, dynamic, conceptual …


Does the composed model produces semantically correct results?

Key issues in validation:◦ Semantic composability is not a closed operation◦ Different validation perspectives:◦ Different validation perspectives: Software: logical correctness Simulation: temporal, formal ….◦ Emergent properties


Semantic validation is a difficult process that requires the presence of a system expert◦ Increased costs◦ No formal guarantee of results◦ No formal guarantee of results

Validation of semantic composability with formal p yguarantees and practical to implement◦ Large models – #components, #states

C l d l l i i i f◦ Complex models – logic, communication, rate of state changes, structure (feedback loops, fork/join connectors, etc.), …


Petty and Weisel’s formal theory◦ Components statically represented as functions of integers;◦ Components statically represented as functions of integers;

Composition = composition of mathematical functions; Validity = close enough to the simulation of a perfect model

◦ Static; composition is based on the linear order of components; validation relation undefinedvalidation relation undefined

Validation of DEVS models◦ A DEVS model is represented a schema in the Z specification

language; composition specification is validated using Z/EVES:language; composition specification is validated using Z/EVES: inconsistencies, theorems, and syntax, type, and domain errors

◦ Asynchronous; assumes knowledge about entire system◦

Validation of BOM models Validation of BOM models◦ Detailed user-specified scenario facilitates individual component

discovery; candidate components executed in all possible combinations and validated against scenarioI f l◦ Informal


CCC t l

MODEL COMPOSITIONConceptual

Model Definition

ModelDiscovery & Selection

Semantic ValidationSyntactic Verification

CC CoDESGUI

Query

attributes

ConceptualModel

ValidatedSimulator

CandidateSimulator

SelectedModel

QueryRequest(s)

CCSyntaxVerifier

ModelLocator

(MI)Model

SelectorSemanticValidator

CoDESFramework

(MI) Selector Validator

Candidate


ModelRepository

CandidateModels

Component: abstracted as a black-box with in and/or t i ti h l d t dout communication channel, and represented as a

meta-component:◦ mandatory & specific attributes

Behavior a finite timed state automaton◦ Behavior - a finite timed state automaton

[I ]S [ ] Cond S [O ][A ] Components are represented using framework COML

standard

[Il ]Sp[t] Cond n St[Ol ][Am ]

standard Simulator: represented by components (base, model)

linked using connectors


Knowledge is captured using component-based

The image part with relationship ID rId4 was not found in the file.

Knowledge is captured using component-based ontology and composition grammars


Queueing Networks◦ Base components: Source, Server, Sink◦ Data-driven modeling: no◦ Complex logic: no◦ Communication: normal



Military Training Simulations◦ Base components: Tank, SoldierTroop◦ Data-driven modeling: yesg y◦ Complex logic: yes◦ Communication: extensive


Base components:

Tank SoldierTroop

Extended Ontology Extended Composition Grammar


INVALID

1INVALIDsyntax ✗VALID

syntax ✓semantic ✓

2INVALIDsyntax ✓semantic ✗

I lid d l ti


- Invalid model properties- Invalid model execution

1 Validate desired model properties1. Validate desired model properties Correct and meaningful component communication Safety, liveness, deadlock-free – with time

- Covers software engineering + simulation perspectivesCovers software engineering simulation perspectives

2. Formal validation of model execution Comparison with reference model using state machine

- New time-based formalism defines validity- 5-step process validates model execution over time


Tank & SoldierTroop components need position information from adversaries to advance/fire towardsinformation from adversaries to advance/fire towards target

Based on received input logic decides: Based on received input, logic decides:◦ Advance toward target◦ Fire if enough ammo◦ After firing – move/stay in the same position◦ After firing move/stay in the same position …

Logic modeled in the COML 2.0 component representation:representation:◦ Auxiliary input parameters◦ Logic inserted in state machine


1 1 V lid t

1. Validation of

invalid1.1 ValidateComponent

Communicationvalid

Composition is invalid

General Model Properties 1.2 ValidateComponentCoordination

invalid Composition is invalid

1.3 ValidateComponentCoordination

valid

invalid Composition is invalidCoordination

over Time

2.1 Exact MatchWith R f

valid

invalid

Composition isexact

not exact

2. Formal Validation of Model Execution

With Reference Model

2 2 Similar to

Composition is valid

exact

similar Composition is


2.2 Similar to Reference

Model

pvalid

not similar Composition is invalid

Validates the logical coordination of components with t t ll ibl bi ti f t trespect to all possible combinations of states;

transitions are instantaneous

2 types of logical properties:◦ Safety – “Nothing bad ever happens” Entire composition: deadlock free, a property holds throughout the

checkcheck Individual components: a property holds throughout the check

◦ Liveness – “Something good will always happen” Individual components: a specific state is always reached

Represent composition using Promela and verify by the Spin model checker


mtype {MSG_POS,MSG_FIRE, MSG_DIE}; ...proctype SOLDIERTR(byte id, health, ..., posX, posY; chan in, out)

mtype {MSG}; chan to1 = [10] of {mtype}; ... proctype CON ONE TO ONE(chan in out)proctype SOLDIERTR(byte id, health, ..., posX, posY; chan in, out)

{bit initial = 1; byte posXFire, posYFire; byte msgPosX, msgPosY, auxX, auxY, auxDistance,...;

S0: atomic{if :: (initial == 1) -> initial = 0;

if :: out ! MSG_POS -> goto S1; fi fi}S1: atomic{ if atomic{if:: in ? MSG FIRE msgPosX msgPosY -> health = health - 10;

proctype CON_ONE_TO_ONE(chan in, out){do :: in ? MSG -> out ! MSG; od}

proctype TANK(byte id; chan in, out){ S1: atomic{ if :: in ? MSG -> if

100 bytes35 states

208 bytes3475 states

:: in ? MSG_FIRE, msgPosX, msgPosY > health health 10; if :: health < health_threshold ->

if :: out ! MSG_DIE -> goto end; fi:: else

if :: out!MSG_POS, posX, posY -> progress: printf("MSG sent\n"); goto S1;fi fi:: in ? MSG_DIE -> out ! MSG_DIE;goto end; //GPS coord

i ? MSG POS P X P Y

:: in ? MSG > if:: out ! MSG -> progress: printf("MSG sent\n"); goto S1; fi fi}}

proctype SOLDIERTR(byte id; chan in, out){bit initial = 1; S0: atomic{

:: in ? MSG_POS, msgPosX, msgPosY ->if :: !(msgPosX<posX-range||msgPosX>posX+range ||msgPosX<posY-range

||msgPosY<posY+range)->if :: ammo>0->out!MSG_FIRE,msgPosX,msgPosY; ammo--; goto S1; fi:: else -> auxDistance = distance;

:: msgPosX<posX->auxX = msgPosX+range; :: else -> auxX = msgPosX - range; fi} ...

S0: atomic{ if :: (initial == 1) -> initial = 0;if :: out ! MSG -> goto S1; fi fi}

S1: atomic{if g g }

//similar to calc nxt position if //broadcast position:: out ! MSG_POS, posX, posY -> goto S1; fifi fi } end: skip; } init{run TANK(1 100 20 5 40 45 to1 from1);

:: in ? MSG -> if :: out ! MSG -> progress: printf("MSG sent\n"); goto S1; fi fi}}

i it{run TANK(1, 100, 20, 5, 40, 45, to1, from1); run SOLDIERTR(2, 100, 10, 5, 15, 20, to2, from2); run CON_ONE_TO_ONE(1, from1, to2);run CON_ONE_TO_ONE(2, from2,to1); }

init{ run TANK(1, to1, from1); run SOLDIERTR(2, to2, from2); run CON_ONE_TO_ONE(from1, to2); run CON_ONE_TO_ONE(from2,to1); }

High level of detail, large state space; difficult to automate

Low level of detail, small state space; easy to automate


Validates safety and liveness through time

3 steps◦ Transform composition into Java classes◦ Specify safety and liveness Safety = validity points – data the user wants to passSafety validity points data the user wants to pass

through specific points in the composition Liveness = transient predicate – must change value during a

time intervaltime interval◦ Validate by running the meta-simulation and observing

the logical properties


Composed Model Reference Model(1) Formal

Representation (2) Unfolding and Sampling

(2) Unfolding and Sampling

(1) Formal Representationsampled

valuesC1 C1*

Composed Model Reference Model

(3) Composition (3) CompositionComposition CompositionC1 C3C2 C1 C3C2

1… …

(4) Simulation (4) Simulation

C1 C3C2 C1 C3C2

L(M) L(M*)

ion

ion Strong equivalence? ValidYes

(5) V

alid

at(5

) Val

idat

L(M) V L(M*)? ValidYes

No


( ) ( )

InvalidNo

2) Unfold & Sample 2) Unfold & Sample) & p ) & p

3) Composition3) Composition

3) Composition

Composed Model Reference ModelMay 18, 2010 24th ACM/IEEE/SCS Workshop on Principles of Advanced and Distributed Simulation 20

4) Simulation 4) Simulation

Composed Model Reference ModelComposed Model Reference Model

May 18, 2010 215) Validation

Validation of general model properties◦ Validation of component communication: Java program queries

the COSMO ontology using the Jena reasoner◦ Validation of component coordination: Instantaneous: Java + Promela specification validated using SPIN

model checker Timed: Java + threaded implementation

Validation of model execution◦ Java program transforms component COML files into p g p

functional representation and subsequently into LTS ◦ Exact match - BISIMULATOR tool from the CADP toolset

determines strong equivalence◦ Close match - Java program determines related states

according to proposed semantic metric relation; related states are subsequently validated by BISIMULATOR


M d l #C State Vector #St t Memory ExecutionModel #Comp State Vector Size (bytes) #States Memory

(MB)Execution

time (s)

Queueing Networks Application Domain

Single-Server 3 520 30 490 39 8 11 54Single-Server Queue#

3 520 30,490 39.8 11.54

Grid System*# 11 1,128 571,343 39.8 17.8

Grid System*# 15 1 556 892 009 39 8 21 6Grid System # 15 1,556 892,009 39.8 21.6

Military Training Application Domain

Tank vs Soldier 2 208 3 475 2 5 0 06Tank vs Soldier troop 2 208 3,475 2.5 0.06

*Grid meta-scheduler with two virtual organizations, two and three nodes respectively#Compiled with DMEMLIM = 40 MB


Compiled with DMEMLIM 40 MB

E tiModel #Components #LTS States Result Execution Time (s)

Queueing Networks Application Domain

Single-Server Queue 3 12 Valid 1.97

Single-Server Queue – 2 job classes 3 12 Invalid 4.72

Grid System* 11 51 Valid 5.54

Grid System* - 2 job classes 11 51 Invalid 8.29

Military Training Application Domain

Tank vs Soldier troop ( & ) 2 21 Valid 6 2(shoot & scoot) 2 21 Valid 6.2

Tank vs. Solider troop 2 21 Invalid 22.3

*G id t h d l ith t i t l i ti t d th d ti l


*Grid meta-scheduler with two virtual organizations, two and three nodes respectively

Data-driven military training application domain◦ Knowledge captured using ontology & compositional

grammarsgrammars

Automated semantic validation of military training iscenarios:

◦ Complex state machines: accuracy vs state-space explosion; accuracy vs automation in the validation of general model propertiesproperties

◦ Large #attributes: accuracy vs runtime in validation of model execution

Open problems◦ Composition using model components

V lid ti f d d l ith t ti◦ Validation of composed models with emergent properties


1. C. Szabo, Y.M. Teo, On Validation of Semantic Composability in Data-driven

email: [email protected]

1. C. Szabo, Y.M. Teo, On Validation of Semantic Composability in Data driven Simulations, Proc. of 24th Workshop on Principles of Advanced and Distributed Simulations, Atlanta, USA, 2010.

2. C. Szabo, Y.M. Teo and S. See, A Time-based Formalism for the Validation of Semantic Composability, Proc. of Winter Simulation Conference, pp 1411 –1422, Austin, USA, 2009 (SIGSIM 2009 Best PhD Student Paper).

3. C. Szabo and Y.M. Teo, An Approach for Validation of Semantic Composabilityi Si l ti M d l P f 23 d W k h P i i l f Ad d din Simulation Models, Proc. of 23rd Workshop on Principles of Advanced and Distributed Simulation, pp 3-10, Lake Placid, USA, 2009.

4. Y.M. Teo and C. Szabo, CODES: An Integrated Approach to ComposableModeling and Simulation Proc of 41st Annual Simulation Symposium ppModeling and Simulation, Proc. of 41st Annual Simulation Symposium, pp 103–110, Ottawa, Canada, 2008.

5. C. Szabo and Y.M. Teo, On Syntactic Composability and Model Reuse, Proc. of International Conference on Modeling and Simulation pp 230–237 Phuketof International Conference on Modeling and Simulation, pp 230 237, Phuket, Thailand, 2007 (invited paper).


Validate component communicationC t t l k th l ( t ) b t◦ Components not only speak the same language (syntax) but also understand each other (semantics)

◦ Calculate degree of semantic data alignment according to the ontologyontology

Validate component coordinationL i l t di ti i th t t f◦ Logical component coordination in the context of instantaneous and timed transitions Instantaneous – all possible interleaved transitions; focus on

computation; safety liveness deadlock freedomcomputation; safety, liveness, deadlock freedom Timed – specific attribute values; user specifies safety & liveness;

based on sampling


Validates the logical coordination of components with t t ll ibl bi ti f t trespect to all possible combinations of states;

transitions are instantaneous

2 types of logical properties:◦ Safety – “Nothing bad ever happens” Entire composition: deadlock free, a property holds throughout the

checkcheck Individual components: a property holds throughout the check

◦ Liveness – “Something good will always happen” Individual components: a specific state is always reached

Represent composition using Promela and verify by the Spin model checker


Claudia Szabo and Yong Meng Teo Department of Computer ...teoym/pub/10/PADS2010-slides.pdf ·...

Documents

Transcript of Claudia Szabo and Yong Meng Teo Department of Computer ...teoym/pub/10/PADS2010-slides.pdf ·...