Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

EXPLICATING AND REASONING WITH MODEL UNCERTAINTY

Marsha ChechikUniversity of Toronto

July, 2014

Acknowledgements: work done in collaboration with R. Salay, M. Famelis, A. DiSandro and many others

Why Do Engineers Build Models?• To understand

• …the interesting characteristics of an existing or desired (complex) system and its environment

• To predict• …the interesting characteristics of the system by analysing its

model(s)• To communicate

• …their understanding and design intent (to others and to oneself!)• To specify

• ...the implementation of the system (models as blueprints)

Source: Bran Selic’s Modeling Course

Software Engineering Models

Requirements

ArchitectureBehaviour Static Design

Use CasesDeployment

Concepts

StructureAbstraction Purposeful Reasoning

Model-Based Software Engineering (MBSE)• An approach to system and software development in which software

models play an indispensable role• Based on two time-proven ideas:

switch (state) { case‘1:action1; newState(‘2’); break; case‘2:action2; newState(‘3’); break; case’3:action3; newState(‘1’); break;}

(2) AUTOMATIONS1

e1/action1e2/action2

e3/action3

switch (state) { case‘1:action1; newState(‘2’); break; case‘2:action2; newState(‘3’); break; case’3:action3; newState(‘1’); break;}

(1) ABSTRACTIONS1

e1/action1e2/action2

e3/action3

Realm of

modelinglanguages

Realm of tools

Source: Bran Selic’s Modeling Course

Model-Based Software Engineering (MBSE) Toolbox (partial list!)Support for:

Development,analysis,

refinement,transformation of models

+Communication,

documentation

This is Natalie.

Natalie is a modeler.

Natalie faces uncertainty in her everyday work

Alternative Designs

Hmm, I don’t know which

one, yet.

Conflicting Stakeholder Opinions

What do I do until they decide?

Incomplete Information

I don’t know everything about

this, yet.

Uncertainty in Software Development

Uncertainty about the content of the model.

Many design alternatives Conflicting stakeholder opinions

Incomplete information

Classifying Natalie’s Uncertainty[Aughenbaugh06] classification

Is inherently unknowable?

In the world(aleatory, “gambler”)

In the knowledge(epistemic, “science”)

Yes(irreducible uncertainty)

No(reducible

uncertainty)

Location of uncertainty

Uncertainty about the content of the model.

MBSE Toolbox for Natalie?Existing tools do not applyOptions:

• Wait until uncertainty gets resolved (how long?)• Make a provisional decision and run with it (need backtracking!)

We propose:• Defer resolution of uncertainty but incorporate

uncertainty handling into the development processto allow progress

• Lift standard MDE tools to handle models with uncertainty• Add new tools:

• Help Natalie express her uncertainty• Help Natalie determine what she does not yet know• Help Natalie manage and reduce uncertainty

• Motivation:• MBSE toolbox• Sources of uncertainty• Goal: MBSE toolbox for uncertainty-aware models

• Creating the toolbox• Modeling uncertainty• Transforming models containing uncertainty• Reasoning about models• Uncertainty-reducing refinement• Tool support

• Putting this into context

EXPRESSING UNCERTAINTY

Representing Uncertainty: ExampleSolver

SolverException

+ effect : String

Unsure if it should be an inner class.

Unsure if we need this field.

Notation means “inner class” in UML

Some Decisions to Make

Options for assigning values to states of confidence

Quantitative

Examples: 0, 0.3, 0.75

Good for capturing likelihood of events

Hard to capture one’s own uncertainty

Qualitative

Examples: yes/no/don’t know strongly … strongly agree disagree

Good for capturing one’s own level of uncertainty

Less precise than quantitative

How to Represent Uncertainty?Ask users

• Goals: model uncertainty explicitly, keep syntax familiar

[MiSE’13]

Representing Uncertainty

Partial Models

• Points of uncertainty (“May elements”) explicated using syntactic annotations

Solver

SolverException

+ effect : String

Unsure if we need this field.

[ICSE12]

Partial Models

Propositional variables: “the element exists”

Solver

SolverException

+ effect : String

[ICSE12]

Representing UncertaintySolver

SolverException

+ effect : String

Solver

SolverException

Solver

SolverException

Solver

SolverException

+ effect : String

Solver

SolverException

+ effect : String

x=F, y=F x=T, y=F

x=F, y=T x=T, y=T

4 concretizations: 4 ways to resolve uncertainty.

Partial Models

• Restrictions to the set of concretizations can be captured in the “May formula”

Solver

SolverException

+ effect : String

Representing UncertaintySolver

SolverException

+ effect : String

Solver

SolverException

Solver

SolverException

Solver

SolverException

+ effect : String

Solver

SolverException

+ effect : String

x=F, y=F x=T, y=F

x=F, y=T x=T, y=TX v Y

-(S)securityRelated : (SV)Types1-numOfPassengers : Int

Vehicle

CarTimeMachine Boat

(M)Hovercraft

(MS)OtherVehicles

-numOfDoors : Int(V)C1

(M) (M)

(M)(M)

(SV)Types1

P1:ClassDiagram

More Complex Design Uncertainty

Not sure which security attributes - but know some will be needed.

Not sure which class will hold the numOfDoors attribute

Not sure if we will have Hovercraft or which class it specializes.

Not sure what other vehicle types there will be

• May• Not sure if a model element should occur• annotations: M – maybe E - exists (default)

• Abs• Not sure how many elements exist that play a particular role• annotations: S - set, P - particular (default)

• Var• Not sure whether the element will be merged with another• annotations: V – variable, C - constant (default)

• OW• Not sure if the model is complete• annotations: M - maybe, E - exists (default)

Capturing Partial Information with MAVO

M – maybe E – exists (default)

S – set P – particular (default)

V – variable C – constant (default)

inc – incomplete comp – complete (default)

A MAVO Class Diagram

Not sure which security attributes - but know some will be needed.

Vehicle

CarTimeMachine Boat

(M)Hovercraft

(MS)OtherVehicles

(M) (M)

(M)(M)

(SV)Types1

P1:ClassDiagram

P1:MAVO(ClassDiagram)

• Benefits• Partial model refinement = uncertainty reduction • Formal: allows analysis and automation• Semantics-independent – applies to models of different types• Use instead of concrete models …

… to allow deferral of decisions without delaying progress

Partial Models

A partial model represents the entire set of possible concrete models

UNCERTAINTY TOOLBOX

Transforming Models with Uncertainty

Natalie should be able to use model transformations

[MODELS’13]

Transforming Models with Uncertainty

We need to lift Natalie’s transformations so that they can apply to models with uncertainty

Existing transformation techniques do not support this!

Natalie should be able to use model transformations

[MODELS’13]

Reminder: Model Transformations With Graph Rewriting (“Classical”)

class1

+ attribute : type

class1

- attribute : type+ getAttribute() :type

class1

class2

NegativeApplicationCondition

LeftHandSide

RightHandSide

EncapsulateVariable refactoring:Make fields private and add getter methodsunless they belong to some inner class

Example rule:

Graph Rewriting: Example Input Model

SolverSolverException

+ effect : String

class1

+ attribute : type

class1

class2

RHSLHSNAC

Graph Rewriting: Example Input Model

SolverSolverException

+ effect : String

class1

+ attribute : type

class1

class2

RHSLHSNAC

NAC also matches! ABORT !

Graph Rewriting: Example Input Model 2

SolverException

+ effect : String

class1

+ attribute : type

class1

class2

RHSLHSNAC

SolverException

+ effect : String

class1

+ attribute : type

class1

class2

RHSLHSNAC

Delete

SolverException

- effect : String

class1

+ attribute : type

class1

class2

RHSLHSNAC

SolverException - effect : String + getEffect() : String

class1

+ attribute : type

class1

class2

RHSLHSNAC

SolverException - effect : String + getEffect() : String

class1

+ attribute : type

class1

class2

RHSLHSNAC

No more LHS matches.

And Now: Transforming Models With Uncertainty

Natalie wants to apply the rule to an input with uncertainty

Why Transforming Models with Uncertainty is Hard?

Solver SolverException

+ effect : String

class1

+ attribute : type

class1

class2

RHSLHSNAC

Match???

YX v Y

+ effect : String

class1

+ attribute : type

class1

class2

RHSLHSNAC

Match???

YX v Y

+ effect : String

class1

+ attribute : type

class1

class2

RHSLHSNAC

Should we delete?Should we add?

YX v Y

Goal: “Lifting”

class1

+ attribute : type

class1

- attribute : type+ getAttribute():type

class1

class2

RHSLHSNAC

Solver

SolverException

+ effect : String

Goal: “Lifting”

class1

+ attribute : type

class1

class2

RHSLHSNAC

Solver

SolverException

+ effect : String

Solver

SolverException

+ effect : StringSolver

SolverException

Goal: “Lifting”

class1

+ attribute : type

class1

class2

RHSLHSNAC

Solver

SolverException

+ effect : String

Solver

SolverException

Solver

SolverException

-effect : String+getEffect() : String

Solver

SolverException

+ effect : String

Solver

SolverException

Goal: “Lifting”

class1

+ attribute : type

class1

class2

RHSLHSNAC

Solver

SolverException

+ effect : String

Solver

SolverException

( X ¬Y ¬a ¬b) ∧ ∧ ∧v(¬X Y ¬a b) ∧ ∧ ∧v( X Y a ¬b)∧ ∧ ∧

Solver

SolverException

+ - effect : String+getEffect() : String

Goal: “Lifting”

class1

+ attribute : type

class1

class2

RHSLHSNAC

Solver

SolverException

+ effect : String

( X ¬Y ¬a ¬b) ∧ ∧ ∧v(¬X Y ¬a b) ∧ ∧ ∧v( X Y a ¬b)∧ ∧ ∧

Solver

SolverException

Lifting Technique

class1

+ attribute : type

class1

class2

RHSLHSNAC

Solver

SolverException

+ effect : String

(a) Find Match

Step 1: Determine applicability

Lifting Technique

class1

+ attribute : type

class1

class2

RHSLHSNAC

Solver

SolverException

+ effect : String

(a) Find Match

(b) Make sure the rule applies to at least one concretization

(requires solvinga SAT problem)

Step 1: Determine applicability

Lifting Technique

class1

+ attribute : type

class1

class2

RHSLHSNAC

Solver

SolverException

+ effect : String

Solver

Step 1: Determine applicabilityStep 2:Transform graph

SolverException

+ effect : StringY

(a) Copy over unchangedparts

Lifting Technique

class1

+ attribute : type

class1

class2

RHSLHSNAC

Solver

SolverException

+ effect : String

Solver

SolverException

Step 1: Determine applicabilityStep 2:Transform graph

(a) Copy over unchangedparts

(b) Perform additions and deletions

Added and deleted elements become Maybe

Lifting Technique

class1

+ attribute : type

class1

class2

RHSLHSNAC

Solver

SolverException

+ effect : String

Step 1: Determine applicabilityStep 2:Transform graphStep 3:Transform formula

( X ¬Y ¬a ¬b) ∧ ∧ ∧v(¬X Y ¬a b) ∧ ∧ ∧v( X Y a ¬b)∧ ∧ ∧

Solver

SolverException

Constrain Maybe elementsto ensure each thateach concrete modelis correctly transformed.

Lifting Technique Summary

Solver

SolverException

+ effect : String

( X ¬Y ¬a ¬b) ∧ ∧ ∧v(¬X Y ¬a b) ∧ ∧ ∧v( X Y a ¬b)∧ ∧ ∧

Solver

SolverException

Step 1: Determine applicability (Henshin + Z3 SMT Solver)Step 2:Transform graph (Hanshin)Step 3:Transform formula (Java + Z3 input strings)

Case Study

• Object–relational mapping (ORM)

• “Translate a class diagram to a relational database

schema.”

• Classic benchmark for model

transformation research

• Triple graph grammar with 5 layered graph rules [Varro06]

(Image from [Varro06])

Case Study• Input model: the Ecore metamodel

• ORM for Ecore is important: cf. CDO and Teneo

• Manually flattened inheritance hierarchy and adapted to the

metamodel in [Varro06]

• Resulting model had 65 model elements:

• 17 classes, 17 associations, 6 generalization links, 25 attributes

• Manually injected points of uncertainty to create partial models with

increasing numbers of concretizations

Setup And Results

• RQ: How does lifting scale with increasing uncertainty?

• Varied: number of concretizations of input

• Measured: time to complete the ORM transformation

• Ran on Intel Core i7-2600 3.40GHz×4core, 8GB RAM, Ubuntu-64 12.10.

• Runtime does not increase dramatically. Approach scales.

# concretizations 1 24 48 108 144 192 256Time (seconds) 32.6 32.8 32.7 32.9 32.6 33.0 48.4

Verification

Example property:“Every inner class has at least one attribute”

Natalie’s favorite analysis

technique

How can you apply it to partial models?

Verification: the Naïve ApproachExample property:

“Every inner class has at least one attribute”

Enumerate:

Verification: LiftingExample property:

“Every inner class has at least one attribute”

Natalie’s favorite analysis technique,

LIFTED

• Applies directly to the partial model

• Does not enumerate concretizations

• Computes result using three-valued logic

…all concretizations

…some but not all

…none

Property holds for…

Verification: Lifting

• Checking syntactic properties [ICSE’12]:• Language independent• SAT-based

• Checking semantic properties:• Depends on language semantics• Requires custom adaptation of verification technique• Examples:

• Goal satisfaction in i* models [RE’12, RE’14]• MTSA: Model checking and other analyses over MTSs [ASE’08]

Scalability of Verification• Q: Is verification of models with uncertainty feasible?

• How does it compare with checking each model individually?• What about comparison with the classical verification?

• Q: How does the level of uncertainty affect this feasibility?

Experiments: with random inputs and real case studies

Random inputs:varying size of the model and level of uncertaintyspeedup = time for set of conventional models

time to check partial model

Results

Case Study: Checking Properties• Why? Triangulate experimental results (random inputs) with

observations from a real-word scenario

• Case study details:• Bug from the UMLet bugzilla• Realistic bug fixes• Uncertainty: which combination of actions that fix the bug

to choose from?• We checked two properties from the literature [SMSJ03]• Resulting partial model: 27,261 elements, 220

concretizations

[SMSJ03] R.V. Der Straeten, T. Mens, J. Simmonds, and V. Jonckers. “Using Description Logic to Maintain Consistency between UML Models”. In Proc. of UML’03, pp 326-340, 2003

Results

Results of Evaluation

• Is there a speedup?- Yes, it is consistently faster than reasoning with the set.

• How is speedup affected by changing model size and levels of uncertainty?- Speedup decreases with model size- Speedup increases with level of uncertainty- No slowdowns!

Reasoning with Partial modelsvs

Reasoning with a set of conventional models

Property-Based RefinementSATSolver

SolverException

+ effect : String

Property Φ1: “Every inner class has at least one attribute”

Goal:Create a partial model of the concretizations for which the property holds.

(X v Y) Λ Φ1

We just need to update the May formula:

Uncertainty-Reducing Refinement• Systematically incorporate

new information…• … Manually

• Edit the uncertainty annotations

• … Using automated transformations

Verify that a transformation is always uncertainty reducing.

Unsure if we need this

field.

You know what? I want to get rid of

effect.[FASE’12, VOLT’12]

-numOfDoors : IntCarTimeMachine Boat (MS)OtherVehicles

-lastLegalEntry : Timestamp-lastIDNumber : (V)IDType-previousIDNumber : (V)IDType-numOfPassengers : Int

Vehicle

(V)IDType

Timestamp

P2:ClassDiagram

A MAVO RefinementNot sure which security attributes - but know some will be needed.

Vehicle

CarTimeMachine Boat

(M)Hovercraft

(MS)OtherVehicles

(M) (M)

(M)(M)

(SV)Types1

P1:ClassDiagram

Decision

Decision: remove it

Partial Decision

Refines

P1:MAVO(ClassDiagram)

Making the Toolbox... MMINT: Model Management INTeractive

• A platform for developing (typed) model management tools• Model management environment:

• Diff, refine, merge, etc.Available at https://github.com/adisandro/MMINT

CSPSAT/SMTHenshinAlloy

Making the Toolbox... Uncertainty-awareMMINT: Model Management INTeractive

• A platform for developing (typed) model management tools• Model management environment:

• Diff, refine, merge, etc.Available at https://github.com/adisandro/MMINTMU-MMINT – lifting of model management operators + new ones

CSPSAT/SMTHenshinAlloy

MU-MMINT(pronounced “Moomin”)

PUTTING UNCERTAINTY INTO CONTEXT

Back to Uncertainty Modeling• Our choice:

• Qualitative, possibilistic uncertainty modeling

• Keeping concrete syntax familiar• Encoding an exact set of potential

classical models• Refinement – reducing the set of

possibilitiesExplicit uncertainty modelingwith Partial Models + May formulas

Representation of Sets

Uncertainty

Product Lines

Metamodels

Megamodels

ConcretizationsPartial model

ProductsProduct Line

InstancesMetamodel

MembersMegamodel

Inspired by: Modal Transition Systems (MTS)

LTS Extension:– Required transitions– Possible transitions– the rest is proscribed (prohibited)

– MTS toolbox – MTSA [ASE’08]

request?

request

[Larsen/Thompson, 1988]

reply? request?

request

MTS = Set of LTS

request?reply?

request

requestreply

MTS = Set of LTS

request?request

reply? request?

request

request?reply?

request

requestreply

Our Modeling of UncertaintyWe generalize MTSs in two ways:1. Make partiality modeling language-independent

• Benefit: consistent applicability across all phases of SE• Cost: reasoning is only about syntactic properties

2. Use different types of partiality for different types of uncertainty

• Benefit: methodological principles for expressing uncertainty

Aside: Complete w.r.t Scope and Span

• Scope – what is of interest (and what is not)• Span – level of abstraction

Valid call sequences of public methods of class Socket Political Map of Africa

sdkfjsdkfj

sdkfj sdkfj

Some Similar Concepts• Non-determinism

• Only in behavioral models• Corresponds to omitted information

• Underspecification• Specification – saying what is known/required• Everything else about a given scope and span is

unspecified (uncertainty is implicit)

• With “our” uncertainty – can say what we do not know• Including statements about a given scope and span• Uncertainty is explicit

Some More Similar Concepts

• Incompleteness• Often same as underspecification• But can also include knowledge about the state of

completeness about the model w.r.t. a given scope and span

• “Our” uncertainty model can explicitly state which parts are incomplete

Uncertainty is a Meta-Property! • Allows the to capture:

uncertainty of the specificationuncertainty in the variability modeluncertainty in a meta-modeluncertainty in parts of a non-deterministic model

• Vagueness in modeling [Hermann]

• Incompleteness in databases• Representing null values in db [Fagin, etc.]

• Local Closed World Assumption [Cortes-Calabuig, etc.]

Other Similar Concepts

Summary

Some Current and Future Work• Front-end for modeling uncertainty [MiSE’13]• Extending the uncertainty toolbox to handle full MAVO models• Lifting other techniques that require semantics to handle

uncertainty• E.g., non-functional analysis

• Understanding origins and models of uncertainty• Mining developer conversations to infer uncertainty in collaborative

design [RSSE’14]• Building uncertainty dashboard to help the design process

• Uncertainty patterns• Bottom-up: useful for tool building• Top down: detecting uncertainty in workflows and conversations

Some Current and Future Work• More complex modeling of uncertainty

• Combining “our” uncertainty with environment uncertainty• Handling richer qualitative models (fuzzy logic? probabilities?)

• Using explicit uncertainty modeling to bridge the gap with informal modeling/sketching• Bran Selic’s “formality slider”

• Uncertainty all the way down to code• Uncertainty as a meta-aspect, to be handled using language

workbenches• Lifting programming languages, e.g., Java• Symbolic execution of partial models

• Lifting analyses to other set-handling formalisms• E.g., transformations in product lines [ICSE’14]• Code analyses?

Questions?

Bibliography I[MODELS13] M. Famelis, R. Salay, A. Di Sandro, M. Chechik. “Transformation of Models Containing Uncertainty”. In Proc. of MODELS’13, 2013.

[ICSE12] M. Famelis, M. Chechik, and R. Salay. “Partial Models: Towards Modeling and Reasoning with Uncertainty”. In Proc. of ICSE’12, 2012.

[ICSE14] R. Salay, M. Famelis, J. Rubin, A. Di Sandro, M. Chechik. “Lifting Model Transformations to Product Lines”. In Proc. of ICSE’14, 2014.

[FASE12] R. Salay, M. Famelis, and M. Chechik. “Language Independent Refinement using Partial Modeling”. In Proc. of FASE’12, 2012.

Bibliography II[Varro06] D. Varro, S. Varro-Gyapay, H. Ehrig, U. Prange, and G. Taentzer. “Termination Analysis of Model Transformations by Petri Nets”. In Proc. of ICGT’06, pages 260–274, 2006.

[Classen09] A. Classen, P. Heymans, P. Schobbens, A. Legay, andJ. Raskin. “Model Checking Lots of Systems: Efficient Verification of Temporal Properties in Software Product Lines”. In Proc. of ICSE'10, pages 335-344, 2010.

[Svahnberg05] M. Svahnberg, J. Van Gurp, and J. Bosch. “A Taxonomy of Variability Realization Techniques”. Software: Practice and Experience, 35(8):705-754, 2005.

I don’t know vs I don’t care• Knowledge vs relevance scale

I care I don’t careI know Specification

(classical)Out of scope

I do not know Uncertainty Out of scope

MAVO • M – default is exists. Explicit only about uncertainty of

existance• A• V• O – by default, the world is closed unless considered

REASONING OVER SETS

“Plural” Models

“Singular” model m Plural

model p

“Deconstruct”D : p {m}

“Extract”E: p x r m

Lifted Transformation

T : p1 p2“Construct”C: {m} p

“Project” P : p1 x r p2

|p1| > |p2|

The intent behind each abstraction determines

pragmatics

Sets of Models – A Useful Abstraction?

• Unified way of looking at set abstractions

• Reuse methods from one abstraction to another

• Chain abstractions?

• Finding connections between fields: principled but practical outcomes

Example: Lifting Transformations for Product Lines

1. Determine applicability2. Transform graph3. Transform formula

1. Determine applicability2. Transform graph3. Transform presence conditions

Transforming Partial Models Transforming Product Lines [ICSE’14]

May elements

May formula

Feature

Presence conditions

Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

Software

Transcript of Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 Keynote)

ECMFA 2015 MoNoGe metamodel extension

On the Combination of Domain Specific Modeling Languages Antonio Vallecillo GISUM/Atenea Research Group ECMFA, Paris, June 2010.

Michalis Famelis, Rick Salay , Alessio Di Sandro , Marsha Chechik University of Toronto

A common meta-model for the interoperation of tools with heterogeneous data models ECMFA 2010 Third Workshop on Model-Driven Tool & Process Integration.

MADES - ECMFA 2012

ECMFA 2016 slides

ENGINEERING OF SOFTWARE FACTORIES FOR HIGH TECH …€¦ · engineering of software factories for high tech industry –experience and challenges ivan kurtev ecmfa 2019

Explicating dynamic capabilities: the nature and ...pages.stern.nyu.edu/~eyoon/seminar/CrossDisciplinary/...Explicating Dynamic Capabilities: Nature and Microfoundations must the innovating

Yu SunUniversity of Alabama at Birmingham PAR Works Jeff Gray University of Alabama ECMFA @ Montpellier, France July 3rd, 2013 This research is supported.

Explicating the Concept of Journalist: How Scholars, Legal Experts

Explicating the Characteristics of STEM Teaching and ...

Explicating Open Innovation

Explicating with Comics and Concept Maps

Explicating Jean Paul Robert Schumann's Program for Papillons, Op. 2

1 MODEL PATTERNS (WHAT STANDS IN THE WAY OF MORE WIDESPREAD ADOPTION OF MODEL-DRIVEN ENGINEERING?) Arend Rensink, University of Twente ECMFA Keynote, 22.

Explicating Web Site Interactivity

Explicating Dynamic Capabilities: The Nature and ...sjbae.pbworks.com/w/file/fetch/93336378/Teece (2007). Explicating... · environment is open to international commerce and fully

Explicating Ethical Corporate Marketing. Insights from the BP … · 2014-11-01 · 1 Explicating Ethical Corporate Marketing. Insights from the BP Deepwater Horizon Catastrophe:

Local one class optimization Gal Chechik, Stanford joint work with Koby Crammer, Hebrew university of Jerusalem.

Playing the Ocarina Across Cultures: Explicating Ludo-Narrative Experience in The Legend of Zelda: Ocarina of Time