ESSA 2013 MASS Workshop on Model Analysis Tools

Discussion - Matthias Meyer

Statistical and Behavioural Model Analysis Tools

Tamás Máhr, Richard Legéndi, László Gulyás

AITIA International, Inc. [email protected]

[email protected]@aitia.ai

Warsaw, 16th September 2013

mailto:[email protected]

mailto:[email protected]

Statistical and Behavioural Model Analysis Tools – Tamás Máhr, Richard Legéndi, László Gulyás

A short introduction to designing and executing computational experiments (i.e., simulations)

Show how to carry out DoE in practice (MEME)

Introduce the Participatory Extension (PET)

Aims of this talk

What are the practical challenges of designing and executing computational experiments?

How to implement and execute the DoE approach in practice?

How to enable human subjects to participate in simulations?


Dowload tools

http://bit.ly/essa2013

http://bit.ly/essa2013


Overview ABM research process


Overview ABM research process5


Model Exploration

(Lorscheid – Heine – Meyer)


The General Approach

Computer simulations are experimentsWhere the experimenter tries to determine

How the systems response (output) depends

On controllable factors (parameters)

One may also want to do replicates (cf. RNG seeds)

System(p1, p2, p3, p4, …) (r1, r2, r3, r4, …)


Model Exploration

(Lorscheid – Heine – Meyer)


Designing a simulation experiment (4)

Select a Factorial Design


Designing a simulation experiment (4)

Select a Factorial Design

Select a design (or fill in the Design Table)We will see a few further desings


Practical Steps of (6) Performing a Simulation Experiment

Set the parameters (factor values)Combinations or levels

WHAT to recordVariables (time series)Agent variables (changing length!)Derived values (statistics)

WHEN to record@end, @timestep, @N timesteps, @condition

WHEN to STOP the simulationFixed number of steps, condition reached, etc.

WHERE to executeLocal computer, local cluster, grid, cloud (comfort, pricing)


Practical Steps of (6) Performing a Simulation Experiment (cont'd)

Collect the resultsWhen using more than a single core/computer, result files end up dispersed

Assemble the result setThe ordering of the records (table rows) could be arbitraryThe number of columns may vary in raw output (e.g., when recording raw agent variables)

Often one also needs to pre-process the result setAggregating, Splitting / Slicing

Archive the experimentKeep a 'logbook' of your experimentsWhat results came from what experiment, when and with what settings


Model Exploration

How to implement and execute computational experiments?Practicalities

Advanced Designs (beyond factorials)Composite CentralBox-BehnkenLatin HyperCubes

„IntelliSweep“ toolsIterative methodsSelf-guided searches


Full Factorial Designs

Classic parameter sweeps „as we know them”


Full Factorial Design

A design in which every setting of every factor appears with every setting of every other factor

A specialized version of the “brute force” strategy

Determines the same number of values (“levels”) for each parameter (factor)


Fractional Factorial Designs

Full factorial designs may be demanding even with two levels only (k=10, 2k=1024)

A fractional factorial experiment is

in which only an adequately chosen fraction of the parameter combinations required for the complete factorial experiment is selected to be runTypically, we pick ¼, ½ of the full factorial


Example: 2-Level Fractional Factorial Experiments with MEME


Central Composite Design 1

The linear fit provided by the 2-level factorial methods may not be enough

To build quadratic, or other higher-order modelswe need new designs



A factorial design with added ‘star points’ on the axis of the parameters + a center point

The star points can make new extreme values for the parameters (both min and max)

The newly added points help to estimate the curvature



There are three different types of CCDs:

Circumscribed (CCC)Face centered (CCF)Inscribed(CCI)

CCC and CCI are rotatable designs, because every design point is at equal distance from the center

The variance of the predicted response of a model based on a rotatable design depends only on the distance from the center point


An Alternative Choice to Fit Quadratic Responses

The Box-Behnken designan independent quadratic design

Does not contain an embedded factorial or fractional factorial

Treatment combinations are at the midpoints of edges and at the center

A sphere that protrudes through each face


Properties of The Box-Behnken Design

Rotatable (or near rotatable) Requires 3 levels of each factor

Have limited capability for orthogonal blocking compared to the central composite designs


Goals and Details of the Box-Behnken Design

The design should be sufficient to fit a quadratic model

The ratio of the number of experimental points to the number of coefficients in the quadratic model should be reasonable

In fact, their designs kept it in the range of 1.5 to 2.6

The estimation variance Should more or less depend only on the distance from the

centre This is achieved exactly for the designs with 4 and 7 factors

Should not vary too much inside the smallest (hyper)cube containing the experimental points


Latin Hypercube Designs, 1

A screening method that

Easily handles more than 2 levels anduses much less runs than the factorial design

LHD designs operate on

A subset of the parameter space defined by a single contiguous interval for each dimension (parameter) – a hypercubeThe subset is defined by giving the low and high values for each tested factor (parameter)



A criterion: non-collapsing design

If one of the parameters has (almost) no influence, then two experiments that differ only in this parameter ‘collapse’

They are like measuring the same point twiceThis is a waste of the resources (in deterministic cases)

Therefore, two design points should not share any coordinate values

If it is not known a priori, which dimensions are important



Definition:

A d-dimensional grid of n levels in every dimensionEach level occurs only once

A non-collapsing design



A desirable property: Space fillingWhen no details on the functional behavior of the response parameters are available, it is important to obtain information from the entire design space

The points of the design should be ‘evenly spread’ over the entire hypercube



A MAXIMIN design is a set of points, such that

The separation distance is maximalI.e., the minimal distance among pairs of points

Assuming that the samples represent their ‘surroundings’ one wants to make sure that we use our sample points efficiently

We maximize the r common radius of spheres around the design points so that they don’t intersect

Any distance metric can be used, but L2 (Euclidean) is a common choice


A Note on MAXIMIN LHDs

Finding a MAXIMIN, non-collapsing design

for many dimensions and a high number of levels

is very hard

Therefore, often

pre-calculated designs are used, and/orthe MAXIMIN property is only approximated


Latin Hypercube Designs in MEME

The LHD plugin in MEME supports

Up to 100 levels andUp to 10 dimensions

Uses predefined experiment designs

Calculated by heuristic methods to approximate MAXIMIN LHD designs

http://www.spacefillingdesigns.nl/maximin/info.html


Latin Hypercube Designs in MEME


Dynamic “IntelliSweep” methods

So far, the entire design was fixed before starting the experiment

There was no feedback from the measured responses to the design

Various (optimization) methods exist that use a different strategy

Hill climbing, simulated annealing, genetic algorithms, etc.


Iterative Uniform Interpolation 1

IUI is a response analysis method

Refines the parameter domain between iterations to achieve better interpolation (of the response value)

Examines “interesting” subintervals by dividing them further

Deviation from the previously observed (assumed) gradient spans new measurements.


Iterative Uniform Interpolation 2


Genetic Algorithm Driven MethodsOptimization

Genetic algorithm (GA)

is a heuristic optimization method

F( o1, …, o

n ) → max

Can be directly used for response analysis

If we are not interested in the entire response surface,

but only in high/low response values


MEME The Model Exploration Module


MEME – The Model Exploration Module

Part of the Multi-Agent Simulation Suite (MASS)Yet, useful with most major agent-based platforms (Repast J, Repast Sym*, NetLogo, MASON, EMIL-S, FABLES)

GOAL:User firendly toolset for ABMHiding coding / implementational difficulties as much as possibleEase of use for non-technical people

MEME is responsible for DesignExecutionData collection

of computational experiments (i.e., agent-based simulations)


MEME – Functions

Assists the research process from the point when the implementation of the model (or a version of it) is complete until the publication of the collected results

Helps configuring the simulation to record the proper variables Data series from program variables or specific statistics of them

Offers wizards for a variety of experimental designs Including fractional factorials and more

Orchestrates the execution of the experiment on a single computer or on cluster or in the cloud

Collects the recorded data in standard data tables


MEME – Functions (cont'd)

Functions to preview the resultsperform exploratory analysis, preliminary charting

Export options and interfaces To standard popular statistical packages like R, SPS, STATA

Optionally, a personal 'laboratory logbook' Archiving and documenting the computational experiments performed by the modeler


MEME History (since 2005)

2005 Tool to administer and process simulation results (Repast J & FABLES)

2006 Setting up config files for parameter sweeps (Repast J)

2007 Distributed execution on local clusters and grids (QosCosGrid)

2007 Design of Experiment (Classic Designs)

2008 Multi-Platform Support (EMIL-S, NetLogo, Custom Java, Repast Sym*)

2008 Advanced statistics in recording

2009 Standard Interface for results processing

2009 Advanced DoE designs (freely extensible architecture)

2010 Intellisweep plugins (iterative, self-guiding exploration of param space)

2011 Execution „in the cloud” (http://modelexploration.aitia.ai/)

2012 Support for the MASON simulation package

2013 MEME goes open source


Design of Experiments in MEME

Classic simulation experiments with „parameter files“

Classic DoE tables

DoE Wizards FactorialsFractional FactorialsMore

IntelliSweep experimentsIterative methodsSelf-guided searching of the parameter spaceOptimization

E.g., Genetic Algorithms


Simulation in the cloud

model exploration servicehttp://modelexplortaion.aitia.ai

runs RepastJ, NetLogo, Java models (Mason support is comming soon)

uses Amazon EC2

user selects the number of machines used

http://www.spacefillingdesigns.nl/maximin/info.html


MEME as a Personal Laboratory Logbook

The more one gets immersed in (computational) experiments, the more results are filling the hard drive

„Nothing can be as alien/unknown than your own code – after two months”

Same applies to experimental results and settings, let alone charts„Hey, this is a nice chart. I wish I remembered what exact parameter settings I user to run the model with!”

Being disciplined always helps, but tools may help being disciplined.

MEME stores all result sets in a DB (together with settings)Grouped by model, version and „batch” (enforced)You can add comments, remarks, descriptions to them


Param Sweeps Param Sweeps Param Sweeps Param Sweeps

Results DB

Charts

Versioning and Merging

Filtering, Processing, Restructuring

Views

Export(Excell, SPSS, etc.)

Import(txt, csv, Excell, etc.)


Practical

Start MEME

Run Simulation...

Open the El Farol model

Select experimental design

Define output

Run experiment

Analyze results


The El Farol bar problem

Parameters:number of agentsovercrowding tresholdmemory sizenumber of strategies

Agents are researchers (N=100)

They visit a popular but small bar in Santa Fé

If attendance > 60 (overcrowded)Who hasn’t come

If attendance <= 60 Who hasn’t come

Each day agents decide individually and in the same time


The El Farol model

MASON implementation of a NetLogo variant

Artifical agents:ARMA-based prediction with history

Players have two actions: No go / go

If +1 Score!

Goal: get max score


Summary

Discussed the technical issues and challenges of executing computational experiments

Explained the challenges of applying the DoE approach in practice

Introduced MEME as a tool to assist from the point when the implementation of the simulation is complete

Discussed the usage of MEME

Together with advanced designs for computational experiments


PET The Participatory Extension


About PET

Participatory Extension (PET)Another component in the MASS toolboxhttp://pet.aitia.ai/

Converts ABM's to Web SimulationsParticipatory ExperimentsLaboratory experiments with human subjectsSome agents in the simulation are artificial, some others are controlled by human agents


Let's play!

http://demo1.aitia.ai


Model List

Create Experiments

Additional Info


Subject list(joined)

Admin tools

Admin page


In-experiment admin page

Player status (moved/waiting)


In-experiment subject page

Status messages


Post-experiment scores


PET software architecture

http://modelexplortaion.aitia.ai/


PET features

Runs Mason simulationsCan be scaled to run on multiple machinesCan be used:

LocallyLaboratory experimentsPolicy makers (scenario analysis with a proper model)

On any webserver to run constantlyGather data (scores from model and all user actions → replay)Dissemination

Questionnaire moduleVerify if subjects understood the rules

Software is already in use by Universiteit van Amsterdam


PET requirements

An ABM model

PET is a generic frameworkcurrently Mason models are supported


PET requirements

An ABM model

PET is a generic frameworkcurrently Mason models are supported

Development of a web interface

no restricitions on tools (HTML5, Javascript, GWT, …)communication is based on standards (AJAX)

config file maps incoming messages to method calls

GWT module available


PET communication

simple messagesinvoked by user events (button click) and at initializationgetters, setters, etc.

call model methodscall agent methods

action messagessimulation is stopped when an action method (decision point!) is hituser events can send action messages

public void updateAttendance()public void updateAttendance(final boolean attend)


PET communication

triggered messagesfrom simulation to browsercall model or agent methodconfigurable triggers

userActed (actionId)messageHandled (messageId)newTurn (actionId)simulationEnduserConnected (playerId, agentType)userDisconnected (playerId, agentType)unhandledExceptionCaught (exceptionType)


PET communication summary

broswer – simulation bidirectional communication

change / query the state of the model or your agent (simple message)

make the simulation move on (action message)

have the simulation send data asynchronously (triggered messages)


Practical

download and start Eclipse

install the right GWT plugin from marketplace!

import the project you downloaded

open the GUI class


Links to Software

http://meme.aitia.ai/

http://modelexploration.aitia.ai/

http://mass.aitia.ai

http://pet.aitia.ai/


Thank you!

Questions?

{tmahr|rlegendi|lgulyas}@aitia.ai

ESSA 2013 MASS Workshop on Model Analysis Tools

Technology

Transcript of ESSA 2013 MASS Workshop on Model Analysis Tools