8/9/2019 Linard_1999-Quality Assurance in System Dynamics Modeling
1/16
1
Quality Assurance in System Dynamics Modelling
Keith LinardSenior Lecturer, School of Civil Engineering
University College UNSW (Australian defence Force Academy)
Email: keithlinard#@#yahoo.co.uk(Remove hashes to email)
SUMMARY
This paper addresses the practical issue of achieving quality assurance in system dynamicsmodelling projects. It first provides a brief overview of system dynamics, then suggests criteria
for determining contexts where system dynamics modelling solutions might be appropriate. The
main focus of the paper is the development of a structured methodology for a system dynamics
project. It summarises the key validation tests that should be undertaken in a typical project and
outlines how a consultant (external or in-house) team might be constituted.
IntroductionWhat is System Dynamics
System dynamics as a management discipline developed in the 1950s with its origins in
engineering control theory (servo-mechanisms and cybernetics), although underlying systems
concepts have been applied rigorously for the past century across most disciplines.
In a nutshell, System Dynamics is the rigorous study of organisational problems, from a holistic or
systemic perspective, using the principles of feedback, dynamics and simulation.
System Dynamics is a methodology for understanding complex problems where there is dynamic
behaviour (quantities changing over time) and where feedback impacts significantly on system
behaviour. It provides a framework and rules for qualitative description, exploration and analysisof systems in terms of their processes, information, boundaries and strategies, facilitating
quantitative simulation modelling and analysis for the design of system structure and control.
Computer simulation is central to the system dynamics discipline. Until 1987 the key software tool
available (Dynamo, a Fortran like language) required skilled programmers and was difficult for line
managers to use without significant support. This inhibited acceptance of the approach.
Powerful graphics software is now available for Macintosh and PC, which allows the modeller to
construct visual and symbolic representation of the system, facilitating both communication of
findings to management and knowledge capture from subject area experts.
8/9/2019 Linard_1999-Quality Assurance in System Dynamics Modeling
2/16
Quality Assurance in System Dynamics Modelling
2
Figure 1: Logical Relationships Developed Graphically
Obviously, models of complex problems require complex mathematics. Models of problems
involving change over time and feedback require the solving of multiple differential equations.
This new generation of graphically oriented software (e.g. the Powersim software used in these
simulations) automatically generates the structure of the nth
order differential equations necessary
for solving complex feedback problems, cutting development time dramatically and reducing the
likelihood of errors. Mathematical knowledge is still critical in fleshing out the interrelationships
between parameters, and it is still possible to build erroneous equations.
8/9/2019 Linard_1999-Quality Assurance in System Dynamics Modeling
3/16
Quality Assurance in System Dynamics Modelling
3
Figure 2: Structure of Equations Automatically Generated from Logic Map
System Behaviour
The essence of systems thinking is that structure influences behaviour or, put another way, when
placed in the same system, different people tend to produce similar results. Indeed, the fundamental
assumption of the federal public service reforms of the 1980s, the Financial Management
Improvement Program (FMIP), was predicated on the assumption that organisational breakdowns
and sub-optimal behaviour do not occur because of management stupidity, but rather are products
of the system. The FMIP reforms focused on modifying those aspects of the system, illustrated
in Figure 3, which encouraged, rewarded or enforced sub-optimal behaviour.
Figure 3: Systems in the public sector with potential to 'teach dysfunctional behaviour
Some Terminology
The fundamental concept in systems thinking is feedback. In systems thinking every influence is
both cause and effect and the key to seeing reality systemically is to see circles of influence
(dynamic thinking) instead of straight lines (linear thinking). By tracing these flows of influence
we often see patterns which repeat themselves, making situations better or worse.
Feedback Loops: The two main building blocks of all system representations are reinforcing andbalancing feedback loops. Reinforcing loops generate exponential growth (positive reinforcement)
and collapse (negative reinforcement) and, as a result, are often represented in systems diagrams by
the snowball effect. Balancing processes generate resistance, maintain stability and help achieve
equilibrium. A reinforcing loop will always, sooner or later, come up against a limiting or
balancing effect.
Delays: A delay is an interruption between an action and its consequences. Delayed feedback is
the key cause of the dynamic behaviour in systems.
Leverage: Sometimes small, well focused actions can produce significant, enduring improvements
- if they are carried out in the right place. This is referred to as leverage. The problem with
leverage is that it is often difficult to find. This is because it is usually not close in space and timeto the symptoms of the problem. As a result, in order to find high-leverage changes it is necessary
Legal, regulatory &budgetary environment
Management systems &
organisational architecture
Perceived standards& practices ('culture')
ACTIONS
8/9/2019 Linard_1999-Quality Assurance in System Dynamics Modeling
4/16
Quality Assurance in System Dynamics Modelling
4
to identify underlying structures in the system. Computer simulation is a key to identification of
leverage points.
As noted above, these concepts formed the foundation of the FMIP management reforms of the
1980s. Figure 4 illustrates, using very simplified causal loop diagrams, the rationale for moving to
user pays for inter-departmental services.
Figure 4: Causal Loop Diagram of 'User Pays' Dynamics
The left hand loop depicts the generic situation then extant across the public service in relation both
to the internal provision of free management services, such as typing, statistical services etc, and
to inter-departmental free services such as commonwealth cars, Foreign Affairs international
communication system, building services etc. The left-hand loop shows self-reinforcing behaviourin relation to these free services. Being free, more of the service is demanded than is really
warranted. This leads to congestion, creating an argument on the part of the supply organisation for
more resources. The increase in resources leads to temporary over capacity, so the agency markets
its improved level of service, leading to increased demand . . . and so on. A second dimension to
this loop, not shown, is that the service provider, being in a monopoly position, could largely
dictate the level and quality of service given.
An understanding of the feedback interrelationships pointed to possible leverage points. First,
sheeting home to the consumer the fact that services cost would (arguably) lead to better
assessment of the quantum of such services really required. Secondly, allowing the recipient to
retain the revenue from user charges would permit more appropriate determination of investment
priorities.
The user pays solution, right hand loop, introduced price as the lever which leads to two balancing
loops.
Contexts Where System Dynamics Modelling Solutions Might Be Appropriate
System dynamic is not appropriate to every problem context. System dynamics may be appropriate
if the problem has the following broad characteristics:
1. The issue is important to organisational objectives.
2. The problem is of an on-going nature rather than a one-time event.
3. The problem has a known history which can be described, both qualitatively and quantitatively.
SurplusCapacity
Demand
Price
CashreservesMaintenance &
expansion capacityDemand
'Congestion'
Pressure forBudget $
DELAY
Marketing ofsurplus capacity"free" of charge
$ forAdditionalcapacity
o
o
os
s
s
s
s
s
s
"User pays" gives genuine marketsignals for demand and provides
reserves for expansion
s
8/9/2019 Linard_1999-Quality Assurance in System Dynamics Modeling
5/16
Quality Assurance in System Dynamics Modelling
5
4. The problem exhibits time dynamics (e.g., time delay between some external trigger and theimpact on system demand, between changes in demand patterns and policy response or between
policy response and policy impact.)
5. Previous attempts have been made to address this problem, but results have been less effective
than desired.
But Why System Dynamics - What About Other Approaches
In some quarters there has been a tendency to set system dynamics modelling in opposition to other
forms of modelling (e.g., econometric modelling). This is a fruitless exercise.
What is important is that the appropriate paradigm, whether it be econometrics, operations research
or system dynamics, is used for a given problem. System dynamics can be misused as easily as
econometrics or any other modelling approach.
The circumstances where system dynamics may be appropriate were noted above. Using it outside
these criteria may be a costly mistake. On the other hand, many dynamic problem situations will
call for a combination of paradigms or time series analysis might be used to identify mathematicalrelationships between some parameters, whilst system dynamics modelling might be implemented
simply for the critical feedback relationships.
In respect of preparedness modelling, the lack of convincing progress by other methodologies, the
fact that all the criteria for use of system dynamics are met and the evidence of the prototypes
developed at ADFA provides a sound case for its application. A further important reason for the
use of the system dynamics modelling tools is their value in communication with stakeholders. In a
complex area such as this, user confidence is critical.
The work at ADFA, however, combines where appropriate supporting operations research and
statistical tools.
Choice of Simulation Software
There are four key graphically oriented system dynamics software packages available in the
marketplace, all of which are in use at ADFA:
Powersim (Norway)
Ithink / STELLA (US)
Vensim (US)
Cosmos /Cosmic (UK).
The Directorate of Army Research & Analysis evaluated the first three of these packages in 1993
and found that Powersim was significantly more useful, recommending that Army standardise on its
use. Developments since that time reinforce this conclusion.
The different packages have advantages and disadvantages reflecting the particular niches they aim
at. The following brief comments relate to their applicability to the Federal budget policy
environment. For a more detailed review reference to the DARA evaluation is suggested.
Cosmos / Cosmic and Vensim are much more geared to background technical use. The fact that
Armys powerful MRU (manpower required in uniform) model, built in Vensim, has fallen into
total disuse within 2 years of its creator leaving is a reflection of its lack of user friendliness.
8/9/2019 Linard_1999-Quality Assurance in System Dynamics Modeling
6/16
Quality Assurance in System Dynamics Modelling
6
Ithink / STELLA, whilst probably the most popular package in the educational context, suffers from
two fatal disadvantages. First, it is not fully networkable in a PC environment; secondly, its array
capability is very limited, leading to excessive complexity in modelling and model presentation.
Powersim has demonstrated its ability to handle the complex modelling. Associated software now
permits publishing of models directly on intranets or the internet.
Figure 5: "Spreadsheet" versus System Dynamics Models
Traditional computer models, whether built in spreadsheets or high level computer languages,tend to focus on the mathematical correlation, hiding the underlying conceptual framework. This
creates problems in validating the model logic with subject area experts and in communicating to
decision makers the reason for behavioural patterns. Statistical correlation, of course, is notsynonymous with causality
100 0.3 30
10000.25 250
500 0.4 200
75 0.4 30
1675
A1 0.3 A1*B2
A2 0.25 A2*B2
5*A1 0.4 A3*B3A1-25 0.4 A4*B4
Sum(x) Sum(y)
Sum(z)100 0.3 30
1000 0.25 250
500 0.4 200
75 0.4 30
1675 510
2185
?
Spreadsheet models:assume uni-directional linear causality
emphasise numerical inputs and outputs
logical relationships between numbersare hidden and difficult to follow
conceptual framework is obscureNumbers
Relationships
Conceptual framework
System Dynamics Modelling starts with the conceptual framework, particularly the feedback
relationships. The logic is mapped diagrammatically, facilitating communication with both
subject area expert and decision makers. The logic map automatically generates the structure of
the underlying mathematics. The tools facilitate deeper level learning.
System Dynamic models:address delayed feedback causality
emphasise meaning and relationships
conceptual framework is 'mapped'
logic is developed & displayeddiagramatically
numbers are kept in background &
are readily called upon
Numbers
Relationships
ConceptualFramework
8/9/2019 Linard_1999-Quality Assurance in System Dynamics Modeling
7/16
Quality Assurance in System Dynamics Modelling
7
System Dynamics is More than Software Simulation
Over the past three decades System Dynamics has been applied to such areas as project
management, business development, government policy analysis, environmental change, economic
development, military strategic and tactical analysis. They have been applied in multi-million andmulti-billion dollar court cases, where they have been developed to withstand scrutiny from
hostile expert witnesses. Especially since the mid 1980s there has been a corresponding growth
in the sophistication of tools and methodologies being developed and applied including computer
simulation tools, soft systems methodology, causal loop diagramming, chaos theory, statistical
analysis and interactive learning environments.
8/9/2019 Linard_1999-Quality Assurance in System Dynamics Modeling
8/16
Quality Assurance in System Dynamics Modelling
8
The diagram below illustrates the tools typically used in the discipline as one moves from
qualitative conceptualisation of the problem to development of rigorous forensic models and
management decision tools.
Figure 6: A palette of systems thinking tools
8/9/2019 Linard_1999-Quality Assurance in System Dynamics Modeling
9/16
Quality Assurance in System Dynamics Modelling
9
System Dynamics Modelling Development Methodology
System Dynamics and Traditional IT Methodologies
System dynamics modelling differs dramatically from traditional computer application
development, and hence the diverse computer business systems frameworks (including standardssuch as AS 3563 (Software Quality Management System) and proprietary methodologies such as
SSADM, JSD, IBM Business Systems Planning, James Martin Information Engineering etc) are
not directly applicable, although elements of these certainly have application.
System dynamics modelling generally takes place in a consulting environment where the client
recognises that there is a problem and the purpose of the modelling is to assist development of an
understanding of the problem context. Accordingly, scope definition, model specification and
model building tend to be an iterative or cyclical process, with mutual learning between the
modeller and the client.
As yet no SDM development methodology has gained widespread acceptance. The following are
the key texts which address the area to some extent:Checkland, P. and J. Scholes: Soft Systems Methodology in Action. Wiley, Chichester,
1990.
Wolstenholme, E: System Enquiry - A System Dynamics Approach. Wiley, Chichester,
1990.
Wolstenholme, E., S. Henderson & A. Gavine: The Evaluation of Management
Information Systems - A Dynamic and Holistic Approach. Wiles, Chichester, 1993.
Richardson, G. and A. Pugh: Introduction to System Dynamics Modelling. Productivity
Press, Portland, 1981.
Roberts, N. et al: Introduction to Computer Simulation - A System Dynamics Modelling
Approach. Productivity Press, Portland, 1994.
In the absence of any formal methodology, the following framework has been developed through
the writers experience as an amalgam of traditional IS planning methodologies, the Lotus
Accelerated Value Method, the Soft Systems Methodology and approaches suggested by the
system dynamics community. It is consistent with the corporate IS strategic planning guidelines
developed by the Federal Department of Finance. The process iterates through 7 broad stages
Figure 7: Seven Step System Dynamics Modelling Methodology
Stage Model Focus Client Focus
Stage 1:
Project Planning
Tools include:
text & flow charts CPM & GANTT charts budget templates risk templates
Outcome objectives for the
modelling project
Project scoping
deliverables timeframe budget skills required risk assessment team specification
Confirm scope and deliverables with
client
clarify clients understanding ofsystem dynamics
seek realistic expectations frommodelling
8/9/2019 Linard_1999-Quality Assurance in System Dynamics Modeling
10/16
Quality Assurance in System Dynamics Modelling
10
Stage 2:
Problem Conceptualisation
Tools include:
text & graphs wire diagrams causal loop diagrams influence diagrams concept mapping SSM rich pictures hexagons cognitive mapping social surveys statistical data
past review reports
State problem contexts, symptoms
and patterns of behaviour over time,
and past solutions
Identify basic organisation structures core business processes optimisation objective functions
(outcome performance measures)
patterns of resource behaviourover time
system boundaries time horizon of study
Identify feedback relationships
key resource states key resource flows
key delays key interrelation-ships
Restate problem, e.g. using SSM
VOCATE framework (Vision,
Owners, Clients, Actors, Transform-
ation processes, Environment)
Confirm understanding of business
with client
Confirm understanding of the
problem with the client
Confirm organisation performance
measures with the client
Stage 3:
Model Formulation
Tools include:
System dynamics software
Output graphs & tables from
the SYSTEM DYNAMICS
model(s)
Initial Prototype(s)
MAP - MODEL - SIMULATE -
VALIDATE - REITERATE
High level system map: Basic
single dimension stock-flow model
of key business processes
20 - 40 variables key stocks (resources) & flows key delays key auxiliaries key targets / goals / performance
indicator(s)
key information or materialfeedbacks
key delaysRun simulation - validate
Where there are a variety of core
processes operating in the
organisation, each of these may need
to be developed as independent sub-
models
Confirm basic logical structure and
model functioning with client
Confirm key variables
Confirm business rules
8/9/2019 Linard_1999-Quality Assurance in System Dynamics Modeling
11/16
Quality Assurance in System Dynamics Modelling
11
Stage 4:
Model Development
Tools include:
System dynamics software
Detailed Prototype
MAP - MODEL - SIMULATE -
VALIDATE -REITERATE
Iteratively elaborate model,challenging
system boundaries stocks, flows, converters complexity / simplicity in
representing business rules
Introduce multi-dimensionalarrays where applicable
Identify & build key policy levers& reports
variables under control of
decision makers output reports of relevance to
decision makers
Confirm basic structure and logic
with subject area experts
Confirm key variables with subject
area experts
Confirm business rules with subject
area experts
Stage 5:
Model Validation Quality Assurance
Undertake validation and verification
tests outlined in Figure 8.
Iteratively revise model
Confirm model outputs with subject
area experts
Independent testing
Stage 6:
Model Handover Installation & Training Installation & Training
Stage 7:
Model in use Experience in use of model identifies
need for fine-tuning.
Significant iteration can be expected between stages 3 - 5. Periodically, issues settled in stage 2
may need to be revisited, especially the key performance indicators.
Details of the actions to be undertaken in stages 2 to 5 are addressed in some detail by Richardson
and Pugh (1980), Wolstenholme et al (1993) and Roberts et al (1994).
8/9/2019 Linard_1999-Quality Assurance in System Dynamics Modeling
12/16
Quality Assurance in System Dynamics Modelling
12
Validation and Verification of System Dynamics Models
Need for Validation
All models are a simplification of the real world. To that extent all models are wrong. What is
important is that the models are useful, that they capture the key elements of real worldbehaviour, so that the patterns of behaviour, the direction of change and the magnitude of change
predicted by the model under specified conditions broadly mimic the expected real world
behaviour. Achieving these, however, is not sufficient. There is an additional, critical, dimension
to usefulness. The clients or users of the model must have confidence in the model.
Validation and verification testing are designed to ascertain whether a model is useful or merely
wrong. Procedures for testing or validating system dynamics models are discussed in a wide
range of literature, including:
Barlas, Y., 1989. Multiple Tests for Validation of System Dynamics Type of Simulation
Models, European Journal of Operational Research, Vol. 42, No. 1, pp. 59-87.
Barlas, Y., 1994. Model Validation in System Dynamics, Proceedings of the 1994International System Dynamics Conference, Methodological Issues Vol. 1, pp. 1-10,
Stirling, Scotland.
Barlas, Y., 1996. Formal aspects of model validity and validation in system dynamics.
System Dynamics Review, Vol 12, No 3. pp. 183-210.
Bell, J.A. and P.M. Senge. 1980. Methods for Enhancing Refutability in System
Dynamics Modelling. TIMS Studies in the Management Sciences, 14 (1), 61-73.
Forrester JW, Senge P, Tests for building confidence in system dynamics models, in A.
Legasto et al. (eds.), TIMS Studies in the Management Sciences (System Dynamics),
North-Holland, The Netherlands, 1980, pp. 209- 228
Graham, A.K. 1980. Parameter Estimation in System Dynamics Modelling. In J. Randers
(Ed.), Elements of the System Dynamic Method. (pp. 143-161). Cambridge, MA:
Productivity Press.
Grcic, B and Munitic, A., 1997. System Dynamics Approach to Validation, Proceedings of
the 1997 International System Dynamics Conference, Istanbul, Turkey.
Peterson, D and Eberline, R, 1994. Reality Check: a bridge between systems thinking and
system dynamics. System Dynamics Review, Vol 10, Nos 2/3.
Richardson GP, Pugh A, Introduction to System Dynamics Modelling with DYNAMO, MIT
Press, Cambridge, MA, 1981
Sargent, R.G., Verification and Validation of Simulation Models, in System Dynamics,
North Holland Publishing Company, Amsterdam, 1980.
Sterman, J.D., A Skeptic's Guide to Computer Models, in Grant, L. et al., Foresight and
National Decisions, University Press of America, Boston, 1988.
Tank-Nielsen, C., Sensitivity Analysis in System Dynamics, in Elements of the System
Dynamics Method, J. Randers (ed), Productivity Press, Connecticut, 1980.
Figure 8 summarises from the above literature the key tests which would seem to be relevant to
assuring the client that the preparedness models are useful. These tests are elaborated below. Not
all these tests are relevant to every situation. However, at least the first six tests should normally beapplied, and the others as appropriate to the particular situation.
8/9/2019 Linard_1999-Quality Assurance in System Dynamics Modeling
13/16
Quality Assurance in System Dynamics Modelling
13
Figure 8: Validation and Verification Tests of Model Adequacy
Structure verification the model structure is consistent with relevant descriptive
knowledge of the system;
Parameter verification the parameters are consistent with relevant descriptive (and,
where available, numerical) knowledge of the system;
Boundary adequacy all important concepts for addressing the policy problem are
endogenous to (included in) the model;
Extreme conditions each equation makes sense, even when inputs take on extreme
values;
Dimensional consistency all equations are dimensionally consistent;
Behaviour reproduction the model generates behaviour modes, phasing, frequencies and
other characteristics of the behaviour of the real system;
Behaviour anomaly anomalous behaviour occurs under standard parameter values,
or anomalous behaviour arises if a key assumption is deleted
Behaviour sensitivity the model behaviour is appropriately sensitive to plausible
variations in input parameters;
Behaviour prediction the model plausibly describes the results of new policy.
Extreme policy the model behaves properly when subjected to extreme policies
or test inputs;
Statistical character the model output has the same statistical character as the
output of the real system;
Structure Verification Tests: Because the foundation for model behavior is the model's structure,
the first test in validating a model is whether the structure of the model matches the structure of the
system being modelled. Every element of the model should have a real-world counterpart, and
every important factor in the real system should be reflected in the model. Although this may seem
like a simple, obvious test, it may not be so. For example, descriptions of how all of the structural
parts of real systems are tied together rarely exist. More often than not, such descriptions must be
based on the concepts, or mental models, of people familiar with the system. Further, important
parts of some systems may lie unrecognized prior to modelling.
Parameter Verification Tests: Parameter values in a model often may be tested in a straightforward
manner, e.g., against historical data. However, even in technical defence systems, the available
data may be suspect. For example, equipment maintenance logs may reflect the total time the
equipment has been in the workshop, not the actual time it was being worked on. Also, in many
instances, there may be variables that are not usually quantified, but that are perceived to be critical
to the system being modelled. These elements must be included in the model. If productivity, for
example, is an important element in assessing optimal workforce structure, it must be included in
the model, and its relationship to other pertinent parts of the system must be specifiedquantitatively. Clearly if the model output is strongly sensitive to such qualitative inputs, further
rigorous analysis of those inputs may be required.
8/9/2019 Linard_1999-Quality Assurance in System Dynamics Modeling
14/16
Quality Assurance in System Dynamics Modelling
14
Boundary Adequacy Test: Model boundaries must match the purpose for which the model is
designed, if the model is to be used with confidence; that is, the model must include all of the
important factors affecting the behavior of interest. In practice, boundaries tend to shift as the
developers' and users' understanding of a problem evolves with the model's development. As model
purpose shifts, changes in the model's boundaries may be required. In many problems, a simple
model with limited boundaries may be expanded, or disaggregated, from time to time, as the modelis used to address problems in greater detail.
Extreme Conditions Test: Tests under extreme conditions, that is applying very high or very low
values to variables, often exposes structural faults or inadequacies and incomplete or erroneous
parameter relationships. The criterion is whether the pattern of system behaviour or values of
dependent variables remain valid (e.g. do not go negative if negativity is theoretically impossible)
or remain plausible. The ability of a model to function properly under extreme conditions
contributes to its utility as a policy evaluation tool as well as user confidence.
Dimensional Consistency Tests of model equations is an important structural test which simply
checks that the dimensions or units for the left hand side of an equation are identical with those on
the right hand side. Errors in dimensional consistency can easily creep into model equations during
model development and, subsequently, during revisions.
Behavior Reproduction Test: The tests relating to model behavior are less technical and, for many
users, more convincing than the structural tests. Foremost among these tests is the comparison of
model behavior with the behavior of the system being modeled. A model whose behavior has little,
or nothing, in common with that of the system of interest generates little, or no, confidence. Where
historical time series data are available, the model must be capable of producing similar data. In
this test, it is again important to keep in mind the purpose of the model -- including the time span of
the areas of behavior that are of interest. Where historical data are very poor or nonexistent, the test
may be one of reasonableness.
Behavior Anomaly Test: When model behavior does not replicate the behavior of the real system,
model structure, parameter values, boundaries, or similar factors must be considered suspect.
Something may have been omitted, improperly specified, or assigned incorrect values. Whether
due to faults in the model or in the real system, the resolution of the discrepancies found through
the anomalous behavior test bolsters confidence and validity.
Behavior Sensitivity Test: Most, but certainly not all, systems are stable. Small, reasonable
changes in a model's parameter values, should normally not produce radical behavior changes.
Typically these are introduced using the STEP, RAMP, PULSE or SINEWAVE functions of the
software. If the model's behavior is not seriously affected by plausible parameter variations,
confidence in the model is increased. On the other hand, dynamic simulation models are often used
to search for parameters that can effect behavior changes. The criterion in the sensitivity test is that
any sensitivity exhibited by the model should not only be plausible, but also consistent with
observed, or likely, behavior in the real system.
Behavior Prediction Test: A fundamental use of dynamic simulation models is predicting how a
system would behave if various policies of interest were implemented. Dynamic simulation models
offer significant advantages when used in this role; they provide a consistent basis for the
predictions. This basis is a consolidation of judgment, experience, and intuition that has been tested
against historical evidence. Confidence in the model is reinforced if the model not only replicates
long-term historical behavior, but also replicates the behaviour in existing systems where policy
changes have been implemented.
Extreme Policy Test: Similar to extreme conditions test, these tests introduce radical policies into
the model to see if the behavior of the model is consistent with what would be expected under these
8/9/2019 Linard_1999-Quality Assurance in System Dynamics Modeling
15/16
Quality Assurance in System Dynamics Modelling
15
conditions.. The criterion again is whether the pattern of system behaviour or values of dependent
variables remain valid or plausible.
Statistical Character Test: The behaviour reproduction test (q.v.) is somewhat qualitative. In some
circumstances it may prove desirable to test rigorously behaviour reproduction. This may be done
by modifying model structure to produce formal statistics on key model output variables which canbe compared with real data.
Consultant Team Role/Function Descriptions
How a consultant approaches and staffs a project will vary according to their particular mode of
operation and the particular staff available, their mix of skills and their competing duties. The
following is indicative only to assist in time / cost estimates.
Figure 9: Role / Function Descriptions
Role/Function Description of Responsibilities Skills/Experience/ Background
Required
Consultant
Project Manager
Responsible for contract administration
Ensures in-budget, on-time, on-quality
delivery of contracted items
Plans the program, allocates and directs staff
and other resources to accomplish tasks, and
maintains control over the program
Tracks project status and works to identify
and resolve road blocks
Ensures client satisfaction
Owns and builds long-term client
relationship
Provides independent review and assessment
to ensure that proper methods are followed:
work conforms to contract risks are discussed and mitigations
achieved
Action plans and targets arecommunicated to team
Results are communicated to client
Proven project management
expertise in defence contracts.
Proven expertise in managing IS
projects
Basic understanding of system
dynamics modelling.
Basic understanding of the
modelling domain (i.e.preparedness)
8/9/2019 Linard_1999-Quality Assurance in System Dynamics Modeling
16/16
Quality Assurance in System Dynamics Modelling
16
Principal system
dynamics analyst
Implements the detailed system dynamics
modelling methodology (steps 2 to 6 in
Figure 6:
problem conceptualisation model formulation model development model validation model handoverLiases directly with the clients subject area
experts to confirm the various stages of the
model development are valid
Technical expert in the 5 stages of
model development listed in Figure
6.
Technical expert in the use of the
modelling software
Technical expertise in the use of
ancillary software (spreadsheets,
statistical packages, database
systems etc)
Significant domain knowledge of
the area being modelled
Assistant system
dynamics analyst
Undertakes less complex aspects of the
modelling, including
background research
documentation of all model parametersand equations development of user help screens development of program menu and
navigation tools
development of user documentation development of draft project report
Intermediate level of expertise in the
5 stages of model development
listed in Figure 6.
Intermediate level of expertise in the
use of the modelling software
Basic domain knowledge of the area
being modelled
_____________________
Top Related