Post on 01-Apr-2015
Teaching an Advanced Simulation Topic
Verification and Validation of Simulation Models
Stewart Robinson
School of Business and Economics
WSC 12, Berlin
Develop an understanding of the concepts of verification, validation and confidence in a model
Understanding some of the methods that can be used in V&V
Session Aim
Aimed at: Specialists: undergraduate and graduate students on
a simulation course; industrial training in simulation Management students: e.g. MBA
Session Outline
Define V&V V&V in the modelling life-cycle Difficulties in performing V&V Impossibility of validating a model! (Techniques of V&V) Role-play illustrating V&V
Verification: The model design (conceptual model) has been satisfactorily converted into a computer model
Validation: The model is sufficiently accurate for the purpose at hand
Verification and Validation
V&V in the Modelling ProcessReal world(problem)
Solutions/understanding
Conceptualmodel
Computermodel
Conceptual
modelling
Mod
el co
ding
Experimentation
Impl
emen
tatio
nSo
lutio
nva
lidat
ion
Experimental
validation
Conceptual
model validation
Verif
icat
ion
Bla
ck-b
ox
Whi
te-b
ox
vali
dati
on
vali
dati
on
Datavalidation
Conceptual Model Validation: determining that the content, assumptions and simplifications of the proposed model are sufficiently accurate for the purpose at hand. Data Validation: determining that the contextual data and the data required for model realisation and validation are sufficiently accurate for the purpose at hand. White-Box Validation: determining that the constituent parts of the computer model represent the corresponding real world elements with sufficient accuracy for the purpose at hand. Black-Box Validation: determining that the overall model represents the real world with sufficient accuracy for the purpose at hand. Experimentation Validation: determining that the experimental procedures adopted are providing results that are sufficiently accurate for the purpose at hand. Solution Validation: determining that the results obtained from the model of the proposed solution are sufficiently accurate for the purpose at hand.
Implications for V&V
Verification and Validation needs to be performed continuously throughout the modelling process.
Key point
Since the modelling process is iterative in nature, so too verification and validation need to be iterated and reiterated from the point of model conception to the implementation of the results.
Difficulties in Performing V&V
1. There is no such thing as general validity
2. There may be no real world to compare against
3. Which real world?
4. Often the real world data are inaccurate
5. There is not enough time
Implications for V&V
It is impossible to validate a model!
Model validation is a process of increasing confidencein a model – to the point where there is a willingness to use it for decision-making.
When validating a model the aim is to demonstrate thatthe model is in fact invalid. The more tests that can beperformed in which it cannot be proved that a model isinvalid, the greater the confidence that can be placed inthat model.
Key points
Natland BankNatland Bank: Planning a New Bank Branch
Question: How many ATMs are required (95% of customers queue for less than 3 minutes)?
ATM 1
ATM 2
QueueCustomers
(Arrival rate)
Service time
Simplifications: 1. No breakdowns of ATMs 2. No customers balk or leave
Proposed model
Natland Bank: Confidence Check
Conceptual Model Validation
High
Medium
Low
Natland Bank: Data
Time of day Average number of arrivals
9:00-10:00 110
10:00-11:00 95
11:00-12:00 140
12:00-13:00 165
13:00-14:00 205
14:00-15:00 145
15:00-16:00 160
16:00-17:00 190
Customer Arrivals
Natland Bank: Data
Service typeService time
(seconds) % of customers
C 30 40
B 20 10
T 30 8
C, B 60 25
C, T 45 10
B, T 40 2
C, B, T 75 5
Service Time
Natland Bank: Confidence Check
Data Validation
High
Medium
Low
Natland Bank
White-Box Validation (also performed in verification)
Watch the model animation: face validation Inspect the model code: correct entry of data Extreme value testing: very high service time
Natland Bank: Confidence Check
White-Box Validation
High
Medium
Low
Black-Box ValidationComparison with the real system
Real systemIR OR
SimulationmodelIS OS
H0: If IS =IR then O S O
R
Black-Box ValidationComparison with other models
Alternativemodel
IA OA
Simulation modelIS OS
H0: If IS =IA then O S O
A
Black-Box Validation
Comparison with other modelsA
ccur
acy
deri
ved
from
com
plex
ity
Simulation
Alternative model
Extreme approach is to make the simulation deterministic
Natland BankBlack-Box Validation: Comparison with Another (Simpler) Model
Deterministic model comparison:Arrival rate = 100/hour2 tellers: service time = 1 minuteCustomers served/hour = 60 x 2 = 120
Expected teller utilisation = 100/120 = 83.3%
Natland BankBlack-Box Validation: Comparison with Another (Simpler) Model
Full model comparison:Mean arrival rate = 157.14/hour2 tellers: mean service time = 40.45 secondsMean customers served/hour = 89.00 x 2 = 178.00
Expected teller utilisation = 157.14/178.00 = 88.28%
Natland Bank: Confidence Check
Black-Box Validation
High
Medium
Low
Will you use my model to determine the number of
ATMs in the bank?