Teaching an Advanced Simulation Topic

Verification and Validation of Simulation Models

Stewart Robinson

School of Business and Economics

WSC 12, Berlin

Develop an understanding of the concepts of verification, validation and confidence in a model

Understanding some of the methods that can be used in V&V

Session Aim

Aimed at: Specialists: undergraduate and graduate students on

a simulation course; industrial training in simulation Management students: e.g. MBA

Session Outline

Define V&V V&V in the modelling life-cycle Difficulties in performing V&V Impossibility of validating a model! (Techniques of V&V) Role-play illustrating V&V

Verification: The model design (conceptual model) has been satisfactorily converted into a computer model

Validation: The model is sufficiently accurate for the purpose at hand

Verification and Validation

V&V in the Modelling ProcessReal world(problem)

Solutions/understanding

Conceptualmodel

Computermodel

Conceptual

modelling

Mod

el co

ding

Experimentation

Impl

emen

tatio

nSo

lutio

nva

lidat

ion

Experimental

validation

Conceptual

model validation

Verif

icat

ion

Bla

ck-b

ox

Whi

te-b

ox

vali

dati

on

vali

dati

on

Datavalidation

Conceptual Model Validation: determining that the content, assumptions and simplifications of the proposed model are sufficiently accurate for the purpose at hand. Data Validation: determining that the contextual data and the data required for model realisation and validation are sufficiently accurate for the purpose at hand. White-Box Validation: determining that the constituent parts of the computer model represent the corresponding real world elements with sufficient accuracy for the purpose at hand. Black-Box Validation: determining that the overall model represents the real world with sufficient accuracy for the purpose at hand. Experimentation Validation: determining that the experimental procedures adopted are providing results that are sufficiently accurate for the purpose at hand. Solution Validation: determining that the results obtained from the model of the proposed solution are sufficiently accurate for the purpose at hand.

Implications for V&V

Verification and Validation needs to be performed continuously throughout the modelling process.

Key point

Since the modelling process is iterative in nature, so too verification and validation need to be iterated and reiterated from the point of model conception to the implementation of the results.

Difficulties in Performing V&V

1. There is no such thing as general validity

2. There may be no real world to compare against

3. Which real world?

4. Often the real world data are inaccurate

5. There is not enough time

Implications for V&V

It is impossible to validate a model!

Model validation is a process of increasing confidencein a model – to the point where there is a willingness to use it for decision-making.

When validating a model the aim is to demonstrate thatthe model is in fact invalid. The more tests that can beperformed in which it cannot be proved that a model isinvalid, the greater the confidence that can be placed inthat model.

Key points

Natland BankNatland Bank: Planning a New Bank Branch

Question: How many ATMs are required (95% of customers queue for less than 3 minutes)?

ATM 1

ATM 2

QueueCustomers

(Arrival rate)

Service time

Simplifications: 1. No breakdowns of ATMs 2. No customers balk or leave

Proposed model

Natland Bank: Confidence Check

Conceptual Model Validation

High

Medium

Low

Natland Bank: Data

Time of day Average number of arrivals

9:00-10:00 110

10:00-11:00 95

11:00-12:00 140

12:00-13:00 165

13:00-14:00 205

14:00-15:00 145

15:00-16:00 160

16:00-17:00 190

Customer Arrivals

Natland Bank: Data

Service typeService time

(seconds) % of customers

C 30 40

B 20 10

T 30 8

C, B 60 25

C, T 45 10

B, T 40 2

C, B, T 75 5

Service Time

Natland Bank: Confidence Check

Data Validation

High

Medium

Low

Natland Bank

White-Box Validation (also performed in verification)

Watch the model animation: face validation Inspect the model code: correct entry of data Extreme value testing: very high service time

Natland Bank: Confidence Check

White-Box Validation

High

Medium

Low

Black-Box ValidationComparison with the real system

Real systemIR OR

SimulationmodelIS OS

H0: If IS =IR then O S O

R

Black-Box ValidationComparison with other models

Alternativemodel

IA OA

Simulation modelIS OS

H0: If IS =IA then O S O

A

Black-Box Validation

Comparison with other modelsA

ccur

acy

deri

ved

from

com

plex

ity

Simulation

Alternative model

Extreme approach is to make the simulation deterministic

Natland BankBlack-Box Validation: Comparison with Another (Simpler) Model

Deterministic model comparison:Arrival rate = 100/hour2 tellers: service time = 1 minuteCustomers served/hour = 60 x 2 = 120

Expected teller utilisation = 100/120 = 83.3%

Natland BankBlack-Box Validation: Comparison with Another (Simpler) Model

Full model comparison:Mean arrival rate = 157.14/hour2 tellers: mean service time = 40.45 secondsMean customers served/hour = 89.00 x 2 = 178.00

Expected teller utilisation = 157.14/178.00 = 88.28%

Natland Bank: Confidence Check

Black-Box Validation

High

Medium

Low

Will you use my model to determine the number of

ATMs in the bank?