Chapter 6 Statistical Analysis of Output from Terminating Simulations.

Chapter 6

Statistical Analysis of Output from Terminating Simulations

Chapter 5 – Detailed Modeling and Terminating Statistical Analysis

Simulation with Arena 2

Statistical Analysis of Output from Terminating Simulations

• Random input leads to random output (RIRO)• Run a simulation (once) — what does it mean?

– Was this run “typical” or not?– Variability from run to run (of the same model)?

• Need statistical analysis of output data– From a single model configuration– Compare two or more different configurations– Search for an optimal configuration

• Statistical analysis of output is often ignored– This is a big mistake – no idea of precision of results– Not hard or time-consuming to do this – it just takes a little planning

and thought, then some (cheap) computer time


Simulation with Arena 3

Output Analysis

Output analysis is concerned with• Designing replications

Obtain most reliable info with minimum number of replications and minimum run length.

• Computing statisticsPoint and confidence interval estimationSize and independency issues

• Presenting them in a textual and graphical format.Aim is to understand the system behavior and

generate predictions for it!


4Simulation with Arena

Time Frame of Simulations

• Terminating: Specific starting, stopping conditions– Run length will be well-defined (and finite)

• Steady-state: Long-run (technically forever)– Theoretically, initial conditions don’t matter (but

practically they usually do)– Not clear how to terminate a simulation run

• This is really a question of intent of the study• Has major impact on how output analysis is done• Sometimes it’s not clear which is appropriate

Asli Sencer 5

Model 6.1• Same as Model 5.3

Number of trunk lines=26No additional staff during 5-8 hrs.

• 10 runs are made

For terminating case, make IID replications Run>Setup>Replication Parameters:

Number of Replications =10

Check both boxes for Initialize Between Replications• Outputs are saved to .dat files Statistics Module, Type=output, Data file name= Filename.dat

Asli Sencer 6

Outputs of Model 6.1• Category Overview report will have some statistical-

analysis results of the output across the replications

Asli Sencer 7

Output Precision in Model 6.1This information (except standard deviation) is in Category Overview report

If > 1 replication specified, Arena uses cross-replication data as aboveFor other confidence levels or graphics – Output Analyzer

8

Confidence Interval EstimationNotation:number of replicationssample meansample standard deviationcritical value for significance level

(1-α)x100% Confidence Interval (CI): for for ,

9Simulation with Arena Chapter 5 – Detailed Modeling and Terminating Statistical Analysis

Interpretation of a Confidence Interval

• A CI is an interval with random (data-dependent) endpoints that’s supposed to have stated probability (1-α) of containing, or covering, the expected values– “Target” expected value is a fixed, but unknown, number– Expected value = average of infinite number of replications

• A CI is not an interval that contains, say, 95% of the data generated by simulation outputs.– That’s a prediction interval … useful too, but different

• 95% CI means: If we make infinitely many batch runs of size and construct C.I. for α=5%, 95% of the time this interval will contain the true (but unknown) population parameter.

10

Required Number of Replications to Achieve a Certain Precision

• Half-width =• We prefer confidence intervals with smaller half width — for

higher precision• Question: How many replications are needed to get the

required (better) precision? We can’t control t or s. We may just increase n — how much?

– Trial and error (now)– Approximate number for acceptable precision (below)– Sequential sampling (Chapter 11)

Want this to be “small,” say< h where h is prespecified

Chapter 5 – Detailed Modeling and Terminating Statistical AnalysisSimulation with Arena



Half Width and Number of Replications

• Let required half-width, solve for• Not really solved for n (t, s depend on n)• Approximation: • Replace t by z, corresponding normal critical value

– Pretend that current s will hold for larger samples

– Get

• Easier but different approximation:

s = sample standarddeviation from “initial”number n0 of replications

h0 = half width from “initial” number n0 of replications

n grows quadraticallyas h decreases.

𝑛=𝑡❑2𝑛− 1,1−∝/2

𝑠2

h2

𝑛≅ 𝑧❑21−∝/2

𝑠2

h2

𝑛≅𝑛0h02

h2

Asli Sencer 12

Number of Replications Needed

• If we require h=$250 rather than $812 for total cost,

replications are needed.

𝑛≅𝑛0h02

h2=10 (812.76 )2

(250 )2=105.7

Asli Sencer 13

Model 6.2

• 110 Runs• 95% CI for the total cost= 22,175.19 +- 369.54, half width=369.54>250! Required accuracy is still not met, why? As more replications are made, in formula, does not change, increases; but , the sample standard deviation might increase. So might decrease.• 95% CI for the %rejected calls= 11.73.96 +- 0.51, accuracy is quite satisfactory!

Asli Sencer 14

Model 6.3

• 1000 Runs-as a trial• Save the output to Total cost.dat• Open Output analyzer as a separate application File>Data File>Export Export binary data in .dat file to a plain ASCII text file and save.• Open Arena Input Analyzer Plot the histogram of the Total Costs

Asli Sencer 15

Histogram of 1000 Total Cost Values

• Since Total Cost values is a sum, law of large numbers apply. We see that the distribution approaches normal as the number of replications increase!

• Same is true for average statistics due to central limit theorem.• It is not true for extreme value statistics like maximum or minimum.



Confidence Intervals (cont’d)

• Usual formulas assume normally-distributed data Never true in simulation Might be approximately true if output is an

average, rather than an extreme Central limit theorem Issues of robustness, coverage, precision –

details in book



Comparison of AlternativesStatistical Hypothesis Test

0

0

211

210

H

H

Reject Ho if is significantly large or small, i.e.,

performance of system 1 is significantly different than system 2!

Here: : Total Cost of Base Model (110 observations)

: Total Cost of Alternative Model (110 observations)

21 XX

1x

is the mean performance of system ii

2x

Asli Sencer 18

Comparing Two Scenarios

• Base Scenario: Model 6.4 (Same as in Model 5.3) -110 runs -26 Trunk Lines, No New Staff between 12:00-16:00• Alternative scenario: Model 6.4 (More-resources scenario) -110 runs -29 Trunk Lines, (Change the capacity from 26 to 29) -Hire three for each of Larry, Moe, Curly, Hermann and Sales Resources. (Change these variables from 0 to 3)• Tradeoff is between increased salary cost but decreased excess

waiting costs. Will the total costs decrease?• Percent Rejected calls will decrease, but how much?

Asli Sencer 19

Comparison of Scenarios• Runs both models for 110 times.• Statistics Data Module Save output files –BaseCase.dat or -MoreResources.dat.• 95% CI for total costs are Base model: 22,175.19 +- 369.54=[21,805, 22,544] Increased resources: 24,542.82 +- 329.11=[24,213, 24,871] Intervals do NOT overlap, hence Total Costs are significantly different at 5% significance level. • 95% CI for percent rejected are Base model: 11.74 +- 0.51=[11.23, 12.25] Increased resources: 1.73 +- 0.31=[1.42, 2.04] Intervals do NOT overlap, hence Percent Rejected are significantly different at 5% significance level.

Asli Sencer 20

Arena Output Analyzer

• Separate application in Arena• Operates in output files (.dat) generated by

Arena through the Statistics data module• Data in .dat file is in binary format to be opened

by Arena Output Analyzer only!• Provides confidence intervals on expected output

statistics as also appear in Arena output reports.• Provides statistical comparison of two scenarios,

and others.

Asli Sencer 21

Comparison of Scenarios with Arena Output Analyzer

• Open Output Analyzer• Select File>New to open a data group, i.e., list of .dat files• Add TotalCost-BaseCase.dat TotalCost-MoreResources.dat PercentRejected-BaseCase.dat PercentRejected-MoreResources.dat• Can save this data group as .dgr file to refer easily

afterwards• Analyze>Compare Means Add each pair of comparisons by choosing ‘lumped’ so that all 110 values are considered in the analysis

Asli Sencer 22

Hypothesis Tests

• Ho: Mean TC of base case = Mean TC of more resources Ha: Mean TC of base case ≠ Mean TC of more resources

• Ho: Mean % rejected of base case = Mean % rejected of more resources case

Ha: Mean % rejected of base case ≠ Mean % rejected of more resources case

Asli Sencer 23

Output Report-Compare Means

Confidence interval on difference misses 0, so conclude that there is a (statistically) significant difference between the base model and the alternative at α=5%

Asli Sencer 24

Evaluating Many Scenarios with Process Analyzer

• Separate application in Arena• Allows making multiple pairwise scenario comparisons

at a time.• PAN operates on Arena program files with .p

extension, generated when .doe model is run.• A PAN scenario includes a program file, a set of values

for the input controls (decision variables in the form of variables and resources), a set of output responses.

• A PAN project is a collection of such scenarios that can be saved by .pan extension for future reference.

Asli Sencer 25

Development of a PAN Project• Use Model 6.5 110 runs Output data files are deleted since they will be useless in PAN.• Open a PAN project File > New, File > Open • Add a new scenario. Double click on the raw Name=Base Case, Program File=Model 6.5.p Replications=110• Add contols Right click in this line OR Insert > Control Under Resources: The capacity of trunk line Under User Specified: New Tech 1, New Tech 2, New Tech 3, New Tech All, New Sales• Add responses Right click on this line OR Insert > Responses Under user specified: Total Cost, Percent Rejected

26

New Scenarios• Suppose you have $1360/week to spend on all additional

resources. To which of the six expandable resources should you allocate the new money?

• Then following 6 alternative scenarios apply in addition to Base Case.

13 more trunk lines ($98 each) 4 more tech 1,2,3 people ($320 each) 3 more tech all people ($360 each) 4 more sales people ( 340 each)• Run the scenarios Check the scenarios to run Run > Go OR play button OR F5 function key

Asli Sencer

Asli Sencer 27

PAN screen

Asli Sencer 28

Generating Reports for Multiple Comparison in PAN

• Insert > Chart OR right click on a response column. Chart type=Box whiskers Check Identify Best Scenarios box Select ‘smaller is the better’• Red boxes are significantly better than blue ones at 5% significance

level.• To decrease the half width of a scenario, increase the number of

replications of that specific one.• Error tolerance is a positive value that represents an amount small

enough that you don’t care if the selected scenarios are actually inferior to the true best one by at most this amount.

A positive error reduce the number of selected scenarios at the risk of being off by a little bit.

Asli Sencer 29

A PAN Report

Chapter 6 Statistical Analysis of Output from Terminating Simulations.

Documents

Transcript of Chapter 6 Statistical Analysis of Output from Terminating Simulations.