A general assistant tool A general assistant tool for the checking results for the checking results

from Monte Carlo simulationsfrom Monte Carlo simulations

A general assistant tool A general assistant tool for the checking results for the checking results

from Monte Carlo simulationsfrom Monte Carlo simulationsKoi, TatsumiKoi, TatsumiSLAC/SCCSSLAC/SCCS

Contents• Motivation• Precision and Accuracy• Central Limit Theorem• Testing Method• Current Status of Development• Summary

Motivation• After a Monte Carlo simulation, we get an answer. Ho

wever how to estimate quality of the answer.

What we must remember is• Large number of history does not valid result of simul

ation.• Small Relative Error does not valid result of simulatio

n

Motivation (Cont.)• To provide “statistical information to as

sist establishing valid confidence intervals for Monte Carlo results” for users, something like MCNPs did.

Subject of this study• Precision of the Monte Carlo

simulation• Accuracy of the result is NOT a

subject of this study

At first we have to define Precision and Accuracy of simulations

True Value Mote Carlo Results

AccuracyPrecision

Precision and Accuracy• Precision: Uncertainty caused by statistical

fluctuation• Accuracy: Difference between expected value

and true physical quantity.

Subject of this study (Cont.)

• Precision of the Monte Carlo simulation is subject of this study.

• To state accuracy of simulations, we should consider details of simulation, i.e., uncertainties of physical data, modeling of physical processes, variance reduction techniques and so on.

• To make a generalized tool, we have to limit subjects only for precision.

Accuracy is a subject for most of presentations in this workshop.

Principal of this study is

Central Limit Theorem

Central Limit Theorem• Every data which are influenced by

many small and unrelated random effects has normally distribution.

• The estimated mean will appear to be sampled from normal distribution with a KNOWN standard deviation when N approaches infinity.

N

Central Limit Theorem (Cont.)

• Therefore, We have to check that N have approached infinity in the sense of the CLT, or not.

• This corresponds to the checking the complete sampling of interested phase space has occurred, or not.

This is not a simple static test

butcheck of results from nature of Monte Carlo

simulations

Checking Values• Mean

• Variance and Standard Deviation

• Relative error

• Variance of Variance

11

2

2

N

xxS

N

ii

N

iixN

x1

1

N

SSwherex

SR xx

22

2

,

4

2

x

x

S

SSVOV

Checking Values (Cont.)

• Figure of Merit

• Scoring EfficiencyRintrinsic and Refficiency

• Shift

• SLOPEFit to the Largest history scores

TRFOM

2

1

NSxxSHIFT i23

2

N

historiesZERONONofnumberq

What we check?• Behavior of MEAN• Values of R• Time profile of R• Values of VOV• Time profile of VOV• Time profile of FOM• Behavior of FOM• Value of SLOPE• Value of SHIFT

• Effect of the largest history score occurs on the next history.– MEAN– R (Rintrinsic and Refficiency)

– VOV– FOM– SHIFT

Boolean Answer

Numeric Answer

Of cause, Boolean check is carried out

mathematically (statistically)

valuebehavior

time profile

For behaviors and time profiles check

• Derive Pearson’s r from data (results and theoretical values)– r=1(-1), perfect positive (negative)

correlation– r=0, uncorrelated

• null hypothesis is set to uncorrelated• Then, follows student t

distribution of degree of freedom • Checking significance of r with null hypothesis.• Rejection level of null hypothesis is 68.28% (1σ)

212 rNrt 2N

Example• Checking value: Observable Energy of Sampling Calorimeter.• Material

– Pb (Lead)-Scinitillator• Thickens

– Pb: 8.0 mm/layer, Sci: 2.0 mm/layer• Layers

– 120 layers– 1 m x 1 m – interaction surface

• Beam– Electon 4 GeV

• Range Cuts– 1 mm

Pb

8mm

2mm

Sci.

・・・・・・・・e-

SD

9.2

9.3

9.4

9.5

9.6

9.7

9.8

9.9

10

0 20 40 60 80 100 120

SD

Example 100 histories

mean

77

77.5

78

78.5

79

79.5

80

0 20 40 60 80 100 120

mean

R

00.0020.0040.0060.0080.01

0.0120.0140.0160.0180.02

0 20 40 60 80 100 120

R

VOV

0

0.005

0.01

0.015

0.02

0.025

0.03

0.035

0.04

0 20 40 60 80 100 120

vov

SD

VOV

MEAN

R

Does not pass most of Boolean tests

SD

99.29.49.69.810

10.210.410.610.8

11

0 200 400 600 800 1000 1200

SD

vov

00.00050.001

0.00150.002

0.00250.003

0.00350.004

0.00450.005

0 200 400 600 800 1000 1200

vov

R

0

0.001

0.002

0.003

0.004

0.005

0.006

0.007

0 200 400 600 800 1000 1200

R

mean

77

77.5

78

78.5

79

79.5

80

0 200 400 600 800 1000 1200

mean

Example 1,000 histories

SD

VOV

MEAN

R

Does not pass some of Boolean tests

vov

00.000050.0001

0.000150.0002

0.000250.0003

0.000350.0004

0.000450.0005

0 2000 4000 6000 8000 10000 12000

vov

R

00.00020.00040.00060.00080.001

0.00120.00140.00160.00180.002

0 2000 4000 6000 8000 10000 12000

R

SD

99.29.49.69.810

10.210.410.610.8

11

0 2000 4000 6000 8000 10000 12000

SD

mean

77

77.5

78

78.5

79

79.5

80

0 2000 4000 6000 8000 10000 12000

mean

Example 10,000 histories

SD

VOV

MEAN

R

Does not pass one of Boolean tests (SLOPE check)

How to apply Energy Spectrum estimation

etc.• Checking each

confidence level of P1, P2, P3, P4,,,,

• Of course, scoring efficiency becomes low.

P1

P2

P3

P4

E

V/E

Unfortunately, this tool does not work well with

some deterministic variance reduction

techniques.This is come from

limitation of CLT (means some variance are

required for distribution), so that we can not over

come.

And some simulations becomes deterministic

without awaking of users.Please check your

simulation carefully.

Current Status of Development

• Most part of developments has been done.

• Following items are remained under development.– Output of testing result– Class or function for minimization of

multi dimensional functions

We want to include this tool in Geant4

butwhat category is suite for this

tool?

Run, SD, Hits and its collections,

Tally??

Summary• We have successfully developed a gene

ral assistant tool for the checking the results from Monte Carlo simulations like MCNPs.

• Through this tool, users easily know the confidence intervals for Monte Carlo results.