Evaluation of Measurement Uncertainty

using Adaptive Monte Carlo Methods

Gerd Wübbeler1, Peter M. Harris2, Maurice G. Cox2, Clemens Elster1

1) Physikalisch-Technische Bundesanstalt (PTB)2) National Physical Laboratory (NPL)

Emerging Topics in Mathematics for Metrology – From Measurement Uncertainty to Metrology of Complex Systems

Physikalisch-Technische Bundesanstalt (PTB)

21-22 June 2010, Berlin, Germany

2

Content

� Evaluation of measurement uncertainty according to GUM S1

� GUM S1 adaptive Monte Carlo scheme

� Alternative approach: Stein’s Two-stage scheme

3

GUM Supplement 1 (GUM S1)

� PDF based method

� Numerical evaluation by a Monte Carlo Method (MCM)

MessgrößeSchätzwert(Messergebnis)

UnsicherheitPDF

Measured data

Further information

Probability density function (PDF)

probabilitydensity

Standarduncertainty

Estimate Measurand

4

Propagation of distributions

PDFs for input quantities PDF for measurandMeasurement model

Change-of-variables

5

GUM S1 Monte Carlo Method (MCM)

Model

),,( 1 NXXfY K=PDF of input quantities

( )NXX Ng ξξ ,,1,,1

K

K

random draw from

evaluation of measurement model ),,( 1 Nf ξξη K=

( )ηYgrandom sample from

( )Nξξ ,,1 K( )NXX N

g ξξ ,,1,,1K

K

η

many repetitions ���� PDF ( )ηYg

[ ] ξξξX d)()()( ∫ −= fggY ηδη

6

321 XXXY =

Illustration

trials

7

)1,0(~ NX i

Convergence

Law of large numbers

8

MCM results exhibit random variations

)1,0(~ NX i

Repetition of the MCM calculation

9

GUM S1 Adaptive Monte Carlo scheme (7.9)

Goal Estimation of the expectation with accuracy with a

coverage probability of about 95 %.

y δ

� Sequential batch-processing mode (e.g. 10 000 trials per batch)

iy� mean of the trials within batch i

iy� for sufficiently large batch size Gaussian distributed (central limit theorem)

),,( 1 hyy K

∑=

−−

=h

iiy hyy

hhs

1

22 ))((1

1)(

∑=

=h

iiy

hhy

1

1)(

10

Start: Batch 1 and 2

Stopping-rule yes

)(new batch hy

δ≤⋅

h

hsy )(2

no

21, yy

2=h

1+= hh

)(ˆ hyy =

GUM S1 Adaptive Monte Carlo scheme (7.9)

11

Assessment of the adaptive schemes

-1 0 1 2 3 4 5 6 70

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

0.5

Y

prob

abili

ty d

ensi

ty

22

21 XXY +=

121 == xx

1)()( 21 == xuxu

Model

Estimates

UncertaintiesGaussian distributions (uncorrelated)

y = 1.812 9

u(y) = 0.844 6

Rice distribution

12

Goal Determination of the expectation value of the Rice

distribution with an accuracy of δ = 0.005 (95 %)

Assessment Adaptive scheme is repeatedly executed 100 000 times

Result per run

� Estimate of the expectation value

� Number of required batches

Assessment of the adaptive schemes

13

Assessment of the GUM S1 adaptive scheme

Distribution of 10 5 estimates

],[ δδ +− yy

Only 80 % of the results found in the specified accuracy interval

95 % Interval

y

y

(Batch size 10 4, δδδδ=0.005)

14

Distribution of the number of batches

Large number of early

terminations at h = 2

batches (about 32 %)

Assessment of the GUM S1 adaptive scheme

(Batch size 10 4, δδδδ=0.005)

15

Reasons for the behavior of GUM S1 adaptive scheme

Sequential estimation scheme

� Random sample size (random number of batches)

� Multiple (dependent) testing for termination of sampling

Resulting confidence level does not necessarily mee t

the confidence level applied in individual test

� Confidence level of GUM S1 stopping rule adequate for fixed sample size

� Multiple testing and random sample size not taken into account

16

Alternative Adaptive Monte-Carlo scheme

Goal Carry out the MCM until a prescribed accuracy is achieved

at a specified confidence level

(with lowest possible numerical effort)

17

Given i.i.d. from unknown

Goal

so that is a confidence interval

for µ at confidence level 1-α

Stein‘s Two-stage scheme

∑=

=h

iiy

hhyh

1

1)( and

])(,)([ δδ +− hyhy

Step 1

Make random draws

�Variance

11 >h

∑=

−−

=1

1

21

11

2 ))((1

1)(

h

iiy hyy

hhs

1,,1 hyy K

K,, 21 yy 22 ,),,( σµσµN

18

Two-stage scheme

Step 2

Number of additionally required drawings

� make further random draws

� no additional random draws are made

( )

+−

⋅= −− 0,1

)(max 12

22/1,11

2

21 h

thsh hy

δα

02 >h 2h

02 =h

αδµδ −≥++≤≤−+ 1))()(Pr( 2121 hhyhhy

Proof C Stein, 1945, Ann. Math. Statist. 16 243-58

)( 21 hhy +

2h

19

Application of the two-stage scheme within GUM S1

Sequential batch-processing mode

Mean of the trials in batch i

Variance of the trials in batch i

sufficiently large batch size (CLT) ���� ,

)(2 yu i

unbiasediy

iy )(2 yu i

Two-stage scheme applicable for

� Estimate

� Squared uncertainty

y

)(2 yu

approximately Gaussian distributed

20

Assessment of the adaptive schemes

-1 0 1 2 3 4 5 6 70

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

0.5

Y

prob

abili

ty d

ensi

ty

22

21 XXY +=

121 == xx

1)()( 21 == xuxu

Model

Estimates

UncertaintiesGaussian distributions (uncorrelated)

y = 1.812 9

u(y) = 0.844 6

Rice distribution

21

Assessment of the two-stage scheme

y

(h1=10, Batch size 10 3, δδδδ=0.005, αααα=0.05)

y

],[ δδ +− yy

Interval covers95 % of the estimates �

Distribution of 10 5 estimates

22

Distribution of the number of batches

Assessment of the two-stage scheme

(h1=10, Batch size 10 3, δδδδ=0.005, αααα=0.05)

No earlyterminations

23

��

�

�� 157 271(196)0.969(0.001)10 000

146 383(218)0.951(0.001)1 000

146 423(218)0.952(0.001)100

146 176(218)0.951(0.001)10

145 802(211)0.952(0.001)1

ANT*Success rateBatch size

Determination of the estimate of the measurand: dependence on batch size

)05.0,005.0,10( 1 === αδh

*) ANT: Average Number of Trials

Result for a requested confidence level of 99.9 %

���� Success rate 0.999 08 (0.000 1)

ANT 654 760 (972)

1000) size Batch ,001.0( =α

�

Assessment of the two-stage scheme

24

Summary: Adaptive schemes for GUM S1

� GUM S1 adaptive scheme does not (intend to) meet a 95 % confidence level

� Alternative approach: Two-stage scheme

� Attains specified confidence level for a Gaussian d istribution (Proof by C. Stein, important for metrological appl ications)

� Applicable for the estimate and the squared uncerta inty

� Allows prediction of computation time (number of tr ials)

Wübbeler, Harris, Cox, Elster Metrologia 47 (2010) 317–324