EU-Twinning Project RO 2006 IB EN 09 Bucharest, 03.02.2009Saxony-Anhalt Wolfgang Garche State...

EU-Twinning Project RO 2006 IB EN 09

Bucharest, 03.02.2009Saxony-Anhalt Wolfgang Garche State Environmental Protection Agency

1

Wolfgang Garche

Saxony-Anhalt Environmental Protection AgencyDepartment Air Quality Monitoring, Information, Assessment

[email protected]

Estimation of Measurement Uncertainties



2

Measurementerror

Systematicerror

Randomerror

Known systematic error

Unknown systematic error

Correction Residual error

Measurement result Measurement uncertainty

(Source: EUROLAB Technical Report 1/2006 „Guide to the Evaluation of Measurement Uncertanty for Qualitative Test Results“)

Types of measurement error and their consideration in determining the result of a measurement and the associated uncertainty



3

Measured values for simultaneous occurring of random and systematic errors

(Source: EUROLAB Technical Report 1/2006 „Guide to the Evaluation of Measurement Uncertanty for Qualitative Test Results“)



4

Uncertainty of measurement is a parameter, associated with the result of a measurement, that characterises the dispersion of the values that could reasonably be attributed to the measurand.

Uncertainty of the resultEstimated quantity intended to characterise a range of values which contains the reference value.

Definitions

Standard uncertainty (u)Uncertainty of the result of a measurement expressed as a standard deviation.

Combined standard uncertainty (uc )Standard uncertainty of the result of a measurement when that result is obtained from the values of a number of other quantities, equal to the positive square root of a sum of terms, the terms being the variances or covariances of these other quantities weighted according to how the measurement result varies with changing these quantities.



5

Expanded uncertainty (Up)Quantity defining an interval about the result of a measurement that may be expected to encompass a large fraction of the distribution of values that could reasonably be attributed to the measurand.

Coverage factor (k)Numerical factor used as a multiplier of the (combined) standard uncertainty in order to obtain an expanded uncertainty.

AccuracyThe closeness of agreement between a test result and the accepted reference value.

TruenessThe closeness of agreement between the average value obtained from a large series of test results and an accepted reference value.

PrecisionThe closeness of agreement between independent test results obtained under stipulated conditions.



6

EN ISO 20988:2007 „Air quality – Guidelines for estimating measurement uncertainty“

A five-step procedure for uncertainty estimation is described:

1. Problem specification

2. Statistical analysis

3. Estimation of variances and covariances

4. Evaluation of uncertainty statements

5. Reporting



7

Objective is to specify

- the measurement

- the wanted uncertainty statement

- the experimental data

- effects not described by experimental data

Problem specification

Element Direct approach Indirect approach

experimental design one design more than one design

experimental data y(j) with j = 1 to NyR (reference value)

x1(j) with j = 1 to N, x1R

x2(j) with j = 1 to N, x2R

….

Additional deviation, if appropriate

Y Y



8

Statistical analysis

Element Direct approach Indirect approach

Method model equation not required y= f(x1, x2,…)

Statistical model equation Y=y + Y Y= f(x1, x2,…)+Y

Variance budget equation var(Y)=var(y)+var(Y) var(Y)=c1² var(x1)+ c2² var(x2)+….

+2c1c2cov(x1,x2)+ … var(Y)

A statistical model equation shall be established to describe the relationship between the statistical population of possible results of measurement Y and the input data.



9

Direct approach

A direct approach provides a series of results of measurement observed in a specified experimental design and, if appropriate, expert judgement of additional deviations Yi caused by effects not described by the series of observations

iYyY

variance budget equation:

where: Y a possible result of measurement y a realized result of measurement (input data) dYi an additional deviation of result of measurement y not described by the experimental data

(can be neglected, if the corresponding variance contributes less than 5 % to the variance estimate var(Y) used in the uncertainty estimation)

statistical model equation:

)var()var()var( iYyY

where: var(Y) estimate of the variance of possible results of measurement Y var(y) estimate of the variance of a series of results of measurement y var(Yi) estimate of the variance of additional deviation Yi , obtained by a type B evaluation



10

Indirect approachprovides for each input quantity xi of a known method model equation y = f(x1,..,xK ) either a series of experimental data xi(j) collected in a specified experimental design, or a variance estimate var(xi). If appropriate, additional deviations δYj , which are not described appropriately by the experimental data to be evaluated, are assessed by expert judgement.

iK YxxfY ),...( 1statistical model equation:where Y possible result of measurement; xi input quantity of the method model equation y = f (x1,.., xK )dYi additional deviation of result of measurement y not described by the experimental data

variance budget equation:

M

jj

K

i

K

i

K

ijKijii YxxccxcY

11 1 11

2 )var(),cov(2)var()var(

where ci sensitivity coefficient with respect to variations of input quantity xi

cov(xi, xj) estimate of covariance between input quantities xi and xj



11

Sensitivity coefficients

The sensitivity coefficient ci is the partial derivative of the method model function

y = f (x1,.., xK )

i

Ki x

xxfc

),....,( 1

ci - can be calculated numerical from the partial derivative

N

j

ii N

jxjyc

1

)(/)(

ci - as mean value of the ratio of observed changes of the results of measurement y(j) to the the changes of the input data xi(j)

V

My Example: y= mass concentration of particles

)var(),cov(2)var()var()var( 22 YVMccVcMcY VMVM

VM

ycM

1

2V

M

V

ycV

0),cov( VM

)var()var()var(1

)var(4

2

2YV

V

MM

VY



12

Estimation of variances and covariances

Type A: Evaluation using statistical analysis of measurement series

The calculation method is depending on the experimental design used to collect the input data.

Type B: Evaluation using means other than statistical analysis of measurement series (expert judgement)

If information are available• on the expected range of variation [min(δYj) < δYj < max(δYj)] of the

deviation δYj

• on the expected type of the statistical population of δYj

Statisticalpopulation

Rangemax(δYj) = – min(δYj)

Estimated variancevar(δYj)

Rectangular a a²/3

Triangular a a²/6



13

Covariances

The covariance associated with the values xi and xk assigned to two input quantities of the applicable method model equation shall be zero if:

• xi and xk have not been observed repeatedly in the same experimental design

• either xi or xk was kept constant, when providing repeated observations of the other quantity repeatedly.

Calculation of covariances can be avoided often by appropriate choice of experimental designs!



14

Evaluation of uncertainty statements

Combined standard uncertainty: )var()( Yyuc

y

yuyu c

r

)()(

where var(Y) is the estimate of the variance of the population of possible results of measurement Y

Relative standard uncertainty:

Expanded Uncertainty: )()( yukyU cp



15

Expanded uncertainty – coverage faktor k

)()( yukyU cp coverage factor k and coverage probability p shall be stated when an expanded uncertainty Up(y) is reported

relationship between k and p:

1. y is a mean value (N>1) of independent observations with the same measuring system

2. y is obtained by single application of a measuring method, distribution of possible results approximately is Gaussian

3. y is obtained by single application of a measuring method, distribution of possible results are not described by a Gaussian distribution



16

Case 1. and 2.: k = t(p,(direct approach)

where:

t(pis the (1-p)-quantile of Students t- distribution of ν degrees of freedom

p is the coverage probability of interval [–t(p,ν); +t(p,ν)] by Students t-distribution with ν degrees of freedom

is the number of degrees of freedom; = N – 1

Uncertainty interval for a coverage probability: )]();([ yukyyuky

Indirect approach: Welch-Satterthwaite-equation:

K

i

M

j j

j

i

i

eff Yxc

Y

1 1

224

2

)(var)(var

)(var



17

A report on execution of a specified task of uncertainty estimation shall include (at least) the following items:

• Problem specification including- method of measurement- wanted uncertainty statement- statistical population of possible results of

measurement considered- experimental data and experimental design- effects not described by experimental data

• statistical analysis, describing the applied statistical model equation and the variance (budget) equation

• evaluation methods, describing the applied evaluation methods

• numerical value of the wanted uncertainty statement and its range of application

Reporting



18

Uncertainty estimation in practise

What is required ??• Analysers have to fulfil the data quality objectives of the Directive

2008/50/EC “on ambient air quality and cleaner air for Europe”.(Uncertainty and minimum data capture)

• Analysers have to fulfil the relevant performance characteristics and criteria of the EN standards.(Type approval test)

Test reports of the type approval should contain all needed uncertainty information for a specific type of analyser.

But the user has to show that an analyser fulfil the requirements on measurement uncertainty also under the specific conditions on the monitoring site. suitability evaluation



19

Type approval of analysers

1. the value of each individual performance characteristic tested in the laboratory shall fulfil the requirements

2. the expanded uncertainty calculated from the standard uncertainties due to the values of the specific performance characteristics obtained in the laboratory tests fulfils the requirements

3. the value of each of the individual performance characteristics tested in the field shall fulfil the requirements

4. the expanded uncertainty calculated from the standard uncertainties due to the values of the specific performance characteristics obtained in the laboratory and field tests fulfils the requirements



20

When a type approved analyser has been chosen for a particular measuring task, then the suitability of this analyser shall be evaluated at a specific measuring location.

The analyser at the specific site has been judged to conform with the EU data quality objectives

An expanded uncertainty calculation for the type approved analyser shall be made according to the specific circumstances at the monitoring station or site.

If the site-specific conditions are outside the conditions for which the analyser is type approved, then the analyser shall be retested under these site-specific conditions and a revised type approval will be issued.

If the analyser complies with the requirements, then that particular analyser may be installed and used at that monitoring station.

General Requirements:



21

Parameters Remarks

Sample pressure variation The sample gas pressure variation during a whole period of a year shall be estimated.

Sample gas temperature

variation

The sample gas temperature variation during a whole period of a year shall be estimated. Sample gas temperature may be controlled by heating or thermostating.

Surrounding air temperature

variation

The temperature fluctuation shall be within the range specified in the type approval test. Temperature may be controlled thermostatically.

Voltage variation The voltage fluctuation shall be within the range in the type approval test. Voltage fluctuations may be controlled by means of voltage stabilizers.

H2O concentration range The H2O concentration range during a whole period of a year shall be estimated.

H2S, NH3, NO, NO2 and

m-xylene concentration ranges

The concentration range of each compound during a whole period of a year shall be estimated.

Expanded uncertainty of the

calibration gas

The expanded uncertainty of the calibration gas shall be included. This implies the uncertainty of the calibration gas itself as well as the uncertainty of any dilution system (where applicable)

Calibration frequency The intended calibration frequency shall be used for the calculation of the influence of the drift.



22

Choosing of the needed performance characteristics from the type approval test report

Evaluation or estimation of the site-specific conditions and of the uncertainty of the span gas used for calibration

Calculation of the combined expanded uncertainty with inclusion of the site-specific conditions

Comparison with the uncertainty requirements

Operating Sequence:



23

Example: Uncertainty of the certification of a transfer standards

Measurand y: NO concentration of the test gasMethod: analyzer according to EN 14211Input data: daily measurements of a certified NO test gas

uncertainty of the test gas concentration

Model equation: )()( jeyjy R Ryjyje )()(

)()()var( 22 euyuy R

with

Variance equation:

0),cov( eyR

Residual standard deviation:

N

jRyjy

Neu

1

2))((1

)(

Standard uncertainty: )()()( 22 euyuyu R



24

Calculated:mean value 132,65 ppb

residual standard dev. u(e)= 0,538 ppb

Date y(j) y(j)-yR

22.10.2008 133,3 0,5

23.10.2008 132,5 -0,3

24.10.2008 133,0 0,2

27.10.2008 133,0 0,2

28.10.2008 133,1 0,3

29.10.2008 133,2 0,4

30.10.2008 132,7 -0,1

03.11.2008 132,7 -0,1

04.11.2008 133,0 0,2

05.11.2008 133,1 0,3

06.11.2008 133,0 0,2

07.11.2008 132,7 -0,1

10.11.2008 132,8 0,0

14.11.2008 131,7 -1,1

17.11.2008 132,9 0,1

18.11.2008 133,1 0,3

19.11.2008 133,2 0,4

20.11.2008 133,3 0,5

21.11.2008 133,0 0,2

24.11.2008 132,7 -0,1

25.11.2008 132,7 -0,1

26.11.2008 131,7 -1,1

27.11.2008 131,6 -1,2

28.11.2008 132,2 -0,6

01.12.2008 131,8 -1,0

02.12.2008 132,2 -0,6

03.12.2008 132,8 0,0

04.12.2008 132,4 -0,4

05.12.2008 132,7 -0,1

08.12.2008 132,5 -0,3

09.12.2008 131,5 -1,3

certified NO concentration yR = 132,8 ppb

stated expanded uncertainty U(yR) = 2% with k=2

standard uncertainty u(yR)=1,328 ppb

ppbyu

euyuyu R

433,1538,0328,1)(

)()()(

22

22

y

Expanded uncertainty:

%2,2022,0)(

92,2)()(

04,2)31;05,0(

95,0

95,0

y

yU

ppbyukyU

tk

y



25

Ozone measurements:(example C.2 from EN ISO 20988)

Method: automatically measurements according EN 14625 (UV-absorption)

function control every 25 h with zero and span gas

daily correction of zero offsets

Measurand: 1-h mean value of the ozone concentration in ambient air

Described effects: variations of surrounding air temperature and pressure

Method model equation: y = x – e(j)

x is the observed value without correction

e(j) is the offset for day j

Wanted uncertainty statement: standard uncertainty u(y), expanded uncertainty U0,95(y)



26

Experiment: 1. every 25 hours providing of zero gas and determination of the zero offset e(j) = x0(j)2. every 25 hours providing of span gas and evaluation of the

span factor ß(j) = xs(j)/ys

Input data: series of offset corrections (N=20)series of span factors (N=20)

Reference values: Zero gas y0 = 0 µg/m³Span gas ys = 280 µg/m³ u(ys) = 2,8 µg/m³

Effects not described: Influence of sampling device(Influence of humidity and other compounds in the

ambient air)

Variance budget equation: var(y) = u²(x) + u²(e) + 2cov(x,e) cov(x,e) = 0



27

Index j Observed measuring value for zero gas e/j)

Span factor ß(j)

1 -0,7 1,00

2 -0,9 0,96

3 -1,4 0,98

4 -0,9 0,99

5 -1,1 1,04

6 -0,3 1,05

7 -0,8 1,04

8 -0,8 1,03

9 -1,0 1,04

10 -1,0 1,03

11 -0,9 1,02

12 -0,8 1,02

13 -1,1 1,03

14 -0,8 1,07

15 -0,8 1,04

16 -0,6 1,02

17 -0,5 1,02

18 -1,0 1,05

19 -0,7 1,05

20 -1,0 0,97

Standard uncertainty of zero offset e:

³/89,0)(1

)(1

20 mµgjx

Neu

N

j

(includes the bias uB(e))

Model equation for x(j): Syjjx )()(

Variance equation:

),cov()(2)()(

)var(22

2S

S

S yjxy

yuuxx

(cov(ß,yS) = 0)

Standard uncertainty of ß:

036,0))(1(1

)(1

2

N

j

jN

u

(includes the bias uB(ß) )



28

Standard uncertainty of y:

)(

)()()( 2

22

2 euy

yuuyyu

S

S

Degrees of freedom:= 20 coverage factor: k = t(0,05,20) = 2,1

Uncertainty depends on the corrected measured value!

Now it is possible to calculate an uncertainty statement for the 1-hour mean values measured in the observed period depending on the corrected measured value.



29

Helpful documents and programs:

EN ISO 20988 “Air quality – Guidelines for estimating measurement uncertainty”

Eurolab Technical Report 1/2006 “Guide to the Evaluation of Measurement Uncertainty for Quantitative Test Results”

Eurolab Technical Report 1/2007 “Measurement Uncertainty Revisited: Alternative Approaches to Uncertainty Evaluation”

Excel-Program Nordtest TR 537 “Measurement Uncertainty Estimation”



30

Many Thanks

for your Attention!

EU-Twinning Project RO 2006 IB EN 09 Bucharest, 03.02.2009Saxony-Anhalt Wolfgang Garche State...

Documents

Transcript of EU-Twinning Project RO 2006 IB EN 09 Bucharest, 03.02.2009Saxony-Anhalt Wolfgang Garche State...