p19 Caeal Unce Pol Rev 1-4

P19 - CAEAL Policy on Uncertainty Rev 1.4 of 39

CAEAL Policy on theEstimation of Uncertainty of Measurement in Environmental

Testing

Requirement from CAN-P-4D

6. The following excerpt from CAN-P-4D articulates the requirement to estimatethe uncertainty of measurement associated with testing.

5.4.6 Estimation of uncertainty of measurement

CAEAL Uncertainty Policy

1. Laboratories accredited under the joint SCC-CAEAL Accreditation Program forEnvironmental Laboratories shall fulfil the requirements of CAN-P-4D (ISO/IEC17025) with respect to the estimation of uncertainty of measurement associated withenvironmental testing for those tests which produce numerical results. This applieswhether the test methods are rational or empirical..

2. They shall report the expanded uncertainty estimate as part of the reported result whenthe reporting of the estimate of measurement uncertainty is• Required by the client, or• Required to establish that the data is 'fit-for-purpose', or• Required because the data is being used to establish compliance (of the body being

represented by the analysed sample) with a requirement.

3. The requirement which underlies this policy is that given in CAN-P-4D, Clause 5.4.6.Other documents and Guides may be used by laboratories to develop methods inmeeting this requirement.

Implementing the CAEAL Uncertainty Policy

4. All laboratories affected by this policy shall submit to CAEAL, on or before 31December 2002, documentary evidence of their commitment to this policy within theirquality system and a plan for its implementation within the laboratory.

5. Beginning with the 2003 assessment year, laboratories shall demonstrate theirimplemented use of adequate procedures for their estimation of the uncertainty ofmeasurement associated with their accredited tests and shall have begun reporting theestimates (as expanded standard uncertainties) in accordance with the requirements ofCAN-P-4D Clause 5.10.3.1 c).


5.4.6.1 A calibration laboratory, or a testing laboratory performing its owncalibrations, shall have and shall apply a procedure to estimate the uncertainty ofmeasurement for all calibrations and types of calibrations.

5.4.6.2 Testing laboratories shall have and shall apply procedures for estimatinguncertainty of measurement. In certain cases the nature of the test method maypreclude rigorous, metrologically and statistically valid, calculation ofuncertainty of measurement. In these cases the laboratory shall at least attempt toidentify all the components of uncertainty and make a reasonable estimation, andshall ensure that the form of reporting of the result does not give a wrongimpression of the uncertainty. Reasonable estimation shall be based on knowledgeof the performance of the method and on the measurement scope and shall makeuse of, for example, previous experience and validation data.

NOTE 1 The degree of rigor needed in an estimation of uncertainty ofmeasurement depends on factors such as:- the requirements of the test method;- the requirements of the client;- the existence of narrow limits on which decisions on conformance to a

specification are based.

NOTE 2 In those cases where a well-recognized test method specifies limits tothe values of the major sources of uncertainty of measurement and specifies theform of presentation of calculated results, the laboratory is considered to havesatisfied this clause by following the test method and reporting instructions (see5.10).

5.4.6.3 When estimating the uncertainty of measurement, all uncertaintycomponents which are of importance in the given situation shall be taken intoaccount using appropriate methods of analysis.

NOTE 1 Sources contributing to the uncertainty include, but are notnecessarily limited to, the reference standards and reference materials used,methods and equipment used, environmental conditions, properties and conditionof the item being tested or calibrated, and the operator.NOTE 2 The predicted long-term behaviour of the tested and/or calibrated itemis not normally taken into account when estimating the measurementuncertainty.NOTE 3 For further information, see ISO 5725 and the Guide to the Expressionof Uncertainty in Measurement (see bibliography) [1].

Guidance and Interpretations adopted by PALCAN for use by laboratories

7. SCC Document D92.5 contains all of the interpretive notes and guidance usedthroughout the PALCAN program and is also for those laboratoriesaccredited under the joint SCC-CAEAL Environmental LaboratoryAccreditation Program. This document is available from the SCC. Relevantcitations are from Clause 5.4.6 of D92.5 [2].


ILAC guidance to clause 5.4.6 (G.5.4.6.)G.5.4.6 Guidance on this aspect is to be found in the ILAC documentExpression of Uncertainty of Measurement in Calibration, and in theEURACHEM document Quantifying the Uncertainty in AnalyticalMeasurement.

SCC In te rpr et ive Note : Measur ement u nce rt ain ty as define d by the G UM, is t he on ly pe rt ine nt pr oduct of c alibr at ion activitie s. I t is impor tant to addr ess bothcalibration done in -hous e by the laborator y its elf and conduct ed by outs ide s upplier s.Te st ing laborator ie s t hat are pr oviding their own calibrations w ill have t o provideme as ure me nt un cer taint ie s for thes e c alibr at ion s. Tes tin g labor atories ar e t o ens ur ethat they also re ce ive appr opriate un ce rtain tie s of me asu re men t from out side sourc es of c alibr ation .

ILAC guidance to clause 5.4.6.2 (G.5.4.6.2)G.5.4.6.2 The complexity involved in estimation of uncertainty ofmeasurement in the case of testing varies considerably from one test field toanother and also within one field itself. It is also often achieved by a lessmetrologically rigorous process than that which can be followed forcalibration. Clause 5.4.6.2 of ISO/IEC 17025 allows for these factors andaccreditation bodies should take them into account during assessments. (ILACLaboratory Liaison Committee is developing a strategy for implementation ofmeasurement uncertainty in testing).

SCC Interpretive Note: this requirement cannot be applied unequivocally atthe present time because of the diversity of the fields of testing that are coveredby the SCC accreditation process. Also, there is a great deal of work that stillneeds to be conducted on the international scene before this requirement canbe applied in a consistent manner for specific fields and specificproduct/services classifications. SCC must remain in line with theinterpretations that will be published by ILAC. SCC can have input into thedevelopment of the ILAC document and will wait for more guidance fromILAC before applying this requirement rigorously. Assessors are reminded toproceed as they have done in the past and to continue to require this where itis common practice. For the other areas where it is not common practice toexpress uncertainty of a test result, assessors are requested to remind thelaboratory of this requirement and to ask them to consider what they feelwould be feasible in their operation and that when guidelines are available,they will be distributed to the laboratories and considered for futureassessments.

SCC In te rpr et ive Note : The Horw itz Trumpe t c an be u sed for an in it iales timat ion of unc er taint y in che mical and microbiology/biology t est in g.(A PL AC Common Ass es sor Training Cours e 2000- 04- 10 to 14)


Estimating the Uncertainty of Measurement in Environmental Laboratories

8. Several organisations and groups have published advice on the estimation ofmeasurement uncertainty to the analytical laboratory community. Theseinclude APLAC and the AOAC, ILAC, NMKL, EURACHEM/CITAC, theSCC TG Labs Mineral Analysis Working Group and the Ontario Ministry ofthe Environment. CAEAL has adopted a single approach which is deemed tohave a minimum impact on the costs incurred by laboratories to meet therequirements of CAN-P-4D. This policy considers the expected costs tolaboratories, is based on consensus from within the Canadian environmentallaboratory community and provides a generic approach from among theorganisations listed above.

9. There are two approaches that can be taken in estimating the uncertainty ofmeasurement associated with testing. The first method, termed Type A,estimates uncertainties through the use of experimental data such as thatfrom routine laboratory QA/QC work (duplicates, reference material usage,method validation studies, and proficiency testing (PT) and other inter-laboratory programs, for example).

10. The second method, Type B, involves the cause and effect-based metrologicalestimation of specific uncertainties from each identified source of uncertainty.It is similar to the approach used by calibration laboratories.

11. The first approach (Type A) is the one used by most specifier agencies whenrequiring estimations of the uncertainty of measurement for analyticallaboratories. It is also the basis of the approach used in this Policy.

Why is the CAEAL Uncertainty Policy based on Type A?

• The EURACHEM CITAC Guide Quantifying Uncertainty in AnalyticalMeasurements states “ Where a factor has been representatively varied during thecourse of a precision experiment … that factor needs no additional study.”

• Virtually all of the data required is already present in laboratory files.• Little time is required to estimate the uncertainty for individual methods using the

laboratory historical data.• Little training is required to enable laboratory staff to do the necessary

calculations.• The resulting estimate is robust and defendable to clients and specifiers.• The resulting estimate is relatively easy to assess during assessments.


15. In cases where test methods generate multi-analyte data of 10 species or more(as from ICP/OES, GC/MS or ICP/MS methods), laboratories shall select 3analytes that represent each of three levels of uncertainty of the results –small, medium and large levels of uncertainty - for which to estimate themeasurement uncertainty. In cases where the analyte is expected to occurover a wide concentration range (more than a factor of 10), the estimation ofuncertainty should be done at low, medium and high concentrations withinthat range. This will reflect the increase in the uncertainty with concentration.The other analytes will be classified within these three categories.

NOTE: The objective of this approach is to minimise the amount of workrequired in the estimation of uncertainty for multi-component tests. It isintended that the uncertainty for all constituents in a category would beassumed to have the uncertainty of the surrogate constituents, expressed asthe pooled SD of the 3 measured SDs at each concentration range.

16. Laboratories will be required to re-estimate measurement uncertainty onlywhen changes to their operations are made that may affect sources ofuncertainty and these sources have not been shown to be unaffected throughmethod validation or other studies. If validation of the method has shownthat the uncertainty of a test is not significantly affected by a change ofanalysts for example, then a change of analysts will not require a new

CAEAL Uncertainty Protocol

12. Measurement uncertainty shall be expressed as a combined Standard Deviation(SD) with the same units as those of the measurand.

13. The final result shall be reported with an expanded uncertainty that

'produces an interval (the expanded uncertainty) about the measurement resultthat may be expected to encompass a large, specified fraction (e.g. 95%) of thedistribution of values that could reasonably be attributed to the measurand.'

Guide to the Expression of Uncertainty in Measurement, ISO,1 ed. 1993, ISBN 92-67-10188-9.

14. The expanded standard uncertainty shall be calculated to give a confidence levelof 95% using an expansion factor 'k' of:

• k = 2 when n is 30 or more (n = number of observations from which the SDis calculated).

• k = the appropriate (95% confidence level) Student distribution 't' (two tailed)factor for n<30.


estimate to be made. Proof that such a change has not affected themeasurement process can be found in control charts monitoring results fromspikes or reference materials. If, according to the definition of the laboratoryquality program, there is no change in the spread of data about the mean (noloss of precision), or no evidence of the introduction of a bias, when such achange has been implemented (a different analyst in this example) then thatfactor does not have to be included as a possible source of uncertainty.Annotated control charts created when such changes occur are consideredsufficient documentary evidence of such a conclusion.

17. Laboratories shall make independent estimates of measurement uncertaintyfor tests performed on samples with significantly different matrices (organicconstituents in ground waters versus waste waters for example) only whensuch differences have been shown to have an effect on the estimate.

18. Laboratories shall make independent estimates of measurement uncertaintyfor analyte concentrations that vary over orders of magnitude (at low,medium and high concentration levels for example). If the relationshipbetween SD and concentration is shown to be linear, the laboratory canestimate an expanded RSD (see Reporting the Result in the Appendix).

19. When laboratories select an analytical portion from a sample that may not behomogeneous, the laboratory shall include sub-sample uncertainty as part ofthe combined standard uncertainty calculation (see Sample duplicateinsertion, under Laboratory Repeat Data Sets in the Appendix) when thatuncertainty is sufficiently large (see 'Tabulate Uncertainty Estimates' in theAppendix).

20. As stipulated in Note 2 to Clause 5.4.6.2 of CAN-P-4D (ISO/IEC 17025): 'Inthose cases where a well-recognized test method specifies limits to the values of themajor sources of uncertainty of measurement and specifies the form of presentationof calculated results, the laboratory is considered to have satisfied this clause byfollowing the reporting instructions.'

Using the Type A Approach

21. The following steps involve the use of experimental data to estimate theuncertainty of measurement for environmental laboratories. (See theAppendix for more detail).

• Using the method SOP and the final result-calculation equation, identifyand list all potential sources of uncertainty.

• Identify and compile recent laboratory repeat analysis and PT data thatis available.

• Match each repeat data set with those sources of uncertainty that arelikely to have varied during the collection of the repeat data and identifydouble counted sources of uncertainty.

• Estimate the magnitude of any source of uncertainty that is not variedduring the collection of any of the repeat data sets.


• Tabulate each source of uncertainty and its associated SD, and/orrelative SD (RSD) derived from the repeat data set(s) matched to it, orfrom the estimate made. Eliminate double counted sources.

• Using only those SDs that are 1/3 or more the size of the largestindividual SD, calculate the combined standard uncertainty usingstandard propagation of error rules (the square root of the sums ofsquares of SDs known as the “root sum of squares” - RSS).

• Apply the appropriate coverage factor 'k'. (see paragraph 14 above)• Report the result with the expanded uncertainty and with a description

of how the uncertainty was calculated.

Reporting the Uncertainty Associated with Environmental Measurement

22. CAN-P-4D, Clause 5.10.3.1 c) details the requirement to report uncertainty ofmeasurement associated with testing and includes the followingcircumstances [1]:

i. When it is relevant to the validity or application of the test result,ii. When so instructed by a client, oriii. When the uncertainty affects compliance to a specification limit.

23. Estimates of measurement uncertainty quoted in reports shall reflectconservative “worst case” scenarios of variability incorporating long termeffects on significant sources of uncertainty such as different analysts,instrument drift and other factors that reflect routine laboratory operations.A short description of how the estimate of uncertainty was determinedshould also be included. This description would include information on thesource(s) of the data used to estimate the SDs included in the calculation ofthe combined standard uncertainty.


APPENDIX 1Definitions of Terms Used in this Policy (reprinted from A2LA Guide[8])

And References

accuracy (of measurement) (VIM 3.5): closeness of the agreement between theresult of a measurement and a true value of the measureand

NOTES:• “Accuracy” is a qualitative concept• The term precision should not be used for “accuracy”.• “an accepted reference value” may be used in place of “a true value” in

this definition.}

bias (ISO 3534-1): the difference between the expectation of the test results from aparticular laboratory and an accepted reference value

NOTE: Bias is the total systematic error as contrasted to random error.There may be one or more systematic error components contributing tothe bias. A larger systematic difference from the accepted reference valueis reflected by a larger bias value.

combined standard uncertainty (GUM 2.3.4): standard uncertainty of the resultof a measurement when that result is obtained from the values of a number ofother quantities, equal to the positive square root of a sum of terms, the termsbeing the variances or covariances of these other quantities weighted accordingto how the measurement result varies with changes in these quantities

correlation (ISO 3534-1): the relationship between two or several randomvariables within a distribution of two or more random variables

NOTE: Most statistical measures of correlation measure only the degree oflinear relationship.

coverage factor (GUM 2.3.6): numerical factor used as a multiplier of thecombined standard uncertainty in order to obtain an expanded uncertainty

NOTE: A coverage factor, k, is typically in the range of 2 to 3.

error (of measurement) (VIM 3.10): result of a measurement minus a true valueof the measureand

NOTES:• Since a true value cannot be determined, in practice a conventional

true value is used.• When it is necessary to distinguish “error” from “relative error”, the

former is sometimes called “absolute error of measurement”. Thisshould not be confused with “absolute value of error”, which is themodulus of the error.

• “an accepted reference value” may be used in place of “a true value” inthis definition.}


expanded uncertainty (GUM 2.3.5): quantity defining an interval about the resultof a measurement that may be expected to encompass a large fraction of thedistribution of values that could reasonably be attributed to the measureand.

NOTES:• The fraction may be viewed as the coverage probability or level of

confidence of the interval.• To associate a specific level of confidence with the interval defined by

the expanded uncertainty requires explicit or implicit assumptionsregarding the probability distribution characterized by themeasurement result and its combined standard uncertainty. The levelof confidence that may be attributed to this interval can be known onlyto the extent to which such assumptions may be justified.

influence quantity (VIM 2.7): quantity that is not the measureand but that affectsthe result of the measurement

EXAMPLES:• temperature of a micrometer used to measure length;• frequency in the measurement of the amplitude of an alternating

electric potential difference;• bilirubin concentration in the measurement of hemoglobin

concentration in a sample of human blood plasma.

level of confidence (GUM C.2.29): The value of the probability associated with aconfidence interval or a statistical coverage interval

NOTE: The value is often expressed as a percentage.

measureand (VIM 2.6): particular quantity subject to measurementEXAMPLE: Vapor pressure of a given sample of water at 20°C.NOTE: The specification of a measureand may require statements aboutquantities such as time, temperature, and pressure.

measurement (VIM 2.1): set of operations having the object of determining avalue of a quantity

precision (ISO3534-1): the closeness of agreement between independent testresults obtained under stipulated conditions

NOTES:• Precision depends only on the distribution of random errors and does

not relate to the true value or the specified value.• The measure of precision is usually expressed in terms of imprecision

and computed as a standard deviation of the test results. Lessprecision is reflected by a larger standard deviation.

• “Independent test results” means results obtained in a manner notinfluenced by any previous result on the same or similar test object.Quantitative measures of precision depend critically on the stipulatedconditions. Repeatability and reproducibility conditions are particularsets of extreme conditions.


repeatability (VIM 3.6): closeness of the agreement between the results ofsuccessive measurements of the same measureand carried out under the sameconditions of measurement

NOTES:• The conditions are called repeatability conditions.• Repeatability conditions include: the same measurement procedure;

the same observer; the same measuring instrument used under thesame conditions; the same location; repetition over a short period oftime.

• Repeatability may be expressed quantitatively in terms of thedispersion characteristics of the results.

reproducibility (VIM 3.7): closeness of the agreement between the results ofmeasurements of the same measureand carried out under changed conditions ofmeasurement

NOTES:• A valid statement of reproducibility requires specification of the

conditions changed.• The changed conditions may include but are not limited to: principle

of measurement; method of measurement; observer; measuringinstrument; reference standard; location; conditions of use; time.

• Reproducibility may be expressed quantitatively in terms of thedispersion characteristics of the results.

• Results are here usually understood to be corrected results.

standard uncertainty (GUM 2.3.1): uncertainty of the result of a measurementexpressed as a standard deviation

trueness (ISO 3534-1): the closeness of agreement between the average valueobtained from a large series of test results and an accepted reference value

NOTES:• The measure of trueness is usually expressed in terms of bias.• Trueness has been referred to as “accuracy of the mean”. This usage is

not recommended.

type A evaluation of uncertainty (GUM 2.3.2): method of evaluation ofuncertainty by the statistical analysis of observations

type B evaluation of uncertainty (GUM 2.3.3): method of evaluation ofuncertainty by means other than the statistical analysis of a series of observations


uncertainty of measurement (VIM 3.9): parameter, associated with the result of ameasurement, that characterises the dispersion of the values that couldreasonably be attributed to the measureand

NOTES:• The parameter may be, for example, a standard deviation (or a given

multiple of it), or the half-width of an interval having a stated level ofconfidence.

• Uncertainty of measurement comprises, in general, many components.Some of these components may be evaluated from the statisticaldistribution of the results of series of measurements and can becharacterised by experimental standard deviations. The othercomponents, which can also be characterised by standard deviations, areevaluated from assumed probability distributions based on experience orother information.

• It is understood that the result of the measurement is the best estimate ofthe value of the measureand, and that all components of uncertainty,including those arising from systematic effects, such as componentsassociated with corrections and reference standards, contribute to thedispersion. This definition is that of the “Guide to the expression ofuncertainty in measurement” in which its rationale is detailed (see inparticular 2.2.4 and Annex D to VIM).

References

1. CAN-P-4D: General Requirements for the Competence of Testing andCalibration Laboratories (Verbatim Canadian adoption of ISO/IEC 17025 –same title). 2000, Standards Council of Canada: Ottawa, ON.

2. SCC: D92.5: PALCAN Interpretations for Conducting Assessments of Testingand Calibration Laboratories. 2000, SCC: Ottawa.

3. Ellison, S.L.R., M. Rosslein, and A. Williams, Editors, QuantifyingUncertainty in Analytical Measurement, 2nd Edition, Eurachem/CITAC,available on internet athttp://www.measurementuncertainty.org/mu/quam2.pdf, 2000.

4. http://www.measurementuncertainty.org/mu/quam2.pdf, 2000.5. ILAC Guide 17: Introducing the Concept of Uncertainty of Measurement in

Testing in Association with the Application of the Standard ISO/IEC 17025.2002, ILAC: Rhodes, NSW, Australia, http://www.ilac.org, 2002.

6. Albert, R. and W. Horwitz, A Heuristic Derivation of the Horwitz Curve.Anal. Chem., 1997. 69(4): p. 789-790.

7. Uncertainties in qualitative testing and analysis. Accreditation and QualityAssurance, 2000. 5( 8.): p. 346-348.

8. APLAC Policy, Interpretation and Guidance on the Estimation of Uncertaintyof Measurement in Testing, Asia-Pacific Laboratory Cooperation, (APLAC)2002.

9. Adams, T.M., A2LA Guide for the Estimation of Measurement Uncertainty InTesting. 2002, American Association for Laboratory Accreditation (A2LA):Frederick, MD. p. 42.


10. Barwick, V.J. and S.L.R. Ellison, VAM Project 3.2.1. Development andharmonisation of measurement uncertainty principles. 2000, LGC, UK,www.vam.org.uk, http://www.caeal.ca/VAM_uncertainty.pdf, 2000.

11. Estimation and Expression of Measurement Uncertainty in ChemicalAnalysis. 1997, NMKL. p. 15.

12. McQuaker, N., Quality Control for Environmental Laboratories. Revision 4.5October 2001, CAEAL: Ottawa, ON.

13. Taylor., J.K., Quality Assurance of Chemical Measurements. 1987, Boca Raton,FL: Lewis Publishers Inc.

14. A2LA Interim Policy on Measurement Uncertainty for Testing Laboratories.2000, American Association for Laboratory Accreditation (A2LA): Frederick,MD.

15. Excel spreadsheet used on the Course in Measurement of uncertainty inmicrobiological examination of food. 2002, NMKL,http://www.nmkl.org/Engelsk/publications.htm,2002.

16. Measurement of uncertainty in microbiological examination of foods. 2nd.Ed. 2002, NMKL: Norway,http://www.nmkl.org/Engelsk/reports.htm,2002.

17. NMKL, Measurement of Uncertainty in Microbiological Examination ofFoods. 1999, NKML (Nordic Committee on Food Analysis. p. 22,www.nmkl.org,1999.

18. Accreditation in Microbiological Laboratories. 2002, European Cooperationfor Accreditation (EA), http://www.europeanaccreditation.org/,2002.

19. Mills, W.J. Uncertainty in Microbiological Analysis of EnvironmentalSamples. in CAEAL Uncertainty Workshop. 2001. Edmonton, AB: CAEAL.

20. Niemela, S.I., A semi-empirical precision control criterion for duplicatemicrobiology colony counts. Letters in Applied Microbiology. 22(4): p. 315-319.1996.

21. Voysey, P.A. and K. Jewell, Uncertainty Associated with MicrobiologicalMeasurement. 1999, Campden & Chorleyword Food Research Association. p.271999.

22. Niemi, R.M. and S.I. Niemela, Measurement Uncertainty in MicrobiologicalCultivation Methods. Accred. Qual. Assur. 6: p. 372-375.2001.

23. Niemela, S.I., Uncertainty of Quantitative Determinations Derived byCultivation of Microorganisms. 2002, Centre for Metrology and Accreditation:Helsinki, Finland. p. 75,http://www.mikes.fi/documents/upload/Publication%20J3%202002_1.pdf,2002.

24. Norli, H.S. NMKL Procedure no 8, 2nd Ed., 2002: Measurement of uncertaintyin microbiological examination of foodsProf. Eystein Skjerve. in AOACAnnual Meeting. 2002. Los Angeles, CA: AOAC.

25. Schop, R., Personal Communication, D.W.J. Mills, Editor. 2002: Toronto,ON2002.

26. USEPA, Membrane Filter Method for the Simultaneous Detection of TotalColiforms and Escherichia coli in Drinking Water. 2000, USEPA, Office ofResearch Environmental Protection and Development Cincinnati OH 45268:Washington, DC. p. 21,http://www.epa.gov/nerlcwww/MI_emmc.pdf,2000.


27. USEPA, Improved Enumeration Methods for the Recreational Water QualityIndicators: Enterococci and Escherichia coli. 2000, United StatesEnvironmental Protection Agency, Office of Science and Technology ,Washington DC 20460,http://www.epa.gov/ost/beaches/rvsdman.pdf,2000.

28. McQuaker, N.R., Measurement Uncertainty for Environmental Laboratories.2000, CAEAL: Ottawa, ON2000.

29. Mills, W.J., Uncertainty Estimate for a Microbiological Dataset. 2002,Unpublished Data 2002.

30. Tholen, D., Telephone Conversation, W.J. Mills, Editor. 2002: Chicago, IL2002.


APPENDIX 2Measurement Uncertainty for Analytical Chemistry

Aim

1. This appendix considers and expands each of the CAEAL Protocol steps, asthey apply to analytical chemistry, in more detail. It also explains what ismeant and how to perform each task required.

Sources of uncertainty:

2. The possible sources of uncertainty for an analytical method are tabulated inmany of the sources listed in paragraph 5 of the Policy. Close examination ofthe steps in the laboratory method SOP, and of the parameters found in thefinal concentration calculation, will usually help to identify the likely sourcesof uncertainty in the method. ILAC Guide 17 lists these as[4]:

a. definition of the measurandb. samplingc. transportation, storage and handling of samplesd. preparation of samplese. environmental and measurement conditionsf. the personnel carrying out the testsg. variations in the test procedureh. the measuring instrumentsi. calibration standards or reference materialsj. software and/or, in general, methods associated with the measurementk. uncertainty arising from correction of the measurement results for

systematic effects.

3. Spike or reference material recovery efficiency is an example of a source ofvariation in the test procedure category above. Inter-laboratory bias andstandard deviations derived from proficiency testing programs are examplesof uncertainty in the correction for systematic effects. Data from referencematerial analyses can also be used for this purpose.

Laboratory Repeat Data Sets

4. These are sources of repeated measurements from which SDs and RSDs canbe calculated. The laboratory can vary one or more of the above sources ofuncertainty during the collection of the repeat data and the SD calculated willinclude uncertainty attributed to the varied source(s). The various repeat datasets include:

• Proficiency testing programs – These are a source of reproducibility SD(SDR) that includes both intra- and inter-laboratory sources of uncertainty.It is larger than the intra-laboratory uncertainty (SDr) – known asrepeatability - of a laboratory whose methods are in statistical control. In


the absence of any other source of repeated data, reproducibility fromproficiency testing and other round robin studies can be used as anestimate of measurement uncertainty provided the laboratory candemonstrate that their bias is within certain bounds, consistent with thecollaborative study estimate of between-laboratory SD. It is, however,very likely to be an over estimate of what the intra-laboratory uncertaintyactually is (by a factor of 2 according to conventional wisdom). Note thepossibility of using PT results for bias detection and correction.

• Reference sample insertion – both certified reference samples and in-house reference samples inserted into routine runs for control chartingapplications are a source of long term uncertainty data. Sources ofuncertainty that can vary during repeated insertion of these samples overtime are analysts, calibration sets, calibration solution sources,environmental conditions, instrument drift and many more. As aconsequence, the standard deviation calculated from this data will reflectthe uncertainty contributions from these sources. Sources of variabilitythat are not included however, are factors that can change from sample tosample such as matrix effects and sample non-homogeneity (orheterogeneity). If a reference sample is to be used to estimate a bias, theuncertainty in the bias estimate must include the uncertainty in thecertified value of the reference material. The combined standarduncertainty must include the bias uncertainty if the result is corrected forbias.

• Spike recovery data – Can give the same information as reference sampleinsertion and in some cases can reflect variability due to different samplematrices. This type of interpretation should be made with cautionhowever since the spike portion may be 100% recovered but the analyteportion may not be (due to speciation differences for example).

• Method validation replicate data – This is a source of data from repeatanalyses run to establish precision estimates at different analyteconcentration levels. The results from those run at low concentrations forthe calculation of detection and quantitation limits can also be used toassess uncertainty at low analyte concentration ranges. The validationdata can also serve as a source of information on the uncertaintycontributed by other sources (such as analyst, instrument, temperature,time etc.) depending on how the validation work was planned andexecuted to include such variables. This is especially the case ifruggedness studies were incorporated as an integral part of the validationprogram to assess the effect of varying parameters likely to be significantsources of uncertainty. A thorough discussion of the use of methodvalidation data in the estimation of uncertainty is VAM Project 3.2.1Development and Harmonization of Measurement Uncertainty Principles; Part(d): Protocol for uncertainty evaluation from validation data, by V.J. Barwickand S.L.R. Ellison, January 2000, Version 5.1. This can be downloaded as apdf file from the VAM web site.


• Sample duplicate insertion – This can be a valuable source of uncertaintydata – known as replicability SDdupl - that reflects the variability due todifferences between analytical portions (non-homogeneity) and otherfactors that can vary between replicates (weighing, volumetricmanipulations, and short term instrument drift are examples). Note: If theduplicates are measured in the same analytical run, as is usually the case,any uncertainty associated with the instrument set up and calibration isnot accounted for. More than 20 duplicate pairs should be run of samplesof a similar concentration. The SDdupl = √(∑R2/2N) where R is thedifference between duplicate pairs and N is the number of duplicate pairs.This should be calculated for low, medium and high concentration rangesto reflect the concentration dependence of the SD. Alternatively, the RSDcan be calculated (at low, medium and high concentration ranges as well)as RSDdupl = √{∑[(ai - bi)/xi]

2/2N} where (ai - bi)/xi is the relativedifference between duplicates for sample “i” and N is the number ofsamples for which duplicates have been run. This value makes allowancesfor the concentration dependence of the SD for concentrations betweenthose at which the calculation was made (see paragraph 9 below).

Match Repeat Data with Uncertainty Sources

5. The objective of this step is to select laboratory Quality Control andvalidation data that includes as many sources of variability as possible so thatthese do not have to be estimated using the more difficult (and timeconsuming) Type B approach. The most effective means of achieving this is todesign the analytical method to ensure spikes, reference samples andduplicates are inserted as early as possible into the analytical run. In addition,from the "Method Validation Replicate Data" section in paragraph 4 above,the method validation program should include the variation of as manypotentially significant sources of uncertainty as possible.

Estimate the Uncertainty for any Sources not Accommodated by RepeatedData

6. In the unusual cases where it is necessary to estimate uncertainties for anysources not accommodated by repeated data, the estimation of theuncertainty from these sources is based on information from manufacturerspecifications that accompany instruments and equipment (such asvolumetric ware), tabulated data from handbooks, experience from othermethods and/or laboratories and other sources. Examples of thesecalculations are found in the EURACHEM/CITAC guide QuantifyingUncertainty in Analytical Measurement available as a pdf file from their webpage.

Tabulate Uncertainty Estimates

7. Compile the values estimated from the repeated experimental data with thatfor each of the potential sources of uncertainty identified as not beingreflected in the repeated data variability (if any) and rank them in decreasing


numerical order. Those sources that have a SD less than 1/3 of the largest SDcan be ignored in the subsequent calculation of the combined uncertaintysince their contribution to the combined uncertainty will be negligible.

Calculation of the Combined Uncertainty

8. SDs cannot be manipulated to calculate the combined standard uncertainty.Instead, the SDs are converted to variances by squaring them and thevariances are used for the calculation of the combined standard uncertainty.The combined standard uncertainty is the square root of the sum of thesquares of the SDs (known as the Root Sum of Squares).

9. If RSDs have been calculated, the SD at a specific concentration C should becalculated by SD = RSD×C. This allows for taking the concentrationdependence of the SD into account. .(NMKL Procedure No. 5 (1997)Estimation and expression of measurement uncertainty in chemical analysis).

10. If no actual data is available, a first approximation of the inter-laboratoryreproducibility RSD is given by RSDR = 2×C(-0.15). The intra-laboratory RSD isone half of that (Official Journal of the European Communities L 77, 16.3.2001,p. 14). The formula is matrix and analyte independent but is unreliable at lowand high concentration extremes.

11. Precautions must be taken to not count the contribution of a source ofuncertainty more than once in the calculation of the combined standarduncertainty. The between run SD calculated from daily spike recoveries forexample, will include the variability found in the entire analytical processprovided the spike was inserted at the very beginning. This is also truehowever, of the SD calculated from the routine inclusion of any referencesample that is inserted at the very beginning of the analytical process.Calculating the combined standard uncertainty by using the SDs from both ofthese sets of data would double count all of the contributing sources andresult in an estimate of the measurement uncertainty that is too large. Theestablished procedure in such an instance is to use the larger of the two SDsin order to give a “worst case” estimate.

As an example, if we have established the between run SD from historicalspike recovery data to be SDspike, the bias uncertainty from a ProficiencyTesting Program to be SDPT and the sample non-homogeneity SD fromsample duplicate insertions to be SDhom and that no other sources ofuncertainty have an SD larger than 1/3 of the largest of these three, thecombined standard uncertainty SDC is given as:

SDC = (SDspike2 + SDPT

2 + SDhom2).

Applying the Coverage Factor “k”

12. The Expanded Uncertainty is derived by multiplying the Combined StandardUncertainty by a coverage factor “k”. The value of k for 95% coverage is


selected on the basis of the number of values “n” that are used for thecalculation for the SDs. If n ≥ 30, k = 2. If n < 30, k is the appropriate Student’st factor for n-1 degrees of freedom and a 95% confidence level.

Reporting the Result

13. The final concentration result C is then reported as C ± k×SDC with adescription of how the measurement uncertainty was calculated.

Uncertainty at the Limit of Detection and at the Limit of Quantitation

14. Only when a measured value is larger than the uncertainty with which it canbe measured does it have any credibility. This point is known as the Limit ofDetection (LOD). The lowest concentration at which a result can have ameaningful uncertainty assigned to it is the Limit of Quantitation (LOQ). TheLOD has been most commonly set at a concentration that gives a signal that is3 times the standard deviation of the measurement process at zeroconcentration or 3so. Similarly, the LOQ has been set at 10so.

15. The value for so is the Method Detection Limit determined as described in theCAEAL document Quality Control for Environmental Laboratories. This gives arelative uncertainty at the LOD and LOQ of ± 100% and ± 30% respectively,both with a 95% confidence level. (J.K. Taylor, Quality Assurance of ChemicalMeasurements Lewis Publishers Inc., pages 79-82 (1987).

Hierarchy of Data Selection for Estimation of Uncertainty

16. The following hierarchy is presented to provide laboratories with guidanceon which types of data they might use to estimate uncertainty within thelaboratory. This list is given in order of priority from (I) Most Suitable, to (IV)Least Suitable.

I. Uncertainty Specified within the Method. In those cases where a well-recognized test method (such as a peer-reviewed AOAC method orone published by agencies such as the Ontario MOE, the US EPA orASTM) specifies limits to the values of the major sources of uncertaintyof measurement and specifies the form of presentation of calculatedresults, the laboratory should follow the reporting instructions (seeNote 2 to Clause 5.4.6.2 of CAN-P-4D (ISO/IEC 17025).

NOTE: The laboratory would be expected to demonstrate that their resultsobtained when using this method have the reliability specified in themethod in order for this clause to apply.

II. Laboratory Control Samples (LCS) and Matrix Spikes. In cases wherematrix specific LCS and/or matrix spike data are available, includeuncertainty estimated from the standard deviation of the LCS ormatrix spikes of more than 50 points collected from their insertion intoroutine analytical runs. See paragraph 4 of this Appendix above.


III. Pooled Sample Replicate Data. In cases where sample replicates areanalysed and there is sufficient data above the limit of quantitation,include pooled sample replicate data to estimate uncertainty thatincorporates sub-sample uncertainty as a source. See paragraph 4 ofthis Appendix above.

IV. Proficiency Testing Sample Data. In cases where the previous options arenot available and where Proficiency Testing samples are analysed withsufficient data above the limit of quantitation, pooled ProficiencyTesting sample data can be used to estimate uncertainty. Seeparagraph 4 of this Appendix above.


Example Table to compile MU informationDescription

ofUncertainty

Source

Value x Uncertaintymeasured or

found

u(x) asStandardDeviation

u(x)/x Source of u(x)information

Steps to using this Table:

4. Define the measureand(s) – the analyte, the measurement objectives required fordata to be “fit-for-purpose” (includes LOD, precision, accuracy, analytical range,selectivity etc.)

5. List the anticipated sources of uncertainty (including parameters found in theequation used to calculate the final result to be reported).

6. List the repeated data sources (spikes, certified reference materials, in-housereference materials. duplicates, method validation files) both short term (one day orone run for example) and long term (over several months or longer).

7. Match the sources of uncertainty with repeat data that was collected while thesources of uncertainty may have varied. Long term spike recovery data may includechanges in analysts, calibration sets, and laboratory environment.

8. Identify those sources of uncertainty that are included in more than one repeat dataset. Both long-term spike and reference material standard deviation values willinclude uncertainty due to different analysts, calibration sets etc.; if these werevaried while the spike and reference material data were being collected in routineruns. Use only one of these two standard deviation values to estimate thecontribution to measurement uncertainty from the sources identified as being varied– usually the larger to be conservative. Alternatively, the two standard deviationscan be pooled and the pooled value included for compilation into the overallestimate of measurement uncertainty.

9. Estimate the uncertainty due to those sources that have not varied during thecollection of repeat data – either during method validation or routine analysis. Thismay involve using certificates for balances and masses or some other source ofuncertainty information.

10. Compile the information into the table above and check to ensure that a source ofuncertainty has not been counted more than once.

11. Remove those sources of uncertainty that have a standard deviation less than 1/3 thelargest standard deviation.

12. Combine the remaining standard deviations using root sum of squares (RSS)technique. (See paragraph 8 of this Appendix)

13. Multiply this combined standard deviation by the appropriate expansion factor todetermine the expanded uncertainty.

14. Ensure the data meets the fit-for-purpose criteria.15. If applicable, report the result with the expanded uncertainty. Indicate the

expansion factor (k) and the confidence interval (usually 95%).


APPENDIX 3Measurement Uncertainty for Microbiological Testing

Aim

1. This Appendix applies to environmental microbiological testing methodswhich:

a. Are quantitative but do not have normally (or Gaussian) distributed data.For example, the results from many microbiological tests, such as totalcoliforms in surface water, are considered to be generally described by thePoisson Distribution [14-20]; and/or

b. Are qualitative or semi-quantitative in nature (for examplepresence/absence types of tests, most probable number (MPN), strains ofbacteria, DNA testing).

Note 1: Data which is normally (Gaussian) distributed can behandled in the manner outlined in Appendix 2, wherever therequired information does exist. Data that are log transformed maybe amenable to data analysis for normally distributed data.Note 2: The Poisson distribution is not completely symmetrical, butcan be considered to be so for mean values greater than 25 to 30[16].

2. Four main types of Microbiological testing have been identified by APLAC[7]and are presented below. Note that Italicized text has been added for CAEALpurposes.

a. General quantitative procedures e.g. total coliforms in Surface waters.Depending upon the range of the colony counts, these general quantitativeprocedures may be considered semi-quantitative (for example in theregion of 1-10 or 20 counts).

b. MPN proceduresc. Qualitative procedures e.g. presence/absence type testing, The

microbiological analysis for E.Coli. in drinking water can be considered aQualitative procedure, since in Ontario at least, the limit is 0.

d. Specialist tests, e.g. pharmaceutical testing, or DNA testing

The applicability of measurement uncertainty (MU) to each of these typesof microbiological tests may differ.

3. This appendix considers and expands each of the CAEAL Protocol steps inmore detail. It also explains what is meant and how to perform each taskrequired.


Sources of uncertainty:

4. The possible sources of uncertainty for an analytical method are tabulated inmany of the sources listed in Paragraph 8 of this Policy. Close examination ofthe steps in the laboratory method SOP, and of the parameters found in thefinal concentration calculation, will usually help to identify the likely sourcesof uncertainty in the method. ILAC [4] lists these as:

a. definition of the measureandb. samplingc. transportation, storage and handling of samplesd. preparation of samplese. environmental and measurement conditionsf. the personnel carrying out the testsg. variations in the test procedureh. the measuring instrumentsi. calibration standards or reference materialsj. software and/or, in general, methods associated with the

measurementk. uncertainty arising from correction of the measurement results for

systematic effects.

5. For microbiological methods additional sources of uncertainty exist,including [16, 20, 21-23]

a. Colony count reliabilityb. microbial distribution

i. single plate ii. multiple plate count

c. Agglomeration and clustering of microbesd. Microbial Growth (Poisson Scattering)

i. difference in incubation time ii. difference in incubation temperature

e. Lack of Reference Materials, especially for quantitativemeasurements

f. Confirmation of identification and countsg. Volumetric issues

i. Dilution factors ii. Volumetric uncertainties

h. Procedural differences.

Niemela [22] and [19] has provided a good discussion of these sources ofuncertainty for microbiological methods. This document is highlyrecommended and can be downloaded from the internet athttp://www.mikes.fi/documents/upload/Publication%20J3%202002_1.pdf (currentlyit is free).


Laboratory Repeat Data Sets

6. As discussed in Paragraph 4 of the Appendix 2 of this Policy, ProficiencyTesting (PT) programs, Reference Samples, Spike Recovery, MethodValidation Replicate and Sample Duplicates are typical sources of LaboratoryRepeat Data Sets and are sources of repeated measurements from which SDsand RSDs can be calculated. The laboratory can vary one or more of the abovesources of uncertainty during the collection of the repeat data and the SDcalculated will include uncertainty attributed to the varied source(s).

Note: There are limitations with the use of any these for microbiologyexcepting sample duplicates:

• Proficiency testing programs – These are a source of reproducibilitySD (SDR) that includes both intra- and inter-laboratory sources ofuncertainty. It is larger than the intra-laboratory uncertainty (SDr) –known as repeatability - of a laboratory whose methods are instatistical control. Unlike the case for typical chemical analysis, the datafrom proficiency test programs alone is often inadequate to establish thequality of a microbiological analysis.

• Reference sample insertion – both certified reference samples and in-house reference samples inserted into routine runs for control chartingapplications are a source of long term uncertainty data in chemicalanalysis. There is a general lack of reference materials available for routineuse in microbiological methods. However there are exceptions, such as areference material with a specified minimum number of organisms, or thepreparation of materials in house [16, 24] At least one vendor claims to be ableto provide freeze dried pellets which will provide a guaranteed range of colonyforming units (CFU).

• Spike recovery data – Can give the same information as referencesample insertion and in some cases can reflect variability due todifferent sample matrices. There is a general lack of materials available forroutine use in quantitative spiking for microbiological methods. Howeverthere are exceptions, the Ontario Ministry of the Environment has preparedspiking solutions for certain bacteria. While the procedure can be timeconsuming and is not performed on a routine basis, properly applied, it canprovide useful information [24]

• Method validation replicate data – This is a source of data from repeatanalyses run to establish precision estimates at different analyteconcentration (or measurement –i.e. colony forming units) levels. Forthe same reasons as the reference materials and spikes, this information isdifficult to obtain and is less useful for microbiological tests. A thoroughdiscussion of the use of method validation data in the estimation ofuncertainty in chemical analysis is VAM Protocol [9] which wasavailable from the VAM web site (www.vam.org.uk).


• Sample duplicate insertion –– known as replicability SDdupl –. TheSDdupl reflects the variability due to differences between analyticalportions (non-homogeneity) and other factors that can vary betweenreplicates and the distribution of counts. In general for microbiologicaltests this will be the most readily available source of uncertainty data

Match Repeat Data with Uncertainty Sources

7. The objective of this step is to select laboratory Quality Control andvalidation data that includes as many sources of variability as possible so thatthese do not have to be estimated using the more difficult (and timeconsuming) Type B approach. The most effective means of achieving this isto design the analytical method to ensure spikes, reference samples andduplicates are inserted as early as possible into the analytical run. In the caseof microbiological testing, sample duplicates will represent the primary source of datafor uncertainty. Microbiological laboratories are encouraged to consider the use ofruggedness testing.[9]

It should be noted here that while this policy recommends the use of the Type"A” approach (use of statistical data to estimate uncertainty), uncertaintiesestimated using the Type “B” approach (estimation of the uncertaintythrough the mathematical manipulation of all relevant contributions to theoverall estimate), are acceptable provided they are complete.

Estimate the Uncertainty for any Sources not Accommodated by RepeatedData

8. In the unusual cases where it is necessary to estimate uncertainties for anysources not accommodated by repeated data, the estimation of theuncertainty from these sources is based on information from manufacturerspecifications that accompany instruments and equipment (such asvolumetric ware), tabulated data from handbooks, experience from othermethods and/or laboratories and other sources. Examples of thesecalculations are found in the EURACHEM/CITAC guide [3] which isavailable for downloading from www.measurementuncertainty.org.Unfortunately none of these references cover microbiological tests. However Niemela[22] has provided some information on observed and computed values of componentsof uncertainty which may be useful to laboratories in the absence of other information.This type of information will be accepted by CAEAL provided it is clearly referenced,copies of the cited reference are maintained, and the information is applied properly tothe uncertainty determination.

Calculation of the Combined Uncertainty

9. Precautions must be taken to not count the contribution of a source ofuncertainty more than once in the calculation of the combined standarduncertainty. The combined uncertainty can be determined from the standarduncertainties for the different components. The combined uncertainty iscalculated base upon the fact that variances are additive. In practice for


microbiological testing it may be found that the uncertainty associated with one of thecomponents overwhelms the contribution of all the other sources (i.e. the RSD orstandard uncertainty is 3 times larger than any other components), in which case theothers may all be considered negligible.

Applying the Coverage Factor “k”

10. The Expanded Uncertainty is derived by multiplying the Combined StandardUncertainty by a coverage factor “k”. The value of k for 95% coverage isselected on the basis of the number of values “n” that are used for thecalculation for the SDs. If n ≥ 30, k = 2. If n < 30, k is the appropriate Student’st factor for n-1 degrees of freedom and a 95% confidence level.

Note: Although it is possible to calculate the Expanded Uncertainty formicrobiological tests in this manner, it will not typically be required to bereported in this manner by CAEAL for these parameters. If the expandeduncertainty is reported at the 95% confidence interval based upon the resultsfrom proficiency testing then there is no need to apply a coverage factor.However if a laboratory is making use of a RSD reported for a using any ofthe other Laboratory Repeat Data Sets described in 22 above, then a coveragefactor of 2 should typically be used.

Uncertainty at the Limit of Detection and at the Limit of Quantitation

11. The concept of LOD and LOQ are much less straightforward in microbiology.The principal application would be to presence absence type of tests (such asfor E.Coli in drinking water). In this case the laboratory should at aminimum provide information on the false positive and false negative rates.If these rates cannot be determined an explanation as to why not should bemaintained in the method records and reviewed on a regular basis.

Reporting the Result

12. For microbiological tests the final result C (in counts per mass or volume) isthen reported asa. C ± k×SDC with a description of how the measurement uncertainty was

calculated. For microbiological tests measurement uncertainty willtypically only be reported for Quantitative measurements (withnumerical uncertainty values)

b. for semi-quantitative the results (in the range of 1-20) in the absence oflaboratory QA data, the results should be noted as semi quantitative innature

c. for qualitative test the negative result should be reported as zero with anupper value for false negative rates( or similar value

Hierarchy of Data Selection for Estimation of Uncertainty

13. The following hierarchy is presented to provide laboratories with guidanceon which types of data they might use to estimate uncertainty for


microbiological measurements within the laboratory. This list is given inorder of priority from (I) Most Suitable, to (IV) Least Suitable.

I. Uncertainty Specified within the Method. In those cases where a well-recognized test method (such as a peer-reviewed AOAC method orone published by agencies such as the Ontario MOE, the US EPA orASTM) specifies limits to the values of the major sources of uncertaintyof measurement and specifies the form of presentation of calculatedresults, the laboratory should follow the reporting instructions (seeNote 2 to Clause 5.4.6.2 of CAN-P-4D (ISO/IEC 17025).

NOTE 1: The laboratory would be expected to demonstrate that theirresults obtained when using this method have the reliabilityspecified in the method in order for this clause to apply.

NOTE 2: The laboratory must review the reported uncertainty todetermine if it is a standard uncertainty, combined uncertaintyor expanded uncertainty. For example the USEPA hasreported a single laboratory precision of 3.3 to 27.3 %(depending upon the range) for E.Coli in drinking water [25]this would need to be multiplied by 2 for the ExpandedUncertainty. Another recent USEPA document [26] indicatesthat a particular E. Coli. method has as 4% false negative rate.

II. Pooled Sample Replicate Data. In cases where sample replicates areanalyzed and there is sufficient data for each of the cases in II (i), (ii) and(iii) below, include pooled sample replicate data to estimate uncertaintythat incorporates sub-sample uncertainty as a source. Sufficient data areconsidered to be a minimum of 30 sets (i.e. minimum of 60 results).i. Samples above 10 Counts: express MU as an uncertainty interval.ii. Samples with 1 to 10 Counts: semi-quantitative, express MU as an

uncertainty interval.iii. Samples with < 1 Counts: qualitative, express MU as a false negative

rate.

III. Laboratory Control Samples (LCS) and Matrix Spikes. In microbiologicaltests, matrix specific LCS and/or matrix spike data are typicallyunavailable. However if available, this type of data can be used toprovide an estimate of MU provided a sufficient number of data pointsexist. Typically data from a minimum of 30 analyses will be required.

IV. Proficiency Testing Sample Data. In cases where the previous options arenot available and where Proficiency Testing samples are analysed withsufficient data for each of II (i), (ii) and (iii) above, pooled ProficiencyTesting sample data can be used to estimate uncertainty. Use of PT sampledata will often result in a much larger MU than a single lab will have. Forexample the results from the analysis of CAEAL PT samples forMicrobiological and Toxicity test parameters [27] are shown in the table 1below. The MU of 0.30 (or 30%) is a 3X larger than the MU calculated forone of the participating laboratories [28]. It is important for a lab to


understand the sources of uncertainty, which were included in the systemthat produced the PT result(s). Since only four samples are analysed perround, and there are two rounds per year, it could take a substantiallength of time to accumulate the 30 samples for each range of analyses.

Table 1: CAEAL Proficiency Testing 1991-1999 Measurement Uncertainty Results [27]Parameter Concentration or

CountsSlope(MU)

Intercept

Low HighFecal Coliform 14 82 0.35 4.5Total Coliform 5 220 0.30 3.6

Average microbiology 0.324Trout LC50 (96 h) 2.077 8.491 0.248 0

Daphnia LC50 (96 h) 2.185 38.357 0.159 1.89Microtox (15 min.) 4.390 10.784 0.301 0

Average ToxicityTesting

0.236

Note: slope is calculated based upon the 95% confidence intervals.

Some Suggested Procedures for Obtaining Measurement Uncertainty Data forMicrobiology Testing

Quantitative Testing

14. Log Transforming Data: A number of authors have recommended the use oflog transformed data [16, 7, 21, 22, 29]for determining statistics, includingstandard uncertainties. The CAEAL policy does not advocate or restrict thistreatment of data. However it is recommended that at least the first set ofdata (minimum 30 sets) be analysed both with and without log transforming.

15. The NMKL procedure[16] provides a simple way for evaluating theuncertainty of quantitative microbiological testing on a day to day basis usingdata from duplicates and/or parallel plating. These include

a. Poisson variance : for plating methodsb. Possion Index of Dispersion (D2)c. Extra Variance (u)d. Calculation of Routine Parameter (Q) and Steering Diagram

Note that for ease of reference, the original Formula number from NMKLhas been retained.

16. Poisson Variance: During the initial method validation, the laboratory shouldcollect data for determining data distribution. Formula 1 provides a simplemethod for determining an approximate of the 95% Confidence Interval forthe case where more than 25 to 30 colony counts are found and the Poissondistribution is applicable.


Where:Y=estimate of probable number of micro organisms per unit (in this case,

Volume (V))Var=varianceSD=standard deviationCI=confidence interval

17. Poisson Index of Dispersion: A test knows as the Dispersion test has beensuggested by Niemela [19] to evaluate if variance is larger in the case of twoparallel results (A and B) than would be expected from the Poissondistribution assumption. The test parameter (D2) is calculated using Formula5.

Where, in this case:A= reading from filter 1B=reading from filter 2D=dispersion parameter

If the D2 value is less than 4, it is approximately 95% probable that the Poissondistribution assumption has been met. It can be seen that as the sum of A andB increases, the deviation that can be tolerated for the Poisson distributionincreases also.


Formula 6 can be used for comparing dilutions for the Poisson distribution,where V1 and V2 represent results from the same sample with differentdilutions.

18. Calculation of extra variance, u: In the case where repeated tests show agreater variance than predicted by the Poisson distribution, the extra varianceis calculated using the factor u from Formula 7 from the NMKL procedure.Note that the factor u is matrix and test method specific.

the factor u can then be used to calculate the 95% CI using Formula 10.

A different u value will result if several analysts are pooling their results, ifdifferent dilutions are being used etc.. The value of u may be found to becount level dependent (e.g. for 1-10, 10-100, 100-1000) in which case differentu values should be used. Note that if u values change significantly over time,with the same procedures, the method must be reviewed.

19. Steering Diagram: The use of the routine parameter Q, suggested by Niemela[19] and NMKL is a precision measure for parallels A and B that is similar inconcept to repeatability and reproducibility. The parameter Q is calculatedusing Formula 8.


The value of Q is expected to be less than 2 and similar for all analyses. Itcan be plotted over time using a so-called “steering diagram”, in a mannersimilar to the typical Shewart control charts used for many chemicalanalysis methods. Note that the use of logarithms serves to minimise theeffects of results which may diverge. NMKL has indicated “this Q variableis probably the one most appropriate for use in routine laboratory operations as itis designed to provide results using the same unit and limit value for allanalyses.”

20. Calculation of Combined Uncertainty: In addition to the uncertaintyassociated with the microbiological colony distribution, microbiologylaboratories must also consider the uncertainty contributions from otheridentified sources. These may include contributions in measuring volume(initial filtration volume, dilutions, parallels volumes, etc.), media, differentsample preparations, and the inter- and intra-analyst variation in colonycounting. The exact factors or sources, which must be examined, will dependon the laboratory.

The combined uncertainty is then calculated from the standarduncertainties for each of the components using Formula 11

As indicated in the main body of this Policy, and in Appendix 2, anystandard uncertainties which are 1/3 or less than the largest uncertaintymay be ignored.

21. Calculation of Expanded Uncertainty: The Expanded Uncertainty can becalculated in several ways. It can be calculated directly from RSD (or SDR)information by multiplying by a coverage factor (k=2) to give the ExpandedUncertainty. In the case where a combined uncertainty has been calculatedthe Expanded Uncertainty is determined using Formula 12 below

22. NMKL has recently updated their 1999 document [23] and has prepared anelectronic calculation aid (in MS-Exceltm) that can be downloaded from theirwebsite at www.nmkl.org and which laboratories may find useful.Laboratories are still responsible for validating the use of the software.


23. In addition to the procedures and Formula provided above, the recentdocument by Niemela [22] is available for download athttp://www.mikes.fi/documents/upload/PublicationJ3202002_1.pdf and provides adetailed discussion on combining uncertainty components from the Poissondistribution, inter-analyst colony counting, volumetric and dilution factorsetc.. While the steps outlined in the Niemela document are not compulsoryfor microbiology testing laboratories, they are highly recommended.

Most Probable Number Methods

24. MPN Methods: The Draft APLAC MU Guidance [7] accepts the data in theMcCrady’s tables as reasonable estimates of MU. For the purposes of thisPolicy, they can be used as estimates of MU for a test, provided the laboratoryreviews the results of unusual combinations and flags or rejects such data. Asknowledge advances in this aspect of MPN testing, this aspect of the Policywill be reviewed.

A procedure has been suggested by NMKL and Niemela [16, 22] whichallows laboratories to calculate a standard deviation for MPN results. Forsamples in which positive results are found, the relationship in Formula 2is considered useful.

Where:Y=most probable numberV=volumen=total no. of tubesq=no. of negative tubes

Note that the CI is skewed.

In many cases this uncertainty will usually overwhelm all other sources(such as volumetric etc.).

Qualitative Tests


25. Qualitative Examinations: For some microbiology tests the result is moresemi-quantitative such as E. Coli in drinking water, where there must be zerocolony forming units; or qualitative such as presence/absence testing. Inthese instances, the “worst” case involves false negatives due to some error.

The following Formula 3 can be used to express an upper limit on thenumber of positives that might be expected in the case where all sampleswere negative. Sample homogeneity information would need to identifyif all samples were from the same location, the same time, etc.

Where:P=confidence level requiredp=maximum possible positive raten= no. of samples that test negative

In order to increase statistical validity using this approach, more tests must berun compared to other approaches.

Laboratories are encouraged to utilise a number of low-level samples todetermine false negative rates.

In addition to the false negative rates, the measurement uncertainty shouldinclude an evaluation of the positive and negative controls.

A Shortcut to Uncertainty of Multiple Plate Instruments

26. Maximum log-likelihood ratio statistic G2: The differences between thecolony numbers observed are partly due to design (volumes and dilutions)and partly due to random variation (Poisson scatter, volume uncertainties,uncertainty of reading counts). The uncertainty due to design can beremoved by using the log-likelihood ratio statistic, G2. The uncertainty of themicrobial count can then be estimated directly without using separateuncertainty components. The microbial concentration of the final suspensioncan be calculated from the weighted mean using formula 63 [22]:

where:zi=the observed colony count of the ith platevi=the volume of the final suspension inoculated into the ith plate

combined relative uncertainty for the test result (wy=ucombined/c) can be calculatedusing Formula 64 [22]


27. To obtain the relative uncertainty estimate that includes all the randomcomponents affecting counts within the instrument, the log-likelihood ratiostatistic G2 is computed using Formula 66 [22].

28. The relative standard uncertainty squared (or variance) of the colony count inFormula 64 is calculated using Formula 67 [22].

Comparisons to Norms, Guidelines or Regulations

29. Comparison to a norm or guideline: In the case where a test result is to becompared to a regulatory limit, norm or guideline, the observed value should


include the confidence limits as calculated in Formula 1. However, analternative method to Formula 1, using the chi-test can be used. The chi-testparameters are shown in Formula 4 below

For a 95% confidence level the critical value is 4 (approximated from 3.84=1.962).

Where:x2=chi squared valueC=no. of coloniesL= limit value

Note that this Policy is based on statistically valid sampling. Regulatory agenciesmay specify other ways of determining compliance with a guideline or standard.Where a regulatory guideline or standard requires an approach different that thisPolicy, the regulatory guideline or standard shall govern.

Limitation on Uncertainty

30. The boundaries for the system for which the uncertainty is being determinedmust be recognised and specified. If the laboratory only determines thesources of uncertainty after the sample has been received, the uncertaintyestimate represents only part (possibly only a small part) of the uncertaintyassociated with the final result.

31. The following Table 2 contains a listing of all equations/formula used in thisappendix, along with the original reference location.


Table 2: Summary of Formula Presented in Appendix 3FormulaNo.

Formula Equation Comments Referenceand Page

1 C= countsV= volumeY=counts pervolumeVary=varianceSD= StandardDeviation

NMKL[16]page 9

2 N=no. of tubesq= no. of negativetubesV= volume ofsample in each tube

NMKL[16] page 11

[22] page 40


Formula No.

Formula Equation Comments OriginalReference andPage

3 P= desiredprobabilityn= no. of negativesamplesp= max. positive rate

NMKL[16]page 12

4 C= countsL= limit value_2= chi squared teststatistic=approximately 4 for95% probability

NMKL[24]page 13

5 D2=dispersion testparameterA= parallel platereadingB= second parallelplate reading

NMKL[16]page 15

6 V1=initial volumeV2= dilution volumeC1=undiluted countsC2=diluted counts

NMKL[24]page 16


Formula No.

Formula Equation Comments Reference andPage

7 u= extra variancemeasureC= counts

NMKL[16]

page 188 Q=test value or

routine parameterNMKL[16]

page 18

10 NMKL[16]page 2

11 uc= combineduncertainty

Adapted fromCITAC [2]

12 k=coverage factor(generally 2-3)uc=combinedstandarduncertainty

Adapted fromCITAC [2]


Formula No.


63 c=meanconcentrationzi=the observedcolony count of theith platevi=the volume of thefinal suspensioninoculated into theith plate

Niemela [22]Page 42

64 wc=relativestandarduncertainty ofcolony countingwF=relativestandarduncertainty ofdilution factor

Niemela [22]Page 42


FormulaNo.


66 zi=colony counton ith platevi=test portionvolume (finalsuspension) ith

plateN=number of testplates(filters)Z=sum of allcolony countsV=sum of all testportion volumes

Niemela [22]Page 42

67 Niemela [22]Page 43

p19 Caeal Unce Pol Rev 1-4

Documents

Transcript of p19 Caeal Unce Pol Rev 1-4