Quality Assurance of Monitoring/Surveillance Data -...
Transcript of Quality Assurance of Monitoring/Surveillance Data -...
June 12, 2013
Quality Assurance of Monitoring/Surveillance Data
Takanori UKENA, Ph.D.
Ministry of Agriculture Forestry and Fisheries
Food Safety and Consumer Affairs Bureau
1 MAFF
Contents
Importance of Quality Assurance
1. Laboratory Management, ISO/IEC 17025
2. Use of Validated Analytical method
3. Internal Quality Control and Proficiency
Testing
4. Measurement Uncertainty
5. Evaluation of Analytical Results
Experience of MAFF Japan 2 MAFF
What is Quality?
“degree to which a set of inherent characteristics fulfils requirement".
⇒ In case of analysis
“delivery of reliable data/results within an agreed span of time under agreed conditions, at agreed costs, and with necessary aftercare". .
The “agreed conditions” should include a specification as to the precision and accuracy of the data
4
(ISO 9000)
MAFF
Quality Management
(ISO/IEC 17025:2005)
Quality management: coordinated activities to direct and control an organization with regard to quality
6 MAFF
(ISO 9000:2005)
The laboratory shall establish, implement and maintain a management system appropriate to the scope of its activities.
Quality management system: management system to direct and control an organization with regard to quality
Note : generally includes establishment of the quality policy, and quality objective, quality planning, quality control, quality assurance, and quality improvement.
Quality Assurance
Quality assurance:
All those planned and systematic actions necessary to provide adequate confidence that analytical results will satisfy given requirements for quality.
GUIDELINES ON ANALYTICAL TERMINOLOGY (CAC/GL 72-2009)
7 MAFF
part of quality management focused on providing confidence that quality requirement will be fulfilled
(ISO 9000:2005)
Quality Control
Quality Control:
part of quality management focused on fulfilling quality requirement
8 MAFF
(ISO 9000:2005)
for chemical analysis laboratories
⇒ Internal Quality Control and Proficiency testing.
Why quality assurance is needed?
Prevention of quality
problems
Risk manager need to make decision based on data obtained from other persons, other laboratories, other countries.
Basis of cconfidence in the data
International requirement
9 MAFF
What is reliable analytical results?
Use of validated method
If you don’t use validated methods, only you will know
how good your numbers are.
By a laboratory with Quality Management Internal quality control and proficiency testing
Fit for purpose Data produced by a measurement process enables a user to make technically and administratively correct decisions for a stated purpose.
10
Back ground information/data MAFF
Use of analytical results
•Conformity assessment
Import/export control
•Risk management activities Occurrence data for estimation of exposure
Evaluation of effectiveness of risk management options
•International activities
GEMS/Food, Codex, JECFA, JMPR, etc.
11 MAFF
Inadequate data (example)
The data from Brazil and Uruguay could not be used as they had been obtained by analytical methods with high LODs, ranging from 5 µg/kg to 50 µg/kg, which are inadequate to detect and measure ochratoxin A at 0.94 µg/kg and 0.19 µg/kg, the weighted mean concen-trations found in Europe and the USA in cereals and cereal products, respectively.
5.3 Distribution
(JECFA 47, 2001)
12 MAFF
Codex guidelines for Import and Export
Control testing laboratories
•Compliance with the general criteria in
ISO/IEC 17025
•Participation in appropriate proficiency
testing schemes
•Use of validated method
•Internal quality control procedures
Guidelines for the Assessment of the Competence of Testing Laboratories Involved in the Import and Export Control of Food (CAC/GL 27-1997) 13 MAFF
Food Control Laboratory Management:
Recommendations
•International Harmonized Protocol for the Proficiency
Testing
Pure & Appl. Chem., 65 (1993) 2132-2144
•Protocol for the Design, Conduct and Interpretation of
Method Performance Studies
Pure & Appl. Chem., 67 (1995) 331-343
•Harmonized Guidelines for Internal Quality Control
Pure & Appl. Chem., 67 (1995) 649-666
Food Control Laboratory Management: Recommendations (CAC/GL 28-1995, Rev.1-1997)
Protocols and Guidelines adopted in Codex
14 MAFF
Guidelines for Settling Disputes over
Analytical (Test) Results
the following information should be shared between competent authorities of the importing and exporting country
(CAC/GL 70-2009)
• validation status of the analytical method used
• raw data
• internal quality assurance/control
(control charts, sequence of analysis, blank data, recovery data, etc.)
•performance in relevant proficiency testing or collaborative studies;
• official accreditation status of the laboratories.
15 MAFF
Why is ISO/IEC 17025 important?
Requirements of Codex guidelines
ISO an International standard / international
recognition
Benefit of accreditation for laboratory
•A marketing advantage for laboratory
often required for international trade
•A bench mark for performance
In depth assessment
Required periodic reassessment
An indicator of technical competence 17 MAFF
Requirements of ISO/IEC 17025
Management (15)
•Organization
•Management system
•Document control
•Review of requests/contracts
•Subcontracting of tests
•Purchasing services/suppliers
•Service to the customer
•Complaints
•Control of non-conforming tests
•Improvement
•Corrective action
•Preventive action
•Control of records
•Internal audits
•Management review
Technical (10)
•General
•Personnel
•Accommodation and environmental conditions
•Test method and method validation
•Equipment
•Measurement traceability
•Sampling
•Handling of test and calibration items
•Assuring quality of test and calibration results
•Reporting the results
18 MAFF
What is Accreditation of ISO/IEC 17025?
Formal recognition by an authoritative body of the competence of a
laboratory to carry out specific tests or types of tests in a reliable,
credible and accurate manner
Evidence of Quality, Credibility and Reliability
•technical competence of staff
•validity and appropriateness of test methods
•suitability, calibration and maintenance of equipment
•appropriate handling of test materials
•quality control procedure, e.g., use of CRM, Proficiency testing, etc.
Actual performance and ethical behavior is critical. Not a guarantee
19 MAFF
Accredited
to each analytical method
each laboratory Need to confirm :
For what method the laboratory is accredited?
Which laboratory is accredited?
If the organization has blanch laboratories.
Notes for Accreditation of ISO/IEC 17025
20 MAFF
ILAC, APLAC
ILAC: International Laboratory Accreditation Cooperation
The international organization on laboratory accreditation with membership of accreditation bodies and affiliated organizations.
Maintain a Mutual Recognition Agreement (MRA) among accreditation bodies to facilitate trade and to eliminate the need for duplicate accreditation.
Assist developing accreditation programs
APLAC: Asia Pacific Laboratory Accreditation Cooperation
21 MAFF
ILAC-MRA, APLAC-MRA
A laboratory accredited by one MRA partner has
equivalent competence to laboratory accredited by the
other MRA partner(s)
https://ilac.org/home.html http://www.aplac.org/home.html
Accreditation body shall comply with ISO/IEC 17011 and that its accredited facilities are in compliance with ISO/IEC 17025 (laboratories), ISO 15189 (medical laboratories) and ISO/IEC 17020 (inspection bodies).
22 MAFF
“Tested once, accepted everywhere”
Chemical Analysis
Qualitative
•the presence or absence of a particular substance, but
not the mass or concentration.
•Identification of the substance
e.g., structure elucidation of natural product
Quantitative
•determination of the absolute or relative abundance
(concentration) of particular substance(s) present in a
sample. 24 MAFF
Analytical method for Risk
Management
Analytical Methods
•Whenever
•Whoever (analytical specialist)
can follow the method procedure and obtain
similar results for the same sample.
(e.g., variability of results fall in expected interval).
25 MAFF
Validated method
“If you don’t use validated methods, only you will
know how good your numbers are.” AOACI
Validated Test Method: An accepted test method for which validation studies have been completed to determine the accuracy and reliability of this method for a specific purpose. Reference: ICCVAM Guidelines for the nomination and submission of new, revised and alternative
Guidelines on Analytical Terminology (CAC/GL 72-2009)
26 MAFF
Validated method is a method tested in many laboratories on the same samples, showing that the method is rugged enough to produce comparable results in different labs, with different operators.
Method Validation Study
Collaboration study, Ring-test- trial
27 MAFF
Validation Study Protocol and Guidelines
AOAC International AOAC Official Methods of Analysis (2002). Interlaboratory Collaborative Study, Appendix D: Guidelines for Collaborative Study Procedures To Validate Characteristics of a Method of Analysis http://www.aoac.org/vmeth/guidelines.htm
ISO 5725-2:1994 Accuracy (trueness and precision) of measurement methods and
results -- Part 2: Basic method for the determination of repeatability and reproducibility of a standard measurement method
IUPAC Protocol for the Design, Conduct and Interpretation of Method-Performance Studies. Pure & Appl. Chem., 67(2), 331-343 (1995)
28
http://www.iupac.org/publications/pac/67/2/0331/
MAFF
Comparison of Protocol for the study
* test statistics and application procedure are different between the protocols . 29 MAFF
General Criteria in Codex Procedure Manual
Single-laboratory Validated Method
single-laboratory validated methods must fulfill the following criteria:
(i)the method is validated according to an internationally recognized protocol (e.g. IUPAC Guidelines)
(ii) the use of the method is embedded in a quality system in compliance with the ISO/IEC 17025
30 MAFF
The method should be complemented with information on accuracy demonstrated for instance with:
–regular participation in proficiency schemes, where available;
–calibration using certified reference materials, where applicable;
–recovery studies performed at the expected concentration of the analytes;
–verification of result with other validated method where available.
31 (Codex Procedural Manual)
MAFF
Harmonized Guidelines for Single-Laboratory Validation of Methods of Analysis
IUPAC/ISO/AOAC, 2002
Guidelines on Single-Laboratory Validation
Performance characteristics
•Applicability •Selectivity •Calibration and linearity •Trueness, Precision •Recovery •Limit of Detection, Limit of Quantification •Sensitivity •Ruggedness
etc.
32 MAFF
Recommended methods in Codex
Recommended Methods of Analysis and Sampling (CODEX STAN 234-1999)
General Methods of Analysis for Contaminants (CODEX STAN 228-2001)
33 MAFF
Criteria approach
Identify method performance characteristics based on existing method validation data.
and establish criteria for evaluating acceptable method of analysis
⇒ laboratory can choose a method meeting criteria (flexibility)
34 MAFF
Guidelines for Criteria approach
<Codex Procedural Manual>
Working Instructions for the Implememtation of
the Criteria Approach in Codex
Guidelines for Establishing Numeric Values for
Method Criteria and/or Assessing Methods for
Compliance Thereof
35 MAFF
Examples for numeric values for the criteria
* The sR should be calculated from the Horwitz / Thompson equation 36 MAFF
Terms related to variability of test results
Accuracy
Trueness Precision
Repeatability Reproducibility
Measurement Uncertainty
Intermediate precision
38 MAFF
Distribution of results for repeated analysis
m
A
C µ: True value m: mean X-axis:
concentration Y-axis: frequency
B D
µ µ
µ µ
m
m m 39 MAFF
accuracy, trueness, precision
accuracy
trueness
precision
○
○
○
×
○
×
×
×
○
µ (true value); m (mean); σ (standard deviation); c (m+s)
m m
σ
m
s
40 MAFF
A B C
Accuracy
The closeness of agreement between a test result or measurement result and a reference value.
•When applied to a test method, the term accuracy refers to a combination of trueness and precision.
Guidelines on Analytical Terminology (CAC/GL 72-2009)
41 MAFF
A reference value is usually provided with reference to a) a certified reference material; b) a reference measurement procedure; c) a comparison of measurement standards.
Trueness The closeness of agreement between the average
of an infinite number of replicate measured
quantity values and a reference quantity value.
•Measurement trueness is inversely related to systematic measurement error, but is not related to random measurement error.
• Measurement accuracy should not be used for ‘measurement trueness’ and vice versa.
Guidelines on Analytical Terminology (CAC/GL 72-2009)
42 MAFF
Precision The closeness of agreement between independent
test/measurement results obtained under
stipulated conditions.
•depends only on the distribution of random errors and does not relate to the true value or to the specified value.
•usually expressed in terms of imprecision and computed as a standard deviation of the test results. Less precision is reflected by a larger standard deviation.
•Quantitative measures of precision depend critically on the stipulated conditions. Repeatability and reproducibility conditions are particular sets of extreme conditions.
Guidelines on Analytical Terminology (CAC/GL 72-2009) 43 MAFF
Repeatability
Precision under repeatability conditions.
Repeatability conditions:
Observation conditions where independent test/measurement results are obtained with the same method on identical test/measurement items in the same test or measuring facility by the same operator using the same equipment within short intervals of time.
Guidelines on Analytical Terminology (CAC/GL 72-2009) 44 MAFF
Reproducibility
Precision under reproducibility conditions.
Reproducibility conditions:
Observation conditions where independent test/measurement results are obtained with the same method on identical test/measurement items in different test or measurement facilities with different operators using different equipment.
Guidelines on Analytical Terminology (CAC/GL 72-2009) 45 MAFF
Relative standard deviation
RSDr : Repeatability relative standard deviation
RSDR : Reproducibility relative standard deviation
Relative standard deviation (%)
= standard deviation (SD) / mean ×100
same as CV (coefficient of variation)
1
221
++
n
)μx()μx(SD n
46 MAFF
Horwitz equation
RSDR (%) = 2 C -0.1505
RSDR (%): reproducibility relative standard deviation
C:concentration ratio
if concentration ratio =100 %, then C = 1
1 %, C = 0.01
A ppm C = A×10-6
C.-
R RSD 10log5012(%)
Horwitz equation
approximation of Horwitz equation
47 MAFF
Graphical expression of Horwitz equation
48 MAFF
-200
-150
-100
-50
0
50
100
150
200
相対標準偏差[%]
1 ppm 1 ppb 1 ppt 1%
16% 45%
128%
4%
RS
DR
Horwitz/Thompson equation
RSDR (%)=
Thompson, M., Analyst, 125, 385-386 (2000)
22
C-0.5
2 C-0.1505
C < 1.2×10-7
C > 0.138
1.2×10-7 ≤ C ≤ 0.138
RSDR (%): Reproducibility relative standard deviation
C:concentration ratios
49 MAFF
Thompson, M., Analyst, 125, 385-386 (2000)
-30
-20
-10
0
10
20
30
Concentration
Relative
standard de
viation[%
]
1 ppt 1 ppb 1%
16%
22%
1 ppm
0.1ppm
Graphical expression of
Horwitz/Thompson equation
50 MAFF
HorRat Value
HorRat(R) = RSDR/PRSDR HorRat(r) =RSDr/PRSDR
RSDr, RSDR: observed relative standard deviation
(sr/C, sR/C)
PRSDR: predicted relative standard deviation
calculated from Horwitz/Thompson
Normal range of HorRat
The ratio of the reproducibility relative standard deviation to that calculated from the Horwitz equation,
Guidelines on Analytical Terminology (CAC/GL 72-2009) 51 MAFF
What is Limit of Detection?
Limit of Detection (LOD) :
The true net concentration or amount of the analyte in the material to be analyzed which will lead, with probability (1-β), to the conclusion that the concentration or amount of the analyte in the analyzed material is larger than that in the blank material.
Critical Value (Lc) :
The value of the net concentration or amount the exceeding of which leads, for a given error probability α, to the decision that the concentration or amount of the analyte in the analyzed material is
larger than that in the blank material.
53 MAFF Guidelines on Analytical Terminology (CAC/GL 72-2009)
LOD and Critical Value (Lc)
Lc Results of blank
α: Type I error (false positive)
β: Type II error (false negative)
0
Results of LOD concentration sample
LOD
Sample of which concentration > LOD will lead to the decision that the concentration of the material is larger than that in the blank material with probability larger than 1-β
If results > Lc, then they will lead to the decision that it is not blank, detected with probability larger than 1-α.
If results are lower than Lc, they are N.D. (could not decided to be detected)
54
β
MAFF
The limit of detection LOD is estimated by,
LOD ≈ 2×t1-αν σo [where α = β], —Where
t1-αν : Student's-t, based on ν degrees of freedom for a one- sided confidence interval of 1-α
σo : the standard deviation of the true value (expectation).
LOD = 3.29 σo, when the uncertainty in the mean (expected) value of the blank is negligible, α = β = 0.05
Usually estimated by using variance of sample (SD of True value is not known.) LOD depends on probability (α and β) Calculate t-value depending on number of data
(LOD should not always equals to 3.29×s (SD. observed))
55 MAFF
α=0.05:
Degree of Freedom = 1 ⇒ t 0.95,1 = 6.31 (2×t-value=12.62)
Degree of Freedom = 10 ⇒ t 0.95,9 = 1.81 (2×t-value=3.62)
Degree of Freedom = ∞ ⇒ t 0.95,∞= 1.64 (2×t-value=3.29)
t-distribution, df=1
t-distribution, df=3
t-distribution, df =10
Standard normal distribution
t-Distribution and Standard Normal Distribution
56 MAFF
Limit of Quantification
Limit of Detection (LOD) : A method performance characteristic generally expressed in terms of the signal or measurement (true) value that will produce estimates having a specified relative standard deviation (RSD), commonly 10% (or 6%).
LOQ is estimated by:
LOQ = kQ ×σQ , —Where kQ = 1/RSDQ
If σ is known and constant (SD of the estimated quantity is independent of concentration) , and σQ = σo :
substitution of kQ = 10% ⇒ LOQ = (10×σQ) = 10 σo
(3.04 ×LOD, α = β = 0.05)
57 MAFF Guidelines on Analytical Terminology (CAC/GL 72-2009)
Harmonized Guidelines for Internal Quality Control in Analytical Chemistry Laboratories, Pure & Appl. Chem. 67 (1995) 649-666.
Introduction Definition Quality assurance practices and internal quality control Internal quality control procedure IQC and within-run precision Control materials in IQC Recommendations Conclusions
Guidelines on internal Quality Control
61 MAFF
Definition
Internal Quality Control
Internal Quality Control:
Set of procedures undertaken by laboratory staff for the continuous monitoring of operation and the results of measurements in order to decide whether results are reliable enough to be released.
62 MAFF
Control materials in IQC
Control materials are characterized substances that are inserted into the run alongside the test materials and subjected to exactly the same treatment.
•appropriate concentration of the analyte
•same matrix in terms of bulk composition, including minor constituents
•similar physical form
•stable and possible to divide the material into effectively identical portions for analysis
63 MAFF
Certified reference material (CRM):
Reference material accompanied by documentation issued by an authoritative body and providing one or more specified property values with associated uncertainties and traceability, using valid procedures
Documentation is given in the form of a “certificate” (ISO guide 30:1992). Procedures for the production and certification of certified reference materials are given, e.g. in ISO Guide 34 and ISO Guide 35.
Use of Certified reference material (1)
GUIDELINES ON ANALYTICAL TERMINOLOGY (CAC/GL 72-2009) 64 MAFF
Metrological Traceability:
Property of a measurement result whereby the result can be related to a reference through a documented unbroken chain of calibrations, each contributing to the stated measurement uncertainty.
Metrological traceability of a measurement result does not ensure that the measurement uncertainty is adequate for a given purpose or that there is an absence of mistakes.
Metrological Traceability
Guidelines on Analytical Terminology (CAC/GL 72-2009) 65 MAFF
Comparability of measurement result, for quantities of given kind. (metrologically traceable to the same reference)
Metrological Traceability (2)
Definition of unit, SI base
Primary national standard
Reference standards
Working standards
Measurement result
BIPM
Primary national standard (other countries)
End user
National Metrology Institute, Designated Institute
Calibration organization
Competent authority, Industry, Academia
Larger Measurement Uncertainty in lower stage 66 MAFF
Use of Certified reference material (2)
COMAR (International Database)
http://www.comar.bam.de/en/ Purchase from reagents manufacturer or reagents sales company
Comparison of the measurement results with the certified value
67 MAFF
Comparison of the measurement results with the certified value
CRMmm CC Δ
http://www.erm-crm.org/ERM_products/application_notes/Pages/index.aspx
22
Δ CRMm uuu +
Δm:absolute difference between mean
measured valued and certified value Cm: mean measured value CCRM : Certified Value
UΔ: expanded measurement uncertainty
um: measurement uncertainty uCRM : uncertainty of the certified value
If Δm ≤ UΔ then there is no significant difference between the measurement result and certified value.
ΔΔ 2uU
68 MAFF
Constraint on the use of CRM
⇒ Preparation of House reference material e.g. materials in proficiency testing, spiked control materials, etc.
For the majority of analysis there is no closely matching CRM available.
Cost to stock every relevant CRMs
Not applicable to unstable materials
Availability of sufficient amounts for IQC to use over extended periods
69 MAFF
If the use of reference material is not practical, limited check on bias is possible by a test of recovery.
Recovery checks
A test portion of the test sample spiked with a known amount of the analyte is analyzed alongside the original test material.
applicable unstable analyte/matrices, ad hoc analysis
70 MAFF
Blank determination
an analysis of a sample without the analyte, or an analysis without a sample, i.e. going through all steps of the procedure with
the reagents only.
1.In many analyses sample results are calculated by subtracting blank readings from sample readings.
2.Blank readings can be excellent monitors in quality control of reagents, analytical processes, and proficiency.
3. They can be used to estimate several types of method detection limits.
71 MAFF
Insertion of control materials for IQC (1)
Recommendation
1.Short (e.g. n<20) frequent runs of similar materials
2. Longer (e.g. n>20) frequent runs of similar materials
•at least once per run •analyze in duplicate at least half of the test materials selected at random •insert at least one blank determination
•at an approximate frequency of one per ten test materials at least once per run •analyze in duplicate at minimum of five test materials selected at random •insert one blank determination per ten test materials
72 MAFF
3. Frequent runs containing similar materials but with a wide range of analyte concentrations
4. ad hoc analysis (statistical control is not applicable)
Insertion of control materials for IQC (2)
•at an approximate frequency of one per ten test materials at least once per run.
•at least two concentration levels, one close to the median level of typical test materials, and the other is approximately at the upper or lower of decile as appropriate.
•duplicate a minimum of five test materials.
•insert one procedural blank per ten test materials.
•duplicate analysis on all of the test materials •recovery tests or use of formulated control material •blank determination
73 MAFF
Use of Control Charts
74
Control Chart (Shewhart Chart):
A statistical tool used to monitor process stability and control. One of the Seven Basic Tools of Quality Control.
MAFF
1. Understanding current and past process performance and its degree of consistency
2. Establishing a "state of statistical control" by identifying and removing causes of unnatural (or "special cause") variation so as to achieve a consistent and predictable level of process quality over time;
77
: The grand mean of all subgroup averages X
X : The mean for each subgroup
n
XXXX n+++
21
n : The number of measurements within a subgroup (here, n=2)
k
XXXX k+++
21
k : The number of subgroups (here, k=9)
: Calculate the following statistics
How to calculate/plot X-Bar and R chart (2)
MAFF
78
R : The average of the ranges for all subgroups
k
RRRR k+++
21
Ri : The individual range for each subgroup
k : The number of subgroups (here, k=9)
How to calculate/plot X-Bar and R chart (3)
MAFF
79
Upper control limit (UCL):
RAXUCL 2+
: Calculation of Upper and Lower control limit
Lower control limit (LCL):
RDUCL 4
For
For R chart:
X chart:
RAXLCL 2
RDLCL 3
For
For R chart:
X chart:
How to calculate/plot X-Bar and R chart (4)
MAFF
80
Constants for Calculating Limits for X-Bar and R charts
How to calculate/plot X-Bar and R chart (5)
http://www.itl.nist.gov/div898/handbook/pmc/section3/pmc321.htm MAFF
Proficiency testing (PT) :
evaluation of participant performance against pre-established criteria by means of interlaboratory comparisons
External Program for Quality Control (Proficiency testing)
84
(ISO/IEC 17043:2010)
•Participating laboratories analyze same sample distributed from proficiency testing provider.
•Reported results were compared to the assigned value and statistically evaluated.
•Participants can confirm their results.
MAFF
How to participate in PT program?
Proficiency testing provider
Internationally renowned: Fera(FAPAS, FEPAS),
AOACI, AOCS, etc.
Domestic (In case of Japan): JSAC, FDSC, JAB, etc.
Participation
Through internet, agency, etc.
Check schedule of individual proficiency testing round.
85 MAFF
example :FAPAS
• Round: Nutritional components, Food ingredients
•contaminants, Pesticides, Veterinary drug residues,
Food additives, etc.
•Participating laboratories analyze distributed sample
using their method. Report the results and the method
information through website.
•Results were Statistically evaluated and z-score were
returned to participating laboratories.
|z|≦2, 2<|z|<3, |z|≧3
Example of Proficiency Testing Scheme
86 MAFF
Result of PT
z = (x - X) / σp x: the participant’s reported result
X:the assigned value e.g., robust mean after excluding results that are clearly spurious and outliers by statistical evaluation.
σp :the target value for standard deviation
e.g., using RSDR from collaborative study, calculation from Horwitz equotion etc.
Usually shown in z-score defined as follows:
87 MAFF
z-Scores for Patulin (80.7 g/L) in Apple Purée Test Material
From: FAPAS PROTOCOL FOR THE ORGANISATION AND ANALYSIS OF DATA SIXTH EDITION, 2002
Graphical expression of z-Scores (example)
88 MAFF
How to interpret a result of PT?
Checking bias of the results
Review of method procedure and management system of
laboratory by periodical participation
Information on methods used by other laboratories
Importance of Participating PT
•A result of |z| >2 is not specially rare. •Is that a usual operation for the laboratory?
z-Scores “|z| ≤ 2” is “satisfactory” ?
89 MAFF
Definition
Non-negative parameter characterizing the dispersion of the values being attributed to a measurand, based on the information used.
Measurement Uncertainty in Codex (1)
Expanded measurement uncertainty:
Measurement uncertainty:
product of a combined standard measurement uncertainty and a factor larger than the number one
Guidelines on Analytical Terminology (CAC/GL 72-2009) 91 MAFF
Estimated range in which the true value would be found
Showing reliability of the results
Estimated by experimental data and statistical evaluation
What is Measurement Uncertainty?
92 MAFF
Analytical results and its variability
•Analytical results of repeated analyses of the
same sample can be found in a normal
distribution.
•In case of chemical quantitative analysis,
reproducibility, between laboratory variation,
depends on concentration of analyte,
independent of variety of food or analyte.
93 MAFF
68%
95%
99%
68% of the observations have values within the range of µ±1σ
95% of the observations have values within the range of µ±2σ
Normal Distribution
Mean : µ
SD : σ
+σ +2σ +3σ +4σ − σ − 2σ − 3σ − 4σ µ
94 MAFF
Measurement Uncertainty and Bias
(Value we want to know)
(unknown)
Population distribution (unknown)
True Value
Sample
distribution
Mean
Bias: difference between the result and the true value
Measurement Uncertainty : Estimated range in which the
true value would be found.
X-axis: Concentration
Y-axis: Density
95 MAFF
Measurement Uncertainty in Codex (2)
Guidelines on Measurement Uncertainty (CAC/GL 54-2004)
Measurement Uncertainty have to be estimated in Codex.
One of the requirements of the ISO/IEC 17025:2005
The measurement uncertainty of a result shall be estimated and then made available if requested.
Guidelines CAC/GL 27-1997 require laboratories involved in the import/export of foods to comply with general criteria in ISO/IEC 17025.
96 MAFF
Measurement Uncertainty in Codex (3)
Reporting the results in a form of “a ± U”
Guidelines on Measurement Uncertainty (CAC/GL 54-2004)
—where a: The best estimate of the true value of the concentration
of the measurand (the analytical result) u: The Standard uncertainty U: The expanded uncertainty (usually equal to 2u).
The range “a ± 2u” represents a 95% level of confidence
in which the true value would be found.
97 MAFF
Guidelines on Measurement Uncertainty (CAC/GL 54-2004)
2. Top-down approach
Use data from collaborative trials, proficiency studies, validation studies or intra-laboratory quality control samples,
1.Bottom-up approach
Identification of uncertainty sources
Estimation of the size of the uncertainty component associated with each potential source.
Calculation of combined uncertainty
Estimating Measurement Uncertainty
98 MAFF
MU in Compliance Decisions
Maximum Level
Result ±U
above ML
Result > ML but ML within U
Result < ML but ML within U
Result ±U
< ML
+U
-U
Guidelines on Measurement Uncertainty (CAC/GL 54-2004) Guidelines on Estimation of Uncertainty of Results (CAC/GL 59-2006)
Situation ii) iii) iv) i)
Result
100
MAFF
Information from Laboratory
Require all relevant data (especially when using 3rd Party laboratory)
•validation status of the methods of analysis used, sample
handling, preparation procedures.
•raw data (including spectral data, calculations, chemical
standards used);
•results of repeat analysis;
•internal quality control (control charts, blank data, recovery
data, uncertainty data, use of RMs);
•performance in proficiency testing
•official accreditation status of the laboratory
102
MAFF
Typical Problem for the Reported
Results (Examples)
•mistakes
e.g., sampling number, classification of commodity
•Ambiguous data
no formula for calculations of LOD, LOQ, recovery, etc.
no information on recovery correction
•Inappropriate significant figures
103
MAFF
Check
Analytical work was correctly done as specified?
•Sample treatment was appropriate ?
•Validated method used? Verified for the commodity?
•LOD and LOQ were properly calculated?
•LOQ was lower than expected occurrence level?
•Calibration was appropriate?
–reliable standard reagents
–linearity of calibration:R2≧0.99 ?
•Recovery corrected?
•Significant figures were properly calculated?
104
MAFF
Recovery
•Recovery/recovery factors:
Proportion of the amount of analyte, present in, added to or present in and added to the analytical portion of the test material, which is presented for measurement.
Guidelines on Analytical Terminology (CAC/GL 72-2009)
105 MAFF
Cf : fortified concentration, analytical results of spiked sample
Cu : unfortified concentration, analytical results of un-spiked sample
CA : added concentration
Food /Feed
Reagents spiked
Measurement Results
106
A
uf
C
CC(%) Recovery Marginal
MAFF
UA
f
CC
C Recovery % Total
+
Recovery Correction
All data, when reported, should
(a)be clearly identified as to whether or not a recovery correction has been applied and (b) if a recovery correction has been applied, the amount of the correction and the method by which it was derived should be included with the report.
This will promote direct comparability of data sets. Correction functions should be established on the basis of appropriate statistical considerations, documented, archived and available to the client.
Harmonized IUPAC Guidelines for the Use of Recovery Information in Analytical Measurement (CAC/GL 37-2001)
108 MAFF
Recovery check as IQC
IQC control charts for recovery should be established during method validation and used in all routine analysis. Runs giving recovery values outside the control range should be considered for re-analysis in the context of acceptable variation, or the results reported as semi-quantitative.
Harmonized IUPAC Guidelines for the Use of Recovery Information in Analytical Measurement (CAC/GL 37-2001)
109 MAFF
Significant Figures(1)
•Addition and Subtraction
–The result should have as many decimal places as the
measured number with the smallest number of decimal
places.
(e.g.) 2.54 cm + 15.75 cm = 18.29 cm
1.46 mg + 12.7 mg = 14.2 mg
(e.g.) 3.58 kg – 0.779 kg = 2.80 kg
1.46
+ 12.7
14.16 2
110 MAFF
Significant Figures(2)
•Multiplication and Division
–The result should have as many significant figures as the measured number with the smallest number of significant figures.
e.g. 1.42 mg/kg × 15 kg = 21 mg 100 m ÷ 12 s = 8.3 m/s
Divided by 4 pieces
8.43cm ÷ 4 = 2.11 cm
1.42
× 15
710
142
2130
111 MAFF
Exact numbers, such as the number of people in a room, have an infinite number of significant figures.
Rounding of numbers (1)
ISO 80000-1: 2009
Rounding
range: 0.1
Rounding
range:10
112 MAFF
The rounding shall always be carried out in only one step. e.g. 12.254 should be rounded to 12,3 and not first to
12,25 and then to 12,2.
Rule A: The even multiple
Rule B: The greater in magnitude
ISO 80000-1: 2009 (ISO 31-0 1992)、JIS Z 8401:1999
If there are two successive integral multiples equally near the given number, two different rules are in use.
113 MAFF
Rounding of numbers (2)
Roles of MAFF and MHLW for food safety
Processing, Distribution Primary Production
Processed Food
Fresh produce Raw material
Food Chain
MHLW (enforcement)
Setting MRLs
Food Inspection, etc.
MHLW (enforcement) Import quarantine Exporting Country
MAFF (enforcement) Fertilizer, Feed, Veterinary drug the use of Pesticide
Cooperation
MAFF (recommendation)
Improvement of production/processing methods of domestically produced food, etc.
Imported food
Consumer Farm
Domestically produced Food
115 MAFF
【Contents】
・Identification of a food safety problem
・Establishment of a risk profile
・Ranking of the hazard for risk management priority
・Establishment of risk assessment policy
・Consideration of the result of the risk assessment
・ Evaluation of risk management options
・ Implementation of management decision
・Risk communication etc
Standard Operating Procedure (SOP)
for MAFF & MHLW
Publication:25 Aug 2005, rev 2006, 2012
116 MAFF
Example of priority list of Chemical
hazards
for food safety (MAFF 2006, revised in
2010) •Metals
–Arsenic
–Cadmium
–Lead
–Methyl mercury
•Mycotoxins
–Aflatoxins
–Zearalenone
–T2-toxin
HT-2 toxin
•Marine toxins –Paralytic shellfish poisoning toxins
–Diarrhetic shellfish poisoning toxins
–Ciguatera poisoning toxins
–Domoic acid
–Brevetoxin
•Process contaminants –Acrylamide
–Polycyclic aromatic hydrocarbons
–Furan
–3-monochloropropane-1,2-diol
(3-MCPD)
3-MCPD fatty acid esters
etc. 117 MAFF
Guideline for the evaluation and publication of analytical results of surveillance/ monitoring (2006, MAFF)
Purpose To Guide MAFF officials for assessing the quality of analytical reports from internal/private laboratories, and for relevant considerations for their publication To provide analytical laboratories with the format for reporting the result of analysis
118 MAFF
Guideline for assessing the result of surveillance/ monitoring (MAFF)
Format for reporting the result of analysis by Laboratories
Laboratory Lab Name, Address, Person in charge
A . Analysis
1. Analyte / Matrix
2 . Laboratory Sample Date of receipt, Condition for storage of laboratory sample and
replicate sample (temperature etc.)
3.Sample Preparation Procedure for preparation of a portion for the measurement from Laboratory sample
119 MAFF
4. Information of analytical method Standard operating procedure of the analytical method
5. Information of method validation Multi Lab / Single Lab / Other
Matricies used for validation
6. LOD, LOQ Value, Calculation method
7. Calibration Used standard reagent, Purity, Protocol for preparation of
standard solution, Linear range
120 MAFF
8. Recovery Matrix, Analyte, Concentration, Number of replication, Value of
recovery rate and RSDr, Raw data or corrected by recovery
9. Measurement Uncertainty
Value of measurement uncertainty Method for estimation
10. Analytical results Number of tested samples, Value of measurement of each sample
121 MAFF
B Quality Control 1. Internal in accordance with GLP guideline by MHLW frequency 2. Proficiency testing provider, matrix, analyte, z-score
※ must participate in the last two years
(same or similar matrix and analyte) 3. ISO/IEC17025 Accreditation
122 MAFF
C Sampling (only when the laboratory itself conducted sampling) Sampling method and it’s reference Sample size for each step in the sampling procedure
123 MAFF
Before establishing guidelines
Large variety but small sample number for one commodity
e.g. only 5 samples for one commodity
No attention to quality of analytical result
Poor requirement of internal quality control,
proficiency testing, precision/recovery data for the used method
Priority to low financial cost
124 MAFF
Change in design of surveillance in MAFF (1)
After establishing guidelines
Statistic based sample number
e.g. enough to obtain occurrence data on 95 percentile value with 95% confidence level
Use of validated method
Requirement of internal/external quality control
• method validation/verification data
• participation to proficiency testing program
• ISO/IEC 17025 accreditation
Priority to data quality
125 MAFF
Change in design of surveillance in MAFF (2)
Challenges for enhancing quality
assurance in Japan
Continued capacity building of risk assessors/
managers
Enhancement of quality assurance of official
and private laboratories and research institutes
Collaboration for analytical method validation
Supply of standard reagents and certified
reference materials
126 MAFF