Post on 30-Jun-2020
© NMISA 2012
“Adequately demonstrating
technical competence by means
of Proficiency Testing – meeting
ISO/IEC 17025 and the SANAS
specific physical testing and
calibration laboratory
requirements”.
Eddie Tarnow
03 September 2012
© NMISA 2012
Presentation Overview
• Introduction
• Historical perspective
• The Proficiency Testing requirement
• The practical implementation of PT
• Interpreting and responding to the results
• Shortcomings of current “informal” laboratory PTS
• Dispelling the myths
• Summary
© NMISA 2012
Introduction
• Calibration & testing labs obliged to assure the validity of their results
• These measures must mitigate the risk of an erroneous measurement result being reported
• Real risk depends on end use of measurement result
• Unfortunately the end use is often transparent to the lab
• Therefore potential certificate or test report recalls are a direct measure of risk for the laboratories
• Therefore risk differ from lab to lab
• Traditionally labs could perform measurements and simply report the results
• Today, labs must do more (compliance testing)
• Testing labs becoming more informed
• Manufacturing tolerances becoming tighter
© NMISA 2012
Historical Perspective
• SANAS calibration labs until recently, were obliged to participate in annual audit programmes
• NMISA was contracted to plan, coordinate and execute these programmes
• Advantageous to labs: no cost, no planning, no selection or characterisation of a suitable artefact necessary
• Disadvantageous because: – Number & scope of labs has grown significantly,
– Technical competency of the labs varies greatly
– Complex matrix of suitable audit programmes necessary
– Costs of suitable programme provision grew rapidly
– May no longer be financially sustainable
– Only applied to accredited calibration laboratories
© NMISA 2012
Historical Perspective (2)
• Internationally a move to making formalised participation in PTS a compulsory requirement
• SANAS has reduced their audit programmes significantly
• ILAC has now introduced policy contained in document P19
• SANAS has responded with regulatory requirements described in documents R48 and R80
© NMISA 2012
Purpose of Proficiency Testing
• PT is one of the most powerful tools in evaluating/proving laboratory competence thereby assuring the quality of results
• It is the “eating of the pudding” so to speak....
• In one exercise it can effectively evaluate the competence of the lab wrt:- – Personnel,
– Accommodation & environment,
– Test and or, calibration methods including measurement uncertainty estimation where relevant,
– Equipment,
– Measurement traceability,
– Handling of test and calibration items,
– Assuring the quality of test and calibration results,
– Reporting the results
© NMISA 2012
ISO/IEC 17025 requirements
• “The lab shall have quality control procedures for monitoring the validity of tests and calibrations undertaken.”
• “The resulting data shall be recorded in such a way that trends are detectable and where practicable, statistical techniques shall be applied to the reviewing of the results.”
© NMISA 2012
ILAC Requirements
• ILAC document P9 states that:
– Laboratories minimum PT activity shall be:-
• PT results available prior to gaining accreditation,
• Planned, scheduled, ongoing PT activity appropriate to their scope of accreditation
© NMISA 2012
SANAS R-48 & R-80 requirements
• Application for accreditation – “All applicant calibration laboratories are required to participate
in PT prior to application.”
• Maintenance of accreditation – “All accredited calibration laboratories are required to participate
in the National Measurement Audit” (R-48)
– “Where appropriate, all accredited laboratories shall participate in PT for items on their schedule of accreditation (R-48) or specific tests or methods (R-80).”
– “The laboratory shall investigate all measurement results that fail to meet the minimum acceptance criteria, and record the root cause analysis conducted and all corrective and preventative action(s) taken.”
© NMISA 2012
SANAS R-48 & R-80 requirements (2)
• PT/ILC Activity Plan (cont.) – “All accredited calibration laboratories shall have available a PT
activity schedule that covers the past 5 year period (where possible) and the plan for the subsequent 5 year period (R-48) or for at least 2 accreditation cycles and shall be accomplished in a period not exceeding 1 accreditation cycle. (R-80).”
© NMISA 2012
SANAS R-48 & R-80 requirements (3)
• PT/ILC Activity Plan (cont.) – “The PT activity plan should address:”
• Identification of participants, and/or potential participants;
• The name/s and or identification of the PT schemes in which the laboratory intends to participate;
• The proposed measurement artefact/s or instrument/s;
• The measurement parameters, including range and measurement points;
• How the reference value is to be established;
• The minimum acceptance criteria;
• Responsibility for issue of the report;
• Where applicable, the typical ranges that cover the scope of accreditation, particularly measurements at extremities.
• Name and make of reference material used; (R-80)
• Any issues experienced with participating in PT; (R-80)
• Frequency of participation per time period justified by the laboratory. (R-80)
© NMISA 2012
SANAS R-48 & R-80 requirements (4)
• During the assessment – “Laboratory’s participation in PT / ILC activities will be evaluated
against their plan”
– “The laboratory shall make available to the assessment team all proficiency testing scheme and ILC reports.”
– “Proficiency testing scheme / ILC reports shall be clear and comprehensive and include at least the following minimum information:”
• Identification of the participants;
• Measurement protocol;
• Identification of the measurement standard or Artefact;
• Measurement results;
• The reference value/s and how these were established;
• Evaluation of the measurement results;
• An indication of the performance of individual participants;
• Minimum acceptance criteria;
• Conclusion.
© NMISA 2012
SANAS R-48 & R-80 requirements (5)
• “The effectiveness of corrective and preventive action taken will be evaluated during the assessment, and taken into consideration during the decision making process.”
© NMISA 2012
Practical Implementation of PT
• In practice a laboratory has two options:
• Participate in an existing PTS, preferably accredited to ISO/IEC
17043 or
• Conduct their own informal PTS
• If an existing scheme is selected, the following
requirements must be met:
• Range and functions of the scheme must be aligned with the
lab’s scope of accreditation
• The scheme Reference Uncertainties must be aligned with the
lab’s accredited CMCs
• The scheme artefact must replicate a typical measurement/test
performed by the laboratory
• The scheme scheduling and cost must be acceptable to the lab
© NMISA 2012
Practical Implementation of PT(2)
• If the lab elects to run their own scheme the following
critical aspects must be considered:
• Establishing the scheme objectives:-
• The measurement functions to be proficiency tested must
be identified,
• The measurement ranges to be proficiency tested must be
identified,
• The measurement uncertainty associated with the PTS
Reference Value must be of same order as accredited CMCs
© NMISA 2012
Practical Implementation of PT(3)
• If the lab elects to run their own scheme the following
critical aspects must be considered (cont):
• Scheme design:-
• Sequential scheme
• Simultaneous scheme
• Split-level scheme
• Split sample scheme
• Partial process scheme
© NMISA 2012
Practical Implementation of PT(4)
• If the lab elects to run their own scheme the following
critical aspects must be considered (cont):
• Selection and characterisation of a suitable scheme artefact:-
• Does a suitable artefact exist in the pilot lab or one of the
participant laboratories?
• Can the artefact be removed from service for the period of
the PTS?
• Will a suitable artefact have to be purchased specifically for
the PTS and if so, who will carry the cost?
• Is the artefact capable of the required technical
performance?
• How and by whom will the artefact be characterised?
© NMISA 2012
Practical Implementation of PT(5)
-10,00
-8,00
-6,00
-4,00
-2,00
0,00
2,00
4,00
6,00
8,00
10,00
2010/01/22 2010/05/02 2010/08/10 2010/11/18 2011/02/26 2011/06/06 2011/09/14 2011/12/23 2012/04/01 2012/07/10
Hu
mid
ity E
rro
r (%
rh
)
Date of Calibration
Humidity drift of Data Logger at 15°C and 70 % rh
© NMISA 2012
Practical Implementation of PT(6)
• If the lab elects to run their own scheme the following
critical aspects must be considered (cont):
• Identification of suitable participants:-
• the benefit to be gained by the participants depends on the
technical capability of each participant
• ideally one participant should have a superior technical
capability – assign Reference Value and Uncertainty
• should preferably always be more then two participants to
mitigate risk of bias going undetected
• participants should preferably use different equipment and
obtain their imported traceability from different sources
© NMISA 2012
Practical Implementation of PT(7)
• If the lab elects to run their own scheme the following
critical aspects must be considered (cont):
• Establishing the Scheme Reference Value:-
• Known Values – by design
• Certified Reference Values – calibration by NMI
• Reference Values – direct comparison with a Reference
Standard
• Consensus values from expert participants – participants
having higher accuracy capability
• Consensus values from participants – statistically
determined
© NMISA 2012
Practical Implementation of PT(8)
3,75000000
3,80000000
3,85000000
3,90000000
3,95000000
4,00000000
4,05000000
4,10000000
4,15000000
4,20000000
4,25000000
2009/10/14 2009/12/03 2010/01/22 2010/03/13 2010/05/02 2010/06/21 2010/08/10 2010/09/29 2010/11/18 2011/01/07 2011/02/26 2011/04/17
Resis
tan
ce E
rro
r (Ω
)
Date of Calibration
Drift of Resistor 100 kΩ
© NMISA 2012
Practical Implementation of PT(9)
• If the lab elects to run their own scheme the following
critical aspects must be considered (cont):
• Scheduling and executing the scheme:-
• Approach depends on scheme design employed
• Sequential Schemes
• Sufficient time must be allowed for artefact
characterisation beforehand
• Scheduling roster must be drawn up
• Start and end date for each participant
• Allow for sufficient time in between participants for
courier and non-adherence to slots
• Time slots to be allocated for intermediate checks on
artefact
© NMISA 2012
Practical Implementation of PT(10)
• If the lab elects to run their own scheme the following
critical aspects must be considered (cont):
• Scheduling and executing the scheme:-
• Simultaneous schemes
• Schedule with start date and results submission deadline
date
• Particular care necessary where artefact stability is time
dependent
• Sufficient time must be allowed for artefact
characterisation beforehand
• Scheme instructions need to be drawn up and distributed
© NMISA 2012
Practical Implementation of PT(11)
• If the lab elects to run their own scheme the following
critical aspects must be considered (cont):
• Analysing the results:-
• Results usually analysed statistically
• Simple difference –
• Percent difference –
• z scores –
• Zeta scores -
)( XxD
100
)(%
X
XxD
)( Xxz
)(
)(22
avlab uu
Xx
© NMISA 2012
Practical Implementation of PT(12)
• If the lab elects to run their own scheme the following
critical aspects must be considered (cont):
• Analysing the results:-
• Results usually analysed statistically (cont)
• Normalised Error (En) -
)(
)(22
reflab
n
UU
XxE
© NMISA 2012
Practical Implementation of PT(13)
• If the lab elects to run their own scheme the following
critical aspects must be considered (cont):
• Generating the report:-
• Results must be formally recorded in a report
• Report should be issued ASAP to mitigate risk in the
participant laboratories
• In sequential schemes “interim report” sometimes possible
© NMISA 2012
Practical Implementation of PT(14)
• If the lab elects to run their own scheme the following
critical aspects must be considered (cont):
• Generating the report (cont):-
• Report must contain the following as a minimum:-
• Period covered by the report
• Identification of the participants
• Measurement protocol
• Identification of the scheme artefact
• Measurement results
• Reference values and how they were established
• Evaluation of the measurement results
• Indication of the relative performance of the participants
• Minimum acceptance criteria
• Conclusion
© NMISA 2012
Interpreting & responding to results
• Criteria for evaluation of the results should be
determined and agreed upon prior to implementation of
the PTS
• The following should considered:-
• Expert consensus to directly determine whether results are fit for
purpose (typically used for qualitative testing)
• Method performance specifications and participants’ recognised
level of operation
© NMISA 2012
Interpreting & responding to results (2)
• The following should considered (cont):-
• Statistical determination for scores as follows:-
• z scores
• |z| ≤ 2,0 performance satisfactory
• 2,0 ≤ |z| ≤ 3,0 performance questionable
• |z| > 3,0 performance unsatisfactory
• Zeta scores
• |ζ| ≤ 2,0 performance satisfactory
• 2,0 ≤ |ζ| ≤ 3,0 performance questionable
• |ζ| > 3,0 performance unsatisfactory
• En numbers
• |En| ≤ 1,0 performance satisfactory
• |En| > 1,0 performance unsatisfactory
© NMISA 2012
Interpreting & responding to results (3)
© NMISA 2012
Interpreting & responding to results (4)
© NMISA 2012
Interpreting & responding to results (5)
© NMISA 2012
Current “informal” lab PT shortcomings
• The required 5 year plans are inadequate
• Don’t cover all ranges
• Don’t cover all CMCs
• Don’t indicate “actual” versus “planned”
• Don’t meet all the SANAS requirements
• PTS Reference Values not adequately defined
• Schemes often only involve two participants
• Uncertainties of participants often significantly different
• Inadequate characterisation of PTS artefact
• Often no formal report issued to participants
© NMISA 2012
Dispelling the myths
• Myth 1
• “I only need to participate in PTS to satisfy a SANAS
requirement”
• Myth 2
• “I can run a PTS with only two participants and it will meet the
SANAS requirements”
• Myth 3
• “Any instrument/test item will be a suitable PTS artefact”
• Myth 4
• “A formal report is just an unnecessary waste of time”
© NMISA 2012
Summary
• PT is one of the most effective tools to assure the quality
of a laboratory’s results
• It replicates the measurements/tests being performed by
the lab for customers
• Assuring the quality of results fundamentally important to
mitigate risk
• PT should be a fully integrated part of any laboratory
quality system
• Participation should be a routine activity and not just to
satisfy the requirements of SANAS