Adequately demonstrating technical competence by means ... · Adequately demonstrating technical...

© NMISA 2012

“Adequately demonstrating

technical competence by means

of Proficiency Testing – meeting

ISO/IEC 17025 and the SANAS

specific physical testing and

calibration laboratory

requirements”.

Eddie Tarnow

03 September 2012

© NMISA 2012

Presentation Overview

• Introduction

• Historical perspective

• The Proficiency Testing requirement

• The practical implementation of PT

• Interpreting and responding to the results

• Shortcomings of current “informal” laboratory PTS

• Dispelling the myths

• Summary

© NMISA 2012

Introduction

• Calibration & testing labs obliged to assure the validity of their results

• These measures must mitigate the risk of an erroneous measurement result being reported

• Real risk depends on end use of measurement result

• Unfortunately the end use is often transparent to the lab

• Therefore potential certificate or test report recalls are a direct measure of risk for the laboratories

• Therefore risk differ from lab to lab

• Traditionally labs could perform measurements and simply report the results

• Today, labs must do more (compliance testing)

• Testing labs becoming more informed

• Manufacturing tolerances becoming tighter

© NMISA 2012

Historical Perspective

• SANAS calibration labs until recently, were obliged to participate in annual audit programmes

• NMISA was contracted to plan, coordinate and execute these programmes

• Advantageous to labs: no cost, no planning, no selection or characterisation of a suitable artefact necessary

• Disadvantageous because: – Number & scope of labs has grown significantly,

– Technical competency of the labs varies greatly

– Complex matrix of suitable audit programmes necessary

– Costs of suitable programme provision grew rapidly

– May no longer be financially sustainable

– Only applied to accredited calibration laboratories

© NMISA 2012

Historical Perspective (2)

• Internationally a move to making formalised participation in PTS a compulsory requirement

• SANAS has reduced their audit programmes significantly

• ILAC has now introduced policy contained in document P19

• SANAS has responded with regulatory requirements described in documents R48 and R80

© NMISA 2012

Purpose of Proficiency Testing

• PT is one of the most powerful tools in evaluating/proving laboratory competence thereby assuring the quality of results

• It is the “eating of the pudding” so to speak....

• In one exercise it can effectively evaluate the competence of the lab wrt:- – Personnel,

– Accommodation & environment,

– Test and or, calibration methods including measurement uncertainty estimation where relevant,

– Equipment,

– Measurement traceability,

– Handling of test and calibration items,

– Assuring the quality of test and calibration results,

– Reporting the results

© NMISA 2012

ISO/IEC 17025 requirements

• “The lab shall have quality control procedures for monitoring the validity of tests and calibrations undertaken.”

• “The resulting data shall be recorded in such a way that trends are detectable and where practicable, statistical techniques shall be applied to the reviewing of the results.”

© NMISA 2012

ILAC Requirements

• ILAC document P9 states that:

– Laboratories minimum PT activity shall be:-

• PT results available prior to gaining accreditation,

• Planned, scheduled, ongoing PT activity appropriate to their scope of accreditation

© NMISA 2012

SANAS R-48 & R-80 requirements

• Application for accreditation – “All applicant calibration laboratories are required to participate

in PT prior to application.”

• Maintenance of accreditation – “All accredited calibration laboratories are required to participate

in the National Measurement Audit” (R-48)

– “Where appropriate, all accredited laboratories shall participate in PT for items on their schedule of accreditation (R-48) or specific tests or methods (R-80).”

– “The laboratory shall investigate all measurement results that fail to meet the minimum acceptance criteria, and record the root cause analysis conducted and all corrective and preventative action(s) taken.”

© NMISA 2012

SANAS R-48 & R-80 requirements (2)

• PT/ILC Activity Plan (cont.) – “All accredited calibration laboratories shall have available a PT

activity schedule that covers the past 5 year period (where possible) and the plan for the subsequent 5 year period (R-48) or for at least 2 accreditation cycles and shall be accomplished in a period not exceeding 1 accreditation cycle. (R-80).”

© NMISA 2012


• PT/ILC Activity Plan (cont.) – “The PT activity plan should address:”

• Identification of participants, and/or potential participants;

• The name/s and or identification of the PT schemes in which the laboratory intends to participate;

• The proposed measurement artefact/s or instrument/s;

• The measurement parameters, including range and measurement points;

• How the reference value is to be established;

• The minimum acceptance criteria;

• Responsibility for issue of the report;

• Where applicable, the typical ranges that cover the scope of accreditation, particularly measurements at extremities.

• Name and make of reference material used; (R-80)

• Any issues experienced with participating in PT; (R-80)

• Frequency of participation per time period justified by the laboratory. (R-80)

© NMISA 2012


• During the assessment – “Laboratory’s participation in PT / ILC activities will be evaluated

against their plan”

– “The laboratory shall make available to the assessment team all proficiency testing scheme and ILC reports.”

– “Proficiency testing scheme / ILC reports shall be clear and comprehensive and include at least the following minimum information:”

• Identification of the participants;

• Measurement protocol;

• Identification of the measurement standard or Artefact;

• Measurement results;

• The reference value/s and how these were established;

• Evaluation of the measurement results;

• An indication of the performance of individual participants;

• Minimum acceptance criteria;

• Conclusion.

© NMISA 2012


• “The effectiveness of corrective and preventive action taken will be evaluated during the assessment, and taken into consideration during the decision making process.”

© NMISA 2012

Practical Implementation of PT

• In practice a laboratory has two options:

• Participate in an existing PTS, preferably accredited to ISO/IEC

17043 or

• Conduct their own informal PTS

• If an existing scheme is selected, the following

requirements must be met:

• Range and functions of the scheme must be aligned with the

lab’s scope of accreditation

• The scheme Reference Uncertainties must be aligned with the

lab’s accredited CMCs

• The scheme artefact must replicate a typical measurement/test

performed by the laboratory

• The scheme scheduling and cost must be acceptable to the lab

© NMISA 2012

Practical Implementation of PT(2)

• If the lab elects to run their own scheme the following

critical aspects must be considered:

• Establishing the scheme objectives:-

• The measurement functions to be proficiency tested must

be identified,

• The measurement ranges to be proficiency tested must be

identified,

• The measurement uncertainty associated with the PTS

Reference Value must be of same order as accredited CMCs

© NMISA 2012



critical aspects must be considered (cont):

• Scheme design:-

• Sequential scheme

• Simultaneous scheme

• Split-level scheme

• Split sample scheme

• Partial process scheme

© NMISA 2012




• Selection and characterisation of a suitable scheme artefact:-

• Does a suitable artefact exist in the pilot lab or one of the

participant laboratories?

• Can the artefact be removed from service for the period of

the PTS?

• Will a suitable artefact have to be purchased specifically for

the PTS and if so, who will carry the cost?

• Is the artefact capable of the required technical

performance?

• How and by whom will the artefact be characterised?

© NMISA 2012


-10,00

-8,00

-6,00

-4,00

-2,00

0,00

2,00

4,00

6,00

8,00

10,00

2010/01/22 2010/05/02 2010/08/10 2010/11/18 2011/02/26 2011/06/06 2011/09/14 2011/12/23 2012/04/01 2012/07/10

Hu

mid

ity E

rro

r (%

rh

)

Date of Calibration

Humidity drift of Data Logger at 15°C and 70 % rh

© NMISA 2012




• Identification of suitable participants:-

• the benefit to be gained by the participants depends on the

technical capability of each participant

• ideally one participant should have a superior technical

capability – assign Reference Value and Uncertainty

• should preferably always be more then two participants to

mitigate risk of bias going undetected

• participants should preferably use different equipment and

obtain their imported traceability from different sources

© NMISA 2012




• Establishing the Scheme Reference Value:-

• Known Values – by design

• Certified Reference Values – calibration by NMI

• Reference Values – direct comparison with a Reference

Standard

• Consensus values from expert participants – participants

having higher accuracy capability

• Consensus values from participants – statistically

determined

© NMISA 2012


3,75000000

3,80000000

3,85000000

3,90000000

3,95000000

4,00000000

4,05000000

4,10000000

4,15000000

4,20000000

4,25000000

2009/10/14 2009/12/03 2010/01/22 2010/03/13 2010/05/02 2010/06/21 2010/08/10 2010/09/29 2010/11/18 2011/01/07 2011/02/26 2011/04/17

Resis

tan

ce E

rro

r (Ω

)

Date of Calibration

Drift of Resistor 100 kΩ

© NMISA 2012




• Scheduling and executing the scheme:-

• Approach depends on scheme design employed

• Sequential Schemes

• Sufficient time must be allowed for artefact

characterisation beforehand

• Scheduling roster must be drawn up

• Start and end date for each participant

• Allow for sufficient time in between participants for

courier and non-adherence to slots

• Time slots to be allocated for intermediate checks on

artefact

© NMISA 2012




• Scheduling and executing the scheme:-

• Simultaneous schemes

• Schedule with start date and results submission deadline

date

• Particular care necessary where artefact stability is time

dependent

• Sufficient time must be allowed for artefact

characterisation beforehand

• Scheme instructions need to be drawn up and distributed

© NMISA 2012




• Analysing the results:-

• Results usually analysed statistically

• Simple difference –

• Percent difference –

• z scores –

• Zeta scores -

)( XxD

100

)(%

X

XxD

)( Xxz

)(

)(22

avlab uu

Xx

© NMISA 2012




• Analysing the results:-

• Results usually analysed statistically (cont)

• Normalised Error (En) -

)(

)(22

reflab

n

UU

XxE

© NMISA 2012




• Generating the report:-

• Results must be formally recorded in a report

• Report should be issued ASAP to mitigate risk in the

participant laboratories

• In sequential schemes “interim report” sometimes possible

© NMISA 2012




• Generating the report (cont):-

• Report must contain the following as a minimum:-

• Period covered by the report

• Identification of the participants

• Measurement protocol

• Identification of the scheme artefact

• Measurement results

• Reference values and how they were established

• Evaluation of the measurement results

• Indication of the relative performance of the participants

• Minimum acceptance criteria

• Conclusion

© NMISA 2012

Interpreting & responding to results

• Criteria for evaluation of the results should be

determined and agreed upon prior to implementation of

the PTS

• The following should considered:-

• Expert consensus to directly determine whether results are fit for

purpose (typically used for qualitative testing)

• Method performance specifications and participants’ recognised

level of operation

© NMISA 2012

Interpreting & responding to results (2)

• The following should considered (cont):-

• Statistical determination for scores as follows:-

• z scores

• |z| ≤ 2,0 performance satisfactory

• 2,0 ≤ |z| ≤ 3,0 performance questionable

• |z| > 3,0 performance unsatisfactory

• Zeta scores

• |ζ| ≤ 2,0 performance satisfactory

• 2,0 ≤ |ζ| ≤ 3,0 performance questionable

• |ζ| > 3,0 performance unsatisfactory

• En numbers

• |En| ≤ 1,0 performance satisfactory

• |En| > 1,0 performance unsatisfactory

© NMISA 2012

Current “informal” lab PT shortcomings

• The required 5 year plans are inadequate

• Don’t cover all ranges

• Don’t cover all CMCs

• Don’t indicate “actual” versus “planned”

• Don’t meet all the SANAS requirements

• PTS Reference Values not adequately defined

• Schemes often only involve two participants

• Uncertainties of participants often significantly different

• Inadequate characterisation of PTS artefact

• Often no formal report issued to participants

© NMISA 2012

Dispelling the myths

• Myth 1

• “I only need to participate in PTS to satisfy a SANAS

requirement”

• Myth 2

• “I can run a PTS with only two participants and it will meet the

SANAS requirements”

• Myth 3

• “Any instrument/test item will be a suitable PTS artefact”

• Myth 4

• “A formal report is just an unnecessary waste of time”

© NMISA 2012

Summary

• PT is one of the most effective tools to assure the quality

of a laboratory’s results

• It replicates the measurements/tests being performed by

the lab for customers

• Assuring the quality of results fundamentally important to

mitigate risk

• PT should be a fully integrated part of any laboratory

quality system

• Participation should be a routine activity and not just to

satisfy the requirements of SANAS

Adequately demonstrating technical competence by means ... · Adequately demonstrating technical...

Documents

Transcript of Adequately demonstrating technical competence by means ... · Adequately demonstrating technical...