Effective Control of theExamination Process

“Reunión de Expertos”

5 Ciclo Internacional De Conferencias De La Calidad

1

Coordinator James Westgard, United States

PARTICIPANTS Laura Mercapide,

Argentina Amadeo Saez, Brasil Aída Porras, Colombia Oscar Martínez, Colombia Enrique Amaya, Perú Margarita Iturriza,

Venezuela Erik Mendoza, México Eduardo Brambila,

México Arturo Terrés, México

Our Focus is on the Examination Process

Pre-analytic and post-analytic errors are also of concern, but our charge is to focus on the analytical part of the examination process

Evidence in scientific literature indicates analytical errors are still major source of problems leading to mistreatment and harm to patients

What errors have been observed in the Total Testing

Process?

Pre-analytic Analytic Post-analytic

• Patient preparation• Specimen acquisition• Specimen processing• Sample transport• Physician test order

• Sample aliquot• Analyzer setup• Test calibration• Quality control• Reportable test

• Test report• Transmittal of report• Receipt of report• Review of test results• Action on test results

60% 15% 25%Plebani & Carraro. Clin Chem 2007:53:1338-42

5

Lab Errors and Patient Care Plebani-Carraro study

Clin Chem 2007:53:1338-42 51,746 tests 393 questionable results 160 confirmed as laboratory errors 46 caused inappropriate patient care 24 of those were analytical errors

Analytical errors are still major cause (over 50% cases) of inappropriate patient care 6

Need for Guidance on “Effective Control of

Examination Process” for Latin American Countries

Important to be relevant and practical for local laboratories

Must consider national and governmental interests and requirements

Must consider professional assessment of needs and practice guidelines

General guidance on “effective control of examination

process”? ISO 15189: Medical Laboratories –

Particular requirements for quality and competence

ISO 15198: Validation of user quality control procedures by the manufacturer

CLSI C24-A3: Statistical Quality Control for Quantitative Measuring Processes – Principles and Definitions

CLSI EP23-P: Laboratory Quality Control based on Risk Management

8

ISO 15189 Guidance for “Assuring Quality” of Examination Process

5.6.1. “The laboratory shall design internal quality control systems that verify the attainment of the intended quality of results…” Medical relevance of laboratory tests

is an important consideration! Comparability of test results is

important for medical relevance! How define medical relevance?

9

ISO 15189 “Assuring quality”

5.6.2 ..determine uncertainty of results, where relevant and possible

5.6.3 …ensure that results are traceable

5.6.4 …participate in interlaboratory comparisons

5.6.5 …if EQA not available, develop a mechanism for determining acceptability

5.6.6 For examinations performed using different procedures or at different sites, define a mechanism for verifying comparability of results

Particular issues assigned to this work group

What frequency of QC is sufficient? How important are recommendations

from manufacturers for QC? Should laboratory modify recommendations?

How often for EQC or PT?

What frequency of QC?

CLSI guidance on defining run length “An analytical run is an interval (i.e., a

period of time or series of measurements) within which the accuracy and precision of the measuring system is expected to be stable; between which events may occur causing the measurement process to be more susceptible (i.e., greater risk) to errors that are important to detect.”

12

Factors Affecting Run Length (1)

Events and non-events Event driven QC

Known, scheduled and expected changes

Non-event driven QC Other things that happen

Parvin’s concepts Ref: Parvin, Gronowski. Effect of

analytical run length on QC performance and the QC planning process. Clin Chem 1997;11:2149-2154.

13

Factors Affecting Run Length (2)

Mode of Operation Batch mode

All patient specimens and control samples are analyzed together

Patient results not reported until control results are validated

Continuous mode Patient results are being reported as

they are determined Control samples are analyzed

periodically14

Factors affecting Run LengthCost-Effectiveness

Number of controls relative to number of patient specimens

Levels of controls

Factors Affecting Run Length (3)

Strategies Manufacturer’s instructions may

provide minimum strategy – e.g., 2 levels per day

+ Event QC to assess significance of changes in the testing process

+ non-event QC to monitor process during routine operation

Multi-stage QC for “startup,” “monitoring,” and “patient data QC”

16

Factors Affecting Run Length (4)

Strategies Sigma QC – relative amount of QC can be

related to method’s sigma-performance Risk analysis and residual risks – guidance

for susceptibility testing Consensus of experts – professional

practice standards Experienced judgment – knowledgeable

analyst has expertise about stability and susceptibility of testing process

17

Run length and frequency of controls

Models

Measures Statistical

Assumptions

Empirical

Economic

Residual risks

Error frequency, f Q/P

Strategies

Sigma

Risk analysis

Consensus of experts

Event+Monitor

Manuf. recommendations

Batch processing

Experienced judgment

Residual risksEvent only

Cost of repeat analysis

Operations

Batch mode

System stability

Continuous mode

Reporting interval

Unexpected events

LIS

Susceptibility

Batch size

Components

Residual risks

Maintenance

Calibration, reagents

Analyte stability

Events

Operators

PartsLab conditions

Stats

QC performance goals

18

What frequency of QC is sufficient?

Recommendations (1) Strategy

Define length of run In terms of time, numbers of patient

samples, mode of operation Importance of “events” or changes that

occur with the process that require verification by controls

Medically important concentrations for controls

General practice to use two levels of controls Sometimes advisable to have three levels

Many factors to consider to optimize run length or frequency of QC

What frequency of QC? Recommendations (2)

For small runs, utilize “batch” strategy QC at beginning QC at end Release results after inspect all controls

and reviewing patient results when necessary

What frequency of QC?Recommendations (3)

For large runs, highly automated systems with continuous reporting of results Controls at beginning of run

Right QC design to detect medically important errors + Controls for events

e.g., Change of reagent lots + Controls to monitor performance during run

Or, possibly use mean or median of patient data to monitor stability during run

+ Controls at end of run

How important are the manufacturer’s QC directions?

Recommendations (4) Provide minimum requirements that the

laboratory must satisfy E.g., calibration, preventive maintenance, etc.

Laboratory is still responsible for design of IQC system Intended clinical use Observed method performance Necessary QC rules and Numbers of

measurements

How important manufacturers QC directions?

Recommendations (5) Need for “independent” control

“Third party control” Traceability is an important responsibility

of manufacturer Calibration materials and process

Verification/validation of method performance is an important responsibility of the laboratory

EQA/PT important responsibility in monitoring/measuring accuracy over time “Commutability” important characteristic of

materials

How often EQA/PT?Recommendations (6)

At least monthly With fast turnaround of results to be useful for

identifying bias and making improvements in the laboratory

Approved EQA program preferred ILAC G13:08/2007 ISO 17043

Most essential information – bias observed vs “assigned value”

Other issues of interest

Medical relevance “Intended use,” “intended quality of test

results” Traceability

Comparability of test results Validation of method performance Design of IQC

Available planning approach, tools

How assure quality?

26

(3) Validate MethodPerformance (CV,bias)

(1) Define Goals for Intended Use

(TEa, Dint)

(3a) Manufacturer’sClaims

(1a) Regulatory &AccreditationRequirements

(1b) Clinical and Medical Applications

(2) Select AnalyticMeasurement

Procedure

(2a) Traceability& Calibration

(2b) Manufacturer’sReference Methods

& Materials

(4) Design SQC(rules, N, F)

(5) FormulateAQC Strategy

(11) Improve AQCEffectiveness

(5b) Lab Evaluationof Residual Risk

(5a) Manufacturer’sRisk Analysis

(6) Develop AQC Plan

(10) Monitor AQCEffectiveness (f), EQA

(7) Implement AQC System

(6a) QC Toolbox

(8) Verify Attainmentof Intended Quality

of Test Results

(9) Measure Quality& Uncertainty

ISO 15198 Validation of QC Procedures

QC procedures shall be validated to assure that failures are not a hazard to patients Recommends use of risk analysis

Conventional statistical quality control procedures (e.g., as described in CLSI C24) are considered adequate Validation may be based on simulated

effects of errors on performance data27

CLSI C24 QC Planning Process

28

Define quality specifications for test

Select appropriate control materials

Determine method performance

Identify quality control strategies

Predict QC performance

Specify goals for QC performance

Select QC to satisfy goals

CalculateSigma

%TEa-%Bias %CV

UtilizeSigma-metricQC Selection

Tool

Sigma-metrics QC Selection Tool

2 Levels Control

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

0.0 1.0 2.0 3.0 4.0

1.65 2.65 3.65 4.65 5.65

13s/2of32s/R4s/31s/6x 0.07 ----- 6 113s/22s/R4s/41s/8x 0.03 ----- 4 213s/22s/R4s/41s 0.03 ----- 4 112.5s 0.04 ----- 4 112.5s 0.03 ----- 2 113s/22s/R4s 0.01 ----- 2 113s 0.00 ----- 2 113.5s 0.00 ----- 2 1

Pfr Ped N R

Pro

bab

ilit

y f

or

Reje

cti

on

(P

)

Systematic Error (SE, multiples of s)

Sigma Scale

DesirableError

Detection

DesirableFalse

Rejection

3s 4s 5s

29

EP22, EP23 on Risk Analysis

Original purpose of CLSI project was to develop scientific approach for defining frequency of QC

Adopted “risk analysis” approach Failure-modes and risk should provide

guidance on need for control mechanisms and frequency of QC

Analytical QC Plan should be the outcome of the risk analysis process

Important Considerations in Future QC Systems

Design Is there a scientific basis and approach for

selecting parameters and setting limits on basis of intended quality of results?

Validation Is there an objective approach for assessing

the reliability of technical and medical decisions on control status?

Control Is there a quantitative process for monitoring

and verifying the attainment of intended quality of test results?

31