IQC Edward Kearney. Why is IQC/EQA important? To ensure the test result is fit for purpose.

IQC

Edward Kearney

Why is IQC/EQA important?

To ensure the test result is fit for purpose

EQAHistoric

Accuracy

Repeated samples may give precision

Who determines if a patient gets a statin?

We do !

Who determines if a patient gets a Dx of DM?

We do !

Who determines if a patient gets a prostrate biopsy?

We do !

Who determines if a patient gets dialysis?

We do !

How do we know we are right?

Do we?We do !

Westgard .com “1575”

Diabetes

Fasting Plasma Glucose > 7.0 mmol/L

On two or more occasions can make the diagnosis of Diabetes Mellitus

Fasting Glucose 6.8 mmol/L = IFG

Fasting Glucose 7.2 mmol/L = DM

Acute Coronary SyndromeTroponin T

> 0.05 ug/L = Biochemical evidence of cardiac damage

< 0.05 ug/L = No Biochemical evidence of cardiac damage

How do we know we are right?Standardisation

Calibrators

Test conditions

Reagents

Quality Control

Internal Quality Control (IQC)

External Quality Assurance (EQA)

IQC

Statistical QC

Electronic QC

Real time

Stable process

Bias

Precision

IQC Materials

Mimic Patients sample

Cover normal and pathological range

Ideal – decision points

Controlling the quality - IQC

Compare process performance with what is expected under stable operation

Stable performance determined by assaying control material over time : Calculate mean and SD

Measurements are made continually and compared with the original distribution

Unexpected results are identified to alert the analyst to possible changes in process performance

Multi-rule QC

Westgard rules

A multi-rule Shewhart chart for quality control in clinical chemistry –

JO Westgard, PL Barry, MR Hunt, T Groth - Clinical Chemistry, 1981

Use and Interpretation of Common Statistical Tests in Method-Comparison Studies - all 2 versions »

JO Westgard, MR Hunt - Clinical Chemistry, 1973

Performance characteristics of rules for internal quality control: probabilities for false rejection

JO Westgard, T Groth, T Aronsson, H Falk, CH de … - Clinical Chemistry, 1977

Assuring analytical quality through process planning and quality control.

JO Westgard - Arch Pathol Lab Med, 1992

European specifications for imprecision and inaccuracy compared with operating specifications

JO Westgard, JJ Seehafer, PL Barry - Clinical Chemistry, 1994

Internal quality control: planning and implementation strategies

JO Westgard - Annals of Clinical Biochemistry, 2003

The Quality of Laboratory Testing Today, An Assessment of σ Metrics for Analytic Quality Using Performance Data From Proficiency Testing Surveys and the CLIA Criteria for Acceptable Performance

James O. Westgard, and Sten A. Westgard. Am J Clin Pathol 2006;125:343-354

• Determine the expected distribution of control values• Calculate mean and SD from control data to establish

control limits for control chart• Expect control values to fall with certain control limits

– 95% within 2 SD– 99.7% within 3 SD

• Plot control values versus time to provide control chart• Identify unexpected values

Very Unexpected

Somewhat Unexpected

XX

X X XX

XX

XX X

X

XXX

XX

XXX

XXXX

250 260245240 255235 265

265

250

260

255

245

1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 20 1 2 3 4

Run Number (or Time, Date)

240

235

X

XX

X

X X

X

X

X

X

XX

X

X

X X

XX

X

X

X

X

X

X

Conceptual basis of the control chart

James O Westgard Internal quality control: planning and implementation strategies Ann Clin Biochem 2003; 40: 593–611

QC Data

12s

13s 22s R4s 41s 10x

Report Results

Take Corrective Action

Flowchart and logic for the multirule IQC procedure commonly known as “Westgard Rules”


IQC2 SD rule for 1 QC sample

Statistically 1 in 20 will be outside this range

This is not unexpected

5% of runs (false) rejected

IQC

2 SD rule for 3 QC samples

This is not unexpected

18% of runs (false) rejected

What actually happens?

• EM Kearney. The South Thames (East) Quality Assurance Liasion Group (1998)

• James O Westgard. Internal quality control: planning and implementation strategies Ann Clin Biochem 2003; 40: 593–611

• David Housley. North Thames audit and QA group: Audit of Internal Quality Control practice and processes. (2005)

5. NUMBER OF SAMPLES USED TO ASSIGN QC VALUES

• 1 = provisionally 3, then re-assessed at 15• 2 = 10 – 15• 1 = 12• 11 = 20 (one 20 days ?); one with minimum of 1

month run in (major variations discussed with manufacturer); one re-assess after 1000; some variation in number of batches / days etc

• 1 = 20 - 25• 1 = > 30• 3 = at least 50• 1 = adjusted every month 9 ( no number stated)

David Housley. North Thames audit and QA group: Audit of Internal Quality Control practice and processes. (2005)

EP5-AClinical and Laboratory Standards Institute (NCCLS) EP5-AEvaluation of precision performance of Clinical Chemistry Devices: Approved guidelines.Precision Evaluation ExperimentRecommendationsMinimum of 20 working days

Batch analysers: two runs per day with two test samples at each of at least two levels.Random access: four samples at each level should be analysed through out the day.

At the end of each 5 days the control limits are recalculated and all data checked for acceptability.The cause of outliers should be determined.Data may not be rejected without valid justification.Learning curve data may be rejected and replaced by a equal number of points at the end.Other methods:Not recommended:

Single run of 20 samples for within runSingle observation each day for 10 – 20 days for day to day

Because:Single runs do not reflect usual operating parameters, thus adversely affecting the estimate.Single observations will be highly dependent on the number of days used.

EP5-ARandom access:

Four samples at each level should be analysed through out the day.

At the end of each 5 days the control limits are recalculated and all data checked for acceptability.

The cause of outliers should be determined.

Data may not be rejected without valid justification.

Learning curve data may be rejected and replaced by a equal number of points at the end.

Criteria use to determine acceptability

• Written procedure 1

• Professional judgement 6

• Combination of above 12

EM Kearney the South Thames (East) Quality Assurance Liasion Group (1998)

Is the procedure always adhered to?

• Yes 2

• Mostly 1

• No 5

• Blank 11


QC rules used for calcium?2SD 6

3SD 1

12s + Multi-rules 1

Westgard 3

Limits professional judgement, Vitros F2/F3


Calcium QC rule violated describe the process for dealing with this

Check QC 3

Rerun 12

Recalibrate 4

Inspect QC history 1


Westgard Rules

1. Check control 2 Make up new

1. Repeat 2 Make up new

1. Repeat 2 Recalibrate


9. ARE WESTGARD RULES USED AS PRIMARY FORM OF DATA EVALUATION?

• YES = 18 (62%)• NO = 11 (38%)

– 1 = Mean and target range (< 2SD)– 5 = single 2SD rule– 1 = single 1.5 SD rule– 1 = not formally but simplified Westgard used– 1 = under consideration– 1 = Instrument set flag points, but no rule stated– 1 = 2SD in first instance, then Westgard for detail bias

profile


REASONS FOR ACCEPTING FAILED QC

• Only if not clinically significant – 7 sites• Lack of reagent prevents change• Lack of staff, pressure to reduce TAT, demotivated staff• Checked with EQA• Various. eg. very close to limit, one QC in/ one QC out, QC

deterioration before new QC is ready e.g. lot no. change until new range established.

• Only if all attempts to correct it have failed• Only after consultation with manufacturer• Only if calibrant returns result to normal• Only if no reason can be found and 1 QC out• eg. If LDH QC is out, the reagent is changed the next day


Planning strategies (doing the right IQC)

Define the quality requirement for the test

Determine the method precision and bias

Identify candidate IQC procedures

Predict IQC performance

Select goals for IQC performance

Select an appropriate IQC procedure

Clinical Outcome Criteria (DInt)

Analytical Outcome Criteria (TEa)

Operating Specifications(smeas, biasmeas, control rules, N)

ProficiencyTestingCriteria

MedicallyImportantChanges

TotalBiologic

Goals

IndividualBiologic

Goals

Performance Criteria SDMax, BiasMax

DiagnosticClassification

Goals

Stateof the

Art

ArbitraryControl

A system of quality requirements and operating specifications


0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

0.0 1.0 2.0 3.0 4.0

12s

12.5s

13s/22s/R4s

13s

13.5s

N

2

2

2

2

2

R

1

1

1

1

1

Thur 11/15/1 - 09:44:08

Systematic Error (SE, multiples of s)

Pro

bab

ilit

y fo

r R

ejec

tion

(P

)

Power Function Graph (SE)

Pfr given byy-intercept

Ped depends on size of

systematic error

Power curves for commonly used IQC procedures having two control measurements per run. Pfr, or the probability for false rejection, is given by the y-intercept. Ped, the probability for error detection, depends on the size error occurring, which is illustrated by the vertical line, and estimated by reading the y-value at the point of intersection with the power curve.


0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

0.0 1.0 2.0 3.0 4.0

13s/2of32s/R4s/31s/6x

13s/22s/R4s/41s

12.5s

12.5s

13s/22s/R4s

13s

13.5s

13s

N

6

4

4

2

2

2

2

1

R

1

1

1

1

1

1

1

1

Mon 4/9/1 - 13:18:00

Pro

bab

ilit

y fo

r R

ejec

tion

(P

)



Probabilityof false

rejection, Pfr given by

y-interceptsProbability of

error detection, Ped , depends

on size of systematic error

Pfr is0.07to

0.01

Ped is 1.00 to 0.40

Example use of a power function graph to assess IQC performance for a cholesterol test where the medically important systematic error is equivalent to 2.85 times the standard deviation of the method.

James O Westgard Internal quality control: planning and implementation strategies Ann Clin Biochem 2003; 40: 593–

611

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

0.0 1.0 2.0 3.0 4.0

13s/2of32s/R4s/31s/6x

13s/22s/R4s/41s

12.5s

12.5s

13s/22s/R4s

13s

13.5s

13s

N

6

4

4

2

2

2

2

1

R

1

1

1

1

1

1

1

1

Mon 4/9/1 - 13:18:00

Pro

bab

ilit

y fo

r R

ejec

tion

(P

)



Lines show SE detectable with

90% chanceby different

QC procedures

Sizes of systematic errors that can be detected with 90% assurance by different IQC procedures


0.0

1.0

2.0

3.0

4.0

5.0

6.0

7.0

8.0

9.0

10.0

0.0 1.0 2.0 3.0 4.0 5.0

13s/2of32s/R4s/31s/6x

13s/22s/R4s/41s

12.5s

12.5s

13s/22s/R4s

13s

13.5s

13s

N

6

4

4

2

2

2

2

1

R

1

1

1

1

1

1

1

1

Mon 1/6/3 - 14:09:43

Pfr

0.07

0.03

0.04

0.03

0.01

0.00

0.00

0.00Operating Point

All

owab

le I

nac

cura

cy (

bia

s, %

)

Allowable Imprecision (s, %)

Mathematical basis for a chart of operating specifications that shows the bias and imprecision that are allowable for different IQC procedures, whose control rules and Ns are given in the key at the right.


611

0.0

1.0

2.0

3.0

4.0

5.0

6.0

7.0

8.0

9.0

10.0

0.0 1.0 2.0 3.0 4.0 5.0

13s/2of32s/R4s/31s/6x

13s/22s/R4s/41s

12.5s

N

6

4

4

R

1

1

1

Mon 1/6/3 - 14:19:03

Pfr

0.07

0.03

0.04Operating Point

All

owab

le I

nac

cura

cy (

bia

s, %

)

Allowable Imprecision (s, %)

OPSpecs Chart TEa 10% with 90%AQA(SE)

Plo

t your o

bserv

ed C

V and b

ias

Application of the OPSpecs tool by plotting the observed method bias as the y-coordinate and the observed method precision as the x-coordinant to describe the “operating point” of an analytical method.


611

1. Define quality requiredfor diagnostic test, %TEa

2. Assess method%bias, %CV

3. Calculate normalizedoperating point

4. Plot on normalizedOPSpecs charts or EZ rules3

5. Inspect normalized OPSpecs charts

6. Select control rules,number of measurements

7. Adopt Total QC strategy

8. Reassess for changes

A practical procedure for planning and selecting IQC procedures.


HI-PedStrategy

IQC

Other QC

QI

MOD-PedStrategy

IQC

Other QC

QI

LO-PedStrategy

IQC

Other QC

QI

Total quality control strategies (TQC) that illustrate the proper combination of IQC with instrument and function checks (Other QC) and need for quality improvement (QI).


611

Minimize cost ofstatistical QC

Minimize cost ofnon-statistical QC

Maximizeerror detection

Maximizenon-statistical QC

Improve methodperformance

Document TQCsystem

Maximizeerror detection

Maximizenon-statistical QC

Improve methodperformance

Optimize QC forprocess stability

Deploy skilledanalysts

Add patientdata QC

Apply QP processwith OPSpecs

charts

AQA orPed

HI-Ped (90%)

MOD-Ped (50%)

LO-Ped (<50%)

A detailed flowchart to guide the development of TQC strategies.


611

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

0.0 1.0 2.0 3.0 4.0

12.5s

13s

13.5s

14s

N

2

2

2

2

R

1

1

1

1

Thur 8/30/1 - 09:49:08

Pro

bab

ilit

y fo

r R

ejec

tion

(P

)

Critical-Error Graph (SE)


Albumin

CO2

Chloride

OTHER TESTSCrit SE > 4.0

Creatinine, T Bil, UAPotassium, GGT,

ALP, PhosphorousBUN, CholesterolAST, LD, Glucose

Total Protein

Calcium

Results of an example application of IQC design for a multitest chemistry analyzer.


Select QC design

from drop-down

list

Select material

from drop-down

list

Entercontrolresult

Programcalculates

z-value

Out-of-controlidentified by

red bar

STARTUPQC design uses

multirule

MONITORQC design uses

2.5s limits

Programdisplaysz-valueon scale-4 to +4

Multi-chart displayshows multiple materials

and multiple designs with multi-rule or single-rule

QC procedures

Implementation of a multistage IQC strategy.


Interfaces orManual data entry

Laboratory testing

processes

Internal QCAccept/Reject

EQA data transfer

EQAData Analysis

IQC PlanningAnd Design

Inservice training

Validation tools

DocumentationValidation Reports, Peer Reports

Toubleshooting

Inventory, ordering

Production, shippingSupplies & materials

Internet SupportServices

A Total Quality Control Support System that integrate IQC and EQA


611

What to do?• Do define the quality required for the test

• Do select QC procedures that minimise false rejections

• Do select QC procedures that detect medically important errors

• Do adopt modern QC planning tools and techniques

• Do standardise QC operations

• Do calculate control limits from your own laboratory data

• Do provide computer support to analyse and interpret QC data

• Do reject out-of-control runs, identify the problem, and eliminate the cause

• Do adopt a Total QC strategy to maximise the cost-effectiveness of QC

• Do calculate daily/weekly/monthly patient means

What not to do?

• Don’t use 2SD control limits

• Don’t just repeat the controls

• Don’t use the same control rules for all tests

• Don’t use bottle values to calculate control limits

• Don’t use medical decision limits as control limits

• Don’t rely on electronic QC alone

• Don’t eliminate SQC in POC applications

Additional methods of IQC

Average of Normals (AoN, Daily patient means)

Standard signals

Experience

Expertise

IQMMonitors system performance real time

Automatically detects potential failures that affect the analytical performance

Automatically performs corrective action

Automatically documents failures and actions taken

Provides quality control reports

IL GEM Premier 4000

IQC Edward Kearney. Why is IQC/EQA important? To ensure the test result is fit for purpose.

Documents

Transcript of IQC Edward Kearney. Why is IQC/EQA important? To ensure the test result is fit for purpose.