Setting of quality standards
Transcript of Setting of quality standards
Setting of quality standards
Graham JonesDepartment of Chemical Pathology
St Vincent’s Hospital, SydneyAACB ASM – Adelaide – October 2014
Setting of Quality Standards - 2013
• The 2013 QC workshop revealed that mostAustralian laboratories are not consciouslyadopting performance goals and many are notensuring that their QC algorithms have highdetection of critical errors.
• This is worrying to say the least ….
• Quality Limits are numerical limits for a testoutside which we do not wish to release results(only one of us had them)
NPAAC
G4 Laboratories should set routine performancegoals … based on the clinical use of the testresults.
Approach to Quality Specifications
IFCC, IUPAC, WHO Consensus Conference, Stockholm 1999www.westgard.com/stockholm.html
Stockholm Consensus Conference on QualitySpecifications in Laboratory Medicine
1. Studies on clinical outcomes2. Clinical decisions in general, data from:
– biological variation– clinicians’ opinions
3. Published professional recommendations4. Performance goals set by regulatory bodies or
organisers of External Quality AssessmentSchemes.
5. Goals based on the current state of the art asdemonstrated by data from EQA or from current
Approaches
• Set / Select Quality Limits - design system tomeet limits– 6 sigma– Capability
• Assess performance against standards (UM)– Optimal, Desirable, Minimal– Improve where needed
• Reverse Engineering– Know how good you are
Quality Limits
• Eg: AST result should be within 20% of correctresult (CLIA Guidelines)
• Questions arising1. What is correct result?2. How do we achieve this?
Capability
ImportantChange
Sigma Metric
σ = Change/SD
Capability
ImportantChange
σ = 6, good
1 SD Shift
2 SD spreadσ = 4, OK
σ = 3, poor
2 SD Shift
4 SD Shift
Required shiftdetection
Sigma Metric
Capability
ImportantChange
Sigma Metric
σ = Change/SD
Capability
ImportantChange
Sigma Metric
σ = (Change-Bias)/SD
Power Function Graph (n=2)
12.5s
13.5s
MR
12s
Quality Limits
• Meets process needs• Set goals based on clinical need in advance• Establish QC protocols based on:
– Defined need– Assay Precision
• Issues:– What are the limits to use?– How do I handle Bias?
Assess performance against standards (MU)
• Run assay over a period of time• Obtain Precision data from QC• Compare performance against Highest level of
Stockholm criteria
• Typically within-subject Biological Variation (Cvi)– Level 2a– Compare assay precision (Cva) against CVi
Precision Goals
0%
5%
10%
15%
20%
25%
30%
35%
40%
0 0.2 0.4 0.6 0.8 1
CVa / CVwi
Incr
ease
in to
tal C
V (%
)
Optimal (0.25, 3%)
Minimum (0.75, 25%)
Desirable (0.5, 12%)
Assess performance against standards (UM)
Stockholm Level 2a
• CVa < 0.25 CVi - Optimal• CVa < 0.5 CVi - Desirable• CVa < 0.75 CVi - Minimal
UM Approach
• UM is required by ISO 15189 and NPAAC• Assess performance using real data
– Typically several month’s QC data– For new assay use limited run-in data
• But– Analysis performed AFTER setting up assay
(can use manufacturers estimate of CVa)– QC data may have outliers excluded– If fails higher standard (eg BV), use lower limit
(eg state of the art)– Assessment not planning
Reverse Engineering
Recommended process:
• Set goals, use processes to meet goals
Reverse
• Assess processes to understand performance
“How bad may a result from my lab be?”
Understanding our assays
• For any assay, with the QC protocol in place, weshould be able to say how much analytical errormay occur.
“These rules have the power to cause a STOP 90% ofthe occasions when there is a shift in the assay of 2.8x LSD and cause a PAUSE 90% of the occasionswhen there is a shift in the assay of 2.6 x LSD.”
SydPath Quality Control SOP
So far…
• Quality Limits
• UM
• Reverse Engineering
Quality Limits
• What Limits?
http://www.datainnovations.com/products/ep-evaluator/allowable-total-error-table
http://www.dgrhoads.com/db2004/ae2004.php?B1=Chemistry+A-C&find=&start=1&NOLINKS=
RCPAQAP(%) 5.0 10.0 8.0 15.0 7.8 15.0 15.0CLIA (%) 10.0 20.0 20.0 17.0 30.0 30.0Range: 3-18 5-14 5-22 3-21 5-18 7-30 10-56
RCPAQAP ALP
Limits/acceptability
Meaning of ALP
Basis“Total Error” – Can share reference interval“Imprecision” – Can Monitor across labs
Level“Optimal” – no need to improve“Desirable” – satisfactory“Minimal” – just satisfactory
Revision of ALP
ALP are applied to Total ErrorUsed in interim reportsSingle results include bias and imprecision
Will use categories of CV:1,2,3,4,5,6,8,10,12,15,20,25,30%
Round to nearest category
Change between absolute and percentagebased on precision profile
Revision of ALPTop category (Imprecision):
Within-Subject Biological Variation (Opt, Des, Min)MonitoringSingle Laboratory reaching standard:
Can monitor a patient at labMany Labs within standard:
Can monitor a patient between labsCan share reference intervals
Revision of ALPNext Category:
Total Error (Opt. Des, Min)Within and between subject BV combinedDiagnosisSingle Laboratory reaching standard:
Satisfactory performanceBias and / or precision target for improvement
Multiple Labs meeting standardLikely to be able to diagnose at different labs
(share reference intervals)
Revision of ALP
Final CategoriesState-of-the-art
If unable to meet a higher category
Expert OpinonIf suitable data not available
Using QAP ALP as Quality Limits
Vanessa Lo
Sigma Metrics as Performance IndicatorContributes to Effective Cost and
Man-hour Saving in Chemical PathologyLaboratory
Department of Clinical PathologyChemical Pathology Laboratory
Hong Kong Sanatorium and HospitalRoche Oral Presentation PrizeAACB ASM 2014
>= 5 Sigma17 analytes
ALTPancreatic Amylase
Total AmylaseAST
Direct BilirubinCK
GlucoseGGTLDH
MagnesiumPhosphateTriglyceride
Uric AcidBicarbonate
HDL-CLDL-CCRPLX
4 Sigma5 analytes
ALPCalcium
CholesterolIron
Total Protein
3 Sigma7 analytes
AlbuminTotal Bilirubin
CreatinineUrea
SodiumPotassiumChloride
Chemistry
16* 7 4
SydPath*
AlbuminSodiumChlorideBicarbonate
>= 5 Sigma12 analytesCKMB-STAT
pro-BNPhs-TNT
beta-HCGE2
FSHLH
ProgesteroneProlactin
ThyroglobulinPTH-STAT
Ferritin
4 Sigma1 analyte
B12
3 Sigma2 analytes
SFolateVit. D Total
Immunoassay
Sigma No. of QCdaily
No. of Analytes
Chemistry Immunoassay
>=5 1 17 12
4 2 5 1
3 3 7 2
Vanessa Lo
• Reduction in QC performance
• Saving of 1.6% of reagent costs
• “Uprising in the emotional status of the staff”
USE OF ALP as Quality Limits
• >5 Sigma: Can have confidence that results willbe within RCPAQAP ALP using simple rules
• 4-5 Sigma: need tighter rules to be sure resultswill meet RCPAQAP ALP
• <4 Sigma: Cannot be confident that Results willmeet RCPAQAP ALP
Will see poor results?
RCPA General Serum Chemistry
CYCLE 90 1,2 3,4 5,6 7,8 9,10 11,12, 13,14 15,16Working mean 0.21 0.25 0.24 0.22 0.26 0.18 0.24 0.29Working 80th 0.33 0.40 0.37 0.40 0.38 0.33 0.40 0.45
Outliers 0 0 0 0 0 0 0 0
No outliers* in entire cycle for any analyte!(35 analytes, 1120 Results)
* Method Specific Targets (includes bias)
RCPA General Serum ChemistryCycle Flagged Test
91 4 PO4(2),FT4,GGT90 089 3 hCG(2), Cl88 11 FT4(5),HDL(2), Bic,Lip,Na,K87 2 Ca, FT485 3 Ca, FT4(2)84 1 Ca79 4 Ca,Ferr,TG,Bil,Ca76 8 Ferr,Cl,Bic(2)gluc,Cbil75 1 Ferr
TOTAL 37 0.66% of total
> 3 years: No albumin failures, 1 Na, 1 CO2, 2 ClAll Incapable (Sigma <4)
Roche Modular BCG Albumin (2014)
Albumin
QC Level1:CVa = 2% @ 31 g/L (ALP 6.4%), Sigma = 3.2
QC Level 2:Cva = 1.9% @ 46 g/L (ALP 6.0%), Sigma = 3.2
Albumin CVi = 3.1%(both QC levels meet minimal Standard)
An incapable assay – how does it succeed?
Assay Characteristics
Stable assays:• Performance defined by mean and SD• QC never fails• Results “always” within +/- 2SD
Stable Assay
14
16
18
20
22
24
26
1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49
QC-3SD-2SDMean+2SD+3SD
Mean = 20, SD = 195% of QC results between 18and 22.
Assay Characteristics
Unstable Assays• Mean drifts over time (fluctuating bias)• QC process used to detect drifts• Variation in results due to scatter plus drift
Unstable Assay
14
16
18
20
22
24
26
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37
-3SD-2SDMean+2SD+3SDQC
Mean = 20, SD = 1, Plus fluctuating mean.Interpretation: Result of 20 has 95% confidence limit of18 – 22, PLUS undetected bias at time of measurement.
Put all this together…
Put all this together…
• 1. Setting Quality Limits provides the mostrobust approach
• What Limits to use?• RCPAQAP (where capable)• Others (where not capable against RCPAQAP)
• ALP different for different analytes, What limits,Complexity…
What I do…• Compare CVa with CVi• If CVa small relative to CVi:
few rules, wide limitsn=2, 1 x approx 3SD
• If CVa not small relative to Cvi:few rules, tighter limitsn=2, 1 x approx 2.2 SD
• Understand performance risks (ReverseEngineering)
• Compare with quality standards regularly(Cvi, RCPAQAP ALP, state of the art – QAP, PI)
What can we do…
• Define limits based on CVi
• Turn internal process into Logical steps
• Remember stat of the art
Bias: what I do
• Consider separately to precision• Base on Fraser concepts: percent of patients
wrongly classified• Eg ~one 10th of population reference interval
(if Gaussian)
Results Change ProtocolANALYTE ALLOWABLE LIMITSAlbumin +/- 4 up to 40 g/L, then +/- 10%ALT +/- 8 up to 60 U/L, then 15% (NCIRI)Amylase +/- 15% (NCIRI)AP +/- 15% (NCIRI)AST +/- 8 up to 60 U/L, then 15% (NCRI)Bicarbonate +/- 4 mmol/L (6 mmol/L IRI)Bilirubin +/- 8 up to 60 umol/L, then 10% (NCIRI)Calcium +/- 0.2 mmol/L (NCIRI)Chloride +/- 5 mmol/L (NCRI)Cholesterol +/- 0.5 up to 10 mmol/L, then 5%CK +/- 15 up to 100 U/L, then 15%CK-MB +/- 2 ug/L to 7 ug/L, then 15% (NTIRI)Creatinine +/- 0.02 up to 0.2 mmol/L, then 10%Fructosamine +/- 30 up to 300 umol/L, then 10%GGT +/- 8 up to 60 U/L, then 15% (NCIRI)Glucose +/- 0.4 up to 3 mmol/L, then 15% (1 mmol/L IRI)Glucose-GTT +/- 0.4 if crosses barrier otherwise 15% (1)HDL-C +/- 0.2 up to 2 mmol/L then 10%Iron +/- 5 umol/L (8 umol/L IRI)Lactate +/- 1 up to 5 mmol/L, then 20%LD +/- 40 up to 200 U/L, then 20% (NCIRI)Lipase +/- 20 up to 100 U/L, then 20% (NCIRI)Lithium +/- 0.2 mmol/LMagnesium +/- 0.12 mmol/L (0.2 mmol/L IRI)Omolality +/- 8 mmol/kgPhosphate +/- 0.1 up to 1.0 mmol/L, then 10% (0.3 mmol/L IRI)Potassium +/- 0.3 mmol/L (0.5 mmol/L IRI)Protein +/- 6 g/L (8 g/L IRI)Sodium +/- 4 mmol/L (6 mmol/L IRI)Transferrin +/- 0.5 g/LTriglyceride +/- 0.4 up to 3.0 mmol/L then 15%Urate +/- 0.05 mmol/L (0.07 IRI)Urea +/- 1 up to 10 mmol/L, then 10% (2 mmol/L IRI)
Free T4 +/- 3 pmol/L up to 20 pmol/L then 15%TSH +/- 0.6 mU/L up to 4 mU/L then 15% (2)
In the event of re-running an assay following the suspicion of an analytical error,significant changes must be changed in the computer and notified to therequesting/treating doctor. Changes equal to, or greater than those shown belowmay be considered significant (these values based on the RCPA-AACB Allowablelimits of performance). If in doubt, consult the Pathologist or Senior Scientist.
• Thank you