Total Analytical Error Evaluation in Quantitative and Qualitative Assays
description
Transcript of Total Analytical Error Evaluation in Quantitative and Qualitative Assays
Total Analytical Error Total Analytical Error Evaluation in Quantitative Evaluation in Quantitative
and Qualitative Assaysand Qualitative Assays May F. MoMay F. Mo
FDA/Industry Statistics WorkshopFDA/Industry Statistics Workshop
Washington, DC - September 29, 2006Washington, DC - September 29, 2006
BackgroundBackground
Total Analytical Error (TAE)Total Analytical Error (TAE) NOT a new conceptNOT a new concept Assay errors from Assay errors from all sourcesall sources of the of the
data collection experiment – including data collection experiment – including systematic and random errorsystematic and random error
CLSI Proposed Guideline January 2002CLSI Proposed Guideline January 2002 CLSI Approved Guideline April 2003CLSI Approved Guideline April 2003 FDA/Industry collaboration in FDA/Industry collaboration in
developing the guidelinedeveloping the guideline
Cause and Effect DiagramCause and Effect Diagramfor Total Analytical Errorfor Total Analytical Error
(from EP21-A)(from EP21-A)
Total Analytical Error
EP21
Random Error Systematic Error
Protocol Specific BiasEP10
ImprecisionEP5
Random Interferences
Protocol Independent
BiasEP9/EP6/EP7
DriftEP10
Sample CarryoverEP10
Constant BiasEP9
Linear BiasEP9
Non-linear BiasEP6
Reagent CarryoverInterferences
EP7
Importance of TAEImportance of TAE
For CliniciansFor Clinicians TAE provides a metric of assay quality TAE provides a metric of assay quality
that can directly equate to medical errorsthat can directly equate to medical errors For LaboratoriesFor Laboratories
TAE provides a simple, cost effective TAE provides a simple, cost effective method for comparing assay productsmethod for comparing assay products
For ManufacturersFor Manufacturers TAE guides manufacturers to set TAE guides manufacturers to set
appropriate design goals and evaluate appropriate design goals and evaluate the product directly to user needsthe product directly to user needs
TAE Evaluation MethodsTAE Evaluation Methods
TAE is evaluated by assessing if the TAE is evaluated by assessing if the combination of errors from all combination of errors from all sources is within some acceptance sources is within some acceptance limit. The common evaluation limit. The common evaluation methods are (Krouwer, 2002):methods are (Krouwer, 2002): Direct EstimationDirect Estimation Indirect EstimationIndirect Estimation Simulated EstimationSimulated Estimation
Note: Note: All methods are based on the All methods are based on the quantitative assays.quantitative assays.
Direct EstimationDirect Estimation
Direct EstimationDirect Estimation ( (Distribution of Distribution of Differences MethodDifferences Method)) EP21 RecommendationEP21 Recommendation Calculate differences of study assay vs. Calculate differences of study assay vs.
reference assay reference assay TAE limit estimated as the CLI of the TAE limit estimated as the CLI of the
differencesdifferences
- No need to estimate and combine individual - No need to estimate and combine individual source errorssource errors
Visual tools - Bland & Altman and “Mountain Visual tools - Bland & Altman and “Mountain Plot” Plot”
Direct Estimation Direct Estimation ((ContinuedContinued))
Is distribution of differences normal?Is distribution of differences normal?Parametric? Nonparametric? or Data Parametric? Nonparametric? or Data
Transformation?Transformation? Tolerance IntervalTolerance Interval: A range that contains a : A range that contains a
specified proportion of a sampled population specified proportion of a sampled population with a specified confidence. (EP21)with a specified confidence. (EP21)
Parametric or nonparametric methods to Parametric or nonparametric methods to report tolerance intervals are covered in report tolerance intervals are covered in Statistical IntervalsStatistical Intervals by Hahn and Meeker by Hahn and Meeker (1991). (1991).
Tolerance Interval (TI) Tolerance Interval (TI) CalculationCalculation
A two-sided 100(1-A two-sided 100(1-)% TI to contain at least 100)% TI to contain at least 100pp% % of the sampled differences from a sample of size n:of the sampled differences from a sample of size n:
Parametric:Parametric:
where k depends on where k depends on , , pp and and n, n, and can be and can be obtained from published tables, or derived using obtained from published tables, or derived using noncentral t-distribution (Odeh and Owen, 1980)noncentral t-distribution (Odeh and Owen, 1980)
NonparametricNonparametric
TI based on rank statistics is [TI based on rank statistics is [xx(l)(l), , xx(u)(u)].].The value of The value of ll and and uu depend on depend on , , pp and and n, n, and can and can be derived using published tables, or using be derived using published tables, or using equation provided in Hahn and Meeker (1991)equation provided in Hahn and Meeker (1991)
)*( skx
Indirect EstimationIndirect Estimation
Indirect EstimationIndirect Estimation ( (Simple Combination Simple Combination ModelModel)) TAE = Bias + k * ImprecisionTAE = Bias + k * Imprecision Simple to applySimple to apply Fits well with the Six-Sigma quality conceptsFits well with the Six-Sigma quality concepts Not recommended by EP21 - Underestimate TAENot recommended by EP21 - Underestimate TAE
Simulation Method (Simulation Method (Research UseResearch Use)) Cost-efficientCost-efficient Model the impact of an individual error component Model the impact of an individual error component
to TAEto TAE Can underestimate TAECan underestimate TAE
Industry PracticesIndustry PracticesQuantitative AssaysQuantitative Assays
Set Product RequirementSet Product Requirement Set allowable TAE (aTAE) limit based on Set allowable TAE (aTAE) limit based on
medical opinion and marketing needsmedical opinion and marketing needs Break aTAE down to precision and bias design Break aTAE down to precision and bias design
requirements using simple combination model. requirements using simple combination model.
Example: TAE = Bias + 2*SD (for 2 sigma Example: TAE = Bias + 2*SD (for 2 sigma performance)performance)
Design Verification and ValidationDesign Verification and Validation Verify design requirements individuallyVerify design requirements individually A direct assessment of total error should be A direct assessment of total error should be
conducted to ensure aTAE is met.conducted to ensure aTAE is met.
Industry PracticesIndustry PracticesQuantitative AssaysQuantitative Assays
TAE Claims in Package InsertsTAE Claims in Package Inserts Not Yet CommonNot Yet Common Concerns?Concerns?
Indirect method – Is it sufficient to establish a Indirect method – Is it sufficient to establish a TAE claim?TAE claim?
Direct method - When the reference assay is Direct method - When the reference assay is not a “gold standard”, TAE has measurement not a “gold standard”, TAE has measurement error from both assays … How to interpret?error from both assays … How to interpret?
User environment – Are our customers ready User environment – Are our customers ready to use TAE claims in the package inserts? to use TAE claims in the package inserts?
Generalize TAE toGeneralize TAE toQualitative AssaysQualitative Assays
MotivationMotivation TAE provides rationale to establish TAE provides rationale to establish
product requirementsproduct requirements TAE provides a final metric to TAE provides a final metric to
evaluate if a product meets medical evaluate if a product meets medical needsneeds
ChallengeChallenge TAE is a quantitative conceptTAE is a quantitative concept TAE does not apply to low negative TAE does not apply to low negative
and high positive samples and high positive samples
TAE in Qualitative AssaysTAE in Qualitative Assays
The performance of a qualitative The performance of a qualitative assay relies on its precision and bias assay relies on its precision and bias near the medical decision point near the medical decision point (cutoff) … (cutoff) … TAE evaluation applicable and desirable TAE evaluation applicable and desirable
for the cutoff zonefor the cutoff zone Direct estimation not realistic due to Direct estimation not realistic due to
limited patient samples around cutofflimited patient samples around cutoff Indirect modeled approach may be more Indirect modeled approach may be more
practicalpractical
aTAE in Qualitative aTAE in Qualitative AssaysAssays
How to define aTAE for the near cutoff How to define aTAE for the near cutoff zone?zone?
EP12-A Guidance:EP12-A Guidance: The 95% interval - the values above
and below the cutoff point at which repeated results are 95% positive or 95% negative, respectively
Grayzone - the zone of uncertainty between the 95% interval limits
aTAE in Qualitative aTAE in Qualitative AssaysAssays
Cutoff
5%5%
Upper 95%Interval
Lower 95%Interval
Grayzone
aTAE in Qualitative aTAE in Qualitative AssaysAssays
Set aTAE for the near cutoff zoneSet aTAE for the near cutoff zone Incorporating measurement errors from both
precision and bias, aTAE can be set as ½ of the grayzone. That is, the test result at 1.2 S/CO should read as positive (>1.0 S/CO) while test results at 0.8 S/CO should read as negative (<1.0 S/CO). Assuming that grayzone is 0.8-1.2 S/CO for an assay, aTAE will be 0.2 S/CO or 20% (assuming cutoff=1.0 S/CO)
When the above aTAE is met, there should be < (i.e., 0.05) probability that a test results will deviate >20% from the true value, and cause a change in the final interpretation.
aTAE in Qualitative aTAE in Qualitative AssaysAssays
Set aTAE for the near cutoff zoneSet aTAE for the near cutoff zone For an assay that reports an indeterminate
result, aTAE can be extended as the size the grayzone.
The test result at 1.2 S/CO should not read as negative (<0.8 S/CO) while test results at 0.8 S/CO should not read as positive (>1.2 S/CO). In this case aTAE can be set at 0.4 S/CO or 50% for results around 0.8 S/CO and 33% for results around 1.2 S/CO.
TAE Evaluation TAE Evaluation for Qualitative assayfor Qualitative assay
Direct Evaluation – Not realisticDirect Evaluation – Not realistic Indirect EvaluationIndirect Evaluation
TAE = Bias + TAE = Bias + * Precision * PrecisionChallenge:Challenge: How to define and measure How to define and measure Bias?Bias?
How to include ALL bias How to include ALL bias sources?sources?
Alternative: TAE = Alternative: TAE = * Reproducibility * ReproducibilityReproducibility includes variations contributed Reproducibility includes variations contributed from instrument, lot, calibration, operator, etc from instrument, lot, calibration, operator, etc … …
Underestimate TAE is still then main concern
TAE in Qualitative AssaysTAE in Qualitative Assays
How do different types of bias impact TAE?How do different types of bias impact TAE? Bias impacts both S and COBias impacts both S and CO
As a proportional biasAs a proportional bias
The will be no bias in S/COThe will be no bias in S/CO As a fixed biasAs a fixed bias
The bias in S/CO is unknown and depends on The bias in S/CO is unknown and depends on the value of S relative to CO (=1.0) and size the value of S relative to CO (=1.0) and size of the biasof the bias
COSCO
S
COCO
SSCOS /*
1
1
*
*/
unknownCO
SCOS
/
CO=1.0 Bias
S -0.2 -0.1 0.1 0.2
0.8 0.75 0.78 0.82 0.83
0.9 0.88 0.89 0.91 0.92
1.0 1.00 1.00 1.00 1.00
1.1 1.13 1.11 1.09 1.08
1.2 1.25 1.22 1.18 1.17
TAE in Qualitative AssaysTAE in Qualitative Assays
How does different type of biases impact How does different type of biases impact TAE?TAE? Fixed bias for both S and CO: The impact on Fixed bias for both S and CO: The impact on
S/CO is relatively small for the near cutoff S/CO is relatively small for the near cutoff range (0.8-1.2 S/CO)range (0.8-1.2 S/CO)
Biases that impact S and CO differently:Biases that impact S and CO differently: InterferencesInterferences Tube TypeTube Type StorageStorage Sample Carry OverSample Carry Over Crossreactivity …Crossreactivity …
ConclusionConclusion
TAE is an useful evaluation method TAE is an useful evaluation method for both quantitative and qualitative for both quantitative and qualitative assays. For qualitative assays, TAE assays. For qualitative assays, TAE should be evaluated for the near should be evaluated for the near cutoff zone using indirect estimation. cutoff zone using indirect estimation. Since indirect method Since indirect method underestimates TAE, it may not be underestimates TAE, it may not be sufficient to support a claim. sufficient to support a claim.
ReferencesReferences NCCLS. Estimation of Total Analytical Error for Clinical NCCLS. Estimation of Total Analytical Error for Clinical
Laboratory Methods; Approved Guideline. NCCLS Laboratory Methods; Approved Guideline. NCCLS Document EP21-A [ISBN 1-56238-502-X]. NCCLS, 940 West Document EP21-A [ISBN 1-56238-502-X]. NCCLS, 940 West Valley Road, Suite 1400, Wayne, Pennsylvania 19087-1898 Valley Road, Suite 1400, Wayne, Pennsylvania 19087-1898 USA, 2003.USA, 2003.
NCCLS. User Protocol for Evaluation of Qualitative Test NCCLS. User Protocol for Evaluation of Qualitative Test Performance; Approved Guideline. NCCLS Document EP12-Performance; Approved Guideline. NCCLS Document EP12-A [ISBN 1-56238-468-6]. NCCLS, 940 West Valley Road, A [ISBN 1-56238-468-6]. NCCLS, 940 West Valley Road, Suite 1400, Wayne, Pennsylvania 19087-1898 USA, 2002.Suite 1400, Wayne, Pennsylvania 19087-1898 USA, 2002.
Hahn GJ, Meeker WQ. Statistical Intervals: A Guide for Hahn GJ, Meeker WQ. Statistical Intervals: A Guide for Practitioners. New York: John Wiley and Sons, 1991.Practitioners. New York: John Wiley and Sons, 1991.
Krouwer JS, Monti KL. A simple graphical method to Krouwer JS, Monti KL. A simple graphical method to evaluate laboratory assays. Eur J Clin Chem Biochem 1995; evaluate laboratory assays. Eur J Clin Chem Biochem 1995; 33:525–7.33:525–7.
Odeh, RE, Owen, DB. Tables for Normal Tolerance Limits, Odeh, RE, Owen, DB. Tables for Normal Tolerance Limits, Sampling Plans, and Screening. New York Marcel Dekker, Sampling Plans, and Screening. New York Marcel Dekker, 1980.1980.