This project has been funded in whole or in part with Federal funds from the Division of AIDS...
-
Upload
marcel-isaac -
Category
Documents
-
view
232 -
download
0
Transcript of This project has been funded in whole or in part with Federal funds from the Division of AIDS...
This project has been funded in whole or in part with Federal funds from the Division of AIDS (DAIDS), National Institute of Allergy and Infectious Diseases, National Institutes of Health, Department of Health and Human Services, under contract No. HHSN272201200009C, entitled NIAID HIV and Other Infectious Diseases Clinical Research Support Services (CRSS).
Verification of Performance Specifications
An Advanced View of Method Validation
Version 5.0, August 2012
2
Identify test classifications Define what each validation experiment details for
testing methods Discuss what is recommended to perform each of the
validation experiments for testing methods Recognize how to evaluate data obtained from each of
the validation experiments
Objectives
3
A rapid Human Immunodeficiency Virus (HIV) test would likely be classified as a:
A. High complexity, modified assayB. Moderate complexity, unmodified assayC. Food and Drug Administration (FDA)-approved,
modified assayD. Waived, FDA-approved, unmodified assay
Pre-Assessment Question #1
4
The precision of a test method gives information related to the method’s:
A. Systematic errorB. Comparison of results to a reference methodC. ReproducibilityD. Likelihood of being affected by hemolysis, lipemia and
icterus E. Both A and B
Pre-Assessment Question #2
5
When transferring reference intervals of 20 specimens used, what is the minimum number that must fall within manufacturer’s reference intervals?
A. 20B. 18 C. 16D. 15
Pre-Assessment Question #3
6
Which linear regression equation component gives information regarding constant bias?
A. yB. xC. m (slope)D. b (intercept)
Pre-Assessment Question #4
7
Selecting a Method
Evaluate diagnostic tests Characteristics of testing
methods References: Technical literature
and manufacturer’s information Select method of analysis Validate method performance Implement method Perform tests with appropriate
Quality Control (QC) and External Quality Assurance (EQA)
Method Validation
Why must we validate?
When should we validate?
What should we validate?
8
What is method validation?
9
Why is validation important? Division of Acquired Immunodeficiency Syndrome
(DAIDS) requirement How important is it that the results produced by the
testing method are reliable? Shouldn’t the laboratory know the level of performance
of an adopted test method?
Method Validation (cont’d)
Tests to Validate
Waived
Non-waived
• Unmodified FDA-approved
• Modified and/or Non-FDA-approved
10
11
Vendor Publications http://www.fda.gov/MedicalDevices/
ProductsandMedicalProcedures/InVitroDiagnostics/LabTest/ucm126079.htm
FDA Approval Resources
12
What would you consider to be the complexity, per Clinical Laboratory Improvement Amendments (CLIA), of the glucose assay in the workbook?
A. WaivedB. ModerateC. High
Skill Check
13
What would you consider to be the complexity of a rapid urine pregnancy assay?
A. WaivedB. ModerateC. High
Skill Check
14
What would you consider to be the complexity of performing a manual white cell differential using a stained whole blood smear?
A. WaivedB. ModerateC. High
Skill Check
15
Before you begin: Be sure you are familiar with the test method before
starting Know what to expect from the method (package insert,
discussions with technical assistance, and field service representatives)
Do not include results outside of stated reportable ranges
Predict your findings; establish limits/evaluation criteria
Method Validation
16
Terms for Discussion
Central Tendency
Dispersion
17
Terms for Discussion (cont’d)V
alue
s
Run
18
Some error is expected Examples
Error must be managed Understanding Defining specifications of allowable error Measurement
Error in Test Methods
19
Total Error of Testing System
Total Allowable Error• CLIA Guidelines per analyte• Other Guidelines
Systematic Error
Random Error Total Error
20
Error Assessment
In either direction,
unpredictable
In one direction, cause results to be high or low
Combined effect
Systematic Error
(SE)
Random
Error
(RE)
Total Error
(TE)
21
Low End Performance Standards Recommendations derived from upper portion of
reportable range are more difficult to achieve at lower concentrations
Maximum Total Error Allowed Considered to be 30% by David Rhoads, except for
amplification methods
Total Error Considerations
22
Systematic Error Slope/Proportional error Intercept/Constant error Bias
Random Error Mean Standard deviation (SD) Coefficient of variation (CV)
Systematic and Random Errors
23
Tools for Use
Spreadsheets
with
calculationsValidation
Software (Westgard, Analyze-
It, EP Evaluator)
Statistical
calculators,
graph paper
Data-Crunching
Tools
24
One quantitative test taken through the validation process
One qualitative method taken through the validation process
How We Will Work Through This Module
Reportable Range
Precision
Accuracy
Reference Intervals
Sensitivity
Specificity
Elements of Validation
25
Repeat testing over short and long term (one day and 20 days, respectively)
20 samples of same material (typically two levels; e.g., Glucose at 50 and 300 mg/dL)
Standard solutions Control materials Pools (short term only)
Precision Definition: Reproducibility Gives information related to random error
Introduction
What is needed
How we perform
the testing
26
27
Precision: How We Evaluate the Data
Mean Standard deviation (SD) Coefficient of Variation (CV)
Short term: 0.25 of allowable total error Long term: 0.33 of allowable total error
Calculate the following:
What amount of random error is allowable, based on CLIA criteria?
28
Link for: Clinical Laboratory Improvement Amendments (CLIA) College of American Pathologists (CAP) Royal College of Pathologists of Australasia (RCPA) Others
http://www.dgrhoads.com/db2004/ae2004.php
Allowable Total Error Database
29
Precision: Levey-Jennings (LJ) ChartsV
alue
s
Run
30
Precision: How We Evaluate the Data
Mean SD CV: More commonly used, allows for
easier comparison
How do we compare to manufacturer’s data?
31
Precision Example
Mean of Level 1 Glucose
CLIA Total Allowable Error
Total Allowable Error Level 1 Glucose
Random error allowed:
90 mg/dL
6 mg/dL or ± 10%
0.1 x 90 = 9 mg/dL
0.25 x total allowable
0.25 x 9 mg/dL
2.25 mg/dL
0.33 x total allowable
0.33 x 9 mg/dL
2.97 mg/dL
Long-term precision
Short-term precision
32
Work with Levey-Jennings graph and data Work with mean and standard deviation to calculate a
coefficient of variation, as well as a mean and a coefficient of variation to calculate a standard deviation
Determine if precision data is acceptable
Activity
Accuracy Definition: How close to the true value Comparison of methods Gives information related to systematic error Potential conflicts on interpretation of results
(reference values)
Introduction
40 different specimens Cover reportable range of method Quality versus quantity
What is needed
Duplicate measurements of each specimen on each method
Minimum of five days, prefer over 20 (since replicate testing is same)
How we perform the
testing
33
Accuracy: How We Evaluate the Data
Graph the Data:
Test method on Y-axis
Reference (comparative) method on X-axis
Shows analytical range of data, linearity of response over range and relationship between methods
Real time Difference plot
Comparison plot Calculate
y = mx + b
b represents constant error
m represents proportional error
34
35
Visual Inspection for AccuracyTe
st M
etho
d
Reference Method
Intercept
(x1, y1)
(x2, y2)Slope = (y2- y1) / (x2- x1)
36
Slope: Usually not significantly different from 1 Intercept: Not significantly different from 0 Significant difference with Medical Decision Points
Accuracy: How We Evaluate the Data
37
Slope Measure of proportional bias
m = (y1-y2)/(x1-x2) or “rise/run” Slope greater than 1 means the Y (Test) values are
generally higher than the X (Comparative) values Slope of 1.11 means the Y (Test) values are on
average 11% higher than the X (Comparative) values
Calculate Appropriate Statistics
38
Intercept of the Line Measure of constant bias between two methods
Y (Test) value at the point where the line crosses the Y axis
If Y intercept is 12, then all Y (Test) values are at least 12 units higher than the X (Comparative) values
Calculate Appropriate Statistics (cont'd)
39
Accuracy
What type of bias do you see?
Accuracy (cont’d)
Constant Bias Proportional Bias
40
41
Can a linear regression formula offer predictive value in relation to method comparisons?
A. YesB. No
Skill Check
42
Create graph based on sample set Determine slope from best-fit line Determine Y-intercept from best-fit line Explain the relationship between comparative and test
results
Activity
CLSI recommends four measurements of each specimen; three are sufficient
Series of samples of known concentrations (e.g., standard solutions, EQA linearity sets)
Series of known dilutions of highly elevated specimen or spiked specimens; EQA specimens
At least four levels (five preferred)
Reportable Range / Linearity Definition: Lowest and highest test results that
are reliable Especially important with two point calibrations Analytical Measurement Range (AMR) and
derived Clinical Reportable Range (CRR)
Introduction
What is needed
How we perform the
testing
43
44
Reportable Range:How We Evaluate the Data
Measured values on Y-axis versus Known or assigned values on X-axis
Plot mean values of:
Compare with expected values (typically provided by manufacturer)
Visually inspect, draw best-fit line, estimate reportable range
45
Reportable Range Activity
AssignedValue
Experimental Results
Average Rep #1 Rep #2 Rep #3 Rep #4
10.0 ____ 11.0 10.0 11.0 10.0
100.0 ____ 99.0 103.0 103.0 101.0
300.0 ____ 303.0 305.0 304.0 306.0
500.0 ____ 505.0 506.0 505.0 506.0
800.0 ____ 740.0 741.0 744.0 742.0
46
Reportable Range Activity (cont'd)
AssignedValue
Experimental Results
Average Rep #1 Rep #2 Rep #3 Rep #4
10.0 10.5 11.0 10.0 11.0 10.0
100.0 101.5 99.0 103.0 103.0 101.0
300.0 304.5 303.0 305.0 304.0 306.0
500.0 505.5 505.0 506.0 505.0 506.0
800.0 741.8 740.0 741.0 744.0 742.0
Reportable Range Activity (cont'd)
Linearity Scatter Plot
0.0
100.0
200.0
300.0
400.0
500.0
600.0
700.0
800.0
0 100 200 300 400 500 600 700 800 900
Assigned Concentrations (units)
Reco
vere
d V
alu
es (
Mean
s)
47
AMR vs. CRR
Analytical Measurement Range (AMR)
Linearity
Clinically Reportable Range (CRR)
Allows for dilution or other preparatory steps beyond routine
48
49
If you do not have enough specimen to perform a dilution, upon which reportable range component must you rely?
A. AMRB. CRRC. Neither A or BD. Both A and B
Skill Check
50
Utilizing the marketing materials from the two chemistry linearity kits in your handouts:
1. Determine which kit would be more appropriate for use with the chemistry assay you chose earlier
2. Explain your reasoning
Linearity Materials
51
Given your choice of linearity kits, you perform your AMR experiments by performing four replicates of each level of known concentration solution. The data you obtain is displayed on the next slide.
1. Review data; record any initial observations2. Graph data on supplied graph paper3. Determine your assay’s AMR
Graph Activity
Level Rep 1 Rep 2 Rep 3 Rep 4
1 24 23 25 24
2 196 197 171 194
3 359 360 358 361
4 530 532 529 535
5 700 695 702 709
Linearity Experiment Results
52
Activity
Using an Excel spreadsheet, create a graph and calculate
linear regression statistics from the data provided
53
54
Rep 1 Rep 2 Rep 3 Rep 4Lab's
AverageKnownConc
24 23 25 24 24 25
196 197 171 194 195.7 200
359 360 358 361 360 375
530 532 529 535 532 550
700 695 702 709 702 725
55
0
100
200
300
400
500
600
700
800
0 200 400 600 800
Re
cove
red
Known Concentration
AMR Verfication
56
Your medical director, in consultation with clinicians, determines that for proper study participant care the Clinically Reportable Range (CRR) for glucose is15 – 1400 mg/dL
Given your linearity experiment results and the package insert, devise a dilution protocol to be contained within our Glucose SOP
Dilution Protocols
57
Given your AMR, CRR, and dilution protocol, how would you handle the following analyzer results?
1. 12 mg/dL2. 800 mg/dL3. 1600 mg/dL
Reportable Results
58
Reference Intervals
Definition: Normal range in healthy population Used for diagnosis/clinical interpretation of
results
Introduction
Pre-defined “normal” criteria for screening purposes
Transferring: 20 “normal” individuals’ specimens Establishing: 120 “normal” individuals’ specimens
What is needed
Perform testing on all samples Document results
How we perform the
testing
Transferring Establishing
18 of 20 must fall within manufacturer’s ranges
Calculate mean and SD of data for each group
Reference Intervals = mean ± 2 SD (if Gaussian Distribution only, otherwise, additional calculations recommended)
59
Reference Intervals:How We Evaluate the Data
60
Activity
Determine if assay is eligible for transference of reference intervals
Review a sample set of data to determine if transference may be performed; if not, determine next step(s)
Sensitivity
Definition: Lowest reliable value; lower limit of detection, especially of interest in drug testing and tumor markers
Different terminologies used by different manufacturers
Introduction
Blank solutions Spiked samples
What is needed
20 replicate measurements over short or long term, depending on focus
How we perform the
testing
61
Sensitivity: How We Evaluate the Data
Lower Limit of Detection (LLD):
Mean of the blank sample, plus two or three SD of blank
sample
Biological Limit of Detection:
LLD plus two or three times SD of
spiked sample with concentration of detection limit
Functional Sensitivity:
Mean concentration for spiked sample whose CV = 20%; lowest limit where quantitative data is
reliable
Three methods used:
62
63
Activity
Using the manufacturer’s package inserts, find the
related information for sensitivity. How was it
calculated?
Specificity Definition: Determination of how well a method
measures the analyte of interest accompanied by potential interfering materials
Introduction
Standard solutions, participant specimens or pools
Interferer solutions (standard solutions, if possible; otherwise, pools or specimens) added at high concentrations
What is needed
Duplicate measurementsHow we
perform the testing
64
Specificity: How We Evaluate the Data
Tabulate results for pairs of samples (dilution and interferent)
Calculate means for each (dilution and interferent)
Calculate the differences Calculate the average interference
of all specimens tested at a given concentration of interference
65
66
Compare diagnosis Assume comparative (reference) method is accurate Determine the following:
True Positives, True negatives False Positives, False negatives
Calculate sensitivity and specificity and compare to manufacturer
Qualitative Assays
67
Negative and Positive Quality Controls Use QC materials recommended by manufacturer for
verification purposes Determine validity of other results, e.g., method
comparisons Evaluate failed runs if they occur during verification
process
Qualitative Assays: Control of Validation
68
How is it performed? Runs of specimens with analyte concentrations near
the cutoff point Three specimens, one at cutoff, one just below cutoff,
and one just above cutoff (± 20% recommended) Replicate measurements of each of three specimens
(20 each, minimum) How is it evaluated?
Determine percentage of positives and negatives for each specimen
Evaluate cutoff, as well as other two specimens
Qualitative Methods: Precision
69
How is it performed? Specimens typical of population (to be tested in future
use of method) 50 positive specimens and 50 negative specimens
recommended; minimum 20 each Performed over 10 to 20 days
How is it evaluated? Discrepant results near cutoff? Most often sensitivity and specificity used to describe
performance
Accuracy/Method Comparisons
70
Qualitative Methods
Comparative or ReferenceMethod Result
Positive Negative
Test Method Result
Positive True Positive False Positive Positive Predictive Value
Negative False Negative True NegativeNegative
PredictiveValue
Sensitivity Specificity
False Positive Rate - False Positives divided by total number of Negatives
False Negative Rate - False Negatives divided by total number of Positives
True vs. False
71
Qualitative Methods (cont'd)
Comparative or ReferenceMethod Result
Positive Negative
Test Method Result
Positive True Positive False Positive Positive Predictive Value
Negative False Negative True NegativeNegative
PredictiveValue
Sensitivity Specificity
Sensitivity = 100 x True Positives divided by (True Positives + False Negatives)
Specificity = 100 x True Negatives divided by (True Negatives + False Positives)
72
Qualitative Methods (cont'd)
Comparative or ReferenceMethod Result
Positive Negative
Test Method Result
Positive True Positive False Positive Positive Predictive Value
Negative False Negative True NegativeNegative
PredictiveValue
Sensitivity Specificity
Predictive Values - Operation of a test on a mixed population of Positive and Negatives
A property of the test and the population; and affected by prevalence of Positives
Positive Predictive Value = True Positives divided by (True Positives + False
Positives) Negative Predictive Value = True Negatives divided by
(True Negatives + False Negatives)
73
High Diagnostic Value 100% Sensitivity 100% Specificity
What happens if True Positive rate is equal to the False Positive rate?
Evaluation Criteria
74
Activity
Estimate sensitivity and specificity of a qualitative method given a data set.
75
Activity (cont’d)
Create a validation plan for a quantitative assay to be performed in your laboratory.
76
Now that you have completed this module, you should be able to:
Identify test classifications Define what each validation experiment details for
testing methods Discuss what is recommended to perform each of the
validation experiments for testing methods Recognize how to evaluate data obtained from each of
the validation experiments
In Closing
77
A rapid HIV test would likely be classified as a:
A. High complexity, modified assayB. Moderate complexity, unmodified assayC. FDA-approved, modified assayD. Waived, FDA-approved, unmodified assay
Post-Assessment Question #1
78
The precision of a test method gives information related to the method’s:
A. Systematic errorB. Comparison of results to a reference methodC. ReproducibilityD. Likelihood of being affected by hemolysis, lipemia and
icterusE. Both A and B
Post-Assessment Question #2
79
When transferring reference intervals of 20 specimens used, what is the minimum number that must fall within manufacturer’s reference intervals?
A. 20B. 18C. 16D. 15
Post-Assessment Question #3
80
Which linear regression equation component gives information regarding constant bias?
A. yB. xC. m (slope)D. b (intercept)
Post-Assessment Question #4
81
DAIDS Good Clinical Laboratory Practice (GCLP) Guidelines. www.westgard.com. Validation of Qualitative Methods. 42 CFR § 493.1253. College of American Pathologists Commission on Laboratory Accreditation,
Accreditation Checklists, April 2006. Westgard, James O. Basic Method Validation 2nd Edition. Madison, WI: Westgard
QC, Inc., 2003. Clinical and Laboratory Standards Institute. User Protocol for Evaluation of
Qualitative Test Performance; Approved Guideline. NCCLS document EP12-A. Clinical and Laboratory Standards Institute, Wayne, PA USA, 2002.
Clinical and Laboratory Standards Institute. Evaluation of Precision. Performance of Quantitative Measurement Methods. NCCLS document EP5-A2.
Clinical and Laboratory Standards Institute, Wayne, PA USA, 2004. Clinical and Laboratory Standards Institute. User verification of Performance for
Precision and Trueness. CLSI document EP15-A2. Clinical and Laboratory Standards Institute, Wayne, PA USA, 2005.
References
Wrap Up
82