Assay Quality Considerations - APHL · 2016. 5. 17. · • EP9-A2: Method Comparison and Bias...
Transcript of Assay Quality Considerations - APHL · 2016. 5. 17. · • EP9-A2: Method Comparison and Bias...
![Page 1: Assay Quality Considerations - APHL · 2016. 5. 17. · • EP9-A2: Method Comparison and Bias Estimation Using Patient Samples • EP10-A2: Preliminary Evaluation of Quantitative](https://reader035.fdocuments.in/reader035/viewer/2022071505/612571308eb4a3086b4f6459/html5/thumbnails/1.jpg)
Christopher N. Greene, PhD Newborn Screening and Molecular Biology Branch,
Division of Laboratory Sciences NCEH, CDC
Tuesday, 9th July 2013
Assay Quality Considerations
National Center for Environmental Health U.S. Centers for Disease Control and Prevention
![Page 2: Assay Quality Considerations - APHL · 2016. 5. 17. · • EP9-A2: Method Comparison and Bias Estimation Using Patient Samples • EP10-A2: Preliminary Evaluation of Quantitative](https://reader035.fdocuments.in/reader035/viewer/2022071505/612571308eb4a3086b4f6459/html5/thumbnails/2.jpg)
Molecular Testing
Testing Parameters Examples Target Analyte Genetic Mutations, Gene Dosage Platform PCR, qPCR, bead arrays, Fluorescent Probe, DNA
Sequencers Complexity Single- or multi-plex of single nucleotides,
Laboratory Developed Test, ASRs, FDA-approved, Gene Sequencing, Next-Gen Sequencing
Multiple molecular assays have been developed for screening procedures. These assays vary on their complexity, the target analyte, as well as the platform used for analysis.
![Page 3: Assay Quality Considerations - APHL · 2016. 5. 17. · • EP9-A2: Method Comparison and Bias Estimation Using Patient Samples • EP10-A2: Preliminary Evaluation of Quantitative](https://reader035.fdocuments.in/reader035/viewer/2022071505/612571308eb4a3086b4f6459/html5/thumbnails/3.jpg)
Molecular Testing
When Adopting a New Test Required Evidence Test Development - LDT Accuracy, precision, reproducibility,
sensitivity, specificity, robustness Test Validation - ASR Clinical and analytic performance and
test limitations Test Verification – FDA approved Identify samples or reference
materials of known quantitative or qualitative values to verify test performance
Molecular tests require verification or validation of assay results. Level of evidence required is related to the type of assay, i.e. laboratory developed test, analyte specific reagents or FDA approved assays.
![Page 4: Assay Quality Considerations - APHL · 2016. 5. 17. · • EP9-A2: Method Comparison and Bias Estimation Using Patient Samples • EP10-A2: Preliminary Evaluation of Quantitative](https://reader035.fdocuments.in/reader035/viewer/2022071505/612571308eb4a3086b4f6459/html5/thumbnails/4.jpg)
Laboratory Regulatory and Accreditation Guidelines
US Food and Drug Administration (FDA): approves kits and reagents for use in clinical testing
Clinical Laboratory Improvement Amendments (CLIA): Regulations passed by Congress1988 to establish quality
standards for all laboratory testing to ensure the accuracy, reliability and timeliness of patient test results regardless of where the test was performed
College of American Pathologists (CAP): Molecular Pathology checklist
State Specific Regulations
![Page 5: Assay Quality Considerations - APHL · 2016. 5. 17. · • EP9-A2: Method Comparison and Bias Estimation Using Patient Samples • EP10-A2: Preliminary Evaluation of Quantitative](https://reader035.fdocuments.in/reader035/viewer/2022071505/612571308eb4a3086b4f6459/html5/thumbnails/5.jpg)
Testing Parameters
Accuracy
Sensitivity
Specificity
Precision
Limit of Detection
Range
Reproducibility
Robustness
![Page 6: Assay Quality Considerations - APHL · 2016. 5. 17. · • EP9-A2: Method Comparison and Bias Estimation Using Patient Samples • EP10-A2: Preliminary Evaluation of Quantitative](https://reader035.fdocuments.in/reader035/viewer/2022071505/612571308eb4a3086b4f6459/html5/thumbnails/6.jpg)
Definitions
• Accuracy: Agreement between test result and the “true” result – Comparison of results between new method and a
reference method – Results of new method on certified reference materials
• Sensitivity: Ability to obtain positive results in
concordance with a positive result of a reference method – can the test detect a positive result
![Page 7: Assay Quality Considerations - APHL · 2016. 5. 17. · • EP9-A2: Method Comparison and Bias Estimation Using Patient Samples • EP10-A2: Preliminary Evaluation of Quantitative](https://reader035.fdocuments.in/reader035/viewer/2022071505/612571308eb4a3086b4f6459/html5/thumbnails/7.jpg)
Definitions
• Specificity: Ability to obtain a negative result for a qualitative test in concordance with reference method
• Precision: Measure of random error in quantitative tests, the closeness of agreement usually expressed as standard deviation of CV
![Page 8: Assay Quality Considerations - APHL · 2016. 5. 17. · • EP9-A2: Method Comparison and Bias Estimation Using Patient Samples • EP10-A2: Preliminary Evaluation of Quantitative](https://reader035.fdocuments.in/reader035/viewer/2022071505/612571308eb4a3086b4f6459/html5/thumbnails/8.jpg)
Definitions
• Limit of Detection: Lowest amount of analyte that is distinguishable from background or a negative control
• Range: Range of test values expected for the test population. For quantitative, high and low range of target, for qualitative, possible detectable alleles of the assay
![Page 9: Assay Quality Considerations - APHL · 2016. 5. 17. · • EP9-A2: Method Comparison and Bias Estimation Using Patient Samples • EP10-A2: Preliminary Evaluation of Quantitative](https://reader035.fdocuments.in/reader035/viewer/2022071505/612571308eb4a3086b4f6459/html5/thumbnails/9.jpg)
Definitions
• Reproducibility: Equivalent to precision for qualitative and semi-quantitative tests. Can be used to indicate with-in or between run agreement of test results
• Robustness: Test precision, given small, deliberate changes in test conditions (e.g. incubation times, temperature variation)
![Page 10: Assay Quality Considerations - APHL · 2016. 5. 17. · • EP9-A2: Method Comparison and Bias Estimation Using Patient Samples • EP10-A2: Preliminary Evaluation of Quantitative](https://reader035.fdocuments.in/reader035/viewer/2022071505/612571308eb4a3086b4f6459/html5/thumbnails/10.jpg)
Pre-Validation
Choose and evaluate assay methodology
Determining analytic performance of an assay involves:
Reviewing professional guidelines and relevant literature
Stringent design/analysis of primers and probes
Quality and quantity of extracted nucleic acid
Appropriate platform for the test
Availability of controls or calibrators
Optimization of amplification and detection
![Page 11: Assay Quality Considerations - APHL · 2016. 5. 17. · • EP9-A2: Method Comparison and Bias Estimation Using Patient Samples • EP10-A2: Preliminary Evaluation of Quantitative](https://reader035.fdocuments.in/reader035/viewer/2022071505/612571308eb4a3086b4f6459/html5/thumbnails/11.jpg)
Verifying Assay Specifications
• Demonstrate that laboratory can obtain specifications comparable to manufacturer – Accuracy – Precision – Reportable Results
• Applies to unmodified FDA-approved tests
• Verify appropriate test results for population
• May require matrix-appropriate materials during verification
![Page 12: Assay Quality Considerations - APHL · 2016. 5. 17. · • EP9-A2: Method Comparison and Bias Estimation Using Patient Samples • EP10-A2: Preliminary Evaluation of Quantitative](https://reader035.fdocuments.in/reader035/viewer/2022071505/612571308eb4a3086b4f6459/html5/thumbnails/12.jpg)
Assay Validation
• Performance characteristics of laboratory developed tests
• Reagents that are the components of a test that is created without regulatory approval – ASR
• Use of FDA-approved tests that have been modified for the testing procedure
Establish that the assay fulfills its intended use with the appropriate sample type and population
![Page 13: Assay Quality Considerations - APHL · 2016. 5. 17. · • EP9-A2: Method Comparison and Bias Estimation Using Patient Samples • EP10-A2: Preliminary Evaluation of Quantitative](https://reader035.fdocuments.in/reader035/viewer/2022071505/612571308eb4a3086b4f6459/html5/thumbnails/13.jpg)
Modified FDA Test
• Alteration of any assay step or analyte for the test that may affect performance
• Changes could include: – Type or concentration of specimen – Storage specifications – Extraction method – Amplification parameters or platforms – Use of test for other purposes
![Page 14: Assay Quality Considerations - APHL · 2016. 5. 17. · • EP9-A2: Method Comparison and Bias Estimation Using Patient Samples • EP10-A2: Preliminary Evaluation of Quantitative](https://reader035.fdocuments.in/reader035/viewer/2022071505/612571308eb4a3086b4f6459/html5/thumbnails/14.jpg)
Assay Validation
Required for: New testing methodology Assay modification – includes cross-checks for different
makes/models of instrumentation
Applies to:
Modified FDA assays Laboratory Developed methods
![Page 15: Assay Quality Considerations - APHL · 2016. 5. 17. · • EP9-A2: Method Comparison and Bias Estimation Using Patient Samples • EP10-A2: Preliminary Evaluation of Quantitative](https://reader035.fdocuments.in/reader035/viewer/2022071505/612571308eb4a3086b4f6459/html5/thumbnails/15.jpg)
Controls for Each Run
Appropriate positive and negative controls should be included for each run of specimens being tested
![Page 16: Assay Quality Considerations - APHL · 2016. 5. 17. · • EP9-A2: Method Comparison and Bias Estimation Using Patient Samples • EP10-A2: Preliminary Evaluation of Quantitative](https://reader035.fdocuments.in/reader035/viewer/2022071505/612571308eb4a3086b4f6459/html5/thumbnails/16.jpg)
Molecular Assay Controls
Positive controls: Inhibitors Component failure Interpretation of results Sources:
Residual positive DBS PT samples QC materials through purchase or exchange
Negative controls: Nucleic acid contamination
![Page 17: Assay Quality Considerations - APHL · 2016. 5. 17. · • EP9-A2: Method Comparison and Bias Estimation Using Patient Samples • EP10-A2: Preliminary Evaluation of Quantitative](https://reader035.fdocuments.in/reader035/viewer/2022071505/612571308eb4a3086b4f6459/html5/thumbnails/17.jpg)
Common Molecular Assay Problems and Trouble Shooting
Temperature errors
Template/Sequence
PCR inhibitors
Buffers
Bad dNTPs
Bad primers
Bad enzyme
![Page 18: Assay Quality Considerations - APHL · 2016. 5. 17. · • EP9-A2: Method Comparison and Bias Estimation Using Patient Samples • EP10-A2: Preliminary Evaluation of Quantitative](https://reader035.fdocuments.in/reader035/viewer/2022071505/612571308eb4a3086b4f6459/html5/thumbnails/18.jpg)
Critical Molecular Assay Components
Nucleic Acids: Prepare aliquots appropriate to workflow to limit freeze-thaw cycles Primers and probes dNTPs Genomic DNA
4-8°C: Up to one year -20°C: Up to seven years
Enzymes Benchtop coolers recommended
Fluorescent reporters Limit exposure to light Amber storage tubes or wrap in shielding (foil)
![Page 19: Assay Quality Considerations - APHL · 2016. 5. 17. · • EP9-A2: Method Comparison and Bias Estimation Using Patient Samples • EP10-A2: Preliminary Evaluation of Quantitative](https://reader035.fdocuments.in/reader035/viewer/2022071505/612571308eb4a3086b4f6459/html5/thumbnails/19.jpg)
Positive Controls
Ideally should represent each target allele used in each run
May not be feasible when:
Highly multiplex genotypes possible
Systematic rotation of different alleles as positives
Rare alleles
Heterozygous or compound heterozygous specimens
![Page 20: Assay Quality Considerations - APHL · 2016. 5. 17. · • EP9-A2: Method Comparison and Bias Estimation Using Patient Samples • EP10-A2: Preliminary Evaluation of Quantitative](https://reader035.fdocuments.in/reader035/viewer/2022071505/612571308eb4a3086b4f6459/html5/thumbnails/20.jpg)
Positive Controls
Assays based on presence or absence of product Internal positive amplification controls to distinguish true
negative from false due to failure of DNA extraction or PCR amplification
PCR amplification product of varying length Specimens representing short and long amplification products to
control for differential amplification
Quantitative PCR Controls should represent more than one concentration
Control copy levels should be set to analytic cut-offs
![Page 21: Assay Quality Considerations - APHL · 2016. 5. 17. · • EP9-A2: Method Comparison and Bias Estimation Using Patient Samples • EP10-A2: Preliminary Evaluation of Quantitative](https://reader035.fdocuments.in/reader035/viewer/2022071505/612571308eb4a3086b4f6459/html5/thumbnails/21.jpg)
Quantitative PCR Controls
• Contamination controls • Calibration control with reliable low limit of
quantification • Measures for consistency, especially for
DBS • Measure of DNA quality – internal control
![Page 22: Assay Quality Considerations - APHL · 2016. 5. 17. · • EP9-A2: Method Comparison and Bias Estimation Using Patient Samples • EP10-A2: Preliminary Evaluation of Quantitative](https://reader035.fdocuments.in/reader035/viewer/2022071505/612571308eb4a3086b4f6459/html5/thumbnails/22.jpg)
False Negative: ADO
Allele drop-out (ADO): the failure of a molecular test to amplify or detect one or more alleles
Potential causes: DNA template concentration
• Incomplete cell lysis • DNA degradation
Non-optimized assay conditions Unknown polymorphisms in target sites Reagent component failure
Major concern for screening laboratories Confirmation of mutation inheritance in families is not an option
![Page 23: Assay Quality Considerations - APHL · 2016. 5. 17. · • EP9-A2: Method Comparison and Bias Estimation Using Patient Samples • EP10-A2: Preliminary Evaluation of Quantitative](https://reader035.fdocuments.in/reader035/viewer/2022071505/612571308eb4a3086b4f6459/html5/thumbnails/23.jpg)
DNA Degradation
Lane 1 + 7: 1kb size standard ladder
Lane 2: 100ng control genomic DNA
Lanes 3-5: Crude cell lysates
![Page 24: Assay Quality Considerations - APHL · 2016. 5. 17. · • EP9-A2: Method Comparison and Bias Estimation Using Patient Samples • EP10-A2: Preliminary Evaluation of Quantitative](https://reader035.fdocuments.in/reader035/viewer/2022071505/612571308eb4a3086b4f6459/html5/thumbnails/24.jpg)
Question?
How can you control for presence of
sufficient amount/quality of DNA for a
PCR based test in a NBS lab?
![Page 25: Assay Quality Considerations - APHL · 2016. 5. 17. · • EP9-A2: Method Comparison and Bias Estimation Using Patient Samples • EP10-A2: Preliminary Evaluation of Quantitative](https://reader035.fdocuments.in/reader035/viewer/2022071505/612571308eb4a3086b4f6459/html5/thumbnails/25.jpg)
PCR Amplification Controls
• Allele-specific amplification • Are there problems with this assay? • What additional controls would be useful?
Allele 1 + 2 Allele 2 Allele 1 Reference Negative
![Page 26: Assay Quality Considerations - APHL · 2016. 5. 17. · • EP9-A2: Method Comparison and Bias Estimation Using Patient Samples • EP10-A2: Preliminary Evaluation of Quantitative](https://reader035.fdocuments.in/reader035/viewer/2022071505/612571308eb4a3086b4f6459/html5/thumbnails/26.jpg)
PCR with Internal Controls
Tetra-primer ARMS-PCR Simultaneous amplification of:
Positive amplification control
Mutation allele
Reference allele
Alternative to tetra-primer ARMS is to
include an additional primer set to amplify a different control sequence
![Page 27: Assay Quality Considerations - APHL · 2016. 5. 17. · • EP9-A2: Method Comparison and Bias Estimation Using Patient Samples • EP10-A2: Preliminary Evaluation of Quantitative](https://reader035.fdocuments.in/reader035/viewer/2022071505/612571308eb4a3086b4f6459/html5/thumbnails/27.jpg)
Allele Drop-out in PCR Testing
ggacatgatGcactacatgctccaaggtagtggag Cgtgatgtacgaggttccat
5’
ggacatgatGcactacatgctccaaggtagtggag cctgtactaCgtgatgtacgaggttccat
5’
![Page 28: Assay Quality Considerations - APHL · 2016. 5. 17. · • EP9-A2: Method Comparison and Bias Estimation Using Patient Samples • EP10-A2: Preliminary Evaluation of Quantitative](https://reader035.fdocuments.in/reader035/viewer/2022071505/612571308eb4a3086b4f6459/html5/thumbnails/28.jpg)
Allele Drop-out in PCR Testing
SNP in primer site
ggacatgatGcGctacatgctccaaggtagtggag gatgtacgaggttccat
5’
ggacatgatGcGctacatgctccaaggtagtggag
Cgtgatgtacgaggttccat 5’
![Page 29: Assay Quality Considerations - APHL · 2016. 5. 17. · • EP9-A2: Method Comparison and Bias Estimation Using Patient Samples • EP10-A2: Preliminary Evaluation of Quantitative](https://reader035.fdocuments.in/reader035/viewer/2022071505/612571308eb4a3086b4f6459/html5/thumbnails/29.jpg)
False Negatives: Deletions
A
G
Reverse Primer
Forward Primer
Reverse Primer
Forward Primer
![Page 30: Assay Quality Considerations - APHL · 2016. 5. 17. · • EP9-A2: Method Comparison and Bias Estimation Using Patient Samples • EP10-A2: Preliminary Evaluation of Quantitative](https://reader035.fdocuments.in/reader035/viewer/2022071505/612571308eb4a3086b4f6459/html5/thumbnails/30.jpg)
False Negatives: Deletions
A
G
Reverse Primer
Forward Primer
Reverse Primer
Forward Primer Deletion
![Page 31: Assay Quality Considerations - APHL · 2016. 5. 17. · • EP9-A2: Method Comparison and Bias Estimation Using Patient Samples • EP10-A2: Preliminary Evaluation of Quantitative](https://reader035.fdocuments.in/reader035/viewer/2022071505/612571308eb4a3086b4f6459/html5/thumbnails/31.jpg)
False Positives
Potential causes:
Non-optimized assay conditions
Unknown polymorphisms in target sites
Gene duplications
Oligonucleotide mis-priming at related sequences
Psuedogenes or gene families
Oligonucleotide concentrations too high
Nucleic acid cross-contamination
![Page 32: Assay Quality Considerations - APHL · 2016. 5. 17. · • EP9-A2: Method Comparison and Bias Estimation Using Patient Samples • EP10-A2: Preliminary Evaluation of Quantitative](https://reader035.fdocuments.in/reader035/viewer/2022071505/612571308eb4a3086b4f6459/html5/thumbnails/32.jpg)
How Many Samples?
• New test or to replace an old test? • How is the test to be used? • What performance criteria are most important? • How available are appropriate samples?
• For screening: daily load vs. weekly load vs.
monthly load – it depends
It depends…
![Page 33: Assay Quality Considerations - APHL · 2016. 5. 17. · • EP9-A2: Method Comparison and Bias Estimation Using Patient Samples • EP10-A2: Preliminary Evaluation of Quantitative](https://reader035.fdocuments.in/reader035/viewer/2022071505/612571308eb4a3086b4f6459/html5/thumbnails/33.jpg)
CLSI Evaluation Protocols recognized by FDA as Consensus Standards
• EP5-A2: Evaluation of Precision Performance of
Quantitative Measurement Methods • EP9-A2: Method Comparison and Bias Estimation Using
Patient Samples • EP10-A2: Preliminary Evaluation of Quantitative Clinical
Methods • EP12-A User Protocol for Evaluation of Qualitative Test
Performance • EP15-A2: User Demonstration of Performance for
Precision and Accuracy
![Page 34: Assay Quality Considerations - APHL · 2016. 5. 17. · • EP9-A2: Method Comparison and Bias Estimation Using Patient Samples • EP10-A2: Preliminary Evaluation of Quantitative](https://reader035.fdocuments.in/reader035/viewer/2022071505/612571308eb4a3086b4f6459/html5/thumbnails/34.jpg)
Molecular Assay Proficiency Testing Material Sources
CDC NSQAP
UKNEQS
EuroGentest
CAP
Maine Molecular
SeraCare
Corielle
ECACC
In-house samples
Round-robin with other NBS laboratories
![Page 35: Assay Quality Considerations - APHL · 2016. 5. 17. · • EP9-A2: Method Comparison and Bias Estimation Using Patient Samples • EP10-A2: Preliminary Evaluation of Quantitative](https://reader035.fdocuments.in/reader035/viewer/2022071505/612571308eb4a3086b4f6459/html5/thumbnails/35.jpg)
Professional Guidelines
American College of Medical Genetics (ACMG) Standards and Guidelines for Clinical Genetics Laboratories
Clinical and Laboratory Standards Institute (CLSI) MM01-A2: Molecular Diagnostic Methods for Genetic Diseases MM13-A: Collection, Transport, Preparation, and Storage of
Specimens for Molecular Methods MM14-A: Proficiency Testing (External Quality Assessment) for
Molecular Methods MM17-A: Verification and Validation of Multiplex Nucleic Acid
Assays MM19-P: Establishing Molecular Testing in Clinical Laboratory
Environments