Post on 22-Sep-2020
Method Validation & Verification
(Introduction to Method Validation)
Wilai Chalermchan Senior Laboratory Advisor (Contractor)
DGHT/Global HIV/AIDS Program Thailand MOPH – U.S. CDC Collaboration (TUC)
1
Contents
• Introduction Standard requirements
• Validation/Verification Parameters, Experiments and Process
• Samples of method validation/verification plan and experiments
• Summary
2
Validation and Verification (MT Std)
Validation (การสอบทวน ) หมายถงกจกรรม หรอกระบวนการ เพอพสจนวา ขนตอนการทดสอบ กระบวนการทดสอบ ระบบการทดสอบ เครองมอ หรอวธการทดสอบทน ามาใช สามารถใชงานไดตามวตถประสงคทก าหนดไว
Verification (การทวนสอบ ) หมายถงการยนยนโดยการตรวจสอบและมหลกฐานแสดงวาเปนไปตามขอก าหนด
3
Validation and Verification (ISO 15189)
VERIFICATION: confirmation, through provision of objective evidence, that specified requirements have been fulfilled การจดหาหลกฐานทยนยนวาวธทดสอบมคณสมบตเปนไปตาม..... ….ความตองการเฉพาะ
VALIDATION: confirmation, through provision of objective evidence, that the requirements for a specific intended use or application have been fulfilled การจดหาหลกฐานทยนยนวาวธทดสอบมคณสมบตเปนไปตาม...... ….ขอก าหนดตางๆในการน ามาใชงานหรอตามความมงหมายในการใชงานทก าหนด
4
Standard Requirements (MT Std.)
5.4.1.1 ....หากใชวธวเคราะหทก าหนดขนเอง ..ไมใชวธสากล หรอดดแปลง ตองมขอมลผานการสอบทวน (validation) เทยบเคยงกบวธทสากลยอมรบในขณะนน บนทกผลการสอบทวนและการทวนสอบ คมอวธปฏบตการวเคราะห • ขอมลการสอบทวน (validation) ไดจากผผลต หรอผพฒนาวธวเคราะห ขอมลตองแสดง ถงความเทยงตรง ถกตอง แมนย า สามารถระบความคลาดเคลอน ความจ าเพาะ ความไว ชวงวเคราะห และความสอดคลองกบอาการทางคลนกได
Standard Requirements (MT Std.) How….
• ขอมลการทวนสอบ (verification) ไดจากขอมลเชงประจกษเพอยนยนไดวาเหมาะสมกบการใชงานในงานบรการ
verification parameter…..? 5.4.1.2 ทบทวนวธวเคราะห อยางนอยปละ 1 ครง... บนทก 5.4.1.3 มการทบทวนคาอางอง อยางนอยปละ 1 ครง
5.4.2 คณลกษณะของแตละวธวเคราะหตองเหมาะสมสอดคลองตามวตถ ประสงคการใชงาน และตามความตองการของผใชบรการ
6
Method Evaluation
Validation
• Establishment of performance characteristic by defining and characterizing the magnitude of the analytical error present
• Performed by the developers of the method (e.g. manufacturers)
• Requires extensive studies to reveal and assess the error present
Verification • Confirmation of known
performance parameters published by the developers of the method.
• Performed by the laboratory to explore error in system
• Requires studies to confirm manufacturer's claims
7 Slide from Anna Murphy, SLAMTA program
Why method need to be validated/verified?
8
Propose of Method Validation/Verification
• Identification of Sources and Quantitation of Potential errors
• Determination if Method is Acceptable for Intended Use • Establish Proof that a Method Can be Used for Decision
Making • Satisfy Regulatory Requirements
9
Process for Establishing a Routine Test
Establishing a Laboratory Routine Test
Select, Evaluate Diagnostic Test
Select Method of Analysis
Validate/Verify Method Performance
Implement Method
Perform Test
Method Improvement
- Maintain Method
- Prevent Problem
Check with Statistical QC
Acquire Specimen
Report Results
A Routine Laboratory Testing Process
Define test SOP - Select appropriate QC -
Personnel training -
10
Why method need to be validated/verified?
11
Validation Parameters and Experiments
12
Common Validation Parameters for a Quantitative Test
• Precision • Accuracy • Linearity Analytical Measurement range Clinical Reportable Range
• Sensitivity • Limit of detection, Limit of quantification
• Specificity • Reference Interval
13
Performance Characteristic required by Thai FDA on HIV Test
• Analytical sensitivity • Diagnostic sensitivity • Diagnostic specificity • Reproducibility • Inhibition • Cross contamination • Whole system failure rate
Precision
14
Performance Characteristics must be relevant to the laboratory intend use. 5.4.2 คณลกษณะของแตละวธวเคราะหตองเหมาะสมสอดคลองตามวตถ ประสงคการใชงาน และตามความตองการของผ ใชบรการ
15
What should I do?
The Guideline
CLIA (Clinical Laboratory Improvement Amendment) Non-waived test
• Application: after April 2003 • Unmodified, FDA-approved test
1. Performance specifications comparable to the manufacturer • Accuracy • Precision • Reportable range
2. Verify manufacturer’s reference interval (normal range)
16
Performance Characteristic to consider Example:
Application
• diseases monitoring
• For diagnosis of infectious diseases
• Blood safety
• High workload
Performance Characteristic
• Reproducibility
• Specificity, False positive
• Sensitivity, LOD
• System failure rate
17
Performance Characteristic and Experiments Performance Characteristic
Experiments
Accuracy • Comparison methods to determine systemic error (SE) or bias
Precision • Repeatability
• Intermediate precision
• Reproducibility
• Replication experiment
Reportable range • Linearity experiment
Reference internal • Study sample from normal/healthy person
• Adopt from the publication or reference lab 18
Verification Process • Establish General Validation/verification procedure; • Define the application, purpose and scope of the method use; • Method selection; (from manufacturer claim) • Establish validation/verification plan/protocol;
• Define the validation/verification parameters and experiments; • State of the acceptability;
• Perform pre-validation experiments: • Adjust method parameters and/or acceptance criteria if
necessary: • Perform full validation experiments: • Record, report with statement of acceptance; • Method approval 19
Define Laboratory Application Requirements
Three types of test characteristic can be considered •Application Characteristic: Cost, type an vol of spec,
TAT, workload, space, portability, personnel •Methodology Characteristic: principle contribute to
the best performance, chemical reaction, rigor of analytical procedure
•Performance Characteristic: reportable range, precision, accuracy, detection limit 20
Sample of defined POC Performance Characteristic or specific requirements
Application
• WB from finger stick or with anti coagulant
• Ease of use for non lab-personnel
• RT° storage
• Minimal maintenance and downtime
• Cost less than..
• Size, weight, portable
Performance
• Specified reportable range
• Reproducibility less than 3%CV
Methodology
• Enzymatic reaction
• Built in calibration and traceable
• TAT of 10 mins or less
21
Considerations Prior to Method Validation/Verification
Consider Suitability of • Instrument
Status of Qualification and Calibration • Materials
Status of Reference Standards, reference material, Reagents lots
• Analyst Status of Training and Qualification Records
• Documentation approved protocol with pre-established acceptance criteria and worksheet 22
Samples of Method Verification Experiment
23
Sample 1: Method evaluation/verification at National level
AIM: Compare two new NAT methods for use in routine infectious screening……….
STUDY DESIGN AND METHODS:
The sensitivity, specificity, and robustness were determined by testing 486,676 seronegative blood donations within one year.
Samples from each day of collection were divided into two sets; the odd-numbered samples were tested individually on the TIGRIS and
the even-numbered samples were tested in pools of 6 on the cobas s 201.
The status of reactive samples was confirmed by… - duplicate testing of samples from the plasma bag to
calculate the test specificity. - tested on the alternate system and followed up.
Reference: Phikulsod et al. One-year experience of nucleic acid technology testing for human immunodeficiency virus Type 1, hepatitis C virus, and hepatitis B virus in Thai blood donations. TRANSFUSION, Volume 49, P 1126-1135 24
Evaluation Parameters
Analytes: HIV-1, HBV and HCV nucleic acid
Analytical sensitivity WHO International Standards for HCV (NIBSC 96/798), HBV (NIBSC 97/746), and HIV-1 (NIBSC 97/760). The standards were diluted to the 95% detection limit for each assay. • 3, 10, and 50 IU/mL for HBV, HCV, and HIV-1 for the Roche MPX test. • 3, 10 and 30 IU/mL for HBV, HCV, and HIV-1, for the Ultrio test IU/mL
for HBV, HCV, and HIV-1, respectively • testing individually 24 replicates
25
Cross contamination studied by including five samples containing high titers of HBV (>108 IU/mL), HCV (105 IU/mL, NIBSC 96/798), and HIV-1 (280,000 copies/mL, determined by the COBAS TaqMan HIV-1 test for use with the High Pure s ystem), which were randomly distributed in the batches of routine blood donor samples that were tested in 1 day.
Genotype detection (detectability) Genotypes of the three viruses, isolated in Thailand, were - 41 HIV-1 isolates, 12 subtype B and 29 subtype A/E (CRF01_AE); - 20 HBV isolates, 8 genotype B and 12 genotype C; and - 20 HCV isolates, 4 genotype 1a, 14 genotype 3a, and 2 genotype 3b, were tested.
26
Clinical specificity and sensitivity - The initial-reactive and repeat-reactive rates for each test were calculated. - Follow-up study of donors with reactive donations All donors with confirmed reactive donations were followed up until seroconversion or until a conclusion about the donor status could be reached. (Sensitivity) - 5000 donor samples, found nonreactive by each test, were retested using the alternate test. The testing was included during the routine testing (Specificity)
27
Whole system failure rate The overall failure rate for the two systems was monitored. Any failure that resulted in the inability to release donor results was regarded as a system failure. In addition, the human errors and the invalid controls (both run controls, internal controls, and in the case of the TIGRIS, the calibrators) were monitored.
28
Sample 2: Verification of new Chemistry Analyzer Validation Plan 1) Aim: The verification will be conducted on the X Chemistry Analyzer , serial
number …. for AST (Aspartate Amino Transferase) compare with in-service method Y during dd/mm/yy…. to dd/mm/yy ……
2) Description …….background, method, scope of use and specific requirements
3, Parameters, experiment methods and result record a. Precision: Repeatability, Intermediate precision
- Description, detail procedure, material and acceptance criteria…….
b. Accuracy/correlation ………. c. Linearity: ………. d. Sensitivity (LOD and LOQ): e. Specificity/interferance: ………. f. Reference Ranges: ……….
4. Study Conclusion and Method Approval: ………. 5. Reference: ……….
29
Sample: Experimental worksheet
30
Analyst
MethodManufacturerAnalyteUnits of MeasureSample name/Description
QC Multi 2; ; Lot# QC223; exp 30-Aug 20XXMaterial 1Material 2Material 3
Precision ExperimentShort Term (within-run)/Repeatability
Miss USA3-Nov-15Date
XYZ Cheistry AnalyzerABCASTU/L
QC Multi 1; Lot# QC123; exp 30-Aug 20XX
Run # Material 1 Material 2 Material 3 Material 1 Material 2 Material 31 35 120
2 34 119 Mean 35.0 U/L 120.0 U/L3 35 120 SD 0.8 U/L 1.5 U/L4 35 121 %CV 2.20% 1.25%5 35 120
6 36 121 Manufacturer's Claim7 35 120 Mean 36.6 U/L 128 U/L8 34 118 SD 0.3 U/L 1.0 U/L9 36 120 %CV 0.80% 0.40%
10 35 122
11 35 119 25% of CLIA Allowable error is 5%12 35 120 Short term precision is acceptable13 35 121
14 35 117
15 34 119
16 35 118
17 35 120
18 37 122
19 36 123
20 34 120
Experiment Results
(acceptable/unacceptable)
Preliminary Estimate of Precision, Short Term
Laboratory
What’s next?
31
On-going Verification
• Test performed by multiple systems required comparison of test results (Comparability study)
• Monitoring of test precision through QC • Monitoring of accuracy PT/ EQA results
No PT program- at least semi annual re-validate/verified
• Kit lot verification
32
Re-validation
When
• Instrument repaired, relocate
• Method performance have been significant
changed
• Scope of application has been change
33
What are the evidence that an assessor looking for?
Minimum • General method validation/verification procedure • Validation/Verification plan/protocol • Records • On-going verification measure(s) and records
34
Summary Method verification • Procedure, plan and records; • Meet specific intend use and extensive; • before conducting the study, qualify instrument and
ensure personnel competency; • observe errors that may have in our laboratory
system; • First step in establish quality control procedure • on-going verification, re-validation
35
References
36
Questions & Comments
37