2013 CAP Accreditation Programs Audioconferences and...

2013 CAP Accreditation Programs Audioconferences and WebinarsCAP Accreditation Requirements for Validation of Laboratory Tests

Stephen J. Sarewitz, MD, FCAP

January 16, 2013

www.cap.org v. 1.0

Obj tiObjectives

• Perform appropriate validation/verification studies in accordance with CAP requirements prior to implementing new tests, including FDA cleared/approved tests and laboratory-developed tests.

• Understand validation requirements for FDA cleared/approved tests modified by the laboratory.

• Appropriately address validation of tests on unusual matrices.

© 2013 College of American Pathologists. All rights reserved. 2

T iTopics

What this webinar covers:

• Validation/verification of the test method itself

What it doesn’t cover: other aspects of implementing new tests, such as:

• Calibration or QC validation

• Documentation of test procedure

• Training of personnel and competency assessment• Training of personnel and competency assessment

• Documenting/modifying environmental conditions

• Proficiency testing or alternative assessmenty g

• Sample handling

• Results reporting


Definitions Used in CAP Accreditation Programsg

• Validation: provision of objective evidence through a defined process that a test performs as intended [CLSI]

• Verification: an abbreviated process to demonstrate that a test • Verification: an abbreviated process to demonstrate that a test performs in substantial compliance to previously established claims

• Sometimes the term “validation” is used to cover both of the above—can be source of confusion


T f V lid tiTypes of Validation

• Analytic validation: “…the process of assessing the assay and its performance characteristics and determining the optimal conditions that will generate a reliable, reproducible, and accurate…assay for the intended application.” --Required by CLIA and CAP Accreditation Checklists Accreditation Checklists

• Clinical validation: determining “the ability of a test to diagnose or predict risk for a particular health condition, measured by clinical (or diagnostic) sensitivity…, specificity, and predictive values.” --Not diagnostic) sensitivity…, specificity, and predictive values. Not addressed by CLIA; in CAP only in Molecular Pathology and Microbiology (in Molecular section) Checklists (for nonwaived tests).

• Clinical utility: “determining the net balance of health-related risks Clinical utility: determining the net balance of health related risks and benefits associated with the test’s use in practice [to] patient, family, healthcare organizations, and society.” --Not addressed by CLIA or CAP. Subjective, complex, involves areas outside of l b t di ilaboratory diagnosis.


P lli Q ti # 1Polling Question # 1

Which of the following is required by CAP and CLIA on a test method Which of the following is required by CAP and CLIA on a test method prior to performing and reporting patient results?


A l ti V lid tiAnalytic Validation

• Required by CLIA and CAP for non-waived tests (No CLIA reqs. for waived tests; CAP has very limited req. for waived tests)

• CLIA and CAP requirements similar

• Distinction between FDA-cleared/approved tests and others:• Distinction between FDA-cleared/approved tests, and others:o For tests cleared or approved by FDA—verification required: i.e.,

laboratory must show that it can obtain performance specifications comparable to those established by the manufacturer

o Requirements the same for both moderate and high complexity tests approved/cleared by FDA

o For tests not approved/cleared by FDA: laboratory must establisho For tests not approved/cleared by FDA: laboratory must establishthe performance specifications (= validating the test)


CAP D fi iti f LDTCAP Definition of LDT

A laboratory-developed test (LDT) is defined as follows - A test used in patient management that has all of the following characteristics:patient management that has all of the following characteristics:

• The test is performed by the clinical laboratory in which the test was developed

• The test is neither FDA-cleared nor FDA-approved, or is an FDA-cleared/approved test modified by the laboratory (sample types or the use of collection devices not listed in manufacturer instructions constitute modifications, by this definition)constitute modifications, by this definition)

• The test was first used for clinical testing after April 23, 2003


A l ti V lid ti ti dAnalytic Validation, continued

• For FDA cleared/approved tests that have been modified by the o c ea ed/app o ed es s a a e bee od ed by e laboratory, lab must validate the modification(s)o Sample types and collection devices not listed in mfg labeling are

considered modifications of the test (ex. body fluids)i t t i ti t l ti l d d i f l b li i o Using a test in a patient population excluded in mfg labeling is a

modification (ex. glucometer in critically ill patients)

• By definition, all modified FDA cleared/approved tests and LDT’s are hi h l it ( l ifi ll t i d b FDA/CLIA high complexity (unless specifically categorized by FDA/CLIA as waived or PPM)


CAP P f S ifi ti f A l ti V lid tiCAP Performance Specifications for Analytic Validation

• Analytic accuracy

• Precision

• Analytic sensitivity (limit of detection, LOD)*

• Analytic specificity (interferences)

• Reportable range

• Reference range• Reference range

• In COM checklist (moved from GEN for 2011 edition)

*Literally means ability of assay to detect a change in concentration of the e a y ea s ab y o assay o de ec a c a ge co ce a o o e analyte; assays with steeper calibration curves are more sensitive in this sense. However this is not the meaning of the term for purposes of validation/verification—LOD is what is meant.


CAP R i t i COM h kli t (i GEN b f 2011)CAP Requirements…in COM checklist (in GEN before 2011)

FDA approved/cleared LDTs & modified FDA tests

Accuracy & Precision (COM.40300, ph II)

Verify Establish (= validation)

Analytic sensitivity (LOD) (COM.40400, ph II)

Verify; literature or manufacturer d t ti OK

Establish

documentation OKAnalytic specificity (interferences) (COM.40500, ph II)

Verify; literature or manufacturer documentation OK

Establish; studies by manufacturer or in literature OK

Reportable range Verify* Establish*Reportable range (AMR)(COM.40600, ph II)

Verify Establish

Reference range (COM.50000, ph II)

Verify ** Establish**

*Reportable range ( = AMR, generally) is the range of values that the method can directly measure without dilution or concentration, while meeting specifications for accuracy & precision

--Details on establishing & validating AMR are in other checklists (ex. CHM, HEM, MOL)

**In some cases labs may use manufacturer or literature data when verification/establishment of


In some cases labs may use manufacturer or literature data when verification/establishment of a reference range is not practical: ex. pediatric blood cell count / index parameters; therapeutic drug levels.

© 2013 College of American Pathologists. All rights reserved.


CAP requires the analytic validation of …..


W i d T t Waived Tests

• No CLIA requirements

• CAP: Verify reference range if practical

POC.04525 (ph II): Reference intervals…are established or verifiedverified….

NOTE: If a…reference study is not…practical….then the POCT site should carefully evaluate the yuse of published data for its own reference ranges…


V lid ti / V ifi ti S ifiValidation / Verification Specifics

• Neither CLIA nor CAP list specifics of validation/verification—numbers of samples, limits of acceptability, number of days required for studies, statistical analysiso Variability across laboratories in terms of environment, patient

populations use of tests – specifics thus need to be determined populations, use of tests specifics thus need to be determined by laboratory director

o Validation = more extensive study than verification

• Exceptions: • Exceptions: o Number of samples—HER2, estrogen receptor, progesterone

receptor tests in breast carcinoma tissueo Concordance acceptability threshold ER/PgR receptor in breast o Concordance acceptability threshold – ER/PgR receptor in breast

carcinomao Number of data points for AMR (3 required)

No requirement for linearity studies per se but AMR is - No requirement for linearity studies per se….but AMR is “linearity lite”


V lid ti /V ifi ti G id liValidation/Verification Guidelines

FDA cleared/approved LDT

A (bi ) 20 40 l At l t 40 l Accuracy (bias) 20-40 samples across AMR

At least 40 samples across AMR; could be > 100

Precision (random error) 2-3 samples at clinical decision points run d il f 5 d

Run study for 20 days

daily for 5 daysReportable range (AMR)* 3 points near low end,

midpoint, and high endSame

Reference range 20 samples 40-60 samples; 120 or g p p ;more ideal

[See CLSI Guidelines (references at end of presentation)]

**CAP checklist requirement [ex. CHM.13600 (II0]

© 2013 College of American Pathologists. All rights reserved. 15© 2013 College of American Pathologists. All rights reserved.


When performing a method validation, what is the required number of specimens for determining accuracy of a FDA cleared/approved

th d?method?


V lid ti G id li tValidation Guidelines, cont.

• Precision (actually, imprecision is what is measured) can be determined within a run (repeatability); across several runs in one day; or across multiple runs across multiple days (repeatability); across several runs in one day; or across multiple runs across multiple days (reproducibility). The latter is most reflective of actual lab practice.

• For AMR, 5 points is closer to ideal but 3 is sufficient (CHM.13600, ph II)

• Reference range classically = central 95% of values for “normal” population For some • Reference range classically = central 95% of values for normal population. For some tests, reference range instead = a cut-off point or decision limit: ex., troponin, cholesterol. Studies for these tests would not be samples from a reference population, but rather studies of samples on either side of the decision point, using clinical data or a comparative method.

• If the laboratory has a valid reference range for an analyte and changes methods, the lab may be able to use method comparison data to establish the reference range for the new method (= “transference by calculation”). (Westgard; CLSI guideline C28-A3).

o Limit transference to one change of methods, to prevent potentially amplifying errors g p p y p y gby multiple transfer procedures (Westgard).

o Recommend verifying transference result by small verification study.


AMR i C l ti N R i t 2012AMR in Coagulation—New Requirements, 2012

• AMR = range of analyte that method can directly measure without dilution or concentration (i e range over which signal from instrument can be reliably related to concentration (i.e., range over which signal from instrument can be reliably related to concentration of analyte)

• Checklist items HEM.38009, 38010, 38011 (II)

• Applies only to tests that directly measure an analyte not to tests that assess analyte • Applies only to tests that directly measure an analyte, not to tests that assess analyte levels through formation of a clot by the coagulation system, nor to platelet function studies

o Ex. immunoassays (EIA, immunoturbidity, chromogenic)− Ex., protein C ag, free & total protein S ag, von Willebrand factor ag, LWM p g p g g

heparin, quant. d-dimero May include tests reported in % if traceable to a standardo Whether or not test involves a reference/standard curve not relevanto Does not apply to analytes measured by functional assay with results convertible to pp y y y y

concentration by a formula

• Directly measured analytes also subject to calibration requirements

• Extensive discussion in the HEM checklist


D fi i All bl T t l E & i iDefining Allowable Total Error…accuracy & precision

• FDA cleared/approved tests: = manufacturer specifications• For LDT’s:

o Decision by Laboratory Directoro Ideally, laboratory defines allowable total error in advance for

th i t d d li ti f th t tthe intended application of the test

• Possible ways to establish allowable total error include:CLIA t bilit li it f i bilit i fi i t ti lt o CLIA acceptability limits for variability in proficiency testing results (403.909 – 403.959)

o Range of actual PT results for the analyte/methodClinical decision points literat re inp t from clinicianso Clinical decision points—literature, input from clinicians

• Observed total error should be less than allowable total error


Allowable Error vs. Observed Error: Using Statistics to Help A l A & P i i ( t ti & d )Analyze Accuracy & Precision (systematic & random error)

• Westgard: “Statistics are just tools for combining many experimental results…and summarizing all that data in just a few numbers…the statistics are used to make reliable estimates of the errors from the data….”

• Imprecision (= random error): calculate standard deviation & coefficient of variation

• Bias or inaccuracy (systematic error): o Average difference between paired samples (for tests covering

narrow range, ex., electrolytes) – t-test can be used to determine if a bias is truly present

o Linear regression statistics: Equation for line of best fit through o Linear regression statistics: Equation for line of best fit through graphed method comparison data (for tests with wide range, ex., cholesterol)

Observed (actual) total error can be defined as: bias + [3 x (sd of Observed (actual) total error can be defined as: bias + [3 x (sd of method)] (Westgard)


Example – New method for serum aspartate aminotransferase (AST)(AST)

• Total allowable error:o Lab determines it should be <= CLIA proficiency testing limit for AST, which

is +/- 20% of target value

• Lab’s new AST method: Bi i + 2 U/L d t f th do Bias is + 2 U/L compared to reference method

o Imprecision = standard deviation = +/- 3 U/L

• Consider target value of 50 U/L:T t l ll bl +/ 20% 50 10 U/Lo Total allowable error = +/- 20% x 50 = 10 U/L

o Observed total error = bias + (3 x SD) = 2 U/L + (3 x 3 U/L) = 11 U/L

• Observed total error of 11 exceeds total allowable error of 10 so test method is not acceptablenot acceptable

(Note: in this calculation, the absolute value of bias and SD are used)



Which statement(s) is (are) true regarding the total allowable error for a nonwaived quantitative test?


Oth R i t i COMOther Requirements in COM

1. Lab director or designee qualified as director must approve validation studies (COM 40000 h I)(COM.40000, ph I).

2. Lab must list all LDT’s implemented in past 2 years for review by inspector (COM.40200, ph I).

3. Lab’s test methods, including performance specifications and supporting validation/verification data, must be available to clients and inspection team (COM.40700, ph II). Lab may require clients to treat information as confidential.

4. If method changes such that results may be significantly altered, the change must be explained to clients (COM.40800, ph II).

5. Lab evaluates reference range and takes corrective action as appropriate 5. ab e a ua es e e e ce a ge a d a es co ec e ac o as app op a e (COM.50100, ph II).

a. Change of method or patient population

6. GEN .20377 (II) Retain validation records while method is in use and at least 2 years afterwards.


Validation items in Other Checklists – Examples (full list at end of t ti )presentation)

• MOL.30900 (II) Validation studies include samples for each reportable tgenotype

o NOTE: Assays for genetic disorders with a limited number of possible genotypes (e.g., hereditary hemochromatosis): confirm ability of assay to detect these genotypes. Assays for genetic disorders with considerable

i i i ifi f i i (allelic heterogeneity and/or significant numbers of private mutations (e.g., cystic fibrosis or hereditary nonpolyposis colorectal cancer) should confirm the accuracy of the methodology used to ensure that the assay will detect a high percentage of possible genotypes. However, it will not b ibl t d t th t h d t t ibl be possible to document that such assays can detect every possible genotype.

• MOL.31015 (II) Validation studies include expected specimen types (frozen, fresh paraffin-embedded blood etc )fresh, paraffin-embedded, blood, etc.)


C li ti Q lit ti T tComplications – Qualitative Tests

• Performance specifications must be established/verified as applicable. Ex. reportable range may not be applicable. For tests with a cut-off, precision is considered to be variation in the test result near the cut-off point.

• Can use kappa statistic to determine the degree to which agreement between 2 comparative methods is result of chance:

kappa = [P(o) – P(e)]/[1-P(e)] where P(e) = agreement by chance and P(o) = observed agreement

P(e) = [%test1(pos) x %test2(pos)] + [%test1(neg) x %test2(neg)]

kappa values: 0 = chance agreement; 1 = perfect agreement

41 60 = moderate agreement.41 - .60 = moderate agreement

.61 - .80 = substantial agreement

See Viera AJ, Garrett JM. Fam Med 2005;37(5):360-363., ; ( )


C li ti “G ld St d d” P blComplications – “Gold Standard” Problem

• New method is considered superior in analytic sensitivity to old established method—how to handle discordant results in method established method how to handle discordant results in method comparison study.o Troponin replacing CK-MB in dx of myocardial infarctiono Molecular tests for microorganisms replacing culture or

immunologic tests—Chlamydia, Trichomonas

• Approaches:o Use clinical data (chart review)( )

- Ex. clinical & EKG evidence of MIo Use reference method if available—must test all samples, not just

discordant ones


Complications – Body Fluids( COM I t d ti t P f S ifi ti ti )(see COM, Introduction to Performance Specifications section)

• General principle: not always possible to establish performance specs in fluids

• General principle: if some or all specs not established, add note to report (ex. “The reference range and reportable range have not been determined for this test in synovial fluid.”)

• Why might lab be unable to establish all specs on a fluid?o Matrix of fluid (esp. pathologic fluids) may vary among patients—so each

sample is to some extent uniqueo Insufficient number of fluid samplespo Reference range may not be relevant if result is compared to blood level

(ex. protein in pleural fluid); for pathologic fluids “normal” may not existo Test is ordered very rarely

− Pathologist clinician discussion may be appropriatePathologist-clinician discussion may be appropriate


C li ti B d Fl id tComplications—Body Fluids, cont.

• Lab should validate fluids to greatest extent possible for:o Normal fluids (urine, CSF) (vs. pathologic fluids, ex. ascites)o Tests on fluids “routinely” offered by labo Lab should document attempts to estab. specifications, incl. literature

review as applicable, and reasons why full validation not practicalreview as applicable, and reasons why full validation not practical

• For tests on a fluid that are FDA-cleared/approved for the fluid, lab must verify mfg performance specifications (ex. urine & CSF total protein)

• Performance specifications for blood may be used for a body fluid if lab can either:

o Exclude matrix interferences in the fluid [spiking studies]o Quantify any matrix interference & factor it into fluid measuremento Quantify any matrix interference & factor it into fluid measurement

• More detail to be added in 2013 checklist edition



You are running pleural fluid pH tests on your ABG analyzer. The manufacturer lists only blood as an acceptable specimen type. Which of the following statements is true?


Mi ti f A l ti V lid ti R i t f GEN t COMMigration of Analytic Validation Requirements from GEN to COM

Requirement GEN no. of citations

COM no. of citationscitations citations

Approval of validation studies by director 94 176Accuracy/precision 18 65Analytic sensitivity 9 29Analytic sensitivity 9 29Interferences 9 24Reportable range 16 33Data available to clients 1 23

Communicate significant method changes 0 2Establish/verify reference intervals 14 30Re-evaluate reference intervals as necessary 3 13Total 164 395Total 164 395

--First seven months of 2011: GEN contains requirements--First seven months of 2012: COM contains requirements--395/164 = 140% increase in citations at on-site inspections


p

Cli i l V lid ti R i t i CAP f LDT’Clinical Validation Requirements in CAP for LDT’s

• Limited to MOL and MIC (molecular section)

MOL 31590 (II) Cli i l f h t i ti f h • MOL.31590 (II) Clinical performance characteristics of each assay are documented

o Diagnostic sensitivity & specificity, and positive & negative predictive values

o May depend on clinical setting, genotype/phenotype associations when these vary with particular mutations or polymorphisms, & genetic/environmental/epigenetic factors affecting clinical expression of a genetic alteration

o For FDA cleared/approved tests, literature or mfg data can be usedo For laboratory-developed tests (LDT’s)—lab should perform studies in-

house- Exceptions when literature can be used: very common conditions p y

with well-established clinical validity in literature and rare conditions for which lab cannot access sufficient no. of patients


CAP Cli i l V lid ti R i t tCAP Clinical Validation Requirements, cont.

• MOL.31710 (II) All clinical claims made by the laboratory about an LDT are validatedLDT are validated.

• NOTE: Clinical claims may include statements about a test's diagnostic sensitivity and specificity, ability to predict the risk of a disease or condition, clinical usefulness, or cost-effectiveness Clinical disease or condition, clinical usefulness, or cost effectiveness. Clinical claims may be found on the test report or in other information distributed by the laboratory (websites, test catalogues, newsletters, memoranda, advertisements, etc.). Laboratories are not required to

k li i l l i b t t t b t l i d b th make clinical claims about a test, but any claims made by the laboratory must be validated. In general, the laboratory should validate claims through a clinical study, but for rare conditions or well-accepted uses of a test, reference to published peer-reviewed p , p pliterature is acceptable.


CAP Cli i l V lid ti R i t i MICCAP Clinical Validation Requirements in MIC

• Introduction to subsection on LDTs in Molecular section of MIC:

• Diagnostic sensitivity and specificity must be determined relative to some "gold standard" (e.g., biopsy findings, clinical findings, etc.). The sensitivity of an assay equals [TP/(TP+FN)] X 100 and the specificity of an assay equals [TN/(TN+FP)] X 100 (TP=true positive, TN=true an assay equals [TN/(TN+FP)] X 100. (TP true positive, TN true negative, FN=false negative, FP=false positive.) Determinations of sensitivity and specificity should be done in a "blinded" fashion (i.e., without prior knowledge of the patient's disease status). For some i f ti it t b ibl t id tif l b f iti infections, it may not be possible to identify large numbers of positives (i.e., patients with the infection) to establish the diagnostic sensitivity of the assay. In such instances, the laboratory should procure as many positive cases as is reasonably possible for method validation y p y pand in addition cite any publications that have investigated the diagnostic sensitivity of the assay.


F th I ith Cli i l V lid tiFurther Issues with Clinical Validation

1. Why restrict clinical validation requirements to MOL and MIC?a The importance of clinical validation is mainly (but not a. The importance of clinical validation is mainly (but not

exclusively) in molecular/genetic LDT’s. b. If applied to all lab areas, clinical validation requirements would

capture LDT’s used for—i. Flow cytometry markersii. Coagulation factor assaysiii. Manual microbiology culturesiv. Immunohistochemistryv. Conventional pap smear

2. Future revisions in clinical validation requirements not clear ---pending FDA decisions on regulation of LDT’spending FDA decisions on regulation of LDT sa. Requirements will most likely be risk-basedb. CAP has proposed partnership with FDA


[email protected]


R G lResources -- General

• Basic Method Validation, 3rd ed., 2008. Westgard J.O. Westgard QC, Inc.

• Burd EM. Validation of Laboratory-Developed Molecular Assays for Infectious Diseases. Clin Micro Reviews. 2010;23(3):550-576.

• Ricos C, Alvarez V, Cava F, Garcia-Lario JV, Hernandez A, Jimenez CV, Minchinela J, Perich C, Simon M. "Current databases on biologic variation: pros, cons and progress." Scand J Clin Lab Invest. 1999;59:491-500. This database was most recently updated in 2012 Available at Westgard QC website: was most recently updated in 2012. Available at Westgard QC website: http://www.westgard.com/biodatabase1.htm


R CLSI id liResources—CLSI guidelines

• C28-A3 Defining, Establishing and Verifying Reference Intervals in the Clinical L b tLaboratory

• EP15-A2 User Verification of Performance For Precision and Truenesso Protocol simple enough for application in variety of labs from POL to large

li i l l b ith h i t id t ti ti ll lid l iclinical lab, with enough rigor to provide statistically valid conclusions.

• EP05-A2 Evaluation of Precision Performance of Quantitative Methodso For those who develop lab methods, or who wish to verify manufacturer

claims More complex than EP15claims. More complex than EP15.

• EP07-A2 Interference Testing in Clinical Chemistryo To assist clinical laboratories in investigating discrepant results due to

interfering substancesinterfering substances.

• EP09-A2 Method Comparison and Bias Estimation Using Patient Samples o To determine if a new method is a suitable replacement for a current

methodmethod.


CLSI id li ti d—CLSI guidelines, continued

• EP10-A3 Preliminary Evaluation of Quantitative Clinical Laboratory M t P dMeasurement Procedures

o A quick check (over 5 days) to rule out major problems -–addresses bias, carry-over, linearity, precision.

• EP12 A2 User Protocol for Evaluation of Qualitative Test Performance• EP12-A2 User Protocol for Evaluation of Qualitative Test Performanceo For comparing a new test against a current or reference method, or

against clinical data.

• EP17-A Protocols for Determination of Limits of Detection and Limits of Quantitation

• MM16-A Verification and Validation of Multiplex Nucleic Acid Assayso Addresses qualitative and semi-quantitative assays.


CLSI id li ti d—CLSI guidelines, continued

• H57-A Protocol for the Evaluation, Validation, and Implementation of C l tCoagulometers

o Addresses instruments using clotting, immunoturbidimetric, chemiluminescent testing.

• H54-A Procedures for Validation of INR and Local Calibration of PT/INR Systems

o Using certified plasmas for local verification of PT/INR.

• H26-A2 Validation, Verification, and Quality Assurance of Automated H26 A2 Validation, Verification, and Quality Assurance of Automated Hematology Analyzers

o Addresses validation by manufacturers and verification by laboratories.

• C49-A Analysis of Body Fluids in Clinical Chemistry• C49-A Analysis of Body Fluids in Clinical Chemistry


Specific Validation Elements in Other ChecklistsGEN C t th l A t i P th lGEN, Cytopathology, Anatomic Pathology

• GEN .20377 (II) Retain validation records while method is in use and at least 2 years afterwardsy

• CYP.05257 (II) Documentation of validation of liquid-based methods & screening instruments (per manufacturer protocol, if provided)

• ANP 22750 (II) Documentation of validation of new IHC antibodiesANP.22750 (II) Documentation of validation of new IHC antibodieso Optimize antibody titration, detection system, antigen retrievalo Analytic sensitivity & specificityo Well-characterized ab with limited targets: 10 pos & 10 neg sampleso Well characterized ab with limited targets: 10 pos & 10 neg sampleso Others—may require more sampleso Exception for rare ag’s not present in normal tissue

• ANP 22956(II) Validation of FISH/ISH probes• ANP.22956(II) Validation of FISH/ISH probes


Specific Validation Elements in Other Checklists, continuedA t i P th lAnatomic Pathology

• ANP.22976 (II) Validation of ER/PgR IHCo 20 pos & 20 neg samples for FDA-approved/cleared assays; more for LDT’so 20 pos & 20 neg samples for FDA approved/cleared assays; more for LDT so Concordance targets: 90% for positives, 95% for negatives

• ANP.22978 (II) Validation of HER2 in breast carcinoma (IHC or FISH/ISH):o 25 sample minimum—compare vs. validated alternative method, or vs.

same method if performed in a different lab; document degree of concordance for each possible result (e.g., 0, 1+, 2+, 3+ for IHC; neg & pos for (F)ISH); must compare test method to an alternative method to define which cases will be reflexed to alternative method. Next edition: Likely will recommend 40 positive & 40 negative cases.

• ANP.23004 (II) o Digital image analysis-preanalytic

Ti ti i & l ti f i i - Tissue prep, preservation, processing, & selection of microscopic fields


Validation Elements in Other Checklists, cont.M l l P th lMolecular Pathology

• MOL.30785 (II) Documentation of validation of performance characteristics for LDT’s and FDA cleared/approved tests modified by lab

• MOL.30900 (II) Validation studies include samples for each reportable genotypeo NOTE: Assays for genetic disorders with a limited number of possible genotypes (e.g.,

hereditary hemochromatosis): confirm ability of assay to detect these genotypes. Assays for genetic disorders with considerable allelic heterogeneity and/or significant y g g y gnumbers of private mutations (e.g., cystic fibrosis or hereditary nonpolyposis colorectal cancer) should confirm the accuracy of the methodology used to ensure that the assay will detect a high percentage of possible genotypes. However, it will not be possible to document that such assays can detect every possible genotype.

• MOL.31015 (II) Validation studies include expected specimen types (frozen, fresh, paraffin-embedded, blood, etc.)

• MOL.31130 (II) Validated method is used as the comparative methodo If no comparative method: then clinical validation (sensitivity/specificity) and run o If no comparative method: then clinical validation (sensitivity/specificity) and run

samples in another lab using same assay


V lid ti El t i Oth Ch kli t tValidation Elements in Other Checklists, cont.

• MOL. 31245 & 31360 (II) Reference & reportable ranges for qual. & quant. testso Qual: define normal result & define reportable outcomes = homozyogous wild,

heterozygous mutant, homozygous mutanto Quant: reference range & AMR

• MOL.31705 (II) For LDT’s, reports contain description of method, statement that assay was developed by lab, and appropriate performance characteristicsdeveloped by lab, and appropriate performance characteristics

• MOL.39323 (II) HER2 Validation in breast carcinoma (see ANP.22978 previous slide)

• MOL.49580 (II) For tests using ASR, report includes FDA-required disclaimero "This test was developed and its performance characteristics determined by p p y

(laboratory name). It has not been cleared or approved by the US Food and Drug Administration.“

o Add’l CAP recommended wording: “"FDA does not require this test to go through premarket FDA review. This test is used for clinical purposes. It should not be

d d i ti ti l f h Thi l b t i tifi d d th regarded as investigational or for research. This laboratory is certified under the Clinical Laboratory Improvement Amendments (CLIA) as qualified to perform high complexity clinical laboratory testing."


Validation Elements in Other Checklists, cont.Mi bi lMicrobiology

• Gas chromatography for microbial detection and HPLC for mycobacterial detection:

• MIC.22840 and MIC.32822 (II) Chromatography method validated using known bacterial strains incl. those expected in clinical samples

Molecular Microbiology* Introduction:Molecular Microbiology Introduction:

• For FDA approved/cleared methods: If lab pools patient samples for testing (ex. Chlamydia trachomatis & Neisseria gonorrhoeae on pooled urine), pooled specs. must be used for validation studies & compared to nonpooled specs incl. entire range of organism concentration seen in clinical specs. (e.g., low pos & high pos specs)

o If limit of detection differs from FDA-approved method for nonpooled specs., validation must follow requirements for LDT’s (see LDT section of (MOL Micro checklist)



Molecular section of MIC: FDA cleared/approved non-amplified methods

MIC 64760 (II) V if f d t i i t bl I l • MIC.64760 (II) Verify mfg data on accuracy, precision, reportable range. Incl. “adequate” no. of samples expected in clinical practice (ex. cervical & urethral swabs, urine). Spiked specs. ok if sufficient no. of pt. samples not available

• MIC 64770 (II) Sample types/collection devices not listed in mfg instructions ok MIC.64770 (II) Sample types/collection devices not listed in mfg instructions ok but must be validated by lab (need not use LDT section of MIC molecular checklist). Complete validation not required if:

oValidation studies in progressoSample type is rare (disclaimer required on report)oSample type is rare (disclaimer required on report)

• MIC.64815 (II) Sample types/collection devices not listed in mfg instructions ok but must be validated by lab (need not use LDT section of MIC molecular checklist). Complete validation not required if:) p q

oValidation studies in progressoSample type is rare (disclaimer required on report)



Molecular section of MIC: FDA cleared/approved non-amplified methods cont.

MIC 64825 (II) L b t lid t t ff f iti lt if difi d f • MIC.64825 (II) Lab must validate any cut-off for positive result if modified from mfg cut-off

• MIC.64860 (II) Documentation of verification study for all testable specimen typestypes

o Qualitative: comparison of pos & neg results to comparative method

o Quantitative: limit of detection*, linearity, reportable range, precision

o Samples can be QC/PT materials, cultured organisms, & must include patient samples

o (*Not in validation requirements in COM for FDA approved/cleared tests)



Molecular section of MIC: Laboratory-developed tests, & FDA cleared/approved tests modified by lab (excluding modifications of sample type/collection device)y ( g p yp )

• MIC.64952 (II) Documentation of validation study

o Analytical and diagnostic sensitivity; analytical and diagnostic specificity, precision linearity (for quantitative tests); reportable range of patient test precision, linearity (for quantitative tests); reportable range of patient test results; reference range (normal values); refer to Test Method Validation Section in GEN [sic]

• MIC.64956 (II) If lab has modified an FDA approved/cleared test, modified test performance must at least equal that of unmodified test

• MIC.64960 (II) Validation studies include adequate no. of specs of each type (blood, fresh, frozen, paraffin, etc); spiked samples ok

• MIC.64964 (II) Validation studies include specs. of each strain or genotype as applicable



• MIC.64968 (II) Validation studies use comparisons to another valid test or to similar test in a different lab

o If there is no comparative method, clinical validation required, plus exchange of specimens with another lab doing similar method

• MIC.64976 (II) For quant. tests, reference & reportable ranges definedo Example, viral load testso Determine how to handle results outside the AMR (=reportable range) o Perform dilution/concentration, or report as greater than or less than the

AMR limit

• MIC.64980 (II) Validation studies include accuracy, analytic sensitivity, analytic specificity, precision, linear range (quant. tests only)

• MIC 64984 (II) LDT reports include method statement that test is LDT and MIC. 64984 (II) LDT reports include method, statement that test is LDT, and appropriate performance characteristics

• MIC.64988 (II) Reports of tests using ASR’s include FDA-required disclaimer


Validation Elements in Other Checklists, cont.Ch i t & Cli i l Bi h i l G tiChemistry & Clinical Biochemical Genetics

• CHM.15300 (II) Reference intervals established or verified

• CHM.15600 If mfg instructions for immunoassay method are modified (incl. modifying dilution ratios), modified test must be validated to show results consistent with mfg performance claims

• CHM.18800 (and CBG.17400) (II) Identification criteria for tandem mass spectrometry are validated

• CHM.30400 (II) Sweat testing methods are validated with specimens equivalent to volume & concentration of patient sweat samples


Validation Elements in Other Checklists, cont.C t tiCytogenetics

• CYG.42700 (II) Procedures for validation of FISH probes

• CYG.46799 (II) If lab modifies FDA approved test, modified procedure is validated to have equivalent or superior performance

• CYG.48399 (II) Validation of HER2 testing for breast carcinoma (see previous ANP item for details)

• CYG.49500 (II) Procedures for validation of microarray genomic copy assessment

• CYG.49525 (II) Nucleic acids extracted & purified by methods from literature, by commercially available method, or by validated method developed in-house


Validation Elements in Other Checklists, cont.F i D T tiForensic Drug Testing

• FDT.19930 (II) Procedure for validation of new methods

• FDT.20030 (II) Validation includes accuracy, precision (at cut-off values), analytic sensitivity (limit of detection) for confirmation procedures, analytic specificity (interferences), linearity for quant. methods including limit of quantitation, carry-overmethods including limit of quantitation, carry over

• FDT.20130 (II) Validation records are complete for all methods & sample types

• FDT.20163 (II) Confirmation method performance validated annuallyfor accuracy, precision, analytic sensitivity, linearity (incl. limit of quantitation), carryover

• FDT.20930 (II) If immunoassay methods modified from mfg instructions, modifications are validated


Validation Elements in Other Checklists, cont.Fl C t tFlow Cytometry

• FLO.30460 (II) Lymphocyte gates validated using linear forward angle light scatter and 90 degree side scatter or by CD45-FITC & angle light scatter and 90 degree side scatter, or by CD45 FITC & CD14-PE monoclonal antibodies

• FLO.30550 (II) Reference intervals established or verified


Validation Elements in Other Checklists, cont.H t lHematology

• HEM.23000 (II) Reference intervals established or verified

• HEM.23453 (II) aPTT-based heparin therapeutic range validated

• HEM.34050 (II) Automated differential method validated against previously validated automated method or manual method

• HEM.35020 (II) Automated reticulocyte method validated against previous manual or automated method

• HEM 35452 (II) Upper & lower limits for automated body fluid cell • HEM.35452 (II) Upper & lower limits for automated body fluid cell counts established

• HEM.37925 (II) If D-dimer method used to exclude thromboembolism, the method is validated for this purposethe method is validated for this purpose

o Mfg data acceptable for FDA-cleared/approved methods; Literature data acceptable for any method


Validation Elements in Other Checklists, cont.Hi t tibilitHistocompatibility

• HSC.21835 (II) Validation of reagents used in different manner from mfg instructionsmfg instructions

• HSC.29997 (II) Lymphocyte gates validated using linear forward angle light scatter and 90 degree side scatter, or by CD45-FITC & CD14-PE monoclonal antibodiesCD14 PE monoclonal antibodies

• HSC.34357 (II) Nucleic acids extracted & purified by methods from literature, by commercially available method, or by validated method developed in-housemethod developed in house

• HSC.36227 (II) Mendelian inheritance of DNA system used is validated by family studies

• HSC.37723 (II) Method of probe labeling validated to detect target sequence without false-positive signal for non-target sequences


Validation Elements in Other Checklists, cont.U i l iUrinalysis

• URN.23150 (II) Establish/verify reference intervals for population being testedtested


Validation Elements in Other Checklists, cont.T f i M di iTransfusion Medicine

• TRM.44925 (II) Platelet counts on platelet components are determined by a method validated to be accurate in the expected concentration rangep g

o Automated whole blood hematology analyzers may yield inaccurate counts in the range of platelet concentrations in platelet components (ex. 1-2 x 106/μL)

( ) i f i i if• TRM.44955 (II) For detection of bacteria in platelet components, if lab or supplier uses a system that is not cleared/approved by FDA, the system is validated to give at least comparable sensitivity to FDA cleared/approved systems


Validation Elements in Other Checklists, cont.I lImmunology

• IMM.32000 (II) Establish/verify reference intervals for population being testedtested


2013 CAP Accreditation Programs Audioconferences and...

Documents

Transcript of 2013 CAP Accreditation Programs Audioconferences and...