TGS-4 Guidance on Test Method Validation for IVDs · Guidance on Test method validation for in...

Technical Guidance Series (TGS)

for WHO Prequalification – Diagnostic

Assessment

Guidance on Test method

validation for in vitro diagnostic

medical devices

TGS–4

Draft for comment 20 December 2016

© World Health Organization 2016

All rights reserved. Publications of the World Health Organization can be obtained from WHO Press, World Health Organization,

20 Avenue Appia, 1211 Geneva 27, Switzerland (tel.: +41 22 791 3264; fax: +41 22 791 4857; e-mail: [email protected]).

Requests for permission to reproduce or translate WHO publications – whether for sale or for non-commercial distribution –

should be addressed to WHO Press, at the above address (fax: +41 22 791 4806; e-mail: [email protected]).

The designations employed and the presentation of the material in this publication do not imply the expression of any opinion

whatsoever on the part of the World Health Organization concerning the legal status of any country, territory, city or area or of

its authorities, or concerning the delimitation of its frontiers or boundaries. Dotted lines on maps represent approximate border

lines for which there may not yet be full agreement.

The mention of specific companies or of certain manufacturers’ products does not imply that they are endorsed or

recommended by the World Health Organization in preference to others of a similar nature that are not mentioned. Errors and

omissions excepted, the names of proprietary products are distinguished by initial capital letters.

All reasonable precautions have been taken by the World Health Organization to verify the information contained in this

publication. However, the published material is being distributed without warranty of any kind, either expressed or implied. The

responsibility for the interpretation and use of the material lies with the reader. In no event shall the World Health Organization

be liable for damages arising from its use.

Contact: Irena Prat, EMP Prequalification Team Diagnostics

WHO - 20 Avenue Appia - 1211 Geneva 27 Switzerland

mailto:[email protected]

Technical Guidance Series for WHO Prequalification – Diagnostic Assessment: Guidance on Test method validation for in vitro diagnostic medical devices TGS–4

of 23 Draft for comment 20 December 2016

Table of contents

Table of contents 3

Acknowledgements 5

1 Definitions 6

2 Introduction 8

3 Scope 8

4 Terminology for test method validation 8

4.1 Explanation of the terms characterisation, verification and validation .................... 8

4.2 Explanation of the terms accuracy, trueness and precision ................................... 10

5 Uses of test method validation in the lifecycle of the IVD 11

6 Test methods 11

6.1 Categories of test methods ..................................................................................... 11

6.2 Statistics and test methods ..................................................................................... 11

6.3 Quantitative and qualitative assays ........................................................................ 11

6.4 Specimen panels and test methods ........................................................................ 12

7 Variability in the test method 12

8 Planning for test method validation 14

9 Examples of test methods and their validation 15

9.1 Validation of test methods related to cleaning processes ...................................... 15

9.2 Validation of test methods for raw materials ......................................................... 17

10 References 22

1



WHO Prequalification Programme: IVD Technical Guidance Series

The WHO Prequalification Programme is coordinated through the Department of

Essential Medicines and Health Products. WHO prequalification of in vitro

diagnostics (IVDs) is intended to promote and facilitate access to safe, appropriate

and affordable IVDs of good quality in an equitable manner. The focus is on IVDs for

priority diseases and their suitability for use in resource-limited settings. The WHO

Prequalification Programme undertakes a comprehensive assessment of individual

IVDs through a standardized procedure that is aligned with international best

regulatory practice. It also undertakes post-qualification activities for IVDs to ensure

their ongoing compliance with prequalification requirements.

Products that are prequalified by WHO are eligible for procurement by United

Nations agencies. The products are then commonly purchased for use in low- and

middle-income countries.

IVDs prequalified by WHO are expected to be accurate, reliable and able to perform

as intended for the lifetime of the IVD under conditions likely to be experienced by a

typical user in resource-limited settings. The countries where WHO-prequalified IVDs

are procured often have minimal regulatory requirements, and the use of IVDs in

these countries presents specific challenges. For instance, IVDs are often used by

health-care workers who do not have extensive training in laboratory techniques, in

harsh environmental conditions, in the absence of extensive pre- and post-test

quality assurance capacity, and for patients with a disease profile that differs from

the profiles encountered in high-income countries. Therefore, the requirements of

the WHO Prequalification Programme may differ from the requirements of high-

income countries, or those of the regulatory authority in the country of

manufacture.

The Technical Guidance Series (TGS) was developed following a consultation held on

10–13 March 2015 in Geneva, Switzerland. The consultation was attended by

experts from national regulatory authorities, national reference laboratories and

WHO prequalification dossier reviewers and inspectors. The guidance series is a

result of the efforts of this and other international working groups.

This guidance is intended for manufacturers interested in WHO prequalification of 2

their IVD. It applies in principle to all IVDs that are eligible for WHO prequalification 3

for use in WHO Member States. This guidance should be read in conjunction with 4

relevant international and national standards and guidance. 5

The TGS guidance documents are freely available on the WHO website.

WHO Prequalification of IVDs

Procurement of prequalified IVDs

Prequalification requirements

About the Technical Guidance Series

Audience and scope



Acknowledgements

The draft document “Technical Guidance Series for WHO Prequalification – Diagnostic Assessment:

Guidance on Test method validation for in vitro diagnostic medical devices” was developed with support

from the Bill & Melinda Gates Foundation and UNITAID. This draft was prepared in collaboration with Dr

J Duncan, London, United Kingdom; D Healy; R Meurant, WHO; and with input and expertise from Dr V

Alcón; D Lepine; IVDD Section, Medical Devices Bureau Health Canada, Ottawa, Canada; Dr S Hojvat,

MD, USA; and Dr S Norman, CA, USA. This document was produced under the coordination and

supervision of Robyn Meurant and Irena Prat, Prequalification team – Diagnostic Assessment, WHO,

Geneva, Switzerland.



1 Definitions

The section below provides definitions which apply to the terms used in this document. 6

Accuracy: The closeness of agreement between a test result and the accepted reference value. 7

(1) 8

Lot: Defined amount of material that is uniform in its properties and has been produced 9

in one process or series of processes. 10

NOTE: The material can be either starting material, intermediate material or finished 11

product. (2) 12

Characteristic: Distinguishing feature 13

Note 1 to entry: A characteristic can be inherent or assigned. 14

Note 2 to entry: A characteristic can be qualitative or quantitative. (3) 15

Note 3 Characterisation: a description of the distinctive nature or features of 16

something. (3) 17

Control material: Substance, material or article used to verify the performance characteristics of an in 18

vitro diagnostic medical device. (4) 19

Control procedure: Activities at the point of use to monitor the performance of an IVD medical device. 20

Note 1 In the IVD medical device industry and in many laboratories that use IVD 21

medical devices, these activities are commonly referred to as quality control. 22

Note 2 Quality control may monitor all or part of the measurement procedure, from 23

the collection of samples to reporting the result of the measurement. (4) 24

In vitro diagnostic medical device (IVD): A medical device, whether used alone or in combination, 25

intended by the manufacturer for the in vitro examination of specimens derived 26

from the human body solely or principally to provide information for diagnostic, 27

monitoring or compatibility purposes. 28

Note 1 IVDs include reagents, calibrators, control materials, specimen receptacles, 29

software, and related instruments or apparatus or other articles and are used, for 30

example, for the following test purposes: diagnosis, aid to diagnosis, screening, 31

monitoring, predisposition, prognosis, prediction, determination of physiological 32

status. 33

Note 2 In some jurisdictions, certain IVDs may be covered by other regulations. (5) 34

In vitro diagnostic reagent/IVD reagent: Chemical, biological or immunological components, solutions, 35

or preparations intended by the manufacturer to be used as an IVD. (2) 36

Life-cycle: All phases in the life of a medical device, from the initial conception to final 37

decommissioning and disposal. (6) 38

Limit of detection, detection limit: Measured quantity value, obtained by a given measurement 39

procedure, for which the probability of falsely claiming the absence of a component 40

in a material is β, given a probability α of falsely claiming its presence. 41



Note 1 IUPAC recommends default values for α and β equal to 0.05. 42

Note 2 The term analytical sensitivity is sometimes used to mean detection limit, but 43

such usage is now discouraged. Modified from (7) 44

Limit of quantitation, quantitation limit: Lowest value of measurand in a sample which can be 45

measured with specified measurement uncertainty, under stated measurement 46

conditions. (7) 47

Measurand: Quantity intended to be measured. (7) 48

Objective evidence: data supporting the existence or verity of something 49

Note 1 Objective evidence can be obtained through observation, measurement, test, 50

or by other means. (2) 51

Performance claim: Specification of a performance characteristic of an IVD medical device as 52

documented in the information supplied by the manufacturer. 53

Note 1 This can be based upon prospective performance studies, available 54

performance data or studies published in the scientific literature. (2) 55

WHO Note “Information supplied by the manufacturer” includes but is not limited 56

to: statements in the instructions for use, in the dossier supplied to WHO and / or 57

other regulatory authorities, in advertising, on the internet referred to simply as 58

“claim” or “claimed” in this document. 59

Precision: The closeness of agreement between independent test results obtained under 60

stipulated conditions. (1) 61

Quality: Degree to which a set of inherent characteristics of an object fulfils requirements. (3) 62

WHO Note: for the purpose of this document these requirements include fitness-for-63

use, safety and performance. 64

Quality assurance: Part of quality management focused on providing confidence that quality 65

requirements will be fulfilled. (3) 66

Ruggedness (robustness): A measure of an analytical procedure’s capacity to remain unaffected by 67

small but deliberate variations in method parameters and provides an indication of 68

its reliability during normal usage. (8) 69

Standard method: A method that is (metrologically) traceable to a recognized, validated method. 70

Non-standard method: A method that is not taken from authoritative and validated 71

sources. This includes methods from scientific journals and unpublished laboratory-72

developed methods. (8) 73

Trueness: The closeness of agreement between the average value obtained from a large series 74

of test results and an accepted reference value. (1) 75

Validation: Confirmation by examination and provision of objective evidence that the 76

requirements for a specific intended use have been fulfilled. (3) 77

Verification: Confirmation through the provision of objective evidence that specified 78

requirements have been fulfilled. (3) 79



2 Introduction

The purpose of test method validation is to ensure that a method consistently provides produces results 80

fit or adequate appropriate for a specific purpose. Testing must have a useful purpose and the result 81

from the test must be shown to be meaningful and to give the expected (and appropriate) information. 82

In order to ensure meaningful results, the test method must be validated; otherwise the measurement 83

has little purpose and no economic value. By using validated test methods, a manufacturer can have 84

confidence that claims made in respect to the quality and performance of an IVD are supported by 85

objective evidence. 86

3 Scope

This document is intended to provide guidance on the validation of the test methods used in 87

manufacturing of an IVD. Sometimes test methods are referred to as analytical methods but in the 88

context of establishing the design, the development and manufacture of an IVD, “test method” is the 89

more commonly used and a more appropriate description since not all testing is analytical. Minimal 90

specific guidance relating to test method validation is available for IVD manufacturers despite the 91

abundance of guidance for the analytical chemistry or pharmaceutical industries (e.g. those from 92

Eurachem (10), Eurolab (11), ICH (12), WHO (13) and FDA (14)) or for clinical laboratories compliant with 93

ISO 15189 (15). This document provides information on validating the test methods used by 94

manufacturers of IVDs in their research and development (R&D), quality control and quality assurance 95

laboratories; it must be read as an adjunct to those formal guides mentioned previously. 96

This document is not intended to give guidance on validation of the IVD itself. For this, it is 97

recommended to refer to “TGS3 Principles of performance studies” (16) in this series. Qualification of 98

instrumentation is outside the scope of this document although the test methods used in qualification 99

must be validated (17). This document does not outline statistical methods for analysis of the required 100

data. 101

4 Terminology for test method validation

4.1 Explanation of the terms characterisation, verification and validation 102

Although internationally accepted definitions exist for the terms characterisation, verification and 103

validation, the following explanation is provided to give greater clarity with relation to test method 104

validation. 105

Characterisation, verification and validation are essential terms. For the purposes of this guidance 106

document “characterisation of a test method” refers to an experimental procedure and the 107

documentation of its characteristics. It is undertaken in order to provide objective evidence of what a 108

method is capable of consistently achieving under defined conditions. The characteristics of the assay 109

are the numerical values proven and documented for each of the method attributes such as sensitivity, 110

specificity, limit of detection etc. Each attribute should be evaluated using an appropriate, validated test 111

method. 112

Verification is the documentary proof that particular specifications have been met. When designing and 113

developing an IVD, relevant attributes such as cost, and those for performance such as precision, 114

sensitivity and stability are identified and given numerical specifications in design input documentation. 115



It is subsequently the role of the R&D department to design an IVD that will meet those specifications. 116

The R&D department consequently identifies valid test methods to demonstrate that the specifications 117

have been met (verificationed) in the new design. Once design has been established, further numerical 118

specifications are produced by the R&D department to ensure that the specifications of each attribute 119

will be met consistently in routine production to ensure quality manufacturing. These new specifications 120

are assigned to control critical production points and may include those for acceptance of raw materials, 121

in-process materials, cleanliness of equipment, qualification of instrumentation and for the finalised IVD 122

to verify its manufacture. Again, it is also the role of the R&D department to identify appropriate test 123

methods to monitor these specifications. An example of verification is related to incoming goods 124

inspections; each time a raw material is purchased its properties will be verified against the specification 125

using a validated test method. 126

Validation is the documentary proof that the particular requirements for a specific intended use can be 127

consistently fulfilled (9). VIM (7) defines validation as “verification against needs for a specific use” (i.e. 128

the specification for that use). Within this guide, consistency is essential: it is an expectation that every 129

lot of an IVD will behave as all other lots and will continue to meet design inputs. To ensure this, it is 130

necessary to have validated test methods for measuring and/or monitoring specifications that will 131

consistently produce results fit for purpose. The test methods must be validated to ensure that the 132

results of measuring and/or monitoring are meaningful. For example, the need for accurate 133

measurement of a raw material weighed in micrograms will not be achieved by using a weighing device 134

with tolerance measured in grams. A test method using such an instrument would not be valid for the 135

intended use. Thus, for the example provided, a test method should be specified that has the necessary 136

accuracy and precision for measuring such weights, and an instrument and procedure identified that will 137

consistently achieve this requirement during its use. The test method is then validated to produce 138

results fit for purpose. 139

Validation of a test method is distinct from its characterisation. Characterisation is documentation of 140

some or all of the features of the method; validation is ensuring that the relevant characteristics are 141

appropriate for the specific intended use. Validation of a method to be used widely, and for standard 142

methods, often begins with complete characterisation. However, for each specific intended use it is 143

likely that only a subset of the characteristics will be relevant and must be evaluated. 144

Published guides to test method validation provide the broad characteristics of assays as set out in Table 145

1. 146



Table 1: Examples of characteristics of assays 147

Characteristic Attributes

Trueness bias, recovery, accuracy, matrix effects

Precision repeatability, intermediate precision, reproducibility

Selectivity specificity, interferences

Sensitivity traceability, working range, linear range, limit of detection, limit of

quantitation, uncertainty at clinically significant threshold values

Reproducibility precision under defined conditions: repeatability, ruggedness, robustness

Stability Ability to maintain characteristics (i.e. reagents, analyte, (specimen types)

throughout the intended use period

Productivity† speed, hazards, cost

† Productivity is not usually mentioned in test method validation texts but is important in

manufacturing environments. Although cost mustis not be a factor considered in risk minimization (6),

it should be a consideration in choice and validation of test methods.

Usually only a small selection of the possible characteristics and attributes will ever be studied for a test 148

method specifically developed for a single purpose. 149

4.2 Explanation of the terms accuracy, trueness and precision 150

The terms accuracy, trueness and precision have specific meaning for technical documentation. 151

Trueness and accuracy are the values obtained for the method under investigation and relative to a 152

value accepted as truth, being established through the use of an accepted traceable calibrator or derived 153

by testing using an accepted reference measurement method on the same item as measured by the 154

method under test. Without an accepted value neither trueness nor accuracy can be given for a test 155

method, only a percent (positive and/or negative) agreement. 156

Trueness is a measure of the closeness of agreement between the accepted value and the average of a 157

large [infinite] number of results from a test or assay method under review. It is expressed as a bias: “the 158

result from this test method has a bias of ±units”. Trueness is a characteristic of the method. 159

Precision is a comparison of the results obtained on the same test method. It does not encompass 160

comparison to another value obtained using a method associated with trueness. It is a measure of the 161

closeness of agreement between independent test results obtained under stipulated conditions using 162

the same test method and encompasses concepts such as repeatability and reproducibility, depending 163

on the specified conditions. It is expressed in terms of a standard deviation or related measures “the 164

precision (or repeatability or reproducibility etc.) of this test method under these conditions is ±y-units”. 165

Precision is a characteristic of the method. 166

Accuracy is a measure of the closeness of agreement between the accepted value as documented and 167

the result of a measurement using the item (i.e. test equipment etc.). It is a characteristic of that single 168

measurement and has components from both trueness and precision of the test method. Each time the 169

test method is performed the accuracy of the measurement is likely to be different because of 170

experimental error and the imprecision of the test method. If a test method requires the documented 171

result to be the average of several individual measurements, the accuracy is related to that average; the 172



fact of limited replication does not convert accuracy to trueness although it may improve the accuracy. 173

Accuracy is expressed in terms of bias: “the accuracy of that measurement was -z-units” (1, 7). 174

Both precision and trueness for a particular method, and subsequently the accuracy of an assay by that 175

method, can be influenced by the concentration of the measurand. As such, when characterising a 176

method, knowledge of the performance of the assay should be obtained over a range of foreseeable 177

measurand concentrations, to ensure the validity of any assumptions regarding the performance of the 178

method. 179

5 Uses of test method validation in the lifecycle of the IVD

Testing in the R&D phase of the life-cycle of a commercial IVD is often to ensure that the work in 180

progress will meet the input requirements or to verify that the test development meets those 181

requirements. Design requirements such as a claim of lack of interference from similar analytes will need 182

to be supported by evidence generated using validated test methods. 183

During production, testing is usually employed to ensure that the material being tested meet its 184

specifications. Test methods will use classical analytical chemistry or biochemistry to evaluate the quality 185

of materials coming into, or synthesised by the factory e.g. commercial chemicals, enzymes, 186

recombinant proteins, peptides or nucleic acids. Validation of the test methods will ensure that the 187

correct attributes are measured appropriately. 188

6 Test methods

6.1 Categories of test methods 189

Test methods can be categorized as standard or non-standard. Standard methods are metrologically 190

traceable to a recognized, validated method and do not require additional characterisation by IVD 191

manufacturers. Pharmacopoeia and various national regulations document approved standard methods. 192

In contrast, non-standard methods must be individually characterised and validated for the intended 193

use. However, all methods must be assessed as appropriate for the specific intended use, and must be 194

verified as being used correctly (6, 13, 19). 195

6.2 Statistics and test methods 196

It is recommended to seek expert statistical advice during the planning stage of all experiments to 197

ensure that sufficient numbers of specimens are tested to provide statistically powered results. These 198

are required to justify any claim, and to provide reasonable estimates of uncertainty. 199

Frequently statistical differences will be found that have no practical consequence. For that reason 200

practical differences, or limits of confidence, should always be defined before experiments are 201

performed. 202

6.3 Quantitative and qualitative assays 203

Most test methods will produce numeric, quantitative results, but some assays can only produce 204

qualitative output: the binary result of analyte present or analyte absent relative to a particular cut-off 205

value. For qualitative assays some of the characteristics listed in section 4 Table 1 cannot be enumerated 206

without applying advanced statistical methods. For guidance on this issue see Valcárcel et al. 2002 (20). 207



Some IVDs are intended only to produce qualitative results in users’ environments but it is usual (and is 208

in most cases probably is essential) that the test methods used in manufacture (quality assurance, 209

quality control) will provide a quantitative result. In most cases qualitative assays can be adjusted to 210

provide a quantitative result, either from an instrumental reading, e.g. for an enzyme immunoassay, or 211

against a graduated reference scale (semi-quantitative reporting of a present/absent result as is the 212

case with many rapid diagnostic tests or photometric reading of an RDT as a measure of the amount of 213

target analyte bound in the test zone) . If a quantitative result cannot be obtained then experiments and 214

results must be designed to be analysed by appropriate qualitative statistical methods. Documenting an 215

outcome as merely positive or negative without giving an uncertainty estimate is rarely sufficient, 216

particularly for test methods intended to characterise an IVD (e.g. for stability, precision, sensitivity) or 217

for release-to-sale testing. For more information, refer to “TGS-6 Panels for quality assurance and quality 218

control of in vitro diagnostic medical devices” (insert ref). 219

6.4 Specimen panels and test methods 220

Test methods used to verify design and consistent production will frequently involve the choice and use 221

of panels of specimens in order to determine and/or monitor quality characteristics of an IVD. Panels 222

must be designed and specimens selected to ensure that data generated usefully demonstrates that the 223

specifications have been met. Designing a valid method to assess sensitivity of antibody detection for 224

example, will need to take into account the fact that testing of dilutions of a strong positive specimen 225

will not produce results that reflect the performance of the assay with respect to seroconversion 226

sensitivity. Similarly, the panel composition for release-to-sale and stability testing must employ panels 227

utilising specimens demonstrated to reflect the state of an IVD relative to real, critical specimens to be 228

valid. It is useful to note that test methods used for an IVD in both the design and development phases 229

as well as during production can have various applications, for example the experiments undertaken at 230

release-to-sale can be also be used with adjusted criteria in proving demonstrating the stability of the 231

IVD. 232

7 Variability in the test method

A critical attribute of all test methods is that they must be less variable than the parameter being 233

evaluated (i.e. have a higher precision). The variability of the test method must not conceal variability in 234

that which is tested. This requirement is usually studied as “gauge R&R” (Gauge Repeatability and 235

Reproducibility, refer to Burdick et al (21)) but the process and methodology applies to any measuring 236

system, not just to gauges. As a rule of thumb, the gauge should have a variance of less than 20% of the 237

variance of the “test-piece”. In this context, it is unreasonable to make claims based on one lot of an IVD 238

evaluated in one or two similar laboratories, regardless of how many individual specimens are tested. It 239

is important to understand the variability between lots of IVD and the test method must be capable of 240

revealing it. For instance, it is an accepted published practice to use three lots of an IVD to demonstrate 241

stability (22), however, no guidance is provided on what actions are required when significant lot-to-lot 242

variability is identified during stability testing. Shelf-life should be assigned statistically taking into 243

consideration the variance between lots (see “TGS-2 Guidance Document for establishing stability of in 244

vitro diagnostics assays and components” in this series (23)). Due to the requirement of high precision of 245

the test method, using a previous lot of the IVD as a simple comparator is unlikely to meet the 246

requirements of being a validated test method unless there is sufficient knowledge and control of the 247



variability associated with each lot. The concerns regarding variability apply equally to all claims, 248

including specificity and sensitivity. 249



8 Planning for test method validation

Figure 1: Test method validation process 250

The flow of the test method validation process is 251

shown in Figure 1. The steps will be described in 252

detail in the examples of test method validation to 253

follow. 254

Understanding the real, intended purpose of the test 255

method is critical. The US FDA (24) states that: 256

“Design input is the starting point for product design. 257

The requirements which form the design input 258

establish a basis for performing subsequent design 259

tasks and validating the design. Therefore, 260

development of a solid foundation of requirements is 261

the single most important design control activity”. 262

ISO 13485 (25) requires that “design and 263

development outputs shall meet the input 264

requirements”. Compiling requirements and 265

comparing outputs to inputs is essential for activities 266

that require detailed planning and execution. 267

Once the input requirements which define the exact 268

purposes for the test method are documented and 269

agreed (e.g. to ensure a particular claim will be met 270

for the whole shelf-life of the IVD), the required 271

characteristics of the test method can be specified 272

and given measurable attributes, usually following 273

risk assessments and development work. 274

The following are examples of input requirements: 275

ability to detect an increase of 0.5% in the invalid 276

result rate of an IVD; ability to detect 10% loss of 277

sensitivity for a particular epitope; evidence that 278

infectivity of a positive control material is reduced by >100-fold. It is important that the assignment of 279

design input specifications, and accordingly the assigned acceptance criteria, are clearly defined as the 280

first ste in the process, rather than by undertaking experimental testing using a rtest method that is 281

under consideration to determine minimum performance requirements. 282

The method can then be developed and validated against these predetermined needs. Without the 283

predetermined needs the method cannot be validated, merely characterised. 284

The analysis of incoming raw materials is usually supported by well-established standard methods 285

and/or published routine test method validation. There are usually a greater array of test methods (e.g. 286

chemical analyses, protein, peptide or nucleic acid sequencing or terminal analyses, spectroscopy, 287

chromatography, electrophoresis, electro-blotting) than for routine quality control or quality assurance 288

of an IVD itself but the link between specifications, predetermined test method characteristics, the 289

capability of the method and the utility of results must still be documented. 290

Identify the required

characteristics of the test method

Compare performance with

requirements

Is themethod fit for

purpose?

YESYES

Use method

Definethe purpose of the

testing

Select or develop the test method

NOselect a different

method or redevelop

Assign numerical values to the

attributes



For quality assurance and quality control of an IVD, the test method frequently requires decisions on the 291

attributes to be measured and numbers of specimens for testing panels. The finalised test methods and 292

specimens to be used are usually developed together during the R&D for an IVD. Given the important 293

relationship between the chosen test method and the testing panel, it is almost always too late to try to 294

find appropriate specimens for the panels after R&D is completed. 295

9 Examples of test methods and their validation

This section gives examples of test methods and their validation for some aspects of IVD manufacture. 296

The examples were generated following WHO analysis of the deficiencies in evidence provided dossiers 297

submitted to support prequalification. The analysis concluded poor understanding of the importance of 298

test method validation, and its necessity in support of claims (for example on lot to lot reproducibility, 299

stability, specimen types). A manufacturer will need to evaluate each phase of work, processes and 300

materials and adapt the procedure outlined in section 08 to the particular IVD. The examples in this 301

document are neither authoritative nor complete. However, if the test method is not valid and 302

documented, the claim is not supported. 303

9.1 Validation of test methods related to cleaning processes 304

Introductory discussion 305

This example of test method validation is of validation of the methods used in verifying cleanliness, not 306

validation of the cleaning process itself. The specifications of what constitutes “clean” must be 307

ascertained on a case by case basis. This is usually from risk analysis based on chemical knowledge of the 308

reasons the cleaning process is necessary and some experimental evidence: why cleaning is essential, 309

the cleaning agents used and the probable subsequent uses of the cleaned item. Once the specifications 310

for cleanliness are known, proven and documented, the requirements of the test methods can be 311

defined. 312

As a simple example consider cleaning a vessel used for preparing conjugates, last used for a conjugation 313

of a monoclonal antibody with an enzyme and now to be used for preparing other conjugates. 314

a) Define the purpose of the testing 315

The residual conjugation chemicals, antibody and enzyme and subsequently any cleaning agents must all 316

be removed in order not to contaminate the next solutions in the vessel. 317

The vessel will be cleaned with pressurised hot water containing an organic anionic detergent followed 318

by alkali and acid rinses and finally rinsing with distilled water and drying. 319

b) Identify the required characteristics 320

The characteristics required are trueness, sensitivity, selectivity and precision for each of the possible 321

analytes. 322

The criteria for successful cleaning could be based in a standard operating procedure requiring vigorous 323

extraction of the cleaned vessel with a defined volume of distilled water prior to any drying stage in 324

order to avoid artefacts from an unclean vessel appearing clean because of difficulty in detecting dried-325

on contaminants. The methods must be able to detect any contamination of the water that could in 326

principle affect subsequent use of the vessel. 327



c) Assign numerical values to the attributes 328

Typically the specification for the water after rinsing could be: less than 10 ppm of total organic carbon, 329

less than 5 ppm of residual protein, less than 1 ppm of residual detergent, less than 1 µM in conjugation 330

related reagents and a conductivity of less than 0.5µS. 331

These are typical specifications for equipment used for fermentation or protein handling and experience 332

shows that if they are achieved the vessel is likely to be acceptably clean for these purposes. However 333

the utility of the cleaning process with these specifications would need to be validated (26) for the 334

specific use before finalising and validating the test methods. 335

d) Select or develop the method(s) 336

The extraction of the water from the vessel is part of the test method: it is required to demonstrate 337

during validation by R&D that a second, similar extraction would contain unmeasurable amounts of the 338

potential contaminants and, independently, that nothing of practical importance would leach into the 339

next solution to be used in the vessel. This aspect of the work requires different methods to those used 340

for the routine verification. These more sensitive methods are required to be validated in the R&D phase 341

of the work (validated for use in the conjugation solution). 342

The validation of the (non-standard) test methods used in this type of work is thoroughly exemplified in 343

the formal pharmaceutical test method validation guides. Validation of the total organic carbon 344

measuring system originates from the manufacturer’s specification and the subsequent performance 345

qualification. 346

As the measurements are made in almost pure water it would not be necessary to verify lack of 347

interference from other constituents of the matrix. Similarly if the conductivity meter was specified as 348

being capable of accurate readings superior than the requirement, further validation would not be 349

necessary. 350

Residual detergent would be measured using an instrument, for example high performance liquid 351

chromatography (HPLC). The method chosen would require characterisation of the sensitivity, accuracy 352

and precision for this purpose and the specific detergent involved. Functionality and conformation of 353

proteins would not survive the acid and alkali washes so any contaminating protein would be measured 354

by a chemical technique, defined in the standard operating procedure for the cleaning process. The 355

definition of the test method is essential as each method of protein quantitation (Lowry, biuret, binding 356

of various dyes and HPLC) provides marginally different results for specific proteins. The sensitivity of the 357

method would be demonstrated to be capable of meeting the requirement and the precision to show 358

that the stated level of protein could be determined with sufficient accuracy. Chemical methods may be 359

used to analysis cleanliness relating to the conjugation reagents. The test method must be defined and 360

the precision and sensitivity in the matrix of distilled water demonstrated to be appropriate. 361

e) Compare performance with requirements and use the methods if adequate 362

Once the methods have been characterised and proven to meet the required numerical attributes they 363

can be used in routine verification of the cleaning process. 364

Over time as the process is shown to consistently produce cleanliness within the required 365

specification, testing would be minimised: i.e. the process itself would be validated 366

(consistently providing cleanliness fit for purpose). However this can only be done by prior 367

use of validated test methods. 368



9.2 Validation of test methods for raw materials 369

9.2.1 Routine commercial materials 370

Most commercial chemicals (salts, acids, alkalis, sugars) have standard analytical methods from 371

pharmacopoeia, (needing no further evaluations except for verification of proper use and documentary 372

evidence of the required level of quality in the materials. The scope of testing for routine commercial 373

chemicals requires individual assessment. However this should be easy with reputable suppliers. 374

9.2.1.1 Components 375

Components to accompany the IVD such as sachets of drying agents, specimen collection devices and 376

tubes, transfer pipettes or dropper bottles will require testing (and documentation) against the specific 377

requirements of the IVD. 378

Example: validation of an incoming test for transfer pipettes used to drop specimen into an IVD. 379


The purpose of the testing is to demonstrate that across the lot of transfer pipettes, the volume 381

delivered meets the specification provided by the R&D department during the development of the IVD. 382

The specifications provided by the R&D department would have been validated to demonstrate that all 383

the claims of the assay (sensitivity, specificity, precision, etc.) are met through the assigned life of the 384

IVD using the pipettes which provide a volume within the specification. 385


The required characteristics are the trueness and the precision of the lot of transfer pipettes, for each 387

specimen type claimed. 388

Further specifications that need to be evaluated for such pipettes may include: orifice diameter and 389

overall length (measures of trueness and sensitivity required), ability to deliver discrete drops easily by 390

untrained individuals (i.e. an in-use precision measure). Each of these requires a validated specification, 391

numerical limits and consequently a test method validated as giving the required information. The 392

following example below is only for the volume measurements. 393


The specification for volume delivered into the IVD, validated by the R&D department, might be “not 395

less than 30 µL and not more than 45 µL of specimen to be delivered in two drops from the pipette” 396

which in the instructions for use would be translated to “add two drops of specimen using the dropper 397

pipette provided”. The specification of the pipettes would be “to deliver 35-40 µL ± 2 µL in two drops 398

and evaluated across the lot”. This would have been validated by R&D for each specimen type claimed. 399

From that specification, the requirement of the test method would be a bias of < 1 µL in the range 30 – 400

45 µL and a precision of < ± 0.8 µL (variance ≈20% of that allowed in the volume specified for the 401

pipettes). 402

d) Select or develop the method 403

The test method (weighing drops of water) is unlikely to introduce bias or imprecision beyond that in the 404

specification (based on the assumption that properly maintained and calibrated weighing 405

instrumentation is accurate) so the most important validation aspect is the relationship between drops 406



of water and drops of each specimen type from the pipettes. The number of randomly selected pipettes 407

and the proportion of lots to be tested are calculated on the basis of acceptable risks (27) as confidence 408

in the supplier increased. 409

It is unlikely that the quality assurance incoming goods inspection team have access to the specimens 410

claimed for the IVD (e.g. fresh whole blood, fresh serum, cerebrospinal fluid). Hence any volume 411

measurements on a substitute liquid (e.g. water) with volume estimated by weight must be validated. 412

Drop volumes and variances of different liquids differ due to density and surface tension effects. 413

The exact method and required specifications would be documented in a standard operating procedure 414

in addition to data recording, monitoring requirements and a reference to the validation of the test 415

method. 416

e) Compare performance with requirements 417

The required characteristics of the test method can be measured and compared against the specification 418

(the precision and the number of pipettes to be tested). Consequently the method may be used if found 419

to be fit for purpose. 420

As can be seen, method validation at this level require planning, experimental work and documentation 421

beyond merely defining the method (“weigh some drops”) and the specification for the component. 422

9.2.1.2 Package labels, instructions for use, vials, stoppers etc. 423

Ancillary materials (e.g. printed matter, packing materials, containers, stoppers) will at a minimum, need 424

inspection prior to acceptance. Inspection is a test method. As usual, the requirements of the test 425

method (the inspection) must be defined once the purpose of the testing is understood. The attributes 426

of the test method can be evaluated (e.g. the pre-defined consumer risk and hence the proportion of the 427

incoming delivery to examine, the capability of the inspectors to distinguish and record the attribute) 428

and the method validated to consistently assure appropriate quality. Regardless of inspection method 429

chosen, its capability to detect flaws at the required level of risk must be documented as must be the 430

standard operating procedure for performing the inspection. 431

9.2.2 Constituents critical to IVD performance 432

Critical constituents must be decided on a risk assessment basis. Nitrocellulose membranes, some 433

detergents and all complex biological reagents (peptides, proteins, and oligonucleotides) are assumed to 434

be critical (unless there is evidence to the contrary). 435

Testing critical constituents typically involves techniques such as HPLC, spectroscopy, mass-436

spectrometry, sequencing and various forms of electrophoresis. All instrumentation is assumed to be 437

correctly documented, qualified and operated and the instrumentation itself will not contribute to bias 438

or uncertainty in the example below. The latter assumptions are usually true for the biological systems 439

described here; it is the nature of the measurements made that requires validation. Assessment of 440

critical constituents should be more detailed than for non-critical constituents (even if the critical 441

materials are obtained commercially). A commercial supplier cannot know the exact use of the material 442

and can only give a general certificate of analysis (COA). Where components are deemed critical, it is 443

necessary to have in place a supplier quality agreement which includes the requirements for a supplier 444

to notify the IVD manufacturer of any proposed changes to a supplied component, or to the process of 445

manufacturing of specific components, This would allow for adequate time for production planning and 446

validation activities. 447



9.2.2.1 Example of acceptance testing for a low molecular weight constituent 448

Some detergents (those containing a polyether bond e.g. Tween, Triton) easily and quickly generate 449

peroxides (28). 450


Peroxides can disrupt enzyme activity and the conformation of recombinant proteins and some 452

peptides. For this reason it may be considered necessary to monitor the peroxide content of detergents 453

used in an IVD, either at the incoming goods check or often just prior to use. 454


The characteristics required are sensitivity (range and uncertainty at specific concentrations), trueness 456

(accuracy, bias) and precision. 457


R&D should have proven the stability of the IVD in studies using various lots, some of these lots at the 459

end of their shelf-lives (18, 23). Preliminary stability experiments should lead to knowledge of the 460

maximum permissible concentration of peroxide, (specified as, for example, < 6.0 µM) in the detergent 461

used routinely in the manufacture of the IVD. 462

An example of specifications for the test method derived from the R&D requirements could be “no bias, 463

sensitivity of 5 µM ± 1 µM at a concentration near 55 µM “(i.e. ability to distinguish between 50 and 55 464

µM and to allow for 10-fold dilution of a stock solution), with a peroxide specification of <55 µM in the 465

stock solution: to give an acceptable margin of safety relative to the permissible concentration and the 466

test method variance. 467

d) Select or develop the method 468

Several suitable methods for measurement of peroxide in aqueous detergent solutions are available. 469

However, as they are not standard methods, they necessitate characterisation and validation of the 470

required sensitivity, bias and precision near the permissible concentration of peroxide in solutions of the 471

particular detergent. 472

e) Compare performance with requirements 473

Clear specifications and justification of both method and expected result are required. 474

9.2.2.2 Acceptance testing of molecules with defined structures 475

For short peptides and oligonucleotides, the COA from an established and reputable supplier usually 476

gives sufficient structural detail (e.g. proof of sequence and terminal residues (usually by mass 477

spectrometry) and freedom from synthetic artefacts and residues (usually by HPLC)). As result, further 478

acceptance measurements are usually not required. However, for peptides containing cysteine (or 479

cystine) residues it might be necessary to monitor the state of the received material to ensure lack of 480

oxidation (or reduction), requiring a validated method for measurement of sulphydryl content. 481

Monoclonal immunoglobulin G class antibodies (but not polyclonal antibodies) are normally robust in 482

production and a COA of identity and purity is generally sufficient. Polyclonal antibodies, which vary in 483

avidity and precise epitopic dependence from animal to animal, may require similar functionality testing 484

to that suggested in the following for recombinant proteins. This is also true for immunoglobulin M class 485

antibodies, whether monoclonal or polyclonal. 486



9.2.2.3 Acceptance testing of recombinant proteins and polynucleotides 487

Recombinant proteins and polynucleotides require more complex testing than for molecules with a 488

simple sequence. The functionality of recombinant proteins and polynucleotides is frequently 489

dependent on the conformation of the macromolecules and specificity depends on the precise 490

impurities present (among other things). The material’s specification must include requirements for 491

functionality, for purity based on similarity of contaminants between lots, measures of sequence 492

integrity and molecular conformation. 493

The test methods involved in preparing satisfactory COAs, or of providing evidence of satisfactory in-494

house preparations, are much more complex than those in the elementary examples given above. 495

However, validation of the methods follows exactly the same principles. Usually the methods are well 496

known analytical procedures. It may be the case that they are not “standard” methods but adequately 497

known and characterised so that if used appropriately they do not require further validation. 498

A problem observed in many submissions to WHO prequalification is that either the methods are not 499

used at all, or the output is not appropriate for the task. The following section discusses these major 500

issues. It does not provide detail of the process of validation, but an expectation of how the well-known 501

test methods will be used. 502

Both purity and conformation are critically lot dependent and are not usually documented in sufficient 503

detail in a commercial COA to provide objective evidence of inter-lot reproducibility. A standard 504

commercial COA for a recombinant protein usually provides a result for purity from a gel after 505

electrophoresis (e.g. “>95 %”) without specifying the exact concentrations and molecular weights of the 506

impurities (so allowing lot-to-lot variation within the specification and hence potential for specificity and 507

stability issues). A COA will also usually provide a molecular weight which is determined from a gel or a 508

Western blot. However, neither technique are adequately sensitive to demonstrate minor post-509

translational modifications, nor capable of providing any information about conformation (allowing 510

potential sensitivity and selectivity issues). Quantitation of results from both stained and blotted gels is 511

not reproducible without special techniques and gives only approximate values (29). A COA should 512

always give some measure of uncertainty in the stated values of both molecular weight and quantity. A 513

competent COA should also include amino- and carboxy- terminal amino acid analyses to ensure 514

absence of minor proteolysis during purification. 515

Choice of correct methods, and knowledge of their limitations, is a major deficiency in most WHO 516

prequalification submissions. There is insufficient proof that there is no lot-to-lot variability, neither in 517

those critical materials nor in the final IVD made from them. 518

Before committing to purchase or process a substantial amount of a new lot of recombinant protein, a 519

careful manufacturer will need to check that the new lot will detect the difficult specimens 520

(seroconversion, latent stage, unusual serotype specimens) with the sensitivity and specificity claimed 521

for the IVD and with approximately the same utilization (devices per milligram) as the lots used to 522

validate the IVD itself. Proficient manufacturers develop testing of identity, integrity and functionality of 523

polynucleotides to be used in IVD. 524

These tests for complex critical reagents can only be specified for commercial or for in-house reagents 525

by the manufacturer of the IVD, since the requirements are unique to the IVD and its validated claims. 526

Nevertheless, the methods used must be validated as suitable for use. 527

The sensitivity, specificity and utilization measurements on new lots of recombinant proteins are usually 528

made by preparing the IVD on a small scale and testing against defined panels of specimens proven to 529



monitor the stated parameters with satisfactory efficiency. Test method validation is to ensure that the 530

panels do indeed monitor the expected parameters. 531



10 References

1. ISO 5725-1:1994: Accuracy (trueness and precision) of measurement methods and results — 532

Part 1: General principles and definitions. Geneva, Switzerland: International Organization for 533

Standardization; 1994 534

2. ISO 18113-1:2009. In vitro diagnostic medical IVDs – Information supplied by the manufacturer 535

(labelling) – Part 1: Terms, definitions and general requirements. Geneva, Switzerland: 536

International Organization for Standardization; 2009. 537

3. ISO 9000:2015. Quality management systems – Fundamentals and vocabulary. Geneva, 538

Switzerland: International Organization for Standardization; 2015. 539

4. ISO 15198:2004. Clinical laboratory medicine – In vitro diagnostic medical IVDs – Validation of 540

user quality control procedures by the manufacturer. Geneva, Switzerland: International 541

Organization for Standardization; 2004. 542

5. GHTF/SC/N4:2012 (Edition 2). Glossary and Definitions of Terms Used in GHTF Documents. 543

Global Harmonization Task Force (GHTF) Steering Committee; 2012. 544

6. ISO 14971:2007. Medical devices – Application of risk management to medical IVDs. Geneva, 545

Switzerland: International Organization for Standardization; 2007. 546

7. ISO/IEC Guide 99:2007: International vocabulary of metrology -- Basic and general concepts and 547

associated terms (VIM). Geneva, Switzerland: International Organization for Standardization; 548

2007. 549

8. US FDA Volume II: Methods, Method Verification and Validation ORA-LAB.5.4.5 October 2003, 550

Revised August 2014. http://www.fda.gov/ScienceResearch/FieldScience/ucm171877.htm 551

9. United States CFR - Code of Federal Regulations Title 21. Sec. 820.3 Definitions. Washington DC, 552

United States of America; 2010. 553

10. Magnusson, B., and Örnemark, U. (eds.) Eurachem Guide: The Fitness for Purpose of Analytical 554

Methods – A Laboratory Guide to Method Validation and Related Topics, (2nd ed. 2014). ISBN 555

978-91-87461-59-0. (http://www.eurachem.org) 556

11. Eurolab: Validation of Test methods. General principles and concepts: EL1545/96 557

(http://www.eurolab.org) 558

12. ICH: Validation of analytical procedures: text and methodology {Q2(R1)} November 1996 559

(http://www.ich.org/products/guidelines/) 560

13. WHO Technical Report Series, N°937, Appendix 4, Analytical method validation, of Annex 4 561

Supplementary guidelines on good manufacturing practices : validation - WHO, 2006 562

14. US FDA Center for Drug Evaluation and Research (CDER): Bioanalytical Method Validation, May 563

2001 & September 2013 (draft update) 564

15. ISO 15189:2012. Medical laboratories -- Requirements for quality and competence. Geneva, 565

Switzerland: International Organization for Standardization; 2012 566

16. WHO Prequalification – Diagnostic Assessment. Technical Guidance Series (TGS). Principles for 567

Performance studies TGS–3. Geneva: World Health Organization; 2016. Available at: 568

https://www.iso.org/obp/ui/#iso:std:iso:18113:-1:ed-1:v1:en

https://www.iso.org/obp/ui/#iso:std:iso:18113:-1:ed-1:v1:en

http://www.iso.org/iso/home/store/catalogue_tc/catalogue_detail.htm?csnumber=45481



http://www.imdrf.org/docs/ghtf/final/steering-committee/procedural-docs/ghtf-sc-n4-2012-definitions-of-terms-121109.pdf

https://www.iso.org/obp/ui/#iso:std:iso:14971:ed-2:v2:en

http://www.fda.gov/ScienceResearch/FieldScience/ucm171877.htm

http://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfcfr/cfrsearch.cfm?fr=820.3

http://www.iso.org/iso/home/store/catalogue_ics/catalogue_detail_ics.htm?csnumber=56115



http://www.who.int/diagnostics_laboratory/guidance/technical_guidance_series/en/, accessed 569

15 July 2016. 570

17. CLSI. Quality Management System: Equipment; Approved Guideline. CLSI document QMS13-A 571

Wayne, PA: Clinical and Laboratory Standards Institute; 2011 572

18. CLSI. Evaluation of Stability of In Vitro Diagnostic Reagents: Approved Guideline. CLSI document 573

EP25-A. Wayne, PA: Clinical and Laboratory Standards Institute; 2009 574

19. CLSI. User Estimation of Precision and Estimation of Bias CLSI document EP15-A3 Wayne, 575 PA: Clinical and Laboratory Standards Institute; 2014 576

20. Valcárcel, M., Cárdenas, S., Barceló, D. et al. Metrology of qualitative chemical analysis, KI-NA-577

20-605-EN-C, ISBN 92-894-5194-7; 2002 Available free of charge from: 578

http://bookshop.europa.eu/en/metrology-of-qualitative-chemical-analysis-pbKINA20605/ 579

21. Burdick, R.K., Borror, C. M., & Montgomery, D. C. (2005). Design and analysis of gauge R and R 580

studies: making decisions with confidence intervals in random and mixed ANOVA models. ASA & 581

SIA Math: ISBN 0898715881 582

22. ISO 23640:2011. In vitro diagnostic medical IVDs - Evaluation of stability of in vitro diagnostic 583

reagents. Geneva, Switzerland: International Organization for Standardization; 2011. 584

23. WHO Prequalification – Diagnostic Assessment. Technical Guidance Series (TGS). Establishing 585

stability of an in vitro diagnostic for WHO Prequalification TGS–2. Geneva: World Health 586

Organization. 2016. Available at: 587

http://www.who.int/diagnostics_laboratory/guidance/technical_guidance_series/en/, 588

Accessed 15 July 2016. 589

24. US FDA Center for Devices and Radiological Health (CDRH): Design control guidance for 590

medical device manufacturers March 1997. Available at: 591

http://www.fda.gov/medicaldevices/deviceregulationandguidance/guidancedocuments/uc592

m070627.htm Accessed 20 December 2016 593

25. ISO 13485:2003. Medical devices – Quality management systems – Requirements for regulatory 594

purposes. Geneva, Switzerland: International Organization for Standardization; 2003. 595

26. WHO Technical Report Series, N°937, Appendix 3, Cleaning validation, of Annex 4 596

Supplementary guidelines on good manufacturing practices : validation - WHO, 2006 597

27. ANSI/ASQ Z1.4–2003 (R2013): Sampling Procedures and Tables for Inspection by Attributes, 598

2013. Available at: https://www.ansi.org/ Accessed 20 December 2016. 599

28. Jaeger J, Sorensen K, Wolff SP. Peroxide accumulation in detergents. J Biochem Biophys 600

Methods. 1994;29:77–81. doi: 10.1016/0165-022X(94)90058-2 601

29. Taylor, SC, Posch, A. (2014) The Design of a Quantitative Western Blot Experiment BioMed 602

Research International, volume 2014, Article ID 361590 8 pages, 2014 603

http://dx.doi.org/10.1155/2014/361590 604

http://bookshop.europa.eu/en/metrology-of-qualitative-chemical-analysis-pbKINA20605/



http://www.fda.gov/medicaldevices/deviceregulationandguidance/guidancedocuments/ucm070627.htm

http://www.fda.gov/medicaldevices/deviceregulationandguidance/guidancedocuments/ucm070627.htm

https://www.ansi.org/

http://dx.doi.org/10.1155/2014/361590

TGS-4 Guidance on Test Method Validation for IVDs · Guidance on Test method validation for in...

Documents

Transcript of TGS-4 Guidance on Test Method Validation for IVDs · Guidance on Test method validation for in...