Risk Analysis Preview

Six Sigma Tools and Metrics

• Learn what ISO standards, CLSI guidelines, and Joint Commission protocols recommend for Risk Analysis• Identify hazards and failure modes in your own processes• Apply the appropriate ranking scale (qualitative, semi- quantitative, and quantitative) to your process• Use Risk Analysis tools to assess and judge the acceptability of risks in your processes.

Risk Analysis is coming to medical laboratories. But for too many labs, Risk Analysis is a buzzword without meaning, an approach without defined technique. In this book, Dr. Westgard surveys the ISO standards (ISO 14971, ISO 22367) as well as the CLSI guidelines (EP18, EP23) and the Joint Commission methodology for Proactive Risk Reduction. After providing an overview of the general approach to Risk Analysis, Dr. Westgard explains how to adapt the principles for the medical laboratory, using data-driven tools and practical implementation tips:

• Process maps, flowcharts and fishbone diagrams • Risk Acceptability matrices • Assessment of hazards through Failure Mode Effect Analysis (FMEA) • Fault Tree Analysis (FTA) and Failure Reporting, Analysis and Corrective Actions System (FRACAS) • Six Sigma metric integration into the Risk Analysis techniques

Using Six Sigma metrics, Dr. Westgard shows how Risk Analysis can be converted from an arbitrary and qualitative technique, into something concrete, quantitative, and relevant to medical laboratories and the patients they serve. For laboratories serious about adopting Risk Analysis in their operations - and manufacturers eager to provide industry-leading support of their instruments - this is an essential reference.

7614 Gray Fox Trail • Madison WI 53717Copyright 2011 Westgard QC Inc.

http://www.westgard.com

SIX SIG

MA

RISK

AN

ALY

SIS W

ESTGA

RD

QC

, INC

.

Six Sigma Risk Analysis

Designing Analytic QC Plans for the Medical Laboratory

James O. Westgard, PhD

Copyright © 20117614 Gray Fox Trail, Madison WI 53717

Phone 608-833-4718 HTTP://WWW.WESTGARD.COM

Library of Congress Control Number: 2011906056

ISBN 1-886958-27-0ISBN-13 978-1-886958-27-2

Published by Westgard QC, Inc.7614 Gray Fox TrailMadison, WI 53717

Phone 608-833-4718

Copyright © 2011 by Westgard QC, Inc. (WQC). All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, or otherwise, without prior written permission of Westgard QC, Inc..

Westgard QC, Inc. Copyright © 2011

Page i

PrefaceWith manufacturers building control mechanisms into their analytical systems, laboratories are interested in customizing their quality control systems on the basis of risk analysis and the remaining failure modes, i.e., errors that still may occur and affect the quality of laboratory testing. The focus in this book is on the development of Analytic Quality Control Plans to fit the needs of a particular analytic instrument as operated in an individual medical laboratory.

The QC issues discussed in this book trace their origin to CMS’s 2004 interpretative guidelines for “Equivalent QC” (EQC), which allow laboratories to reduce the frequency of QC from 2 lev-els of controls per day to 2 per week or even 2 per month. There is no scientific evidence to support the equivalence of these practices to traditional QC procedures. There is no proof that these reduced QC frequency protocols provide adequate error detection or patient safety, even though CMS prescribed “validation protocols” in order to qualify for reduced QC frequency. Those validation protocols themselves are not valid.

The EQC guidelines seem to be driven by a desire to simplify QC practices, particularly for Point-of-Care applications where op-erators have little experience in doing laboratory tests and minimal knowledge of traditional QC practices. Because of concerns and urgings of both manufacturers and laboratories, CLSI initiated a project to develop a new guideline for QC procedures based on risk analysis. That guideline is known as EP23 “Laboratory Quality Control based on Risk Management.”

Manufacturers are generally familiar with risk analysis be-cause of practices recommended in ISO 14971 – Application of risk management to medical devices. Medical laboratories, on the other hand, have little or no experience with formal risk analysis. We, like others, had to start from scratch to study risk analysis in order to understand its potential application for developing Analytic QC Plans. In the process of learning about risk analysis, we examined the risk models and the reliability of particular techniques for es-timation or calculation of risk. We assessed the practicality of the risk analysis methodology that was being recommended. On the


Page ii

basis of our studies, we concluded there could be serious problems in implementing the CLSI guidance unless additional educational materials and training programs are made available to laboratories.

This book is part of our efforts to provide more practical and quantitative guidance for the application of risk analysis in medical laboratories. In Part I, “First, do no harm!”, we consider the problems and issues that must be addressed within the overall framework for managing analytical quality in a medical laboratory. In Part II, “ISO and CLSI Guidance,” we review several international and national consensus standards, including ISO 14971, ISO 22367, ISO 15198, CLSI C24, CLSI EP18, and CLSI EP23. In Part III, “Methodology and Tools,” we recommend adoption and adaptation of the JC (Joint Commission) Proactive Risk Reduction methodology and illustrate the application of many of the tools that are useful for developing Analytic QC Plans. Our resulting methodology employs a more rigorous risk analysis model and a more quantitative approach for assessment of the residual risk of an Analytic QC Plan.

As quality management systems evolve, risk analysis should be integrated with existing practices, particularly with Six Sigma concepts, principles, tools, and metrics. That is the basis of our approach in Six Sigma Risk Analysis. Six Sigma is inherently risk oriented in its definition of tolerance limits, estimation of defects, and characterization of defect rates. Risk models, when properly applied, can provide estimates of the number of defective test results that may produced by a laboratory. Such defective test results are potentially harmful or hazardous to the health of our patients.

Given that Six Sigma Risk Analysis is a more advanced and complex subject, it will require more extensive and more thorough study than some of our previous books. This book assumes a basic knowledge of quality control, such as found in our book on Basic QC Practices that describes the principles and procedures for imple-menting Statistical QC to monitor the performance of the methods in your laboratory. It assumes knowledge of the experimental and statistical techniques for evaluating the analytical performance of measurement procedures, such as found in our book Basic Method Validation. It also builds on Six Sigma concepts, principles, and tools, as found in our book on Six Sigma Quality Design and Control


Page iii

and in our book on Assuring the Right Quality Right, which focuses on the design of Statistical QC procedures to verify the attainment of the intended quality of test results (the technical requirement for QC as stated in ISO 15189).

A Note on ISO standards and CLSI guidelinesAs of the time of this printing, the CLSI EP23 was not yet finalized and accepted as a guideline. Nevertheless, the final shape of the document is complete.

But in that spirit, readers should note that each standard, guideline, and regulation is inevitably a moving target. ISO and CLSI continuously review and attempt to improve their documents, and every few years they issue an update. The regulatory and accredi-tation bodies (CLIA, CAP, JC, etc.) do the same. Thus, the specific language of some of these standards will change. However, as you are probably well aware, large changes in regulatory policy are rare. It is unlikely that major changes will occur that change the goals of these organizations and their recommendations.

It is also important to note that this book is NOT meant to replace or substitute for ISO standards or CLSI guidelines. Laboratories are strongly encouraged to purchase the spe-cific documents that they intend to use in their operations. For a manufacturer or laboratory that intends to implement Risk Analysis, it will not be sufficient to read just this book.

Where this book can be helpful is to give an overview and a comparison of ISO, CLSI, and JC recommendations. Laboratories may be able to decide which documents to purchase, as well as how to reconcile the differences between the different standards and guidelines.

The other unique feature of this book is the Sigma-metric ap-proach. While the other recommendations tend to be vague on how to rank and judge the acceptability of risk, this book is very quantitative and data-driven. Assessing your risk on the Sigma-scale will give you a concrete estimate of its “residual risk” and impact on patient care. The combination of Six Sigma and Risk Analysis can provide powerful tools and techniques to the medical laboratory.


Page iv

AcknowledgmentsFirst and foremost, Sten Westgard made this book possible

through his perspective on risk analysis in the world outside the laboratory, his interest that we find a way to help laboratories do a better job of risk analysis, and his commitment to supporting pub-lication of these new materials, both as Internet courses and this book. I am fortunate that once these words are typed into a word processor, they can be transformed into educational materials and quickly made available to the laboratory community. Sten makes that happen and these materials wouldn’t exist without him.

In developing Six Sigma Risk Analysis, we have been stimu-lated by discussions with many people, including Jan Krouwer, Don Powers, Greg Cooper, Tina Krenc, Mike Noble, and Jim Nichols. We have also been stimulated by the evolving standards and guidelines for application of risk analysis with medical devices and medical laboratories, and appreciate the time, effort, and hard work that has gone into their development. We hope that our distillation of the current guidelines and our recommendations for adapting the JC methodology will support the implementation of a more rigorous and objective approach for risk analysis in medical laboratories, particularly for the design of Analytic QC Plans that will verify the attainment of the intended quality of results, as recommended in ISO 15189.

But wait, there’s more!This book is not long enough to contain all the tools and lessons that we want to make available to you. So we’ve put some more online.

For readers who have purchased this book, additional tools available online. For example, a set of power function curves that show the error detection and false rejection capabililites of Average of Normals (AoN) QC procedures.

Visit http://www.westgard.com/risk-extras.htm for more de-tails. You’ll be given a link to a special location, along with password instructions.

Westgard QC, Inc, Copyright © 2011

Table of Contents

1. Controlling Quality .....................................................................................................................................1

2. Managing Analytical Quality ......................................................................................................13

3. Analyzing and Assessing Risk .................................................................................................25

4. A Safety Net to Catch Analytical Errors ........................................................................51

5. ISO 14971 Risk Management for Medical Devices .........................................67

6. ISO 15198 and CLSI C24 Guidance for Safe Use and QC .......................95

7. EP18 & EP22 Guidance for Risk Analysis and QC Plans .....................113

8. ISO 22367 Guidance for Risk Management .......................................................... 137

9. Adopting the JC Risk Analysis Methodology .................................................... 147

10. Diagramming a Laboratory Process .......................................................................... 163

11. Identifying Failure Modes ........................................................................................................ 177

12. Prioritizing Failure Modes ....................................................................................................... 189

13. Determining Root Causes ....................................................................................................... 203

14. Mitigating Risks with an Analytical QC Plan .................................................... 213


15. Estimating Detection and Evaluating Residual Risks .......................... 233

16. Monitoring Failures and Measuring Performance .................................... 259

17. Implementing Analytical QC Plans .............................................................................. 275

18. Integrating Six Sigma into Risk Analysis ........................................................... 283

Index ................................................................................................................................................291


Page 25

3: Analyzing and Assessing Riskwith Sten Westgard, MS

What is Risk Management?

Existing ISO guidelines and emerging CLSI guidelines recom-mend that risk management be applied in healthcare laboratories. Laboratories face a learning curve that includes basic concepts and terminology, as well as practical applications for the total testing process. Our particular interest here is the application of risk man-agement to the analytical portion of the testing process, particularly the development of an Analytical QC Plan that takes into account the potential risk of a particular test, its analytical methodology, and its application in a laboratory environment.

We want to be forthright about our concern about the ap-plication of risk management in healthcare laboratories. We agree with the principles of risk management – to prevent problems from occurring and reduce harm to patients when problems do occur. It would be ideal if errors can be prevented by manufacturers in their design of analytic systems and minimized by built-in controls and instrument checks, but laboratories are still responsible to “verify the attainment of the intended quality of test results,” according to ISO 15189 [1]. We believe that means Statistical QC should be a major part of any Analytical QC Plan. Also, we have reservations about practice guidelines that are emerging, particularly in the US where the drive has been to reduce the amount of QC performed, rather than to optimize QC to guarantee the quality of test results.

Our purpose here is to demonstrate that the principles of risk management can be related and applied quantitatively as part of analytical quality management and with the application of tradi-tional SQC. To accomplish this, we will review the principles of risk management, define the terms to clarify the risk management Process, focus on the steps involved with risk assessment and a


Page 26

commonly-used tool – Failure Mode Effects Analysis (FMEA), then demonstrate how its application to analytical testing can be related to the Sigma performance of analytical methods and systems.

An all-too-brief history of Risk ManagementPeter Bernstein, author of Against the Gods: The Remarkable Story of Risk [2], places the birth of Risk (as we know it) in a coffee house in London around the year 1696. Edward Lloyd, the enterprising owner of the Starbucks’ of his day, noticed his customers – mostly sailors and ship captains, since he was near the docks – had an insatiable thirst for information about the arrivals and departures of ships, as well as details on sea conditions and news in foreign countries. He published all of this information in what became known as “Lloyd’s List.” All of this information, assembled together, dramatically improved the ability to launch new ventures.

“[A]nyone who was seeking insurance would go to a broker, who would then hawk the risk to the individual risk-takers who gathered in the coffee houses or in the precincts of the Royal Exchange. When a deal was closed, the risk-taker would confirm the agreement to cover the loss in return for a specified premium by writing his name under the terms of the contract; soon these one-man insurance operators came to be known as ‘underwriters.’”

In 1771, nearly one hundred of the underwriters formally created the Society of Lloyds. Over 300 years since Edward Lloyd began serving information along with coffee, Lloyds of London has grown into a leading supplier of specialist insurance.

Of course, you might protest, that’s the birth of insurance, not risk management. But the two concepts are intertwined. Insurance is an answer to the problem of risk. It’s illustrative to see that at the very birth of the risk and risk management, information was critical to the process. Even if sea merchants lacked the informa-tion or ability to mitigate their hazards by redesign (i.e. change the weather so their ships would avoid storms), they gained some idea of what to avoid.


Page 27

As we return to the history of risk, we need to connect the birth of risk and risk management to its formal adoption and implementation in healthcare. To do that, we have to leap forward several hundred years to the malpractice crisis of the 1970s and 1980s. During that time, verdicts and settlements against hospitals and other health-care institutions threatened to overwhelm their financial reserves. Risk management became a way to address the threat of litigation. While at first this was a defensive, reactive move, a way to shield the institution from litigation and losses, eventually healthcare institutions began to use risk management proactively [3]:

“Professionals with clinical experience were hired with the hope that they could identify the systemic problems in specific clinical areas (primarily obstetrics, anesthesia, and the emergency department), engage clinicians and educate them about the need to modify specific behaviors, and work collaboratively with others on the clinical and administrative teams to help design environments that would be more conducive to the delivery of safe care.”

When the reports of the Institute of Medicine, To Err is Hu-man [4] and Crossing the Quality Chasm [5] were issued in 2000 and 2001, respectively, both the public and healthcare professionals alike were shocked by the frequency and severity of medical errors. That heightened awareness motivated the healthcare field to search for new tools to combat and prevent medical errors. Risk management became one of the new tools to address the problem.

In healthcare, the Joint Commission recommended the use of risk management as part of the Patient Safety Movement. Its ac-creditation guidelines for 2002 included a requirement that health-care organizations should perform at least one Failure Mode Effects Analysis (FMEA) each year [6]. Also in response to patient safety issues, the Institute for Healthcare Improvement (IHI) began provid-ing education, training, and support for FMEA via its website [7]. In addition, the Veterans Affairs National Center for Patient Safety supported the use of FMEA throughout its healthcare institutions[8].


Page 28

Finally, still another thread of history helped bring risk man-agement into healthcare institutions – global standards courtesy of ISO. Long accepted by industry, ISO sets rigorous guidelines for processes and products marketed worldwide. Adherence to ISO standards is often a de facto requirement for businesses to compete globally. As ISO standards were expanded and applied to more and different segments of industry, they developed standards for risk management in Medical Devices (ISO 14971)[9]. The medical device industry, already an industry where litigation worries mandated a robust analysis of potential design flaws and device hazards, found that the risk management techniques married well with their ex-isting efforts to improve quality. From the medical device industry to the medical device marketplace was only a small step. Already ISO 15189 had specified particular requirements for quality and competence in medical laboratories. A further standard, ISO 22367 [10], specified techniques for the “Reduction of error through Risk Management and continual improvement.”

For laboratories outside the US, ISO standards often replace, supplement, or substitute for local government regulations. Some countries simply point to ISO standards and adopt them in their entirety for accreditation of medical laboratories. In the US, how-ever, ISO standards have not been widely adopted because the CLIA regulations have been dominant. With the CLIA Final Rule in 2003 and the subsequent proposal for “Equivalent Quality Control” prac-tices, the door opened wide for alternative quality regulations. Faced with scientific and professional debate about the adequacy of the new EQC guidelines, CLSI began to develop an alternate approach for defining QC Plans based on risk management guidelines that adhered to the ISO standards. These CLSI guidelines are intended to supplement, if not replace, the CLIA guidelines for EQC. CMS will decide whether the new risk-based QC guidelines can be used to provide “equivalent quality testing.”

What is Risk?Risk is both a noun and verb, a concept and an action. We can take risks and we can risk disaster. We can speculate that analytical error is one of the biggest risks in laboratory testing. We can try to


Page 177

11. Identifying Failure Modes

What tools can be used to identify failure modes?

An essential part of risk analysis is identifying what might go wrong, i.e., the potential failure modes in the process under study. The process should first be diagrammed to clarify how the process works. Next, each step of the process should be examined to iden-tify what might go wrong. The technique for doing this begins with brainstorming by the project team to identify potential failure modes, then summarizing and organizing those potential failure modes us-ing a cause-and-effect diagram, also known as a fishbone diagram.

In any risk analysis methodology, the process of interest must first be diagrammed to identify the critical steps or operations, as described in the previous chapter. The next step in the methodology is to identify potential failure modes, i.e., what might go wrong and cause a delay or an error in reporting a test result. The identifica-tion of failure modes often makes use of brainstorming, followed by a graphical summary by “fishbone” diagram – two tools that will be described in this chapter. Later chapters will consider the prioritiza-tion of failure modes to assess their relative importance and guide efforts to reduce the risks of failures, the identification of root causes of the high priority failure modes, and the utilization of risk control option analysis (ISO/CLSI terminology) [1-4] or process redesign strategies (JC terminology) [5] to eliminate causes when possible, detect failures and implement corrective actions for recovery, and reduce the risk of patient harm by providing information for the safe use of test results.


Page 178

Patient testing processThe overall patient testing process involves many steps and can be extremely complicated. To simplify any risk analysis project, it will be critical to confine the project to a part of the patient testing process, otherwise the project may become unmanageable (i.e. the scope will be so large as to be overwhelmed by a multitude of failure modes). An overview of the patient testing process is shown in Figure 11-1, which outlines two major processes – the clinical testing process and the laboratory testing process, the latter often called the “total testing process” by the personnel in a medical laboratory. This laboratory process consists of the pre-analytic, analytic, and post-analytic phases of the patient testing process. It typically begins with the receipt of an order for a test and ends with the report of results. This is the way personnel outside the laboratory usually see the laboratory – a department or “black box” with an input for test requisitions and an output for test results. Their focus is on the “clinical testing process,” or the pre-pre-analytic and post-post-analytic phases of the patient testing process, though they won’t necessarily describe it using this terminology.

In applying risk analysis in a medical laboratory, it will be critical to delineate the scope of the project, particularly whether it crosses the departmental boundaries between the clinical and laboratory processes. Across-department projects will require a more diverse and larger team. For example, projects on patient identification and test turnaround time are inherently difficult be-cause of the number of people and departments that are involved. Even within the laboratory, it is necessary to carefully delineate the start and stop steps to narrow the focus when possible to a primary phase. For example, when the purpose of the risk analysis project is to develop an Analytic QC Plan, the focus will be primarily on the analytic phase of the laboratory testing process. Still, there could be overlap with the pre-analytic phase related to sample quality and overlap with the post-analytic phase related to implementation of control mechanisms (e.g., delta checks), review of test reports, and provision of information for safe use via the LIS or HIS. The proj-ect team may need some representation from different areas in the laboratory department.


Page 179

Pre-AnalyticReceive test order

Identify patientCollect specimen

Transport specimen Process specimen Prepare samples

Distribute samples

Post-AnalyticReview test results

Monitor qualityAdd safety informationEnter in patient recordTransmit test reports

AnalyticReceive & Inspect samples Prepare reagents & controls

Setup analyzerCalibrate methodAnalyze samples

Verify analyzer operationCheck QC

Release test results

Pre-Pre-AnalyticCollect patient info

Select testPrepare patientEnter test orders

Post-Post-AnalyticReview test reportsInterpret test results

Plan treatmentTreat patient

END

NO

YES

Laboratory Testing Process

Clinical Testing Process

Need Laboratory

Test?

START

Figure 11-1. Diagram of the patient testing process, showing the clinical testing process (composed of the pre-pre-analytic and post-post-analytic phases) and the laboratory testing process (composed of the pre-analytic, analytic, and post-analytic phases).

This “big picture” of the patient testing process should provide the frame of reference for narrowing the focus of any risk analysis project. Next, a more detailed description in the form of a flowchart is needed of the specific process of interest. This chapter focuses on the analytic phase of the patient testing process and the identifica-tion of potential failure modes that lead to delayed test results and delayed patient diagnosis or treatment.


Page 180

Analytic testing processThe analytic phase begins with the receipt of samples and ends with the release of test results, as shown in Figure 11-2.

Receive Samples

Inspect samples

Samplesadequate?

Prepare reagents , calibrators, &

controls

NO

Setup analyzer & Perform function

checks

Calibrate & Verify analyzer performance

Analyze samples & controls

Analyzer performance

OK?

Release test results

Trouble-shoot, Take corrective

actionNO

Re-process,Re-collect

AnalyzerReady?

YES

YES

Repeat tests

Post-analyticreview

NO

YES

Figure 11-2. Flowchart for the analytic phase of an example laboratory testing process.


Page 233

15. Estimating Detection and Evaluating Residual Risks

How should the risk mitigations be tested and evaluated?

After identifying the risk mitigation strategies or redesign options for addressing the high priority failure modes, the effective-ness of the redesigned process should be tested and evaluated. For laboratory applications where the redesigned process is actually an Analytic QC Plan, the residual risks should be evaluated to determine acceptability. The Joint Commission and ISO provide some guid-ance on how to test the process, but little useful advice for judging the acceptability of residual risks. The laboratory should assure that the risks are acceptable for the particular clinical or medical applications of the tests performed. In the case of laboratory tests, quality requirements can be defined for the intended use of each test, the detection of control mechanisms can be characterized by their probabilities of rejection, risk can be estimated in the form of a defect rate, and the residual risks can be described in terms of the number of potentially harmful patient test results.

This step is described as “analyzing and testing the redesigned process” in the Joint Commission (JC) methodology [1]. The gen-eral approaches recommended include pilot testing, simulations, and paper testing – which in this case means preparing a second FMEA. All these approaches have some utility in the evaluation of an Analytic QC Plan in a medical laboratory if they are focused on estimation of detection and evaluation of the risks that remain after implementation of the controls, i.e., residual risks. ISO 15198 [2] provides additional guidance on how this might be done, recom-mending studies that include the introduction of failure modes or error conditions, or simulated challenges, and in the case of SQC procedures, “validation may be based on statistical evaluation of the simulated effects of imprecision and/or bias on actual performance data obtained in routine operating conditions.”


Page 234

The combined JC and ISO guidance suggests the following approaches for analyzing and testing an Analytic QC Plan:

1. Evaluate a manufacturer’s information on the performance of built-in controls, which should be based on experimental studies to characterize their detection.

2. Characterize QC performance for stable sample and patient data control procedures on the basis of simulated effects of errors and estimate detection for medically important errors.

3. Prepare a second FMEA using a 3-factor risk model that includes the estimates of detection for the various controls in order to evaluate residual risks.

It is important to distinguish the two uses of FMEAs. In the earlier step, a FMEA was used to prioritize the failure modes that were then addressed by risk mitigation actions. In the latter step, the purpose is to evaluate the effectiveness of those risk mitigation actions and determine the acceptability of the residual risks that remain. JC provides little guidance on how to make this decision. ISO 14971 [3] and CLSI [4,5] describe risk acceptability matrices that are qualitative (and somewhat arbitrary). And though they talk about clinically acceptable risk or medically acceptable risk, the guidelines provide no quantitative relationship between the acceptability matrix and definitions of the quality needed for the intended clinical and medical use of laboratory tests. Despite the risk numbers, it boils down to a personal decision (or a committee vote). You (or the team) decide that the risk is okay, based on your judgment.

There should be a more quantitative, objective way to do this. Analytic QC Plans should incorporate a definition of the allowable total error for the quality required for a test. The expected defect rate provides a rational metric for estimating the risk due to delayed and/or erroneous test results. Defect rate can be used to calculate the number of test results that are potentially harmful to patients, which provides an understandable estimate of residual risk. In this way, laboratories can make an informed judgment on the accept-ability of residual risks, rather than resorting to an arbitrary risk acceptability matrix.


Page 235

1. Evaluate the manufacturer’s information about built-in controlsIdeally, a manufacturer will disclose the risk analysis studies that describe the effectiveness of its built-in controls, but remember a manufacturer is not required to do this. If such information is not available, it may be possible for a medical laboratory to estimate detection for some of these controls by performing the appropriate experimental studies. For example, the effectiveness of serum indices for detection of abnormal sample conditions could be determined in the laboratory by preparing a series of samples having different levels of hemolysis, lipemia, and bilirubin. Such experiments would be a natural part of the method validation studies that should be performed in the laboratory. Experimental testing of other built-in controls might be possible but will be more difficult. It would be much better if the manufacturer performed that testing and provided clear documentation.

If this information is not available from the manufacturer, then the laboratory must implement at least one independent control. And if this risk information is not presented in a practically useful manner (for instance, in terms of a defect rate), laboratories will have little ability to understand the residual risks – again pointing to a need for at least one independent control mechanism whose detec-tion can be determined and optimized for the intended use of the test and the performance of the method. The easiest independent control mechanism in your laboratory is SQC. With a statistical QC procedure, you can determine its detection and verify the attainment of the intended quality of results in your laboratory.

2. Characterize QC detection of statistical control procedures In evaluating the effectiveness of a QC Plan, a laboratory should optimize the probability of detecting medically important errors (true alarms) and minimize the probability of false error detection (false alarms) [5]. Manufacturers are expected to provide information about the performance of their electronic checks, built-in controls, and integrated liquid controls. Laboratories, likewise, must consider


Page 275

17. Implementing Analytical QC Plans

How do you implement a QC Plan?

Implementation of an Analytic QC Plan may be done using manual tools, offline computer tools, or online data processing tools. The available tools will depend on software that resides in an ana-lytic system, computerized work stations, middleware, Laboratory Information System (LIS), or Hospital Information System (HIS). Implementation may involve qualitative, semi-quantitative, or quantitative applications of Six Sigma Risk Analysis. The complex-ity of the analytic systems and the capabilities of the laboratory analysts will affect how risk analysis is applied and how QC Plans are implemented in the medical laboratory.

Applications of Six Sigma Risk Analysis depend on the re-sources available in an individual laboratory, particularly the data analysis programs and information systems for managing the testing processes. Manual systems may be sufficient for low volume labo-ratories and Point-of-Care applications, but computer support will be needed for high volume testing and complex analytic systems. Computer support may be found in the analytic system itself, of-fline and/or online programs. As the volume and complexity of the testing processes increase, online tools that are integrated into the quality management process become increasingly important. Given the many different analytic systems and the many different data processing configurations, the capabilities for implementing QC Plans will vary with the systems available to your laboratory. In the discussion here, we use the terms qualitative, semi-quantitative, and quantitative to refer to the rigor of the risk analysis methodol-ogy and the objectivity in the estimation of risk, not to identify the type of laboratory tests (i.e. qualitative or quantitative lab tests).


Page 276

Qualitative risk applicationsLet’s start with the simplest case of a manual method that performs only one test. This category applies to many Point-of-Care devices and Physician Office Laboratories. The test systems are often “waived,” which means there are minimal regulatory requirements, usually only that the laboratory must follow the manufacturer’s instructions for use. Nevertheless, a good quality system requires that method performance be validated, which should include the definition of the quality requirements for the intended clinical use of the test. With the information on the quality requirement for the test and the precision and accuracy observed for the method, the SQC procedure should be designed to assure detection of medically important errors. Next particular failure modes and events that require testing of control materials should be identified to determine when controls need to be analyzed (the basic safety net for catching errors). The manufacturer’s recommended control mechanisms should be reviewed, along with regulatory and accreditation requirements, to define a QC Plan. The Plan should specify frequency of monitoring, corrective action and recovery procedures, and reporting guidelines that include information for safety.

For simple applications, method validation should include experiments for reportable range, replication, comparison of meth-ods, and verification of reference intervals. Data analysis may be performed using an electronic spreadsheet or Internet tools, such as the method validation calculators provided on the Westgard website (www.westgard.com). Decisions on acceptable performance depend on the definition of the quality required for the intended use, which may be different in Point-of-Care applications and Physician Office Laboratories. Calculation of a Sigma-metric helps the decision-making process, as well as the selection of an appropriate SQC procedure using the Sigma-Metric QC Selection tool from CLSI C24 [1]. Control charts can be prepared for manual plotting of control measurements and should include a tabular record of any changes to the process and all corrective actions. Other control mechanisms recommended by the manufacturer should be integrated into the QC Plan and documented in the control records. Operator training should include training on all of the control mechanisms included in the QC Plan.


Page 291

Index

A

Acceptability matrix, see risk acceptablity matrix

AdvaMed 7Allowable total error (TEa) 21, 58, 222, 285Analytical performance characteristics 52Analytical QC Strategy 19, 22Analytical quality requirements 56Analytic errors 143Analytic QC Plan 19, 21, 151,

178, 191, 222- 226, 233, 259–261Developing 213–230Implementing 275–281The plan, the plan, the plan, the plan! 270

Analytic QC strategy 53, 283Analytic system checks 130Analytic testing process 180–181A safety net to catch analytical errors 51–66Autoverification programs 280

B

Block diagrams and Top-Down Flow-charts 166–169

Blood transfusion process 150Brainstorming 33, 152, 168

basics 181–183Brooks 99

C

Carey 131Cassidy 284Cembrowski 131CLIA 1, 2, 7, 10, 18, 53, 124, 156, 157,

219, 261QC possibilities 3–5

CLIA criteria for acceptable performance 21CLIA Final Rule 28clinical decision interval 21CLSI 5, 9, 17, 28, 32CLSI C24 10, 92, 95CLSI C24 Guidance for SQC Proce-

dures 101–111CLSI EP15 266

CLSI EP18 9, 34, 36, 45, 67, 113–123, 153, 186, 214, 278, 280

CLSI EP22 83, 132CLSI EP23 9, 34, 45, 67, 97, 113-114,

124-132, 163, 183, 199, 213, 219, 280

Cognitive error 139Competency evaluation 268–269Continual improvement 138Control procedure

definition 96Corrective action 138Corrective Actions and Preventive Actions

(CAPA) 264Criticality 31, 36, 117, 118, 153, 191

definition 117Criticality matrix 36, 117Critical systematic error 240–241Crossing the Quality Chasm 27

D

Default QC (DQC) 5Defect rates 285, 287

converting ISO/CLSI ratings 247Delta checks 216Deming 14Design for Six Sigma 17Detailed process flowchart 172–173Detailed process outline 169Detection 33, 42, 43, 132, 153, 189–

200, 214, 216–220, 245, 246, 286estimating 233–256

Developing Analytical QC Plans and Quality Systems 19–24

Device risk mitigation features 132Diagramming a Laboratory Process 152,163–

174Disclosure 221, 262–263

E

Electronic QC 18Eliminating or reducing occurrence 215–216“Equivalent QC” (EQC) 5, 28, 124, 219 options 7

“Option 4” 7“Equivalent quality testing” 5, 28Error assessment 57Error budgets 16


Page 292

Estimating Detection 233–256Examination failure 86External Quality Assessment (EQA) 56,

217, 267

F

Failure Mode and Effects Analysis (FMEA) 17, 27, 32-33, 114-115, 143, 147, 148, 153-154, 189–200, 272

3-factor model 245definition 115example for Turnaround Time 194example tables 250–253preparing a second FMEA 234Preparing a table 192

Failure modes 33, 168, 189-200, 213, 270Identifying 177–187List of potential failures and specific

causes 120–123Potential failure modes causing a delay in

reporting test results 182Failure Reporting, Analysis, and Corrective

Action Systems (FRACAS) 114, 263definition 115

False rejection rate 38Fault Tree Analysis (FTA) 114, 155, 207–

212definition 115example FT for identifying root cause of

delayed test analysis 211FDA 1, 6, 7Fishbone diagram 33, 177–187, 183–186Five-Whys 155, 204, 206–207Flowcharts 15, 33

basics 164–165Formulation of an Analytical Quality Control

Strategy 61–66Fraser 262Frequency of controls 260

G

Guidance to Manufacturers on Validating User QC Procedures 96

H

Harm 29

Hazard 29, 69definition 71related to performance 87–88

Hazard analysis 31, 140Hazard Identification 127Hazard matrix 117Hazard score 117Healthcare FMEA (HFMEA) 117hemolysis 157, 197, 221, 235HFEMA risk scoring matrix 117High risk methods 66Hubbard 286

I

Identification and control of non-conformi-ties 138

Improving detection 216–220Inadequate volume 197Information for safety 90–91Inspection 16Instructions for use

definition 97Integrating Six Sigma into Risk Analy-

sis 283–289Intended purpose 71Intended use 71, 86ISO 17, 34ISO 14971 6, 28, 29, 34, 36, 67–

93, 129, 143, 213, 221General Guidance and Approach for Risk

Management 70Informative annexes 68–69

ISO 15189 3, 9, 22, 25, 28, 36, 52, 67, 92, 137, 221, 262, 266

ISO 15198 6, 91, 95–99, 233Validation of QC Procedures 98–100

ISO 22367 6, 28, 35, 36, 153, 264, 269ISO/TS 22367 131, 137–145IVD failure 86

J

JC Risk Analysis Methodology 147–161Joint Commission 27, 147, 233

rating scale for severity, occurrence, and detection 154

Juran 14


Page 293

K

Koshy 132Krouwer 32, 207, 261

L

Laboratory error 139Latent error 139Lean 17, 159Lipemia 235Lloyds 26Low risk methods 64

M

Management Responsibilities 140Manufacturer’s Risk Report 131McDermott 45Measurement uncertainty 262, 266Measuring Performance 259–272Medically important systematic errors 240Method Decision 285Moderate risk methods 65Monitoring Failures 259–272Multistage QC 46

N

Need for objective and quantitative estimates of risk 285–287

Need for scientific management of qual-ity 284–286

Need for uniform standards of quality in labo-ratory testing 287–289

Non-cognitive error 139Non-statistical QC procedures 53

O

Occurrence 31, 33, 38, 39, 117, 128, 153, 189–200, 216, 245, 246, 286

qualitative ranking 78semi-quantitative ranking 78

Operating point 61OPSpecs chart 61, 285

P

Pareto analysis 31definition 117

Pareto diagrams 15Parvin 46, 99, 261Patient data QC 130, 242, 280Patient Safety 27Patient testing process 178–179Peer comparison programs 267, 280Performance specifications 53Periodic review of QC data 267Post-analytic errors 143Power function graph 236-237Powers 32, 35, 36, 45Pre-analytic errors 142Prevention 214Preventive action 138Prioritizing Failure Modes 168, 189–200Proactive Risk Reduction 147–

161, 163, 166, 221, 259Probability for error detection 236Probability for false rejection 236Process capability index, Cpk 39Process redesign options 214Proficiency Testing (PT) 217, 267Protective measures 90

Q

QC Design and Planning 15, 102–105QC Tools

example assessment of feasibility of differ-ent tools 227

Qualitative ranking 34Qualitative risk applications 276–277Quality Assessment (QA) 15quality by design 266Quality Compliance 2Quality Control (QC)

definition 2, 101event-driven 46

Quality Control Plan (QCP or QC Plan) 114, 169, 216, 235

Quality Control rule (QC procedures) 53definition 101

Quality Control Strategy (QC strategy)definition 102high error detection 62low error detection 63moderate error detection 62Sigma-metric relationship 63

Quality Goals (QG) 15, 52, 54–66


Page 294

Quality Improvement (QI) 15Quality Indicators 17, 269Quality Laboratory Processes (QLP) 15Quality Requirements

Biologic variation 56European recommendations for biologic

goals 56Medically important changes 57Stockholm Consensus Conference 54

“Quality triology” 14Quantitative risk applications 278–279

R

Recovery 214, 261Redesigning the process 156–157Redesign options 156Reducing severity 221–222Re-engineering 17Reference change value 262Residual risk 19, 30, 191, 199, 248–

249, 254definition 73evaluating 233–256

Risk 28definition 71

Risk acceptability matrices 36, 82, 118, 143, 191, 234, 255

comparison of ISO and CLSI 129qualitative 80semi-quantitative 81, 144

Risk analysis 30-31, 143comparison of JC and EP23 methods 168definition 72integrating Six Sigma 283–289outline of integrated JC/CLSI/ISO ap-

proach 170–172Risk assessment 30-31, 35

number of factors 34prioritization 35ranking scales 34

Risk control 30, 32, 168definition 73

Risk control option analysis 155, 213Risk estimation 31, 168

definition 72Risk evaluation 30, 35, 36

definition 72Risk information 45, 285

Risk management 17, 26-50comparison of different models 44

Risk management 17Risk Management QC Plan (QCP) 5Risk matrix

qualitative 79semi-quantitative 80

Risk Mitigation 213–230, 286Risk Monitoring 32Risk Priority Number (RPN) 31, 33,

118, 132, 153, 191, 248Risk Reduction 32Risk Score 191Robust Process Improvement 159Root Cause Analysis (RCA) 155, 168, 183,

203–212, 215differences between JC and CLSI 204–205

Roubini 287, 288

S

Safety by design 89–90, 156Selection of Statistical QC procedures 59–66Semi-quantitative ranking 34Semi-quantitative risk applications 277–278Sentinel event 151Severity 29, 31,

33, 37, 117, 118, 128, 153, 189–200, 245, 246, 286

qualitative ranking 77reducing 221–222semi-quantitative ranking 78

Shadow testing 288Sigma-metric 40, 79, 223–225, 240–

241, 245, 285equation 58priority of QC tools in relation to Sigma

performance 225QC recommendations 59

Sigma-Metrics QC Selection Tool 60, 105–109, 285

Six Sigma 16, 17, 275Specific monitors or quality indicators

(list) 265Stable sample controls 130Stamatis 32, 45, 236State Operations Manual (SOM) 7Statistical QC 18Statistical QC (SQC) 2, 5, 219–


Page 295

220, 235, 261, 272, 285definition 102

Surrogate QC 3Surrogate sample 3

T

Tague 207To Err is Human 27Total Quality Management (TQM) 14Total testing process 178Trueness 10Trueness controls 217, 266Trueness controls 268Truth in labeling 1Turbid sample 197

U

Uncertainty 10Use error

definition 71Use errors 86–87

V

Veterans Administration National Center for Patient Safety 117

W

Wrong specimen type 197

Y

Young 262

Westgard QC Order Form Westgard QC, 7614 Gray Fox Trail, Madison WI 53717

CALL 1-608-833-4718 if you wish to pay by purchase order or other means. Fax: 608-833-0640 or 203-389-6753

Item Price (US$) Quantity Subtotal

Westgard software and reference manuals

EZ Rules® 3 QC Design software $695.00

NEW! Six Sigma Risk Analysis (save $20 if you order before September 30th)

$90.00 $70.00

Six Sigma Quality Design & Control, 2nd Edition $90.00

Assuring the Right Quality Right (Advanced Quality Management) $90.00

Basic QC Practices manual, 3rd Edition (most popular) $80.00

Basic Method Validation, 3rd Edition $80.00

Basic Planning for Quality $60.00

CLIA Final Rules for Quality manual $60.00

Westgard Online Courses

Introduction to Risk Analysis for the Laboratory (4 credits) $155.00

Quality Management and Design of Analytical Systems (5 credits) $75.00

"Westgard Rules" and Levey-Jennings Charts mini-course (3 credits) $75.00

Basic QC Practices – complete online course (14 credits) $135.00

Basic Method Validation – complete online course (15 credits) $175.00

Grand Subtotal

Shipping & Handling: Within US = $9 Canada & Mexico = Add 10%; Europe, Asia, & all other countries = Add 20%

Sales Tax (Add 5.5% in WI, 6.0% in CT)

GRAND TOTAL

Save more when you order online! Visit http://www.westgard.com/store.htm

to see specials and coupons available only online

Your Name

Institution

Department

Street Address

City State Zip Code

Country

Business Phone/Business Fax

E-mail Address

Credit Card Type (circle one) VISA Mastercard American Express

Credit Card Number Exp. Date

Signature

**Westgard QC ships domestic orders via US Priority Mail with Delivery Confirmation. For other shipping methods, please call. Fax this to 608-833-0640 or 203-389-6753.

Risk Analysis Preview

Documents

Transcript of Risk Analysis Preview