Defining and Measuring the “Value” of Diagnostic Imaging The “Value” Working Group

Optimization of diagnostic and therapeutic strategies is an important element of medical science (1). For diag- nostic imaging, Fryback and Thombury (2) have de- scribed a six-stage hierarchical model of efficacy in which diagnostic technology can be assessed. The initial stages of the model describe efficacy in terms of physical image quality and diagnostic accuracy (ie. sensitivity and specificity). The middle stages assess the impact of diagnostic information on the clinician’s thinking and therapeutic planning. The later stages consider patient and society outcomes (ie, mortality, morbidity, and cost). These stages are interconnected such that efficacy at lower levels is necessary, although not sufficient, to ensure efficacy at higher levels. There are several im- plications of this model. First, it is clear that most of the research that radiologists currently do (ie, diagnostic accuracy assessment) is an intermediate outcome. Sec- ond, imaging is dimcult to assess beyond such inter- mediate outcomes because of confounding variables. Thus, using conventional measurements, one could be left with the impression that diagnostic information, which does not impact therapeutic planning, is not ef- ficacious to the patient and society.

In November 1994, representatives from imaging, clinical practice, industry, government, payors. and re- search methodology participated in a 3-day session to discuss methods for the evaluation of diagnostic imag- ing. Our subgroup was charged with the assessment of methods for evaluating the ‘*value” of diagnostic imag- ing. The consensus of this subgroup was that the meth- ods currently used in the radiological literature for assessing the value of diagnostic imaging are insuffi-

Index terms: Patient outcomes - Societal outcomes Physician outcomes * Diagnostic accuracy * Decision analysis

JlLRI I996 1:7-9

Abbreviations: MRI = magnetic resonance Imaging.

From MR Facilities (M.F.). Waukesha. WI: Value Health Services (G.G.). Fairfax, VA, the Food and Drug Administration (R.H.], Rockville, MD: University of Gron- ingen (M.G.M.H.). Groningen, the Netherlands; Cleveland Clinic Foundation (M.M.. M.T.M., N.0.1, Cleveland, OH; Aetna Health Plan (S.P.). Middletown, Cr; and New York Hospital-ComeU Medical Center (H.D.S.). New York, N.Y. Received July 17. 1995 accepted July 17. The ‘Value” Working Group includes: Mary Rtzpatrlck, BS. George Goldberg. MD, Robert Hirsch, PhD. Maria G. M. Hunlnk, MD, PhD. MaurIe Markman. MD, MichaelT. Modlc. MD, Nancy Obuchowski. PhD, Stephanie Plent. MD. and H. Dirk Sostman, MD. M d n ~ nprlnt requents to N.O.. Depart- ment of BiostaUstics and Epidemiology. Cleveland Clinic Foundation, Desk P88. 9500 Euclid Avenue, Cleveland, OH 44195-5196.

6 ISMRM, 1996

Aent. Specifically, studies of patient outcome (ie, late stages of the hierarchical model) focus on mortality and morbidity, and overlook the value of patient reassur- ance and reduction of stress and anxiety attributable to diagnostic information. The terms “outcomes” and ”val- ues” soon were applied synonomously. The specific aims of the group then were to (1) define outcomes of diagnostic imaging, (2) identi@ those outcomes that may be particularly difficult to quantify because of their intangible nature, and (3) propose methods to evaluate such outcomes. This report summarizes the conclu- sions of this group.

DEFINING OUTCOMES OF DIAGNOSTIC IMAGING Table 1 summarizes diagnostic imaging outcomes ac-

cording to whom they impact. Stratification of outcomes by population (ie, patient, physician, and society) is useful for several reasons. First, when attempting to measure these outcomes, this stratification clarifies the target population. Second, this scheme is useful for weighting the advantages of diagnostic information for one population against the disadvantages to another.

Diagnostic accuracy is probably the most widely measured intermediate outcome of diagnostic imaging. Others include the impact of diagnostic information on diagnostic thinking and patient management. These outcomes are labeled intermediate because they only indirectly impact the patient, physician, and/or society. For example, an altered management strategy impacts the patient only through its subsequent impact on the patient‘s mortality and morbidity.

Survival and quality of life are commonly measured pa- tient outcomes. Survival includes both short-term (survival of the diagnostic test) and long-term survival (suMval of the disease after the indicated treatment). Quality of life describes short- and long-term functioning and disability; it encompasses aspects of physical, mental, emotional, and social well-being. However, the information provided by a diagnostic test (ie, information per se) may be valu- able, independent of its effect on survival and quality of life. For example, a true-positive test result provides an explanation of symptoms and allows a patient to plan his/her life. Although theoretically this outcome should affect quality of life, currently used quality of life measures are unlikely to capture this nuance unless specifically de- signed to do so. There are other important patient out- comes associated with pure information: a true-negative result provides reassurance and reduction of stress; a false-negative result can lead not only to inappropriate treatment, but also to false reassurance and later regret; and a false-positive is associated with anxiety and dis-

7

utility from the work-up in addition to potential morbidity and mortality associated with inappropriate treatment. Customer satisfaction describes the patient‘s satisfaction with care received (or denied) and is often quantified by the frequency of changes in health plan or occurrences of litigation. Customer satisfaction may be closely tied to the value of information per se.

Physician outcomes are also defined by the value of the diagnostic information per se. For example, a true-posi- tive implies a “job well done,” a false-negative means a missed diagnosis with subsequent associated regret and possible legal action, a false-positive can lead to physi- cian anxiety, and a true-negative provides reassurance and avoidance of legal action. The downstream costs of various physician outcomes to the health care system and society are not well understood.

Societal outcomes include direct costs and resource utilization (immediate and downstream costs of tests and treatment and downstream costs of health-seeking be- havior), time costs (lost productivity), induced health care costs, and the value of information for understanding dis- ease and avoiding its spread.

Whereas patient outcomes of mortality and morbidity are commonly measured, alone they are incomplete in- dices of the full worth of diagnostic imaging. McNeil and Pauker (3) recognized the underestimation of the worth of diagnostic tests through the omission of outcomes, such as reduction of anxiety and self-perception. In ad- dition, they point out that negative factors related to downstream effects, such as false reassurance and in- appropriate labeling, are likely to be underestimated.

In assessing these intangible outcomes, it must be rec- ognized that they may violate the hierarchy of Fryback and Thornbury [ie, the patient can receive reassurance (level 5: patient efficacy) without an impact on patient management (level 4: therapeutic efficacy)].

ASSESSING OUTCOMES OF DIAGNOSTIC IMAGING

There are several examples in the literature whereby intangible patient outcomes have been assessed or in- corporated into decision analysis models. Two examples follow herein.

In 1981, Sox et al(4) described a study of 176 patients with nonspecific chest pain who were at very low risk for ischemic heart disease. The patients were randomly as- signed either to undergo routine testing or no testing. De- spite the fact that both groups received similar therapeu- tic care, patients in the testing group experienced a significant reduction in short-term disability and greater satisfaction with care, compared with the no testing group. The results of this study suggest that test results with no apparent diagnostic value can affect patient out- come.

In 1990, Mooney et a1 (5) applied decision-analytic methods to assess the efficacy of magnetic resonance im- aging (MRI) in suspected multiple sclerosis. The authors compared two situations: the expected utility and cost of immediate use of MRI vs a wait for further symptoms be- fore testing approach. Patient outcomes, such as the value of information in resolving uncertainty and the im- pact of being labeled with a diagnosis of MS, were as- signed utility scores and considered with outcomes such a s survival. Although the authors found little net benefit for immediate testing, they concluded that more work was needed to assess the utility of diagnostic uncertainty, because this variable was pivotal for decisions on the use of MlU.

Whereas randomized studies like Sox et al‘s can be used to assess the impact of diagnostic imaging on pa- tient outcome, such studies are costly and time-consum- ing. Alternatively, decision-analytic modeling can be applied. Such methods can provide a mechanism to con- solidate information from a variety of sources, and allow consideration of a variety of different paths and out- comes. In this fashion, one may have a logical means to analyze seemingly confounding or intangible factors to minimize the possibility of over- or undervaluing a diag- nostic test. Once formed, these models provide a template for future modifications as new information and/or treat- ments become available.

There are a number of tools available for assessing quality of life and intangible outcomes. Table 2 sum- marizes some of these methods. Health status meas- urements are often patient questionnaires assessing components of physical, emotional, mental, and social health. There are general and disease-specific forms. Several utility assessment tools exist; they are com- monly used to elicit patient preferences for various health states (6). The standard reference gamble is the- oretically the correct method for eliciting preferences but may require some modification for assessing the utility of reassurance and disutility of uncertainty. Other methods, specifically assessment of willingness to pay and resource utilization, may also be appropri- ate.

The suitability of the methods in Table 2 for assessing the value of diagnostic imaging needs to be addressed, along with several other pertinent questions. (1) Whose utility should be assessed when comparing different mental, physical, and emotional health states: sympto- matic patients or healthy representatives from society3 (2) What method is the most appropriate to elicit prefer- ences and values for intangible outcomes? (3) How can we avoid double counting (ie, outcomes such as anxiety may be captured in the quality of life measure)?

CONCLUSIONS 1. The process of defining outcomes for diagnostic im-

aging is complicated. Good data on the accuracy of the diagnostic test, the natural history of the suspected disease (including its prevalence, incidence, and be- havior over time), as well as the risks and benefits of available treatment options are required.

2. Reliance on survival rates and life expectancy leads to underestimation of the true worth of diagnostic im- aging. The full value of information needs to be meas- ured, but is easily overlooked in classical methods of assessment, such as randomized controlled trials. In contrast, decision modeling can be applied to deline- ate and quant@ all outcomes of the diagnostic pro- cess.

3. We cannot stand still. We must move, now, beyond measurement of the intermediate outcomes of diag- nostic imaging.

Acknowledgments: The views presented in this report are those of the authors and do not necessarily reflect the views of the Food and Drug Administration.

References 1. Arrow KL. Government decision making and the preciousness of

life. In: Tancredi L, ed: Papers of the Conference on Health Care and Changing Values. Washington, DC: Institute of Medicine, National Academy of Sciences, November 27-29, 1973; 33-47.

2. Fryback DG, Thornbury JR The efficacy of diagnostic imaging. Med Dec Mahing 1991; 11:88-94.

8 9 JMRl JanuarylFebruary 1996

3. McNeil BJ, Pauker SG. The patient’s role in assessing the value of diagnostic tests. Radiology 1979: 132:605-610.

4. Sox HC, Margulies I, Sox CH. Psychologically mediated effects of diagnostic tests. Ann Intern Med 1981: 95:680-685.

5. Mooney C. Mushl’in AI, Phelps CE. Targeting assessments of mag- netic resonance imaging in suspected multiple sclerosis. Med Dec

6. Keeney RL, Raiffa H. Decisions with multiple objectives: prefer- ences and value tradeoffs. New York, Ny: Wiley. 1976: 131-218.

Making 1990; 10:77-94.

Volume 6 Number 1 JMRl 9