Edspt

orattciimgw

ci

a

sb

Rp

5

Turf Wars in Radiology: The Qualityof Interpretations of Imaging Studies

by Nonradiologist Physicians—A Patient Safety Issue?

David C. Levin, MDa,b, Vijay M. Rao, MDa

Previous publications have demonstrated that in side-by-side testing, radiologists clearly outperform nonradi-ologist physicians at interpreting plain radiographs. Other articles, mostly in the nonradiologic literature, haveshown that in actual practice settings, nonradiologist physicians make frequent errors in image interpretation.On the other hand, several other studies have concluded that such errors are infrequent and generally do notaffect patient outcomes. However, most of the latter studies have had methodologic flaws. Radiologists need tobe aware of the evidence that exists in the medical literature on errors made by nonradiologists in imageinterpretation and to bring it to the attention of hospital authorities or medical policy–making groups that arecharged with improving patient safety.

Key Words: Self-referral, medical economics, diagnostic radiology, departmental management, socioeco-nomic issues

J Am Coll Radiol 2004;1:506-509. Copyright © 2004 American College of Radiology

TPswpdbrfm(ocgwtccdTcat

uT

ditor’s note: This is the fourth in a series of articles by the same authorsealing with aspects of self-referral. The intent is that by the end of theeries, the reader should have a comprehensive view of this importanthenomenon. I have allowed the authors considerable latitude to expressheir own views.

In the last two articles in this series, we discussed theverutilization of diagnostic imaging resulting from self-eferral and other causes. Overutilization is a troublingnd seemingly inevitable consequence of self-referral, buthere is another aspect that must be considered as well:he quality of patient care. When nonradiologist physi-ians self-refer their patients for diagnostic imaging andnterpret those studies themselves, how accurate are theirnterpretations? This article presents evidence from the

edical literature showing that compared with radiolo-ists, physicians in other disciplines generally do poorlyhen they try to interpret imaging studies.The best way to make such an analysis is a side-by-side

omparison of the performance of the two groups innterpreting a test set of images with known diagnoses.

Department of Radiology, Thomas Jefferson University Hospital and Jeffer-on Medical College, Philadelphia, Pennsylvania.

HealthHelp Networks, Inc., Houston, Texas.

Corresponding author and reprints: David C. Levin, MD, Department ofadiology, Gibbon 3390, Thomas Jefferson University Hospital, Philadel-

fhia, PA 19107; e-mail: david.levin@mail.tju.edu.

06

here are only two such studies of which we are aware.otchen et al. [1] tested three physician panels using atandardized set of 60 chest radiographs, half of whichere normal and half of which contained lesions such asulmonary infiltrates, pneumothoraces, masses, and car-iac abnormalities. The first panel consisted of 111oard-certified radiologists, the second of 29 radiologyesidents, and the third of 22 nonradiologist physiciansrom private practice and academic medical groups. Theedical specialties of the latter panel were family practice

n � 6), emergency medicine (n � 3), radiation oncol-gy (n � 6), pulmonology (n � 4), and internal medi-ine (n � 3). Each participant interpreted all 60 radio-raphs. Receiver operator characteristic (ROC) curvesere calculated for each participant and for each of the

hree panels as a whole. The mean areas under the ROCurves for the three panels were 0.860 for the 111 board-ertified radiologists, 0.746 for the 29 radiology resi-ents, and 0.657 for the 22 nonradiologist physicians.he differences between the three panels were statisti-

ally significant. Thus, even the radiology residents (withmean training length of 2.4 years) performed better

han the nonradiologists.A similar study was conducted by Eng et al. [2]. They

sed four physician panels at an academic medical center.he panels consisted of (1) four faculty radiologists, (2)

our radiology residents, (3) four emergency medicine

© 2004 American College of Radiology0091-2182/04/$30.00 ● DOI 10.1016/j.jacr.2004.01.005

fdewncncaet00[an0awgg

ssafgmsg

ptpcssdstitqa(gfac3c2ir

tdMcpqwrwiidiioEttetcbaoifriistpuwpirhsbb

posleEshbtt

Levin, Rao/Turf Wars in Radiology 507

aculty members, and (4) four emergency medicine resi-ents. Panelists interpreted 120 plain radiographs (skel-tal, chest, and abdominal), approximately half of whichere normal and half of which contained clinically sig-ificant abnormalities, such as fractures, pneumothora-es, pulmonary infiltrates or masses, pneumoperito-eum, or small bowel obstructions. Receiver operatorharacteristic curves were constructed for the four panels,nd the findings proved quite similar to those of Potchent al. [1]. The mean areas under the curves were 0.85 forhe faculty radiologists, 0.78 for the radiology residents,.70 for the emergency medicine faculty members, and.65 for the emergency medicine residents. As Eng et al.2] pointed out, an area of 1.0 under the ROC curvemounts to a perfect diagnostic performance (almostever achievable in this kind of test), whereas an area of.50 is the equivalent of a random guess. The overallccuracy of the faculty radiologists was 80%, comparedith 71% for the radiology residents, 59% for the emer-ency medicine faculty members, and 57% for the emer-ency medicine residents.

Both the aforementioned studies demonstrate that inide-by-side comparisons of interpretive abilities usingtandardized sets of images, board-certified radiologistsnd even radiology residents will substantially outper-orm nonradiologist physicians. And because plain radio-raphs are considered less complex than computed to-ography (CT), magnetic resonance imaging, or

onography, it is likely that the differences would be evenreater if these studies had been included [2].

A number of other articles have been published com-aring the initial interpretations of imaging examina-ions by nonradiologist physicians with subsequent inter-retations by radiologists. These studies, performed inlinical settings rather than test situations, generally con-idered the radiologists’ interpretations to be the “goldtandard” and had little long-term clinical follow-up toetermine ultimate accuracy. Although some might con-ider this a drawback, it is a reasonable approach, givenhe findings of Potchen et al. [1] and Eng et al. [2] thatnterpretations by radiologists are far more likely to behe correct ones. Most of the articles demonstrated fre-uent errors by nonradiologists. For example, Vincent etl. [3] assessed the accuracy of emergency departmentED) house staff members in interpreting plain radio-raphs. Among 6448 plain radiography studies per-ormed in an ED over an 8-month period, 505 hadbnormalities, and 200 of these were considered clini-ally significant. The ED house staff members missed5% of all 505 positive findings and 39% of the 200linically significant positive findings. Snow [4] analyzed48 plain radiography cases in an ED that were firstnterpreted by medical house staff members, then by

adiologists. Discrepancies between the two interpreta- e

ions occurred in 114 (46%) cases, and in 28 (11%), theiscrepancies were judged to be clinically significant.cLain and Kirkwood [5] compared readings of 294

onsecutive plain radiography studies by primary carehysicians (PCPs) in a small, rural hospital with subse-uent readings by consulting radiologists who came thereeekly. The PCPs’ readings differed from those of the

adiologists in 27 (9.2%) cases, and 24 (8.2%) of themere determined to be clinically significant. In a review of

nterpretations of 1471 plain radiography examinationsn a pediatric ED by pediatric ED specialists and resi-ents, Walsh-Kelly et al. [6] found that 14% were mis-

nterpreted. A study by Nolan et al. [7] evaluated thenterpretations by pediatric residents of 1411 plain radi-graphy examinations performed in conjunction withD or pediatric outpatient clinic visits. Compared with

he formal reports by radiologists, which were consideredhe correct interpretations, the pediatric residents maderrors in 16% (11% false positives and 5% false nega-ives). Gatt et al. [8] compared interpretations of 509hest radiographs by a group of ED physicians with thosey radiologists. The sensitivities for detecting differentbnormalities among the ED physicians ranged fromnly 20% to 65%, depending on the type of abnormal-ty; that is, many abnormalities were missed, resulting inalse-negative interpretations. Specificities of the groupanged from 95% to 99%, indicating that there was a lowncidence of false positives. In an ED setting, sensitivityn x-ray interpretation is a more important measure thanpecificity. Alfaro et al. [9] reported on the interpreta-ions of cranial CT examinations by ED physicians com-ared with those by radiologists. Among 555 patientsndergoing cranial CT during evaluation in an ED, thereere clinically significant misinterpretations by the EDhysicians in 24%. Among the missed findings were crit-cal lesions such as infarcts, masses, cerebral edema, pa-enchymal hemorrhages, contusions, and subarachnoidemorrhages. As a group, these studies and others havehown that the interpretation of imaging examinationsy nonradiologist physicians results in a substantial num-er of errors.It is noteworthy that all the studies alluded to in the

receding paragraph were published in medical journalsf specialties other than radiology. There have also beeneveral publications addressing this issue in the radiologyiterature. Reinus [10] reported on 953 plain radiographyxaminations taken in an ED that were first evaluated byD physicians, then subsequently by radiologists. His

tudy was somewhat different in that the ED physiciansad the option of requesting immediate interpretationsy radiologists rather than trying to interpret the filmshemselves. They did so in 11% of the cases (presumablyhe more difficult ones). Among the remaining, suppos-

dly easier studies, the ED physicians still made clinically

s[forfi0

bpra[ioitrtnadc[wTspattawoyhvtw

gPtetwjotwopb

r2mstttcoissplat

R

1

1

1

1

1

508 Journal of the American College of Radiology/Vol. 1 No. 7 July 2004

ignificant errors in 7%. In another study, Hopper et al.11] conducted a quality audit of chest radiographs per-ormed and interpreted by internists and other PCPs andf other chest radiographs performed and interpreted byadiologists. They demonstrated serious errors in 7% oflms interpreted by the nonradiologists compared with% by radiologists.Other research studies have used similar methodology

ut instead concluded that nonradiologists (usually EDhysicians or PCPs) make few errors in interpreting plainadiographs and that even when they do, it rarely has andverse impact on the clinical management of patients12-16]. However, these articles generally are character-zed by two methodologic flaws. First, from 6% to 68%f the films covered by the period of the studies were notnterpreted by nonradiologist physicians. This could leado selection bias wherein nonradiologists refrain fromeading the more difficult cases for fear of making mis-akes. Second, and perhaps even more important, theumbers of examinations whose results were negativend positive were not indicated; that is, the prevalence ofisease is unknown. To see how the prevalence of diseasean affect the results, we review the data of Vincent et al.3] alluded to earlier, in which the numbers of studieshose results were negative and positive were specified.heir study covered plain radiographs in 6448 patients

een in an ED in London, England, over an 8-montheriod. Of these, 505 (8%) showed some abnormalities,nd 200 (3%) showed clinically significant abnormali-ies. With this low prevalence of significant abnormali-ies, an untrained reader could adopt a policy of callingll examination results negative, except perhaps for thoseith blatantly obvious positive findings. In the detectionf clinically significant abnormalities, this policy wouldield an accuracy rate of at least 97% and probably asigh as 98% to 99%. A result such as this might soundery commendable at first glance, yet it would mean thathe relatively few patients with serious diseases or injuriesould often be misdiagnosed.Of course, it must also be acknowledged that radiolo-

ists, like all physicians, make mistakes. The studies ofotchen et al. [1] and Eng et al. [2] clearly demonstratedhat. The nature and causes of radiologic errors have beenxtensively studied [17-25] and are beyond the scope ofhis discussion. Nevertheless, it is clear that radiologists,ho undergo 4 to 6 years of intensive training, do better

obs of interpreting imaging studies than physicians inther disciplines. This statement is not meant to impugnhe abilities of those other physicians; just as radiologistsould do poor jobs if they attempted to do brain surgeryr deliver babies, other physicians cannot be expected toerform well at interpreting imaging studies without the

enefit of adequate training.

Since the publication of the Institute of Medicine’seport To Err is Human: Building a Safer Health System in000 [26], a great deal of attention has been focused onedical error reduction and ways to improve patient

afety. It is unfortunate that this effort has largely ignoredhe question of who should interpret imaging studies. Inhis paper, we have presented incontrovertible evidencehat interpretations of imaging studies by radiologists areonsiderably more accurate than those by physicians inther disciplines, who receive far less formal training inmaging (or in many cases none at all). Radiologistshould bring this evidence to the attention of hospitalafety officers, third-party insurance carriers, large em-loyer groups, and other policy makers in their practice

ocales and point out that a beneficial and easily achiev-ble way of reducing medical errors would be to requirehat all imaging studies be interpreted by radiologists.

EFERENCES

1. Potchen EJ, Cooper TG, Sierra AE, et al. Measuring performance in chestradiography. Radiology 2000;217:456-9.

2. Eng J, Mysko WK, Weller GER, et al. Interpretation of emergencydepartment radiographs: a comparison of emergency medicine physicianswith radiologists, residents with faculty, and film with digital display.Am J Roentgenol 2000;175:1233-8.

3. Vincent CA, Driscoll PA, Audley RJ, Grant DS. Accuracy of detection of radio-graphic abnormalities by junior doctors. Arch Emerg Med 1988;5:101-9.

4. Snow DA. Clinical significance of discrepancies in roentgenographic filminterpretation in an acute walk-in area. J Gen Intern Med 1986;1:295-9.

5. McLain PL, Kirkwood CR. The quality of emergency room radiographinterpretations. J Fam Practice 1985;20:443-8.

6. Walsh-Kelly CM, Melzer-Lange MD, Hennes HM, et al. Clinical impactof radiograph misinterpretation and the effect of physician training level.Am J Emerg Med 1995;13:262-4.

7. Nolan TM, Oberklaid F, Boldt D. Radiological services in a hospitalemergency department—an evaluation of service delivery and radiographinterpretation. Aust Paediatr J 1984;20:109-12.

8. Gatt ME, Spectre G, Paltiel O, Hiller N, Stalnikowicz R. Chest radio-graphs in the emergency department: is the radiologist really necessary?Postgrad Med J 2003;79:214-7.

9. Alfaro D, Levitt MA, English DK, Williams V, Eisenberg R. Accuracy ofinterpretation of cranial computed tomography scans in an emergencymedicine residency program. Ann Emerg Med 1995;25:169-74.

0. Reinus WR. Emergency physician error rates for interpretation of plainradiographs and utilization of radiologist consultation. Emerg Radiol1995;2:207-13.

1. Hopper KD, Rosetti GF, Edmiston RB, et al. Diagnostic radiology peerreview: a method inclusive of all interpreters of radiographic examinationsregardless of specialty. Radiology 1991;180:557-61.

2. O’Leary MR, Smith MS, O’Leary DS, et al. Application of clinical indi-cators in the emergency department. JAMA 1989;262:3444-7.

3. Brunswick JE, Ilkhanipour K, Seaberg DC, McGill L. Radiographic inter-pretation in the emergency department. Am J Emerg Med 1996;14:346-8.

4. Klein EJ, Koenig M, Diekema DS, Winters W. Discordant radiographinterpretation between emergency physicians and radiologists in a pedi-

atric emergency department. Ped Emerg Care 1999;15:245-8.

1

1

1

1

1

2

2

2

2

2

2

2

Levin, Rao/Turf Wars in Radiology 509

5. Halvorsen JG, Kunian A, Gjerdingen D, et al. The interpretation of officeradiographs by family physicians. J Fam Practice 1989;28:426-32.

6. Warren JS, Lara K, Connor PD, Cantrell J, Hahn RG. Correlation ofemergency department radiographs: results of a quality assurance reviewin an urban community hospital setting. J Am Board Fam Pract 1993;6:255-9.

7. Herman PG, Gerson DE, Hessel SJ, Mayer BS, Watnick M, Blesser B,Ozonoff D. Disagreements in chest roentgen interpretation. Chest 1975;68:278-82.

8. Swensson RG, Hessel SJ, Herman PG. Omissions in radiology: faultysearch or stringent reporting criteria? Radiology 1977;123:563-7.

9. Renfrew DL, Franken EA Jr, Berbaum KS, Weigelt FH, Abu-YousefMM. Error in radiology: classification and lessons in 182 cases presentedat a problem case conference. Radiology 1992;183:145-50.

0. Berlin L. Malpractice issues in radiology. The importance of proper radio-graphic positioning and technique. Am J Roentgenol 1996;166:769-71.

1. Berlin L. Malpractice issues in radiology. Perceptual errors. Am J Roent-genol 1996;167:587-90.

2. Berlin L. Malpractice issues in radiology. Errors in judgement. Am JRoentgenol 1996;166:1259-61.

3. Berlin L. Malpractice issues in radiology. Possessing ordinary knowledge.Am J Roentgenol 1996;166:1027-9.

4. Robinson PJA. Radiology’s Achilles heel: error and variation in the inter-pretation of the Roentgen image. Brit J Radiol 1997;70:1085-98.

5. Fitzgerald R. Error in radiology. Clin Radiol 2001;56:938-46.

6. Kohn LT, Corrigan JM, Donaldson MS. To err is human: building a saferhealth system. Washington (DC): National Academy Press; 2000.