ARTICLE IN PRESS

Evidence-Based Healthcare & Public Health (2005) 9, 20–22

KEYWORDHospital qMortality rQuality indSurgery;Cohort stu

1744-2249/$ - sdoi:10.1016/j.e

$Abstractedsample size. JA

www.elsevier.com/locate/ebhph

EVIDENCE-BASED HEALTH POLICY

Surgical death rates are not a reliable measure ofhospital quality$

Julie Bruce, BSc MSc PhD

Research Fellow in Epidemiology Department of Public Health University of Aberdeen, UK

Suality;ate;icators;

dy

ee front matter & 2005hbc.2004.11.011

from: Dimick JB, WelchMA 2004; 292: 847-851

Question: Are enough surgical procedures performed to use surgical mortality ratesas a reliable measure of hospital quality?

Study design: Cohort study

Main results: In a 3-year observation period, 90% of hospitals performing coronaryartery bypass graft (CABG) surgery exceeded the minimum number of cases neededto reliably detect poor performance (see Results table). However, only a smallproportion of hospitals exceeded the minimum number of cases for craniotomy,paediatric heart surgery, repair of abdominal anuerysum, pancreatic resection,oesophageal resection and hip replacement. Altering the definition of poorperformance from twice to 1.5 times the national average death rate for eachprocedure significantly reduced the number of hospitals that exceeded the minimumnumber of cases (data not shown). When the observation period was increased to 5years, CABG surgery was still the only procedure that exceeded the minimumnumber of cases (data not shown).

Authors’ conclusions: Mortality from CABG surgery may be a useful measure ofhospital quality. Craniotomy, paediatric heart surgery, repair of abdominalaneurysm, pancreatic resection, oesophageal resection or hip replacement are notperformed frequently enough and an alternative quality indicator for theseprocedures should be found.& 2005 Elsevier Ltd. All rights reserved.

Further details

Setting

994 hospitals, USA.

Elsevier Ltd. All rights reserv

HG, Birkmeyer JD. Surgical mo.

ed.

rtality as an indicator of hospital quality. The problem with small

ARTICLE IN PRESS

Surgical death rates are not a reliable measure of hospital quality 21

Results table

Hospitals meeting or exceeding minimum caseloads in a 3-year period.*

Surgery

National average deathrate (%)

Minimum number ofcases

Proportion of hospitals(%)

CABG surgery

3.5 219 90 Craniotomy 10.7 64 33 Hip replacement 0.3 2668 o1 Oesophageal resection 9.1 77 1 Paediatric heart surgery 5.4 138 25 Pancreatic resection 8.3 86 2 Repair of abdominalaneurysm

3.9

195 8

*Poor performance defined as twice the national average death rate from that procedure.

Participants

People undergoing coronary bypass graft surgery, craniotomy, paediatric heart surgery, repair of abdominalanuerysum, pancreatic resection, oesophageal resection or hip replacement.

Analysis

The national average death rate for each of the seven surgical procedures was calculated using data fromthe 2000 Nationwide Inpatient Sample. Sample size calculations were used to determine the minimumnumber of cases needed to detect increased deaths from each procedure in poor performing hospitals. Thenumber of hospitals that met or exceeded the minimum number of cases was then identified. Poor hospitalperformance was defined as twice the national average death rate for each procedure. Sensitivity analyseswere carried out for a period of 1, 3 or 5 years. Repeat analysis was carried out with poor performancedefined as 1.5 times the national average death rate for each procedure.

Main outcomes

Proportion of hospitals exceeding the minimum caseload for each operation; death from surgery duringhospital stay.

Notes

Hospitals chosen were intended to represent all hospitals in the US. The seven surgical procedures used inthis study are Agency for Healthcare Research and Quality Inpatient Quality Indicators.

Sources of funding: Research Enhancement Award from the Department of Veterans Affairs, USA.

Abstract provided by Bazian Ltd, London

Commentary

Performance monitoring, by comparing individualor institutional rates to a standard or nationalaverage, is at the cornerstone of efforts to measurequality and efficiency of clinical care. Surgicalmortality rates, either in-hospital mortality or 30-day mortality after surgery, are widely used as

indicators of surgical quality and hospital perfor-mance. Cardiothoracic surgeons were the first to bepublicly judged on postoperative mortality rates,spurring the development of risk-adjustment in-dices.

Dimick et al. analysed data from the NationwideInpatient Sample, representing 20% of all acutecare hospitalisations in the USA, to identify the

ARTICLE IN PRESS

J. Bruce22

minimum sample size required for detection ofincreases in mortality after seven surgical proce-dures, advocated by the AHRQ as quality indicators.Other than for coronary artery bypass surgery(CABG), less than half of hospitals met theminimum sample size required to detect a doublingof the national average mortality rate for eachsurgical procedure. Despite sensitivity analysis byreducing the benchmark rate and aggregating dataover 3 and 5-year periods to account for smallsample size and random variation, the majority ofhospitals failed to reach the minimum caseloadthreshold. Minimum caseload required is based onhospital operative volume and baseline event rate;in this series a minimum annual hospital caseload of2668 hip replacements was required to detect adoubling of the 0.3% national mortality rate.

This paper is one of the first to investigatesample size requirements for quality measurementin surgery. In-hospital death is an objective out-come, formally registered and recorded withinnational administrative systems but criticised forbeing an incomplete measure. Thirty-day mortalityis a more comprehensive measure, although effortsin tracing deaths after hospital discharge are knownto vary among monitoring systems.1 However,death is a rare event after some surgical proce-dures, presenting difficulties for measurement,analysis and interpretation of data. Debate con-tinues over the most appropriate statistical methodto detect increases in surgical mortality and whatlevels of departure from national average or otherbenchmark are deemed acceptable.2 The AHRQrecommend measurement of in-patient surgicalmortality for selected procedures to highlightdeficiencies in quality of healthcare provision, butDimick et al. provide clear evidence that smallsample sizes and low event frequency invalidatethe use of mortality for non-CABG procedures.

Clinical indicators used for performance monitor-ing should have the statistical power to exhibit oridentify change within the intended timescale.3

Aggregated rates may be required to explorevariations in surgical practice, but combining ratesfrom heterogeneous procedures could mask under-lying sub-trends. If risk-adjusted procedure-specificrates are used, aggregated data must be achievedwithin a clinically relevant time period to be usefulfor monitoring performance. The evidence fromthis study suggests that other clinically relevantindicators should be considered, such as morbid-ities, surgical adverse events, process measures orother indirect markers of healthcare. Morbiditiesoccur more frequently, are known to be sensitive tovariations in patient care and are considered directconsequences of healthcare.

The main implication from this study is that largeproportions of hospitals adhering to quality mea-surement guidelines may be falsely reassured ofgood surgical practice when they have failed todetect an effect. In-hospital mortality may be a validoutcome indicator for CABG surgery but variation inmortality rates after other procedures should beinterpreted with caution particularly if used withinthe context of performance monitoring.

References

1. Russell EM, Bruce J, Krukowski ZH. The quality of surgicalmortality monitoring: a systematic review. Br J Surg 2003;90:527–32.

2. Poloniecki J, Sismanidis C, Bland M, Jones P. Retrospectivecohort of false alarm rates associated with a series of heartoperations: the case for hospital mortality monitoring groups.BMJ 2004;328:375–80.

3. Royal Statistical Society Working Party on PerformanceMonitoring in the Public Services. Performance indicators:good, bad and ugly (2003). Available at http://www.rss.org.uk/archive/performance/.