Performance Reports

Andy Bindman MD

Department of Medicine, Epidemiology and Biostatistics

UCSF

Calculating an Individual’s Risk

Solve the multivariate model incorporating an individual’s specific characteristics

For continuous outcomes the predicted values are the expected values

For dichotomous outcomes the sum of the derived predictor variables produces a “logit” which can be algebraically converted to a probability (enat log odds /1 + enat log odds )

Aggregating to the group level Sum observed events (eg deaths) for sub-group

Sum expected probability of events for same

Probabilities used to calculate expected events derived from entire data set and applied to individuals in sub-group (eg defined by care site)

The overall expected number of events must equal the observed number of events but this need not be the case at the level of subgroups

Comparing observed and expected outcomes to assess quality

Observed events or rates of events Expected events or rates of events Better quality implied when observed is lower

than expected (worse quality when observed higher than expected)

Risk adjusted rates

Standardizes rates across sub-groups so that they can be directly compared with a single number

Observed rate/expected rate of subgroup

x overall observed rate

Observed CABG Mortality Rates, NY 1989-1992

1989 1990 1991 1992

Volume 12,269 13,946 14,944 16,028

Deaths 432 437 460 446

Observed Mortality,%

3.52 3.14 3.08 2.78

Calculating Expected CABG Mortality Rates in New York by Year

Pool all 4 years of CABG patients Develop risk adjustment model for CABG patients Apply risk adjustment model for CABG patients

sub-grouped by year to determine expected number of deaths for each year

Divide expected number of deaths by number of cases per year to get expected death rate

Observed and Expected CABG Mortality Rates, NY 1989-1992

1989 1990 1991 1992

Volume 12,269 13,946 14,944 16,028

Observed Mortality,%

3.52 3.14 3.08 2.78

Expected Mortality,%

2.62 2.97 3.16 3.54

Annual Risk Adjusted Mortality Rate for CABG in New York

Observed rate per year/expected rate per year

X average death rate over 4 year period (3.1)

Observed, Expected and Risk-Adjusted CABG Mortality Rates, NY 1989-1992

1989 1990 1991 1992

Volume 12,269 13,946 14,944 16,028

Observed Mortality,%

3.52 3.14 3.08 2.78

Expected Mortality,%

2.62 2.97 3.16 3.54

Risk adj Mortality,%

4.17 3.28 3.03 2.45

What Happened with CABG Surgery Over Time in New York?

Operated on sicker patients

Observed mortality rate declined over time

Risk adjusted mortality rate declined even more

Did quality of CABG care improve over time?

NY CABG Risk Adjustment Model

Well designed model

C index = .787; Hosmer-Lemeshow chi square p=.16

Mortality is not a subjective outcome- hard to fake

Gaming might be possible with coding some predictors

Interpreting Risk Adjusted CABG Outcomes

Public reporting on hospital CABG mortality began in 1989

Low volume hospitals had higher mortality rates and some stopped performing CABG over time

Process indicators of cardiac care (beta blocker post MI) also improved in NY hospitals over time

Hospitals documented more co-morbidities over time resulting in inflated expected death rates

Some sick NY cardiac patients operated on in NJ

Applying Results To Providers

Possible to aggregate observed and expected rates of events to hospital, physician, or some other provider level grouping

Statistical problems arise when total number of expected events are small

Minimum of five expected events per group as a rule of thumb

Naming Names

Assigning assessments of quality to specific providers increases the stakes

Need to demonstrate validity of analytic approach

Reporting Results

Public reporting vs internal quality improvement

Data users tend to want gradations of quality along a continuum (excellent to poor)

However denoting those within a 95% confidence interval of the expected as average is less sensitive to noise in data

Bootstrap Procedure:Deriving Confidence Intervals

Multiple (e.g. 1000) random samples of same size of original derived from original sample with replacement

Calculate expected rate for each “new” sample

Create frequency distribution of expected rates

Empirically derive 95% CI (950 of 1000 centered around mean)

Consistency in the evidence Differences between observed and expected may be due

to things other than ‘quality’

Are the results consistent over time

Are results consistent with prior expectations such as volume-outcome relationships

Confirmation through very different types of evidence is a major goal- “external validation”

Observed / Expected Mortality

Volume-Outcome

Relationship between high volume providers and better outcomes

Most often studied in relationship to procedures

Consistent with notion that practice makes perfect

Hospital Volume and CABG Mortality in California Hospitals Using Registry, 2000-02

Volume Hospitals

N (%)

Patients

N (%)

OR

(CI)

>=600 6 (7) 16,145 (28) 0.56

(0.40-0.79)

300-599 16 (19) 17,052 (30) 0.80

(0.63-1.02)

200-299 14 (17) 8,168 (14) 0.74

(0.57-0.97)

<200 47 (57) 16,022 (28) referent

External validation of data Link hospital discharge data with CABG registry data Looking for missing cases, deaths and highly predictive risk

factors– 221 cases in discharge data not reported to registry– 26 additional deaths (498 total)– 63 undercodes and 123 overcodes of cardiogenic shock – 29 overcodes of salvage (51 total)

Direct auditing – Deaths– Highly weighted predictors particularly if subjective

Growing Number of Quality Initiatives Provide Opportunity for Cross Comparisons

AHRQ Quality Indicators JCAHO ORYX Hospital Core Performance CMS Hospital Quality Alliance National Quality Forum (NQF) Leapfrog NCQA HEDIS

JCAHO - Hospital AMI indicators

Process elements:

Aspirin at arrival, Aspirin at discharge, ACEI for LVSD, Smoking cessation counseling, Beta-blocker at discharge, Beta-blocker at arrival, Thrombolytic within 30 minutes, PCI within 120 minutes

Who uses these reports and how Patients

– Slow to catch on

– More important for those without other ways to judge quality Managers

– Aim to improve quality to avoid “naming and shaming” Payers (health plans)

– Selective contracting

– Pay for performance

How much do these reports matter?

California has lowered isolated CABG mortality by ~1% (from 3% to 2 %) during public report period

Approximately 20,000 procedures per year Reduction from 600 to 400 deaths Average survival ~5 years Even if half the change is due to gaming, 500 life

years saved

My Reflections on Performance Reports

View the risk adjusted estimates as ‘yellow flags’, not ‘smoking guns’

Risk models will probably improve with ability to add more clinical data available through electronic records

Research may not need to be perfect to bring about public health benefits

Attempts to improve quality need to consider unintended consequences on access/efficiency

Outcomes Research Opportunities

Validate risk adjustment models for new conditions

Are health care outcomes changing over time and if so why?

How can performance reports on health outcomes be used to create better health care quality?