Assessing Performance: New Strategies

Robin Blackstone, MD, FACS, FASMBSPresident, ASMBS

MISS 7:30am Saturday February 25

Bariatric Surgical QualityOld Paradigm was a very good start……..now it is time to evolve

On February 6, 2012 the ASMBS Executive Council agreed to merge the BSCOE program with the American College of Surgeons(ACS) BCSN program.On February 10, the ACS Board of Regents agreed.

821 programs will merge April 1, 2012 based on the current volume standards.

Foundation of New Program

All current programs in either BCSN or BSCOE will be accredited in the new program under the current volume based standards

Content and Direction of the program will be through committees that are being established out of the current ASMBS and ACS bariatric committees

Joint (ACS/ASMBS/SAGES) Credentialing Recommendations will be made

A new outcomes based standard will evolve to replace the current volume based standards

The program will have a joint data registry

Robust Bariatric Quality Improvement on a National/Regional and State wide basis will be put in place

“Knowing is not enough, we must apply.

Willing is not enough, we must do.”

-Goethe

Mentor Leadership Data Registry

Quality Improveme

nt Collaborativ

es

Outcome Based

Accreditation Standards

Credentialing

Joint Task Force

Charge: To develop recommendations for facilities for credentialing of bariatric surgeons

reports.asmbs.org

“Knowing is not enough, we must apply.

Willing is not enough, we must do.”

-Goethe

Mentor Leadership Data Registry

Quality Improveme

nt Collaborativ

es

Outcome Based

Accreditation Standards

Credentialing

Community surgeons established a culture of reporting dataThe most important change that has resulted from the registry

10

Database launched in 2007; Entry of data into BOLD became a requirement for BSCOE designation in early 2008 Surgeons/surgical practices responsible for data entry

Source of initial data elements and definitions: Research Advisory Committee (RAC), Bariatric Surgery Review Committee (BSRC), SRC Research Department Staff Sought compatibility with NIH LABS data collection

Primary purpose was to monitor compliance with BSCOE requirements; Secondary purpose was for research Compliance monitoring focused only on surgical volume and

30-day serious complications (deaths, transfers, readmissions, revisions and reoperations)

Many questions asked in the database were intended for research

History of BOLDResearch Director SRC: Debbie Winegar

11

Initial IRB situation complex Informed consent required for all patients

IRB oversight required for all: Sites had option to waive oversight to central IRB or retain local IRB oversight

IRB situation simplified in 2010 with move to Copernicus Group IRB

Waiver of documentation of informed consent

Oversight of sites not required by central IRB

No specific qualifications required for data entry personnel Each program determined the appropriate skill set of

individual(s) who would enter their data

In 2010, requirement added that a BOLD Administrator be named for each practice

History of BOLD - 2

12

BSCOE participants questioned the value of data collection relative to the burden

BOLD reports not widely utilized

Felt that too many questions were research-orientedo Patient information/demographics – 13-19 questionso Preoperative visit – 38 questions o Hospital visit – 24 questionso Postoperative visit – 36 questions

A review of BOLD data elements launched in 2010 Surgeon experts proposed a significant reduction in the

number of core data elements Plan approved by BSRC and EC in 4Q2010 Improvements not executed

History of BOLD -3

13

Data self-reported by programs Potential for bias when data entered by key program personnel

with vested interest in outcomes

Heterogeneity of data entry personnel No specific qualifications (e.g., health/science background) have

been required for data entry personnel; each program has selected whomever they feel can fulfill the role

While this policy accommodates all types of program structures, it can lead to variations in data entry practices that can affect data quality

Inadequate training of data entry personnel and lack of data entry support materials Although basic data entry training has been accessible, neither a

software manual nor guidelines on how to enter specific cases have been provided to data entry personnel

Inadequate training allows room for self-interpretation of data definitions and leads to variations in data entry practices

Limitations of the Database

14

Non-standard data definitions The use of non-standard data definitions in BOLD (e.g., comorbidity

severity scale; certain complications) limits its utility for making comparisons with or for combining with data from with other registries

Lack of sensitivity of the comorbidity severity scale The severity scale captures improvement in many comorbidities

only when there is a major change in treatment regime (e.g., whether medications are required) unlike changes in standard measures (e.g., HbA1c)

Leads to potential underestimation of comorbidity resolution

Potential under-representation of complications Surgical programs have been responsible for entering all

complications in BOLD, even those managed by other health care providers

The likelihood is high that bariatric programs may not be aware of all events that have occurred thus, complications in BOLD are potentially under-represented

Limitations of the Database - 2

15

Variations in the time to data entry There has never been a required time frame with which data must be

entered into BOLD following a patient visit

Leads to the appearance of missing follow-up data at a given point in time

Frequency of on-site data auditing The current schedule for on-site data auditing (every three years as

part of site inspection) may not be sufficient to detect program-specific issues and to keep programs honest

Thoroughness of data auditing BOLD data auditing has been limited to surgeries performed during a

one-year period preceding the site inspection and has been focused on a subset of data elements: procedures, complications, readmission, reoperations and transfers

In general, ~10% of cases selected for random chart review

A 100% chart review triggered by unreported events

Limitations of the Database - 3

16

Inconsistent follow-up intervals among programs Necessitates the use of wide visit windows for the assessment of

outcomes

Incomplete follow-up data Lack of a follow-up tracking report in BOLD has been an

inconvenience for programs however incomplete follow-up is a common problem faced by all bariatric databases

Inadequate feedback to programs through BOLD reports- The BOLD reports provided to date have not shown a program’s

risk-adjusted outcomes vs national risk-adjusted benchmarks nor have they presented all program data in a useful format (e.g., comorbidity data)

- The usefulness of BOLD data for quality improvement has been limited

Limitations of the Database - 4

Potential settlement agreement will provide the data through March 31, 2012 to ASMBS for all programs (unless you specifically opt out)

All programs will receive a risk adjusted and reliability adjusted composite measure and comprehensive data report for each surgeon and program participating on June 22, 2012 at the annual meeting during the first Annual National Bariatric Quality Forum from 10am - 12 noon. All programs should make sure the bariatric coordinator and medical director of the program are there to participate in that meeting.

The data will not be migrated due to the limitations as noted in the previous slides

What will happen to the BOLD data in the transition

18

BOLD Data Dissemination Committees through Research Committee

Data Access Committee (DAC) reviewed requests for BOLD data

Through Sept 2011, 106 requests for data received, 30% for research, 17% for quality improvement

The majority of requests (58%) were received from BSCOE participants

Data Dissemination Committee (DDC) reviewed requests for publication of information derived from BOLD data analysis

Through Sept 2011, 20 abstracts/presentations and 6 manuscripts containing BOLD data were reviewed by the DDC

BOLD data provided for payor negotiations; Data on sleeve provided to CMS for NCD consideration

Dissemination of BOLD Data

19

Programs genuinely want to do the right thing in terms of data collection but they need clear information on what is expected

Data collection is expensive for everyone but is essential for quality assessment and improvement

Have a clear plan as to what the purpose of the data collection is and communicate that to program participants

Tie data collection to accreditation to emphasize its importance

Less is more - focus on fewer data points and do a better job with them

Proper training of data entry personnel is critical for obtaining high quality data

Feedback of information to programs demonstrates the value of the data collection effort and keeps them engaged

Lessons Learned

New Data Registry April 1 2012

Strong and specific data definitions

Parsimonious data set

No IRB required for CQI except as required locally

Data validation (digital oversight) ongoing through data safety monitoring board – site visited as needed to assure compliance

Collection methods – certified data entry person

MOC

Analytics and Reporting The Digital Camera Model vs. Film

“Knowing is not enough, we must apply.

Willing is not enough, we must do.”

-Goethe

Mentor Leadership Data Registry

Quality Improveme

nt Collaborativ

es

Outcome Based

Accreditation Standards

Credentialing

Volume and relationship to mortality

The mortality of bariatric surgery has declined dramatically from 2004 (o.2%) to 2009 (0.1%)

The primary driver of the decline was the wide spread adoption of laparoscopic surgery from 2004 (29.9%) to 2008 (90.2%)

The second biggest driver was the inclusion of a very low risk procedure (Adjustable Gastric Band) 2004 (1%) vs. 2008 (29%)

Not clear how much additional decrease in mortality was due to volume based accreditation

Failure to rescue patients may be of critical importance – emphasizing the importance of collaborative networks of care

Because of the current accreditation stands ASMBS does not have data on programs doing les than 125 cases per year (ACS data seems to indicate that down to 25 cases a year the outcomes are very similar, in a risk adjusted environment. Nguyen, J Am Coll Surg 2011;213:261–266

Lessons from LABS

Historic surgeon experience was not related to adverse event rate

After adjusting for patient risk, the effect of surgeon volume on outcomes for RYGP procedures in LABS showed that for each increase by 10-case per year in surgeon volume the rate of composite events improved by 10 percent.

The observed relationship between surgeon RYBP volume and CE rates was continuous, illustrating that there was no satisfactory level of annual case volume that could act as a threshold for surgeon credentialing within the BSCOE

No significant differences were observed in mortality between low and high volume surgeons.

Smith MK, Patterson E, Wahed AS, Belle SH, Bessler M, Courcoulas AP, Flum D, Halpin V, Mitchell JE, Pomp A, Pories WJ, Wolfe B. Relationship between surgeon volume and adverse outcomes after RYGP in Longitudinal Assessment of Bariatric Surgery (LABS) study. Surg Obes Rel Dis 2010; 6:118-125.

The Role of Risk Adjustment

Important because surgeons feel that if their outcomes are poor it is because they are operating on sicker patients

Challenges in Risk adjustment there is no one factor that has emerged to adjust risk Risk adjustment has changed with time and the data set The most significant predictor of risk in any report has

been the procedure itself The adverse event curve is “J” shaped In order to use data to do risk adjustment the data

definitions have to be crystal clear and the number of patients in follow up has to be high – pulmonary hypertension example

The challenge of Risk Adjustment

Reliability AdjustmentWhen data is unreliable

The “Shrinkage” Debate

• In traditional statistics, the observed rate is thought to best represent the truth (n outcomes/k trials)

• Bayesian statistics considers observed data in the context of prior information

• Empirical Bayes derives prior information from the data– e.g., the “best” guess is a rate

somewhere between the observed rate and the overall rate

• Accomplished using hierarchical modeling

Traditional ApproachShrink to the average mortality

0%

20%

Mo

rtal

ity

rate

(%

)

Mortality rates forhigh-risk surgery 10%

Mo

rtal

ity

rate

(%

)

15%

5%

Overall mean mortality rate

Adjusted for reliability

Observed mortality rates

Box size is proportional to the hospital caseload

Composite Measure Approach: Shrink to the mortality for volume group

0%

20%M

ort

alit

y ra

te (

%)

Mortality rates forhigh-risk surgery 10%

Mo

rtal

ity

rate

(%

)

15%

5%

Observed mortality rates

Low volume

Medium volume

High volume

Mortality rates

Composite mortality

Develop accreditation standards based on outcomes measuresDevelopment of a Bariatric Surgery Composite Measure

Promise of composite outcome measures

• Global indicators of performance—combine multiple domains of quality (structure, process, outcome) into a single quality score• Empirically weight input measures• Filter out statistical noise• Extract as much quality “signal” as possible

from existing data• Designed to optimally describe and forecast

hospital outcomes

Composite Measures are the best predictors of quality to use for accreditation

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

4.5

5.0

Hospital volume Serious complications Composite measure

3-star2-star1-star

Hospital rankings (2008-09)

Risk-adjusted serious complications

(2010)

3.3

2.72.4

3.4 3.43.0

3.2

4.0

4.6

Hospital volumeSerious

complicationsCompositemeasure

Odds Ratio (95% CI), 1-star vs. 3-star 0.85 (0.43-1.68) 1.56 (0.84-2.91) 1.99 (1.14 -3.47)

% VariationExplained 0% 28% 89%

MBSC Data

Volume still counts

Only counts as much as the data dictates it should

With the new program we will get data from the first case

If Volume is found to matter it can be added back based on the data

Key components: Data/structure and process set up at beginning Somewhere between 25-40 cases you will be able to

generate a composite outcomes score

Prediction of serious complication after RYGB using hospital volume from

2008-2009 (BOLD)

1 2 3 4 50.0

1.0

2.0

3.0

4.0

5.0

6.0

7.0

8.0

9.0

10.0

5.2 4.9 4.9 5.24.8

Hospital Volume Quintile (08-09)

Ris

k A

dju

sted

Seri

ou

s C

om

plica

tion

in

20

10

Surgeon volume

1 2 3 4 50.0%

2.0%

4.0%

6.0%

8.0%

10.0%

4.9%4.3% 4.6%

5.2%

4.0%

Surgeon Volume Quintile

Ris

k a

dju

sted

seri

ou

s co

m-

plica

tion

BOLD data through 2011

Prediction of serious complication after RYGB using composite measure

from 2008-2009 (BOLD)

1 2 3 4 50.0

1.0

2.0

3.0

4.0

5.0

6.0

7.0

8.0

9.0

10.0

2.1

3.4

5.25.8

9.1

Composite Measure Quintile (08-09)

Ris

k A

dju

ste

d S

eri

ou

s C

om

pli

-ca

tio

n i

n 2

01

0

Risk Adjusted/Reliability Adjusted

Broader composite measures?

Global composite measure

Quality of lifeFunctional outcomes

Perioperative safety

Long-term effectiveness

Patient-centered results

MorbidityReoperation

Weight lossResolution of comorbid diseases

reports.asmbs.org

“Knowing is not enough, we must apply.

Willing is not enough, we must do.”

-Goethe

Mentor Leadership Data

Registry

Quality Improvement Collaborative

s

Outcome Based

Accreditation Standards

Credentialing

Goal of the MBSQIP

Establish national/regional and state collaboratives to improve care Decrease morbidity by 50% over

five years Decrease readmissions,

reoperations Improve VALUE of metabolic

surgery by increasing safety, improving efficacy and decreasing cost of care

Share best practices

Collaboratives for Quality ImprovementWhy are they different?

Strategies for improving outcomes

Steer patients to the best hospitals

Improve care everywhere

Accreditation Programs

Pay for Performance

Public Reporting

Michigan Bariatric Surgery CollaborativeExample of the Quality Improvement Process that will be established

The Michigan program in regional collaborative improvement

Partnership between BCBSM, Michigan hospitals, and clinician scientists Pilot test with PCI in1998, broad implementation 2005-

6

$30 million annual investment from BCBSM

12 collaborative quality improvement programs PCI /PVI, Cardiac, NSQIP, bariatrics, breast cancer,

cardiac CT, trauma, joint replacement, and medical admissions

50+ hospitals 200,000+ pts / year

Collaborative quality improvement

Basic idea: Physicians/hospitals collaborate with and learn from each other in improving outcomes

Robust data registry with “digital” capability

Empirical and non-empirical identification of best practices Leveraging “natural experiments” associated with variation in

practice across hospitals and physicians Non-empirical learning

Performance feedback, collaborative meetings, site visits, etc.

Empirical identification and dissemination of best practices Process-outcomes linkage, leveraging “natural experiments” associated

with variation in practice Guideline implementation and evaluation

Health Affairs, April, 2011

Michigan Bariatric Surgery Collaborative (MBSC)

29 hospitals

65 surgeons

7,000 pts / yr

Trends in VTE Guideline Adherence

2006 2007 2008 2009 2010 20110%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Intervention

*Based on random site audit of 1,148 charts

Trends in VTE Rates

QI intervention

Will collaborative quality improvement reduce costs, or just save lives?

Inferior vena cava (IVC) Filters

Aim to prevent fatal pulmonary embolism after surgery

Used commonly in bariatric surgery (10% in Michigan)

Effectiveness unclear

Total BCBSM payments with gastric bypass (2006)

$32,008

$45,559

Variation in the use of IVC filters before gastric bypass

Low use hospitals

High use hospitals

Complications in gastric bypass patients with and without IVC filters

OR=1.3 (0.5-3.2)

OR=1.4 (0.9-2.2)

OR=2.5 (1.0-6.3)

Birkmeyer NJO et al., Ann Surg, July 2010

Over half of deaths and permanent disability directly attributable to the filter itself

Trends in the Use of IVC Filters

Q1 20

07

Q3 20

07

Q1 20

08

Q3 20

08

Q1 20

09

Q3 20

09

Q1 20

10

Q3 20

10

Q1 20

11

Q3 20

110

2

4

6

8

10

Time Period

Perc

en

t

IVC filter intervention

Data first presented at Collaborative Meeting

Net savings to BCBSM

$2.6 million / yr.

The Payor CommunityImprove Access to Care by giving VALUE (COST/QUALITY) back to payors

Quality Events at ASMBS Annual Meeting

Sunday, June 17 ASMBS Presents Quality Improvement Workshop, Nuts and Bolts

(free to all registered attendees)

Monday, June 18 National Bariatric Surgery Collaborative; The Next Level of

Excellence

Wednesday, June 19 ASMBS Town Hall Meeting; New Direction for ASMBS

Thursday, June 20 Quality in Bariatric Surgery Mason Lecture, John Birkmeyer, MD

Friday, June 21 ASMBS National Collaborative Process Improvement Initiative

Thank you