Are you good enough?

Ryutaro Hirose MD

UCSF Transplant QI committee, ChairUCSF Dept of Surgery QI committeeUCSF Clinical performance improvement committee, ChairUCSF Patient Safety CommitteeFormer Chair, ASTS Standards and Quality committee

How do you know if you’re good?

• Look in the mirror– You ask people– Pt satisfaction surveys – important for CMS…but correlate with

quality??• CGCAHPS, HCAHPS

• National rankings• You don’t make errors – pt safety

– You don’t make errors that hurt people or kill people– Failure Mode Effect Analysis (FMEA)– RCA

• Processes and Outcomes– Benchmarks and comparators

Patients tell you that you are good?

Archives of Internal Medicine, Fenton et al 2012

Patients with highest satisfaction had increased mortality, worse outcomes

Comparison of Four National Rating Systems

National Hospital Rankings• US News and World Report, HealthGrades, Leapfrog

group, Consumer Reports• Also CMS Hospital Compare• NO hospital was ranked a high performer by all 4

rankings• ONLY 10% of hospitals (844) ranked high performer

by one ranking was ranked high performer by another

• Different rating methods, different focus, stresses different measures of quality

Quality Assessment and Performance Improvement

• Institute of Medicine– To Err is Human• Released 2000• Focused attention on medical errors, preventable

deaths• Systems approach to patient safety• Improving processes, not blaming individuals• Individuals must be held accountable and held

responsible

– Crossing the Quality Chasm

How do you judge quality? Team metric?

• 2014 – 2015 U of Kentucky Men’s Basketball• Overall 34-0• #1 Seed in NCAA tournament• National Championship?

Are you good enough?

• New England Patriots (2014-2015)• Super bowl championships– 2001, 2003, 2004, 2014

Are you good enough

• Individual provider metrics are confused with team metrics

• Baseball –– Pitchers are judged by W-L record– More appropriate metrics• ERA• WHIP

Individual metrics

• Michael Jordan– 6 time NBA champion– 10 x scoring title, PPG: 30.2 (career)

• Tom Brady– 4 Superbowl rings– 63.5 completion %, 53,258 yrds, 392 TDs, 143

INTs

Effect of environment on quality

• Domains of quality– Patient safety/safe care– Practice consistent with current medical

knowledge– Customization, ability to meet customer-specific

values• External forces to drive quality– Regulatory/legislative pressures– Economic and other incentives

Professional societies

• Set norms, standards of practice• Expectation that delivery of safe, high quality

care is standard • Promote culture of safety and improvement

Lapses in quality

• Misuse - Avoidable complications• Overuse – excess provision of service, not

supported by evidence• Underuse - failure to provide a service that

would have provided a favorable outcome

• First addressed by pt safety initiatives• Second and third addressed by evidence based

practice.

Quality Assurance and Performance Improvement

• QAPI programs– Mandated by CMS– Industry models– PDSA (plan-do-study-act)– Six Sigma (DMAIC)

Define/Measure/Analyze/Improve/Control– Lean Six Sigma

• LEAN – preserving value with less work• Waste reduction, increase efficiency, improve work flow• Production time/costs reduced

Cycle of continuous improvementPlan – Do – Study – Act

• Plan – Identify goal/purpose, define metrics of success

• Do– Implement components of plan

• Study– Monitor outcomes, test validity – ID areas for improvement

• Act– Close cycle, adjust goal– Change methods, reformulate theory

Six Sigma/Total Quality Management

• Seeks to improve qulaity by identifying and removing errors and minimizing variability

• DMAIC– Define, Measure, Analyze, Improve, Control

• Combined with lean manufacturing• LEAN Six Sigma– Address flow and waste

QAPI programs

• Mandated to have process AND outcome measures – for three phases

• Pre transplant• Transplant• Post transplant

18

How is quality measured at liver transplant centers

• Mandated data collection and submission to UNOS• Transplant Centers– Pretransplant – Transplant Candidate Registration form– Transplant – Transplant Recipient Registration form– Post transplant – Transplant Followup forms

• Organ Procurement Organizations– Deceased donor data– Performance data

What do we do now?

SRTR PSR’s– Waitlist mortality– Transplant rates– 1 yr and 3 yr patient and graft survival

• Used by UNOS/OPTN to flag programs, used by CMS as well

Center specific results

• Candidates– ‘shot selection’

• Referring physicians• Nephs, hepatologists• Selection committee/selection

critieria• Surgeon

• Donors– Donor demographics– Donor management

• Affects outcome

– Donor selection• In general• For specifc recipient

• Donor/recip interactions– Matching– Size

• Operative course– Redo, blood loss– Technical performance– Anesthesia

• Post op management– ICU– Surgeons/medical– ID

Variation in outcomes

• UNOS/MPSC• CMS– National coverage decisions e.g.• Liver - 1991 - if meets survival minimum

– (1-Yr: 77%, 2-yr: 60%)

– Two consecutive reporting periods of worse than expected outcomes

– O-E > 3.0; O/E ratio of events > 1.5, p<0.05– Results in review, possible SIA

Center specific results

• Integrated signals– We play a team sport

• Wait list mortality– Referral patterns

• Who are referred to us

– Selection criteria/behavior• Who are listed

– Wait list management• PCPs, referring nephs, heps• De-listing• Workup• Accurate documentation of comorbidities

26

Scientific Registry of Transplant Recipients

• Current Contract is held by the Chronic Disease Research Group of the Minneapolis Medical Research Foundation (Used to be held by the U of Michigan group)

• Responsible for designing and carrying out rigorous scientific analyses of data and disseminating information to the transplant community – including– transplant programs– organ procurement organizations– policy makers– transplant professionals– transplant recipients– organ donors and donor families– and the general public.

27

SRTR roles

• Policy development – evidence based• Collaborative efforts between the transplant

community, the SRTR, and the OPTN. • Policy-making is the OPTN's responsibility, the SRTR

plays a critical role in policy development through ongoing data analyses designed to provide policy makers with the information necessary to make informed decisions– LSAM modeling to predict changes in allocation policy, and

effects on wait list mortality, transplant rates, overall outcomes

28

SRTR – Scientific Registry of Transplant Recipients

• Risk stratified results– Wait list mortality– Transplant rates– Post transplant

• 1 month, 1 year, 3 year• Graft survival• Patient survival

• CUSUM charts– Real time analysis to trend events over time– Signals potentially concerning event rates

29

How is the expected pt and graft survival derived?

• Risk stratification model• E.g 1 yr graft survival model• Multiple factors• Donor factors • Recipient factors• Interaction between donor and recipient

30

Model for 1 year graft survival

• ABO compatible• Cold ischemia time• Local vs Regional• Recip factors

– Diagnosis of recip– PVT– Previous surgery– Alb, INR– Functional status– Race– Cr/dialysis– ICU/life support

• Donor age• Donor ht• Donor race• Donor use of drugs

(cocaine)• Donor hx of cancer• Hx of HTN• CDC high risk• Donor pressors/ddAVP• Split vs whole

Liver – adult 1 yr pt and graft survival

• Only one center had lower than expected 1 year patient survival on the last SRTR PSR

• Not a sensitive metric for quality

32

Risk stratification

• Too little?• Risk averse behavior• Inhibits innovation

• Too much?• Risky behavior• Futile transplants• Not best use of organs?

Should there be an absolute non risk stratified standard?

• Observed/Expected ratios• Limited/scarce resources– Lower absolute rates of success– Best use of limited resources?

• Argument that innovation is stifled

34

Quality control

• Available Statistical Techniques • Basic techniques

– Average mortality rate – Rolling average (last 10 cases) – Adjusted average mortality rates – # of consecutive failures

• Cox proportional hazard models – Adjusted for clinical characteristics – Incorporates risk adjustment

• Statistical process control – Continuous monitoring techniques (e.g. CUSUM)

CUSUM technique

• Statistical control charts developed to study industrial processes– Designed to ‘signal’ if there is a deviation from

accepted production standards• CUmulative SUM – used in quality control in

industry to trend events over real time• CUSUM techniques recently being used in

medicine and to track surgical outcomes

36

CUSUM – cumulative sum

Cumulative Sum (CUSUM) is technique of monitoring outcomes – Began as a method to monitor industrial processes – Produces a graphical output that can be tracked over time.

• CUSUM monitoring utility in health care recognized in 1970s – Limited by poor data collection – Lack of risk adjustment techniques

• Recent innovations let to expanded interest – Steiner et al. Incorporated risk adjustment – Axelrod et al. Effective in multi-center assessment in a retrospective

study – Kalbfleisch and Biswas- now utilizes survival analysis rather than a

binary (alive or dead) logistic analysis

37

CUSUM charts

• Graphical representation of outcomes for process – Can be risk-adjusted charts for important donor and recipient

characteristics – Plot outcomes over time to compare the results with expected

outcomes based on a national model of mortality or graph failure – 2 types: O - E charts and One-sided charts – Trends in the plot line suggest improving or declining outcomes – Once the trend line reaches a certain predefined level (one-sided

charts) or exceeds a certain slope (O - E charts) the CUSUM signals

CUSUM chart

• Graphical representation of observed vs expected events over time

• Risk adjusted CUSUM chartgs• One-sided CUSUM– Control limit is set, defines signaling threshold

• Two-sided or O-E CUSUM

43

A kidney transplant center

44

CUSUM conclusions

• CUSUM charting provides a reliable, risk adjusted method of tracking outcomes of a clinical process

• Can be “tuned” to balance the need for sensitively to detect clinical failures with the requirement to limit the number of false positive signals

• Graphical output is easily interpretable with a minimal amount of training

• Providing outcomes are promptly reported the CUSUM can provide real time insight into TC outcomes

National Transplant QI program

• Why?• 1 yr graft and pt survival – Gross outcomes– Team outcomes

• Provider specific outcomes• Need Benchmarks• Identify outliers• Identify areas for PI• Identify best practices

Surgical complications

• Identification of surgical complications is poor– In most cases, done retrospectively– By coders/billing personnel

• Poor risk adjustment• Lack of relevant benchmarks– Rate of SSI– Rate of thrombosis– Rate of biliary/ureteral complications– readmissions

Provider specific results

• Blood loss– Donor and recip characteristics

• Need for reoperation/intervention• Vascular complications• Bile duct/Ureter complications• Readmissions

TransQIP

• Accurate prospective data collection• Risk adjustment models to be developed and

tested• ACS/NSQIP tests inter-rater reliability and

accuracy• Establishment of national benchmarks,

comparison across centers/providers• Identification of best practices

The future of quality and reimbursement

• Value based purchasing• Expansion of PQRS program– From 0.5% incentive to 2% penalty– 3-8% penalty/incentives

• Maintenance of competency– Quality assurance activities

Summary

• The environment is changing• Quality will take center stage• Legislative/Regulatory environment• Reimbursement/Financial disincentives and

incentives• Value = Quality/Cost

Future directions• Patient safety

– Operative debriefings– Preventable errors in the OR – retained FB, errors of omission in the OR

• National TransQIP program– Collaboration between ASTS and ACS– Relevant outcomes– Benchmarks– Identify areas for improvement, identify best practices

• We can always do better, refine processes, increase pt safety• Patient satisfaction – modify Surgical Care CAHPS• Improve access, eliminate disparities• Composite Pre transplant Metric

– Waitlist mortality– Offer acceptance rate– Transplant rate (geographically adjusted)