Post on 13-Aug-2020
The Science Behind ClinicalSystems Improvement
Donald M. Berwick, MD, MPP, FRCPInstitute for Healthcare Improvement
Clinical Systems Improvement SeminarUniversity of Warwick
Coventry, UK: 23 January 2008
William Stewart Halsted
2
3
Canadian Task Force:Grades of Evidence
I: One Properly Randomized Controlled TrialII: Well-Designed Controlled Trials without
RandomizationII-2: Well-Designed Cohort or Case-Control
Analytic StudiesII-3: Multiple Time Series with or without the
Intervention; or Dramatic Results in UncontrolledExperiments
III: Opinions of Respected Authorities Based onClinical Experience; Descriptive Studies andCase Reports; Reports of Expert Committees
4
Institute of Medicine:Crossing the Quality Chasm
AIMS FOR IMPROVEMENT OF HEALTH CARE
• Safety• Effectiveness• Patient-Centeredness• Timeliness• Efficiency• Equity
5
Four Concerns
• Inference and gaining knowledge
• Thresholds for action
• Bias and trust
• Hope, teamwork, civility
Dr. Hillman’s Letter: July 23, 1982
6
Dr. Hillman’s Letter
7
Dr. Hillman’s Letter
“This is a different concept to the cardiacarrest team. It is hoped that providingearlier and more expert care,cardiopulmonary arrest may be avoided.The doctors called to seriously ill medicalor surgical patients would provide rapidresuscitation and assessment.”
8
Rapid Response Teams
BMC Code Rate
0.00
1.00
2.00
3.00
4.00
5.00
6.00
7.00
8.00
9.00
10.00
J an . 05F eb . 05
M a r . 0 5Ap r . 0 5
M ay . 05J un . 05
J u l. 0 5
Aug . 05
Sep t. 0 5O ct.
0 5
N ov . 05D ec . 05
J an -0 6F eb -06
M a r -06Ap r -0
6M ay -06
J un -0 6J u l- 0 6
Aug -06Sep -06
O ct- 06N ov -06
D ec -06J an -0 7
F eb -07M a r -0
7Ap r - 0
7M ay - 07
J un -0 7J u l-0 7
Aug -07Sep t-0 7
Co
de
pe
r1
00
0P
ati
en
tD
ay
sCode Rate
Mean
9
Rapid Response Teams
Codes Outside the ICU and RRS UtilizationDelmarva Foundation
2.4 2.2 2.3
1.9
1.4
1.9 2.01.7
2.812.1
14.3
18.6
14.4
9.7 9.710.6
11.8
9.9
0
1
2
3
4
5
Jan-06 Feb-06 Mar-06 Apr-06 May-06 Jun-06 Jul-06 Aug-06 Sep-06 Oct-06 Nov-06 Dec-06
Co
des
Out
sid
eth
eIC
Up
er1,
000
Dis
char
ges
0
2
4
6
8
10
12
14
16
18
20
RR
SC
alls
per
1,00
0D
isch
arge
s
Number of codes outside the ICU per 1,000 dischargesNumber of RRS calls per 1,000 discharges
10
Rapid Response Teams
Crash Call Rate
1
2
3
4
5
6
7
8
9
Jan-
05
Feb
-05
Mar
-05
Apr
-05
May
-05
Jun-
05
Jul-0
5
Aug
-05
Sep
-05
Oct
-05
Nov
-05
Dec
-05
Jan-
06
Feb
-06
Mar
-06
Apr
-06
May
-06
Jun-
06
Jul-0
6
Aug
-06
Sep
-06
Oct
-06
Nov
-06
Dec
-06
Period
Ind
ivid
ua
lV
alu
e
11
Rapid Response Teams
12
Rapid Response Teams
13
Rapid Response Teams
14
Rapid Response Teams
15
Rapid Response Teams
Catholic Health Initiatives System Mortality Rates
July02-Jun04 July04-Jun05 July05-June06 July06-Jun07 July07-Oct07
Baseline Mortality
IHI 100K Lives Campaign
Introduction to RRTs
100% CHI Beds
Covered By RRTsIHI 5M Lives Campaign
600+ RRT Calls
Q1 FY 08
16
17
The “100,000 Lives Campaign”
Six Changes That Save Lives
1. Deploy Rapid Response Teams
2. Deliver Reliable, Evidence-Based Care forAcute Myocardial Infarction
3. Prevent Adverse Drug Events (ADEs)
4. Prevent Central Line Infections
5. Prevent Surgical Site Infections
6. Prevent Ventilator-Associated Pneumonia
18
19
MERIT Study
“The MET system greatly increasesemergency team calling, but does notsubstantially affect the incidence ofcardiac arrest, unplanned ICU admissions,or unexpected deaths.”
MERIT Study Investigators. Introduction of the medical emergency team(MET) system: a cluster-randomized controlled trial. Lancet. 2005; 365:2091-2097
19
20
Johns Hopkins Researchers
“Given the equivocal evidence supportingthe effectiveness of RRT programs, withthe largest best-designed study showingno benefits, it is unclear why there is suchinterest in implementing this interventionand making it a care standard…”
Winters BD, Pham J, Pronovost PJ. Rapid response teams – walk, don’trun. JAMA. 2006; 296: 1645-1647.20
21
New England Journal of MedicineSounding Board
“Early trials of medical emergency teamssuggested a large potentialbenefit…However, a large, randomizedtrial subsequently showed that medicalemergency teams had no effect on patientoutcomes.”
Auerbach AD, Landefeld CS, Shojania KG. The tension between needingto improve and knowing how to do it. N Engl J Med. 2007; 357: 608-613.21
22
Consensus Conference Report
“It was emphasized that findings for oneresponse team might not…begeneralizable to other settings or overtime, and that the intervention itself shouldcontinually evolve and improve…”
Øvretveit J, Suffoletto J. Improving rapid response systems: progress,issues, and future directions. Jt Comm J Qual Patient Saf. 2007; 33: 512-519.
22
23
Consensus Conference Report
“ An RRS (rapid response system) is not adrug treatment, a hospital is not a body.Both are continually changing, complexsocial interventions and cannot beunderstood through randomized controlledtrials.” (John Øvreveit, MD)
Øvretveit J, Suffoletto J. Improving rapid response systems: progress,issues, and future directions. Jt Comm J Qual Patient Saf. 2007; 33: 512-519.
23
24
Pawson and Tilley: Realistic Evaluation
24
25
Pawson and Tilley
Pre-Test Treatment Post-Test
ExperimentalGroup
O1 X O2
Control Group O1 O2
The Classic Experimental Design: “OXO”
Pawson R, Tilley N. Realistic Evaluation. London: Sage Publications,Ltd.; 1997.25
26
Pawson and Tilley
“For us, the experimental paradigmconstitutes a heroic failure, promising somuch and yet ending up in ironicanticlimax. The underlying logic…seemsmeticulous, clear-headed, and militarilyprecise, and yet findings seem to emergein a typically non-cumulative, low-impact,prone-to-equivocation sort of way.”
Pawson R, Tilley N. Realistic Evaluation. London: Sage Publications,Ltd.; 1997.26
27
Pawson and Tilley
Context + New Mechanism = Outcome
C + M = O
Pawson R, Tilley N. Realistic Evaluation. London: Sage Publications,Ltd.; 1997.27
28
Pawson and Tilley
“In other words, programs work (havesuccessful ‘outcomes’) only in so far asthey introduce the appropriate ideas andopportunities (‘mechanisms’) to groups inthe appropriate social and culturalconditions (‘contexts’).”
Pawson R, Tilley N. Realistic Evaluation. London: Sage Publications,Ltd.; 1997.28
29
Implications of Pawson and Tilley
“In other words, Rapid Response Systemswork (have successful ‘outcomes’) only inso far as they introduce the appropriateideas and opportunities (‘mechanisms’)to groups in the appropriate social andcultural conditions (‘contexts’).”
Pawson R, Tilley N. Realistic Evaluation. London: Sage Publications,Ltd.; 1997.29
30
Pawson and Tilley
“…(E)xperimentalists have pursued toosingle-mindedly the question of whether aprogram works at the expense of knowingwhy it works.”
Pawson R, Tilley N. Realistic Evaluation. London: Sage Publications,Ltd.; 1997.30
31
Sir Austin Bradford Hill
“All scientific work is incomplete – whether itbe observational or experimental. Allscientific work is liable to be upset ormodified by advancing knowledge. Thatdoes not confer upon us a freedom toignore the knowledge we already have, orto postpone action that it appears todemand at a given time.”
Hill AB. The environment and disease: association or causation? ProcRoyal Soc Med. 1965; 58: 295-300.31
32
Four Changes
1. Broaden methods of inference andembrace a wider range of ways to growknowledge (with no relaxation of rigor)
32
Is life this simple?
X Y
(If only it was this simple!)
Understanding Profound Knowledge
33
No, it looks more like this…
X3
X2
X1
X5
X4
Y
In this model there are numerous direct effects between theindependent and variables (the Xs) and the dependent variable (Y).
Time 1
Time 3
Time 2
Dependentor outcomevariable
Ind
ep
en
de
nt
Va
ria
ble
s
34
In this case, there are numerous direct and indirect effects betweenthe independent variables and the dependent variable. For example, X1
and X4 both have direct effects on Y plus there is an indirect effect dueto the interaction of X1 and X4 conjointly on Y.
Y
Or, probably more like this…
X3
X2
X1
X5
X4
Time 1
Time 3
Time 2
R3
R2
R1
R5
R4
RY
Key Reference on Causal Modeling
Blalock HM, ed. Causal Models in theSocial Sciences. Chicago: Aldine; 1999.
R = residuals or error termsrepresenting the effects of variables
omitted in the model.
35
You need to enjoy the messiness of life!
36
Rigor and Complexity: View #1
SimpleLinear
Cause-and-Effect
Informal Narrative StudyStories and Anecdote
“Non-Rigorous”
ComplexNon-Linear
Chaotic
Formal Evaluative DesignsRCT’s
“Rigorous”
The “Evaluation Project” –Add Rigor to the Complex
37
Rigor and Complexity – View #2
SimpleLinear
Cause-and-Effect
ComplexNon-Linear
Chaotic
“Rigorous” Learning
Poor Learning38
Evidence in Simple Systems
SimpleLinear
Cause-and-Effect
ComplexNon-Linear
Chaotic
“Rigorous” Learning
Poor Learning
Fisher’sRCT’s
Halsted’sConclusions
39
Evidence in Complex Systems
SimpleLinear
Cause-and-Effect
ComplexNon-Linear
Chaotic
“Rigorous” Learning
Poor Learning
Fisher’sRCT’s
Halsted’sConclusions
RCT’s to StudyRapid Response Teams
40
Eating Soup with a Fork
SimpleLinear
Cause-and-Effect
ComplexNon-Linear
Chaotic
“Rigorous” Learning
Poor Learning
Fisher’sRCT’s
Halsted’sConclusions
RCT’s to StudyRapid Response Teams
41
Rigorous Learning inComplex Systems
SimpleLinear
Cause-and-Effect
ComplexNon-Linear
Chaotic
“Rigorous” Learning
Poor Learning
Fisher’sRCT’s
Halsted’sConclusions
RCT’s to StudyRapid Response Teams
What Methods ofLearning Go Here?
42
Rigorous Learning inComplex Systems
SimpleLinear
Cause-and-Effect
ComplexNon-Linear
Chaotic
“Rigorous”Learning
Poor Learning
Fisher’sRCT’s
Halsted’sConclusions
RCT’s to StudyRapid Response Teams
•Statistical Process Control•Time Series Methods•Off-Line Design•Anthropology•Ethnography•Journalism•Etc…Etc…Etc…
43
44
Four Changes
1. Broaden methods of inference andembrace a wider range of ways to growknowledge (with no relaxation of rigor)
2. Reconsider our attitudes towardthresholds for action
44
Primum non noncere
“When harm is underway, proceed urgentlyto learn how to stop it, and act urgently on
the learning.”
45
Science in Cycles of Action and Learning
PLAN – DO – STUDY - ACT
P
A
S
D
46
47
Four Changes
1. Broaden methods of inference andembrace a wider range of ways to growknowledge (with no relaxation of rigor)
2. Reconsider our attitudes towardthresholds for action
3. Reconsider our view of “bias” andincrease trust in knowledge of contextand mechanism
47
Auerbach AD, Landefeld CS, Shojania KG. Thetension between needing to improve andknowing how to do it. N Engl J Med. 2007; 357:608-613.
48
One View of Bias and Trust
“Even when direct financial conflicts ofinterest do not exist, any organization thathas undertaken a major campaign toimprove the quality of care has littleincentive to invest resources in a rigorousevaluation of the effects of its efforts.”
48
49
Possible Inferences…?
• “Parents have little incentive to evaluate theeffects of their parenting.”
• “Golfers have little incentive to study the effectsof their stroke.”
• “Armies have little incentive to study the effectsof their tactics.”
• “Politicians have little incentive to study theeffects of their campaign plans.”
• “Environmentalists have little incentive to studytheir effects on the environment.”
49
50
Four Changes
1. Broaden methods of inference andembrace a wider range of ways to growknowledge (with no relaxation of rigor)
2. Reconsider our attitudes towardthresholds for action
3. Reconsider our view of “bias” andincrease trust in knowledge of contextand mechanism
4. Reconsider mood, affect, and civility
50