Post on 21-Mar-2017
Are We Making a Difference? Using Measurement to Guide Improvement
Andrew Wray
Quality Forum 2017
Presenter Disclosure
Presenter: Andrew Wray
Relationships with commercial interests: – Nothing to disclose
Performance Measurement in Health Care
• Accountability: for judgement, to reassure the public, to decide how much people are paid, to choose between two options.
• Research: to develop new knowledge, often generalizable
• Improvement: to guide our efforts to change, to monitor our progress, to understand the problem.
Reference: The Improvement Guide, 2nd ed. Langley,
Moen, Nolan, Nolan, Norman & Provost, p. 24
Purpose of Measurement for Improvement
Data can tell us:
- How we are progressing over time
- Where we can get some change ideas
What to measure?
• Each project will need to identify a couple of key measures related to the work
• Will be a mix of process, outcome and balancing measures
• Needs to be meaningful for the team, and reflective of the processes you are improving
We’ll also have measures as part of our PDSA cycles – more later.
Guideline: between 3-8 measures per project
• at least one outcome measure
• at least one process measure
A Few key indicators to track project progress
Family of Measures
Example: Improving Diabetes Care
• Outcome: – HbA1C meeting clinical target
• Process:
– % who have attended structured education session – % screened for retinopathy in the last year – % reporting sufficient social and spiritual supports
• Balancing:
– Staff satisfaction
• For your work, what sorts of indicators are important to track?
Operational Definitions
• Full description of a measure: – What is the measure
– Inclusion/exclusion criteria
– Calculation
– Sample size
– Sampling strategy
– Subgroup frequency
– Data collection strategy
– By whom
– Etc.
Measurement Plan Worksheet
Measure Operational
Definition Outcome, Process or Balancing
Data Collection Strategy
Frequency of Data Collection
How will measure be displayed
Baseline result Target result
Operational Definitions • Option 1
– Number of people who fell
– All patients over 5yrs
– Number of patients with a fall/total number of patients
– Complete enumeration
– Monthly – Data source: DAD
• Option 3
– Number of falls with an injury
– All patients
– Count of all falls in the facility that caused harm
– Sample of 50 charts
– Audit by quality department staff
– Quarterly
• Option 2
– Number of falls
– Patients 70+
– Count of all falls in the facility
– Complete enumeration
– Reports to PSLS
– Monthly
Sampling?
• Complete enumeration is best
• Sampling saves resources – usually recommended – use judgment samples
DATA COLLECTION
• Start right away
• Small, frequent measures
• Integrate into workload
• Timely
• For the indicators you started thinking about, use the measurement plan worksheet to start working on an operational definition.
• So we’re collecting data…
what do we do with it?
• So we’re collecting data…
what do we do with it?
Share it with your team!
The Run Chart: Tracking progress over time
Data displayed in time order
Data is collected and displayed weekly or monthly.
Pre and Post Change Bar Chart – What is the Interpretation?
Scenario 1 . Data displayed in a run chart over time.
0
2
4
6
8
10
change made between week 7 and 8
Scenario 1 . Pre-post data.
Let’s Look at the Data in a Run Chart: Scenario 1
0
1
2
3
4
5
6
7
8
9
10
Scenario 2 . Pre-post data.
Scenario 2 . Data displayed in a run chart over time.
Scenario 2
change made between week 7 and 8
Scenario 3. Pre-post data.
Scenario 3. Data displayed in a run chart over time.
change made between week 7 and 8
Scenario 3
What if we use a t-test? Average Before
Change =70.0
Average After Change =30.1
0
10
20
30
40
50
60
70
80
The t-test shows a significant difference:
t(22)=7.6, p<.001
Adapted from Perla R.J., Provost L.P., & Murray S.K.
(2011). The run chart: a simple analytical tool for
learning from variation in healthcare processes. BMJ
Quality & Safety, 20(1):46-51.
0
10
20
30
40
50
60
70
80
90
Week1
Week2
Week3
Week4
Week5
Week6
Week7
Week8
Week9
Week10
Week11
Week12
Week13
Week14
Week15
Week16
Week17
Week18
Week19
Week20
Week21
Week22
Week23
Week24
pre-change
post-change
Average Before Change
=70.0
Average After
Change =30.1
0
10
20
30
40
50
60
70
80
t(22)=7.6, p<.001 Adapted from Perla R.J., Provost L.P., & Murray S.K.
(2011). The run chart: a simple analytical tool for
learning from variation in healthcare processes. BMJ
Quality & Safety, 20(1):46-51.
Data in real-time
• Data in real time allows us to track progress of the work
• Tells us if we getting better or worse
• Tells us if the changes we are testing are working
– The longer the lag, the slower the learning cycle
Visual analysis of run charts
No improvement. Random fluctuation.
Improvement. Trend going up.
Run charts: Evidence of non-random patterns
0
5
10
15
20
25
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
Rule 1. Shift
0
5
10
15
20
25
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
Rule 2. Trend
0
5
10
15
20
25
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
Rule 3. Runs
Data line crosses once Too few runs: total 2 runs
0
5
10
15
20
25
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
Rule 4. Astronomical Point
“Measurement should be used to speed things up, not to slow them down”
• - IHI Breakthrough Series Guide
Reference: The Improvement Guide, 2nd ed. Langley,
Moen, Nolan, Nolan, Norman & Provost, p. 24
What about PDSA cycles?
• We’ll have a family of measures to track over the duration of the project but we will also need other measurement
– We’ll want to use measurement to learn what is working when we test changes.
PDSA measurement
• Change idea: phone call reminders
• P: try 5 phone calls
• D: first 5 patients on Monday
• S: Number reached, length of call, did they attend
• A: What next?
Generate light, not heat, with data
• Reinertsen JL, Gosfield AG, Rupp W, Whittington JW. Engaging Physicians in a Shared Quality Agenda. IHI Innovation Series white paper. Cambridge, Massachusetts: Institute for Healthcare Improvement; 2007