How Will You Know That a Change Is An...

9/15/2014

1

How Will You Know That a Change Is An Improvement?

Robert Lloyd, PhD

John Boulton, MD

Day 2Concurrent Breakout Session15 September 2014

How will you know…

1. If the change(s) you have made signal a true

improvement? If you have sustained

improvement?

2. If it is the right time to implement the

change(s)

3. If it is time to spread the change(s) to other

areas?

4. If it is time to stop measuring?

9/15/2014

2

“There are many hazards to the use of

common sense. Common sense cannot be

measured. You have to be able to define

and measure what is significant. Without

statistical methods you don’t know what the

numbers means.”

Source: Dr. W. E. Deming as quoted by N. Mann, The Keys to

Excellence: The Story of the Deming Philosophy. Mercury Books,

London, England, 1989: 62.

An Option: Use Common Sense? 4

1. By understanding the variation

that lives within your data

2. By making good management

decisions about this variation (i.e.,

don’t overreact to a special cause and don’t

think that random movement of your data up

and down is a signal of improvement).

How will we know that a

change is an improvement?

9/15/2014

3

Understanding Variation:

You have a choice!

5

STATIC VIEW

Descriptive StatisticsMean, Median & Mode

Minimum/Maximum/RangeStandard Deviation

Bar graphs/Pie charts

DYNAMIC VIEWRun Chart

Control Chart

(plot data over time)

Statistical Process Control (SPC)

Rat

e pe

r 100

ED

Pat

ient

s

Unplanned Returns to Ed w/in 72 Hours

M

41.78

17

A

43.89

26

M

39.86

13

J

40.03

16

J

38.01

24

A

43.43

27

S

39.21

19

O

41.90

14

N

41.78

33

D

43.00

20

J

39.66

17

F

40.03

22

M

48.21

29

A

43.89

17

M

39.86

36

J

36.21

19

J

41.78

22

A

43.89

24

S

31.45

22

Month

ED/100

Returns

u chart

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19

0.0

0.2

0.4

0.6

0.8

1.0

1.2

UCL = 0.88

Mean = 0.54

LCL = 0.19

6

How will we know that a

change is an improvement?

Which time period

is better?

Now, which time

period is better?

2013

2013

2014

2014

Percent of

patients

assessed

for

pressure

ulcers

Percent of

patients

assessed

for

pressure

ulcers

This is the difference

between static and

dynamic displays of data!

Source: R. Lloyd. Quality Health Care: A Guide to Developing and

Using Indicators. Jones and Bartlett Publishers, 2004.

9/15/2014

4

Walter

Shewhart

(1891 – 1967)Joseph Juran

(1904 - 2008)

W. Edwards Deming

(1900 - 1993)

The Pioneers of Statistical Thinking 8How do we analyze variation for

quality improvement?

Run and Control Charts are

the best tools to determine

if our improvement

strategies have had the

desired effect.

9/15/2014

5

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29

Point Number

Po

un

ds o

f R

ed

Ba

g W

aste

3.25

3.50

3.75

4.00

4.25

4.50

4.75

5.00

5.25

5.50

5.75

6.00

Median=4.610

Measu

re

Time

Four simple run rules are used to determine if

non-random data patterns are present

X (CL)~

Elements of a Run Chart

The centerline (CL) on a

Run Chart is the Median

Jan01 Mar01 May01 July01 Sept01 Nov01 Jan02 Mar02 May02 July02 Sept02 Nov02

Month

Nu

mb

er

of

Co

mp

lain

ts

5.0

10.0

15.0

20.0

25.0

30.0

35.0

40.0

45.0

50.0

A

B

C

C

B

A

UCL=44.855

CL=29.250

LCL=13.645

Elements of a Control Chart

X (Mean)

Me

as

ure

Time

An indication of a

special cause(Upper

Control Limit)

(Lower Control

Limit)

9/15/2014

6

11

AIM (How good? By when?)

Concept

Measure

Operational Definitions

Data Collection Plan

Data Collection

Analysis ACTION

But, you need a Roadmap for your

Quality Measurement Journey



12

AIM (How good? By when?)

Concept

Measure

Operational Definitions

Data Collection Plan

Data Collection

Analysis ACTION

The Quality Measurement Journey



9/15/2014

7

13

Types of Variation

Common Cause VariationIs inherent in the design of the process

Is due to regular, natural or ordinary causes

Affects all the outcomes of a process

Results in a “stable” process that is predictable

Also known as random or unassignable variation

Special Cause Variation

Is due to irregular or unnatural

causes that are not inherent in

the design of the process

Affect some, but not necessarily all aspects of the process

Results in an “unstable”

process that is not predictable

Also known as non-random or

assignable variation


1. If the change(s) you have made signal a

true improvement?

2. If you have sustained improvement?


change(s)


areas?

5. If it is time to stop measuring? SPSP

9/15/2014

8

Source: The Data Guide by L. Provost and S. Murray, Jossey-Bass Publishers, 2011.

A Shift:

6 or more

An astronomical

data point

Too many or

too few runs

A Trend

5 or more

Run Chart Rules are used to determine

if a change has occurred

Use the run chart rules to determine if a change has

occurred

A shift = 6 or more data points above or below the

baseline median (centerline)

A trend = 5 data points constant going up or down

16

Random Variation (common cause)

Nothing has changed here!

9/15/2014

9



improvement?



change(s)


areas?


160

180

200

220

240

260

280

300

320

LO

S (

min

ute

s)

2/16/11 3/16 4/13 5/11 6/8

Week

Measu

re

Time

Median

A downward shift in the

data (6 data points below

the median)

Sustained Improvement

• First identify a shift or a trend in the data

• Then look to see if 3 or more data point have stayed at

the new level.

3 more data

points staying

at the new level

of performance

9/15/2014

10



improvement?



change(s)


areas?


Degree of Belief When

Making Changes to Improve

Developing a change Testing a change - cycle 1,

cycle 2, cycle 3Implementing a Change

Degree of

belief that a

change will

result in

improve-

ment

LOW

MO

DE

RA

TE

HIGH

Change needs

further tesing

Unsuccessful

change!

Successful

change!

Source: The Improvement

Guide, Langley, J. et al,

Jossey-Bass, 2009: 145.

9/15/2014

11

Implementing a Change

Baseline

Testing

Successful

Testing

Begin implementation

on pilot unit

Evidence of improvement

during implementation

Note that when you move to full implementation things may

actually get worse for a little bit.

Current Situation Resistant Indifferent Ready

Low

Confidence that

current change

idea will lead to

Improvement

Cost of

failure

large

Very Small

Scale Test

Very Small

Scale Test

Very Small

Scale Test

Cost of

failure

small

Very Small

Scale Test

Very Small

Scale Test Small Scale

Test

High

Confidence that

current change

idea will lead to

Improvement

Cost of

failure

large

Very Small

Scale Test Small Scale

Test

Large Scale

Test

Cost of

failure

smallSmall Scale

Test

Large Scale

TestImplement

Conditions for Implementing a Change

Note the conditions for Implementing a change!

Risk of not

succeeding

large

Risk of not

succeeding

large

Risk of not

succeeding

small

Risk of not

succeeding

small

Risk of not

succeeding

small

9/15/2014

12



improvement?



change(s)

4. If it is time to spread the change(s) to

other areas?


Collaborative

John Whittington OSF Healthcare

Holding the Gains

Spreading a Change

• First identify a shift or a trend in the data.

• Then look to see if 6 or more data point have stayed at the new

level.

• This indicates that you are ‘holding the gains.’

A downward shift in the

data (6 data points below

the median)

6 more data

points staying

at the new level

of performance

9/15/2014

13



improvement?



change(s)


areas?


Two Simple Rules for Measuring

Outcome Measures – always!

Process Measures – it depends!

9/15/2014

14

How often do you need to measure?

It is not uncommon for a team to want to stop

collecting data, especially after they have

been at it for a year or two!

The reliability of the process and your

need to know how the process is

functioning should drive the frequency of

data collection and analysis.

A Simple Rule for Outcomes

Outcome Measures – Always!

As long as you are concerned about the quality and

safety of the care that you deliver, you should

continue to track the outcomes!

For example, how long should these outcomes be measured?

• When do you stop measuring your financial results?

• When should a diabetic patient stop tracking his or her blood glucose?

• How long should we monitor the vital signs of an ICU patient ?

• When should airport security stop assessing passengers for weapons?

• How long does a local water authority need to measure the quality of the

water going through its pipes?

• When should schools stop measuring the progress of students?

9/15/2014

15

Process Measures – it depends!• Process measures usually demonstrate improvement before

outcome measures.

• Process measures may be revised during an improvement

project; new data will then need to be collected and tracked.

• A process measure should demonstrate improvement

(against the run chart rules) and then STAY at the new level

of performance for at least 3 reporting periods to be

considered “sustained.”

A Simple Rule for Processes Frequency of Process Measures

Regularly (daily, weekly or monthly)Done to improve a specific measure (reduce variation or shift the

centerline of process performance)

Periodically (once every 2 - 3 months)Done when statistical improvement has been noted, sustained AND

the process is highly reliable (audit approach can be used here)

Once or twice a year (why bother?)

Stop measuring! Done when performance is so reliable, stable and capable that it is

time to move on to measure something new.

9/15/2014

16

Process Improvement: Isolated Femur Fractures

0

200

400

600

800

1000

1200

1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55 58 61 64Sequential Patients

Min

ute

s E

D t

o O

R p

er

Patient

Holding the Gain: Isolated Femur Fractures

0

200

400

600

800

1000

1200


Min

ute

s E

D t

o O

R p

er

Patient

3. Determine if we are

holding the gains

Current Process Performance: Isolated Femur Fractures

0

200

400

600

800

1000

1200


Min

ute

s E

D t

o O

R p

er

Patient

The Steps in

Statistical

Thinking

2. Determine if a change is an improvement

1. Make process performance visible

Baseline

Data

Change

Introduced

here

Sustained

Improvement

The Statistical Thinking Framework

Systems Thinking

Analysis ActionPhilosophy of

Measurement

Statistical Methods

Process ImprovementVariation Data

Statistical Thinking will be found in all aspects of

organizational behavior and performance

Source: American Society for Quality, Statistics Division.

9/15/2014

17

Attributes of a Person Who

Understands Variation

They understand the different ways in which variation is viewed (static versus dynamic).

They explain changes in terms of common causes and special causes.

They use graphical methods to plot data over time, learn from data and expect others to consider variation in their decisions and actions.

They understand the concept of stable and unstable processes and the potential losses due to tampering.

They understand the capability of a process or system before changes are attempted.

34

Sustaining improvements

and Spreading changes to

other locations

Developing

a change

Implementing a

change

Testing a

changeTheory and

Prediction

Test under a

variety of

conditions

Make part of

routine

operations

The Sequence of Improvement

requires Measurement

9/15/2014

18

35

In short, statistical thinking is a way to approach all

aspects of work.

It is a way of thinking about numbers and how they

can be used to make improvements.

Statistical thinking is the primary way to know

if a change has led to improvement!

Statistical Thinking is a Way of Life!

How Will You Know That a Change Is An...

Documents

Transcript of How Will You Know That a Change Is An...