STATISTICAL TOOLS FOR QUALITY IMPROVEMENT

Ok, this is the week that many find the most challenging. So first of all, relax, we will repeat these concepts again and again until you are sick of hearing them.

As we have discussed, a major focus of quality management is understanding variation. (I told you that you would be sick of hearing this) It will not surprise you then, to know that we are again going to talk about variation.

Let’s begin this week by reviewing 2 key points that we have already covered.

First—when we talk about quality improvement in healthcare remember:

Quality Improvement= Process ImprovementRecall that we have noted that a gap or variation is the difference between how our process is working and how it should work. Our goal is to modify the process to eliminate unwarranted variation and make the outcome more predictable. (Hopefully this sounds familiar)

Second- we can identify two types of variation.

Random (common cause variation) that is inherent in every process and can’t be eliminated without changing the process. Note that there are different terms to describe the same phenomenon. Sorry for the confusion.

Assignable (special cause variation) that indicates that something has changed in the process. Again, there are different terms to describe the same phenomenon. The previous apology applies here as well.

A big mistake that people make in their quality improvement efforts is confusing Common Cause Variation with Special Cause Variation. This week we will see a practical experiment of why this confusion can create problems in the video entitled “The Funnel Experiment”

Let’s just think about this for a moment.

Take a look at the following bar chart. It shows the percent mortality rate in a before period and in an after period. Mortality rate of what, you ask. Let’s say its 30-day surgical mortality rate, for the sake of argument. The question that we want to answer is “did this organization show improvement?”

Before After0

1

2

3

4

5

6

Percent Mortality

The temptation here is to say “yes” because the before percentage is higher than the after percentage. And this is precisely why this topic is so confusing to many of us. What we need to understand before we can determine if there has been improvement is—statistically speaking, are the before and after numbers really different? If the difference can be explained by common cause, random variation, then really the fluctuations between the before and after are not surprising—and we can’t really say that improvement has occurred.

If that makes sense to you, great- you can skip ahead. If you are like me and that does not make intuitive sense- let’s look at a bunch of examples to try and make this point clearer.

How do we distinguish Common Cause and Special Cause Variation? Before we look at the example in the text, let’s consider a simplified version of the problem.

You have probably seen this ad for Domino’s Pizza:

As you can imagine, Domino’s has a strong interest in ensuring that pizzas are delivered in 30 minutes or less. They therefore want to measure the time that their delivery process takes. It turns out they deliver one pizza a day to the same address, so they record the amount of time it takes to deliver a pizza each day. The driver takes the same route each day. (as I’ve said, we have simplified the story here)

Days 1 2 3 4 5 6 7 8 9Minutes

26 28 31 29 26 32 28 26 29

Days 10 11 12 13 14 15 16 17 18Minutes

27 28 30 32 27 29 29 31 28

Days 18 20Minutes

60 27

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 200

10

20

30

40

50

60

70

Delivery Times

As you can see the gray line represents the goal of 30 minutes. You can also see that sometimes (Day 3, Day 6, Day 13, Day 17) it takes longer than 30 minutes to deliver a pizza. On those days the store manager yells at the driver for taking longer than 30 minutes—but if you look at the graph, you can see that is really not fair. The time it takes for the pizza to be delivered generally ranges from 26 minutes to 32 minutes. That appears to be the normal, random variation (or common cause variation) that is inherent in driving a pizza to that same address. (we will talk in a minute how we can tell for sure that this is the case.) Blaming the driver for the days that delivery takes more than 30 minutes—or praising the driver for the days that delivery takes 30 days or less is pointless. If this is common cause variation, the process as it is now is always going to bounce around between 26 and 32 minutes. If the manager at Domino’s wants to guarantee that delivery will always be less than 30 minutes, he or she is going to have to improve the entire process. Maybe instruct the driver to take a different route, maybe change how long pizzas cook. Without changing the process though, delivery will always occur between 26 and 32 minutes.

But take a look at Day 19. Delivery on that day took way longer than expected! Looking at the graph we are left thinking that something weird must have happened that day. Maybe the delivery person

stopped to take pick up his daughter on the way to delivering the pizza. Or maybe there was an accident on the road. Or a snowstorm. Whatever it was, something was different—meaning that this was special cause variation. If the manager wants to improve in this case, he or she needs to figure out what was different and address that specific problem.

The manager of Domino’s is making a common mistake of confusing common cause and special cause variation. People often mistake of not recognizing fluctuations in performance as just the random variation that occurs in any process and they react to the last value that they see. We are going to explain why this is a problem in a minute. It will also be covered in the video entitled Burnt Toast for this week.

Again, if this is all making sense to you, some of these videos will be very repetitious and you may be able to skip them. But many people are like me—and this stuff is not intuitive.

Common Cause and Special Cause Variation in Healthcare Quality Improvement and The Run Chart

The text presents a typical situation where a hospital seeks to decrease the number of adverse events occurring each year by 10%. The hospital creates a great scorecard—green when they’ve met their target (yay!), red when they haven’t (boo!). Hospital executives love these types of charts because they are easy to read. Let’s see why they can be misleading.

The hospital raw data looks like this:

The hospital decreased the number of adverse events by 8 (130- 122). You can see that info in the “Total column on the far right hand of the chart. Sadly, they conclude, they did not meet their target reduction.

The hospital then graphs their results to see if they can identify any patterns. The two approaches they try are here:

As the hospital leadership looks at the data and the charts, they congratulate themselves on their great performance in months 4, 9 and 12 of the 2nd year. They can tell they have done well because those months are colored green meaning they met their target. However, they bemoan their terrible performance in months 3 and months 11. They got a red color which advertises their poor performance. They shake their collective heads and try to determine who is to blame for the poor performance on those months. Sadly, they are making the same mistake that the manager at Domino’s made. They are confusing random variation and common cause variation and it is causing them to misinterpret their data.

But how do we know this? Well it turns out there are some statistical tools that let us distinguish between these two types of variation.

The first tool that we will look at is the run chart. The run chart plots the measured value over time and then adds a line to represent the median (the point where half the values lie above, and half the values lie below the line)

The above data is represented in a run chart below:

We can then use this run chart to determine if the variation we are seeing is the normal random variation that we expect to see or if something else is going on.

The way we determine this is through a series of tests. If any of these tests are positive, this is an indication that the variation we are seeing is NOT random. “Not random” means that the variation is not due to Common Cause—ergo, there must be something strange happening, that is special cause variation.

The tests that indicate special cause variation include:

Finding a Trend: Six successive points that increase or decrease. If two or more consecutive points are the same, only the first one counts.

Finding a Shift: Eight-points-in-a-row all above or all below median. Lines on the median don’t count- they are ignored, they neither break a shift nor count as part of the shift.

Finding an Astronomical Point: a significant outlier

Finally, we want to look for non-random patterns, that is too many or too few runs. A run is a series of points in a row on one side of the median.

Returning to our example, for 24 data points, we should have more than 8 runs and less than 17 (some charts say 18) runs- which we do. (The table showing the number of runs per number of data points is available in the other handout in the weekly overview section.)

In the case we are looking at, none of the tests are positive. It seems therefore all the differences in the data points can be explained by random (common cause) variation.

An important point to note is that special cause variation is not good or bad—it merely tells you that something has changed. In fact, quality improvement teams often look at run charts to find proof that the changes that they made to the process have resulted in improvement. As you can see in the run chart below, there is clear evidence of a shift, which in this instance has brought the organization closer to their goal of 90%. It should be noted that in situations where there is baseline data and a clear intervention, the rules regarding special cause (which can indicate an improvement) are a bit less rigorous than if the chart is being used to demonstrate if special cause variations exists.

The Control Chart

The Control Chart (or the statistical process control chart, or the Shewart Chart—it is a little confusing how in Quality Improvement there are multiple names for the same item—maybe that is an opportunity for improvement :-) also considers the issue of common cause variation and special cause variation but it provides an additional feature.

Control charts have the additional feature of statistical control limits—and thus is better able to answer the questions

1) Is this process in control (that is free from special cause variation)?2) Have our changes resulted in improvements?

A control chart, like a run chart, plots the measured element over time. Unlike the run chart, the control chart then calculates the mean. It also makes some additional calculations.

In your prior statistic classes, you have likely encountered the idea of a normal distribution curve (a bell- shaped curve)—in the center of the curve is a line representing the mean (average) of all the values under the curve. In addition, you may recall the idea of the standard deviation (the spread around the mean of the bell-shaped curve). You might remember something that looks like this:

You also probably remember that 3 standard deviations on either side of the mean include 99.7% of all the values under the curve.

Now imagine for a moment, we tilt this bell-shaped curve on its side and place it next to the graph with the plotted data

Let’s now call the 3 standard deviation points (circled in red) above—the upper and lower control limits. (incidentally, you will notice you have 3 standard deviations on each side of the central line—for a total of six standard deviations. That is where the term “six sigma” comes from in the six-sigma improvement model) (technically the control limits are not exactly equal to 3 standard deviation—but its close enough for our purposes)

The construction of control charts can be a little tricky—and requires special software. For the purposes of this course, you will not be expected to create a control chart. But by the end of this week you should feel comfortable reading a control chart and being able to explain what a control chart is actually telling us.

To do this—let us look at the control chart from the text. You will recognize this data from our hospital that was trying to decrease their number of adverse events. You remember the run chart that we just looked at-

Recall that in the Run Chart—the purple line is the median—the point where half the data points are above, and half the data points are below.

Compare the run chart with the chart below (which is a control chart). What do you notice that is different?

If you look closely you will see there is a green line that represents the average of all the data points rather than the median that we saw in the control chart. But you will also notice 2 red dotted lines—one toward the top of the chart and one toward the bottom. These are the upper and lower control limits.

The interesting thing about the data points on this graph—and the thing that is hard for many people to understand--- because the data points lie between the red lines, they are indistinguishable:

(1) from each other

(2) from process average

Let’s repeat that again for emphasis. Because the data points all lie between the red control lines, that means that statistically speaking there is no difference between the high value of 16 and the low value of 7. This is the normal common cause variation that we expect to see in a process. By now, you should be sick of hearing about common cause variation- but you should feel comfortable defining it.

Now let’s go back to the original data where the hospital leadership was patting themselves on the back in the months that were colored green—and grinding their teeth in the months that were colored red.

The control chart shows us that statistically speaking these two numbers are not different. Freaky huh? Therefore, the red and green colors are meaningless. This is a mistake that people interpreting quality data make all the time. Fortunately, you now won’t make this mistake.

When a process is charted on a control chart and all the points are between the upper and lower control limit, the process is said to be “in control” or “stable.” That is the variation you are seeing is common cause variation.

If you are confused, don’t worry. Robert Lloyd will walk through this again in this week’s videos.

One other area of confusion that arises in the interpretation of control charts is regarding benchmarking.

The chart below shows percent-compliance with the Diabetic Guidelines. A goal of 75% compliance has been set. It looks like a change in process was made somewhere around month 10. Notice the control limits have changed—and they have gotten tighter. It looks like something has improved. However, look at the mean of the new process. It looks like it is around 68%. That means, this process is not going to consistently meet the goal of 75%. Another change in the process is going to have to occur. But the upper control limit looks about 77%. That means, as a result of random, common cause variation- this process will occasionally give results of 75%, making it appear that the process is meeting the benchmark. If the quality improvement team misinterprets this information, they are going to be frustrated. Remember, all data points within the control limits are the same from a statistical point of view—so 76% is the same as 65%. If the improvement team forgets this, they will be pulling out their hair trying to figure out why they have met the target one month- but not met it the next month. To

truly meet the 75% target, the team has to change the process again until they develop a process where the average performance of the process is 75%.

USING DATA FOR IMPROVEMENT

With all this talk about run and control charts, it is easy to forget the purpose of this data—and that is use data in order to improve.

Let us know assume that we have collected data and it shows that we have a process that is stable, but that is performing below where we want it to be.

But how do we know where to focus our improvement efforts.

We could guess.

We could brainstorm. Brainstorming is not a bad approach and will generate a lot of ideas. One tool for brainstorming is using a fishbone diagram.

But is there a way to focus our efforts even further to get the biggest bang from our improvement buck?

The answer, as you may have guessed, is yes. Recall back to our first lecture when we discussed the idea of the Pareto analysis.

The textbook presents a case of a team that was trying to decrease hospital acquired infections in the neonatal intensive care unit of its hospital systems. To decrease the rate of infections, the NICUs had implemented infection control Bundles. Bundles, you will recall, are a series of evidence-based steps that have been shown to decrease infection rates. Bundles are often operationalized as a checklist. Here is a commonly used bundle for prevention of ventilator acquired pneumonia.

People check off each box as it is done. This allows the quality team to perform an audit on each step—assuming people fill out the form correctly.

The NICU team described in the text thought they had made an improvement, however, a run chart demonstrated that the variation that they interpreted as improvement was only random, common cause variation.

How should the improvement team proceed?

Like most improvement teams, the NICU team started with a fishbone diagram. This generated a lot of ideas—but which ideas would have the most impact. In terms of a Pareto chart—what is the 20% problem that will result in 80% improvement.

To answer this question, the team looked at 50 episodes of “non-compliance”—that is, just enough data.

What they noted was this:

It appeared two steps in the Bundle were most frequently missed. Step number 5, and step number 3

The team also looked at the compliance rates of different hospitals in the system and noticed this

The biggest rates of non-compliance occurred at Hospital E and Hospital B.

This information helps the team develop a “laser” strategy rather than a “shot gun” strategy.

The team then constructed a Pareto Matrix—comparing the Failures of Bundle Compliance to the Different Hospitals

The Matrix identifies areas of interest. Take a look at the highlighted areas which represent special cause variation. Do you notice any patterns?

Note that all hospitals appear to have difficulty with Bundle Element 5 (circled in red)—This is probably an area that needs more exploration. Simply telling people to do a better job with Element E will probably not solve this issue.

The other interesting thing to notice is Hospital E, it appears to be having trouble with all the elements of the Bundle. (blue circle) Again, additional exploration as to the cause is needed—but now the team knows where to focus their effort

Finally, the other area of note is that it appears that Hospital B is having difficult with compliance on Element 3—again an area that needs further analysis.

As you can see, this approach focuses the efforts of the quality improvement team bringing their attention to the areas that are most likely to have the biggest impact on improvement.

This is an information dense week. Don’t worry if you didn’t understand this information here. It is repeated in the text—and the videos include some excellent explanation by Robert Lloyd of IHI. We will then walk through a run chart exercise to be sure you have the concepts down.

By the end of the week, you should understand the concept and avoid making the mistakes of the people in this week’s examples.