The Run Chart a Simple Analytical Tool

The run chart: a simple analytical toolfor learning from variation inhealthcare processes

Rocco J Perla,1 Lloyd P Provost,2 Sandy K Murray3

Background: Those working in healthcare today are

challenged more than ever before to quickly and

efficiently learn from data to improve their services and

delivery of care. There is broad agreement that

healthcare professionals working on the front lines

benefit greatly from the visual display of data

presented in time order.

Aim: To describe the run chartdan analytical tool

commonly used by professionals in quality

improvement but underutilised in healthcare.

Methods: A standard approach to the construction, use

and interpretation of run charts for healthcare

applications is developed based on the statistical

process control literature.

Discussion: Run charts allow us to understand

objectively if the changes we make to a process or

system over time lead to improvements and do so with

minimal mathematical complexity. This method of

analyzing and reporting data is of greater value to

improvement projects and teams than traditional

aggregate summary statistics that ignore time

order. Because of its utility and simplicity, the

run chart has wide potential application in

healthcare for practitioners and decision-makers.

Run charts also provide the foundation for more

sophisticated methods of analysis and learning such

as Shewhart (control) charts and planned

experimentation.

The skills associated with using data forimprovement vary widely among thoseworking to improve healthcare. We describea simple analytical tool commonly used byprofessionals in quality improvement, butunderutilised in healthcaredthe run chart.1

For those health professionals that use runcharts, they provide a valuable source ofinformation and learning for both practi-tioner and patient. The following scenariodescribed by Neuhauser and Diaz2 providesone example of the simplicity of run chartsand their potential for wide application inhealthcare:

Susan Cotey is a diabetes educator at Huron

Hospital in Cleveland, Ohio. She gives out

graph paper to elderly diabetic patients who

live in the most impoverished part of her city.

She uses a self-help book designed specifi-

cally for her patients. Each patient gets

a copy. She asks them to plot their blood

sugar measures over time, connect the dots

and bring their graphs in to small discussion

groups of similar patients who share their

experience and learn about diabetes self-

management (diet, exercise, weight

control). Nearly every patient brings in their

graph. The large majority of patients

improve their diabetic control. This hospital

has made diabetes management a centre of

its healthcare mission.

The use of run charts by these patients withdiabetes summarises the spirit of our paperdthe run chart has a role to play in healthcareimprovement work.Although many healthcare professionals

now recognise the value of statistical processcontrol methods, applications and tools inimproving the quality of care, much of thisfocus in the healthcare improvement litera-ture is on Shewhart (control) charts andtheir various derivatives (such as cumulativesummation charts and funnel plots).3 Verylittle has been written about the use andapplication of run charts.The run chart allows us to learn a great deal

about the performance of our process withminimal mathematical complexity. Specifi-cally, it provides a simplemethod to determineif a process is demonstrating non-randompatterns, what we term a ‘signal’. By focussingon the time order that data are collected, therun chart can be applied when traditionalmethods to determine statistical significance(t-test, chi-square, F test) are not useful.Important uses of the run chart for improve-ment activities include the following4:

< An additional text box ispublished online only. Toview this file please visit thejournal online (http://qualitysafety.bmj.com).1UMass Memorial HealthCare, Worcester,Massachusetts, USA2Associates in ProcessImprovement, Austin, Texas,USA3Corporate TransformationConcepts, Eugene, Oregon,USA

Correspondence toDr Rocco J Perla, Office ofQuality and Patient Safety,Center for Innovation andTransformational Change, 22Shattuck Street, Worcester,MA 01605, USA; [email protected]

Accepted 26 July 2010

46 BMJ Qual Saf 2011;20:46e51. doi:10.1136/bmjqs.2009.037895

ORIGINAL RESEARCH

group.bmj.com on April 8, 2011 - Published by qualitysafety.bmj.comDownloaded from

http://qualitysafety.bmj.com/

http://group.bmj.com/

< Displaying data to make process performance visible< Determining if changes tested resulted in improve-

ment< Determining if we are holding the gains made by our

improvement< Allowing for a temporal (analytic) view of data versus

a static (enumerative) viewDisplaying data on a run chart is often the first step in

developing more complex Shewhart (control) charts4 5

and in the design of planned experiments.6 7 In thispaper, we briefly outline the construction, interpretationand use of run charts.

DEFINITION AND CONSTRUCTION OF A RUN CHART

A run chart is a graphical display of data plotted in sometype of order. The horizontal axis is most often a timescale (eg, days, weeks, months, quarters) but could alsoinclude sequential patients, visits or procedures. Thevertical axis represents the quality indicator beingstudied (eg, infection rate, number of patient falls,readmission rate). Usually, the median is calculated andused as the chart’s centreline. The median is requiredwhen using the probability-based rules to interpret a runchart (see below). The median is used as the centerlinebecause (1) it provides the point at which half theobservations are expected to be above and below thecenterline and (2) the median is not influenced byextreme values in the data. Goal lines and annotations ofchanges and other events can also be added to the runchart. Figure 1 shows an example of a run chart. As shownin figure 1, the run chart helps us understand and visu-alise the impact of different interventions and tests ofchange over time. To determine objectively when thesedata signal a process improvement, we use the medianand run chart rules described in the next section.The primary advantage of using a run chart is that it

preserves the time order of the data, unlike statistical testsof significance that generally compare two or more

aggregated sets of data. For example, the summarystatistic presented in figure 2 looks like there is improve-ment in the before to after data attributed to a change inthe system. Summary statistics for each of the three unitsshown in figure 2 where the change was tested producethe same pre-test mean and SD (70 min, 11.3 min) andpost-test mean and SD (30 min, 13.15 min). Further,a t-test produced a highly significant result (t22¼7.88,p<0.001). The question we want answered, however, is notwhether our change was statistically significant butwhether the change is associated with a sustainableimprovement in each unit where the change was tested.Data from Unit 1 would yield the bar chart in figure 2 andsupport the conclusion that we have achieved a sustain-able improvement. Data fromUnit 2 would yield the samebar chart but the data here reveal that improvement wasalready occurring before the change was tested. Data fromUnit 3 would also result in the same bar chart. In Unit 3,the change did result in improvement, however, it was notsustained. Viewing data over time rather than in summarystatistics yields richer data andmore accurate conclusionsfor improvement projects.

RULES TO HELP INTERPRET A RUN CHART

When improvement data are presented in healthcare(eg, clinical reports, dashboards, project updates, andboard reports), people will often over- or under-react toa single or most recent data point (and begin tampering,possibly making things worse).8 The terms ‘shift’ and‘trend’ are often used indiscriminately on a subjectivebasis as a means for moving a conversation or decisionforward, without recognition that statistical definitionsof such terms exist and rely on more than a single datapoint. The three probability-based rules below are usedto objectively analyse a run chart for evidence of non-random patterns in the data based on an a error ofp<0.05. Although there is nothing magical about the0.05 level of significance, it provides an objective statis-tical threshold for whether changes are leading toimprovement or degradation in a process and is consis-tent with typical practice in research. The thresholdcould be made more or less stringent based on theparticular situation, but an agreed upon approach iscritical to establish especially if many people are lookingat, and acting on, the data being presented. The rulesbelow are appropriate for quality improvement projects(where improvement is planned and expected) and havebeen shown to be effective in detecting signals in a widerange of healthcare applications.9 10

Rule 1dshiftSix or more consecutive points either all above or allbelow the median. Values that fall on the median do not

Figure 1 Example of a run chart demonstrating compliancewith a standard procedure.

BMJ Qual Saf 2011;20:46e51. doi:10.1136/bmjqs.2009.037895 47

ORIGINAL RESEARCH




add to nor break a shift. Skip all values that fall on themedian and continue counting.

Rule 2dtrendFive or more consecutive points all going up or all goingdown.11 If the value of two or more consecutive points isthe same, only count the first point and ignore therepeating values; like values do notmake or break a trend.

Rule 3drunsA non-random pattern is signalled by too few or too manyruns, or crossings of the median line.12 A run is a seriesof points in a row on one side of the median. If onlychance is influencing the process being measured witha run chart, then there should be a regularity at whichdata points go above and below the median to satisfy thiscondition. Some points can fall exactly on the medianline, which makes it hard to decide which run thesepoints belong to. An easy way to determine the numberof runs is to count the number of times the lineconnecting the data points crosses the median and addone. Tabled critical values are used to determine if toomany or too few runs exist (see table 1). Figure 3 showsan example of a run chart with too few runs, where it ispossible that an intervention is keeping the data fromdipping back down below the median which is where itwould tend to go if the data were random.

Rule 4dastronomical pointUsed in detecting unusually large or small numbers. Anastronomical data point is one that is obviously, even

blatantly, different from the rest of the points; allstudying the chart would agree the point is unusual.Astronomical points should not be confused with thehighest or lowest data points, which every run chart willhave. While Rules 1, 2 and 3 are probability based, Rule 4is subjective and recognises the importance of the visualdisplay of the data in a run chart.Figure 3 shows the four rules for identifying non-

random signals with actual run charts. Of course theserules, like any others, can only be understood in thecontext in which they are being applied. Nevertheless,having simple and consistent guidelines to distinguishbetween random variation and non-random signals is animportant part of learning from data.Good technique in using a run chart to learn about

the impact of changes is to create an initial median usingbaseline data. If the baseline data come from a processexhibiting no signals (shift, trend, runs, astronomicaldata point), extend or ‘freeze’ this initial median intothe future.4 By using the baseline data and extending themedian into the future, new data are not allowed toinfluence the initial median. Any changes in the newdata stand out against the baseline median more clearlyallowing for more accurate detection of signals ofimprovement (the probability-based rules are relative tothis median value). This is particularly important as wetry to understand the impact of different changes toa system over time.With a small amount of data, the median (and thus the

above rules) may not be useful. The shift and run rulesrequire more than 10 points before they are applicable.

Figure 2 Summary statisticsversus time-ordered data. (Eachunit has the same 24 data valuesordered differently over time.)

Cyc

le T

ime

(min

)

SD = 13.15SD = 11.36

t 887=)22( <p 100. , .


ORIGINAL RESEARCH




But the user should decide when to calculate the medianwhen starting the run chart. There are many applica-tions (eg, patient monitoring of annual PSA tests) wherethe run chart is useful with just three or four data pointsin order to get an early indication of central tendencyand trend. In the case of PSA testing, we usually do nothave the luxury of many data points but still need tolearn from the data to help form a clinical decision.

LIMITATIONS

As with any analytical tool, there are limitations to runcharts. First, run charts are designed for the earlydetection of signals of improvement or degradation ina process over time. However, run charts are not capableof determining if a process is stable (as defined byShewhart in relation to control charts only). Usingcontrol chart language with run charts can createconfusion because the two methods include differentrules for identifying non-random patterns. Shewhartcharts identify deviations from the centreline (mean, notthe median) using control limits. Figure 4 shows a runchart with no signals of non-random variation. Somemight be tempted to declare this process stable. Thesesame data when displayed on the appropriate Shewhartchart, however, reveal special causes. This process is notstable. Determining if a process is stable is important tounderstand if improvements have been sustained and topredict future performance which will impact decision-making. To determine if a process or system is in a stablestate, a Shewhart (control) chart is needed.4 5 13 In usingrun charts, we recommend avoiding the terms specialand common cause and stable or unstable, reservingtheir use for Shewhart (control) charts.Second, there are situations in healthcare settings

where the data are discrete and can make use of the runchart rules more complex. For example, if 50% or moreof the data on a run chart represents the absoluteextreme values on the scale (eg, 0 or 100% ona percentage scale), the criteria for detecting a non-random statistical signal using the median cannot beapplied. In these cases, the data will not yield a usefulmedian making application on the rules useless. Themedian will be the extreme absolute value itself (0% or100%). In these cases, one could use the mean as thecenterline (if no extreme data values were observed).Another option in this case would be to display the time(eg, days) or workload (eg, number of cases) between theevent on the run chart, plotting each time an eventoccurs.4 5 More time or workload between undesirableevents may be a sign of improvement. Although beyondthe scope of this paper, strategies to modify run chartsto address these types of unusual data situations areavailable.4 6

Table 1 Checking for too many or too few runs on a runchart. Table is based on about a 5% risk of falling the runtest for random patterns of data

Total number ofdata points onthe run chartthat do not fallon the median

Lower limit forthe number ofruns (< than thisnumber runs is‘too few’)

Upper limit forthe number ofruns (> than thisnumber runs is‘too many’)

10 3 911 3 1012 3 1113 4 1114 4 1215 5 1216 5 1317 5 1318 6 1419 6 1520 6 1621 7 1622 7 1723 7 1724 8 1825 8 1826 9 1927 10 1928 10 2029 10 2030 11 2131 11 2232 11 2333 12 2334 12 2435 12 2436 13 2537 13 2538 14 2639 14 2640 15 2741 15 2742 16 2843 16 2844 17 2945 17 3046 17 3147 18 3148 18 3249 19 3250 19 3351 20 3352 20 3453 21 3454 21 3555 22 3556 22 3557 23 3658 23 3759 24 3860 24 38

Source: Adapted from Swed and Eisenhart.12


ORIGINAL RESEARCH




Data line crosses onceToo few runs: total 2 runs

Figure 3 Rules for identifying non-random signals with run charts.

Figure 4 The same dataanalysed using a run chart anda Shewhart chart.

55

Run Chart(n=500)

40

45

50

% Unreconciled Medications

25

30

35

52423222120291817161514131211101987654321

55

Shewhart Chart(n=500)

40

45

50

Mean 41.9

UL 48.5

30

35

% Unreconciled Medications

25

52423222120291817161514131211101987654321


ORIGINAL RESEARCH




Third, as was shown in the example in figure 2, runcharts require judgement and understanding of thecontext and situation in which the data are collected andpresented because it is ultimately the context of a situa-tion that drives our predictions and goals. Lastly,healthcare providers and professionals are largelytrained in aggregate summary statistics and hypothesistesting paradigms which focus often on larger amountsof data at distant intervals. Using run charts, and otherstatistical process control tools, requires more regularmonitoring and data collection for the purposes ofbetter understanding the voice of the process (andsometimes the voice of the patient). Those leadingimprovement efforts using such tools should recognisethe potential for more frequent data collection overshorter time periods (see box online).

CONCLUSION

Run charts are easy to construct and simple to interpret.Since improvements are made over time, plotting dataover time using a run chart is a fundamental method toevaluate the success of improvement efforts in anobjective way. The run chart is therefore an importanttool with wide potential application in healthcareimprovement. Some healthcare organizations use runcharts as part of their process improvement effortspresently, but far more do not. Without some objectiveand simple measure of change and improvement we areleft with speculation, intuition, subjective assessments orthe application of inappropriate statistical approaches.There is growing recognition that quality measurementtools, such as run charts, are important for research andplanned experimentation that informs how we thinkabout and understand the systems and outcomes wewant to improve.6 It has long been advocated that

aggregate summary statistics always include measures ofdata in their natural time order as a means of acquiringknowledge.13 14

The value of a run chart is its simplicity and versatility inletting us learn from our data. By adding some proba-bility-based rules to aid interpretation, we get a picture ofthe process over time and a method to systematicallyidentify non-random signals. The information providedin this paper provides the necessary information to beginexploring and using run charts as an analytical tool forimprovement projects in healthcare.

Competing interests None.

Provenance and peer review Not commissioned; externally peer reviewed.

REFERENCES1. Ott E. Process Quality Control. New York: McGraw-Hill Book

Company, 1975:34e44.2. Neuhauser D, Diaz M. Quality improvement research: are

randomised trials necessary? Qual Saf Health Care 2007;16:77e80.3. Thor J, Lundberg J, Ask J, et al. Application of statistical process

control in healthcare improvement: systematic review. Qual SafHealth Care 2007;16:387e99.

4. Provost L, Murray S. The Healthcare Data Guide: Learning from Datafor Improvement. San Francisco: Jossey-Bass, Publication, 2011.

5. Benneyan JC, Lloyd RC, Plsek PE. Statistical process control asa tool for research and healthcare improvement. Qual Saf HealthCare 2003;12:458e64.

6. Moen RD, Nolan TW, Provost LP. Quality Improvement ThroughPlanned Experimentation. McGraw-Hill, 1999.

7. Speroff T, O’Connor G. Study designs for PDSA quality improvementresearch. Qual Manag Health Care 2004;13:17e32.

8. Deming WE. The New Economics: For Industry, Government,Education. Cambridge, MA: MIT Press, 1994.

9. Langley GJ, Moen RD, Nolan KM, et al. The Improvement Guide: APractical Approach To Enhancing Organizational Performance. SanFrancisco, CA: Jossey-Bass, 2009.

10. Lloyd RC. Quality Health Care: A Guide To Developing And UsingIndicators. Boston, MA: Jones and Bartlett, 2004.

11. Olmstead Pl. Distribution of sample arrangements for runs up anddown. Ann Math Stat 1945;17:24e33.

12. Swed FS, Eisenhart C. Tables for testing randomness of grouping ina sequence of alternatives. Ann Math Stat 1943;14:66e87.

13. Shewhart WA. The Economic Control Of Quality Of ManufacturedProduct. New York: D Van Nostrand, 1931.

14. Deming WE. On probability as a basis for action. Am Stat1975;29:146e52.


ORIGINAL RESEARCH




doi: 10.1136/bmjqs.2009.037895 2011 20: 46-51BMJ Qual Saf

Rocco J Perla, Lloyd P Provost and Sandy K Murray processeslearning from variation in healthcare The run chart: a simple analytical tool for

http://qualitysafety.bmj.com/content/20/1/46.full.htmlUpdated information and services can be found at:

These include:

Data Supplement http://qualitysafety.bmj.com/content/suppl/2011/01/20/20.1.46.DC1.html

"Web Only Data"

References http://qualitysafety.bmj.com/content/20/1/46.full.html#ref-list-1

This article cites 7 articles, 3 of which can be accessed free at:

serviceEmail alerting

the box at the top right corner of the online article.Receive free email alerts when new articles cite this article. Sign up in

Notes

http://group.bmj.com/group/rights-licensing/permissionsTo request permissions go to:

http://journals.bmj.com/cgi/reprintformTo order reprints go to:

http://group.bmj.com/subscribe/To subscribe to BMJ go to:


http://qualitysafety.bmj.com/content/20/1/46.full.html

http://qualitysafety.bmj.com/content/suppl/2011/01/20/20.1.46.DC1.html

http://qualitysafety.bmj.com/content/20/1/46.full.html#ref-list-1

http://group.bmj.com/group/rights-licensing/permissions

http://journals.bmj.com/cgi/reprintform

http://group.bmj.com/subscribe/



The Run Chart a Simple Analytical Tool

Documents

Transcript of The Run Chart a Simple Analytical Tool