7 Outcome Measure..

8/13/2019 7 Outcome Measure..

1/25

Six Sigma Basics: Outcome

Measurement of ProcessPerformance Two Approaches for

Measuring ProcessPerformance A Well Known Example DPM Conversion Table Laboratory PerformanceIndicators as OutcomeMeasures

Benchmarks for ProcessPerformance Measurement Considerations What's the point? References


2/25

James O. Westgard, Ph.D.

Six Sigma Quality Management provides ageneral goal for process performance - six

sigmas of process variation should fit withinthe tolerance limits or quality requirement forthe product. Given this goal, it is important to

measure process performance to determinewhether improvement is needed. There aretwo different methodologies - one based on

counting the defects produced by theprocess (an outcome measure) and theother based on measuring the variability of

the process directly (a predictive measure).


3/25

Two Approaches for Measuring Process

PerformanceThe two approaches are illustrated in theaccompanying figure. The first approach is based on

the measurement of the outcome of the process,whereas the second is based on measurement of thevariation of the process. The outcome measurementapproach is applicable to any process, but usually

requires extensive efforts to collect and analyze thedata. Direct measurement of process variability allowsfor determination of process capability. This approachassumes a process whose distribution is stable and

can be characterized by repetitive measurements. Thissecond approach is more limited in application, but isadvantageous in order to plan, design, evaluate, andoptimize process performance prior to routine

operation and production.


4/25

This lesson is about the first methodology thatinspects the output of the process, estimates thedefect rate in terms of defects per million (DPM), thenconverts the defect rate to a sigma-metric to

characterize process performance.


5/25

A Well Known Example

A well-known tire manufacturer produced 6,000,000tires for a particular brand of sports utility vehicle. Itwas reported that there have been 2000 accidents

and 100 deaths due to failures of the tires. What's theperformance of the production process?

Step 1. Inspect the data to determine the number

of defects. In this case, a defect should be definedas a tire that causes an accident. The severity of theaccident determines the number of deaths.

Step 2. Calculate the defects per million (DPM).There were 2000 accidents observed in thosevehicles equipped with 6,000,000 tires.

2000/6,000,000 = 333 DPM


6/25

Step 3. Convert from DPM to a sigma metric

using a standard table from one of the Six Sigmatexts. Here's a short version of such a table.

Sigma Metric DPM without shift DPM with 1.5s shift

1.00 317,400 697,7002.00 45,400 308,6372.50 12,419 158,686

3.00 2,700 66,8073.50 465 22,7504.00 63 6,210

4.50 6.8 1,3505.00 0.57 2335.50 0.038 32

6.00 0.002 3.4


7/25


8/25

DPM Conversion TableEvery Six Sigma text includes a table forconverting DPM to a sigma metric, such as theone linked at the end of this lesson.The nomenclature often varies from table totable. Some tables used the terms Defects Per Million Opportunties (DPMO) and parts per million (ppm) , and some tables give the figures interms of percentages that can be related toprocess yield or percent within specifications , as

well as percent defectives . The sigma-metric isvariously referred to as sigma, sigma value,process sigma level, sigma level, and sigma

quality level.


9/25

The tables often contain two sets of conversions,one that corresponds to the defect rate that wouldbe observed if the process were to shift as muchas 1.5 sigma, and the other corresponding to the

defect rate expected if the process were properlycentered. These two distributions are shown inthe accompanying figure. The tabulated values in

the conversion tables correspond to right half ofthe distribution curves, with the DPM values fromthe y-scale being tabulated versus the sigma

values on the x-scale. It is obvious that theconversion based on the shifted distribution giveshigher values for the sigma metric than that fromthe centered distribution.


10/25

The rationale for using a 1.5s offset is discussed by Harry andSchroeder [1]:"By offsetting normal distribution by a 1.5 standard deviation on either side,the adjustment takes into account what happens to every process overmany cycles of manufacturing Simply put, accommodating shift and driftis our "fudge factor," or a way to allow for unexpected errors or movementover time. Using 1.5 sigma as a standard deviation gives us a strongadvantage in improving quality not only in industrial process and designs,but in commercial processes as well. It allows us to design products andservices that are relatively impervious, or "robust", to natural, unavoidable

sources of variation in processes, components, and materials."


11/25

Strictly speaking, it is more rigorous to use the "centered"conversion, but it is more common to use the conversion thatallows for a 1.5s shift. So common, in fact, that some of the SixSigma texts only provide a conversion table that allows for the1.5s shift. A more obvious reason for favoring that conversion isthat process performance always looks better, i.e., the sigma-

metric will always be higher. Harry and Schroeder explain thedifference between the two measures as follows [2]:

"So when companies claim that their processes are at six sigma,what they are really saying is that the short-term capability of their

processes is six sigma; the long-term performance, however, is 4.5sigma because of process centering errors.

The difference between the two conversions is important inapplications in healthcare laboratories. Outcome measures of

process performance may not be consistent with the predictivemeasures used in process design. To be consistent between themeasurement of process performance via outcomes or via predictiveperformance characteristics (i.e., direct estimation of the standarddeviation of the process), it would be preferable to use the"centered" conversion.


12/25

Laboratory Performance Indicators as Outcome

MeasuresThe use of outcome measures to characterize theperformance of laboratory related processes has beenillustrated by Nevalainen et al [3] in what appears tobe the first paper (April, 2000) published on Six SigmaQuality Management in the laboratory medicineliterature. The authors analyzed data from three

individual laboratories, as well as summaries ofperformance from 300 to 500 laboratories participatingin the College of American Pathologist's Q-Probeprogram. The original paper provides information on

sample size and the number of defects or errors,which were then converted into percent errors anddefects per million. DPM figures for representativequality indicators from the Q-Probe data are converted

to sigma values in the tables below.


13/25

Q-Probe QUALITY INDICATOR % ERROR DPM SIGMA*

Order accuracy 1.8 % 18,000 3.60Duplicate test orders 1.52 15,200 3.65Wristband errors (not banded) 0.65 6,500 4.00

TDM timing errors 24.4 244,000 2.20Hematology specimen acceptability 0.38 3,800 4.15

Chemistry specimen acceptability 0.30 3,000 4.25Surgical pathology specimen

accessioning 3.4 34,000 3.30

Cytology specimen adequacy 7.32 73,700 2.95

Laboratory proficiency testing 0.9 9,000 3.85Surg path froz sect diagnosticdiscordance 1.7 17,000 3.60PAP smear rescreening false negatives 2.4 24,000 3.45

Reporting errors 0.0477 477 4.80

*Conversion using table with allowance for 1.5s shift


14/25


15/25

Benchmarks for Process Performance

To provide some perspective on the observed sigma-metricsfor production processes, a number of benchmarks are oftencited:

"World class quality" is represented by 3.4 DPM or six

sigma performance. Airline baggage handling shows a 0.4% or 4000 DPMmishandling rate, which corresponds to a 4.15 sigmaprocess.

Airline safety has a very low fatality rate - 0.43 deaths permillion passenger flights, which is better than a 6 sigmaprocess.

Typical process performance in business and industry isoften suggested to be about 4 sigma.

Improving business process performance to achieve 5sigma is considered a significant improvement and often isset as the first level goal in a six sigma program.

Firestone production of tires for Ford Explorers appears tobe near the 5 sigma level on the basis of availableinformation on accident rates in the popular press.


16/25

With respect to these benchmarks, laboratory

processes are seen to approach averageperformance of business processes. Opportunitiesfor improvement abound! Achieving 5 sigmaperformance would be a significant improvement formost laboratory processes.However, the typical perspective in the healthcaresetting is that the observed quality is good enough,given the cost-control environment, demandingwork conditions, overloaded workers andmanagers, and burden of regulatory requirements

[4]. The "we are different" attitude still prevails, asshown by the statement "I'm not sure it's reasonableto expect laboratories to achieve the same defectrate as activities such as baggage handling." It isn't - we should do better!


17/25

Measurement Considerations

It appears to be simple procedure to count defects orerrors and convert them to a sigma-metric for processperformance. However, the design of the experiment,collection of data, and the analysis of that data are stillcritical.What's a defect? It's critical to define a defect or error.Remember the recent national dilemma concerning the

presidential vote in Florida and the difficulty of defininga defective ballot or vote. Similar things havehappened in the laboratory. One widely quoted studyabout the frequency of laboratory errors was a reviewof 363 incidents captured by the hospital's qualityassurance program [5]. Pre-analytic mistakesaccounted for 45.5% of those incidents, analytic errorsaccounted for 7.3%, and post-analytic for 47.2%.


18/25

How do you define an incident? Do you think the definition is uniform from hospital to

hospital? How do you define a medically significant analytic error? Does the size of a medically significant error vary from test

to test?

Is it possible that a medically significant analytic error couldoccur and there be no incident report?The study reported that the overall defect rate was 37.5 per100,000, or 375 DPM, which corresponds to an

approximately 4.9 sigma process. Considering that thismeasure of performance would be affected by all the pre-analytic, analytic, and post-analytic factors discussed earlier,the overall performance of the laboratory testing process inthis hospital far exceeds the sigma-metrics observed in Q-Probe studies. Is laboratory testing that much better in thishospital? Is it possible that "incidents" are not a reliable wayto count defects? Do you think that the accidents reported forFirestone tires on Ford Explorer vehicles identified all thedefective tires?


19/25

What sample size is needed? We sample 100

units and find 1 defect, which gives a 1% defect rateor 10,000 DPM or 3.8 sigma process performance.Would it be better to sample 1000 units or 10,000units or 100,000 units? Of course! But, given the

cost of sampling, is it necessary to sample 100,000units?One way to see the importance of sample size is todetermine the confidence interval for an estimatedproportion. This can be done by calculating thestandard deviation from the expression (pq/n)1/2,where p is the proportion, q is 1-p, and n is the

sample size [6]. An approximate 95% confidenceinterval can be described by taking the range p+2sto p-2s. For a process that has a true defect rate of1% or 0.01 proportion defective, the confidence

intervals would be as follows:


20/25

For n=100, the observed defect rate could be from 3.0%to 0.0%, giving estimates of process performance from4.25 sigma to greater than 6 sigma.

For n=1000, the observed defect rate would be from 1.6%to 0.37%, for process performance from 4.45 to 4.85sigma.

For n=10,000, the observed defect rate could be from1.2% to 0.8%, for process performance from 4.50 to 4.65

sigma. For n=100,000, the observed defect rate could be from

1.06% to 0.96%, for process performance from 4.55 to4.60 sigma.

Clearly a sample size of 100 is not sufficient to properlycharacterize the sigma capability of the process. A samplesize of 1000 gets us close. A sample size of 10,000 nails it

down pretty solid. A sample size of 100,000 isn't necessary.


21/25

Are there alternative measures to DPM? The DPMparameter is most commonly used to describe the qualityper unit of product. In situations where there may be severalquality attributes that could result in a defect for a given unit,the number of quality attributes are multiplied by the numberof units to express defects per million opportunities (DPMO) .In measuring the outcome of many healthcare processes,DPMO may be the better measure to take into account themany dimensions of quality that impact the outcome.For industrial processes, it is also common to describeprocess performance in terms of yield [7]. For example, aprocess with a 1% defect rate would provide a yield of 0.99(yield = 1 - proportion defective). The yield of sub-processesmay be measured, often called first pass yield or throughput

yield . The expected outcome of a process can be calculatedby combining the yields of the sub-processes to give what isvariously called a final yield , rolled yield , or rolled throughput yield . Some conversion tables include % Yield and DPMO asthe measures to be converted to a sigma metric.


22/25

What's the point?

Six Sigma Quality Management provides ageneral methodology for measuring the

outcome of any process and using that data tocharacterize process performance. Themeasure is the number of defective units,

services, or results, expressed per million.Standard tables are available in all Six Sigmatexts to convert the measured DPM to a

sigma-metric that describes the performanceof the process. This is a powerful methodologythat applies to virtually any kind of process.

The drawbacks are that:


23/25

the measurement occurs on the end-product ofthe process, therefore the process must be inproduction before it's performance can bemeasured;

the nature of a defect must be carefully defined;

the inspection process must be carefully plannedto properly assess the defects; a relatively large number of units must be

produced to get a reliable estimate of processperformance.

Fortunately for analytical testing processes, there aredirect measures of process performance that can be

used to predict the outcome and plan the processesto produce the desired quality. However, for otherservice processes in the laboratory, themeasurement of outcomes will be essential to

characterize the performance of the processes.


24/25

References

1. Harry M, Schroeder R. Six Sigma: The Breakthrough ManagementStrategy Revolutionizing the World's Top Corporations. NewYork:Currency, 2000, page 144.

2. Ibid, page 143.3. Nevalainen D, Berte L, Kraft C, Leigh E, Morgan T. Evaluating

laboratory performance on quality indicators with the six sigma scale.Arch Pathol Lab Med 2000;124:516-519.

4. Sarewitz SJ. Letter in response to Evaluation laboratory performancewith the six sigma scale. Arch Pathol Lab Med 2000;124:1748.

5. Ross JW, Boone DJ. Assessing the effect of mistakes in the totaltesting process on the quality of patient care. Proceedings of the1989 Institute on Critical Issues in Health Laboratory Practice.Centers for Disease Control, Atlanta, GA, 1991, page 173.

6. Davies OL, Goldsmith PL. Statistical Methods in Research andProduction. New York:Hafner Publishing Company, 1972, page 309.

7. Pande PS, Neuman RP, Cavanagh RR. The Six Sigma Way: HowGE, Motorola and Other Top Companies are Honing Their

Performance. New York:McGraw Hill, 2000, pp 229-231.


25/25

Other Six Sigma articles:Six Sigma DPM TableSix Sigma Quality ManagementSix Sigma and Requisite Laboratory QCSix Sigma Basics: Process improvement, goals, andmeasurements

Copyright 2001. All rights reserved.Westgard QC, 7614 Gray Fox Trail, Madison WI 53717

Call 608-833-4718 or e-mail us at [email protected] A Message from JOW

QC Lessons | QC Applications | Questions | MultiruleCLIA | What's New? | Catalog | Demo Download

Home | Glossary | ARCHIVES | Links | Feedback

7 Outcome Measure..

Documents

Transcript of 7 Outcome Measure..