Eight Common Statistical Traps -...
Transcript of Eight Common Statistical Traps -...
Eight Common Statistical Traps
Presented by Bob MitchellPast Chair, ASQ Statistics Division
Fellow, ASQ
Hiawatha SectionJanuary 8, 2004
ASQ Statistics Division
Vision:Statistical Thinking Everywhere
Process Variation Data Improvement
Statistical Thinking
Philosophy ActionAnalysis
Statistical Methods
Motto:We help keep you employable
Data Sanity
Statistical Thinking applied to everyday data
Statistical Thinking
• All work is a process• Variation exists in all processes• Knowledge and management of
variation are keys to success.
Process
ProcessOutputs
Customers
Feedback
Inputs
Suppliers
S I P O C
Process Metrics
Statistics
Statistics is not merely the science of analyzing data, but the art and science of collecting and analyzing data.
Use of Data as a Process
Given any improvement situation one must be able to:
1. Choose and define the problem in a process/ systems context
2. Design and manage a series of simple, efficient data collections
3. Use comprehensive methods presentable and understood across all layers of the organization (graphical analyses)
4. Numerically assess the current state, assess the effects of interventions, and hold the gains of any improvements made.
Use of Data as a ProcessPeople, Methods, Materials, Machines, Environment and Measurements inputs can be a source of variation for any one of the measurement, collection, analysis or interpretation processes!
Statistical TrapsCommon errors in data use, display and collection.
Organizations tend to have wide gaps in knowledge regarding the proper use, display and collection of data. These result in a natural tendency to either react to anecdotal data, or “tamper”.
Trap 1Treating all observed variation in a time series data sequence as Special Cause
• Most common form of “tampering” –treating common cause variation as special cause.
• Given two numbers, one will be bigger!
Two-Point Comparison
What Action is Appropriate?What Action is Appropriate?So
met
hing
Impo
rtan
t
This PeriodLast Period
Common vs. Special
Common CauseCommon Cause Special CauseSpecial Cause
It Depends!It Depends!
Example – Trap 1
Change from Change fromRegion Q4 (000) Q3 Q4 last year
Northeast 1148 17.6% 20.6%Southwest 1337 11.7% 11.8%Northwest 806 17.2% (8.2%)North Central 702 (5.5%) 4.7%Mid-Atlantic 781 (3.2%) (2.6%)South Central 359 (19.7%) (22.3%)
Northeast
Observation
Ind
ivid
ua
l V
alu
e
2018161412108642
1400
1200
1000
800
600
_X=1000.0
UC L=1367.5
LC L=632.4
Observation
Mo
vin
g R
an
ge
2018161412108642
600
450
300
150
0
__MR=138.2
UC L=451.6
LC L=0
1
1
I-MR Chart of Northeast
Southwest
Observation
Ind
ivid
ua
l V
alu
e
2018161412108642
1600
1400
1200
1000
_X=1195.9
UC L=1534.5
LC L=857.3
Observation
Mo
vin
g R
an
ge
2018161412108642
400
300
200
100
0
__MR=127.3
UC L=416.0
LC L=0
I-MR Chart of Southwest
Northwest
Obser vation
Ind
ivid
ua
lV
alu
e
2018161412108642
1400
1200
1000
800
600
_X=1025.9
UC L=1298.0
LC L=753.8
Obser vation
Mo
vin
gR
an
ge
2018161412108642
300
200
100
0
__MR=102.3
UC L=334.3
LC L=0
1
1
I-MR Chart of Northwest
North Central
Observation
Ind
ivid
ua
l V
alu
e
2018161412108642
700
600
500
400
_X=519.6
UC L=669.7
LC L=369.4
Observation
Mo
vin
g R
an
ge
2018161412108642
200
150
100
50
0
__MR=56.5
UC L=184.5
LC L=0
1
1
11
I-MR Chart of N. Central
Mid-Atlantic
Observation
Ind
ivid
ua
l V
alu
e
2018161412108642
800
700
600
500
_X=656.7
UC L=750.2
LC L=563.2
Observation
Mo
vin
g R
an
ge
2018161412108642
100
75
50
25
0
__MR=35.2
UC L=114.9
LC L=0
1
1
1
1
111
1
I-MR Chart of Mid-Atlantic
South Central
Observation
Ind
ivid
ua
l V
alu
e
2018161412108642
500
450
400
350
300
_X=420.3
UC L=534.2
LC L=306.4
Observation
Mo
vin
g R
an
ge
2018161412108642
160
120
80
40
0
__MR=42.8
UC L=140.0
LC L=0
I-MR Chart of S. Central
Trap 2
• Fitting inappropriate trend lines to a time-series data sequence
• Another form of tampering –attributing a specific type of special cause (linear trend) to a set of data that contains only common cause.
Trends
Example – Trap 2
It generally takes a run of length seven to declare a sequence a true trend. If the total number of observations is 20 or less, SIX continuously increasing or decreasing points can be used to declare a trend.
Trap 3
• Unnecessary obsession with and incorrect application of the Normal distribution
• A case of “reverse” tampering – treating special cause as common cause.
• Can cause misleading estimates and inappropriate prediction of process outputs.
Normal Distribution
The “Normal” Distribution is a distribution of data that has certain consistent properties.
These properties are very useful in our understanding of the characteristics of the underlying process from which the data were obtained.
Many natural phenomena and man-made processes are distributed normally, or can be represented as approximately normal.
ND PropertiesProperty 1Property 1:: A normal distribution can be described completely by knowing only the:
− Mean
− Standard deviation
Distribution OneDistribution One
Distribution TwoDistribution Two
Distribution ThreeDistribution Three
ND PropertiesProperty 2Property 2:: The area under sections of the curve can be used to estimate the cumulative probability of a certain “event” occurring
-4 -3 -2 -1 0 1 2 3 4
95%
99.73%
68%
Empirical RuleThe previous rules of probability apply even when a set of data is not perfectly normally distributed
Number of Standard
Deviations
Theoretical
Normal
Empirical – Almost any distribution
+/- 1σ 68%
60-75%
+/- 2σ 95%
90-98%
+/- 3σ 99.7%
99-100%
Example – Trap 33 potential vendors for new process
Customer information for paper tearTarget 26.5 g/cmSpecifications 23-30 g/cm
QC summary data (based on 30 jumbos from each vendor):
Vendor Mean StdevA 26.5 0.98B 26.6 0.67C 26.6 0.82
Cost of A < Cost of B < Cost of C
Graphical Analysis
24.4 25.4 26.4 27.4 28.4
Dotplot for Vendor A-Vendor C
Vendor A
Vendor B
Vendor C
Vendor CVendor BVendor A
28.5
27.5
26.5
25.5
24.5
Vendor
Tear
ANOVAOne-way ANOVA: Tear versus VendorSource DF SS MS F PVendor 2 0.160 0.080 0.12 0.890Error 87 59.931 0.689
Total 89 60.091
Individual 95% CIs For Mean Based on
Pooled StDev
Level N Mean StDev ---------+---------+---------+---------+
Vendor A 30 26.523 0.976 (--------------*--------------)
Vendor B 30 26.573 0.668 (--------------*--------------)
Vendor C 30 26.627 0.817 (--------------*--------------)
---------+---------+---------+---------+
26.40 26.60 26.80 27.00
p-value > .05; therefore, “no statistically significant difference”
Plot the Bloody Dots !
10 20 3024
25
26
27
28
29
Index
Vend
or A
10 20 3024
25
26
27
28
29
Index
Vend
or B
10 20 3024
25
26
27
28
29
Index
Vend
or C
Trap 4
• Incorrect calculation of standard deviation and sigma limits
• The traditional calculation of standard deviation typically yields a grossly inflated variation limit.
• Some people have arbitrarily changed decision limits to two standard deviations from the average.
Example – Trap 4
Observation
Indi
vidu
al V
alue
30272421181512963
5
4
3
2
1
_X=3.039
UCL=5.043
LCL=1.035
I Chart of Length of Stay
Decision limits based on overall standard deviation = 0.668Too wide for the process.
Example – Trap 4
Observation
Indi
vidu
al V
alue
30272421181512963
4.5
4.0
3.5
3.0
2.5
2.0
_X=3.039
UCL=4.049
LCL=2.028
11
1
1111
I Chart of Length of Stay
Decision limits based on process sigma.
Trap 5
• Misreading special cause signals on a control chart
• Just because an observation is outside the calculated three standard deviation decision limits does not necessarily mean that the special cause occurred at that point.
Example - Trap 5
Observation
Indi
vidu
al V
alue
30272421181512963
4.5
4.0
3.5
3.0
2.5
2.0
_X=3.039
UCL=4.049
LCL=2.028
11
1
1111
I Chart of Length of Stay
Special cause signals are the result of mean shifts
Example – Trap 5
Observation
Indi
vidu
al V
alue
30272421181512963
5
4
3
2
1
_X=4
UCL=4.754
LCL=3.246
1 2 3
I Chart of Length of Stay by Level
Correct Interpretation
Trap 6
• Choosing arbitrary cutoffs for “above” average and “below” average values
• There is actually a “dead band” of common cause variation on either side of an average that is determined from the data themselves.
Dead BandCoin Toss
With a “fair” coin, we expect about 50% headsWith 20 flips, would you be suspicious about the coin’s fairness…
– If someone flipped 11 heads?– If someone flipped 19 heads?
When would you become suspicious? – Draw lines below
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Confidence IntervalsFlipping 20 coins and looking for heads produces the following distribution:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Empirical Rule - Revisited
−3σ −2σ −1σ 0 −3σ −2σ −1σ 0 1σ 2σ 3σ 1σ 2σ 3σ
µ + 1σ (60-75% of Obs)
µ + 2σ (90-98% of Obs)
µ + 3σ (99-100% of Obs)
Example – Trap 6MD Incidents PTCA Incid / PTCA
1 1 36 2.78 %
2 4 53 7.55 %
3 4 79 5.06 %
4 2 58 3.45 %
5 5 110 4.55 %
Total 16 336 4.76 %
Are physicians 2 & 3 really “above average”?
By definition, approx 50% of the data will be below and 50% above the mean
Example – Trap 6
Trap 7
• Improving processes through arbitrary numerical goals and standards
• Any process output has a natural, inherent capability within a common cause range.
• Goals are merely wishes.
Trap 8
• Using statistical techniques on “rolling” or “moving” averages
• Another form of tampering –attributing special cause to a set of data that may only contain common cause variation, plus some structure.
• The rolling average technique creates the appearance of special cause
Rolling Averages
Structural VariationExamples:
– Sales steadily increasing (true trend)– Seasonal pattern in revenues– Business cycle patterns for orders
SALES
-1000
0
1000
2000
3000
4000
5000
6000
Mar
-95
Apr-9
5
May
-95
Jun-
95
Jul-9
5
Aug-
95
Sep-
95
Oct
-95
Nov
-95
Dec
-95
Jan-
96
Feb-
96
Mar
-96
Apr-9
6
May
-96
Jun-
96
Jul-9
6
Aug-
96
Sep-
96
Oct
-96
Nov
-96
Dec
-96
Jan-
97
Feb-
97
Mar
-97
MONTH, YR
Indi
vidu
al V
alue
Special Cause Flag
Seasonal or periodic patternTrend over time
For more information…
www.asqstatdiv.org
Thank You !