Post on 30-Dec-2015
description
Hypothesis Testing for Simulation 1
hypothesis testing with special focus on
simulation
Hypothesis Testing for Simulation 2
Hypothesis Test answers yes/no question with some statistical certainty
H0 = default hypothesisis a statement
Ha = alternate hypothesisis the precise opposite
Hypothesis Testing for Simulation 3
X = test statistic (RANDOM!)sufficient (uses all avail. data)often Z, T, N are used as notation
FX = its probability distribution
a = P[reject H0 | H0 true]
Hypothesis Testing for Simulation 4
ca = critical region for a
a = P[X in ca | H0]
a is our (controllable) risk
Hypothesis Testing for Simulation 5
TWISTED LOGIC
We WANT to reject H0 and conclude Ha, so...We make a very small, so...If we can reject, we have strong
evidence that Ha is true
This construct often leads to inconclusive results“There is no significant statistical
evidence that...”
Hypothesis Testing for Simulation 6
IMPORTANT
Inability to reject <> H0 true
Hypothesis Testing for Simulation 7
POWER OF THE TEST
b = P[X not in ca | Ha]
1-b = P[correctly rejecting]
Hypothesis Testing for Simulation 8
VENACULAR
a is type I errorProbability of incorrectly rejecting
b is type II errorProbability of incorrectly missing the opportunity to reject
Hypothesis Testing for Simulation 9
UNOFFICIAL VENACULAR
type III error – answered the wrong question
type IV error – perfect answer delivered too late
Hypothesis Testing for Simulation 10
EXAMPLE!
Dial-up ISP has long experience & knows...
50
)]([
STHdownloadE
Hypothesis Testing for Simulation 11
We get DSL, observe 12 samples
9.11ˆ
0.42
X
Hypothesis Testing for Simulation 12
IS DSL FASTER?
H0: mDSL = 50
Ha: mDSL < 50
test with P[type I] = 0.01
Hypothesis Testing for Simulation 13
PROBABILITY THEORY
Z ~ tn-1Must know the probability distribution of the
test statistic IOT construct critical region
n
XZ
ˆ
Hypothesis Testing for Simulation 14
for n = 12, a = 0.01, ca = -2.718
99% of the probabilityabove -2.718
Hypothesis Testing for Simulation 15
our test statistic-2.33
33.2
129.11
)5042(
Hypothesis Testing for Simulation 16
0.021 called the p-value
Given H0, we expect to see a test statistic as extreme as Z roughly 2% of the time.
-2.718(0.01)
-1.796(0.05)-2.33
(0.021)
Hypothesis Testing for Simulation 17
CONFIDENCE INTERVALS
la ua
For a given aP[la <= m <= ua] = 1-a
mBased on the sampleSo they are RANDOM!
Hypothesis Testing for Simulation 18
GOODNESS-OF-FIT TEST
Discrete, categorized dataRolls of diceMiss distances in 5-ft. increments
H0 assumes a fully-specified probability modelHa: the glove does not fit!
Hypothesis Testing for Simulation 19
TEST STATISTIC
2
1
22 ~
exp
)exp(
n
i i
ii
ected
ectedobsX
“chi-squared distribution with gnu degrees of freedom”
Hypothesis Testing for Simulation 20
n = observations - estimated param
Did you know... if Zi~N(0, 1), then
Z12+ Z2
2+...+ Zn2 ~ cn
2
Hypothesis Testing for Simulation 21
CELLS
H0 always results in a set of category cells with expected frequencies
EXAMPLECoin is tossed 100 timesH0: Coin Fair
Hypothesis Testing for Simulation 22
CELLS AND EXPECTED FREQUENCIES
EXPECT
H 50
T 50
Hypothesis Testing for Simulation 23
EXAMPLE
Cannon places rounds around a targetH0: miss distance ~ expon(0.1m)
Record data in 5m intervals(0-5), (5-10), ...(25+)
Hypothesis Testing for Simulation 24
EXPONENTIALS
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 10 20 30 40 50 60
F(x)
f(x)
x
x
exf
exF
)(
1)(
E(X)=1/l
Hypothesis Testing for Simulation 25
RESULTS
RIGHT OBS 1-exp(-0.1x) PROB EXPECT (OBS-EXPECT)^2
0.00
5.00 30 0.39 0.39 39.35 2.22
10.00 17 0.63 0.24 23.87 1.97
15.00 21 0.78 0.14 14.47 2.94
20.00 11 0.86 0.09 8.78 0.56
25.00 11 0.92 0.05 5.33 6.05
30+ 10 1.00 0.08 8.21 0.39
100.00 14.14
Hypothesis Testing for Simulation 26
0
5
10
15
20
25
30
35
40
45
0 10 20 30 40
observed
expected
Hypothesis Testing for Simulation 27
TEST RESULTS
Degrees of Freedom6 cells0 parameters estimatedn = 6
For the c62 distribution, the p-
value for 14.14 is about p=0.025
REJECT at any a > 0.025
Hypothesis Testing for Simulation 28
DIFFERENT H0
H0: the miss distances are exponentially distributed
Ha: the exponential shape is incorrect
We estimate the parameter, we lose one degree of freedom
Hypothesis Testing for Simulation 29
RESULTS 2
LEFT RIGHT OBS 1-exp(-0.0738x) PROBEXPE
CT (OBS-EXPECT)^2
0.00
0.00 5.00 30 0.31 0.31 30.86 0.02
5.00 10.00 17 0.52 0.21 21.34 0.88
10.00 15.00 21 0.67 0.15 14.75 2.65
15.00 20.00 11 0.77 0.10 10.20 0.06
20.00 25.00 11 0.84 0.07 7.05 2.21
25.00 30+ 10 1.00 0.16 15.80 2.13
7.95
Hypothesis Testing for Simulation 30
0
5
10
15
20
25
30
35
0 10 20 30 40
observed
expected
Hypothesis Testing for Simulation 31
n = 5
p-value for 7.83 is larger than 0.05
CANNOT REJECT
CONCLUSION?
Hypothesis Testing for Simulation 32
SIMULATION vs. STATISTICS
StatisticsSample is fixed and givenConclusion is unknownSignificance is powerful
SimulationSample is arbitrarily largeConclusion is knownWe need another thought about what is
meaningful
Hypothesis Testing for Simulation 33
SAMPLE SIZE EFFECT
m = 100s = 10
Hypothesis Testing for Simulation 34
HOW LARGE IS A DIFFERENCE BEFORE IT IS MEANINGFUL?
mu lower upper sigma
10 101.0468 98.16152 103.9321 5.547101100 101.3426 99.8384 102.8468 9.144828500 101.0861 100.3455 101.8266 10.06773
1000 100.8007 100.2762 101.3253 10.0847
97
98
99
100
101
102
103
104
105
0 200 400 600 800 1000
x-bar
lower
upper
Hypothesis Testing for Simulation 35
SUMMARY
You probably knew the mechanics of HT
You might have a new perspective