Scientific Methods 1
-
Upload
daquan-banks -
Category
Documents
-
view
31 -
download
0
description
Transcript of Scientific Methods 1
10 Dec 2012 COMP80131-SEEDSM8 1
Scientific Methods 1
Barry & Goran
‘Scientific evaluation, experimental design
& statistical methods’
COMP80131
Lecture 8: Statistical Methods-Significance tests & confidence limits
www.cs.man.ac.uk/~barry/mydocs/MyCOMP80131
10 Dec 2012 COMP80131-SEEDSM8 2
Introduction
• Statistical significance testing has so far been applied on the assumption of a
1) discrete population with binomial distribution
2) continuous population with known normal pdf & stdev.• Before proceeding further, take a quick look at a few more
prob distributions & pdfs.• Significance testing can be adapted to any of these.
10 Dec 2012 COMP80131-SEEDSM8 3
Exponential pdf• Lifetimes, e.g. of light bulbs, follow an exponential distribution:
0:)/1(
0: 0 )( / xe
xxpdf x
0 1 2 3 4 5 6 7 8 9 100
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
0.5
x
mean = 2;x = 0:0.1:10;y = exppdf(x,mean);plot(x,y);
Mean =
Stdev = also
10 Dec 2012 COMP80131-SEEDSM8 4
Poisson Distribution
• λ, is both mean & variance of the distribution. • Poisson & exponential distributions are related. • If number of counts follows a Poisson distribution, then interval
between individual counts follows exponential distribution.• As λ gets larger, Poisson pdf normal with µ = λ, σ2 = λ.
integeran is x where!
)(x
exprob
xx
• For applications that involve counting number of times a random event occurs in a given amount of time, e.g. number of people walking into a store in an hour.
10 Dec 2012 COMP80131-SEEDSM8 5
Poisson distributions in MATLABx=0:16y = poisspdf(x,5);stem(x,y);
0 10 20 30 40 50 600
0.01
0.02
0.03
0.04
0.05
0.06
0.07
0.08
0.09
prob
(x)
x0 2 4 6 8 10 12 14 16
0
0.02
0.04
0.06
0.08
0.1
0.12
0.14
0.16
0.18
prob
(x)
x
x=0:60y = poisspdf(x,20);stem(x,y);
10 Dec 2012 COMP80131-SEEDSM8 6
Chi-squared distribution
0:
)2/(2
10: 0
)( 2/12/2/
2
xexV
xx xV
VV
•Given a population of normally distrib random variables with mean = 0 & stdev =1. •Randomly choose a sample of V observations of them.•Let x be the sum of their squares. •Then pdf of x has the 2 distribution:
(‘Gamma function’ (x) is generalisation of x! to non-integers).
If s = stdev of the V observations, pdf(s2) (1/V)V2(s2)
If pop mean = & stdev = , pdf (s2 ) (1/V)V2(s2/2+ 2)
10 Dec 2012 COMP80131-SEEDSM8 7
Plot chi2 pdf with V = 4
0 5 10 150
0.02
0.04
0.06
0.08
0.1
0.12
0.14
0.16
0.18
0.2
x
x = 0:0.2:15; y = chi2pdf(x,4); plot(x,y)
10 Dec 2012 COMP80131-SEEDSM8 8
Student’s t-distribution pdf
tVtVV
Vtpdf
V: )/1
)2/(
) 2/)1( ()(
2/)1(2
Depends on a single parameter V (degrees of freedom).
As V, t-pdf approaches standard normal distribution
If x is random sample of size n from a normal distribution with mean μ, then the t-statistic
stdev) samples&mean samplex(with /
ns
x
has Student's t-pdf with V = n – 1 degrees of freedom.
10 Dec 2012 COMP80131-SEEDSM8 9
Compare t-pdf(V=5) with normal
x = -5:0.1:5;y = tpdf(x,5);z = normpdf(x,0,1);plot(x,y,'b',x,z,'r');
-5 -4 -3 -2 -1 0 1 2 3 4 50
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
T-p
df(b
lue)
N
orm
(red
)
10 Dec 2012 COMP80131-SEEDSM8 10
MATLAB functions for t-dist
• pdf for t-distribution with V degrees of freedom: y = tpdf ( t,V);
(With samples with n values, V = n-1)
• Cumulative df with V degrees of freedom p = tcdf ( t , V) Prob of rand var being t
• Complementary df (area under ‘tail’ from t to ) p = 1 – tcdf ( t , V) Prob of rand var being > t
10 Dec 2012 COMP80131-SEEDSM8 11
Inverse-cdf in MATLAB
• If p = tcdf(t,V) then t = tinv(p,V)
Value of t such that prob of rand var being t is p• If p = normcdf(z,m,) then z = norminv(p,m, )
Value of z such that prob of rand var being z is p
• Complementary version:
t = tinv(1-p,V)
Value of t such that prob of rand var being > t is p.• Similarly for complementary version of norminv
t-pdf
x
t
p• Inverse of cumulative distrib function
t-pdf
x
t
p
10 Dec 2012 COMP80131-SEEDSM8 12
Significance testing: z-test• Assume Normal population with known stdev = .• Null-hypothesis: pop-mean = 0
• Alternative hyp: pop-mean < 0• Take one sample of n values & calculate the z-statistic:
stdev) pop&mean samplex(with /
0
n
xz
If pop-mean = 0, dist of z will be standard Normal (mean=0, std=1)
-2 -1 0 1 2 40
0.1
0.2
0.3
0.4
Std
Nor
mal
z
If mean of z is 0, how likely is a value z as just calculated?
p-value = prob (x z)
= 1-normcdf(z,0,1)
If p-value < significance level alpha () reject null-hyp.
10 Dec 2012 COMP80131-SEEDSM8 13
Alternative formulation
stdev) pop&mean samplex(with /
0
n
xz
Assuming we need 95% confidence, = 0.05Let z() = norminv(1-, 0, 1) = 1.65Prob of getting rand var 1.65 is less than 0.05If z 1.65, it is outside our 95% ‘confidence limit’ that the null-hyp may be true.So reject null-hyp.Confidence limit is for z is - to 1.65Neglect possibility that z may be negative.(1-tailed test)Confidence limit for sample-mean is - to 1.65/n + 0
10 Dec 2012 COMP80131-SEEDSM8 14
2-tailed teststdev) pop&mean samplex(with
/0
n
xz
Assuming we need 95% confidence, = 0.05.Allowing possibility that z < 0, extreme portions of tails are for z > z(/2)) and for z < -z(/2)). prob(z z(/2)) + prob(z -z((/2) ) = 2 prob(z z(/2))Now, z(/2) = norminv(1-/2,0,1) = 1.96Prob of getting rand var 1.96 or -1.96 is 0.05If z > 1.96 or z < - 1.96, it is outside our 95% ‘confidence limit’ that the null hyp may be true. So reject null-hyp.Confidence limits for z are -1.96 to 1.96Confidence limits for sample-mean are: 0 - 1.96/n to 0 + 1.96/n
10 Dec 2012 COMP80131-SEEDSM8 15
Significance testing: t-test• Assume Normal population with unknown stdev.• Null-hypothesis: pop-mean =0
• Alternative hyp: pop-mean < 0• Take one sample of n values & calculate the t-statistic:
stdev) sample&mean samplex(with /
0
sns
xt
If pop-mean = 0, dist of t will be standard t-pdf (blue) with V=n-1.
How likely is calculated value of t?
‘1-tailed’ p-value = prob (x t)
= 1 - tcdf(t , n-1)
If p-value < significance level alpha () reject null hyp.
t
-5 -4 -3 -2 -1 0 1 2 3 4 50
0.1
0.2
0.3
0.4
T-p
df(b
lue)
Nor
m-p
df(r
ed)
10 Dec 2012 COMP80131-SEEDSM8 16
Alternative formulation (2-tailed)
stdev) sample&mean samplex(with /
0
sns
xt
• Assuming we need 95% confidence, = 0.05• Confidence limits for 0 is:
• Null-Hyp is that pop-mean is 0
If value of 0 is outside these limits, reject the null-hyp that population mean is 0
If 0 is within these confidence limits, cannot reject null-hyp.
nsntinvxnsntinvx /)1,2/1( to/)1,2/1(
10 Dec 2012 COMP80131-SEEDSM8 17
Difference betw z-test & t-test(2-tailed)• With z-test pop-std () is known; with t-test is unknown.
stdev) pop&mean samplex(with /
0
n
xz
stdev) sample&mean samplex(with /
0
sns
xt
For z-test, p-value = prob ( x z) = 1- normcdf(z,0,1)For t-test, p-value = prob( x t) = 1 – tcdf(t,n-1)Same Null-hyp: pop-mean = 0 : reject if 0 outside conf limits
Confidence limits for z-test:
Confidence limits for t-test:
nxnx /)1,0 ,2/1(norminv to/)1,0, 2/1(norminv
nsntinvxnsntinvx /)1,2/1( to/)1,2/1(
10 Dec 2012 COMP80131-SEEDSM8 18
Non-Gaussian populations
• If samples of size n are ‘randomly’ chosen from a pop with mean & std , the pdf of their sample-means approaches a Normal (Gaussian) pdf with mean & stdev /n as n ∞.
• Regardless of whether the population is Gaussian or not!• This is Central Limit Theorem• Tests can be made to work for non-Gaussian populations
provided n is ‘large enough’.
10 Dec 2012 COMP80131-SEEDSM8 19
Meaning of confidence limits
If =0.5, there is 95% probability that the confidence limits for a given sample will contain the true population statistic say.
10 Dec 2012 COMP80131-SEEDSM8 20
A really subtle point
• Does this mean that there a 95% probability that lies within the 95% confidence limits for the given sample?
10 Dec 2012 COMP80131-SEEDSM8 21
A really subtle point
• Does this mean that there a 95% probability that lies within the 95% confidence limits for the given sample?
• No! A common mistake!• We have just one sample – we have no idea whether it is
one whose confidence limits contain or not.• Only 95% of possible samples will have conf limits which
contain .
10 Dec 2012 COMP80131-SEEDSM8 22
P-values & confidence limits in MATLAB
• Come for free with most measurements. For example: x= [1;2;3;4;5;6]; y =[1.1; 3;2;4;6;4]; [R, p_value, Rlo, Rup] = corrcoef(x,y)• Returns Pearson corr coeff R= 0.79, • p_value = 0.061, • Also 95% confidence limits: Rlo=-0.06, Rup = 0.98• 95% prob that the true corr lies between -0.06 & 0.98
• “ Returns p-values for testing the hypothesis of no correlation. Each p-value is probability of getting a correlation as large as the observed value by random chance, when the true correlation is zero. If p_value is small, say < 0.05, then the correlation is significant”.
10 Dec 2012 COMP80131-SEEDSM8 23
Credibility limits
• Baysian equivalent of ‘confidence limits’• If limits are C1 to C2, & = 0.05• Now there is 95% probability of the statistic, say, lying
between C1 & C2.• ‘Confidence limits are ‘frequentist’• Jonas explained why many people distrust the
frequentist approach and consider the Bayesian approach to be much more reliable.
10 Dec 2012 COMP80131-SEEDSM8 24
Reminder: Binomial distribution• If p=prob(Heads), prob of getting Heads exactly r times in n
independent coin-tosses is:
nCr pr (1-p)(n-r)
• For a fair coin. p=0.5, this becomes nCr /2n
0 2 4 6 8 10 12 14 16 18 2000.020.04
0.10.12
0.16
0.2
No of heads obtainable with n coin-tosses
Tru
e pr
obab
ility
of g
ettin
g th
at n
o of
hea
ds
10 Dec 2012 COMP80131-SEEDSM8 25
Binomist dist with n=6
0 1 2 3 4 5 60
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
No of heads obtainable with n coin-tosses
Tru
e pr
obab
ility
of g
ettin
g th
at n
o of
hea
ds
0.0156
10 Dec 2012 COMP80131-SEEDSM8 26
MATLAB Script
p = 0.5; % for coin tossing
n=6;
for r=0:n
nCr = prod(n:-1:(n-r+1))/prod(1:r);
Prob(1+r) = nCr * (p^r) * (1-p)^(n-r);
end;
Prob
figure(2); stem(0:n,Prob);
10 Dec 2012 COMP80131-SEEDSM8 27
Geometric distribution• p(x) = (1-p)px-1 (p = prob of success).• Number of trials (coin tosses) up to & including that in
which first failure occurs
1 2 3 4 5 6 7 8 9 100
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
0.5
x: number of trials
prob
of
first
fai
lure
at
x
p = 0.5x=1:10; prob = (1-p)*p.^(x-1);stem(x,prob);
10 Dec 2012 COMP80131-SEEDSM8 28
Geometric distribution (again)
1 2 3 4 5 6 7 8 9 100
0.05
0.1
0.2
0.3
0.4
0.5
x: number of trials
prob
of
first
fai
lure
at
x
prob(6) = 0.0156
prob(5) = 0.0313
10 Dec 2012 COMP80131-SEEDSM8 29
Barry’s Assignment
• Deadline 20 Dec 2012
• Email to [email protected] with ‘SEEDSM12’ in title
• or
• Hand in paper copy to SSO
• Exam statistics are in examdata.dat and examdata.xls in
• www.cs.man.ac.uk/~barry/mydocs/MyCOMP80131
• (or navigate from www.cs.man.ac.uk/~barry)