Likelihood Ratio Tests
-
Upload
boris-dean -
Category
Documents
-
view
88 -
download
1
description
Transcript of Likelihood Ratio Tests
Likelihood Ratio Tests
The origin and properties of using the likelihood ratio in hypothesis testing
Teresa Wollschied
Colleen Kenney
Outline
Background/History Likelihood Function Hypothesis Testing Introduction to Likelihood Ratio Tests Examples References
Jerzy Neyman (1894 – 1981)
Jerzy Neyman (1894 – 1981) April 16, 1894: Born in Benderry, Russia/Moldavia (Russian version:Yuri
Czeslawovich) 1906: Father died. Neyman and his mother moved to Kharkov. 1912:Neyman began study in both physics and mathematics at University
of Kharkov where professor Aleksandr Bernstein introduced him to probability
1919: Traveled south to Crimea and met Olga Solodovnikova. In 1920 ten days after their wedding, he was imprisoned for six weeks in Kharkov.
1921: Moved to Poland and worked as an asst. statistical analyst at the Agricultural Institute in Bromberg then State Meteorological Institute in Warsaw.
Neyman biography
1923-1924:Became an assistant at Warsaw University and taught at the College of Agriculture. Earned a doctorate for a thesis that applied probability to agricultural experimentation.
1925: Received the Rockefeller fellowship to study at University College London with Karl Pearson (met Egon Pearson)
1926-1927:Went to Paris. Visited by Egon Pearson in 1927, began collaborative work on testing hypotheses.
1934-1938: Took position at University College London 1938: Offered a position at UC Berkeley. Set up Statistical Laboratory
within Department of Mathematics. Statistics became a separate department in 1955.
Died on August 5, 1981
Egon Pearson (1895 – 1980)
August 11, 1895: Born in Hampstead, England. Middle child of Karl Pearson
1907-1909: Attended Dragon School Oxford 1909-1914: Attended Winchester College 1914: Started at Cambridge, interrupted by influenza. 1915: Joined war effort at Admiralty and Ministry of Shipping 1920: Awarded B.A. by taking Military Special Examination;
Began research in solar physics, attending lectures by Eddington
1921: Became lecturer at University College London with his father
1924: Became assistant editor of Biometrika
Pearson biography
1925: Met Neyman and corresponded with him through letters while Neyman was in Paris. Also corresponded with Gosset at the same time.
1933: After father retires, becomes the Head of Department of Apllied Statistics
1935: Won Weldon Prize for work done with Neyman and began work on revising Tables for Statisticians and Biometricians (1954,1972)
1939: Did war work, eventually receiving a C.B.E. 1961: Retired from University College London 1966: Retired as Managing Editor of Biometrika Died June 12, 1890
Likelihood and Hypothesis Testing
“On The Use and Interpretation of Certain Test Criteria for Purposes of Statistical Inference, Part I,” 1928, Biometrika: Likelihood Ratio Tests explained in detail by Neyman and Pearson “Probability is a ratio of frequencies and this relative measure
cannot be termed the ratio of probabilities of the hypotheses, unless we speak of probability a posteriori and postulate some a priori frequency distribution of sampled populations. Fisher has therefore introduced the term likelihood, and calls this comparative measure the ratio of the two hypotheses.
Likelihood and Hypothesis Testing
“On the Problem of the most Efficient Tests of Statistical Hypotheses,” 1933, Philosophical Transactions of the Royal Society of London: The concept of developing an ‘efficient’ test is expanded upon. “Without hoping to know whether each hypothesis is true or false,
we may search for rules to govern our behavior with regard to them, in following which we insure that, in the long run of experience, we shall not be too often wrong”
Likelihood Function
n
i
ixf1
n1
2
)|()|f(),...xx|L(
:by defined offunction the isfunction likelihood theobserved, isx =Xgiven
Then, ).|f( p.m.f.or p.d.f.on with distributi a from sample random a is X,...,X,X Suppose n1
x
x
Hypothesis Testing
Define T=r(x) R={x: T>c} for some constant c.
aa
o
H
H
:
: 0
Power Function
The probability a test will reject H0 is given by:
Size test:
Level test:
)()( RXP
10 ,)(sup o
10 ,)(sup o
Types of Error
Type I Error:Rejecting H0 when H0 is true
Type II Error: Accepting H0 when H0 is false
Likelihood Ratio Test (LRT)
LRT statistic for testing H0: 0 vs. Ha: a is:
A LRT is any test that has a rejection region of the form {x: (x) c}, where c is any number such that 0 c 1.
x)|L( sup
x)|L( sup )(
0
x
Uniformly Most Powerful (UMP) Test
Let be a test procedure for testing H0: 0 vs. Ha: a, with level of significance 0. Then , with power function (), is a UMP level 0 test if:(1) () 0
(2) For every test procedure ′ with (′) 0, we have
′() () for every a.
Neyman-Pearson Lemma
Consider testing H0: = 0 vs. Ha: = 1, where the pdf or pmf corresponding to i is f(x|i), i=0,1, using a test with rejection region R that satisfies
xR if f(x|1) > k f(x|0)
(1) and
xRc if f(x|1) < k f(x|0),
for some k 0, and
(2) ).(0 RXP
Neyman-Pearson Lemma (cont’d)
Then
(a) Any test that satisfies (1) and (2) is a UMP level test.
(b) If there exists a test satisfying (1) and (2) with k>0, then every UMP level test satisfies (2) and every UMP level test satisfies (1) except perhaps on a set A satisfying .0)()( 10 AXPAXP
Proof: Neyman-Pearson Lemma
)).(')(()(')(
x)]|x()|x()][x(')x([0 (3)
Thus, every x.for 0))|x()|x())(x(')x(( 1,)x('0 Since
).(' and )( are functionspower ingcorrespond the where test, levelother any for function test thebe )x('
and (2) and (1) satisfying test a offunction test thebe )x(Let
.R xif 0 (x) R, xif 1)x( :as function test theDefine
point. oneonly has since ,R)(XPR)(XPsup because test level
a hence and test size a have weR),(XP if that,Note
0011
01
01
c
0
0 0
0
k
dkff
kff
Proof: Neyman-Pearson Lemma (cont’d)
test.an UMP is ,in point only theis andarbitrary is ' Since ).(')( showing
)(')())(')(()(')(0
,(3) and 0 with Thus
.0)(' )(')( notingby proved is (a)
0
111
110011
000
c
k
k
Proof: Neyman-Pearson Lemma (cont’d)
A.set aon 0x)|x( whereperhaps, except, (1) satisfies ' ifonly 0 be will(3)in integrand
enonnegativ But the equality.an is (3) that implyingtest size a is ' is,that ,)(' test, level a is ' Since
.0)(')( )(' ,0 and (3), this, Using).(')( So, test.
level UMPa also is (2), and (1) satisfying test the, (a),By
test. level any UMPfor function test thebe 'let Now,(b) of Proof
A
i
0
000
11
df
k
LRTs and MLEs
x)|L(
x)|L( (x)
is statistic LRT Then the. of MLE thebelet and of MLE thebeLet 0
o
oo
Example: Normal LRT
2)(
20
1 1
220
2
)()(
12/2)(
n/2-
12/2)(
n/2-
0100
n1
20
1 1
220
0
(x)
.)()()( that Note
e
)(2
)(2
x)|L(
x)|L( (x)
:is statistic LRT Then the. :H versus :H :Test
.population 1) ,N( a from sample random a be ,...XXLet
xn
n
i
n
iii
xxx
n
ixx
n
ix
o
e
xnxxx
e
e
x
n
i
n
iii
i
i
Example: Normal LRT (cont’d)
We will reject H0 if (x) c. We have:
Therefore, the LRTs are those tests that reject H0 if the sample mean differs from the value 0 by more than
./)ln2(0 where}/)ln2( |:|{
}/)ln2( )(:{
}:{})(:{
0
20
2)( 2
0
ncncxx
ncxx
cexcxxxn
./)ln2( nc
Example: Size of the Normal LRT
test. size a is N(0,1),~th Z wi
2
)P(Z satisfies where
, || ifreject
: testThe. if )1,0(~)(n and },{
:have weexample, previous For the .c)(X)(P supsuch that c Chose
2/ 2/
2/0
0000
0
zz
n
zX
NX
Sufficient Statistics and LRTs
Theorem: If T(X) is a sufficient statistic for , and *(t) and (t) are the LRT statistics based on T and X, respectively, then *(T(x))=(x) for every x in the sample space.
Example: Normal LRT with unknown variance
statistic. t sStudent' on the based test a toequivalent iswhich
if
x)|,L(
x)|,L(
if 1
x)|,L(
x)|,L( max
x)|,L( max
x)|,L( max (x)
:is statistic LRT Then the.parameter) nuisance a is :(Note . :H versus :H :Test
.population ) ,N( a from sample random a be ,...XXLet
02
2
00
0
2
2
}0,: ,{
2
}0,: ,{
2
}0,: ,{
20100
2n1
20
2
22
20
2
Example: Normal LRT with unknown variance (cont’d)
2
0n/2-2
n/2-0
1
2)(
)(2n/2-
2
1
2)(
)(2n/2-
0
12/2)(
n/2-2
102/2)(
n/2-0
2
2
00
0
)(
)(
)(2
)(2
)(2
)(2
x)|,L(
x)|,L( (x)
then, If
1
2
0
1
20
0
n
n
ix
x
n
n
ix
x
n
n
ix
n
ix
in
i
i
in
i
i
i
i
e
e
e
e
Example: Normal LRT with unknown variance (cont’d)
.-X
when H rejecting toanalogous is this
and ')(1
1
)(1
1
(x)
whenc(x) Therefore,
)(1
)()(1
)(1
and
1)(
1
: thatNote
,12
00
;0
2
20
20
2
2
0
2
20
2
1
20
2
1
20
2
0
2
1
22
n
n
ii
n
ii
n
ii
t
nS
xc
Sx
nn
nn
xSn
n
Sn
n
xSn
nxnxx
nx
n
Sn
nxx
n
Asymptotic Distribution of the LRT – Simple H0
.[M(X)]Eth wi
c,c- ,x )M()|(log
:such that )on depend(both )M(function a and 0c , (5) sign. integral under the times threeateddifferenti be
can )|( and ,in continuous is derivative third the, to
respect with abledifferenti times threeis )|(density thex (4) point.interior an is valueparameter
true which theof set open an contains spaceparameter The (3) .in abledifferenti is
)|( and support,common some have )|( densities The (2) ).'|()|(then ,' if i.e., le;identifiab isparameter The (1)
:conditions regularity following the thesatisfies )|( and , of MLE theis ),|( iid are
,...XX suppose , :H versus :H gFor testin :Theorem
0
003
3
0
0
n10a 00
X
X
xxf
x
dxxf
xf
xfxfxfxf
xfxf
,
Asymptotic Distribution of the LRT – Simple H0 (cont’d)
.by specified parameters free ofnumber theandby
specified parameters free ofnumber e between th difference theison distributi limiting theof where
,(X) log 2- then , instead, If,
(X) log 2-
,n as ,Hunder Then
0
20
21
0
D
D
Restrictions
When a UMP test does not exist, other methods must be used. Consider subset of tests and search for a UMP test.
References
Cassella, G. and Berger, R.L. (2002). Statistical Inference. Duxbury:Pacific Grove, CA.
Neyman, J. and Pearson, E., “On The Use and Interpretation of Certain Test Criteria for Purposes of Statistical Inference, Part I,” Biometrika, Vol. 20A, No.1/2 (July 1928), pp.175-240.
Neyman, J. and Pearson, E., “On the Problem of the most Efficient Tests of Statistical Hypotheses,” Philosophical Transactions of the Royal Society of London, Vol. 231 (1933), pp. 289-337.
http://www-history.mcs.st-andrews.ac.uk/Mathematicians/Pearson_Egon.html
http://www-history.mcs.st-andrews.ac.uk/Mathematicians/Neyman.html