1

STAT 400 UIUC 6.4 - Maximum Likelihood Estimation,

Method of Moments Stepanov

Dalpiaz Nguyen

p.m.f. or p.d.f. f ( x ; q ), q Î W. W – parameter space. Example 1: Suppose W = { 1, 2, 3 } and the p.d.f. f ( x ; q ) is q = 1: f ( 1 ; 1 ) = 0.6, f ( 2 ; 1 ) = 0.1, f ( 3 ; 1 ) = 0.1, f ( 4 ; 1 ) = 0.2.

q = 2: f ( 1 ; 2 ) = 0.2, f ( 2 ; 2 ) = 0.3, f ( 3 ; 2 ) = 0.3, f ( 4 ; 2 ) = 0.2.

q = 3: f ( 1 ; 3 ) = 0.3, f ( 2 ; 3 ) = 0.4, f ( 3 ; 3 ) = 0.2, f ( 4 ; 3 ) = 0.1. What is the maximum likelihood estimate of q ( based on only one observation of X ) if … a) X = 1; b) X = 2; c) X = 3; d) X = 4.

Peyton : pep= population mean , 6 = pop .

Std.

T

parameter parameter

(santa. '¥÷:* mean,

"

iii.us.#> Iopredation is normally distributed .

N ( µ , 6)T t

⇒ Use sample data to estimate pi, 6. parameters of thea-what we want to estimate distribution

if -0=1 , then

I@tweeps's hat

x-- 4 is 0.2.

probabilityto estimate

of(x -- l; f =D = O.G ← I = I f (x- 2 ; -0=1) -- O. I

f (x - l ; 0=2) = 0.2 # f ( x=L; 0=2) = 0.3

f ( x- l ; 0=3) = 0.3 f- ( x=L; f -- 3) = O. y ← I =3

f (x=3; ⑦ =D = O.I

-

f (x -- 4; f -- l) = 0.2f (x=3; 0=2) - O. 3 ← I = 2 } ← I = I or 2f ( x=3; 8=3) = O. 2

f ( x = 4; f -- 2) = 0.2

f- ( x = 4; f =3) = O. I

* Note Maximum likelihood estimate may not be unique .

( ML E)

2

Likelihood function:

L ( q ) = L ( q ; x 1 , x 2 , … , x n ) = f ( x i ; q ) = f ( x 1 ; q ) × … × f ( x n ; q )

It is often easier to consider ln L ( q ) = ln f ( x i ; q ).

Maximum Likelihood Estimator: = arg max L ( q ) = arg max ln L ( q ). Method of Moments: E ( X ) = g ( q ). Set = g ( ). Solve for . Example 0: Consider a single observation X of a Binomial random variable with n trials and probability of “success” p. That is,

, k = 0, 1, … , n. a) Obtain the method of moments estimator of p, . b) Obtain the maximum likelihood estimator of p, .

Õ=

n

i 1

å=

n

i 1

θ̂

X θ~ θ~

knk ppkCnk --== 1 XP )(

)(

p~

p̂

( MLE)sample data ( n observations)-

Tparameter to be estimated

Tnatural log

estimates .P Idea: E (X) = I( Mom) t -

g CE) = I

population moment = sample moment

X - Binom ( n, p) astfwme n is known

Population moment Sample momentF- CX) = np I = X

so, rip = x ⇒F

Likelihood function :L ( p) = L ( p ; X) -- f ( X; p) = nc×p× ( I- p )

"-X

Elp) = In L ( p ) = lnlncxltxlnlpt In- X) ln ( I - p)

optdecp) = Ip - np = X-XP-nptXP-axp-n.PE Etop CI

-

p)

⇒p-.

d.a

check 2nd derivative:#

dpz-llp7-inplt-PJ-npllll-p7-pygopr.CI-

p)'

3

Example 2: Let X1 , X2 , … , X n be a random sample of size n from a Poisson distribution with mean l, l > 0. That is,

P ( X = k ) = , k = 0, 1, 2, 3, … .

a) Obtain the method of moments estimator of l, .

b) Obtain the maximum likelihood estimator of l, .

!λ

λ

kek -

λ~

l̂

-

unknown=

known f=

①← pmf of Poisson partmeter of

Poisson

Population:sampleth-ment.EC/7--④ I = E. Xi

⇒ I -- I ⇒=

does not have id

LCH -

- iitfcxisi)=T*e"

e-in .IT ÷

ICX -- Xi ;D)i'¥

=

lnlab) -- blnla) t=axle- X XX - e- d

. . . .

Ine- d-- -

incl) = O Poisson Cd) X , !'

Xz ! Xn !

lnfab) - lnlaltlnlb)= e-dndh.E.xi.nlIn (F) = Inca) - lncb) TIX ; !

Ince) = I i -- i

Nd)=lnLCN= - inkle) t.E.pl/ilnd-lniIIXi ! )= - in t Ii

,

Xi Ind - II. In ( Xi ! )

¥ecN= - nt ?÷ Eto ⇒ I=iEfi

4

Example 3: Let X 1 , X 2 , … , X n be a random sample of size n from a Geometric distribution with probability of “success” p, 0 < p < 1. That is, P ( X = k ) = ( 1 – p ) k – 1 p, k = 1, 2, 3, … . a) Obtain the method of moments estimator of p, .

b) Obtain the maximum likelihood estimator of p, .

p~

p̂

:ftp.Mor-nent : S#t :

Efx) = IF I = III⇒ feet I ⇒

"

Llp) = IT f ( Xi sp) = ¥1 ( i - p)"- ' pIC X -- Xi )r

= ( I - p )" - l

p. ( I - p)"- '

p. . . ( I -p)×n

- '

pGeom (p)

= ph ( i - p ) Ei, Xi-

n

f. ( p) = In Llp) = nlnlp) t ( FE,Xi - n) ln

#elp) -- F t C -D= F - E;YipI=nCtp)-tExinp ( I - p)

n - PEIXi - O= to

⇒ n - ri E. xi=o⇒p=Exi

5

Example 4: Let X 1 , X 2 , … , X n be a random sample of size n from the distribution with probability density function

f ( x ; q ) =

0 < q < ¥.

a) Obtain the method of moments estimator of q, .

Method of Moments: E ( X ) = g ( q ). Set = g ( ). Solve for . b) Obtain the maximum likelihood estimator of q, .

Likelihood function:

L ( q ) = L ( q ; x 1 , x 2 , … , x n ) = f ( x i ; q ) = f ( x 1 ; q ) × … × f ( x n ; q )

Maximum Likelihood Estimator: = arg max L ( q ) = arg max ln L ( q ).

ïïî

ïïí

ì -

££×

otherwise0

10θ1

θ θ1

xx

θ~

X θ~ θ~

θ̂

Õ=

n

i 1

θ̂

continuous

ftp.#ent : E (x) -- f! x - tf x'

dx -- tf fo"

x't

d× = to fix % dx=¥l÷÷÷l : I ⇐ ÷.S#ment : F ⇒ If = I ⇒ E=¥-i=t x

Lee ) = II fix is e) = II. et Xi'

= Ix ,'¥

. I x.'

. . . ¥xn¥= In I÷Xi¥ - ' inch - uh =

- in Co)

eco) = in Hot = nln (to) t in ( II Xi't ') = -nonce) t.E.in ( Xi't

'

)= - nine I - I ) ln (Xi ) = -mln E t ( to - I) Ezln (Xi)

#eco- I ⇐ -F -Eihf Eto ⇒ E=-IiEw x

6

c) Suppose n = 3, and x 1 = 0.2, x 2 = 0.3, x 3 = 0.5. Compute the values of the method of moments estimate and the maximum likelihood estimate for q. Example 5: Let X 1 , X 2 , … , X n be a random sample of size n from N ( q 1 , q 2 ), where

W = { ( q 1 , q 2 ) : – ¥ < q 1 < ¥, 0 < q 2 < ¥ }. That is, here we let q1 = µ and q2 = s2 . a) Obtain the maximum likelihood estimator of q 1 , , and of q 2 , .

b) Obtain the method of moments estimator of q 1 , , and of q 2 , .

1 θ̂ 2 θ̂

1 θ~

2 θ~

X = XltXzgtX3 = 0.2+0.33+-0.5 = ÷I.

'

I= - I[ ln (O. 2) t Cn (O. 3) t In (0.5)) €1.16890

% a"

Likelihood function :Pdf of Normal Co , .az ,

HG . a) = II. this G. ed = exp f-H]tex -- expend =f!⇒ )!It expf-lxijo.FI)

h

=H⇒) e-'

e

-¥¥". . . .

e-":*

.

→

=÷.sn#.e.E.-cxiiEiI=q.o..jnh.e-Eo.Eicxi-a5ICE.. Ez) = In L CQ . Ez) = -Z ln (2nA) -2¥77, ( Xi - G)

-

ln (e)

= - Eln (2nA) - If.

IF, ( Xi - fi )'

¥,

elf.. a) = - Izz ¥,

2 ( Xi - G) C- D=# II. ( Xi - fi)

⇐ eco-i.az) = - 2¥,

Ldn) t# ÷zCxi - ED'n

= -

Iot # Ei, Hi - -0,5set # eco

. .= o ⇒ i¥¥=o⇒÷icxi -G) - o

⇐ II. Xi - no , -- o ⇐o*

set # eco , .es -

- o ⇒ - Io.

+E7o⇒I¥¥iIo205 2-05

⇒- n + Eat:O ⇒ oI=÷E"→ this is not

the samplevariance

(b) Mom Xi . . . . . Xn N CQ , QD

Population :samplemo-ment.ECX) = f ,← first I ← first sample moment

2nd → ECX'

) = Var ( x) t# (XD- XT = ht Eax:

= Gz t -0T

⇒ s :÷÷: ⇐ *⇒H÷÷÷÷÷i*