STATISTICS POINT ESTIMATION Professor Ke-Sheng Cheng Department of Bioenvironmental Systems...

STATISTICS POINT ESTIMATION

Professor Ke-Sheng ChengDepartment of Bioenvironmental Systems Engineering

National Taiwan University

04/10/23Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ.

Annual flood peak water level (in cm)230 288 295282 275 462309 294 400249 245 299348 305 285360 375 330220 287 237295 210 278255 286 307195 500 300

2


Is it originated from a normal (or gamma) distribution???

3


Statistical inference(統計推論 )

• Given a random sample from the distribution of a population, we often are interested in making inferences about the population.

• Two important statistical inferences are – Estimation (推估 )– Test of hypotheses (假設檢定 ).

?

300Normal distribution?

4


Parameter estimation

• Assume that some characteristics of the elements in a population can be represented by a RV X with pdf fX

( ． ;θ), where the “form” of the density is assumed known except that it contains some unknown parameters θ.

• We want to estimate an unknown parameter θ or some function of the unknown parameter, τ(θ), using observed values of a random sample, x1,x2,…,xn.

5


• Point estimationLet the value of some statistic, say , represent the unknown parameter θ or τ(θ); such a statistic is called a point estimator.

),,( 1 nXXt

For example, and

respectively are point estimator of and .

n

iin x

nx

1

1

n

inin xx

ns

1

2)(1

1

6


• Interval estimationTwo statistics and , where , so that

( , ) constitutes an interval for which the probability can be determined that it contains the unknown parameter θ or

τ(θ).

),,( 11 nXXt ),,( 12 nXXt

),,( 11 nXXt ),,( 12 nXXt ),,( 11 nXXt ),,( 12 nXXt

7


Point Estimation of Distribution Parameters

• What parameters are to be estimated? – Parameters are variables that characterize

distributions. For example mean and standard deviation are parameters.

• How can we estimate the parameters?– methods of finding estimators – desired properties of point estimators

8


• To estimate parameters we need to have a random sample. For example, a random sample (x1,x2,…,xn) of f( ． ;θ) is collected in order to estimate the parameter θ. Therefore,

• There can be many (or infinite) ways of estimation and we need to establish some kind of criteria in order to have an adequate estimator.

),...,,(ˆ21 nxxxt

9


Estimators and Estimates

• Estimator: Any “statistic” whose values are used to estimate τ(θ), where τ( ． ) is some function of the parameter θ, is defined to be an estimator of . Note that an estimator is a random variable.

• Estimate: The estimated value given by an estimator.

)(

10


Methods of finding estimator

• Assume that x1,x2,…,xn is a random sample from a density f( ． ;θ), where the form of the density is known but the parameter θ is unknown. Further assume that θ is a vector of real numbers, say θ=(θ1,θ2,…,θk).

• We often let , called the parameter space, denote the set of possible values that the parameter θ can assume. Our objective is to find statistics to be used as estimators of certain functions, say , of θ= (θ1,…,θk).

)(),(),( 21 r

11


Method of moments

• Let f( ． ;θ1,θ2,…,θk) be the density of random variable X which has k parameters .

• In general will be a known function of the k parameters , i.e.

• Let x1, x2,…, xn be a random sample from the density f( ． ; ). From the k equations, we have

)( rr XE

k ,,1

k ,,1

k ,,1

12

krf krr ,...,2,1),...,(' 1


• We can solve a solution of expressed in terms of x1, x2,…, xn. The solution, i.e.

, is called an estimator of .

krfXn

M kr

n

i

rir ,...,2,1)ˆ,...,ˆ(

1' 1

1

),,( 1 k

),,( 1 k )ˆ,,ˆ( 1 k

13

04/10/23Laboratory for Remote Sensing Hydrology and Spatial Modeling, Dept of Bioenvironmental Systems Engineering, National Taiwan Univ. 14


Given a random sample of a normal distribution with mean μ and variance σ2. Estimate the parameters μ and σ by the method of moments.

ˆ1

','1

11

n

ini XX

nM

n

iiX

nM

1

2222

222122

1ˆˆ)(

n

XX

n

XnXXX

nX

nX

nn

ii

n

iin

ii

n

i

n

iii

1

2

2

1

2

2

1

2

1

2

1

22

)(

)()

1()

1(

1

15


• Note that the method of moment estimator of σ2 is NOT the sample variance.

n

ii XX

n 1

22 )(1

n

ii XX

nS

1

22 )(1

1

16


• Let x1, x2,…, xn be a random sample from a

Poisson distribution with parameter λ.

Estimate using MOM.

!);(

x

exf

x

,...2,1,0x

ˆ1''

111

n

iiX

nM

nX

17


Let be a random sample from a uniform distribution on . What are the method of

moments estimator of and ?

nXXX ,...,, 21

]3,3[

)(XE

3

3

22

2

)(32

1

)()(

dxx

XEXVar

n

ii XX

nM

111 ˆ

1

22

1

22

222 ˆˆ

1

n

iiX

nM

n

ii XX

n 1

221

18


Maximum Likelihood Method Definition of the likelihood function

19


Rationale of using likelihood function for parameter estimation

21


Definition of the maximum likelihood estimator

25


Example

28


Example

31


Example

32


21 2

1

Lower bound for y1

Upper bound for yn

21

1 y 21ny

21

21

1 yyn

)(

)();(),...,;(

],[

1],[

11

21

121

21

21

yy

n

ii

n

iin

nI

xIxfxxL

33


Example

36


)()()()32

1(

)()()32

1(),...,;,(

),[),

3[),

3[

]3,[1],3[1

11

1

nyyyn

nyy

nn

yIII

yIyIxxL

n

n

39


Parameter estimators of various distributions

42


Properties of point estimators

• Among different estimators, we want to know whether one estimator is “better” than others or what properties an estimator may or may not possess.

• Consider the case that we estimate using a statistic of a random sample from a density . Intuitively, we look for an estimator that is “close” to . However, what is the definition of “closeness”? (Or, how do we measure closeness?)

)(

)(

),...,( 1 nXXt);( xf

44


Most concentrated estimator

45


Pitman-closer and Pitman-closest

46


Unbiased estimator

• An estimator ( ) of is said to be unbiased if

• An unbiased estimator is said to be more efficient than any other unbiased estimator of , if

for all .

.)ˆ( E

ˆ )ˆ()ˆ( VarVar

47


Mean squared error of an estimator

48


• Naturally, we would prefer to find an estimator that has the smallest mean-squared error, however, such estimators rarely exist. In general, the mean-squared error of an estimator depends on .

• What should we look for if a uniformly minimum MSE estimator rarely exists?

49


Uniformly minimum MSE estimator

50


• Let’s define = bias of T . [Note: is an estimator of .] Then,

2

22

22

22

12

1

)]()([)(

)]()([})]({[

})]()([)]()()][([2)]({[

])]}()([)]([{[})]({[

)),...,((})](),...,({[)(

TETVar

TETETE

TETETETTETE

TETETETE

XXtTLetXXtEMSE nnt

)]ˆ([)()()( bTETb )(T )ˆ(

22 )ˆ()()]([)()( bTVarTbTVarMSEt

51


Bias


• We look for an estimator that has a uniformly minimum MSE within the class of unbiased estimators. Such an estimator is called a uniformly minimum-variance unbiased estimator (UMVUE).

55


Example

56


• The MSE of is

n

ii XX

n 1

22 )(1

2

1

22

1

2

2

1

22

]})(1

[{])(1

[

}])(1

{[

n

ii

n

ii

n

ii

XXn

EXXn

Var

XXn

E

2

44

42

2

22222

)1

3(

1)1(

]1

[)()1

(

nn

n

nn

n

n

nSVar

n

nn

58


Standard Error

• The “standard error” of a statistic is the standard deviation of its sampling distribution. If the standard error involves unknown parameters whose values can be estimated, substitution of those estimates into the standard error results in an “estimated standard error”.

60


Mean-squared-error consistency

• We discuss the mean-squared error of an estimator derived from a random sample of “fixed” size n. Properties of point estimators that are defined for a “fixed” sample size are referred to as “small-sample” properties, whereas properties that are defined for increasing sample size are referred to as “large-sample” properties.

62

Properties of the maximum likelihood estimators

• Asymptotic properties of the MLEs


• Invariance property of the MLEs

• The invariance property does not hold for unbiasedness.


STATISTICS POINT ESTIMATION Professor Ke-Sheng Cheng Department of Bioenvironmental Systems...

Documents

Transcript of STATISTICS POINT ESTIMATION Professor Ke-Sheng Cheng Department of Bioenvironmental Systems...