Bounded normal mean minimax estimation

SECTION 1SECTION 2

Estimating a Bounded Normal Mean

Jacopo Primavera

TSI-EuroBayes StudentUniversity Paris Dauphine

21 November 2011 / Reading Seminar on Classics

Jacopo Primavera Estimating a Bounded Normal Mean

SECTION 1SECTION 2

"ALL MODELS ARE WRONG, BUT SOME AREUSEFUL"

G. E. P. Box


SECTION 1SECTION 2

PRESENTING THE PROBLEM

One observation x ∼ N(θ,1)

θ ∈ [−m,m] ⊂ RSquared loss (θ − δ(x))2

R(θ, δ) = MSE(δ) = BIAS(δ) + VAR(δ)

MINIMAX ESTIMATOR δMM = argminδ

[supθ

R(θ, δ)]


SECTION 1SECTION 2

SUMMARY


SECTION 1SECTION 2

Outline

1 SECTION 1THE CANDIDATES2-POINTS PRIOR

2 SECTION 2AND WHEN m GETS LARGE?CLOSING THE LOOP


SECTION 1SECTION 2

THE CANDIDATES2-POINTS PRIOR

Outline




SECTION 1SECTION 2


SAMPLE MEAN

δSM = x

Main characteristicDOES NOT INVOLVEPRIOR INFORMATION

PropertiesΘ BOUNDED⇒ NOT Minimax


SECTION 1SECTION 2


MLE

δMLE (x)−m for x ≤ −mx for x ∈ (−m,m)

m for x ≥ m

Main characteristicA SELECTOR ESTIMATOR

PropertiesΘ BOUNDED⇒ δMLE dominates δSM


SECTION 1SECTION 2


DECISION THEORY

FORMALIZING THE CHOICE

Frequentist approachRESTRICT ∆

U = SET UNBIASED δ

CHOOSE UMVE

Decision-oriented approach(i) K OPTIMAL CRITERIA(ii) CHOOSE δ

MINIMIZING R(θ, δ)W.R.T K


SECTION 1SECTION 2


BAYES ESTIMATOR

(i) PROBABILITY MEASURE τ ON Θ

(ii) CRITERIA = Eτ

(iii) Eτ [R(θ, δ)] = r(τ, δ)

(iv) minδ

[r(τ, δ)] = r(τ, δB) = r(τ)

(v) δB BAYES RULE

Bayes method and Decision theoryNATURAL ORDERING CRITERIATHE VERSATILITY OF τ MAKES BAYESIAN METHODCOHERENT WITH DECISION THEORYWHICH PRIOR INDUCES MINIMAXITY ?


SECTION 1SECTION 2


GAME THEORY

Two-person zero-sum game

Θ Set of all possible strategies player 1A Set of all possible strategies player 2L Gain function (pl. 1) and loss function (pl.2)

RATIONAL PLAYER LOOK FOR A GUARANTEEWHATEVER OPPONENT’S MOVEMINIMAX STRATEGY ARISE NATURALLYMINIMAX STRATEGY FOR PLAYER TWO ≡ MAXIMINSTRATEGY FOR PLAYER ONE


SECTION 1SECTION 2


THE LINK

THE LINK

PLAYER II

PLAYER IMAXIMIN STRATEGYGAIN-ORIENTEDRATIONALITY

STATISTICIAN

NATURELEAST FAVORABLESTATE OF NATURELEAST FAVORABLEDISTRIBUTION


SECTION 1SECTION 2


THE LINK

THE LINK

PLAYER IIPLAYER I

MAXIMIN STRATEGYGAIN-ORIENTEDRATIONALITY

STATISTICIANNATURE

LEAST FAVORABLESTATE OF NATURELEAST FAVORABLEDISTRIBUTION


SECTION 1SECTION 2


THE LINK

THE LINK

PLAYER IIPLAYER IMAXIMIN STRATEGY

GAIN-ORIENTEDRATIONALITY

STATISTICIANNATURELEAST FAVORABLESTATE OF NATURE

LEAST FAVORABLEDISTRIBUTION


SECTION 1SECTION 2


THE LINK

THE LINK

PLAYER IIPLAYER IMAXIMIN STRATEGYGAIN-ORIENTEDRATIONALITY

STATISTICIANNATURELEAST FAVORABLESTATE OF NATURELEAST FAVORABLEDISTRIBUTION


SECTION 1SECTION 2


SIMPLE EXAMPLE

UNDER MINIMAXCOMPARE sup

θ[R]

δ1 OPTIMAL

UNDER BAYES

τ p.d.f. on Θ

COMPARE Eτ [R]

τ SUCH THAT δ1 � δ1

SUFFICIENT BIAS TOθ0


SECTION 1SECTION 2


SIMPLE EXAMPLE

UNDER MINIMAXCOMPARE sup

θ[R]

δ1 OPTIMAL

UNDER BAYESτ p.d.f. on Θ

COMPARE Eτ [R]

τ SUCH THAT δ1 � δ1

SUFFICIENT BIAS TOθ0


SECTION 1SECTION 2


LEAST FAVORABLE PRIOR

LEAST FAVORABLE FOCUSES ON θ’s MAXIMAL RISKPOINTS FOR A GENERIC BAYES RULE

LEAST FAVORABLE MAXIMIZE THE BAYES RISK

Lemmar(τ, δB

τ ) ≥ R(θ, δBτ ) ∀θ ∈ Θ ⇒

δBτ MINIMAX

τ LEAST FAVORABLE


SECTION 1SECTION 2


GUESSING MINIMAX DECISION

GUESS MAX. RISK PTS.

Suppose θ = +mLIKELY SAMPLES∈ [m − 1,m + 1]

HIGHLY BIASEDINTERVAL


SECTION 1SECTION 2




Suppose θ = +m

LIKELY SAMPLES∈ [m − 1,m + 1]



SECTION 1SECTION 2




Suppose θ = +mLIKELY SAMPLES∈ [m − 1,m + 1]



SECTION 1SECTION 2


Outline




SECTION 1SECTION 2


A BAYES RULE

CONCENTRATING ONTHE BOUNDS

τ◦m TWO-POINTS

PRIORδ◦m(x) =

m × tanh(m × x)


SECTION 1SECTION 2


A BAYES RULE

CONCENTRATING ONTHE BOUNDSτ◦m TWO-POINTS

PRIOR

δ◦m(x) =

m × tanh(m × x)


SECTION 1SECTION 2


A BAYES RULE

CONCENTRATING ONTHE BOUNDSτ◦m TWO-POINTS

PRIORδ◦m(x) =

m × tanh(m × x)


SECTION 1SECTION 2


SHRINKING TO THE BOUNDS


SECTION 1SECTION 2


CONDITIONS FOR MINIMAXITY

minimaxity of δ◦m depends on the interval width

Theorem

x ∼ N(θ,1)

θ ∼ [−m,m]

m ≤ m0

L Gaussian loss

δ◦m minimaxτ◦m least favorable

Numerical solution for m0

1.056742


SECTION 1SECTION 2


NUMERICAL EVIDENCE


SECTION 1SECTION 2


SKETCHING THE PROOF - 1ST STEP

Prove that r(τ◦m, δ

◦m) ≥ R(θ, δ

◦m) ∀θ ∈ Θ

R′ At most 3 sign chg(−+)(+−)(−+)

R′(0) = 0

R′(θ) = −R′(−θ)

Extremum for θ > 0 is(−+)

R even function⇒ Maximum attainedat 0 or at the bounds


SECTION 1SECTION 2




◦m) ≥ R(θ, δ

◦m) ∀θ ∈ Θ

R′ At most 3 sign chg

(−+)(+−)(−+)

R′(0) = 0

R′(θ) = −R′(−θ)




SECTION 1SECTION 2




◦m) ≥ R(θ, δ

◦m) ∀θ ∈ Θ


R′(0) = 0

R′(θ) = −R′(−θ)




SECTION 1SECTION 2




◦m) ≥ R(θ, δ

◦m) ∀θ ∈ Θ


R′(0) = 0

R′(θ) = −R′(−θ)


R even function

⇒ Maximum attainedat 0 or at the bounds


SECTION 1SECTION 2




◦m) ≥ R(θ, δ

◦m) ∀θ ∈ Θ


R′(0) = 0

R′(θ) = −R′(−θ)




SECTION 1SECTION 2


SKETCHING THE PROOF - CONCLUSION


◦m) ≥ R(θ, δ

◦m) ∀θ ∈ Θ

∃m0 such that R(m)≥ R(0) ∀m ≤ m0

r(τ◦m, δ

◦m) = 1

2R(−m) +12R(m) = R(m, δ

◦m)

= implies ≥⇒ Theorem is proved


SECTION 1SECTION 2




◦m) ≥ R(θ, δ

◦m) ∀θ ∈ Θ


r(τ◦m, δ

◦m) = 1

2R(−m) +12R(m) = R(m, δ

◦m)

= implies ≥

⇒ Theorem is proved


SECTION 1SECTION 2




◦m) ≥ R(θ, δ

◦m) ∀θ ∈ Θ


r(τ◦m, δ

◦m) = 1

2R(−m) +12R(m) = R(m, δ

◦m)

= implies ≥⇒ Theorem is proved


SECTION 1SECTION 2

AND WHEN m GETS LARGE?CLOSING THE LOOP

Outline




SECTION 1SECTION 2


HOW TO PROCEED

AS LONG AS Θ IS COMPACTR(θ, δτ ) analytic and 6= cost

⇓

THE NUMBER OF MAXIMAL RISK θ’s ISFINITESτ =SUPPORT OF τLS FINITE

IN GENERALcard{Sτ} INCREASES AS m INCREASES


SECTION 1SECTION 2


HOW TO PROCEED


⇓THE NUMBER OF MAXIMAL RISK θ’s ISFINITE

Sτ =SUPPORT OF τLS FINITE



SECTION 1SECTION 2


HOW TO PROCEED


⇓THE NUMBER OF MAXIMAL RISK θ’s ISFINITESτ =SUPPORT OF τLS FINITE



SECTION 1SECTION 2


GUESSING MINIMAX DECISION m>1

GUESS THE NEXT MAX.RISK PT.

Suppose θ = 0LIKELY SAMPLES∈ [−1,1]

LARGE RANGE (= 2)suppose θ 6= 0 and6= ±mSMALLER RANGE(≤ 2)


SECTION 1SECTION 2




Suppose θ = 0

LIKELY SAMPLES∈ [−1,1]



SECTION 1SECTION 2







SECTION 1SECTION 2





LARGE RANGE (= 2)

suppose θ 6= 0 and6= ±mSMALLER RANGE(≤ 2)


SECTION 1SECTION 2





LARGE RANGE (= 2)suppose θ 6= 0 and6= ±m

SMALLER RANGE(≤ 2)


SECTION 1SECTION 2







SECTION 1SECTION 2


3-POINTS PRIOR

the expression

δαm(x) = (1−α)mtanh(mx)

1−α+αexp(m22 )sech(mx)

when 1.4 ≤ m ≤ 1.6∃α∗ such that δαm(x) is minimax and ταm is least favorable


SECTION 1SECTION 2


NUMERICAL EVIDENCE


SECTION 1SECTION 2


4-PTS PRIOR

θ’s max risk pts w.r.t a generic bayes rule

MAX BIAS → BOUNDSMAX VARIANCE → PTS FAR FROM BOUNDS

Due to normal symmetryθMS ’s pop up pairwise around zero


SECTION 1SECTION 2


Outline




SECTION 1SECTION 2


GENERALIZED BAYES RULE

Generalizing the approach

(τn) SEQUENCE OFPROPER PRIORSlim−→ δn = δ0

δ0 GENERALIZEDBAYES RULE

Lemma

δ0 Generalized bayesrulelim−→[rn] ≥ Rδ0(θ) ∀θ<∞

δ0 MINIMAX RULE


SECTION 1SECTION 2




(τn) SEQUENCE OFPROPER PRIORS

lim−→ δn = δ0


Lemma


δ0 MINIMAX RULE


SECTION 1SECTION 2






Lemma


δ0 MINIMAX RULE


SECTION 1SECTION 2






Lemma

δ0 Generalized bayesrule

lim−→[rn] ≥ Rδ0(θ) ∀θ<∞

δ0 MINIMAX RULE


SECTION 1SECTION 2






Lemma

δ0 Generalized bayesrulelim−→[rn] ≥ Rδ0(θ) ∀θ

<∞

δ0 MINIMAX RULE


SECTION 1SECTION 2






Lemma


δ0 MINIMAX RULE


SECTION 1SECTION 2


NO MORE BOUNDS

Generalized bayesian approach

τn ∼ N(0,n)τn(θ|x) ∼ N( xn

n+1 ,n

n+1)

δn = xnn+1 → x = δ0

r(τn) =n

n+1 → 1R(θ, δ0) = 1⇒ δ0 MINIMAX


SECTION 1SECTION 2


NO MORE BOUNDS


τn ∼ N(0,n)

τn(θ|x) ∼ N( xnn+1 ,

nn+1)

δn = xnn+1 → x = δ0

r(τn) =n

n+1 → 1R(θ, δ0) = 1⇒ δ0 MINIMAX


SECTION 1SECTION 2


NO MORE BOUNDS



n+1 ,n

n+1)

δn = xnn+1 → x = δ0

r(τn) =n

n+1 → 1R(θ, δ0) = 1⇒ δ0 MINIMAX


SECTION 1SECTION 2


NO MORE BOUNDS



n+1 ,n

n+1)

δn = xnn+1 → x = δ0

r(τn) =n

n+1 → 1

R(θ, δ0) = 1⇒ δ0 MINIMAX


SECTION 1SECTION 2


NO MORE BOUNDS



n+1 ,n

n+1)

δn = xnn+1 → x = δ0

r(τn) =n

n+1 → 1R(θ, δ0) = 1

⇒ δ0 MINIMAX


SECTION 1SECTION 2


NO MORE BOUNDS



n+1 ,n

n+1)

δn = xnn+1 → x = δ0

r(τn) =n

n+1 → 1R(θ, δ0) = 1⇒ δ0 MINIMAX


SECTION 1SECTION 2


EVERY GOOD ESTIMATOR IS BAYES

Berger and Srinivasan 1978

θ natural parameter exponential familyquadratic lossEVERY ADMISSIBLE ESTIMATOR = GENERALIZEDBAYES ESTIMATOR

Wald 1950

Θ COMPACTRISK SET R CONVEXALL ESTIMATORS HAVE CONTINUOUS RISKFUNCTION⇒ BAYES ESTIMATORS’ SET = COMPLETE CLASS


SECTION 1SECTION 2




θ natural parameter exponential family

quadratic lossEVERY ADMISSIBLE ESTIMATOR = GENERALIZEDBAYES ESTIMATOR

Wald 1950



SECTION 1SECTION 2




θ natural parameter exponential familyquadratic loss

EVERY ADMISSIBLE ESTIMATOR = GENERALIZEDBAYES ESTIMATOR

Wald 1950



SECTION 1SECTION 2





Wald 1950



SECTION 1SECTION 2





Wald 1950Θ COMPACT

RISK SET R CONVEXALL ESTIMATORS HAVE CONTINUOUS RISKFUNCTION⇒ BAYES ESTIMATORS’ SET = COMPLETE CLASS


SECTION 1SECTION 2





Wald 1950Θ COMPACTRISK SET R CONVEX

ALL ESTIMATORS HAVE CONTINUOUS RISKFUNCTION⇒ BAYES ESTIMATORS’ SET = COMPLETE CLASS


SECTION 1SECTION 2





Wald 1950Θ COMPACTRISK SET R CONVEXALL ESTIMATORS HAVE CONTINUOUS RISKFUNCTION

⇒ BAYES ESTIMATORS’ SET = COMPLETE CLASS


SECTION 1SECTION 2





Wald 1950Θ COMPACTRISK SET R CONVEXALL ESTIMATORS HAVE CONTINUOUS RISKFUNCTION⇒ BAYES ESTIMATORS’ SET = COMPLETE CLASS


SECTION 1SECTION 2


AS m GOES TO INFINITY

Bickel 1981Minimize Fisherinformation w.r.t. anypriorcos2(π/2)× x , |x | ≤ 1

ρ(m) = 1− π2

m2 + o(m−2)

as m→∞


SECTION 1SECTION 2


THANK YOU

FOR

YOUR ATTENTION


Bounded normal mean minimax estimation

Education

Transcript of Bounded normal mean minimax estimation