Testing point null hypothesis, a discussion by Amira Mziou

Testing a point null hypothesis: IPE

Testing a point null hypothesis: TheIrreconcilability of P-values and Evidence

Authors: James O.Berger & Thomas SellkeSource: Journal of the American Statistical Association 1987

Presented by: MZIOU Amira

Reading Seminar in Statistical Classics: C.P RobertUniv Paris Dauphine

February

Outline

1 IntroductionStatistical Hypothesis TestP-value and Evidence in Statistics

2 Problematic

3 Posterior Probability, Bayes Factor and Posterior ODDS

4 Irreconcilability and Conflit between P-value and Pr(H0 \ x)

5 Various lower bounds on Posterior Probabilities

6 Solution

7 Conclusion

Outline

2 Problematic

6 Solution

7 Conclusion

Outline

2 Problematic

6 Solution

7 Conclusion

Outline

2 Problematic

6 Solution

7 Conclusion

Outline

2 Problematic

6 Solution

7 Conclusion

Outline

2 Problematic

6 Solution

7 Conclusion

Outline

2 Problematic

6 Solution

7 Conclusion

Introduction

2 Problematic

6 Solution

7 Conclusion

Introduction

Statistical Hypothesis Test

Let X be a characteristic of the population whose distributiondepends on an unknown parameter θ. We want to make adecision about the value of this parameter θ from a sample.

Question :How to decide on a population from the examinationof a sample from this population ?

Definition

A statistical hypothesis test is a method of making decisionsusing data, whether from a controlled experiment or anobservational study.

Introduction

Definition

Introduction

Definition

Introduction

Definition

Introduction

There are six steps to do a Statistical Hypothesis Test :

1 State hypothesis

2 Identify test statistic

3 Specify significance level

4 State decision rule

5 Collect data and perform calculations

6 Make statistical decision : reject or accept H0

Introduction

There are six steps to do a Statistical Hypothesis Test :1 State hypothesis

Introduction

P-value and Evidence in Statistics

2 Problematic

6 Solution

7 Conclusion

Introduction

P-Value and Evidence in statistics

P-value

Statistical hypothesis tests answer the question : Assuming thatthe null hypothesis is true, what is the probability of observing avalue for the test statistic that is at least as extreme as the valuethat was actually observed ?That probability is known as the P-value

Statistical evidence

A set or collection of numbers that prove a theory or story to betrue.

Introduction

P-value

Statistical hypothesis tests answer the question : Assuming thatthe null hypothesis is true, what is the probability of observing avalue for the test statistic that is at least as extreme as the valuethat was actually observed ?

That probability is known as the P-value

Introduction

P-value

Introduction

P-value

Introduction

Example

Let x = (x1,x2,...,xn) a sample of X = (X1,X2,...,Xn) where the Xi areiid N(θ,σ2)It’s desired to test the null hypothesis :

H0 : θ = θ0 VS H1 : θ 6= θ0

A classical test is based on consideration of test statistic T (X) wherelarge values of T (X) cause doubt on H0 .The P-value of observed data x is then

p = Prθ=θ0(T (X)≥ t = T (x)) (1)

T (X) =√

n|X −θ0|/σ (2)

Problematic

2 Problematic

6 Solution

7 Conclusion

Problematic

Most statisticians prefer use of P-values feeling it to be important toindicate how strong the evidense against H0

Given the observed value, is it likely that H0 is truei.e how high is P(H0\x) : the posterior probability of H0 giving x ?

Irreconcilability

By a Bayesian analysis with a fixed prior and for values of t chosen to yield agiven fixed p, the posterior probability of H0 is greater than p.=⇒ P-values can be highly misleading measures of the evidenceprovided by the data against the null hypothesis

Relationship between the P-value and conditional and Bayesianmeasures of evidence against the null hypothesis.

Problematic

Given the observed value, is it likely that H0 is true

i.e how high is P(H0\x) : the posterior probability of H0 giving x ?

Irreconcilability

Problematic

Irreconcilability

Problematic

Irreconcilability

Problematic

Irreconcilability

By a Bayesian analysis with a fixed prior and for values of t chosen to yield agiven fixed p, the posterior probability of H0 is greater than p.

=⇒ P-values can be highly misleading measures of the evidenceprovided by the data against the null hypothesis

Problematic

Irreconcilability

Problematic

Irreconcilability

Problematic

Irreconcilability

Posterior Probability, Bayes Factor and Posterior ODDS

2 Problematic

6 Solution

7 Conclusion

Posterior probability

X has density f (x \θ) , θ ∈Θ

π0 :Prior probability of H0(θ = θ0) θ0 ∈Θ

π1 :Prior probability of H1(θ 6= θ0) : π1 = 1−π0

We suppose that the mass on H1 is spread out according to a density g(θ).

The marginal density of X is :

m(x) = f (x/θ0)π0 + (1−π0)mg(x) (3)

where mg(x) =∫

f (x/θ)g(θ)dθ

Assuming that f (x/θ) > 0, the posterior probability of H0 is given by :

Pr(H0\x) = f (x\θ0)× π0

m(x)= [1 +

1−π0

f (x/θ0)]−1 (4)

m(x) = f (x/θ0)π0 + (1−π0)mg(x) (3)

where mg(x) =∫

f (x/θ)g(θ)dθ

Pr(H0\x) = f (x\θ0)× π0

m(x)= [1 +

1−π0

f (x/θ0)]−1 (4)

m(x) = f (x/θ0)π0 + (1−π0)mg(x) (3)

where mg(x) =∫

f (x/θ)g(θ)dθ

Pr(H0\x) = f (x\θ0)× π0

m(x)= [1 +

1−π0

f (x/θ0)]−1 (4)

m(x) = f (x/θ0)π0 + (1−π0)mg(x) (3)

where mg(x) =∫

f (x/θ)g(θ)dθ

Pr(H0\x) = f (x\θ0)× π0

m(x)= [1 +

1−π0

f (x/θ0)]−1 (4)

m(x) = f (x/θ0)π0 + (1−π0)mg(x) (3)

where mg(x) =∫

f (x/θ)g(θ)dθ

Pr(H0\x) = f (x\θ0)× π0

m(x)= [1 +

1−π0

f (x/θ0)]−1 (4)

m(x) = f (x/θ0)π0 + (1−π0)mg(x) (3)

where mg(x) =∫

f (x/θ)g(θ)dθ

Pr(H0\x) = f (x\θ0)× π0

m(x)= [1 +

1−π0

f (x/θ0)]−1 (4)

Posterior ODDS Ratio

Defintion

The Odds Ratio is a measure of effect size, describing the strength ofassociation or non-independence between two binary data values. It is usedas a descriptive statistic, and plays an important role in logistic regression.

The posterior ODDS ratio of H0 to H1 is :

PosteriorODDSRatio =Pr(H0/x)

1−Pr(H0/x)=

1−π0︸︷︷︸Prior ODDS Ratio

× f (x/θ0)

mg(x)︸︷︷︸Bayes Factor Bg(x)

Defintion

1−Pr(H0/x)=

× f (x/θ0)

Defintion

1−Pr(H0/x)=

× f (x/θ0)

Remark

• The Bayes factor Bg(x) =f (x/θ0)mg(x)

does not involve the priorprobabilities of the hypotheses. It’s the evidence reported by thedata alone.It can be interpreted as the likelihood ratio where thelikelihood of H1 is calculated with respect to g(θ)

Irreconcilability and Conflit between P-value and Pr(H0 \ x)

2 Problematic

6 Solution

7 Conclusion

Astronomer’s Example

An astronomer how learned that many statistical users rejectednull normal hypothesis at the 5% level when t = 1,96 wasobserved. He decides to do an experiment to verify the validityof rejecting H0 when t = 1,96 by testing normal hypothesis. Hesupposes that about half of the point nulls are false and half aretrue. When he concentrates attention on the subset in which t isbetween 1,96 and 2 , he discovers that 22% of null hypothesesare true.

Astronomer’s Example

An astronomer how learned that many statistical users rejectednull normal hypothesis at the 5% level when t = 1,96 wasobserved. He decides to do an experiment to verify the validityof rejecting H0 when t = 1,96 by testing normal hypothesis. Hesupposes that about half of the point nulls are false and half aretrue. When he concentrates attention on the subset in which t isbetween 1,96 and 2 , he discovers that 22% of null hypothesesare true.

Example(continued)

Again X ∼ N(θ,σ2), X ∼ N(θ, σ2

Suppose that π0 is arbitrary and g is N(θ0,σ2) ,

We have that

mg(x)∼ N(θ0,σ2(1 + n−1))

Bg(x) = f (x/θ0)mg(x)

= (1 + n)12 exp{− 1

2 t2/(1 + n−1)}

P(H0\x) = [1+ 1−π0π0Bg

]−1 =[1+ 1−π0π0

(1+n)−12 ×exp[ 1

2 t2/(1+n−1)]]−1

Example(continued)

Again X ∼ N(θ,σ2), X ∼ N(θ, σ2

Suppose that π0 is arbitrary and g is N(θ0,σ2) ,

We have that

mg(x)∼ N(θ0,σ2(1 + n−1))

Bg(x) = f (x/θ0)mg(x)

= (1 + n)12 exp{− 1

2 t2/(1 + n−1)}

P(H0\x) = [1+ 1−π0π0Bg

]−1 =[1+ 1−π0π0

(1+n)−12 ×exp[ 1

2 t2/(1+n−1)]]−1

Jeffrey’s Bayesian Analysis

Let π0 = 12 and g is N(θ0,σ

=⇒ Pr(H0 \ x) = (1 + (1 + n)−12 ×exp[ 1

2 t2(1 + n−1)])−1

FIGURE : Pr(H0 \ x) for Jeffreys Type prior

If n = 50 and t = 1,96 : classically we reject H0 at significance levelp = 0,05 , but Pr(H0) = 0,52 which indicates the evidence favors H0 .=⇒ the conflict between p and Pr(H0\x).

=⇒ Pr(H0 \ x) = (1 + (1 + n)−12 ×exp[ 1

2 t2(1 + n−1)])−1

If n = 50 and t = 1,96 : classically we reject H0 at significance levelp = 0,05 , but Pr(H0) = 0,52 which indicates the evidence favors H0 .

=⇒ the conflict between p and Pr(H0\x).

=⇒ Pr(H0 \ x) = (1 + (1 + n)−12 ×exp[ 1

2 t2(1 + n−1)])−1

If n = 50 and t = 1,96 : classically we reject H0 at significance levelp = 0,05 , but Pr(H0) = 0,52 which indicates the evidence favors H0 .=⇒ the conflict between p and Pr(H0\x).

Bad choice of priors

To prevent having a non-Bayesian reality, working withlowers bounds on Pr(H0\x) (translate into lower bounds onBg), and raisonable densities’ classes can be considered tobe objective.

Bad choice of priors

To prevent having a non-Bayesian reality, working withlowers bounds on Pr(H0\x) (translate into lower bounds onBg), and raisonable densities’ classes can be considered tobe objective.

Various lower bounds on Posterior Probabilities

2 Problematic

6 Solution

7 Conclusion

Lower bounds

This section examine some lower bounds on Pr(H0\x) when g(θ) thedistribution of θ given that H1 is true is alowed to vary within some classof distributions G.

1 GA : all distributions2 GS : all distributions symmetric about θ0

3 GUS : all unimodal distributions symmetric about θ0

4 GNOR :all N(θ0,τ2)

Letting

Pr(H0\x ,G)= infg∈GPr(H0\x)

B(x ,G) = infg∈GBg(x) =⇒ B(x ,G) = f (x\θ0)/supg∈Gmg(x)

Pr(H0\x ,G) = [1 + 1−π0π0× 1

B(x ,G) ]−1

Lower bounds

This section examine some lower bounds on Pr(H0\x) when g(θ) thedistribution of θ given that H1 is true is alowed to vary within some classof distributions G.

1 GA : all distributions2 GS : all distributions symmetric about θ0

3 GUS : all unimodal distributions symmetric about θ0

4 GNOR :all N(θ0,τ2)

Letting

Pr(H0\x ,G)= infg∈GPr(H0\x)

B(x ,G) = infg∈GBg(x) =⇒ B(x ,G) = f (x\θ0)/supg∈Gmg(x)

Pr(H0\x ,G) = [1 + 1−π0π0× 1

B(x ,G) ]−1

GA : all distribution

FIGURE : Comparison of P-values and Pr(H0 \ x ,GA) when π0 = 12

GS : all distributions symmetric about θ0

FIGURE : Comparison of P-values and Pr(H0 \ x ,GS) when π0 = 12

GUS : all unimodal distributions symmetric aboutθ0

FIGURE : Comparison of P-values and Pr(H0 \ x ,GUS) when π0 = 12

GNOR :all N(θ0,τ2)

FIGURE : Comparison of P-values and Pr(H0 \ x ,GNOR) when π0 = 12

Comparaison of the lower bounds B(x,G) for thefour G’s considered

FIGURE : Values of B(x ,G) in the normal example for different choicesof G.

Solution

2 Problematic

6 Solution

7 Conclusion

Solution

Again B(x ,G) can be considered to be a reasonable lower bound on the comparativelikelihood measure of evidence against H0 (under an unimodal symmetric prior on H1).

Replacing t by (t−1)+, the lower bound on Bayes factor is quite similar to the p-valueobtained by (t−1)+.

FIGURE : Comparison of P-values and B(x ,GUS)

Solution

Again B(x ,G) can be considered to be a reasonable lower bound on the comparativelikelihood measure of evidence against H0 (under an unimodal symmetric prior on H1).Replacing t by (t−1)+, the lower bound on Bayes factor is quite similar to the p-valueobtained by (t−1)+.

FIGURE : Comparison of P-values and B(x ,GUS)

Conclusion

2 Problematic

6 Solution

7 Conclusion

Conclusion

P-values can be dangerous to quantify evidence against apoint null hypothesis because they lead to talk aboutposterior distribution without going through BayesianAnalysis.What should a statistician desiring a point null hypothesisdo ?Lower bound of Pr(H0 \ x) can be argued to be useful totest evidence : if the lower bound is large we know not toreject H0 but if the lower bound is small we still do not knowif H0 can be rejected.For normal distribution, replacing t by (t−1)+,p-value= 0,05 can confirms the rejection of H0

Conclusion

REFERENCES

JAMES O.BERGER, THOMAS SELKE, TheIrreconcilability of P-value and Evidence, JOURNAL OFAMERICAN STATISTICAL ASSOCIATION (March 1987).

MATAN GAVISH, A Note on Berger and Selke

LINDLEY, D.V, The Use of Prior Probability Distributions inStatistical Inference and Decision, UNIVERSITY OFCALIFORNIA (1961)

LINDLEY, D.V, A Statistical Paradox, BIOMETRIKA(1957)

CHRISTIAN P.ROBERT,Faut-il Accepter ou rejeter lesp-values ? , PRIX DU STATISTICIEN DE LANGUEFRANCAISE

WIKIPEDIA31

Conclusion

THANK YOU FOR YOUR ATTENTION

Testing point null hypothesis, a discussion by Amira Mziou

Education

Transcript of Testing point null hypothesis, a discussion by Amira Mziou