Preferences of goodness-of-fit tests: A survey about the ... · A survey about the analysis of...

A survey about the analysis of nonparametric power functions

Preferences of goodness-of-fit tests: A surveyabout the analysis of nonparametric power

functions

Arnold Janssen

Mathematical Institute, Heinrich-Heine-University Duesseldorf

Ulm, September , 2015

Table of contents

1 Introduction

2 Local comparision of asymptotic nonparametric powerfunctions

3 Global nonparametric power functions are almost flat

4 Tests for hazard rates

5 Recommendation in practice

6 Making two-sample GOF tests distribution free if ties arepresent

7 References

Introduction

Outline

1 Introduction

7 References

Introduction

Kolmogorov-Smirnov (KS) test

asymptotics of a normalized empirical process under the null(critical regions), B0: Brownian bridge

omnibus tests are typically consistent

Introduction

What is the difference for more general critical lines?

Which type of tests can be recommended?

The statistician should analyze the goodness-of-fit tests of hiscomputer package in order to get some knowledge and animpression about their preferences.

Introduction

What is the difference for more general critical lines?

Which type of tests can be recommended?The statistician should analyze the goodness-of-fit tests of hiscomputer package in order to get some knowledge and animpression about their preferences.

Introduction

Discussion under local alternatives

about nonparametric power functionsclassification of tests: preferences for special alternativesmeaning and statistical interpretation of preferedalternatives

Introduction

History

Hájek/ Šidák (1967): asymptotic power of two-sample linearrank tests

Pitman asymptotic relative efficiency AREefficiency (ARE=1) for special alternativesalso ARE=0 show up

For KS test: Hájek/ Šidák (1967), Andel (1967):one-sided KS test is close to the median test(via local asymptotic relative efficiency)two-sided: Milbrodt and Strasser (1990)

Tool: the three Lemmata of Le Cam

Introduction

History

Hájek/ Šidák (1967): asymptotic power of two-sample linearrank tests

Pitman asymptotic relative efficiency AREefficiency (ARE=1) for special alternativesalso ARE=0 show up

For KS test: Hájek/ Šidák (1967), Andel (1967):one-sided KS test is close to the median test(via local asymptotic relative efficiency)two-sided: Milbrodt and Strasser (1990)

Tool: the three Lemmata of Le Cam

Local comparision of asymptotic nonparametric power functions

Outline

1 Introduction

7 References

“local parameters of order 1√n ” score functions g : (0,1)→ R

(or tangents g)H0 : λλ|(0,1) = F0 (after quantile transformation)

dFn,i.gdF0

(x) = 1 + cni g(x) + . . . 1 ≤ i ≤ n

cni =1√n one sample

Nonparametrics: parametric submodels are still present

Under local alternatives Png/√

n with g ∈ L02(0,1), where

L02(0,1) := {h ∈ L2(0,1) :

∫ 10 h(u) du = 0}, < h,g >=

∫ 10 hg dλλ:

√n(Fn(t)− F0(t))

D−−−−→Pn

X0(t) = B0(t)︸︷︷︸noise

0g(u) du︸︷︷︸signal

, t ∈ [0,1],

F0 continuous.

limit experiment: signal detection problem (SDP) for B0.Milbrodt and Strasser (1990)

Reparametrization

testing problem for the limit experiment:

H0 : Sg ≤ 0 versus H1 : Sg ≥ 0,Sg 6= 0,

whereSg(t) :=

0g(u) du, 0 ≤ t ≤ 1,

is the signal for g ∈ L02(0,1).

Le Cam theory: State of the art.

Convergence of the experiments, SDP limit experimentMain theorem of TestingConvergence of power functions towards the powerfunction of a limit test

Two-sided goodness-of-fit tests with convex acceptanceregions on suitable function spaces are asymptoticallyadmissible (Le Cam). Thus, there exists no overallasymptotically efficient goodness-of-fit test for all possibledirections of alternatives with dominating power function.

Le Cam theory: State of the art.

Convergence of the experiments, SDP limit experimentMain theorem of TestingConvergence of power functions towards the powerfunction of a limit testTwo-sided goodness-of-fit tests with convex acceptanceregions on suitable function spaces are asymptoticallyadmissible (Le Cam). Thus, there exists no overallasymptotically efficient goodness-of-fit test for all possibledirections of alternatives with dominating power function.

Example: One-sided Kolmogorov-Smirnov-Type Test

Let ρ : [0,1]→ R be a boundary function

ϕnKS =

supt∈[0,1](√

n (Fn(t)− t)− ρ(t)) 0 .0 ≤

Limit test for SDP is Kolmogorov-Smirnov-Type Test:

ϕKS =

supt∈[0,1](X0(t)− ρ(t)) 00 ≤

with X0(t) = B0(t) +∫ t

0 g(u) du for 0 ≤ t ≤ 1.

Let Pg be the distribution on C[0,1] of

X (t) = B0(t) +∫ t

0g(u)du = B0(t) + Sg(t) , 0 ≤ t ≤ 1

power of test ϕ for SDP (Girsanov formula)

g 7→ EPg (ϕ) =

∫ϕexp

(∫ 1

0gdB0 −

||g||2

principal component decompositionintegral test statistics: compact operators, Shorack/WellnerCramér von Mises test statistics, Neuhaus (1976)not for sup-statistics (KS-test), Milbrodt/Strasser (1990)

Let Pg be the distribution on C[0,1] of

X (t) = B0(t) +∫ t

0g(u)du = B0(t) + Sg(t) , 0 ≤ t ≤ 1

power of test ϕ for SDP (Girsanov formula)

g 7→ EPg (ϕ) =

∫ϕexp

(∫ 1

0gdB0 −

||g||2

principal component decompositionintegral test statistics: compact operators, Shorack/WellnerCramér von Mises test statistics, Neuhaus (1976)not for sup-statistics (KS-test), Milbrodt/Strasser (1990)

Comparison of power functions of tests ϕ along a path ϑ 7→ νϑof distributions

oracle test : Neyman Pearson test or(benchmark) best two-sided unbaised test

principle of local comparison at H0 : ϑ = 0

slope: ddϑEϑ (ϕ)|ϑ=0 curvature: d2

dϑ2 Eϑ (ϕ)|ϑ=0

Theorem (principal components of tests, J.(1995))

H0 = P0, h tangent h ∈ L(0)2 , ray t 7→ Pth. There exist a gradient

h0 ∈ L(0)2 and a Hilbert-Schmidt operator T : L(0)

2 → L(0)2 with

Eth(ϕ) = E0(ϕ)︸︷︷︸level α

+t < h,h0 >︸︷︷︸slope

2< h,T (h) >︸︷︷︸

curvature

+o(t2)

< h,T (h) > =∞∑

λi < h,hi >2, ||hi || = 1, λ1 ≥ λ2 ≥ . . . ≥ 0

||h0||2 +12

∞∑i=1

λ2i < α(1− α), α = E0(ϕ)

one-sided tests: h0||h0|| direction with highest slope

two-sided unbaised test: h0 = 0: h1 direction with highestcurvature (=highest preference for h0, h1 respectively)

Theorem (principal components of tests, J.(1995))

H0 = P0, h tangent h ∈ L(0)2 , ray t 7→ Pth. There exist a gradient

h0 ∈ L(0)2 and a Hilbert-Schmidt operator T : L(0)

2 → L(0)2 with

Eth(ϕ) = E0(ϕ)︸︷︷︸level α

+t < h,h0 >︸︷︷︸slope

2< h,T (h) >︸︷︷︸

curvature

+o(t2)

< h,T (h) > =∞∑

λi < h,hi >2, ||hi || = 1, λ1 ≥ λ2 ≥ . . . ≥ 0

||h0||2 +12

∞∑i=1

λ2i < α(1− α), α = E0(ϕ)

one-sided tests: h0||h0|| direction with highest slope

two-sided unbaised test: h0 = 0: h1 direction with highestcurvature (=highest preference for h0, h1 respectively)

Results for one- and two-sided KS-type tests

one-sided gradient h(a,b)0 (approximately), J./Kunz (2002),

Rahnenführer (2003)

h(a,a)0 ∼ sign(2u − 1)

score function of the mediantest

one-sided ϕ: local asymptotic relative efficiency in direction h

AREL(ϕ,h) =(

< h,h0 >

‖h‖ f (u1−α)

if < h,h0 > ≥ 0

f , u1−α N(0,1) density, 1− α quantile

AREL = 1 iff ϕ Neyman Pearson test (familiar Pitmaninterpretation of ARE)

numerical result: α = 0.05

(ϕKS,one-sided,h(a,a)

)≈ 0.715

unbaised two-sided GOFtests gradient: h0 = 0

first principal component h1 ofthe Hilbert-Schmidt operatorh1 ≈ h(a,b)

0 , J. (1995), Rah-nenführer (2003).

A.Connection to the score function of locationfamilies (x , ϑ) 7→ f (x − ϑ)

score function h ↑ : there exists a location family with score

function − f ′(F−1(u))f(F−1(u))

= h(u), Hájek/ Šidák (1967):

h(a,b)0 : f(a,b)(x) =

1a + b

)1(−∞,0](x) + exp

)1(0,∞)(x)

J./Ünlü (2008)

f(a,a) double exponential

B.Connection to differentiable stat. functionals κ

Koshevnik and Levit (1976)Pfanzagl, Wefelmeyer (1982)Bickel, Klaassen, Ritov, Wellner (1993)

κ : P → R (differentiable functional)

ddϑκ (Pϑ)|ϑ=0 =< g, κ >, κ canonical gradient (preference of κ)

quantile function: κ(F ) = F−1(

), κ(x) = h(a.b) ◦ F (x)

Conclusion: KS type test is a testfor F 7→ F−1

Global nonparametric power functions are almost flat

Outline

1 Introduction

7 References

SDP: B0(t) +∫ t

0 g(u) du,0 ≤ t ≤ 1, distribution Pg

H0 : P0

Theorem J. (2000) (almost flat global power)

Let ϕ be a test for the SDP {Pg : g ∈ L02(0,1)} with EP0(ϕ) = α.

For all ε > 0 and K > 0 there exists a subspace V ⊂ L02(0,1) of

finite dimension with

sup{|EPh(ϕ)− α| : h ∈ V⊥, ‖h‖ ≤ K} ≤ ε.

Moreover,

dim(V )− 1 ≤ ε−1α(1− α)(exp(K 2)− 1).

already contained in the text bookLehmann and Romano (2005), Chap. 14.6,Testing statistical hypotheses, third edition.also holds uniformly in the sample size n (J. (2003) )trick: C[0,1] and RN are Borel-isomorphic as polish spaces

Explaination: Nonparametric asymptotic power functions ofgoodness-of-fit tests are almost flat. The statistician has a“total amount of power” which can be distributed along theorthonormal directions. Each test distributes the power in adifferent way.Dual description via level points of GOF tests

already contained in the text bookLehmann and Romano (2005), Chap. 14.6,Testing statistical hypotheses, third edition.also holds uniformly in the sample size n (J. (2003) )trick: C[0,1] and RN are Borel-isomorphic as polish spacesExplaination: Nonparametric asymptotic power functions ofgoodness-of-fit tests are almost flat. The statistician has a“total amount of power” which can be distributed along theorthonormal directions. Each test distributes the power in adifferent way.Dual description via level points of GOF tests

Tests for hazard rates

Outline

1 Introduction

7 References

Gaussian shift experiment: SPD with Brownian motion B

X (t) := B(t) +∫ t

0h(u)du, 0 ≤ t ≤ 1

limit model for hazard rates λ(t) = f (t)1−F (t)

λϑλ(u)− 1 =

h(u)√n

+ . . .

the same result for global power functions

KS type test for SDP for Brownian motion:

ϕ = 1(

supt≤1

Xt ≥ c)

Rényi test

Rahnenführer (2003): h0 ≈ constant“close to the log-rank test (Savage test)”

Related to the K-transformation of Khmaladze (1981, 1983)

Fn = Mn + An Mn martingale, An compensator

“tests based on Mn instead of Fn”= testing hazards instead of densities

Two-sided: Drees/Milbrodt (1994)

KS type test for SDP for Brownian motion:

ϕ = 1(

supt≤1

Xt ≥ c)

Rényi test

Rahnenführer (2003): h0 ≈ constant“close to the log-rank test (Savage test)”Related to the K-transformation of Khmaladze (1981, 1983)

Fn = Mn + An Mn martingale, An compensator

“tests based on Mn instead of Fn”= testing hazards instead of densities

Two-sided: Drees/Milbrodt (1994)

Recommendation in practice

Outline

1 Introduction

7 References

Ask the scientist about his preferences for alternatives interms of statistical functionals (median, quantiles,interquartile distance, . . . )calculate the canonical gradients of the functionals,g1, . . . ,gk (= prefered directions of the functionals)

select a GOF test with high power on the subspace ofalternatives span(g1, . . . ,gk ) or

one-sided: Likelihood ratio test for the cone{∑ki=1 βigi : βi ≥ 0

}two-sided: h1, . . . ,hd O.N. basis of span(g1, . . . ,gk )

Neyman’s smooth test Tn =∑d

(1√n

∑ni=1 hj(xi)

Ask the scientist about his preferences for alternatives interms of statistical functionals (median, quantiles,interquartile distance, . . . )calculate the canonical gradients of the functionals,g1, . . . ,gk (= prefered directions of the functionals)select a GOF test with high power on the subspace ofalternatives span(g1, . . . ,gk ) or

one-sided: Likelihood ratio test for the cone{∑ki=1 βigi : βi ≥ 0

}two-sided: h1, . . . ,hd O.N. basis of span(g1, . . . ,gk )

Neyman’s smooth test Tn =∑d

(1√n

∑ni=1 hj(xi)

Making two-sample GOF tests distribution free if ties are present

Outline

1 Introduction

7 References

two i.i.d. samples n = n1 + n2, min(n1,n2)→∞

X1, . . . ,Xn1 Xn1+1, . . . ,Xn

edf Gn edf Hn

Theorem (J. 1994)Under the null H0 : Xi ∼ F i.i.d.

Tn :=(n1n2

supt∈R

∣∣∣Gn(t)− Hn(t)∣∣∣ D−−−→

n→∞supt∈R|B0 (F (t))|

“depends on F for tied data.”works also for other norms on D[0,1].

How to get critical values at finite sample size?

Fix X1, . . . ,Xn and use the permutation test. Let

σ : {1, . . . ,n} → {1, . . . ,n}

be random uniformly distributed permutation, independentof the X ’s.Critical values are taken from the permutation distribution

σ 7→ Tn(Xσ(1),Xσ(2), . . . ,Xσ(n)

Result: Power of the permutation test is equal to the power ofthe unconditional test if F for H0 would be known.

How to get critical values at finite sample size?

Fix X1, . . . ,Xn and use the permutation test. Let

σ : {1, . . . ,n} → {1, . . . ,n}

be random uniformly distributed permutation, independentof the X ’s.Critical values are taken from the permutation distribution

σ 7→ Tn(Xσ(1),Xσ(2), . . . ,Xσ(n)

Result: Power of the permutation test is equal to the power ofthe unconditional test if F for H0 would be known.

References

Outline

1 Introduction

7 References

References

Andel, J., 1967. Local asymptotic power and efficiency of tests of Kolmogorov-Sminov type. Ann. Math. Stat.38, 1705–1725.

Bickel, P.J., Klaassen, C.A.J., Ritov, Y., Wellner, J.A., 1993. Efficient and Adaptive Estimation forSemiparametric Models. Johns Hopkins University Press, Baltimore.

Drees, H., Milbrodt, H., 1994. The one-sided Kolmogorov-Smirnov test in signal detection problems withGaussian white noise. J. Stat. Plann. Inference 29, 325–335

Hájek, J., Šidák, Z., 1967. Theory of rank tests. Academic Press, New York.

Janssen, A., 1994:Two-sample goodness-of-fit tests when ties are present. J. Stat. Plann. Inference 39,399-424.

Janssen, A., 1995. Principal component decomposition of non-parametric tests. Probab. Theory RelatedFields 101, 193–209.

Janssen, A., 1999. Testing nonparametric statistical functionals with applications to rank tests. J. Stat.Plann. Inference 81, 71-93. Erratum (2001), 92, 297.

Janssen, A., 2000. Global power functions of goodness of fit tests. Ann. Statist.28, 239–253.

Janssen, A., 2003. Which power of goodness of fit tests can really be expected:Intermediate versuscontiguous alternatives. Statist. Decisions 21, 301–325.

Janssen, A., Kunz, M., 2002. Global extrapolations for power functions of one-sided nonparametric tests.Stat. Decis. 20, 153-176.

References

Janssen, A., Kunz, M., 2004: Brownian Type Boundary Crossing Probability for Piecewise Linear BoundaryFunctions. Commun. Statist. - Theory Meth., Vol.33, 1445-1464.

Janssen, A., Ünlü, H., 2008: Regions of alternatives with high and low power for goodness-of-fit tests. J.Stat. Planning Inference 138, 2526-2543.

Khmaladze, É.V., 1981. A martingale approach in the theory of goodness-of-fit-tests. (Russian) Teor.Veroyatnost. i Primenen. 26, 246–265.

Khmaladze, É.V., 1983. Martingale limit theorems for decomposable statistics. (Russian) Teor. Veroyatnost. iPrimenen. 28, 504–520.

Koshevnik, Y.A., Levit, B.Y., 1976. On a nonparametric analogue of the information matrix. Theory Probab.Appl. 21, 738–753.

Lehmann, E.L., Romano, J.P., 2005. Testing Statistical Hypotheses. 3rd edition, Springer, New York.

Milbrodt, H., Strasser, H., 1990. On the asymptotic power of the two-sided Kolmogorov-Smirnov test. J.Statist. Plann. Inference 26, 1-23.

Neuhaus, G., 1976. Asymptotic power properties of the Cramér-von Mises test under contiguousalternatives. J. Multivariate Anal. 6, 95–110.

Pfanzagl, J., Wefelmeyer, W., 1982. Contributions to a general asymptotic statistical theory. Lecture Notes inStatistics 13, Springer, Berlin.

Rahnenführer, J., 2003. On preferences of general two-sided tests with applications to Kolmogorov-Smirnov-type tests. Statist. Decisions 21, 149–170.

Shorack, G.R., Wellner, J.A., 1986. Empirical processes with applications to statistics. John Wiley & Sons,New York.

Strasser, H., 1985. Mathematical Theory of Statistics. De Gruyter Studies in Mathematics.

Thank you for your attention!

Preferences of goodness-of-fit tests: A survey about the ... · A survey about the analysis of...

Documents

Transcript of Preferences of goodness-of-fit tests: A survey about the ... · A survey about the analysis of...

AP Supplier Int Comparision

Karnal_quality Comparision Output

ERP Comparision

teradata oracle comparision

A Critical Survey on Detection of Object and Tracking of Object With Different Technique With Comparision

Open ERP comparision

GFSI comparision

Flipkart and Amazon Comparision

Comparision of concentrating collectors

Garch (1,1) comparision

comparision MPvsFT

Comparision 802.11af and 802.22

Autopilot Comparision To Carsforsale.Com

Linux Windows Unix Comparision

Finnish and Tamil comparision

NEC and IEC Comparision

Interfirm Comparision Final

Budget Comparision NFCL Synopsis

Mutual Funds Comparision

s comparision