Preferences of goodness-of-fit tests: A survey about the ... · A survey about the analysis of...
Transcript of Preferences of goodness-of-fit tests: A survey about the ... · A survey about the analysis of...
A survey about the analysis of nonparametric power functions
Preferences of goodness-of-fit tests: A surveyabout the analysis of nonparametric power
functions
Arnold Janssen
Mathematical Institute, Heinrich-Heine-University Duesseldorf
Ulm, September , 2015
A survey about the analysis of nonparametric power functions
Table of contents
1 Introduction
2 Local comparision of asymptotic nonparametric powerfunctions
3 Global nonparametric power functions are almost flat
4 Tests for hazard rates
5 Recommendation in practice
6 Making two-sample GOF tests distribution free if ties arepresent
7 References
A survey about the analysis of nonparametric power functions
Introduction
Outline
1 Introduction
2 Local comparision of asymptotic nonparametric powerfunctions
3 Global nonparametric power functions are almost flat
4 Tests for hazard rates
5 Recommendation in practice
6 Making two-sample GOF tests distribution free if ties arepresent
7 References
A survey about the analysis of nonparametric power functions
Introduction
Kolmogorov-Smirnov (KS) test
asymptotics of a normalized empirical process under the null(critical regions), B0: Brownian bridge
omnibus tests are typically consistent
A survey about the analysis of nonparametric power functions
Introduction
What is the difference for more general critical lines?
Which type of tests can be recommended?
The statistician should analyze the goodness-of-fit tests of hiscomputer package in order to get some knowledge and animpression about their preferences.
A survey about the analysis of nonparametric power functions
Introduction
What is the difference for more general critical lines?
Which type of tests can be recommended?The statistician should analyze the goodness-of-fit tests of hiscomputer package in order to get some knowledge and animpression about their preferences.
A survey about the analysis of nonparametric power functions
Introduction
A survey about the analysis of nonparametric power functions
Introduction
Discussion under local alternatives
about nonparametric power functionsclassification of tests: preferences for special alternativesmeaning and statistical interpretation of preferedalternatives
A survey about the analysis of nonparametric power functions
Introduction
History
Hájek/ Šidák (1967): asymptotic power of two-sample linearrank tests
Pitman asymptotic relative efficiency AREefficiency (ARE=1) for special alternativesalso ARE=0 show up
For KS test: Hájek/ Šidák (1967), Andel (1967):one-sided KS test is close to the median test(via local asymptotic relative efficiency)two-sided: Milbrodt and Strasser (1990)
Tool: the three Lemmata of Le Cam
A survey about the analysis of nonparametric power functions
Introduction
History
Hájek/ Šidák (1967): asymptotic power of two-sample linearrank tests
Pitman asymptotic relative efficiency AREefficiency (ARE=1) for special alternativesalso ARE=0 show up
For KS test: Hájek/ Šidák (1967), Andel (1967):one-sided KS test is close to the median test(via local asymptotic relative efficiency)two-sided: Milbrodt and Strasser (1990)
Tool: the three Lemmata of Le Cam
A survey about the analysis of nonparametric power functions
Local comparision of asymptotic nonparametric power functions
Outline
1 Introduction
2 Local comparision of asymptotic nonparametric powerfunctions
3 Global nonparametric power functions are almost flat
4 Tests for hazard rates
5 Recommendation in practice
6 Making two-sample GOF tests distribution free if ties arepresent
7 References
A survey about the analysis of nonparametric power functions
Local comparision of asymptotic nonparametric power functions
A survey about the analysis of nonparametric power functions
Local comparision of asymptotic nonparametric power functions
“local parameters of order 1√n ” score functions g : (0,1)→ R
(or tangents g)H0 : λλ|(0,1) = F0 (after quantile transformation)
dFn,i.gdF0
(x) = 1 + cni g(x) + . . . 1 ≤ i ≤ n
cni =1√n one sample
Nonparametrics: parametric submodels are still present
A survey about the analysis of nonparametric power functions
Local comparision of asymptotic nonparametric power functions
Under local alternatives Png/√
n with g ∈ L02(0,1), where
L02(0,1) := {h ∈ L2(0,1) :
∫ 10 h(u) du = 0}, < h,g >=
∫ 10 hg dλλ:
√n(Fn(t)− F0(t))
D−−−−→Pn
g/√
n
X0(t) = B0(t)︸ ︷︷ ︸noise
+
∫ t
0g(u) du︸ ︷︷ ︸signal
, t ∈ [0,1],
F0 continuous.
limit experiment: signal detection problem (SDP) for B0.Milbrodt and Strasser (1990)
A survey about the analysis of nonparametric power functions
Local comparision of asymptotic nonparametric power functions
Reparametrization
testing problem for the limit experiment:
H0 : Sg ≤ 0 versus H1 : Sg ≥ 0,Sg 6= 0,
whereSg(t) :=
∫ t
0g(u) du, 0 ≤ t ≤ 1,
is the signal for g ∈ L02(0,1).
A survey about the analysis of nonparametric power functions
Local comparision of asymptotic nonparametric power functions
Le Cam theory: State of the art.
Convergence of the experiments, SDP limit experimentMain theorem of TestingConvergence of power functions towards the powerfunction of a limit test
Two-sided goodness-of-fit tests with convex acceptanceregions on suitable function spaces are asymptoticallyadmissible (Le Cam). Thus, there exists no overallasymptotically efficient goodness-of-fit test for all possibledirections of alternatives with dominating power function.
A survey about the analysis of nonparametric power functions
Local comparision of asymptotic nonparametric power functions
Le Cam theory: State of the art.
Convergence of the experiments, SDP limit experimentMain theorem of TestingConvergence of power functions towards the powerfunction of a limit testTwo-sided goodness-of-fit tests with convex acceptanceregions on suitable function spaces are asymptoticallyadmissible (Le Cam). Thus, there exists no overallasymptotically efficient goodness-of-fit test for all possibledirections of alternatives with dominating power function.
A survey about the analysis of nonparametric power functions
Local comparision of asymptotic nonparametric power functions
Example: One-sided Kolmogorov-Smirnov-Type Test
Let ρ : [0,1]→ R be a boundary function
ϕnKS =
1 >
supt∈[0,1](√
n (Fn(t)− t)− ρ(t)) 0 .0 ≤
Limit test for SDP is Kolmogorov-Smirnov-Type Test:
ϕKS =
1 >
supt∈[0,1](X0(t)− ρ(t)) 00 ≤
with X0(t) = B0(t) +∫ t
0 g(u) du for 0 ≤ t ≤ 1.
A survey about the analysis of nonparametric power functions
Local comparision of asymptotic nonparametric power functions
Let Pg be the distribution on C[0,1] of
X (t) = B0(t) +∫ t
0g(u)du = B0(t) + Sg(t) , 0 ≤ t ≤ 1
power of test ϕ for SDP (Girsanov formula)
g 7→ EPg (ϕ) =
∫ϕexp
(∫ 1
0gdB0 −
||g||2
2
)dP0
principal component decompositionintegral test statistics: compact operators, Shorack/WellnerCramér von Mises test statistics, Neuhaus (1976)not for sup-statistics (KS-test), Milbrodt/Strasser (1990)
A survey about the analysis of nonparametric power functions
Local comparision of asymptotic nonparametric power functions
Let Pg be the distribution on C[0,1] of
X (t) = B0(t) +∫ t
0g(u)du = B0(t) + Sg(t) , 0 ≤ t ≤ 1
power of test ϕ for SDP (Girsanov formula)
g 7→ EPg (ϕ) =
∫ϕexp
(∫ 1
0gdB0 −
||g||2
2
)dP0
principal component decompositionintegral test statistics: compact operators, Shorack/WellnerCramér von Mises test statistics, Neuhaus (1976)not for sup-statistics (KS-test), Milbrodt/Strasser (1990)
A survey about the analysis of nonparametric power functions
Local comparision of asymptotic nonparametric power functions
Comparison of power functions of tests ϕ along a path ϑ 7→ νϑof distributions
oracle test : Neyman Pearson test or(benchmark) best two-sided unbaised test
principle of local comparison at H0 : ϑ = 0
slope: ddϑEϑ (ϕ)|ϑ=0 curvature: d2
dϑ2 Eϑ (ϕ)|ϑ=0
A survey about the analysis of nonparametric power functions
Local comparision of asymptotic nonparametric power functions
Theorem (principal components of tests, J.(1995))
H0 = P0, h tangent h ∈ L(0)2 , ray t 7→ Pth. There exist a gradient
h0 ∈ L(0)2 and a Hilbert-Schmidt operator T : L(0)
2 → L(0)2 with
Eth(ϕ) = E0(ϕ)︸ ︷︷ ︸level α
+t < h,h0 >︸ ︷︷ ︸slope
+t2
2< h,T (h) >︸ ︷︷ ︸
curvature
+o(t2)
< h,T (h) > =∞∑
i=1
λi < h,hi >2, ||hi || = 1, λ1 ≥ λ2 ≥ . . . ≥ 0
||h0||2 +12
∞∑i=1
λ2i < α(1− α), α = E0(ϕ)
one-sided tests: h0||h0|| direction with highest slope
two-sided unbaised test: h0 = 0: h1 direction with highestcurvature (=highest preference for h0, h1 respectively)
A survey about the analysis of nonparametric power functions
Local comparision of asymptotic nonparametric power functions
Theorem (principal components of tests, J.(1995))
H0 = P0, h tangent h ∈ L(0)2 , ray t 7→ Pth. There exist a gradient
h0 ∈ L(0)2 and a Hilbert-Schmidt operator T : L(0)
2 → L(0)2 with
Eth(ϕ) = E0(ϕ)︸ ︷︷ ︸level α
+t < h,h0 >︸ ︷︷ ︸slope
+t2
2< h,T (h) >︸ ︷︷ ︸
curvature
+o(t2)
< h,T (h) > =∞∑
i=1
λi < h,hi >2, ||hi || = 1, λ1 ≥ λ2 ≥ . . . ≥ 0
||h0||2 +12
∞∑i=1
λ2i < α(1− α), α = E0(ϕ)
one-sided tests: h0||h0|| direction with highest slope
two-sided unbaised test: h0 = 0: h1 direction with highestcurvature (=highest preference for h0, h1 respectively)
A survey about the analysis of nonparametric power functions
Local comparision of asymptotic nonparametric power functions
Results for one- and two-sided KS-type tests
one-sided gradient h(a,b)0 (approximately), J./Kunz (2002),
Rahnenführer (2003)
h(a,a)0 ∼ sign(2u − 1)
score function of the mediantest
A survey about the analysis of nonparametric power functions
Local comparision of asymptotic nonparametric power functions
one-sided ϕ: local asymptotic relative efficiency in direction h
AREL(ϕ,h) =(
< h,h0 >
‖h‖ f (u1−α)
)2
if < h,h0 > ≥ 0
f , u1−α N(0,1) density, 1− α quantile
AREL = 1 iff ϕ Neyman Pearson test (familiar Pitmaninterpretation of ARE)
numerical result: α = 0.05
AREL
(ϕKS,one-sided,h(a,a)
)≈ 0.715
A survey about the analysis of nonparametric power functions
Local comparision of asymptotic nonparametric power functions
unbaised two-sided GOFtests gradient: h0 = 0
first principal component h1 ofthe Hilbert-Schmidt operatorh1 ≈ h(a,b)
0 , J. (1995), Rah-nenführer (2003).
A survey about the analysis of nonparametric power functions
Local comparision of asymptotic nonparametric power functions
A.Connection to the score function of locationfamilies (x , ϑ) 7→ f (x − ϑ)
score function h ↑ : there exists a location family with score
function − f ′(F−1(u))f(F−1(u))
= h(u), Hájek/ Šidák (1967):
h(a,b)0 : f(a,b)(x) =
1a + b
[exp
(xa
)1(−∞,0](x) + exp
(−x
b
)1(0,∞)(x)
]
J./Ünlü (2008)
f(a,a) double exponential
A survey about the analysis of nonparametric power functions
Local comparision of asymptotic nonparametric power functions
B.Connection to differentiable stat. functionals κ
Koshevnik and Levit (1976)Pfanzagl, Wefelmeyer (1982)Bickel, Klaassen, Ritov, Wellner (1993)
κ : P → R (differentiable functional)
ddϑκ (Pϑ)|ϑ=0 =< g, κ >, κ canonical gradient (preference of κ)
quantile function: κ(F ) = F−1(
aa+b
), κ(x) = h(a.b) ◦ F (x)
Conclusion: KS type test is a testfor F 7→ F−1
(a
a+b
)
A survey about the analysis of nonparametric power functions
Global nonparametric power functions are almost flat
Outline
1 Introduction
2 Local comparision of asymptotic nonparametric powerfunctions
3 Global nonparametric power functions are almost flat
4 Tests for hazard rates
5 Recommendation in practice
6 Making two-sample GOF tests distribution free if ties arepresent
7 References
A survey about the analysis of nonparametric power functions
Global nonparametric power functions are almost flat
SDP: B0(t) +∫ t
0 g(u) du,0 ≤ t ≤ 1, distribution Pg
H0 : P0
Theorem J. (2000) (almost flat global power)
Let ϕ be a test for the SDP {Pg : g ∈ L02(0,1)} with EP0(ϕ) = α.
For all ε > 0 and K > 0 there exists a subspace V ⊂ L02(0,1) of
finite dimension with
sup{|EPh(ϕ)− α| : h ∈ V⊥, ‖h‖ ≤ K} ≤ ε.
Moreover,
dim(V )− 1 ≤ ε−1α(1− α)(exp(K 2)− 1).
A survey about the analysis of nonparametric power functions
Global nonparametric power functions are almost flat
already contained in the text bookLehmann and Romano (2005), Chap. 14.6,Testing statistical hypotheses, third edition.also holds uniformly in the sample size n (J. (2003) )trick: C[0,1] and RN are Borel-isomorphic as polish spaces
Explaination: Nonparametric asymptotic power functions ofgoodness-of-fit tests are almost flat. The statistician has a“total amount of power” which can be distributed along theorthonormal directions. Each test distributes the power in adifferent way.Dual description via level points of GOF tests
A survey about the analysis of nonparametric power functions
Global nonparametric power functions are almost flat
already contained in the text bookLehmann and Romano (2005), Chap. 14.6,Testing statistical hypotheses, third edition.also holds uniformly in the sample size n (J. (2003) )trick: C[0,1] and RN are Borel-isomorphic as polish spacesExplaination: Nonparametric asymptotic power functions ofgoodness-of-fit tests are almost flat. The statistician has a“total amount of power” which can be distributed along theorthonormal directions. Each test distributes the power in adifferent way.Dual description via level points of GOF tests
A survey about the analysis of nonparametric power functions
Tests for hazard rates
Outline
1 Introduction
2 Local comparision of asymptotic nonparametric powerfunctions
3 Global nonparametric power functions are almost flat
4 Tests for hazard rates
5 Recommendation in practice
6 Making two-sample GOF tests distribution free if ties arepresent
7 References
A survey about the analysis of nonparametric power functions
Tests for hazard rates
Gaussian shift experiment: SPD with Brownian motion B
X (t) := B(t) +∫ t
0h(u)du, 0 ≤ t ≤ 1
limit model for hazard rates λ(t) = f (t)1−F (t)
λϑλ(u)− 1 =
h(u)√n
+ . . .
the same result for global power functions
A survey about the analysis of nonparametric power functions
Tests for hazard rates
KS type test for SDP for Brownian motion:
ϕ = 1(
supt≤1
Xt ≥ c)
Rényi test
Rahnenführer (2003): h0 ≈ constant“close to the log-rank test (Savage test)”
Related to the K-transformation of Khmaladze (1981, 1983)
Fn = Mn + An Mn martingale, An compensator
“tests based on Mn instead of Fn”= testing hazards instead of densities
Two-sided: Drees/Milbrodt (1994)
A survey about the analysis of nonparametric power functions
Tests for hazard rates
KS type test for SDP for Brownian motion:
ϕ = 1(
supt≤1
Xt ≥ c)
Rényi test
Rahnenführer (2003): h0 ≈ constant“close to the log-rank test (Savage test)”Related to the K-transformation of Khmaladze (1981, 1983)
Fn = Mn + An Mn martingale, An compensator
“tests based on Mn instead of Fn”= testing hazards instead of densities
Two-sided: Drees/Milbrodt (1994)
A survey about the analysis of nonparametric power functions
Recommendation in practice
Outline
1 Introduction
2 Local comparision of asymptotic nonparametric powerfunctions
3 Global nonparametric power functions are almost flat
4 Tests for hazard rates
5 Recommendation in practice
6 Making two-sample GOF tests distribution free if ties arepresent
7 References
A survey about the analysis of nonparametric power functions
Recommendation in practice
Ask the scientist about his preferences for alternatives interms of statistical functionals (median, quantiles,interquartile distance, . . . )calculate the canonical gradients of the functionals,g1, . . . ,gk (= prefered directions of the functionals)
select a GOF test with high power on the subspace ofalternatives span(g1, . . . ,gk ) or
one-sided: Likelihood ratio test for the cone{∑ki=1 βigi : βi ≥ 0
}two-sided: h1, . . . ,hd O.N. basis of span(g1, . . . ,gk )
Neyman’s smooth test Tn =∑d
j=1
(1√n
∑ni=1 hj(xi)
)2.
A survey about the analysis of nonparametric power functions
Recommendation in practice
Ask the scientist about his preferences for alternatives interms of statistical functionals (median, quantiles,interquartile distance, . . . )calculate the canonical gradients of the functionals,g1, . . . ,gk (= prefered directions of the functionals)select a GOF test with high power on the subspace ofalternatives span(g1, . . . ,gk ) or
one-sided: Likelihood ratio test for the cone{∑ki=1 βigi : βi ≥ 0
}two-sided: h1, . . . ,hd O.N. basis of span(g1, . . . ,gk )
Neyman’s smooth test Tn =∑d
j=1
(1√n
∑ni=1 hj(xi)
)2.
A survey about the analysis of nonparametric power functions
Making two-sample GOF tests distribution free if ties are present
Outline
1 Introduction
2 Local comparision of asymptotic nonparametric powerfunctions
3 Global nonparametric power functions are almost flat
4 Tests for hazard rates
5 Recommendation in practice
6 Making two-sample GOF tests distribution free if ties arepresent
7 References
A survey about the analysis of nonparametric power functions
Making two-sample GOF tests distribution free if ties are present
two i.i.d. samples n = n1 + n2, min(n1,n2)→∞
X1, . . . ,Xn1 Xn1+1, . . . ,Xn
edf Gn edf Hn
Theorem (J. 1994)Under the null H0 : Xi ∼ F i.i.d.
Tn :=(n1n2
n
) 12
supt∈R
∣∣∣Gn(t)− Hn(t)∣∣∣ D−−−→
n→∞supt∈R|B0 (F (t))|
“depends on F for tied data.”works also for other norms on D[0,1].
A survey about the analysis of nonparametric power functions
Making two-sample GOF tests distribution free if ties are present
How to get critical values at finite sample size?
Fix X1, . . . ,Xn and use the permutation test. Let
σ : {1, . . . ,n} → {1, . . . ,n}
be random uniformly distributed permutation, independentof the X ’s.Critical values are taken from the permutation distribution
σ 7→ Tn(Xσ(1),Xσ(2), . . . ,Xσ(n)
).
Result: Power of the permutation test is equal to the power ofthe unconditional test if F for H0 would be known.
A survey about the analysis of nonparametric power functions
Making two-sample GOF tests distribution free if ties are present
How to get critical values at finite sample size?
Fix X1, . . . ,Xn and use the permutation test. Let
σ : {1, . . . ,n} → {1, . . . ,n}
be random uniformly distributed permutation, independentof the X ’s.Critical values are taken from the permutation distribution
σ 7→ Tn(Xσ(1),Xσ(2), . . . ,Xσ(n)
).
Result: Power of the permutation test is equal to the power ofthe unconditional test if F for H0 would be known.
A survey about the analysis of nonparametric power functions
References
Outline
1 Introduction
2 Local comparision of asymptotic nonparametric powerfunctions
3 Global nonparametric power functions are almost flat
4 Tests for hazard rates
5 Recommendation in practice
6 Making two-sample GOF tests distribution free if ties arepresent
7 References
A survey about the analysis of nonparametric power functions
References
References
Andel, J., 1967. Local asymptotic power and efficiency of tests of Kolmogorov-Sminov type. Ann. Math. Stat.38, 1705–1725.
Bickel, P.J., Klaassen, C.A.J., Ritov, Y., Wellner, J.A., 1993. Efficient and Adaptive Estimation forSemiparametric Models. Johns Hopkins University Press, Baltimore.
Drees, H., Milbrodt, H., 1994. The one-sided Kolmogorov-Smirnov test in signal detection problems withGaussian white noise. J. Stat. Plann. Inference 29, 325–335
Hájek, J., Šidák, Z., 1967. Theory of rank tests. Academic Press, New York.
Janssen, A., 1994:Two-sample goodness-of-fit tests when ties are present. J. Stat. Plann. Inference 39,399-424.
Janssen, A., 1995. Principal component decomposition of non-parametric tests. Probab. Theory RelatedFields 101, 193–209.
Janssen, A., 1999. Testing nonparametric statistical functionals with applications to rank tests. J. Stat.Plann. Inference 81, 71-93. Erratum (2001), 92, 297.
Janssen, A., 2000. Global power functions of goodness of fit tests. Ann. Statist.28, 239–253.
Janssen, A., 2003. Which power of goodness of fit tests can really be expected:Intermediate versuscontiguous alternatives. Statist. Decisions 21, 301–325.
Janssen, A., Kunz, M., 2002. Global extrapolations for power functions of one-sided nonparametric tests.Stat. Decis. 20, 153-176.
A survey about the analysis of nonparametric power functions
References
Janssen, A., Kunz, M., 2004: Brownian Type Boundary Crossing Probability for Piecewise Linear BoundaryFunctions. Commun. Statist. - Theory Meth., Vol.33, 1445-1464.
Janssen, A., Ünlü, H., 2008: Regions of alternatives with high and low power for goodness-of-fit tests. J.Stat. Planning Inference 138, 2526-2543.
Khmaladze, É.V., 1981. A martingale approach in the theory of goodness-of-fit-tests. (Russian) Teor.Veroyatnost. i Primenen. 26, 246–265.
Khmaladze, É.V., 1983. Martingale limit theorems for decomposable statistics. (Russian) Teor. Veroyatnost. iPrimenen. 28, 504–520.
Koshevnik, Y.A., Levit, B.Y., 1976. On a nonparametric analogue of the information matrix. Theory Probab.Appl. 21, 738–753.
Lehmann, E.L., Romano, J.P., 2005. Testing Statistical Hypotheses. 3rd edition, Springer, New York.
Milbrodt, H., Strasser, H., 1990. On the asymptotic power of the two-sided Kolmogorov-Smirnov test. J.Statist. Plann. Inference 26, 1-23.
Neuhaus, G., 1976. Asymptotic power properties of the Cramér-von Mises test under contiguousalternatives. J. Multivariate Anal. 6, 95–110.
Pfanzagl, J., Wefelmeyer, W., 1982. Contributions to a general asymptotic statistical theory. Lecture Notes inStatistics 13, Springer, Berlin.
Rahnenführer, J., 2003. On preferences of general two-sided tests with applications to Kolmogorov-Smirnov-type tests. Statist. Decisions 21, 149–170.
Shorack, G.R., Wellner, J.A., 1986. Empirical processes with applications to statistics. John Wiley & Sons,New York.
Strasser, H., 1985. Mathematical Theory of Statistics. De Gruyter Studies in Mathematics.
A survey about the analysis of nonparametric power functions
Thank you for your attention!