An Empirical Investigation of Statistical Significance in NLP

Taylor Berg-Kirkpatrick David Burkett Dan KleinComputer Science Division

University of California at Berkeley{tberg, dburkett, klein}@cs.berkeley.edu

Abstract

We investigate two aspects of the empiricalbehavior of paired significance tests for NLPsystems. First, when one system appearsto outperform another, how does significancelevel relate in practice to the magnitude of thegain, to the size of the test set, to the similar-ity of the systems, and so on? Is it true that foreach task there is a gain which roughly impliessignificance? We explore these issues acrossa range of NLP tasks using both large collec-tions of past systems’ outputs and variants ofsingle systems. Next, once significance lev-els are computed, how well does the standardi.i.d. notion of significance hold up in practicalsettings where future distributions are neitherindependent nor identically distributed, suchas across domains? We explore this questionusing a range of test set variations for con-stituency parsing.

1 Introduction

It is, or at least should be, nearly universal that NLPevaluations include statistical significance tests tovalidate metric gains. As important as significancetesting is, relatively few papers have empirically in-vestigated its practical properties. Those that dofocus on single tasks (Koehn, 2004; Zhang et al.,2004) or on the comparison of alternative hypothe-sis tests (Gillick and Cox, 1989; Yeh, 2000; Bisaniand Ney, 2004; Riezler and Maxwell, 2005).

In this paper, we investigate two aspects of theempirical behavior of paired significance tests forNLP systems. For example, all else equal, largermetric gains will tend to be more significant. How-ever, what does this relationship look like and howreliable is it? What should be made of the conven-tional wisdom that often springs up that a certainmetric gain is roughly the point of significance fora given task (e.g. 0.4 F1 in parsing or 0.5 BLEU

in machine translation)? We show that, with heavycaveats, there are such thresholds, though we alsodiscuss the hazards in their use. In particular, manyother factors contribute to the significance level, andwe investigate several of them. For example, whatis the effect of the similarity between the two sys-tems? Here, we show that more similar systems tendto achieve significance with smaller metric gains, re-flecting the fact that their outputs are more corre-lated. What about the size of the test set? For ex-ample, in designing a shared task it is important toknow how large the test set must be in order for sig-nificance tests to be sensitive to small gains in theperformance metric. Here, we show that test sizeplays the largest role in determining discriminationability, but that we get diminishing returns. For ex-ample, doubling the test size will not obviate theneed for significance testing.

In order for our results to be meaningful, we musthave access to the outputs of many of NLP sys-tems. Public competitions, such as the well-knownCoNLL shared tasks, provide one natural way to ob-tain a variety of system outputs on the same testset. However, for most NLP tasks, obtaining out-puts from a large variety of systems is not feasible.Thus, in the course of our investigations, we proposea very simple method for automatically generatingarbitrary numbers of comparable system outputs andwe then validate the trends revealed by our syntheticmethod against data from public competitions. Thismethodology itself could be of value in, for exam-ple, the design of new shared tasks.

Finally, we consider a related and perhaps evenmore important question that can only be answeredempirically: to what extent is statistical significanceon a test corpus predictive of performance on othertest corpora, in-domain or otherwise? Focusing onconstituency parsing, we investigate the relationshipbetween significance levels and actual performance

on data from outside the test set. We show that whenthe test set is (artificially) drawn i.i.d. from the samedistribution that generates new data, then signifi-cance levels are remarkably well-calibrated. How-ever, as the domain of the new data diverges fromthat of the test set, the predictive ability of signifi-cance level drops off dramatically.

2 Statistical Significance Testing in NLP

First, we review notation and standard practice insignificance testing to set up our empirical investi-gation.

2.1 Hypothesis TestsWhen comparing a new system A to a baseline sys-tem B, we want to know if A is better than B onsome large population of data. Imagine that we sam-ple a small test set x = x1, . . . , xn on which Abeats B by δ(x). Hypothesis testing guards againstthe case where A’s victory over B was an unlikelyevent, due merely to chance. We would thereforelike to know how likely it would be that a new, in-dependent test set x′ would show a similar victoryfor A assuming that A is no better than B on thepopulation as a whole; this assumption is the nullhypothesis, denoted H0.

Hypothesis testing consists of attempting to esti-mate this likelihood, written p(δ(X) > δ(x)|H0),where X is a random variable over possible test setsof size n that we could have drawn, and δ(x) is aconstant, the metric gain we actually observed. Tra-ditionally, if p(δ(X) > δ(x)|H0) < 0.05, we saythat the observed value of δ(x) is sufficiently un-likely that we should reject H0 (i.e. accept that A’svictory was real and not just a random fluke). Werefer to p(δ(X) > δ(x)|H0) as p-value(x).

In most cases p-value(x) is not easily computableand must be approximated. The type of approxi-mation depends on the particular hypothesis testingmethod. Various methods have been used in the NLPcommunity (Gillick and Cox, 1989; Yeh, 2000; Rie-zler and Maxwell, 2005). We use the paired boot-strap1 (Efron and Tibshirani, 1993) because it is one

1Riezler and Maxwell (2005) argue the benefits of approx-imate randomization testing, introduced by Noreen (1989).However, this method is ill-suited to the type of hypothesis weare testing. Our null hypothesis does not condition on the testdata, and therefore the bootstrap is a better choice.

1. Draw b bootstrap samples x(i) of size n bysampling with replacement from x.

2. Initialize s = 0.3. For each x(i) increment s if δ(x(i)) > 2δ(x).4. Estimate p-value(x) ≈ s

b

Figure 1: The bootstrap procedure. In all of our experimentswe use b = 106, which is more than sufficient for the bootstrapestimate of p-value(x) to stabilize.

of the most widely used (Och, 2003; Bisani and Ney,2004; Zhang et al., 2004; Koehn, 2004), and be-cause it can be easily applied to any performancemetric, even complex metrics like F1-measure orBLEU (Papineni et al., 2002). Note that we couldperform the experiments described in this paper us-ing another method, such as the paired Student’s t-test. To the extent that the assumptions of the t-testare met, it is likely that the results would be verysimilar to those we present here.

2.2 The Bootstrap

The bootstrap estimates p-value(x) though a com-bination of simulation and approximation, drawingmany simulated test sets x(i) and counting how oftenA sees an accidental advantage of δ(x) or greater.How can we get sample test sets x(i)? We lack theability to actually draw new test sets from the un-derlying population because all we have is our datax. The bootstrap therefore draws each x(i) from xitself, sampling n items from x with replacement;these new test sets are called bootstrap samples.

Naively, it might seem like we would then checkhow often A beats B by more than δ(x) on x(i).However, there’s something seriously wrong withthese x(i) as far as the null hypothesis is concerned:the x(i) were sampled from x, and so their averageδ(x(i)) won’t be zero like the null hypothesis de-mands; the average will instead be around δ(x). Ifwe ask how many of these x(i) have A winning byδ(x), about half of them will. The solution is a re-centering of the mean – we want to know how oftenA does more than δ(x) better than expected. We ex-pect it to beat B by δ(x). Therefore, we count uphow many of the x(i) have A beating B by at least2δ(x).2 The pseudocode is shown in Figure 1.

2Note that many authors have used a variant where the eventtallied on the x(i) is whether δ(x(i)) < 0, rather than δ(x(i)) >2δ(x). If the mean of δ(x(i)) is δ(x), and if the distribution ofδ(x(i)) is symmetric, then these two versions will be equivalent.

As mentioned, a major benefit of the bootstrap isthat any evaluation metric can be used to computeδ(x).3 We run the bootstrap using several metrics:F1-measure for constituency parsing, unlabeled de-pendency accuracy for dependency parsing, align-ment error rate (AER) for word alignment, ROUGEscore (Lin, 2004) for summarization, and BLEUscore for machine translation.4 We report all met-rics as percentages.

3 Experiments

Our first goal is to explore the relationship be-tween metric gain, δ(x), and statistical significance,p-value(x), for a range of NLP tasks. In order to sayanything meaningful, we will need to see both δ(x)and p-value(x) for many pairs of systems.

3.1 Natural Comparisons

Ideally, for a given task and test set we could obtainoutputs from all systems that have been evaluatedin published work. For each pair of these systemswe could run a comparison and compute both δ(x)and p-value(x). While obtaining such data is notgenerally feasible, for several tasks there are pub-lic competitions to which systems are submitted bymany researchers. Some of these competitions makesystem outputs publicly available. We obtained sys-tem outputs from the TAC 2008 workshop on auto-matic summarization (Dang and Owczarzak, 2008),the CoNLL 2007 shared task on dependency parsing(Nivre et al., 2007), and the WMT 2010 workshopon machine translation (Callison-Burch et al., 2010).

For cases where the metric linearly decomposes over sentences,the mean of δ(x(i)) is δ(x). By the central limit theorem, thedistribution will be symmetric for large test sets; for small testsets it may not.

3Note that the bootstrap procedure given only approximatesthe true significance level, with multiple sources of approxima-tion error. One is the error introduced from using a finite num-ber of bootstrap samples. Another comes from the assumptionthat the bootstrap samples reflect the underlying population dis-tribution. A third is the assumption that the mean bootstrap gainis the test gain (which could be further corrected for if the metricis sufficiently ill-behaved).

4To save time, we can compute δ(x) for each bootstrap sam-ple without having to rerun the evaluation metric. For our met-rics, sufficient statistics can be recorded for each sentence andthen sampled along with the sentences when constructing eachx(i) (e.g. size of gold, size of guess, and number correct are suf-ficient for F1). This makes the bootstrap very fast in practice.

0.5

0.6

0.7

0.8

0.9

1

0 0.5 1 1.5 2

1

- p

-val

ue

ROUGE

Different research groupsSame research group

Figure 2: TAC 2008 Summarization: Confidence vs.ROUGE improvement on TAC 2008 test set for comparisonsbetween all pairs of the 58 participating systems at TAC 2008.Comparisons between systems entered by the same researchgroup and comparisons between systems entered by differentresearch groups are shown separately.

3.1.1 TAC 2008 SummarizationIn our first experiment, we use the outputs of the

58 systems that participated in the TAC 2008 work-shop on automatic summarization. For each possi-ble pairing, we compute δ(x) and p-value(x) on thenon-update portion of the TAC 2008 test set (we or-der each pair so that the gain, δ(x), is always pos-itive).5 For this task, test instances correspond todocument collections. The test set consists of 48document collections, each with a human producedsummary. Figure 2 plots the ROUGE gain against1 − p-value, which we refer to as confidence. Eachpoint on the graph corresponds to an individual pairof systems.

As expected, larger gains in ROUGE correspondto higher confidences. The curved shape of the plotis interesting. It suggests that relatively quickly wereach ROUGE gains for which, in practice, signif-icance tests will most likely be positive. We mightexpect that systems whose outputs are highly corre-lated will achieve higher confidence at lower met-ric gains. To test this hypothesis, in Figure 2 we

5In order to run bootstraps between all pairs of systemsquickly, we reuse a random sample counts matrix between boot-strap runs. As a result, we no longer need to perform quadrat-ically many corpus resamplings. The speed-up from this ap-proach is enormous, but one undesirable effect is that the boot-strap estimation noise between different runs is correlated. As aremedy, we set b so large that the correlated noise is not visiblein plots.

0.5

0.6

0.7

0.8

0.9

1

0 0.5 1 1.5 2 2.5 3 3.5 4

1

- p

-val

ue

Unlabeled Acc.

Different research groupsSame research group

Figure 3: CoNLL 2007 Dependency parsing: Confidence vs.unlabeled dependency accuracy improvement on the ChineseCoNLL 2007 test set for comparisons between all pairs of the21 participating systems in CoNLL 2007 shared task. Com-parisons between systems entered by the same research groupand comparisons between systems entered by different researchgroups are shown separately.

separately show the comparisons between systemsentered by the same research group and compar-isons between systems entered by different researchgroups, with the expectation that systems entered bythe same group are likely to have more correlatedoutputs. Many of the comparisons between systemssubmitted by the same group are offset from themain curve. It appears that they do achieve higherconfidences at lower metric gains.

Given the huge number of system comparisons inFigure 2, one obvious question to ask is whetherwe can take the results of all these statistical sig-nificance tests and estimate a ROUGE improvementthreshold that predicts when future statistical sig-nificance tests will probably be significant at thep-value(x) < 0.05 level. For example, let’s say wetake all the comparisons with p-value between 0.04and 0.06 (47 comparisons in all in this case). Eachof these comparisons has an associated metric gain,and by taking, say, the 95th percentile of these met-ric gains, we get a potentially useful threshold. Inthis case, the computed threshold is 1.10 ROUGE.

What does this threshold mean? Well, based onthe way we computed it, it suggests that if somebodyreports a ROUGE increase of around 1.10 on the ex-act same test set, there is a pretty good chance that astatistical significance test would show significanceat the p-value(x) < 0.05 level. After all, 95% of

the borderline significant differences that we’ve al-ready seen showed an increase of even less than 1.10ROUGE. If we’re evaluating past work, or are insome other setting where system outputs just aren’tavailable, the threshold could guide our interpreta-tion of reports containing only summary scores.

That being said, it is important that we don’t over-interpret the meaning of the 1.10 ROUGE threshold.We have already seen that pairs of systems submit-ted by the same research group and by different re-search groups follow different trends, and we willsoon see more evidence demonstrating the impor-tance of system correlation in determining the rela-tionship between metric gain and confidence. Addi-tionally, in Section 4, we will see that properties ofthe test corpus have a large effect on the trend. Thereare many factors are at work, and so, of course, met-ric gain alone will not fully determine the outcomeof a paired significance test.

3.1.2 CoNLL 2007 Dependency ParsingNext, we run an experiment for dependency pars-

ing. We use the outputs of the 21 systems that par-ticipated in the CoNLL 2007 shared task on depen-dency parsing. In Figure 3, we plot, for all pairs,the gain in unlabeled dependency accuracy againstconfidence on the CoNLL 2007 Chinese test set,which consists of 690 sentences and parses. Weagain separate comparisons between systems sub-mitted by the same research group and those submit-ted by different groups, although for this task therewere fewer cases of multiple submission. The re-sults resemble the plot for summarization; we againsee a curve-shaped trend, and comparisons betweensystems from the same group (few that they are)achieve higher confidences at lower metric gains.

3.1.3 WMT 2010 Machine TranslationOur final task for which system outputs are pub-

licly available is machine translation. We run an ex-periment using the outputs of the 31 systems par-ticipating in WMT 2010 on the system combinationportion of the German-English WMT 2010 news testset, which consists of 2,034 German sentences andEnglish translations. We again run comparisons forpairs of participating systems. We plot gain in testBLEU score against confidence in Figure 4. In thisexperiment there is an additional class of compar-

0.5

0.6

0.7

0.8

0.9

1

0 0.2 0.4 0.6 0.8 1

1

- p

-val

ue

BLEU

Different research groupsSame research groupSystem combination

Figure 4: WMT 2010 Machine translation: Confidence vs.BLEU improvement on the system combination portion of theGerman-English WMT 2010 news test set for comparisons be-tween pairs of the 31 participating systems at WMT 2010.Comparisons between systems entered by the same researchgroup, comparisons between systems entered by different re-search groups, and comparisons between system combinationentries are shown separately.

isons that are likely to have specially correlated sys-tems: 13 of the submitted systems are system com-binations, and each take into account the same setof proposed translations. We separate comparisonsinto three sets: comparisons between non-combinedsystems entered by different research groups, com-parisons between non-combined systems entered bythe same research group, and comparisons betweensystem-combinations.

We see the same curve-shaped trend we saw forsummarization and dependency parsing. Differ-ent group comparisons, same group comparisons,and system combination comparisons form distinctcurves. This indicates, again, that comparisons be-tween systems that are expected to be specially cor-related achieve high confidence at lower metric gainlevels.

3.2 Synthetic Comparisons

So far, we have seen a clear empirical effect, but, be-cause of the limited availability of system outputs,we have only considered a few tasks. We now pro-pose a simple method that captures the shape of theeffect, and use it to extend our analysis.

3.2.1 Training Set ResamplingAnother way of obtaining many different sys-

tems’ outputs is to obtain implementations of a

handful of systems, and then vary some aspect ofthe training procedure in order to produce many dif-ferent systems from each implementation. Koehn(2004) uses this sort of amplification; he uses a sin-gle machine translation implementation, and thentrains it from different source languages. We takea slightly different approach. For each task we picksome fixed training set. Then we generate resampledtraining sets by sampling sentences with replace-ment from the original. In this way, we can gen-erate as many new training sets as we like, each ofwhich is similar to the original, but with some vari-ation. For each base implementation, we train a newsystem on each resampled training set. This resultsin slightly tweaked trained systems, and is intendedto very roughly approximate the variance introducedby incremental system changes during research. Wevalidate this method by comparing plots obtained bythe synthetic approach with plots obtained from nat-ural comparisons.

We expect that each new system will be differ-ent, but that systems originating from the same basemodel will be highly correlated. This provides a use-ful division of comparisons: those between systemsbuilt with the same model, and those between sys-tems built with different models. The first class canbe used to approximate comparisons of systems thatare expected to be specially correlated, and the latterfor comparisons of systems that are not.

3.2.2 Dependency ParsingWe use three base models for dependency parsing:

MST parser (McDonald et al., 2005), Maltparser(Nivre et al., 2006), and the ensemble parser of Sur-deanu and Manning (2010). We use the CoNLL2007 Chinese training set, which consists of 57Ksentences. We resample 5 training sets of 57K sen-tences, 10 training sets of 28K sentences, and 10training sets of 14K sentences. Together, this yieldsa total of 75 system outputs on the CoNLL 2007Chinese test set, 25 systems for each base modeltype. The score ranges of all the base models over-lap. This ensures that for each pair of model typeswe will be able to see comparisons where the metricgains are small. The results of the pairwise compar-isons of all 75 system outputs are shown in Figure5, along with the results of the CoNLL 2007 sharedtask system comparisons from Figure 3.

0.5

0.6

0.7

0.8

0.9

1

0 0.5 1 1.5 2 2.5 3 3.5 4

1

- p

-val

ue

Unlabeled Acc.

Different model typesSame model type

CoNLL 2007 comparisons

Figure 5: Dependency parsing: Confidence vs. unlabeled de-pendency accuracy improvement on the Chinese CoNLL 2007test set for comparisons between all pairs of systems gener-ated by using resampled training sets to train either MST parser,Maltparser, or the ensemble parser. Comparisons between sys-tems generated using the same base model type and compar-isons between systems generated using different base modeltypes are shown separately. The CoNLL 2007 shared task com-parisons from Figure 3 are also shown.

The overlay of the natural comparisons suggeststhat the synthetic approach reasonably models therelationship between metric gain and confidence.Additionally, the different model type and samemodel type comparisons exhibit the behavior wewould expect, matching the curves corresponding tocomparisons between specially correlated systemsand standard comparisons respectively.

Since our synthetic approach yields a large num-ber of system outputs, we can use the proceduredescribed in Section 3.1.1 to compute the thresh-old above which the metric gain is probably signifi-cant. For comparisons between systems of the samemodel type, the threshold is 1.20 unlabeled depen-dency accuracy. For comparisons between systemsof different model types, the threshold is 1.51 un-labeled dependency accuracy. These results indi-cate that the similarity of the systems being com-pared is an important factor. As mentioned, rules-of-thumb derived from such thresholds cannot beapplied blindly, but, in special cases where two sys-tems are known to be correlated, the former thresh-old should be preferred over the latter. For example,during development most comparisons are made be-tween incremental variants of the same system. Ifadding a feature to a supervised parser increases un-

0.5

0.6

0.7

0.8

0.9

1

0 0.2 0.4 0.6 0.8 1

1

- p

-val

ue

BLEU

Different model typesSame model type

WMT 2010 comparisons

Figure 6: Machine translation: Confidence vs. BLEU im-provement on the system combination portion of the German-English WMT 2010 news test set for comparisons between allpairs of systems generated by using resampled training sets totrain either Moses or Joshua. Comparisons between systemsgenerated using the same base model type and comparisons be-tween systems generated using different base model types areshown separately. The WMT 2010 workshop comparisons fromFigure 4 are also shown.

labeled accuracy by 1.3, it is useful to be able toquickly estimate that the improvement is probablysignificant. This still isn’t the full story; we willsoon see that properties of the test set also play amajor role. But first, we carry our analysis to sev-eral more tasks.

3.2.3 Machine TranslationOur two base models for machine translation

are Moses (Koehn et al., 2007) and Joshua (Li etal., 2009). We use 1.4M sentence pairs from theGerman-English portion of the WMT-provided Eu-roparl (Koehn, 2005) and news commentary corporaas the original training set. We resample 75 trainingsets, 20 of 1.4M sentence pairs, 29 of 350K sentencepairs, and 26 of 88K sentence pairs. This yields atotal of 150 system outputs on the system combi-nation portion of the German-English WMT 2010news test set. The results of the pairwise compar-isons of all 150 system outputs are shown in Figure6, along with the results of the WMT 2010 workshopsystem comparisons from Figure 4.

The natural comparisons from the WMT 2010workshop align well with the comparisons betweensynthetically varied models. Again, the differentmodel type and same model type comparisons formdistinct curves. For comparisons between systems

0.5

0.6

0.7

0.8

0.9

1

0 0.5 1 1.5 2

1

- p

-val

ue

AER

Different model typesSame model type

Figure 7: Word alignment: Confidence vs. AER improve-ment on the Hansard test set for comparisons between all pairsof systems generated by using resampled training sets to traineither the ITG aligner, the joint HMM aligner, or GIZA++.Comparisons between systems generated using the same basemodel type and comparisons between systems generated usingdifferent base model types are shown separately.

of the same model type the computed p-value <0.05 threshold is 0.28 BLEU. For comparisons be-tween systems of different model types the thresholdis 0.37 BLEU.

3.2.4 Word AlignmentNow that we have validated our simple model of

system variation on two tasks, we go on to gen-erate plots for tasks that do not have competitionswith publicly available system outputs. The firsttask is English-French word alignment, where weuse three base models: the ITG aligner of Haghighiet al. (2009), the joint HMM aligner of Liang et al.(2006), and GIZA++ (Och and Ney, 2003). The lasttwo aligners are unsupervised, while the first is su-pervised. We train the unsupervised word alignersusing the 1.1M sentence pair Hansard training cor-pus, resampling 20 training sets of the same size.6

Following Haghighi et al. (2009), we train the super-vised ITG aligner using the first 337 sentence pairsof the hand-aligned Hansard test set; again, we re-sample 20 training sets of the same size as the origi-nal data. We test on the remaining 100 hand-alignedsentence pairs from the Hansard test set.

Unlike previous plots, the points correspondingto comparisons between systems with different base

6GIZA++ failed to produce reasonable output when trainedwith some of these training sets, so there are fewer than 20GIZA++ systems in our comparisons.

0.5

0.6

0.7

0.8

0.9

1

0 0.5 1 1.5 2

1

- p

-val

ue

F1

Different model typesSame model type

Figure 8: Constituency parsing: Confidence vs. F1 improve-ment on section 23 of the WSJ corpus for comparisons betweenall pairs of systems generated by using resampled training setsto train either the Berkeley parser, the Stanford parser, or theCollins parser. Comparisons between systems generated us-ing the same base model type and comparisons between sys-tems generated using different base model types are shown sep-arately.

model types form two distinct curves. It turns outthat the upper curve consists only of comparisonsbetween ITG and HMM aligners. This is likely dueto the fact that the ITG aligner uses posteriors fromthe HMM aligner for some of its features, so thetwo models are particularly correlated. Overall, thespread of this plot is larger than previous ones. Thismay be due to the small size of the test set, or possi-bly some additional variance introduced by unsuper-vised training. For comparisons between systems ofthe same model type the p-value < 0.05 thresholdis 0.50 AER. For comparisons between systems ofdifferent model types the threshold is 1.12 AER.

3.2.5 Constituency ParsingFinally, before we move on to further types of

analysis, we run an experiment for the task of con-stituency parsing. We use three base models: theBerkeley parser (Petrov et al., 2006), the Stanfordparser (Klein and Manning, 2003), and Dan Bikel’simplementation (Bikel, 2004) of the Collins parser(Collins, 1999). We use sections 2-21 of the WSJcorpus (Marcus et al., 1993), which consists of 38Ksentences and parses, as a training set. We resample10 training sets of size 38K, 10 of size 19K, and 10of size 9K, and use these to train systems. We teston section 23. The results are shown in Figure 8.

For comparisons between systems of the same

model type, the p-value < 0.05 threshold is 0.47F1. For comparisons between systems of differentmodel types the threshold is 0.57 F1.

4 Properties of the Test Corpus

For five tasks, we have seen a trend relating met-ric gain and confidence, and we have seen that thelevel of correlation between the systems being com-pared affects the location of the curve. Next, welook at how the size and domain of the test set playa role, and, finally, how significance level predictsperformance on held out data. In this section, wecarry out experiments for both machine translationand constituency parsing, but mainly focus on thelatter because of the availability of large test corporathat span more than one domain: the Brown corpusand the held out portions of the WSJ corpus.

4.1 Varying the Size

Figure 9 plots comparisons for machine translationon variously sized initial segments of the WMT2010 news test set. Similarly, Figure 10 plots com-parisons for constituency parsing on initial segmentsof the Brown corpus. As might be expected, thesize of the test corpus has a large effect. For bothmachine translation and constituency parsing, thelarger the corpus size, the lower the threshold forp-value < 0.05 and the smaller the spread of theplot. At one extreme, the entire Brown corpus,which consists of approximately 24K sentences, hasa threshold of 0.22 F1, while at the other extreme,the first 100 sentences of the Brown corpus have athreshold of 3.00 F1. Notice that we see diminishingreturns as we increase the size of the test set. Thisphenomenon follows the general shape of the cen-tral limit theorem, which predicts that variances ofobserved metric gains will shrink according to thesquare root of the test size. Even using the entireBrown corpus as a test set there is a small rangewhere the result of a paired significance test was notcompletely determined by metric gain.

It is interesting to note that for a fixed test size,the domain has only a small effect on the shape ofthe curve. Figure 11 plots comparisons for a fixedtest size, but with various test corpora.

0.5

0.6

0.7

0.8

0.9

1

0 0.2 0.4 0.6 0.8 1

1

- p

-val

ue

BLEU

100 sent.200 sent.400 sent.800 sent.

1600 sent.

Figure 9: Machine translation; varying test size: Confidencevs. BLEU improvement on portions of the German-EnglishWMT 2010 news test set.

0.5

0.6

0.7

0.8

0.9

1

0 0.5 1 1.5 2

1

- p

-val

ue

F1

100 sent.200 sent.400 sent.800 sent.

1600 sent.3200 sent.

Entire Brown corpus

Figure 10: Constituency parsing; varying test size: Con-fidence vs. F1 improvement on portions of the Brown corpus.

0.5

0.6

0.7

0.8

0.9

1

0 0.5 1 1.5 2

1

- p

-val

ue

F1

WSJ sec 22WSJ sec 23WSJ sec 24

Brown corpus

Figure 11: Constituency parsing; varying domain: Confi-dence vs. F1 improvement on the first 1,600 sentences of sec-tions 22, 23, and 24 of the WSJ corpus, and the Brown corpus.

4.2 Empirical Calibration across Domains

Now that we have a way of generating outputs forthousands of pairs of systems, we can check empir-ically the practical reliability of significance testing.Recall that the bootstrap p-value(x) is an approxi-mation to p(δ(X) > δ(x)|H0). However, we oftenreally want to determine the probability that the newsystem is better than the baseline on the underlyingtest distribution or even the distribution from anotherdomain. There is no reason a priori to expect thesenumbers to coincide.

In our next experiment, we treat the entire Browncorpus, which consists of 24K sentences, as the truepopulation of English sentences. For each systemgenerated in the way described in Section 3.2.5 wecompute F1 on all of Brown. Since we are treat-ing the Brown corpus as the actual population of En-glish sentences, for each pair of parsers we can saythat the sign of the F1 difference indicates which isthe truly better system. Now, we repeatedly resam-ple small test sets from Brown, each consisting of1,600 sentences, drawn by sampling sentences withreplacement. For each pair of systems, and for eachresampled test set, we compute p-value(x) using thebootstrap. Out of the 4K bootstraps computed in thisway, 942 had p-value between 0.04 and 0.06, 869of which agreed with the sign of the F1 differencewe saw on the entire Brown corpus. Thus, 92% ofthe significance tests with p-value in a tight rangearound 0.05 correctly identified the better system.

This result is encouraging. It suggests that sta-tistical significance computed using the bootstrap isreasonably well calibrated. However, test sets arealmost never drawn i.i.d. from the distribution of in-stances the system will encounter in practical use.Thus, we also wish to compute how calibration de-grades as the domain of the test set changes. In an-other experiment, we look at how significance nearp-value = 0.05 on section 23 of the WSJ corpuspredicts performance on sections 22 and 24 and theBrown corpus. This time, for each pair of generatedsystems we run a bootstrap on section 23. Out ofall these bootstraps, 58 system pairs had p-value be-tween 0.04 and 0.06. Of these, only 83% had thesame sign of F1 difference on section 23 as they didon section 22, 71% the had the same sign on sec-tion 23 as on section 24, and 48% the same sign on

Sec. 23 p-value % Sys. A > Sys. BSec. 22 Sec. 24 Brown

0.00125 - 0.0025 97% 95% 73%0.0025 - 0.005 92% 92% 60%0.005 - 0.01 92% 85% 56%0.01 - 0.02 88% 92% 54%0.02 - 0.04 87% 78% 51%0.04 - 0.08 83% 74% 48%

Table 1: Empirical calibration: p-value on section 23 of theWSJ corpus vs. fraction of comparisons where system A beatssystem B on section 22, section 24, and the Brown corpus. Notethat system pairs are ordered so that A always outperforms B onsection 23.

section 23 as on the Brown corpus. This indicatesthat reliability degrades as we switch the domain. Inthe extreme, achieving a p-value near 0.05 on sec-tion 23 provides no information about performanceon the Brown corpus.

If we intend to use our system on out-of-domaindata, these results are somewhat discouraging. Howlow does p-value(x) have to get before we start get-ting good information about out-of-domain perfor-mance? We try to answer this question for this par-ticular parsing task by running the same domain cal-ibration experiment for several different ranges ofp-value. The results are shown in Table 1. Fromthese results, it appears that for constituency pars-ing, when testing on section 23, a p-value level be-low 0.00125 is required to reasonably predict perfor-mance on the Brown corpus.

It should be considered a good practice to includestatistical significance testing results with empiri-cal evaluations. The bootstrap in particular is easyto run and makes relatively few assumptions aboutthe task or evaluation metric. However, we havedemonstrated some limitations of statistical signifi-cance testing for NLP. In particular, while statisticalsignificance is usually a minimum necessary condi-tion to demonstrate that a performance difference isreal, it’s also important to consider the relationshipbetween test set performance and the actual goalsof the systems being tested, especially if the systemwill eventually be used on data from a different do-main than the test set used for evaluation.

5 ConclusionWe have demonstrated trends relating several im-portant factors to significance level, which include

both properties of the systems being compared andproperties of the test corpus, and have presented asimple approach to approximating the response ofthese factors for tasks where large numbers of sys-tem outputs are not available. Our results revealthat the relationship between metric gain and sta-tistical significance is complex, and therefore sim-ple thresholds are not a replacement for significancetests. Indeed, we strongly advocate the use of statis-tical significance testing to validate metric gains inNLP, but also note that informal rules-of-thumb doarise in popular discussion and that, for some set-tings when previous systems are unavailable, theseempirical results can supplement less sensitive un-paired tests (e.g. bar-overlaps-point test) in evalua-tion of progress. Finally, even formal testing has itslimits. We provide cautionary evidence to this ef-fect, showing that the information provided by a testquickly degrades as the target corpus shifts domain.

Acknowledgements

This work was partially supported by NSF fellow-ships to the first and second authors and by the NSFunder grant 0643742.

References

D.M. Bikel. 2004. Intricacies of collins’ parsing model.Computational Linguistics.

M. Bisani and H. Ney. 2004. Bootstrap estimates forconfidence intervals in asr performance evaluation. InProc. of ICASSP.

C. Callison-Burch, P. Koehn, C. Monz, K. Peterson,M. Przybocki, and O.F. Zaidan. 2010. Findings ofthe 2010 joint workshop on statistical machine trans-lation and metrics for machine translation. In Proc. ofthe Joint Fifth Workshop on Statistical Machine Trans-lation and MetricsMATR.

M. Collins. 1999. Head-driven statistical models fornatural language parsing. Ph.D. thesis, University ofPennsylvania.

H.T. Dang and K. Owczarzak. 2008. Overview of thetac 2008 update summarization task. In Proc. of TextAnalysis Conference.

B. Efron and R. Tibshirani. 1993. An introduction to thebootstrap. Chapman & Hall/CRC.

L. Gillick and S.J. Cox. 1989. Some statistical issues inthe comparison of speech recognition algorithms. InProc. of ICASSP.

A. Haghighi, J. Blitzer, J. DeNero, and D. Klein. 2009.Better word alignments with supervised ITG models.In Proc. of ACL.

D. Klein and C.D. Manning. 2003. Accurate unlexical-ized parsing. In Proc. of ACL.

P. Koehn, H. Hoang, A. Birch, C. Callison-Burch,M. Federico, N. Bertoldi, B. Cowan, W. Shen,C. Moran, R. Zens, et al. 2007. Moses: Open sourcetoolkit for statistical machine translation. In Proc. ofACL.

P. Koehn. 2004. Statistical significance tests for machinetranslation evaluation. In Proc. of EMNLP.

P. Koehn. 2005. Europarl: A parallel corpus for statisti-cal machine translation. In MT summit.

Z. Li, C. Callison-Burch, C. Dyer, J. Ganitkevitch,S. Khudanpur, L. Schwartz, W.N.G. Thornton,J. Weese, and O.F. Zaidan. 2009. Joshua: An opensource toolkit for parsing-based machine translation.In Proc. of the Fourth Workshop on Statistical MachineTranslation.

P. Liang, B. Taskar, and D. Klein. 2006. Alignment byagreement. In Proc. of NAACL.

C.Y. Lin. 2004. Rouge: A package for automatic evalu-ation of summaries. In Proc. of the Workshop on TextSummarization.

M.P. Marcus, M.A. Marcinkiewicz, and B. Santorini.1993. Building a large annotated corpus of english:The penn treebank. Computational linguistics.

R. McDonald, F. Pereira, K. Ribarov, and J. Hajic. 2005.Non-projective dependency parsing using spanningtree algorithms. In Proc. of EMNLP.

J. Nivre, J. Hall, and J. Nilsson. 2006. Maltparser: Adata-driven parser-generator for dependency parsing.In Proc. of LREC.

J. Nivre, J. Hall, S. Kubler, R. McDonald, J. Nilsson,S. Riedel, and D. Yuret. 2007. The conll 2007 sharedtask on dependency parsing. In Proc. of the CoNLLShared Task Session of EMNLP-CoNLL 2007.

E.W. Noreen. 1989. Computer Intensive Methods forHypothesis Testing: An Introduction. Wiley, NewYork.

F.J. Och and H. Ney. 2003. A systematic comparison ofvarious statistical alignment models. Computationallinguistics.

F.J. Och. 2003. Minimum error rate training in statisticalmachine translation. In Proc. of ACL.

K. Papineni, S. Roukos, T. Ward, and W.J. Zhu. 2002.Bleu: a method for automatic evaluation of machinetranslation. In Proc. of ACL.

S. Petrov, L. Barrett, R. Thibaux, and D. Klein. 2006.Learning accurate, compact, and interpretable tree an-notation. In Proc. of ACL.

S. Riezler and J.T. Maxwell. 2005. On some pitfalls inautomatic evaluation and significance testing for mt.In Proc. of ACL Workshop on Intrinsic and ExtrinsicEvaluation Measures for Machine Translation and/orSummarization.

M. Surdeanu and C.D. Manning. 2010. Ensemble mod-els for dependency parsing: cheap and good? In Proc.of NAACL.

A. Yeh. 2000. More accurate tests for the statistical sig-nificance of result differences. In Proc. of ACL.

Y. Zhang, S. Vogel, and A. Waibel. 2004. Interpretingbleu/nist scores: How much improvement do we needto have a better system. In Proc. of LREC.