Proceedings of Workshop for NLP Open Source Software, pages 7–12Melbourne, Australia, July 20, 2018. c©2018 Association for Computational Linguistics

7

Stop Word Lists in Free Open-source Software Packages

Joel NothmanSydney Informatics Hub

University of Sydneyjoel.nothman@gmail.com

Hanmin QinPeking University

qinhanmin2005@sina.com

Roman YurchakSymerio

rth.yurchak@gmail.com

Abstract

Open-source software (OSS) packagesfor natural language processing often in-clude stop word lists. Users may applythem without awareness of their surprisingomissions (e.g. hasn’t but not hadn’t) andinclusions (e.g. computer), or their incom-patibility with particular tokenizers. Mo-tivated by issues raised about the Scikit-learn stop list, we investigate variationamong and consistency within 52 popularEnglish-language stop lists, and proposestrategies for mitigating these issues.

1 Introduction

Open-source software (OSS) resources tend to be-come de-facto standards by virtue of their avail-ability and popular use. Resources include tok-enization rules and stop word lists, whose precisedefinitions are essential for reproducible and in-terpretable models. These resources can be se-lected somewhat arbitrarily by OSS contributors,such that their popularity within the communitymay not be a reflection of their quality, universal-ity or suitability for a particular task. Users maythen be surprised by behaviors such as the wordcomputer being eliminated from their text analy-sis due to its inclusion in a popular stop list.

This paper brings to the community’s attentionsome issues recently identified in the Scikit-learnstop list. Despite is popular use, the current Scikit-learn maintainers cannot justify the use of this par-ticular list, and are unaware of how it was con-structed. This spurs us to investigate variationamong and consistency within popular English-language stop lists provided in several popular lan-guage processing, retrieval and machine learninglibraries. We then make recommendations for im-proving stop list provision in OSS.

2 Background

Stop words are presumed to be not informative asto the meaning of documents, and hence are de-fined by being unusually frequent, or by not being“content words”. Saif et al. (2014) lists severalmethods for constructing a stop word list, includ-ing: manual construction; words with high docu-ment frequency or total term frequency in a corpus;or by comparing term frequency statistics from asample of documents with those in a larger col-lection.1 In practice, Manning et al. (2008) in-dicate that statistical approaches tend not to beused alone, but are combined with manual filter-ing. This paper notes ways in which statisticalconstruction of stop lists may have introduced re-grettable errors.

Stop lists have been generated for other lan-guages, such as Chinese (Zou et al., 2006),Thai (Daowadung and Chen, 2012) andFarsi (Sadeghi and Vegas, 2014), using usessimilar frequency threshold approaches, aresusceptible to the same issues discussed here.

Most prior work focuses on assessing or im-proving the effectiveness of stop word lists, suchas Schofield et al.’s (2017) recent critique of stoplists in topic modeling. Our work instead exam-ines what is available and widely used.

3 Case Study: Scikit-learn

Having become aware of issues with the Scikit-learn (Pedregosa et al., 2011) stop list,2 we be-gin by studying it. Scikit-learn provides out-of-the-box feature extraction tools which convert acollection of text documents to a matrix of tokencounts, optionally removing n-grams containing

1They also investigate using supervised feature selectiontechniques, but the supervised learning context is inapplica-ble here.

2As at version 0.19.1

8

given stop words. Being a popular library for ma-chine learning, many of its users take a naive ap-proach to language processing, and are unlikely totake a nuanced approach to stop word removal.

History While users are able to provide theirown stop list, Scikit-learn provides an English-language list since July 2010. The list was initiallydisabled by default since the contributing authorclaimed that it did not improve accuracy for textclassification (see commit 41b0562). In Novem-ber 2010, another contributor argued to enable thelist by default, saying that stop word removal is areasonable default behavior (commit 41128af).The developers disabled the list by default againin March 2012 (commit a510d17).

The list was copied from the Glasgow Informa-tion Retrieval Group,3 but it was unattributed untilJanuary 2012 (commit d4c4c6f). The list wasaltered in 2011 to remove the content word com-puter (commit cdf7df9), and in 2015 to correctthe word fify to fifty (commit 3e4ebac).

This history gives a sense of how a stop wordlist may be selected and provided without greatawareness of its content: its provenance was ini-tially disregarded; and some words were eventu-ally deemed inappropriate.

Critique Currently, the list in Scikit-learn hasseveral issues. Firstly, the list is incompatiblewith the tokenizer provided along with it. It in-cludes words discarded by the default tokenizer,i.e., words less than 2 chars (e.g. i), and someabbreviated forms which will be split by the tok-enizer (e.g. hasnt). What’s more, it excludes encl-itics generated by the tokenizer (e.g. ve of we’ve).In April 2017, a maintainer proposed to add ve tothe list.4 Contributors argued this would break re-producibility across software versions, and the is-sue remains unresolved.

Secondly, there are some controversial words inthe list, such as system and cry. These words areconsidered to be informative and are seldom in-cluded in other stop lists. In March 2018, a userrequested the removal of system and has gainedapproval from the community.5

Another issue is that the list has some surpris-

3http://ir.dcs.gla.ac.uk/resources/linguistic_utils/stop_words

4https://github.com/scikit-learn/scikit-learn/issues/8687

5https://github.com/scikit-learn/scikit-learn/issues/10735

1970 1980 1990 2000 2010

Snowball

Glasgow IR

rank.nl(Google)

scikit-learn

MySql(InnoDB)

MySql(MyIsam)SMART

NLTK

CoreNLP

Lucene / Solr

Stone, et al spaCy

gensim

postgres

Figure 1: Family tree of popular stop word lists.

ing omissions. Compared to extensions of theGlasgow IR list from Stone et al. (2010) used byspaCy (Honnibal and Montani, 2017) and gen-sim (Rehurek and Sojka, 2010), the list in Scikit-learn includes modal has, but lacks does; includesintensifier very but excludes really; and includeslight verb get but excludes make.

The Glasgow IR list appears to have been con-structed from corpus statistics, although typo-graphic errors like fify suggest manual editing.However, we have not found any documentationabout how the Glasgow IR list was constructed.Hence we know little about how to generate com-parable lists for other languages or domains.

In the remainder of this paper, we consider howsimilar issues apply to other open-source stop lists.

4 Datasets

We conduct our experiments on Igor Brigadir’scollection of English-language stop word lists.6

We exclude 1 empty list, 2 lists which contain n-grams (n > 1) and 1 list which is intended to aug-ment other lists (i.e. LEMUR’s forumstop). Fi-nally, we get 52 lists extracted from various searchengines, libraries, and articles. The size of the listsvaries (see the right part of Figure 2), from 24words in the EBSCOhost medical databases list,to 988 words in the ATIRE search engine list.

5 Stop List Families

Through digging into project history, we constructa family tree of some popular stop lists (Figure 1)to show how popular OSS packages adopt or adaptexisting lists. Solid lines in the figure correspondto inclusion without major modification, whiledashed lines correspond to a more loose adapta-tion. For instance, the Glasgow IR list used byScikit-learn was extended with 18 more words by

6https://github.com/igorbrigadir/stopwords/tree/21fb2ef

9

Stone et al. (2010), and this list was adopted byOSS packages gensim and spaCy in turn.

A more data driven approach identifies similar-ities among stop lists by clustering them with theJaccard distance metric (JD(A,B) := 1− |A∩B|

|A∪B|where A and B are sets of stop words). InFigure 2, we have plotted the same data with aheatmap of word inclusion in order of descendingdocument frequency in the NYT section of Giga-word 5.0 (Parker et al., 2011). Here we take themaximum frequency under three tokenizers fromLucene, Scikit-learn and spaCy. Each of themhas different approaches to enclitics (e.g. hasn’t istreated as hasn’t in Lucene; hasn in Scikit-learnand has n’t in spaCy).

Looking at the heatmap, we see that stop wordsare largely concentrated around high documentfrequency. Some high frequency words are ab-sent from many stop lists because most stop listsassume particular tokenization strategies (See Sec-tion 6.2). However, beyond the extremely frequentwords, even the shortest lists vary widely in whichwords they then include. Some stop lists includemany relatively low-frequency words. This ismost noticeable for large lists like TERRIER andATIRE-Puurula. TERRIER goes to pains to in-clude synthesized inflectional variants, even con-cerninger, and archaic forms, like couldst.

Through the clusermap, we find some lists withvery high within-cluster similarity (JD < 0.2):Ranks.nl old Google list and MySQL/InnoDB list;PostgreSQL list and NLTK list; Weka list, MAL-LET list, MySQL-MyISAM list, SMART list andROUGE list; Glasgow IR list, Scikit-learn list andspaCy/Gensim list. Beyond these simple clusters,some lists appear to have surprisingly high over-lap (usually asymmetric): Stanford CoreNLP listappears to be an extension of Snowball’s originallist; ATIRE-Puurula appears to be an extension ofthe Ranks.nl Large list.

6 Common Issues for Stop Word Lists

In section 3, we find several issues in the stop wordlist from Scikit-learn. In this section, we explorehow these problems manifest in other lists.

6.1 Controversial Words

We consider words which appear in less than 10%of lists to be controversial.7 After excluding words

7Some false negatives will result from the shared originsof lists detailed in the previous section, but we find very simi-

which do not begin with English characters, we get2066 distinct stop words in the 52 lists. Amongthese words, 1396 (67.6%) words only appear inless than 10% of lists, and 807 (39.1%) words onlyappear in 1 list (see the bars at the top of Figure 2),indicating that controversial words cover a largeproportion. On the contrary, only 64 (3.1%) wordsare accepted by more than 80% lists. Among the52 lists, 45 have controversial words.

We further investigate the document frequencyof these controversial words using Google BooksNgrams (Michel et al., 2011). Figure 3 shows thedocument frequency distribution. Note: We scaledocument frequency of controversial words by themax document frequency among all the words.Although peaked on rare words, some words arefrequent (e.g. general, great, time), indicating thatthe problem is not trivial.

6.2 Tokenization and Stop Lists

Popular software libraries apply different tok-enization rules, particularly with respect to word-internal punctuation. By comparing how differentstop lists handle the word doesn’t in Figure 4, wesee several approaches: most lists stop doesn’t. Afew stop doesn or doesnt, but none stop both ofthese. Two stop doesn’t as well as doesnt, whichmay help them be robust to different choices oftokenizer, or may be designed to handle informaltext where apostrophes may be elided.

However, we find tools providing lists that areinconsistent with their tokenizers. While most listsstop not, Penn Treebank-style tokenizers – pro-vided by CoreNLP, spaCy, NLTK and other NLP-oriented packages – also generate the token n’t.Of our dataset, n’t is only stopped by CoreNLP.8

Weka and Scikit-learn both have default tokeniz-ers which delimit tokens at punctuation including’, yet neither stops words like doesn.

We find similar results when repeating this anal-ysis on other negated models (e.g. hasn’t, haven’t,wouldn’t), showing that stop lists are often tunedto particular tokenizers, albeit not always thedefault tokenizer provided by the correspondingpackage. More generally, we have not found anyOSS package which documents how tokenizationrelates to the choice of stop list.

lar results if we remove near-duplicate lists (Jaccard distance< 0.2) from our experiments.

8We are aware that spaCy, in commit f708d74, recentlyamended its list to improve consistency with its tokenizer,adding n’t among other Penn Treebank contraction tokens.

10

0.00.20.40.6Jaccard distance

,yo

use

epa

stca

n't

prob

lem

idea

gone

wro

ngop

ened

chec

kst

udie

sno

v 85lu

nch

resu

lted

back

sse

cw

owfif

teen

parte

das

ides

furth

ered th

uar

ent

fore

sm

auge

rm

yse

over

alle

stin

war

der ¹

Words in descending Gigaword frequency

0

50

# lis

ts

0 500 1000# words

t101_minimalokapi_cacm_expanded

terrierreuters_wos

okapi_sampleokapi_sample_expanded

onixgalago_rmstop

indrivoyant_taporware

taporwarescikitlearn

glasgow_stop_wordsspacy_gensim

atire_ncbichoi_2000naacl

zettairtonybsk_1rouge_155

smartmysql_myisam

malletweka

ranksnl_largetonybsk_6

atire_puurulagate_keyphrase

cook1988_function_wordsnltk

postgresqlcorenlp_acronym

snowball_expandedranksnl_default

snowball_originalcorenlp_stopwords

99webtoolstextfixer

sphinx_astellarvw_lda

lingpipeokapi_cacm

lexisnexisbow_short

ovidmysql_innodb

ranksnl_oldgooglelucene_elastisearchcorenlp_hardcoded

ebscohost_medline_cinahlokapiframework

sphinx_500datasciencedojo

Figure 2: Word inclusion in clustered English stop word lists. Words are ordered by descending docu-ment frequency. The dendrogram on the left indicates minimum Jaccard distance between stop list pairswhen merged. The bars on the right show the number of words in each list, and the bars on the topindicate the number of lists each word is found in.

Figure 3: Document frequency distribution of con-troversial words

6.3 Incompleteness

Stop lists generated exclusively from corpus statis-tics are bound to omit some inflectional formsof an included word, as well as related lexemes,such as less frequent members of a functional syn-tactic class. In particular, stop word list con-struction prefers frequency criteria over contex-tual indicators of a word’s function, despite Har-ris’s (1954) well-established theory that similarwords (e.g. function words, light verbs, negatedmodals) should appear in similar contexts.

To continue the example of negated modals, wefind inconsistencies in the inclusion of have and itsvariants, summarized in Figure 5. Weka includeshas, but omits its negated forms, despite includingnot. Conversely, Okapi includes doesnt, but omits

11

Figure 4: Number of stop lists that include variantsof doesn’t and their combinations.

Figure 5: Number of stop lists that include variantsof have and their combinations.

does. Several lists include has, hasnt, have andhad, but omits havent and hadnt. Several lists thatinclude has and have forms omit had forms. Theseinclusions and omissions seem arbitrary.

Some negated modals like shan’t and mustn’tare absent more often than other modals (e.g.doesn’t, hasn’t), which may be an unsurprising ar-tifact of their frequency, or may be an ostensiveomission because they are more marked.

TERRIER list (Ounis et al., 2005) appears tohave generated inflectional variants, to the extentof including concerninger. This generally seemsan advisable path towards improved consistency.

7 Improving Stop List Provision in OSS

Based on the analysis above, we propose strategiesfor better provision of stop lists in OSS:

Documentation Stop lists should be docu-mented with their assumptions about tokenizationand other limitations (e.g. genre). Documentationshould also include information on provenanceand how the list was built.

Dynamic Adaptation Stop lists can be adapteddynamically to match the NLP pipeline. For ex-ample, stop lists can be adjusted according to thetokenizer chosen by the user (e.g. through apply-ing the tokenizer to the stop list); a word which isan inflectional variant of a stop word could also beremoved

Quality Control The community should de-velop tools for identifying controversial terms instop lists (e.g. words that are frequent in onecorpus but infrequent in another), and to assistin assessing or mitigating incompleteness issues.For instance, future work could evaluate whetherthe nearest neighborhood of stop words in vectorspace can be used to identify incompleteness.

Tools for Automatic Generation A major limi-tation of published stop lists is their inapplicabilityto new domains and languages. We thus advocatelanguage independent tools to assist in generatingnew lists, which could incorporate the quality con-trol tools above.

8 Conclusion

Stop word lists are a simple but useful tool formanaging noise, with ubiquitous support in natu-ral language processing software. We have foundthat popular stop lists, which users often applyblindly, may suffer from surprising omissions andinclusions, or their incompatibility with particulartokenizers. Many of these issues may derive fromgenerating stop lists using corpus statistics. Wehence recommend better documentation, dynami-cally adapting stop lists during preprocessing, aswell as creating tools for stop list quality controland automatically generating stop lists.

Acknowledgments

We thank Igor Brigadir for collecting and provid-ing English-language stop word lists along withtheir provenance. We also thank Scikit-learn con-tributors for bringing these issues to our attention.

12

References

P. Daowadung and Y. H. Chen. 2012. Stop word inreadability assessment of thai text. In Proceedingsof 2012 IEEE 12th International Conference on Ad-vanced Learning Technologies, pages 497–499.

Zelig Harris. 1954. Distributional structure. Word,10(23):146–162.

Matthew Honnibal and Ines Montani. 2017. spaCy 2:Natural language understanding with bloom embed-dings, convolutional neural networks and incremen-tal parsing. To appear.

Christopher D. Manning, Prabhakar Raghavan, andHinrich Schutze. 2008. Introduction to InformationRetrieval. Cambridge University Press.

Jean-Baptiste Michel, Yuan Kui Shen, Aviva PresserAiden, Adrian Veres, Matthew K. Gray, Joseph P.Pickett, Dale Hoiberg, Dan Clancy, Peter Norvig,Jon Orwant, Steven Pinker, Martin A. Nowak, andErez Lieberman Aiden. 2011. Quantitative analysisof culture using millions of digitized books. Sci-ence, 331(6014):176–182.

Iadh Ounis, Gianni Amati, Vassilis Plachouras, BenHe, Craig Macdonald, and Douglas Johnson. 2005.Terrier information retrieval platform. In Proceed-ings of the 27th European Conference on Advancesin Information Retrieval Research, pages 517–519.

Robert Parker, David Graff, Junbo Kong, Ke Chen, andKazuaki Maeda. 2011. English Gigaword fifth edi-tion LDC2011T07. Linguistic Data Consortium.

Fabian Pedregosa, Gael Varoquaux, Alexandre Gram-fort, Vincent Michel, Bertrand Thirion, OlivierGrisel, Mathieu Blondel, Peter Prettenhofer, RonWeiss, Vincent Dubourg, Jake Vanderplas, Alexan-dre Passos, David Cournapeau, Matthieu Brucher,Matthieu Perrot, and Edouard Duchesnay. 2011.Scikit-learn: Machine learning in Python. Journalof Machine Learning Research, 12:2825–2830.

Radim Rehurek and Petr Sojka. 2010. Software Frame-work for Topic Modelling with Large Corpora. InProceedings of the LREC 2010 Workshop on NewChallenges for NLP Frameworks, pages 45–50, Val-letta, Malta. ELRA.

Mohammad Sadeghi and Jess Vegas. 2014. Automaticidentification of light stop words for persian infor-mation retrieval systems. Journal of InformationScience, 40(4):476–487.

Hassan Saif, Miriam Fernandez, Yulan He, and HarithAlani. 2014. On stopwords, filtering and data spar-sity for sentiment analysis of Twitter. In Pro-ceedings of the Ninth International Conference onLanguage Resources and Evaluation. Proceedings.,pages 810–817.

Alexandra Schofield, Mans Magnusson, and DavidMimno. 2017. Pulling out the stops: Rethinkingstopword removal for topic models. In Proceedingsof the 15th Conference of the European Chapter ofthe Association for Computational Linguistics: Vol-ume 2, Short Papers, pages 432–436.

Benjamin Stone, Simon Dennis, and Peter J. Kwantes.2010. Comparing methods for single paragraphsimilarity analysis. Topics in Cognitive Science,3(1):92–122.

Feng Zou, Fu Lee Wang, Xiaotie Deng, Song Han, andLu Sheng Wang. 2006. Automatic construction ofchinese stop word list. In Proceedings of the 5thWSEAS International Conference on Applied Com-puter Science.