Inducing Domain-Specific Sentiment Lexicons from Unlabeled Corpora

William L. Hamilton, Kevin Clark, Jure Leskovec, Dan JurafskyDepartment of Computer Science, Stanford University, Stanford CA, 94305

wleif,kevclark,jure,jurafsky@stanford.edu

Abstract

A word’s sentiment depends on the domain inwhich it is used. Computational social scienceresearch thus requires sentiment lexicons thatare specific to the domains being studied. Wecombine domain-specific word embeddingswith a label propagation framework to in-duce accurate domain-specific sentiment lex-icons using small sets of seed words, achiev-ing state-of-the-art performance competitivewith approaches that rely on hand-curated re-sources. Using our framework we performtwo large-scale empirical studies to quantifythe extent to which sentiment varies acrosstime and between communities. We induceand release historical sentiment lexicons for150 years of English and community-specificsentiment lexicons for 250 online communi-ties from the social media forum Reddit. Thehistorical lexicons show that more than 5%of sentiment-bearing (non-neutral) Englishwords completely switched polarity during thelast 150 years, and the community-specificlexicons highlight how sentiment varies dras-tically between different communities.

1 Introduction

Inducing domain-specific sentiment lexicons is cru-cial to computational social science (CSS) research.Sentiment lexicons allow us to analyze key subjec-tive properties of texts like opinions and attitudes(Taboada et al., 2011). But lexical sentiment ishugely influenced by context. The word soft has avery different sentiment in an online sports commu-nity than it does in one dedicated to toy animals (Fig-ure 1). Terrific once had a highly negative conno-

r/sportsr/mylittlepony

−6−4−2

0246

Sen

timen

tofs

oft

Figure 1: The sentiment of soft in different online commu-nities. Sentiment values computed using SENTPROP (Section3) on comments from Reddit communities illustrate how sen-timent depends on social context. Bootstrap-sampled standarddeviations provide a measure of confidence with the scores.

tation; now it is essentially synonymous with good(Figure 2). Without domain-specific lexicons, socialscientific analyses can be misled by sentiment as-signments biased towards domain-general contexts,neglecting factors like genre, community-specificvernacular, or demographic variation (Deng et al.,2014; Hovy, 2015; Yang and Eisenstein, 2015).

Using experts or crowdsourcing to constructdomain-specific sentiment lexicons is expensiveand often time-consuming (Mohammad and Turney,2010; Fast et al., 2016), and is especially prob-lematic when non-standard language (as in histori-cal documents or obscure social media forums) pre-vents annotators from understanding the sociolin-guistic context of the data.

Web-scale sentiment lexicons can be automat-ically induced for large socially-diffuse domains,such as the internet-at-large (Velikovich et al., 2010)or all of Twitter (Tang et al., 2014). However, tostudy sentiment in domain-specific cases—financialdocuments, historical texts, or tight-knit social me-

arX

iv:1

606.

0282

0v2

[cs

.CL

] 2

4 Se

p 20

16

dia forums—such generic lexicons may be inaccu-rate, and even introduce harmful biases (Loughranand McDonald, 2011).1 Researchers need a princi-pled and accurate framework for inducing lexiconsthat are specific to their domain of study.

To meet these needs, we introduce SENTPROP, aframework to learn accurate sentiment lexicons fromsmall sets of seed words and domain-specific cor-pora. SENTPROP combines the well-known methodof label propagation with advances in word em-beddings, and unlike previous approaches, is de-signed to be accurate even when using modestly-sized domain-specific corpora (∼107 tokens). Ourframework also provides confidence scores alongwith the learned lexicons, which allows researchersto quantify uncertainty in a principled manner.

The key contributions of this work are:

1. A simple state-of-the-art sentiment induction al-gorithm, combining high-quality word vectorembeddings with a label propagation approach.

2. A novel bootstrap-sampling framework for infer-ring confidence scores with the sentiment values.

3. Two large-scale studies that reveal how sentimentdepends on both social and historical context.(a) We induce community-specific sentimentlexicons for the largest 250 “subreddit” commu-nities on the social-media forum Reddit, reveal-ing substantial variation in word sentiment be-tween communities.(b) We induce historical sentiment lexicons for150 years of English, revealing that >5% ofwords switched polarity during this time.

To the best of our knowledge, this is the first workto systematically analyze the domain-dependency ofsentiment at a large-scale, across hundreds of yearsand hundreds of user-defined online communities.

All of the inferred lexicons along with code forSENTPROP and all methods evaluated are madeavailable in the SOCIALSENT package released withthis paper.2 The SOCIALSENT package provides abenchmark toolkit for inducing sentiment lexicons,including implementations of previously publishedalgorithms (Velikovich et al., 2010; Rothe et al.,2016), which are not otherwise publicly available.

1http://brandsavant.com/brandsavant/

the-hidden-bias-of-social-media-sentiment-analysis2http://nlp.stanford.edu/projects/socialsent

Figure 2: The sentiment of terrific changed from negative topositive over the last 150 years. Sentiment values and boot-strapped confidences were computed using SENTPROP on his-torical data (see Section 6).

2 Related work

Our work builds upon a wealth of previous researchon inducing sentiment lexicons, along two threads:

Corpus-based approaches use seed words andpatterns in unlabeled corpora to induce domain-specific lexicons. These patterns may rely on syn-tactic structures (Hatzivassiloglou and McKeown,1997; Jijkoun et al., 2010; Rooth et al., 1999; The-len and Riloff, 2002; Widdows and Dorow, 2002),which can be domain-specific and brittle (e.g., insocial media lacking usual grammatical structures).Other models rely on general co-occurrence (Igoand Riloff, 2009; Riloff and Shepherd, 1997; Tur-ney and Littman, 2003). Often corpus-based meth-ods exploit distant-supervision signals (e.g., reviewscores, emoticons) specific to certain domains (As-ghar et al., 2015; Blair-Goldensohn et al., 2008;Bravo-Marquez et al., 2015; Choi and Cardie, 2009;Severyn and Moschitti, 2015; Speriosu et al., 2011;Tang et al., 2014). An effective corpus-based ap-proach that does not require distant-supervision—which we adapt here—is to construct lexical graphsusing word co-occurrences and then to performsome form of label propagation over these graphs(Huang et al., 2014; Velikovich et al., 2010). Re-cent work has also learned transformations of word-vector representations in order to induce sentimentlexicons (Rothe et al., 2016). Fast et al. (2016) com-bine word vectors with crowdsourcing to producedomain-independent topic lexicons.

Dictionary-based approaches use hand-curatedlexical resources—usually WordNet (Fellbaum,1998)—in order to propagate sentiment from seed

Domain Positive seed words Negative seed words

StandardEnglish

good, lovely, excellent, fortunate, pleasant,delightful, perfect, loved, love, happy

bad, horrible, poor, unfortunate, unpleasant,disgusting, evil, hated, hate, unhappy

Finance successful, excellent, profit, beneficial, im-proving, improved, success, gains, positive

negligent, loss, volatile, wrong, losses, dam-ages, bad, litigation, failure, down, negative

Twitter love, loved, loves, awesome, nice, amazing,best, fantastic, correct, happy

hate, hated, hates, terrible, nasty, awful,worst, horrible, wrong, sad

Table 1: Seed words. The seed words were manually selected to be context insensitive (without knowledge of the test lexicons).

labels (Esuli and Sebastiani, 2006; Hu and Liu,2004; Kamps et al., 2004; Rao and Ravichandran,2009; San Vicente et al., 2014; Takamura et al.,2005; Tai and Kao, 2013). There is an implicit con-sensus that dictionary-based approaches will gener-ate higher-quality lexicons, due to their use of theseclean, hand-curated resources; however, they are notapplicable in domains lacking such a resource (e.g.,most historical texts).

Most previous work seeks to enrich or enlarge ex-isting lexicons (Qiu et al., 2009; San Vicente et al.,2014; Velikovich et al., 2010), emphasizing recallover precision. This recall-oriented approach is mo-tivated by the need for massive polarity lexicons intasks like web-advertising (Velikovich et al., 2010).In contrast to these previous efforts, the goal of thiswork is to induce high-quality lexicons that are ac-curate to a specific social context.

Algorithmically, our approach is inspired by Ve-likovich et al. (2010). We extend Velikovichet al. (2010) by incorporating high-quality wordvector embeddings, a new graph construction ap-proach, an alternative label propagation algorithm,and a bootstrapping method to obtain confidencevalues. Together these improvements, especiallythe high-quality word vectors, allow our corpus-based method to even outperform the state-of-the-artdictionary-based approach.

3 Framework

Our framework, SENTPROP, is designed to meetfour key desiderata:1. Resource-light: Accurate performance without

massive corpora or hand-curated resources.2. Interpretable: Uses small seed sets of

“paradigm” words to maintain interpretabil-ity and avoid ambiguity in sentiment values.

3. Robust: Bootstrap-sampled standard deviationsprovide a measure of confidence.

4. Out-of-the-box: Does not rely on signals that arespecific to only certain domains.SENTPROP involves two steps: constructing a

lexical graph from unlabeled corpora and propagat-ing sentiment labels over this graph.

3.1 Constructing a lexical graph

Lexical graphs are constructed from distributionalword embeddings learned on unlabeled corpora.

Distributional word embeddingsThe first step in our approach is to build high-

quality semantic representations for words using avector space model (VSM). We embed each wordwi ∈ V as a vector wi that captures informationabout its co-occurrence statistics with other words(Landauer and Dumais, 1997; Turney and Pantel,2010). This VSM approach has a long history inNLP and has been highly successful in recent appli-cations (see Levy et al., 2015 for a survey).

When recreating known lexicons, we used a num-ber of publicly available embeddings (Section 4).

In the cases where we learned embeddings our-selves, we employed an SVD-based method to con-struct the word-vectors. First, we construct a matrixMPPMI ∈ R|V|×|V| with entries given by

MPPMIi,j = max

{log

(p̂(wi, wj)

p̂(w)p̂(wj)

), 0

}, (1)

where p̂ denotes smoothed empirical probabilities ofword (co-)occurrences within fixed-size sliding win-dows of text.3 MPPMI

i,j is equal to a smoothed vari-ant of the positive pointwise mutual information be-tween words wi and wj (Levy et al., 2015). Next,we compute MPPMI = UΣV>, the truncated sin-gular value decomposition of MPPMI . The vector

3We use contexts of size four on each side and context-distribution smoothing with c = 0.75 (Levy et al., 2015).

a. Run random walks from seed words. b. Assign polarity scores based on frequency of random walk visits.

love

hate

despise

adore

loath

dislike

like

find

appreciate

acknowledge

see

+++

idolize

relish

abhor

notice

---

execrate

disapprove

uncover

love

hate

despise

adore

loath

dislike

like

find

appreciate

acknowledge

see

+++

idolize

relish

abhor

notice

---

execrate

disapprove

uncover

--

-----

---

++

++

++ +

++

Figure 3: Visual summary of the SENTPROP algorithm.

embedding for word wi is then given by

wSVDi = (U)i . (2)

Excluding the singular value weights, Σ, has beenshown known to dramatically improve embeddingquality (Turney and Pantel, 2010; Bullinaria andLevy, 2012). Following standard practices, we learnembeddings of dimension 300.

We found that this SVD-based method signif-icantly outperformed word2vec (Mikolov et al.,2013) and GloVe (Pennington et al., 2014) on pre-liminary experiments with the domain-specific datawe used (see Section 4).

Defining the graph edgesGiven a set of word embeddings, a weighted lex-

ical graph is constructed by connecting each wordwith its nearest k neighbors within the semanticspace (according to cosine-similarity). The weightsof the edges are set as

Ei,j = arccos

(− wi

>wj

‖wi‖‖wj‖

). (3)

3.2 Propagating polarities from a seed setOnce a weighted lexical graph is constructed, wepropagate sentiment labels over this graph using arandom walk method (Zhou et al., 2004). A word’spolarity score for a seed set is proportional to theprobability of a random walk from the seed set hit-ting that word (Figure 3).

Let p ∈ R|V| be a vector of word-sentimentscores constructed using seed set S (e.g., ten nega-tive words); p is initialized to have 1

|V| in all entries.And let E be the matrix of edge weights given byequation (3). First, we construct a symmetric tran-sition matrix from E by computing T = D

12 ED

12 ,

where D is a matrix with the column sums of E onthe diagonal. Next, using T we iteratively update puntil numerical convergence:

p(t+1) = βTp(t) + (1− β)s, (4)

where s is a vector with values set to 1|S| in the en-

tries corresponding to the seed set S and zeros else-where. The β term controls the extent to which thealgorithm favors local consistency (similar labels forneighbors) vs. global consistency (correct labels onseed words), with lower βs emphasizing the latter.

To obtain a final polarity score for a word wi,we run the walk using both positive and negativeseed sets, obtaining positive (pP (wi)) and nega-tive (pN (wi)) label scores. We then combine thesevalues into a positive-polarity score as p̄P (wi) =

pP (wi)pP (wi)+pN (wi)

and standardize the final scores tohave zero mean and unit variance (within a corpus).

3.3 SENTPROP variantsMany variants of the random walk approach and re-lated label propagation techniques exist in the lit-erature (San Vicente et al., 2014; Velikovich et al.,2010; Zhou et al., 2004; Zhu and Ghahramani, 2002;Zhu et al., 2003). For example, there are differ-ences in how to normalize the transition matrix inthe random walks (Zhou et al., 2004) and variantsof label propagation, e.g. where the labeled seedsare clamped to the correct values (Zhu and Ghahra-mani, 2002) or where only shortest-paths throughthe graph are used for propagation (Velikovich et al.,2010).

We experimented with a number of these ap-proaches and found little difference in their perfor-mance. We opted to use the random walk methodbecause it had a slight edge in terms of performance

in preliminary experiments4 and because it pro-duces well-behaved distributions over label scores,whereas Zhu and Ghahramani (2002)’s method andits variants produce extremely peaked distributions.We do note report in detail on all the label propa-gation variants here, but the SOCIALSENT packagecontains a full suite of these methods.

3.4 Bootstrap-sampling for robustness

Propagated sentiment scores are inevitably influ-enced by the seed set, and it is important for re-searchers to know the extent to which polarity valuesare simply the result of corpus artifacts that are cor-related with these seeds words. We address this issueby using a bootstrap-sampling approach to obtainconfidence regions over our sentiment scores. Webootstrap by running our propagation over B ran-dom equally-sized subsets of the positive and neg-ative seed sets. Computing the standard deviationof the bootstrap-sampled polarity scores provides ameasure of confidence and allows the researcher toevaluate the robustness of the assigned polarities.We set B = 50 and used 7 words per random subset(full seed sets are size 10; see Table 1).

4 Recreating known lexicons

We validate our approach by recreating known sen-timent lexicons in the three domains: Standard En-glish, Twitter, and Finance. Table 1 lists the seedwords used in each domain.

Standard English: To facilitate comparison withprevious work, we focus on the well-known GeneralInquirer lexicon (Stone et al., 1966). We also use thecontinuous valence (i.e., polarity) scores collectedby Warriner et al. (2013) in order to evaluate thefine-grained performance of our framework. We testour framework’s performance using two differentembeddings: off-the-shelf Google news embeddingsconstructed from 1011 tokens5 and embeddings weconstructed from the 2000s decade of the Corpus ofHistorical American English (COHA), which con-tains ∼2 × 107 words in each decade, from 1850to 2000 (Davies, 2010). The COHA corpus allowsus to test how the algorithms deal with this smaller

4>2% improvement across metrics on the standard and his-torical English datasets described in Section 4.

5https://code.google.com/p/word2vec/

historical corpus, which is important since we willuse the COHA corpus to infer historical sentimentlexicons (Section 6).

Finance: Previous work found that general pur-pose sentiment lexicons performed very poorly onfinancial text (Loughran and McDonald, 2011), soa finance-specific sentiment lexicon (containing bi-nary labels) was hand-constructed for this domain(ibid.). To test against this lexicon, we constructedembeddings using a dataset of∼2×107 tokens fromfinancial 8K documents (Lee et al., 2014).

Twitter: Numerous works attempt to induceTwitter-specific sentiment lexicons using supervisedapproaches and features unique to that domain (e.g.,follower graphs; Speriosu et al., 2011). Here, weemphasize that we can induce an accurate lexiconusing a simple domain-independent and resource-light approach, with the implication that lexiconscan easily be induced for related social mediadomains without resorting to complex supervisedframeworks. We evaluate our approach using thetest set from the 2015 SemEval task 10E competition(Rosenthal et al., 2015), and we use the embeddingsconstructed by Rothe et al. (2016).6

4.1 Baselines and state-of-the-art comparisons

We compare SENTPROP against standard baselinesand state-of-the-art approaches. The PMI base-line of Turney and Littman (2003) computes thepointwise mutual information between the seeds andthe targets without using propagation. The base-line method of Velikovich et al. (2010) is simi-lar to our method but uses an alternative propa-gation approach and raw co-occurrence vectors in-stead of learned embeddings. Both these methodsrequire raw corpora, so they function as baselinesin cases where we do not use off-the-shelf embed-dings. We also compare against DENSIFIER, a state-of-the-art method that learns orthogonal transforma-tions of word vectors instead of propagating labels(Rothe et al., 2016). Lastly, on standard English wecompare against a state-of-the-art WordNet-basedmethod, which performs label propagation over aWordNet-derived graph (San Vicente et al., 2014).Several variant baselines, all of which SENTPROP

6The official SemEval task 10E involved fully-supervisedlearning, so we do not use their evaluation setup.

Method AUC Ternary F1 τ

SENTPROP 90.6 58.6 0.44DENSIFIER 93.3 62.1 0.50WordNet 89.5 58.7 0.34Majority – 24.8 –

(a) Corpus methods outperform WordNet on standardEnglish. Using word-vector embeddings learned on a mas-sive corpus (1011 tokens), we see that both corpus-basedmethods outperform the WordNet-based approach overall.

Method AUC Ternary F1 τ

SENTPROP 86.0 60.1 0.50DENSIFIER 90.1 59.4 0.57Sentiment140 86.2 57.7 0.51Majority – 24.9 –

(b) Corpus approaches are competitive with a distantlysupervised method on Twitter. Using Twitter embeddingslearned from ∼109 tokens, we see that the semi-supervisedcorpus approaches using small seed sets perform very well.

Method AUC Ternary F1

SENTPROP 91.6 63.1DENSIFIER 80.2 50.3PMI 86.1 49.8Velikovich et al. (2010) 81.6 51.1Majority – 23.6

(c) SENTPROP performs best with domain-specific financeembeddings. Using embeddings learned from financial cor-pus (∼2×107 tokens) , SENTPROP significantly outperformsthe other methods.

Method AUC Ternary F1 τ

SENTPROP 83.8 53.0 0.28DENSIFIER 77.4 46.6 0.19PMI 70.6 41.9 0.16Velikovich et al. (2010) 52.7 32.9 0.01Majority – 24.3 –

(d) SENTPROP performs well on standard English evenwith 1000x reduction in corpus size. SENTPROP maintainsstrong performance even when using embeddings learnedfrom the 2000s decade of COHA (only 2×∼107 tokens).

Table 2: Results on recreating known lexicons.

outperforms, are omitted for brevity (e.g., usingword-vector cosines in place of PMI in Turney andLittman (2003)’s framework). Code for all thesevariants is available in the SOCIALSENT package.

4.2 Evaluation setupWe evaluate the approaches according to (i) theirbinary classification accuracy (ignoring the neutralclass, as is common in previous work), (ii) ternaryclassification performance (positive vs. neutral vs.negative)7, and (iii) Kendall τ rank-correlation withcontinuous human-annotated polarity scores.

For all methods in the ternary-classification con-dition, we use the class-mass normalization method(Zhu et al., 2003) to label words as positive, neutral,or negative. This method assumes knowledge of thelabel distribution—i.e., how many positive/negativevs. neutral words there are—and simply assigns la-bels to best match this distribution.

4.3 Evaluation resultsTables 2a-2d summarize the performance of ourframework along with baselines and other state-of-

7Only GI contains words explicitly marked neutral, so forternary evaluations in Twitter and Finance we sample neutralwords from GI to match its neutral-vs-not distribution.

the-art approaches. Our framework significantlyoutperforms the baselines on all tasks, outperforms astate-of-the-art approach that uses WordNet on stan-dard English (Table 2a), and is competitive withSentiment140 on Twitter (Table 2b), a distantly-supervised approach that uses signals from emoti-cons (Mohammad and Turney, 2010). DENSIFIER

also performs extremely well, outperforming SENT-PROP when off-the-shelf embeddings are used (Ta-bles 2a and 2b). However, SENTPROP significantlyoutperforms all other approaches when using thedomain-specific embeddings (Tables 2c and 2d).

Overall our results show that SENTPROP— a rel-atively simple method, which combines high-qualityword vectors embeddings with standard label prop-agation — can perform at a state-of-the-art level,even performing competitively with methods rely-ing on hand-curated lexical graphs. Unlike previouspublished approaches, SENTPROP is able to main-tain high accuracy even when modest-sized domain-specific corpora are used. In cases where very largecorpora are available and where there is an abun-dance of training data, DENSIFIER performs ex-tremely well, since it was designed for this sort ofsetting (Rothe et al., 2016).

We found that the baseline method of Velikovichet al. (2010), which our method is closely relatedto, performed relatively poorly with these domain-specific corpora. This indicates that using high-quality word-vector embeddings can have a drasticimpact on performance. However, it is worth notingthat Velikovich et al. (2010)’s method was designedfor high recall with massive corpora, so its poor per-formance in our regime is not surprising.

Lastly, we found that the choice of embeddingmethod could have a drastic impact. Prelimi-nary experiments on the COHA data showed thatusing word2vec SGNS vectors (with default set-tings) instead of our SVD-based embeddings led toa >40% performance drop for SENTPROP acrossall measures and a >10% performance drop forDENSIFIER. It is possible that certain settings ofword2vec could perform better, but previous workhas shown that SVD-based methods have superiorresults on smaller datasets and rare-word similaritytasks (Levy et al., 2015; Hamilton et al., 2016), sothis result is not surprising.

5 Inducing community-specific lexicons

As a first large-scale study, we investigate how sen-timent depends on the social context in which aword is used. It is well known that there is sub-stantial sociolinguistic variation between differentcommunities, whether these communities are de-fined geographically (Trudgill, 1974) or via under-lying sociocultural differences (Labov, 2006). How-ever, no previous work has systematically investi-gated community-specific variation in word senti-ment at a large scale. Yang and Eisenstein (2015)exploit social network structure in Twitter to infera small number (1-10) of communities and analyzedsentiment variation via a supervised framework. Ouranalysis extends this line of work by analyzing thesentiment across hundreds of user-defined commu-nities using only unlabeled corpora and a small setof “paradigm” seed words (the Twitter seed wordsoutlined in Table 1).

In our study, we induced sentiment lexicons forthe top-250 (by comment-count) subreddits from thesocial media forum Reddit.8 We used all the 2014comment data to induce the lexicons, with words

8Subreddits are user-created topic-specific forums.

lower cased and comments from bots and deletedusers removed.9 Sentiment was induced for the top-5000 non-stop words in each subreddit (again, bycomment-frequency).

5.1 Examining the lexicons

Analysis of the learned lexicons reveals the ex-tent to which sentiment can differ across com-munities. Figure 4 highlights some words withopposing sentiment in two communities: inr/TwoXChromosomes (r/TwoX), a communitydedicated to female perspectives and gender issues,the words crazy and insane have negative polarity,which is not true in the r/sports community, and,vice-versa, words like soft are positive in r/TwoXbut negative in r/sports.

To get a sense of how much sentiment differsacross communities in general, we selected a ran-dom subset of 1000 community pairs and examinedthe correlation in their sentiment values for highlysentiment-bearing words (Figure 5). We see that thedistribution is noticeably skewed, with many com-munity pairs having highly uncorrelated sentimentvalues. The 1000 random pairs were selected suchthat each member of the pair overlapped in at leasthalf of their top-5000 word vocabulary. We thencomputed the correlation between the sentiments inthese community-pairs. Since sentiment is noisy andrelatively uninteresting for neutral words, we com-pute τ25%, the Kendall-τ correlation over the top-25% most sentiment bearing words shared betweenthe two communities.

Analysis of individual pairs reveals some inter-esting insights about sentiment and inter-communitydynamics. For example, we found that the sentimentcorrelation between r/TwoX and r/TheRedPill(τ25% = 0.58), two communities that hold con-flicting views and often attack each other10, wasactually higher than the sentiment correlation be-tween r/TwoX and r/sports (τ25% = 0.41), twocommunities that are entirely unrelated. This re-sult suggests that conflicting communities may have

9https://archive.org/details/2015_reddit_comments_corpus

10This conflict is well-known on Reddit; for example, bothcommunities mention each others’ names along with fuck-basedprofanity in the same comment far more than one would expectby chance (χ2

1 > 6.8, p < 0.01 for both). r/TheRedPill isdedicated to male empowerment.

Figure 4: Word sentiment differs drastically between a community dedicated to sports (r/sports) and one dedicated tofemale perspectives and gender issues (r/TwoX). Words like soft and animal have positive sentiment in r/TwoX but negativesentiment in r/sports, while the opposite holds for words like crazy and insane.

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8

Community-community sentiment correlation (τ25%)

0

1

2

3

4

5

6

Cou

nt

Figure 5: There is a long tail of communities with very dif-ferent word sentiments. Some communities have very similarsentiment (e.g., r/sports and r/hockey), while other com-munity pairs differ drastically (e.g., r/sports and r/TwoX).

more similar sentiment in their language comparedto communities that are entirely unrelated.

6 Inducing diachronic sentiment lexicons

Sentiment also depends on the historical time-periodin which a word is used. To investigate this depen-dency, we use our framework to analyze how wordpolarities have shifted over the last 150 years. Thephenomena of amelioration (words becoming morepositive) and pejoration (words becoming more neg-ative) are well-discussed in the linguistic literature(Traugott and Dasher, 2001); however, no compre-hensive polarity lexicons exist for historical data(Cook and Stevenson, 2010). Such lexicons are cru-cial to the growing body of work on NLP analyses of

historical text (Piotrowski, 2012) which are inform-ing diachronic linguistics (Hamilton et al., 2016),the digital humanities (Muralidharan and Hearst,2012), and history (Hendrickx et al., 2011).

Our work is inspired by the only previous workon automatically inducing historical sentiment lexi-cons, Cook and Stevenson (2010); they use the PMImethod and a full modern sentiment lexicon as theirseed set, which relies on the assumption that allthese words have not changed in sentiment. In con-trast, in addition to our different algorithm, we use asmall seed set of words that were manually selectedbased on having strong and stable sentiment over thelast 150 years (Table 1; confirmed via historical en-tries in the Oxford English Dictionary).

6.1 Examining the lexicons

We constructed lexicons from COHA, since it wascarefully constructed to be genre balanced (e.g.,compared to the Google N-Grams; Pechenick et al.,2015). We built lexicons for all adjectives withcounts above 100 in a given decade and also for thetop-5000 non-stop words within each year. In boththese cases we found that>5% of sentiment-bearing(positive/negative) words completely switched po-larity during this 150-year time-period and >25%of all words changed their sentiment label (includ-ing switches to/from neutral).11 The prevalence of

11We defined the thresholds for polar vs. neutral using theclass-mass normalization method and compared scores aver-

(a) Lean becomes more positive. Lean underwent amelioration,becoming more similar to muscular and less similar to weak.

(b) Pathetic becomes more negative. Pathetic underwent pejo-ration, becoming similar to weak and less similar to passionate.

Figure 6: Examples of amelioration and pejoration.

full polarity switches highlights the importance ofhistorical sentiment lexicons for work on diachroniclinguistics and cultural change.

Figure 6a shows an example amelioration de-tected by this method: the word lean lost its neg-ative connotations associated with “weakness” andinstead became positively associated with conceptslike “muscularity” and “fitness”. Figure 6b showsan example pejoration, where pathetic, which usedto be more synonymous with passionate, gainedstronger negative associations with the conceptsof “weakness” and “inadequacy” (Simpson et al.,1989). In both these cases, semantic similaritiescomputed using our learned historical word vectorswere used to contextualize the shifts.

Some other well-known examples of sentimentchanges captured by our framework include the se-mantic bleaching of sorry, which shifted from nega-tive and serious (“he was in a sorry state”) to usesas a neutral discourse marker (“sorry about that”)and worldly, which used to have negative conno-tations related to materialism and religious impu-rity (“sinful worldly pursuits”) but now is frequentlyused to indicate sophistication (“a cultured, worldlywoman”) (Simpson et al., 1989). Our hope is thatthe full lexicons released with this work will spurfurther examinations of such historical shifts in sen-timent, while also facilitating CSS applications thatrequire sentiment ratings for historical text.

7 Conclusion

SENTPROP allows researchers to easily induce ro-bust and accurate sentiment lexicons that are rel-

aged over 1850-1880 to those averaged over 1970-2000.

evant to their particular domain of study. Suchlexicons are crucial to CSS research, as evidencedby our two studies showing that sentiment dependsstrongly on both social and historical context.

Our methodological comparisons show that sim-ply combining label propagation with high-qualityword vector embeddings can achieve state-of-the-art performance competitive with methods that relyon hand-curated dictionaries, and the code pack-age released with this work contains a full bench-mark toolkit for this area, including implementationsof several variants of SENTPROP. We hope thesetools will facilitate future quantitative studies on thedomain-dependency of sentiment.

Of course, the sentiment lexicons induced bySENTPROP are not perfect, which is reflected in theuncertainty associated with our bootstrap-sampledestimates. However, we believe that these user-constructed, domain-specific lexicons, which quan-tify uncertainty, provide a more principled founda-tion for CSS research compared to domain-generalsentiment lexicons that contain unknown biases.In the future our method could also be integratedwith supervised domain-adaption (e.g.,Yang andEisenstein, 2015) to further improve these domain-specific results.

Acknowledgements

The authors thank P. Liang for his helpful com-ments. This research has been supported in partby NSF CNS-1010921, IIS-1149837, IIS-1514268NIH BD2K, ARO MURI, DARPA XDATA, DARPASIMPLEX, Stanford Data Science Initiative, SAPStanford Graduate Fellowship, NSERC PGS-D,Boeing, Lightspeed, and Volkswagen.

References

[Asghar et al.2015] Muhammad Zubair Asghar, Au-rangzeb Khan, Shakeel Ahmad, Imran Ali Khan,and Fazal Masud Kundi. 2015. A Unified Frame-work for Creating Domain Dependent Polarity Lex-icons from User Generated Reviews. PLOS ONE,10(10):e0140204, October.

[Blair-Goldensohn et al.2008] Sasha Blair-Goldensohn,Kerry Hannan, Ryan McDonald, Tyler Neylon,George A. Reis, and Jeff Reynar. 2008. Building asentiment summarizer for local service reviews. InWWW Workshop on NLP in the Information ExplosionEra.

[Bravo-Marquez et al.2015] Felipe Bravo-Marquez, EibeFrank, and Bernhard Pfahringer. 2015. From Unla-belled Tweets to Twitter-specific Opinion Words. InSIGIR.

[Bullinaria and Levy2012] John A. Bullinaria andJoseph P. Levy. 2012. Extracting semantic represen-tations from word co-occurrence statistics: stop-lists,stemming, and SVD. Behavior Research Methods,44(3):890–907, September.

[Choi and Cardie2009] Yejin Choi and Claire Cardie.2009. Adapting a polarity lexicon using integer lin-ear programming for domain-specific sentiment clas-sification. In EMNLP.

[Cook and Stevenson2010] Paul Cook and SuzanneStevenson. 2010. Automatically Identifying Changesin the Semantic Orientation of Words. In LREC.

[Davies2010] Mark Davies. 2010. The Corpus of Histor-ical American English: 400 million words, 1810-2009.http://corpus.byu.edu/coha/.

[Deng et al.2014] Lingjia Deng, Janyce Wiebe, and Yoon-jung Choi. 2014. Joint inference and disambiguationof implicit sentiments via implicature constraints. InCOLING.

[Esuli and Sebastiani2006] Andrea Esuli and Fabrizio Se-bastiani. 2006. Sentiwordnet: A publicly availablelexical resource for opinion mining. In LREC.

[Fast et al.2016] Ethan Fast, Binbin Chen, and Michael S.Bernstein. 2016. Empath: Understanding Topic Sig-nals in Large-Scale Text. In CHI.

[Fellbaum1998] Christiane Fellbaum. 1998. WordNet.Wiley Online Library.

[Hamilton et al.2016] William L. Hamilton, JureLeskovec, and Dan Jurafsky. 2016. DiachronicWord Embeddings Reveal Statistical Laws of Seman-tic Change. In ACL.

[Hatzivassiloglou and McKeown1997] Vasileios Hatzi-vassiloglou and Kathleen R. McKeown. 1997.Predicting the semantic orientation of adjectives. InEACL.

[Hendrickx et al.2011] Iris Hendrickx, Michel Gnreux,and Rita Marquilhas. 2011. Automatic pragmatictext segmentation of historical letters. In LanguageTechnology for Cultural Heritage, pages 135–153.Springer.

[Hovy2015] Dirk Hovy. 2015. Demographic factors im-prove classification performance. In ACL.

[Hu and Liu2004] Minqing Hu and Bing Liu. 2004. Min-ing and summarizing customer reviews. In KDD.

[Huang et al.2014] Sheng Huang, Zhendong Niu, andChongyang Shi. 2014. Automatic constructionof domain-specific sentiment lexicon based on con-strained label propagation. Knowledge-Based Sys-tems, 56:191–200.

[Igo and Riloff2009] Sean P. Igo and Ellen Riloff. 2009.Corpus-based semantic lexicon induction with web-based corroboration. In ACL Workshop on Unsuper-vised and Minimally Supervised Learning of LexicalSemantics.

[Jijkoun et al.2010] Valentin Jijkoun, Maarten de Rijke,and Wouter Weerkamp. 2010. Generating focusedtopic-specific sentiment lexicons. In ACL.

[Kamps et al.2004] Jaap Kamps, M. J. Marx, Robert J.Mokken, and M. de Rijke. 2004. Using wordnet tomeasure semantic orientations of adjectives. In LREC.

[Labov2006] William Labov. 2006. The social stratifica-tion of English in New York City. Cambridge Univer-sity Press.

[Landauer and Dumais1997] Thomas K. Landauer andSusan T. Dumais. 1997. A solution to Plato’s prob-lem: The latent semantic analysis theory of acquisi-tion, induction, and representation of knowledge. Psy-chol. Rev., 104(2):211.

[Lee et al.2014] Heeyoung Lee, Mihai Surdeanu, BillMacCartney, and Dan Jurafsky. 2014. On the Impor-tance of Text Analysis for Stock Price Prediction. InLREC.

[Levy et al.2015] Omer Levy, Yoav Goldberg, and IdoDagan. 2015. Improving distributional similarity withlessons learned from word embeddings. Trans. Assoc.Comput. Ling., 3.

[Loughran and McDonald2011] Tim Loughran and BillMcDonald. 2011. When is a liability not a liability?Textual analysis, dictionaries, and 10-Ks. The Journalof Finance, 66(1):35–65.

[Mikolov et al.2013] Tomas Mikolov, Ilya Sutskever, KaiChen, Greg S. Corrado, and Jeff Dean. 2013. Dis-tributed representations of words and phrases and theircompositionality. In NIPS.

[Mohammad and Turney2010] Saif M. Mohammad andPeter D. Turney. 2010. Emotions evoked by commonwords and phrases: Using Mechanical Turk to createan emotion lexicon. In NAACL.

[Muralidharan and Hearst2012] Aditi Muralidharan andMarti A Hearst. 2012. Supporting exploratory textanalysis in literature study. Literary and LinguisticComputing.

[Pechenick et al.2015] Eitan Adam Pechenick, Christo-pher M. Danforth, and Peter Sheridan Dodds. 2015.Characterizing the Google Books corpus: Strong lim-its to inferences of socio-cultural and linguistic evolu-tion. PLoS ONE, 10(10).

[Pennington et al.2014] Jeffrey Pennington, RichardSocher, and Christopher D. Manning. 2014. Glove:Global vectors for word representation. In EMNLP.

[Piotrowski2012] Michael Piotrowski. 2012. Natural lan-guage processing for historical texts. Synthesis Lec-tures on Human Language Technologies, 5(2):1–157.

[Qiu et al.2009] Guang Qiu, Bing Liu, Jiajun Bu, andChun Chen. 2009. Expanding Domain SentimentLexicon through Double Propagation. In IJCAI.

[Rao and Ravichandran2009] Delip Rao and DeepakRavichandran. 2009. Semi-supervised polarity lexi-con induction. In EACL.

[Riloff and Shepherd1997] Ellen Riloff and Jessica Shep-herd. 1997. A corpus-based approach for buildingsemantic lexicons. arXiv preprint cmp-lg/9706013.

[Rooth et al.1999] Mats Rooth, Stefan Riezler, DetlefPrescher, Glenn Carroll, and Franz Beil. 1999. In-ducing a semantically annotated lexicon via EM-basedclustering. In ACL.

[Rosenthal et al.2015] Sara Rosenthal, Preslav Nakov,Svetlana Kiritchenko, Saif M. Mohammad, Alan Rit-ter, and Veselin Stoyanov. 2015. SemEval-2015 task10: Sentiment analysis in Twitter. SemEval-2015.

[Rothe et al.2016] Sascha Rothe, Sebastian Ebert, andHinrich Schutze. 2016. Ultradense Word Embeddingsby Orthogonal Transformation. In NAACL-HLT.

[San Vicente et al.2014] Inaki San Vicente, RodrigoAgerri, German Rigau, and Donostia-San Sebastin.2014. Simple, Robust and (almost) UnsupervisedGeneration of Polarity Lexicons for Multiple Lan-guages. In EACL.

[Severyn and Moschitti2015] Aliaksei Severyn andAlessandro Moschitti. 2015. On the automaticlearning of sentiment lexicons. In NAACL-HLT.

[Simpson et al.1989] John Andrew Simpson, Edmund SCWeiner, et al. 1989. The Oxford English Dictionary,volume 2. Clarendon Press Oxford, Oxford, UK.

[Speriosu et al.2011] Michael Speriosu, Nikita Sudan, SidUpadhyay, and Jason Baldridge. 2011. Twitter po-larity classification with label propagation over lexicallinks and the follower graph. In ACL Workshop onUnsupervised Learning in NLP.

[Stone et al.1966] Philip J Stone, Dexter C Dunphy, andMarshall S Smith. 1966. The General Inquirer: AComputer Approach to Content Analysis.

[Taboada et al.2011] Maite Taboada, Julian Brooke, Mi-lan Tofiloski, Kimberly Voll, and Manfred Stede.2011. Lexicon-based methods for sentiment analysis.Comput. Ling., 37(2):267–307.

[Tai and Kao2013] Yen-Jen Tai and Hung-Yu Kao. 2013.Automatic domain-specific sentiment lexicon genera-tion with label propagation. In Proceedings of Inter-national Conference on Information Integration andWeb-based Applications & Services, page 53. ACM.

[Takamura et al.2005] Hiroya Takamura, Takashi Inui,and Manabu Okumura. 2005. Extracting semantic ori-entations of words using spin model. In ACL.

[Tang et al.2014] Duyu Tang, Furu Wei, Bing Qin, MingZhou, and Ting Liu. 2014. Building Large-ScaleTwitter-Specific Sentiment Lexicon: A RepresentationLearning Approach. In COLING.

[Thelen and Riloff2002] Michael Thelen and Ellen Riloff.2002. A bootstrapping method for learning semanticlexicons using extraction pattern contexts. In EMNLP.

[Traugott and Dasher2001] Elizabeth Closs Traugott andRichard B Dasher. 2001. Regularity in SemanticChange. Cambridge University Press, Cambridge,UK.

[Trudgill1974] Peter Trudgill. 1974. Linguistic changeand diffusion: Description and explanation in soci-olinguistic dialect geography. Language in Society,3(2):215–246.

[Turney and Littman2003] Peter D. Turney andMichael L. Littman. 2003. Measuring praiseand criticism: Inference of semantic orientation fromassociation. ACM Trans. Inf. Sys., 21(4):315–346.

[Turney and Pantel2010] Peter D. Turney and PatrickPantel. 2010. From frequency to meaning: Vec-tor space models of semantics. J. Artif. Intell. Res.,37(1):141–188.

[Velikovich et al.2010] Leonid Velikovich, Sasha Blair-Goldensohn, Kerry Hannan, and Ryan McDonald.2010. The viability of web-derived polarity lexicons.In NAACL-HLT.

[Warriner et al.2013] Amy Beth Warriner, Victor Kuper-man, and Marc Brysbaert. 2013. Norms of valence,arousal, and dominance for 13,915 English lemmas.Behavior research methods, 45(4):1191–1207.

[Widdows and Dorow2002] Dominic Widdows and BeateDorow. 2002. A graph model for unsupervised lexicalacquisition. In COLING.

[Yang and Eisenstein2015] Yi Yang and Jacob Eisenstein.2015. Putting Things in Context: Community-specificEmbedding Projections for Sentiment Analysis. arXivpreprint arXiv:1511.06052.

[Zhou et al.2004] Dengyong Zhou, Olivier Bousquet,Thomas Navin Lal, Jason Weston, and BernhardScholkopf. 2004. Learning with local and global con-sistency. In NIPS.

[Zhu and Ghahramani2002] Xiaojin Zhu and ZoubinGhahramani. 2002. Learning from labeled and un-labeled data with label propagation. Technical report.

[Zhu et al.2003] Xiaojin Zhu, Zoubin Ghahramani, JohnLafferty, and others. 2003. Semi-supervised learn-ing using Gaussian fields and harmonic functions. InICML.