Proceedings of the 2019 EMNLP and the 9th IJCNLP (System Demonstrations), pages 205–210Hong Kong, China, November 3 – 7, 2019. c©2019 Association for Computational Linguistics

205

A Stylometry Toolkit for Latin Literature

Thomas J. Bolt,1 Jeffrey H. Flynt,2 Pramit Chaudhuri,1 Joseph P. Dexter3†1 Department of Classics, University of Texas at Austin

2 Department of Molecular Biosciences, University of Texas at Austin3 Neukom Institute for Computational Science, Dartmouth College† Corresponding author: joseph.p.dexter@dartmouth.edu

Abstract

Computational stylometry has become an in-creasingly important aspect of literary criti-cism, but many humanists lack the technicalexpertise or language-specific NLP resourcesrequired to exploit computational methods.We demonstrate a stylometry toolkit for analy-sis of Latin literary texts, which is freely avail-able at www.qcrit.org/stylometry.Our toolkit generates data for a diverse rangeof literary features and has an intuitive point-and-click interface. The features includedhave proven effective for multiple literarystudies and are calculated using custom heuris-tics without the need for syntactic parsing. Assuch, the toolkit models one approach to theuser-friendly generation of stylometric data,which could be extended to other premod-ern and non-English languages underserved bystandard NLP resources.

1 Introduction

Stylometry, the quantitative analysis of writingstyle, is an longstanding yet active area of re-search in literary studies. Traditional applicationsof stylometry in both classical and modern literaryscholarship have focused on authorship attributionand establishing relative chronology (Mostellerand Wallace, 1964; Marriott, 1979; Fitch, 1981;Vickers, 2004; Jockers and Witten, 2010; Stoveret al., 2016). In recent years, new digital toolsand computational methods, especially machinelearning (Long and So, 2016; Dexter et al., 2017),have allowed researchers to address more fine-grained literary critical questions and have alsogiven rise to novel frameworks for literary anal-ysis, such as ‘distant reading’ and ‘macroanaly-sis’ (Moretti, 2013; Jockers, 2013; Piper, 2018;Underwood, 2019).

Much research in computational stylometry hasfocused on English literature due in part to the rich

NLP resources available for the English language,especially high-quality syntactic parsing. NLP re-sources for many premodern and non-English lan-guages are, by contrast, at an earlier stage of de-velopment or entirely lacking. Moreover, manyof the academic disciplines studying these lan-guages are smaller than for English, and thus thecommunity of potential developers is correspond-ingly reduced. These factors suggest the need foruser-friendly stylometric tools, which can providea wide range of literary data for under-resourcedlanguages and are suitable for use by humanistslacking a computational background.

Syntactic parsing, which remains at an earlystage of development for Latin,1 is not a prereq-uisite for the successful application of computa-tional stylometry to literary problems. Our priorwork has shown that custom heuristics can enableextraction of a wide range of features useful forthe study of Latin literature, in particular syntac-tic markers, non-content words, and elements ofsound and rhythm (Dexter et al., 2017; Chaudhuriet al., 2018). Here we report development of apoint-and-click stylometry toolkit to enable easygeneration of such data for a corpus containing al-most all major classical Latin texts.

Other recently developed stylometry packages,such as the “stylo” R package and Lexomics, areaimed at audiences with a range of computationalexpertise (Eder et al., 2016; Drout et al., 2007).These packages, however, have typically been de-veloped for general-purpose application to multi-ple languages instead of a single language. Focus-ing on the latter creates opportunities for targetinglanguage-specific features, which often play a cru-cial role in literary style.

1See, for example, the recent progress of the Clas-sical Language Toolkit (CLTK) (www.cltk.org) andStanfordNLP (https://stanfordnlp.github.io/stanfordnlp/index.html).

206

The need for a point-and-click toolkit is partic-ularly acute in classical studies. Although clas-sical philologists have long applied stylometry toshed light on questions of authorship, relativelyfew studies have employed digital tools. Excep-tions have tended to focus on a restricted set offeatures, such as relative word frequency (Stoveret al., 2016) or average sentence length (Marriott,1979; Clayman, 1981). Such limitations may bedue in part to the absence of an accurate methodfor syntactic parsing, and in part to a more gen-eral lack of collaboration to date between clas-sical philologists and NLP specialists. By im-proving the accessibility of rich philological data,our toolkit should further promote the adoptionof quantitative approaches by literary critics. Atthe same time, the toolkit bridges the gap betweenclassical studies and research on English, in whichcomputational approaches are more common andare supported by a more extensive technical appa-ratus.

2 Toolkit

Our toolkit provides researchers working withLatin literature access to large-scale stylomet-ric data difficult to acquire by non-computationalmethods and enables humanists without specialistdigital training to construct custom datasets.

The design goal for the toolkit is to providean intuitive and easy-to-use interface hosted in aweb browser. The interface is point-and-click andcan be used by researchers with no prior program-ming or NLP experience. Users can choose fromover 700 Latin texts, which comprise almost allof the surviving corpus of classical Latin. Thetexts were originally digitized by the Perseus Dig-ital Library and further developed by the TesseraeProject (Crane, 1996; Coffee et al., 2012). Textscan be selected by author, text, or book (roughlythe ancient equivalent of a chapter). Searches canbe as fine-grained as examining a single book, oras large-scale as analyzing the entire built-in cor-pus in one go (Figure 1).

Next, users select the stylometric features to an-alyze for their chosen corpus. They can run anal-yses using any combination of the twenty-six fea-tures (Figure 2 shows a sample output). The re-sults are displayed on a spreadsheet in the webbrowser and can be downloaded as a CSV file. Inaddition, a user can produce simple visualizations(e.g., a bar chart comparing the values of a partic-

ular feature across a set of texts) inside the toolkit.The ease with which the toolkit can be used

does not limit its versatility. A user can create acustom corpus of texts preselected from the ex-isting database, which is close to comprehensivefor canonical material, or upload texts of theirown for analysis. This latter functionality is es-pecially important for understudied texts, such asthose produced in Late Antiquity and during theRenaissance, the sum total of which far exceedsthe quantity of extant classical Latin. While dig-ital versions are available for many post-classicaltexts, for the most part the later periods are notwell served by the prominent tools or repositoriesin the field, which maintain a classical focus. Ourtoolkit allows users to analyze any text availablein electronic form. Furthermore, if a work is notavailable online, a user may upload a plain text fileor transcribe it directly into the upload interface.

3 Features

Our feature set comprises twenty-six stylometricfeatures across four broad syntactic and grammat-ical categories (pronouns and non-content adjec-tives, subordinate clauses, conjunctions, and mis-cellaneous, as listed in Table 1) and is described indetail in a previous publication (Chaudhuri et al.,2018). Some features are lexical (e.g., preposi-tions), while others are syntactic (e.g., sentencelength) or address semantic and rhetorical aspectsof the texts (e.g., superlatives and interrogativesentences). Taken together, the features offer arich and diverse, albeit necessarily partial, profileof Latin literary style.

An important aspect of our toolkit is that itdoes not depend on syntactic parsing, named en-tity recognition, or other NLP methods that havenot been developed fully for classical Latin (Erd-mann et al., 2016). We employ three strate-gies to circumvent current technical limitations.The majority of features (Alius, Idem, Ipse, Iste,Quidam, Demonstrative Pronouns, Personal Pro-nouns, Third-Person Pronouns, Atque + Conso-nant, Antequam, Cum, Dum, Priusquam, Quin,Quominus, Ut, and Prepositions) are computed us-ing hard-coded lists of almost all possible formsof the relevant Latin words. While some features(e.g., Quin or Dum) are frequencies of a singleform, others (e.g., Demonstrative Pronouns) in-volve long lists of morphological variants. Otherfeatures are estimated based on the frequency of a

207

Figure 1: On the left, drop-down menu for text selection; on the right, a list of texts that have been selected.

Figure 2: Sample output from the toolkit for a selection of Latin literary texts.

signal n-gram. For instance, all regular superlativeadjectives include the n-gram -issim- (e.g., largis-simus, “most abundant” or clarissima, “clearest”).As this n-gram is extremely rare outside of su-perlatives, we could curate a near-comprehensivelist of exclusions (e.g., dissimilis, “unlike”). Weuse a similar strategy to capture the instances ofselected gerunds and gerundives, which containthe n-grams -ndus, -ndum, -ndarum, or -ndorum.A third class of features are determined usingpunctuation (e.g., question marks to assess the fre-quency of direct interrogative sentences or to filterinterrogative pronouns, which have many formsin common with relative pronouns, from relativeclause counts).

The precision and recall of each of these heuris-tics is discussed in detail in (Chaudhuri et al.,2018). We emphasize that these approaches arenot intended as a substitute for NLP, but ratheras a stopgap for philologists until more substan-tial resources become available for classical lan-

guages. We expect that the overall usefulness ofthe toolkit will increase as our heuristics are ren-dered obsolete by improvements in part-of-speechtagging and dependency parsing for Latin.

Our features are drawn from a wide array ofsources in order to maximize the capture of infor-mation pertinent to Latin literary style. Some fea-tures, such as prepositions, are inspired by stud-ies of other languages, where they have provenuseful for the characterization of genres or sub-genres (Jockers, 2013). Most features, however,are based on previous studies of Latin style and aredesigned to capture aspects specific to the Latinlanguage (Adams, 1972; Adams et al., 2005). Forexample, atque (“and”) followed by a word begin-ning with a consonant is a stylistic feature that isassociated with certain influential figures writingearly in the tradition. When later authors employatque + consonant, they do so either in imitation ofthese figures specifically, or to recall an archaizingstyle more generally.

208

FeaturePronouns and non-content adjectives

1 Alius2 Idem3 Ipse4 Iste5 Quidam6 Demonstrative Pronouns7 Personal Pronouns8 Third-Person Pronouns

Conjunctions9 Atque + Consonant10 Conjunctions

Subordinate clauses11 Antequam12 Cum13 Dum14 Priusquam15 Quin16 Quominus17 Conditional Markers18 Fraction of Sentences with Relative Clauses19 Mean Length of Relative Clauses

Miscellaneous20 Ut21 Interrogative Sentences22 Mean Length of Sentences23 Prepositions24 Regular Superlatives25 Selected Gerunds & Gerundives26 Selected Vocatives

Table 1: Full set of Latin stylometric features.

4 Literary Importance

The stylometric data generated by the toolkit shedslight on a variety of literary problems. The sim-plest type of analysis involves a single feature cal-culated across a small number of texts. Past re-search in Ancient Greek stylometry, for instance,has shown that sentence length constitutes onemeaningful difference between the early Home-ric hexameter tradition and the Hellenistic tradi-tion, since later writers use longer sentences evenas they retain other core aspects such as formu-laic language and meter (Clayman, 1981). Fig-ure 3 shows the mean sentence length of most ofthe surviving classical Latin epics as calculatedby the toolkit. Three texts, De Rerum Naturaby Lucretius, Astronomicon by Manilius, and the

Georgics by Vergil, have noticeably longer sen-tences on average (mean length >140 characters,compared to <125 characters for the other epics).An attractive explanation for the three anomaloustexts is that they are all identified with a sub-genre of epic known as “didactic,” a specific classwhich purports to teach its readers philosophy ora specialized technical skill, such as astrology orfarming. The sentences are longer plausibly be-cause detailed treatment of intricate philosophi-cal or technical issues requires more complex sen-tences than typically more straightforward narra-tive action or direct speech, which represent theprincipal content of the other epics.

Figure 3: Mean sentence length of Latin epic poems (incharacters). Error bars denote one s.d. across the eightpoems.

The toolkit also reveals that Latin drama has ahigher frequency of personal pronouns than otherverse genres, as shown in Figure 4. This is nodoubt due to drama’s dialogic form: charactersspeak to each other directly, often employing first(“I” or “we”) and second person (“you”) pro-nouns. Many other literary genres primarily em-ploy a narrative structure in which a narrator de-scribes the action. This narrative type often usesthird person pronouns (“he,”“she,” “it”), but rarelyuses first or second person pronouns. Accordingly,the frequency of personal pronoun use is higher indrama. While this difference may be intuitive to areader, the large-scale data generated by the toolkitoffers quantitative evidence of a genre’s formalstyle, which would otherwise be difficult if not im-

209

possible to calculate by hand.

Figure 4: Mean per-character frequency of personalpronouns in the major genres of Latin literature. Er-ror bars denote one s.d. across the texts within each ofthe four genres.

Finally, the toolkit can also generate input datafor supervised and unsupervised machine learninganalyses. In our recent study of Latin prose andverse, we trained a random forest classifier usingall 26 features to distinguish the two genres withhigh (>97%) accuracy (Chaudhuri et al., 2018).The underlying data can now be produced easilyusing the toolkit, and similar datasets can be con-structed for other machine learning applications.

5 Conclusion and Future Work

This paper introduces a stylometry toolkit forLatin literature, which incorporates a diverse fea-ture set demonstrably useful for literary criticism.The toolkit includes a point-and-click interfaceto maximize usage among core domain special-ists, principally researchers in the humanities, whomay not have specialized computational training.Future versions of the toolkit will further diver-sify the feature set, incorporating high-frequencyn-grams and sense-pauses alongside the existingcategories (Fitch, 1981; Dexter et al., 2017), andwill leverage expected advances in Latin NLP toimprove the methods for calculation of existingfeatures.

In related work, we have developed a similarfeature set for Ancient Greek, which has been usedto classify prose and verse and, at a more fine-grained level, epic and drama (Gianitsos et al.,

2019). Our work on Old English has demonstratedthe utility of related features for various literaryand attribution studies (Neidorf et al., 2019). Afterextension of the current toolkit to Ancient Greekand Old English, we plan in due course to incor-porate other underserved languages, in particularBengali.

Acknowledgments

This work was conducted under the auspices of theQuantitative Criticism Lab (www.qcrit.org),an interdisciplinary group co-directed by P.C. andJ.P.D. and supported by a National Endowment forthe Humanities Digital Humanities Start-Up Grant(grant number HD-10 248410-16) and an Ameri-can Council of Learned Societies (ACLS) DigitalExtension Grant. T.J.B. was supported by an En-gaged Scholar Initiative Fellowship from the An-drew W. Mellon Foundation, P.C. by an ACLSDigital Innovation Fellowship and a Mellon NewDirections Fellowship, and J.P.D. by a NeukomFellowship.

ReferencesJ.N. Adams. 1972. The language of the later books

of Tacitus’ Annals. The Classical Quarterly,22(2):350–373.

J.N. Adams, M. Lapidge, and T. Reinhardt. 2005. In-troduction. In J.N. Adams, M. Lapidge, and T. Rein-hardt, editors, Aspects of the Language of LatinProse. Proceedings of the British Academy, 129,pages 1–36. Oxford University Press, Oxford.

P. Chaudhuri, J.P. Dexter, T. Dasgupta, and K. Iyer.2018. A small set of stylometric features differen-tiates Latin prose and verse. Digital Scholarship inthe Humanities.

D.L. Clayman. 1981. Sentence length in Greek hex-ameter poetry. In R. Grotjahn, editor, HexameterStudies. Quantitative Linguistics 11, pages 107–136.Brockmeyer, Bochum.

N. Coffee, J.-P. Koenig, S. Poornima, R. Ossewaarde,C. Forstall, and S. Jacobson. 2012. Intertextuality inthe digital age. Transactions of the American Philo-logical Association, 142(2):383–422.

G. Crane. 1996. Building a digital library: The PerseusProject as a case study in the humanities. In Pro-ceedings of the First ACM International Conferenceon Digital Libraries, pages 3–10.

J.P Dexter, T. Katz, N. Tripuraneni, T. Dasgupta,A. Kannan, J.A. Brofos, J.A. Bonilla Lopez, L.A.

210

Schroeder, A. Casarez, M. Rabinovich, A. Haim-son Lushkov, and P. Chaudhuri. 2017. Quantita-tive criticism of literary relationships. Proceed-ings of the National Academy of Sciences USA,114(16):E3195–204.

M.D.C. Drout, M.J. Kahn, M.D. LeBlanc, and C. Nel-son. 2007. Of dendrogrammatology: Lexomicmethods for analyzing relationships among Old En-glish poems. Journal of English and GermanicPhilology, 110(3):301–336.

M. Eder, J. Rybicki, and M. Kestemont. 2016. Sty-lometry with R: A package for computational textanalysis. The R Journal, 8(1):107–121.

A. Erdmann, C. Brown, B. Joseph, M. Janse, P. Ajaka,M. Elsner, and M.-C. de Marneffe. 2016. Chal-lenges and solutions for Latin named entity recogni-tion. In Proceedings of the Workshop on LanguageTechnology Resources and Tools for Digital Human-ities (LT4DH), pages 85–93, Osaka, Japan.

J.G. Fitch. 1981. Sense-pauses and relative datingin Seneca, Sophocles and Shakespeare. AmericanJournal of Philology, 102(3):289–307.

E.T. Gianitsos, T.J. Bolt, P. Chaudhuri, and J.P. Dexter.2019. Stylometric classification of Ancient Greekliterary texts by genre. In Proceedings of the 3rdJoint SIGHUM Workshop on Computational Lin-guistics for Cultural Heritage, Social Sciences, Hu-manities and Literature, pages 52–60, Minneapolis,USA.

M. Jockers. 2013. Macroanalysis. University of Illi-nois Press, Champaign, IL.

M. Jockers and D.M. Witten. 2010. A compara-tive study of machine learning methods for author-ship attribution. Literary and Linguistic Computing,25(2):215–223.

H. Long and R.J. So. 2016. Literary pattern recog-nition: Modernism between close reading and ma-chine learning. Critical Inquiry, 42(2):235–267.

I. Marriott. 1979. The authorship of the Historia Au-gusta: Two computer studies. Journal of RomanStudies, 69:65–77.

F. Moretti. 2013. Distant Reading. Verso, London.

F. Mosteller and D.L. Wallace. 1964. Inference andDisputed Authorship: The Federalist. Addison-Wesley, Reading, MA.

L. Neidorf, M.S. Krieger, M. Yakubek, P. Chaudhuri,and J.P. Dexter. 2019. Large-scale quantitative pro-filing of the Old English verse tradition. Nature Hu-man Behaviour, 3(6):560–567.

A. Piper. 2018. Enumerations: Data and LiteraryStudy. University of Chicago Press, Chicago.

J. Stover, Y. Winter, M. Koppel, and M. Keste-mont. 2016. Computational authorship verificationmethod attributes a new work to a major 2nd centuryAfrican author. Journal of the Association for Infor-mation Science and Technology, 67(1):239–243.

T. Underwood. 2019. Distant Horizons: Digital Evi-dence and Literary Change. University of ChicagoPress, Chicago.

B. Vickers. 2004. Shakespeare, Co-author: A Histori-cal Study of Five Collaborative Plays. Oxford Uni-versity Press, Oxford.