Annotating Shakespeare’s Sonnets with Appraisal Theory to Detect Irony

Nicolò BusettoDepartment of Linguistic Studies

Ca Foscari UniversityCa Bembo - Venezia

830070@stud.unive.it

Rodolfo DelmonteDepartment of Linguistic Studies

Ca Foscari UniversityCa Bembo - Venezia

delmont@unive.it

Abstract

English. In this paper we propose an ap-proach to irony detection based on Ap-praisal Theory(Martin and White(2005))in Shakespeare’s Sonnets, a well-knowndata set that is statistically valuable. Inorder to produce meaningful experiments,we created a gold standard by collectingopinions from famous literary critics onShakespeare’s Sonnets focusing on irony.We started by manually annotating thedata using Appraisal Theory as a refer-ence theory. This choice is motivated bythe fact that Appraisal annotation schemesallow smooth evaluation of highly elab-orated texts like political commentaries.The annotation is then automatically com-piles and checked against the gold stan-dard in order to verify the persistence ofcertain schemes that can be identified asironic, satiric or sarcastic. Upon observa-tion, irony detection reaches a final matchof 80%1.

Italiano. In questo articolo si propone unapproccio basato sulla Appraisal Theoryper l’individuazione dell’ironia nei Sonettidi Shakespeare, un dataset che è statistica-mente valido. Allo scopo di produrre es-perimenti significativi, abbiamo creato ungold standard raccogliendo le opinioni difamosi critici letterari sullo stesso corpus,con l’ironia come tema. Abbiamo poi an-notato manualmente i sonetti utilizzandogli strumenti e i tratti della Appraisal The-ory che permettono di ottenere una valu-tazione di testi altamente elaborati comegli articoli di politica. L’annotazione è

1Copyright c© 2019 for this paper by its authors. Usepermitted under Creative Commons License Attribution 4.0International (CC BY 4.0)

stata poi raccolta automaticamente e con-frontata con il gold standard per verificarela persistenza di certi schemi che possonoessere identificati come ironici, satirici osarcastici, raggiungendo una corrispon-denza finale del 80%.

1 Introduction

Shakespeare’s Sonnets are a collection of 154 po-ems which is renowned for being full of ironiccontent (Weiser(1983)), (Weiser(1987)) and for itsambiguity thus sometimes reverting the overall in-terpretation of the sonnet. Lexical mbiguity, i.e.a word with several meanings, emanates from theway in which the author uses words that can beinterpreted in more ways not only because inher-ently polysemous, but because sometimes the ad-ditional meaning meaning they evoke can some-times be derived on the basis of the sound, i.e.homophone (see “eye”, “I” in sonnet 152). Thesonnets are also full of metaphors which manytimes requires contextualising the content to thehistorical Elizabethan life and society. Further-more, there is an abundance of words related tospecific language domains in the sonnets. For in-stance, there are words related to the language ofeconomy, war, nature and to the discoveries of themodern age, and each of these words may be usedas a metaphor of love. Many of the sonnets areorganized around a conceptual contrast, an oppo-sition that runs parallel and then diverges, some-times with the use of the rhetorical figure of thechiasmus. It is just this contrast that generatesirony, sometimes satire, sarcasm, and even par-ody. Irony may be considered in turn as: whatone means using language that normally signifiesthe opposite, typically for humorous or emphaticeffect; a state of affairs or an event that seemscontrary to what one expects and is amusing asa result. As to sarcasm this may be regarded the

use of irony to mock or convey contempt. Par-ody is obtained by using the words or thoughtsof a person but adapting them to a ridiculouslyinappropriate subject. There are several types ofirony, though we select verbal irony which, in thestrict sense, is saying the opposite of what youmean for outcome, and it depends on the extra-linguistics context(Attardo(1994)). As a result,Satire and Irony are slightly overlapping but con-stitute two separate techniques; eventually Sar-casm can be regarded as a specialization or a sub-set of Irony. It is important to remark that in manycases, these linguistic structures may require theuse of nonliteral or figurative language, i.e. the useof metaphors. This has been carefully taken intoaccount when annotating the sonnets by meansof Appraisal Theory Framework (hence ATF). Inour approach we will follow the so-called incon-gruity presumption or incongruity-resolution pre-sumption. Theories connected to the incongruitypresumption are mostly cognitive-based and re-lated to concepts highlighted for instance, in (At-tardo(2000)). The focus of theorization under thispresumption is that in humorous texts, or broadlyspeaking in any humorous situation, there is an op-position between two alternative dimensions. As aresult, we will look for contrast in our study of thesonnets, produced by the contents of manual clas-sification. The purpose of this study is to showhow ATF can be useful for detecting irony, con-sidering its ambiguity and its elusive traits.

2 Producing the Gold Standard

In order to produce a gold standard that may en-compass strong hints to classification in terms ofhumour as explained above, we collected literarycritics’ reviews of the sonnets. We used criticismfrom a set of authors including (Frye(1957))(Calimani(2009)) (Melchiori(1971)) (Ea-gle(1916)) (Marelli(2015)) (Schoenfeldt(2010))(Weiser(1987)) (Serpieri(2002)) all listed in thereference section. The gold standard classificationhas been produced by second author and checkedby first author. It is organized into a numberof separate fields in a sequence to allow thereader to get a better picture of the sonnet in thecollection. All classifications are reported in asupplementary file in the Appendix. Here beloware the classifications for two sonnets:

• SONNET 8SEQUENCE: 1-17 Procreation MAIN

THEME: One against many ACTION: Youngman urged to reproduce METAPHOR:Through progeny the young man will not bealone NEG.EVAL: The young man seemsto be disinterested POS.EVAL: Young manpositive aesthetic evaluation CONTRAST:Between one and many

• SONNET 21SEQUENCE: 18-86 Time and Immortal-ity MAIN THEME: Love ACTION: TheYoung man must understand the sincerityof poet’s love METAPHOR: True love issincere NEG.EVAL: The young man listensthe false praise made by others POS.EVAL:Young Man positive aesthetic evaluationCONTRAST: Between true and fictitious love

As can be seen, we indicate SEQUENCE forthe thematic sequence into which the sonnet is in-cluded; this is followed by MAIN THEME whichis the theme the sonnet deals with; ACTION re-ports the possible action proposed by the poetto the protagonist of the poem; METAPHOR isthe main metaphor introduced in the poem some-times using words from a specialized domain;NEG.EVAL and POS.EVAL stand for NegativeEvaluation and Positive Evaluation contained inthe poem in relation to the theme and the protag-onist(s); finally, CONTRAST is the key to signalpresence of opposing concrete or abstract conceptsused by Shakespeare to reinforce the argumentspurported in the poem. Many sonnets have re-ceived more than one possible pragmatic category.This is due to the difficulty in choosing one cate-gory over another. In particular, it has been par-ticular hard to distinguish Irony from Satire, andIrony from Sarcasm. Overall, we ended up with 54sonnets receiving a double marking over 98, rep-resenting the total number of sonnets with somekind of pragmatic label by the literary critics, witha ratio of 98/154, corresponding to a percentage of63.64%. We ended up with the count of annotatedsonnets reported above in Table 1.

Eventually, as commented in the section be-low, the introduction of annotations based on Ap-praisal Theory has helped in choosing best prag-matic classification. In fact, literary critics weresimply hinting at "irony" or "satire", but the anno-tation gave us a precise measure of the level ofcontrast present in each of the sonnets regardedgenerically as "ironic".

Table 1: Final distribution of sonnets in the 5 prag-matic categories

Type QuantityBlank 57Irony 73Satire 20Parody 4Sarcasm 47Duplicated 54

2.1 Appraisal Theory for Poetry andLiterary Texts

The experiment we have been working on is anattempt to describe irony, parody and sarcasm interms of a strict scientifically viable linguistic the-ory, the Appraisal Framework Theory (Martin andWhite(2005)), as has already been done in the pastby other authors (see (Taboada and Grieve(2004))(Read and Carrol(2012)) but also (Stingo and Del-monte(2016)) (Delmonte and Marchesini(2017)) .The idea is as follows: produce a complete anno-tation of the sonnets using the tools made avail-able by the theory and then verify how well it fitsinto the gold standard produced. The primary pur-pose of the Appraisal Framework Theory(henceAFT) is to delineate the interpersonal dimensionof communication, supplying schemes by whichit is possible to recognize evaluative sequenceswithin texts and information about the positioningof the author in relation to evaluated targets.2

The annotation has been organized around onlyone category, Attitude, and its direct subcate-gories, in order to keep the annotation at a moreworkable level, and to optimize time and space inthe XML annotation. Attitude includes differentoptions for expressing positive or negative evalua-tion, and expresses the author’s feelings. The maincategory is divided into three primary fields withtheir relative positive or negative polarity, namely:

• Affect is every emotional evaluation ofthings, processes or states of affairs, (e.g.like/dislike), it describes proper feelings andany emotional reaction within the text aimedtowards human behaviour/process and phe-nomena.

2Further information can be found on the dedicatedwebsite dedicated to the Appraisal Framework Theory:http://www.languageofevaluation.info/appraisal/

• Judgement is any kind of ethical evaluation ofhuman behaviour, (e.g. good/bad), and con-siders the ethical evaluation on people andtheir behaviours.

• Appreciation is every aesthetic or functionalevaluation of things, processes and state ofaffairs (e.g. beautiful/ugly; useful/useless),and represent any aesthetic evaluation ofthings, both man-made and natural phenom-ena.

Eventually, we end up with six different classes:Affect positive, Affect Negative, Judgement Pos-itive, Judgement Negative, Appreciation Positive,Appreciation Negative. Overall in the annotationthere is a total majority of positive polarities witha ratio of 0.511, in comparison to negative anno-tations with a ratio of 0.488. In short, the wholeof the positive poles is 607, and the totality of thenegative poles is 579 for a total number of 1186annotations. Judgement is the more interestingcategory because it allows social moral sanction,in that it refers to two subfields, Social Esteemand Social Sanction - which however we decidednot to mark. In particular, whereas the positivepolarity annotation of Judgement extends to Ad-miration and Praise, the negative polarity annota-tion deals with Criticism and Condemnation or So-cial Esteem and Social Sanction (see (Martin andWhite(2005)), p.52). In particular, Judgement isfound mainly in the final couplet of the sonnets.

The annotation work on the texts has beenaccomplished by first author and checked bysecond author. Given the level of objectivedifficulty in understanding the semantic contentof the sonnets, we have decided not to resort toadditional annotators - second author producedthe annotation as part of his Master thesis work.So far, we have not been able to produce a mea-sure for interannotator agreement: however, sinceI was obliged to correct 35% of all annotationsthat measure could be approximated by 65% ofagreement. The tags we used for the annotationinclude a tag for <text> contains the whole textof the sonnet; <p> to mark stanzas, and <s>to mark lines. Focusing on the annotation ofthe evaluative sequences instead, every time wefound an evaluative word (or sequence of words),we delimited the item/phrase within the tags<apprsl></apprsl>. Subsequently, following thegeneral indications mentioned above provided by

(Martin and White(2005)), we assigned one ofthe three subcategories – affect, judgement andappreciation – as an attribute of the tag <apprsl>,also providing the positive/negative sentimentorientation as a value of the attribute. Here belowwe show the annotation for Sonnet 40 which ishighly contrasted:

<?xml version="1.0" encoding="ISO-8859-1"?> <text> <p> <s> Take all my loves, my love,yea take them all, </s> <s> What hast thou thenmore than thou hadst before? </s> <s> No love,my<apprsl affect="positive">love,</apprsl>thatthou mayst<apprsl appreciation="positive">true</apprsl>love call, </s> <s> All mine wasthine, before thou hadst this more: </s></p> <p> <s> Then if for my<apprsl af-fect="positive">love,</apprsl>thou my<apprslaffect="positive">love</apprsl>receivest,</s> <s> I cannot<apprsl judg-ment="negative">blame</apprsl>thee, formy<apprsl affect="positive">love</apprsl>thouusest, </s> <s> But yet<apprsl judge-ment="negative">be blamed,</apprsl>ifthou thy self<apprsl judge-ment="negative">deceivest</apprsl></s> <s> By<apprsl apprecia-tion="negative">wilful</apprsl>tasteof what thy self<apprsl apprecia-tion="negative">refusest</apprsl> </s> </p><p> <s> <apprsl judgement="positive">Ido forgive</apprsl><apprsl judge-ment="negative">thy robbery</apprsl> <ap-prsl appreciation="positive">gentle</apprsl>thief </s> <s> textbfAlthough<apprsljudgement="negative">thou steal theeall my poverty:</apprsl> </s> <s>And yet love knows it is a<apprsl af-fect="negative">greater grief</apprsl> </s><s> To<apprsl appreciation="negative">bearlove’s wrong,</apprsl>than<apprsl apprecia-tion="negative">hate’s known injury</apprsl>.</s> </p> <p> <s> <apprsl apprecia-tion="negative">Lascivious</apprsl>grace,in whom <apprsl apprecia-tion="negative">all ill</apprsl> wellshows, </s> <s> Kill me with<apprsl af-fect="negative">spites</apprsl>yet <ap-prsl judgement="positive">we must not befoes</apprsl>. </s> </p> </text>

In the choice of which and how many itemsto annotate, we adopted the following linguisticcriteria to enhance the notational analysis.

• Semantic criteria:Anytime one or more verb/noun modifiers arefound, when they do not represent meaning-ful evaluation by themselves, they are anno-tated together with the part of speech thatthey contribute to modify. Any instance ofevaluation of a multiword expression, is an-notated as a single appraisal unit. Any in-stance of evaluation of rhetorical or figurativelanguage, is annotated as a single appraisalunit. When possible, the evaluations are em-bedded so as to include appraisal units into abigger evaluative unit, in order to fully cap-ture figures of speech such as oxymora, apa-goges, rhetorical questions, interjections andthe like.

• Syntactic Criteria:Without exceeding the length of the propo-sition, it is allowed to annotate phrases assingle appraisal unit up until a clause-level,whenever they express opinions or evalua-tions. Additionally, for those cases wherecomplex phrasal structures were found, welimited ourselves to the annotation of themost evaluative part within the overall se-quence, so as to avoid overproduction oflong annotation. Again, when possible,the clauses have been de-structured so thatthrough embedding we were able to capturethe evaluation on a clause-level in greater de-tail. It is allowed to annotate evaluative se-quences on a clause level even beyond thepunctuation marks limits. However, these an-notations are very rare. In case of dyad/triadof items, whenever they share the same at-tribute and the same polarity orientation, theyare annotated as single evaluative units. Incase of more than three items in a row thatshare the same attribute and the same polarityorientation, they were annotated separately.

As to interpretation criteria, we assumed thatsonnets with the highest contrast could belong tothe category of Sarcasm. The reason for this isjustified by the fact that a high level of NegativeJudgements accompanied by Positive Apprecia-tions or Affect is by itself interpretable as the in-tention to provoke a sarcastic mood. As a final

result, there are 44 sonnets that present the highestcontrast and are specifically classified accordingto the six classes above (see Figure 1 in the Ap-pendix). There is also a group that contains am-biguous sonnets which have been classified witha double class, mainly by Irony and Sarcasm. Asa first remark, in all these sonnets, negative polar-ity is higher than positive polarity with the excep-tion of sonnet 106. In other words, if we considerthis annotation as the one containing the highestlevels of Judgement, we come to the conclusionthat possible Sarcasm reading is mostly associatedwith presence of Judgement Negative and in gen-eral with high Negative polarity annotations (seetable 2 below). As a first result, we may noticea very high convergence existing between critics’opinions as classified by us with the label highestcontrast and the output of manual annotation byAppraisal classes.

Table 2: Quantitative data for six appraisal classesfor sonnets with highest contrast

Classes Sum Mean St.Dev.Appr.Pos 56 2.534 8.199Appr.Neg 25 1.134 3.691Affct.Pos 53 2.4 7.733Affct.Neg 77 3.467 11.202Judgm.Pos 32 1.445 4.721Judgm.Neg 122 5.467 17.611

In the group of 50 sonnets classified, mainly orexclusively, with Irony, the presence of JudgementNegative is much lower than in the previous ta-ble for Sarcasm (see Figure 2 in the Appendix).In fact only half of them – 25 – has annotationfor that class, the remaining half introduces twoother negative classes: mainly Affect Negative,but also Appreciation Negative - see table 3 be-low. As to the main Positive class, we can see thatit is no longer Judgement Positive, but Apprecia-tion Positive which is present in 33 sonnets. Thisis followed by Affect Positive which is better dis-tributed.

In other words we can now consider that Sar-casm is characterized by a majority of negativeevaluations 224 over 141; while Irony is charac-terized by a majority of Positive evaluations 262over 183 and that the values are sparse and un-equally distributed. The final table concerns thenumber of sonnets with blank evaluation by criticswhich amount to 60. As a rule, this group of son-

Table 3: Quantitative data for six appraisal classesfor sonnets with lowest contrast

Classes Sum Mean St.Dev.Appr.Pos 139 5.346 18.821Appr.Neg 65 2.5 8.844Affct.Pos 64 2.462 8.708Affct.Neg 81 3.115 11.009Judgm.Pos 59 2.269 8.029Judgm.Neg 37 1.423 5.047

Table 4: Quantitative data for six appraisal classesfor sonnets with no contrast

Classes Sum Mean St.Dev.Appr.Pos 88 3.034 1.269Appr.Neg 59 2.034 7.638Affct.Pos 89 3.069 11.483Affct.Neg 109 3.759 14.052Judgm.Pos 49 1.689 6.367Judgm.Neg 8 0.276 1.079

nets look different from the two groups we alreadyanalysed. The prevailing trait is Affect Negative;Judgement Negative is only occasionally present;the second preminent trait is Affect Positive. Inorder to know how much the difference is, we canjudge from the quantities shown in table 3 above(but see also Figure 3 in the Appendix).

In particular, in this case the ratio Nega-tive/Positive is more balanced 226 over 176 with amajority of Positive annotations as happened withIrony but with a lower gap. The appraisal categorywith highest number of annotations is now Affect,whereas in the case of Irony it was Appreciation,and in Sarcasm it was Judgement. So eventuallywe have been able to differentiate the three mainand more frequent pragmatic categories by meansof Appraisal Framework features: they are char-acterized by a different distribution of positive vs.negative evaluations and also by a prominent pres-ence of one of the three main subcategories intowhich Appraisal has been subdivided that is Ap-preciation for Irony, Judgement for Sarcasm andAffect where no evaluation has been expressed.

3 Conclusion

In this paper we have presented work carried out toannotate and experiment with the theme of irony inShakespeare’s Sonnets. The gold standard for the

experiment has been created by collecting com-ments produced by literary critics on the presenceof some kind of thematic, semantic and syntac-tic opposition in the sonnets as to produce somesort of irony. At first the sonnets have been an-notated using the framework of Appraisal Theoryand then we checked the results: we obtained avery high level of matching with the critics’ opin-ions at 80%. Eventually, Appraisal framework hasshown its ability to classify and diversify differentlevels of irony effectively.

ReferencesSalvatore Attardo. 1994. Linguistic Theories of Hu-

mor. Mouton de Gruyter, Berlin – New York.

Salvatore Attardo. 2000. Irony as relevant inappropri-ateness. Journal of Pragmatics, 84(32).

Dario Calimani. 2009. William Shakespeare, I sonettidella menzogna. Carrocci, Roma.

Rodolfo Delmonte and Giulia Marchesini. 2017. Asemantically-based approach to the annotation ofnarrative style. In Proceedings of the 13th Joint ISO-ACL Workshop on Interoperable Semantic Annota-tion (ISA-13), pages 14–25, Stroudsburg, PA, USA.ACL.

R.L. Eagle. 1916. New light on the enigmas of Shake-speare’s Sonnets. John Long Limited, London.

Northrop Frye. 1957. Anatomy of Criticism: Four Es-says. Princeton University Press.

Maria Antonietta Marelli. 2015. William Shakespeare,I Sonetti – con testo a fronte. Garzanti.

J. Martin and P.R. White. 2005. Language of Eval-uation, Appraisal in English. Palgrave Macmillan,London and New York.

Giorgio Melchiori. 1971. Shakespeare s Sonnets.Adriatica Editrice, Bari.

J. Read and J. Carrol. 2012. Annotating expressions ofappraisal in english. Language Resources and Eval-uation, 46:421–447.

Michael Schoenfeldt. 2010. Cambridge introduction toShakespeare’s poetry. Cambridge University Press,Cambridge.

Alessandro Serpieri. 2002. Polifonia Shakespeariana.Bulzoni, Roma.

Michele Stingo and Rodolfo Delmonte. 2016. Anno-tating satire in italian political commentaries withappraisal theory. In Natural Language Processingmeets Journalism - Proceedings of the Workshop,NLPMJ-2016, pages 74–79, Stroudsburg, PA, USA.ACL.

M. Taboada and J. Grieve. 2004. Analyzing appraisalautomatically. In Proceedings of the AAAI SpringSymposium on Exploring Attitude and Affect in Text:Theories and Applications, pages 158–161. AAAIPress.

David K. Weiser. 1983.http://www.jstor.org/stable/43343552 Shake-spearean irony: The ’sonnets’. NeuphilologischeMitteilungen, 84(4):456–469.

David K. Weiser. 1987. Mind in Character – Shake-speare s Speaker in the Sonnets. The University ofMissouri Press.

APPENDIX.Figures Of the Six Pragmatic Categories for Appraisal-Based Classification

Figure 1: Subdivision into six appraisal classes for sonnets with highest contrast

Figure 2: Subdivision into six appraisal classes for sonnets with lowest contrast

Figure 3: Subdivision into six appraisal classes for sonnets with no contrast