Natural Language Processing of Argumentation. Adam Wyner. Summer School Lecture 2014

Argument Extrac.on from Social Media Using GATE

Adam Wyner Compu.ng Science, University of Aberdeen Summer School on Argumenta.on: Computa.onal and Linguis.c Perspec.ves University of Dundee Sept. 7, 2014

Goals

Argumenta.on Summer School, Dundee A. Wyner, Univ of Aberdeen 07/09/2014

•  Iden.fy materials (social media) and generic issues. •  Outline linguis.c issues. •  Outline GATE methodology. •  Provide some examples.

2

Materials

Argumenta.on Summer School, Dundee A. Wyner, Univ of Aberdeen 07/09/2014 3

Where Arguments Appear


•  Consumer websites: Amazon, eBay,... •  Law: policy making, Supreme Court transcripts, case based

reasoning, regula.ons. •  BBC's Have Your Say and Moral Maze. •  Medical diagnosis. •  Current events. •  Making plans. •  Debatepedia, Wikipedia, mee.ng annota.ons, web-‐forums,... •  Social media: Facebook, da.ng

Current Events


•  ScoZsh Independence and Currency •  h[p://www.bbc.co.uk/news/uk-‐scotland-‐scotland-‐

poli.cs-‐2622589

5

ScoZsh Independence 2014


The issue of what currency an independent Scotland would use has become the key ba[leground of the referendum debate. The ScoZsh government is in favour of a sterling zone, saying it would be in the interests of both Scotland and the UK to con.nue to formally share the pound if the former votes for independence, ensuring stability for both states. UK chancellor George Osborne has said the UK would not enter into a currency union with Scotland if it voted 'Yes' in September's referendum, claiming such a union would be against the economic interests of England, Wales and Northern Ireland. Mr Osborne's statement was the UK government's strongest interven.on in the debate yet, and his posi.on was supported by both Labour and the Liberal Democrats. First Minister Alex Salmond countered Mr Osborne's claims in a speech to pro-‐independence business leaders in Aberdeen on Monday, which he said had "deconstructed" the case against a currency union. So what are Mr Osborne's key arguments and how has Mr Salmond sought to counter them? Claim: Trade with Scotland is important to the UK, but the overall propor;on is small George Osborne: "I'm the first to say that our deeply integrated businesses and their suppliers are compelling reasons for keeping the UK together -‐ 70% of ScoZsh trade is with the rest of the UK. That is a massive propor.on. "And trade with Scotland is important to the rest of the UK -‐ but at only 10% of the total trade, it is a much smaller propor.on. These trade figures don't make the unanswerable case for a shared currency that the ScoZsh government assume." Alex Salmond: "I am publishing an es.mate of the transac.ons costs he would poten.ally impose on businesses in the rest of the UK. They run to many hundreds of millions of pounds. My submission is that this charge -‐ let us call it the George tax -‐ would be impossible to sell to English business. "In fact if you remove oil and gas from the equa.on, Scotland is one of the very few countries in the world with which England has a balance of trade surplus."

6

Arguments in debategraph.org


Current Method -‐ read text -‐ manually analyse -‐ manually enter text into tool -‐ manually annotate.

Problems -‐ slow, costly, error-‐prone, ad hoc, must search for 'place' of new addi.ons, etc....

Consumer Comments on Amazon


Pro and Con


Comments on Comments


Policy Consulta.ons -‐ LIBER on Copyright


-‐ Ques;on 9. Should the law be clarified with respect to whether the scanning of works held in libraries for the purpose of making their content searchable on the Internet goes beyond the scope of current excep;ons to copyright? -‐ Yes. -‐ Not all the material digi.sed by publishers is scanned with OCR (Op.cal Character Recogni.on) with the purpose of making the resul.ng content searchable. If the rights holders will not do this, libraries should be able to offer this service. It would have a transforma.ve effect on research, learning and teaching by opening up a mass of content to users which can be searched using search engines. The interests of copyright holders will not be harmed, because the resul.ng output will act as marke.ng material for their materials.

What Needs to be Done?


•  Annotate textual passages for argument relevant por.ons (premise, claim)

•  Annotate rela.ons amongst passages (premise of what argument)

•  Represent in some machine readable form.

•  Thought experiments to objec7fy and abstract the issues.

12

Generically What Needs to be Done?


What Needs to be Done? Basic


andalka nadlka fa adlkaa la lkd alkdj a a akj dal;k fda ada. a;lkd a andalkda anda;k a jad ie ae. a;lkd. nainea ; alkei nai lalin oa nekn. ake anoiiena dk aieane0-‐a an a;kl aeu ajena. ;oi anoi alkd ao;na oen oiana oin. l ;kja dka j ajda djflka kle ak kad la ien ae n en. lkj ad ad fa ;adja dfakd.

Source Text A

a1.p.1. -‐ andalka nadlka fa adlkaa la lkd alkdj a a akj dal;k fda ada. a1.p.2. -‐ a;lkd a andalkda anda;k a jad ie ae. a;lkd. a1.c. -‐ nainea ; alkei nai lalin oa nekn. a2.p.1 -‐ ake anoiiena dk aieane0-‐a an a;kl aeu ajena. a2.p.1 -‐ ;oi anoi alkd ao;na oen oiana oin. a2.e.3 -‐ l ;kja dka j ajda djflka kle ak kad la ien ae n en. a1.c -‐ lkj ad ad fa ;adja dfakd.

Annotated Text A Key: premise, excep.on, claim

What Needs to be Done? Ques.ons


•  How do we know (as readers) which is a premise, which is a claim, and which is an excep.on? –  explicit linguis.c markers (e.g. assuming X, therefore Y) –  order of sentences? –  other, e.g. context?

•  If we scrambled the order of the sentences, could we recons.tute the argument annota.on? –  Engineer – "Doesn't happen, not relevant. Build for par.culars." –  Scien.st – "Does it happen? If it does or could, how do we address it?

Explore for principles."

15

What Needs to be Done? Scramble


andalka nadlka fa adlkaa la lkd alkdj a a akj dal;k fda ada. nainea ; alkei nai lalin oa nekn. a;lkd a andalkda anda;k a jad ie ae. a;lkd. ;oi anoi alkd ao;na oen oiana oin. lkj ad ad fa ;adja dfakd. l ;kja dka j ajda djflka kle ak kad la ien ae n en. ake anoiiena dk aieane0-‐a an a;kl aeu ajena.

Source Text A

a1.p.1. -‐ andalka nadlka fa adlkaa la lkd alkdj a a akj dal;k fda ada. a1.p.2. -‐ a;lkd a andalkda anda;k a jad ie ae. a;lkd. a1.c. -‐ nainea ; alkei nai lalin oa nekn. a2.p.1 -‐ ake anoiiena dk aieane0-‐a an a;kl aeu ajena. a2.p.1 -‐ ;oi anoi alkd ao;na oen oiana oin. a2.e.3 -‐ l ;kja dka j ajda djflka kle ak kad la ien ae n en. a1.c -‐ lkj ad ad fa ;adja dfakd.

Annotated Text A Key: premise, excep.on, claim

Scramble in Comment Update


andalka nadlka fa adlkaa la lkd alkdj a a akj dal;k fda ada. a;lkd a andalkda anda;k a jad ie ae. a;lkd. nainea ; alkei nai lalin oa nekn.

Source Text A

a1.p.1. -‐ andalka nadlka fa adlkaa la lkd alkdj a a akj dal;k fda ada. a1.p.2. -‐ a;lkd a andalkda anda;k a jad ie ae. a;lkd. a1.p.3. -‐ n=jja nmae a;kda nIanl. a1.e.2. -‐ nada dnana a kkkd andai ;a. a1.c. -‐ nainea ; alkei nai lalin oa nekn.

Annotated Text A plus

Key: premise, excep.on, claim

nada dnana a kkkd andai ;a.

Source Text B

Source Text C n=jja nmae a;kda nIanl. a;lkd a andalkda likalaka anda;k a jad ie ae. a;lkd. (contrary to a1.p.2) Source Text D

Scrambling Ques.ons


•  How do we know when two premises are/are not the same? •  How do we know what argument to a[ach a proposi.on to?

•  Addressing these ques.ons may require some deep syntac.c and seman.c analysis (hint, I think it does and can be done....eventually).

•  BUT VERY HARD!!

•  Find a less demanding, near term approach towards similar objec.ves.

18

Generic Issues


•  Reconstruc.on of arguments from textual sources: –  extrac.on for argument evalua.on. –  Discon.nuity of arguments in textual source. –  Knowledge base construc.on and dynamics.

•  Linguis.c issues: –  Domain terminology. –  Linguis.c informa.on and variety (many-‐to-‐one sentence-‐

proposi.on). –  Argument rela.ons (premise, claim, excep.on, contrary). –  Sources of defeasibility (epistemic 'strength'). –  Other argument component, e.g. proposi.onal aZtudes (e.g believe,

know), speech act verbs (e.g. assert, grant).

19

Argument Pipeline


Loca.ng the Problem and Engineering a Solu.on


•  The knowledge acquisi.on bo[leneck from NL to some formal representa.on.

•  Rela.onship to other parts of the argumenta7on processing pipeline.

Three Stages Graph – Structured or Instan.ated AFs


�[]qYIGOI��<hI �rjI[hQ][h�]N��gOkZI[jh

�rjI[hQ][h�]N� ][EYkhQ][h

/jId�Â��E][hjgkEj�<gOkZI[jh�<[G�<jj<EXh

�gOkZI[j<jQ][��g<ZIq]gX

/jId�Ã��QGI[jQNs�hIjh�]N�<EEIdjIG�<gOkZI[jh

/jId�Ä��QGI[jQNs�hIjh�]N�<EEIdjIG�E][EYkhQ][h

Three Stages -‐ Caminada and Wu 2011

Knowledge Acquisi.on Bo[leneck: .me, labour, exper.se to construct a KB at scale.

Logic-‐based Instan.ated Argumenta.on Besnard and Hunter


•  An argument is an ordered pair <ψ, α>; ψ is a subset of a given KB and α is an atomic proposi.on from the KB; ψ is a minimal set of formulae such that ψ implies α, and ψ does not imply a contradic.on. ψ is said to support the claim α.

•  Where p and q are atoms, and where the KB is comprised of p and p→q, then <{p, p→q}, q> is an argument.

•  We could have a KB from which we can form an argument which supports ¬q, <{p, p→¬q}, ¬q>. In addi.on and with respect to this argument, suppose we can form an undercu>er <{r, r→¬p}, ¬p> and a rebu>al <{r, r→¬p, ¬p→q}, q}>.

•  KBs (even rela.vely small ones) generate lots of arguments and a[ack rela.onships which can be structured in a tree.

Abstract Argumenta.on


Preferred extension: {a, c, d, h, i, k}

Zeroing In


Source text Knowledge base & argumenta.on

schemes

Generated arguments (abstract or instan.ated).

25

Context with Respect to Analysis and Argumenta.on Schemes


Current Tools to Extract and Structure Arguments from Text


•  Ra.onale, Araucaria, Carneades (Gordon 2007), IMPACT Project, Legal Appren.ce, Argument Wall,....

•  Pale[e of annota.ons and templates. •  All manual. No NLP.

Argumenta.on Schemes


•  Pa[erns of presump.ve (defeasible) reasoning (Walton 1996) •  Prac.cal Reasoning with values:

–  Do ac.on (transi.on) because: •  Current circumstances -‐ a list of literals. •  Consequences – a list of literals. •  Values (promoted, demoted, neutral wrt ac.ons) – a list of terms.

•  Credible Source: –  Z is accepted because:

•  X is an expert in domain Y. •  X stated literal Z •  Z is about domain Y.

28

Overall Proposal


•  Normalise natural language source material into argumenta.on schemes.

•  Formalise argumenta.on schemes in terms of roles of proposi.ons in the scheme and internal structure of proposi.ons (predicates and typed variables).

•  Connect argumenta.on schemes to abstract arguments. •  Relate one scheme to another in terms of contrariness. •  Extract scheme relevant informa.on from the source. •  Create a knowledge base to instan.ate variables.

29

Caveat


•  Low level automa.on, using high level structures as guides. •  For example, no automa.c search for scheme filling,

grounding of variables, contrast iden.fica.on. •  Progress can be made on these (and for contrast

iden.fica.on, there is significant work already).

30

Normalise for Argumenta.on Schemes


Annotate – Query – Extract


•  Annotate with respect to Argumenta.on Schemes. –  characteris.c terminology of the scheme. –  generalise the terminology to cover varia.on. –  dis.nguish "domain" from "generic" terminology.

•  Complex, flexible queries over the annota.ons. –  Low level (atomic) and high level (molecular) construc.ons.

–  Interac.ve, semi-‐automa.c. •  Export to some machine readable format -‐ XML.

Language Issues


Problems with Language I

•  Iden.fica.on, implicit informa.on, mul.ple forms with the same meaning, the same form with mul.ple meanings:

•  En.ty ID: Jane Smith, for plain.ff.

•  Rela.on ID: Edgar Wilson disclosed the formula to Mary Hays.

•  Bill drove the car into Phil at 60 MPH. (agent, instrument, killing)

•  Jane Smith, Jane R. Smith, Smith, A[orney Smith....

•  Jane Smith in one case decision need not be the same Jane Smith in another case decision.


Problems with Language II

•  Concepts, dispersed meanings, rules, diathesis:

•  Plain.ff, judge, a[orney.

•  Jane Smith represented Jones Inc. She is a partner at Dewey, Chetum, and Howe. To contact her, write to [email protected].

•  If a woman is over 62 years old and lives in the UK, she is a pensioner.

•  Diathesis: alterna.ve sentence forms with (almost) synonymous meaning: Bill pushed Jill; Jill was pushed by Bill.


Problems with Language III

•  Ambiguity, vagueness, underspecifica.on:

•  The man saw the woman with binoculars.

•  It is illegal to leave a heap of shoes on the sidewalk.

•  Vehicles may not be driven in the park.

•  Sarcasm, irony.

•  Interpreta.on.

•  Context dependence, subjec.vity, arbitrary meaning, when I was at school, I know language....


Problems with Language IV

•  Complexity, length, and layout (see our Camera example).

•  Intersenten.al connec.ons:

•  Bill le� the house. He drove home.

•  Bill le� the house. He didn't feel comfortable there.

•  Bill le� the house. It was an old house, once owned by a wealthy merchant.

•  Synonymy, antonyms, meronyms (finger part of hand), etc.

•  Repe..on.


Problems for Annota.on

•  Annotate large legacy corpora.

•  Address growth of corpora.

•  Reduce number of human annotators and tedious work.

•  Make annota.on systema.c, automa.c, and consistent.

•  Annotate fine-‐grained informa.on:

•  Names, loca.ons, addresses, web links, organisa.ons, ac.ons, argument structures, rela.ons between en..es.

•  Map from well-‐dra�ed documents in NL to RDF/OWL/XML.


Addressing the Problems

•  Decompose big problems down to smaller problems.

•  Modularise problems.

•  Address the smaller, modular problems.

•  Compose solu.ons from parts.

•  Iden.fy (set aside, address, assign to someone else) remaining and/or highly problema.c issues.


Methodology


Approaches

•  Knowledge light in terms of knowledge of the domain or of language – sta.s.cal or machine learning approaches. Algorithmically compare and contrast large bodies of textual data, iden.fying regulari.es and similari.es. Sparse data problem. Need a ‘gold standard’. No rules extracted. Opaque. Hard to modify.

•  Knowledge heavy in terms of lists, rules, and processes. Labour and knowledge intensive. Creates gold standards. Transparent. Can jus.fy outcomes. Can 'correct' solu.ons.

•  Can do either. Where textual traceability (jus.fica.on) is essen.al, knowledge heavy is important.


Overall Approach

•  Decompose large complex problems into smaller, manageable problems for which we can create solu.ons.

•  So�ware engineering approach.

•  Papers by Wyner and Peters (2010, 2011).


Development Caveat

•  Developing working prototypes (much less public and/or commercial tools) takes resources.

•  Tool development

•  Corpus development

•  Language analysis

•  It is a slow, painstaking, and gradual process of construc.ng modules to do the small tasks you need to build the large applica.ons you want.

•  Not a simple phone app.


Development Cycle

Source Text

Linguis.c Analysis

Tool Construc.on

Knowledge Extrac.on

Evalua.on


Whazza Methodology?

07/09/2014 Argumenta.on Summer School, Dundee A. Wyner, Univ of Aberdeen 45

Linguis.c Processing


Computa.onal Linguis.c Cascade I

•  Sentence segmenta.on -‐ divide text into sentences.

•  Tokenisa.on -‐ words iden.fied by spaces between them.

•  Part of speech tagging -‐ noun, verb, adjec.ve....

•  Morphological analysis -‐ singular/plural, tense, nominalisa.on, ...

•  Shallow syntac.c parsing/chunking -‐ noun phrase, verb phrase, subordinate clause, ....

•  Named en.ty recogni.on -‐ the en..es in the text.


Computa.onal Linguis.c Cascade II

•  Dependency analysis – sentence subject, subordinate clauses, pronominal anaphora,...

•  Rela.onship recogni.on – X is president of Y; A hit B with a car and killed B.

•  Enrichment -‐ add lexical seman.c informa.on to verbs or nouns.

•  Supertagging – adding conceptual annota.ons to text.

•  Transla.on to logic for reasoning.

•  Each step guided by pa[ern matching and rule applica.on.


Overall Processing Strategy

•  Make implicit informa.on explicit by adding machine readable annota7ons.


A Tool


GATE

•  General Architecture for Text Engineering (GATE) -‐ open source framework which supports plug-‐in NLP components to process a corpus of text.

•  GATE Training Courses h[ps://gate.ac.uk/

•  A GUI to work with the tools. •  A Java library to develop further applica.ons. •  Components and sequences of processes, each process

feeding the next in a “pipeline”. •  Annotated text output or other sorts of output.


GATE Benefits


•  No need for parsed, pre-‐structured text. •  Generic components apply anywhere. •  No need for a gold standard. •  Low entry point, no programming required. •  Useful interface for analysis and demonstra.on. •  Lots of public resources and open to build more add-‐ons. •  Connects to other tools, widely used....

GATE Basic Process Flow


Can add further processing components to pipeline, e.g. NER, co-‐reference, other other annota.ons,...

GATE -‐ Gaze[eers

•  Gaze[eers are lookup lists that add features -‐ when a string in the text is located in a lookup list, annotate the string in the text with the feature. Conceptual covers.

•  Feature: list of items...

•  Obliga.on: ought, must, obliged, obliga.on.... •  Excep.on: unless, except, but, apart from.... •  Verbs according to thema.c roles: lists of verbs and

their associated roles, e.g. run has an agent (Bill ran), rise has a theme (The wind blew).


GATE – JAPE Rules

•  JAPE Rules (finite state transduc.on rules) create overt annota.ons and reuse other annota.ons (e.g. Parser Output):


GATE – Building an Applica.on

•  Have Gaze[eer lists and JAPE rules for:

•  lists in various forms; •  excep.on phrases in various forms; •  condi.onals in various forms; •  deon.c terms; •  associa.ng gramma.cal roles (e.g. subject and object)

with thema.c roles (agent and theme) in various forms.


Example -‐ Camera


Argument Fragment for a Camera


Pro and Con


Comments on Comments


Goals

•  Extract arguments distributed across a corpora and evaluate them with formal, automated tools.

•  Speed the work of human analysts. •  Provide semi-‐automa3c support. •  Use aspects of NLP to incrementally address a range of

problems (ambiguity, structure, contrasts,....)

•  Wyner, Schneider, Atkinson, and Bench-‐Capon (2012).


Consumer Argumenta.on Scheme

Variables in schemes as targets for extrac7on. Premises: •  Camera X has property P. •  Property P promotes value V for agent A. Conclusion: •  Agent A should Ac;on Camera X.


Analyst’s Goal: Instan.ate

Premises: •  The Canon SX220 has good video quality. •  Good video quality promotes image quality for casual

photographers. Conclusion: •  Casual photographers should buy the Canon SX220.


Annota.ng Text

•  Annotate text: –  Simple or complex annota.ons. –  Highlight annota.ons with –  Search for and extract text by annota.on.

•  GATE “General Architecture for Text Engineering”. – Works with large corpora of text. –  Rule-‐based or machine-‐learning approaches.


To Find Argument Passages •  Use:

–  Indicators of aJer, as, because, for, since, when, .... –  Indicators of therefore, in conclusion, consequently, ....


Rhetorical Terminology


To Find What is Being Discussed

•  Use : –  Has a flash –  Number of megapixels –  Scope of the zoom –  Lens size –  The warranty


Domain Terminology


To Find A[acks Between Arguments

•  Use contrast terminology: –  Indicators but, except, not, never, no, .... –  Contras.ng sen.ment The flash worked . The flash worked .


Sen.ment Terminology


, ,


Query for Pa[erns


An Argument for Buying the Camera

Premises: The pictures are perfectly exposed. The pictures are well-‐focused. No camera shake. Good video quality. Each of these proper.es promotes image quality.

Conclusion: (You, the reader,) should buy the CanonSX220.


An Argument for NOT Buying the Camera

Premises: The colour is poor when using the flash. The images are not crisp when using the flash. The flash causes a shadow. Each of these proper.es demotes image quality.

!Conclusion: (You, the reader,) should not buy the CanonSX220.


Counterarguments to the Premises of “Don’t buy”

The colour is poor when using the flash. For good colour, use the colour seZng, not the flash.

The images are not crisp when using the flash. No need to use flash even in low light.

The flash causes a shadow. There is a correc.ve video about the flash shadow.


In More Detail


ANNIC Movie


Example -‐ Rules

•  Rule iden.fica.on in regula.ons; what one can 'argue' for and against.

•  Using previous modules.

•  Wyner and Peters (2011)


Sample Outputs

Consequence, list structure, and conjuncts of the antecedent.

Excep.on, agent NP, deon.c concept, ac.ve main verb, theme. 07/09/2014 Argumenta.on Summer School, Dundee

A. Wyner, Univ of Aberdeen 92

Sample Output

Theme, deon.c modal, passive verb, agent with complex rela.ve clause.


Sample Output -‐ Overall


Sample Output -‐ XML


This is an inline representa.on, and not 'pure' XML as tags can overlap. There is also offset, which can be modified easily.

Sample Output – ANNIC Search


Gold Standards


Teamware to Create Gold Standards


Results of Annota.on

•  The annotators carry out their task and complete the project.

•  Carry out inter-‐annotator agreement analysis.

•  Curate the disagreements to create a Gold Standard corpus. Can use this for machine learning.

•  Search the annota.ons using an online tool, e.g. ANNIC.


Addi.ons


Add to Explorer (or Teamware)

•  Verbs for proposi.onal aZtudes, e.g. believe, know, hope and speech acts, e.g. stated, men7oned, guessed.

•  Opinion adverbials -‐ obviously, so far as I know, scien7fically. •  Ques.on words and markers – who, why, ? •  Rhetorical connec.ves -‐ elabora7on, example, contrast. •  Others....


References


•  Wyner, van Engers, Hunter (2010) •  Wyner and Peters (2010, 2011) •  Wyner, Schneider, Atkinson, and Bench-‐Capon (2012)

102

Thanks for your a[en.on!

•  Questions? •  Contacts:

–  Adam Wyner [email protected]


Natural Language Processing of Argumentation. Adam Wyner. Summer School Lecture 2014

Science

Transcript of Natural Language Processing of Argumentation. Adam Wyner. Summer School Lecture 2014