VINCENZO TOMBOLINI CATTEDRA DI RADIOTERAPIA Università Sapienza di Roma.
Linguistic Computing Laboratory Dipartimento di Informatica Universit à di Roma “La Sapienza”
34
© Johan Bos April 2006 When logical inference helps in determining textual entailment (and when it doesn’t) Johan Bos & Katja Markert Linguistic Computing Laboratory Dipartimento di Informatica Università di Roma “La Sapienza” Natural Language Processing Group Computer Science Department University of Leeds
description
When logical inference helps in determining textual entailment ( and when it doesn’t) Johan Bos & Katja Markert. Linguistic Computing Laboratory Dipartimento di Informatica Universit à di Roma “La Sapienza”. Natural Language Processing Group Computer Science Department Universit y of Leeds. - PowerPoint PPT Presentation
Transcript of Linguistic Computing Laboratory Dipartimento di Informatica Universit à di Roma “La Sapienza”
© J
oh
an
Bos
Ap
ril 2
00
6
When logical inference helps in determining textual entailment
(and when it doesn’t)
Johan Bos & Katja Markert
Linguistic Computing LaboratoryDipartimento di Informatica
Università di Roma “La Sapienza”
Natural Language Processing GroupComputer Science Department
University of Leeds
© J
oh
an
Bos
Ap
ril 2
00
6
Aristotle’s Syllogisms
All men are mortal.
Socrates is a man.
-------------------------------
Socrates is mortal.
ARISTOTLE 1 (TRUE)
© J
oh
an
Bos
Ap
ril 2
00
6
Talk Outline
• Hybrid system combining:– Shallow semantic approach – Deep semantic approach
• Machine Learning– Features of both approaches are
combined in one classifier
© J
oh
an
Bos
Ap
ril 2
00
6
Shallow Semantic Analysis
• Primarily based on word overlap• Using weighted lemmas• Weights correspond to inverse doc. freq.
– Web as corpus– Wordnet for synonyms
• Additional features– Number of words in T and H– Type of dataset
© J
oh
an
Bos
Ap
ril 2
00
6
Deep Semantic Analysis
• Compositional Semantics– How to build semantic representations
for the text and hypothesis– Do this in a systematic way
• Logical Inference– FOL theorem proving– FOL model building
© J
oh
an
Bos
Ap
ril 2
00
6
Compositional Semantics
• The ProblemGiven a natural language expression, how do we convert it into a logical formula?
• Frege’s principleThe meaning of a compound expression is a function of the meaning of its parts.
© J
oh
an
Bos
Ap
ril 2
00
6
Compositional Semantics
• We need a theory of syntax, to determine the parts of a natural language expression
• We will use CCG
• We need a theory of semantics, to determine the meaning of the parts
• We will use DRT
• We need a technique to combine the parts• We will use the Lambda-calculus
© J
oh
an
Bos
Ap
ril 2
00
6
Combinatorial Categorial Grammar
• CCG is a lexicalised theory of grammar (Steedman 2001)
• Deals with complex cases of coordination and long-distance dependencies
• Lexicalised, hence easy to implement– English wide-coverage grammar– Fast robust parser available
© J
oh
an
Bos
Ap
ril 2
00
6
Discourse Representation Theory
• Well understood semantic formalism– Scope, anaphora, presupposition, tense, etc. – Kamp `81, Kamp & Reyle `93, Van der Sandt `92
• Semantic representations (DRSs) can be build using traditional tools– Lambda calculus– Underspecification
• Model-theoretic interpretation – Inference possible– Translation to first-order logic
© J
oh
an
Bos
Ap
ril 2
00
6
CCG/DRT example
NP/N:a N:spokesman S\NP:lied
p. q. ;p(x);q(x) z. x. x(y. )spokesman(z)
x e
lie(e)
agent(e,y)
© J
oh
an
Bos
Ap
ril 2
00
6
CCG/DRT example
NP/N:a N:spokesman S\NP:lied
p. q. ;p(x);q(x) z. x.x(y. )
-------------------------------------------------------- (FA)
NP: a spokesman
p. q. ;p(x);q(x)(z. )
spokesman(z)
x
spokesman(z)
e
lie(e)
agent(e,y)
x
© J
oh
an
Bos
Ap
ril 2
00
6
CCG/DRT example
NP/N:a N:spokesman S\NP:lied
p. q. ;p(x);q(x) z. x.x(y. )
-------------------------------------------------------- (FA)
NP: a spokesman
q. ; ;q(x))
spokesman(z)
x
spokesman(x)
e
lie(e)
agent(e,y)
x
© J
oh
an
Bos
Ap
ril 2
00
6
CCG/DRT example
NP/N:a N:spokesman S\NP:lied
p. q. ;p(x);q(x) z. x.x(y. )
-------------------------------------------------------- (FA)
NP: a spokesman
q. ;q(x)
spokesman(z)
x
x
spokesman(x)
e
lie(e)
agent(e,y)
© J
oh
an
Bos
Ap
ril 2
00
6
CCG/DRT example
NP/N:a N:spokesman S\NP:lied
p. q. ;p(x);q(x) x. x.x(y. )
-------------------------------------------------------- (FA)
NP: a spokesman
q. ;q(x)
-------------------------------------------------------------------------------- (BA)
S: a spokesman lied
x.x(y. ) (q. ;q(x))
spokesman(z)
x
x
spokesman(x)
e
lie(e)
agent(e,y)
e
lie(e)
agent(e,y)
x
spokesman(x)
© J
oh
an
Bos
Ap
ril 2
00
6
CCG/DRT example
NP/N:a N:spokesman S\NP:lied
p. q. ;p(x);q(x) x. x.x(y. )
-------------------------------------------------------- (FA)
NP: a spokesman
q. ;q(x)
-------------------------------------------------------------------------------- (BA)
S: a spokesman lied
;
spokesman(x)
x
x
spokesman(x)
e
lie(e)
agent(e,y)
e
lie(e)
agent(e,x)
x
spokesman(x)
© J
oh
an
Bos
Ap
ril 2
00
6
CCG/DRT example
NP/N:a N:spokesman S\NP:lied
p. q. ;p(x);q(x) x. x.x(y. )
-------------------------------------------------------- (FA)
NP: a spokesman
q. ;q(x)
-------------------------------------------------------------------------------- (BA)
S: a spokesman lied
spokesman(x)
x
x
spokesman(x)
e
lie(e)
agent(e,y)
x e
spokesman(x)
lie(e)
agent(e,x)
© J
oh
an
Bos
Ap
ril 2
00
6
The Clark & Curran Parser
• Use standard statistical techniques– Robust wide-coverage parser – Clark & Curran (ACL 2004)
• Grammar derived from CCGbank– 409 different categories– Hockenmaier & Steedman (ACL 2002)
• Results: 96% coverage WSJ– Bos et al. (COLING 2004)– Example output:
© J
oh
an
Bos
Ap
ril 2
00
6
Logical Inference
• How do we perform inference with DRSs? – Translate DRS into first-order logic– Use off-the-shelf inference engines
• What kind of inference engines?– Theorem Prover:
Vampire (Riazanov & Voronkov 2002)
– Model Builder: Paradox
© J
oh
an
Bos
Ap
ril 2
00
6
Using Theorem Proving
• Given a textual entailment pair T/H:– Produce DRSs for T and H– Translate these DRSs into FOL– Give to the theorem prover: T’ H’
• If a proof is found, then T entails H
• Good results for examples with:
– apposition, relative clauses, coordination
– intersective adjectives, noun noun compounds
– passive/active alternations
© J
oh
an
Bos
Ap
ril 2
00
6
Example (Vampire: proof)
On Friday evening, a car bomb exploded
outside a Shiite mosque in Iskandariyah,
30 miles south of the capital.
-----------------------------------------------------
A bomb exploded outside a mosque.
RTE-2 112 (TRUE)
© J
oh
an
Bos
Ap
ril 2
00
6
Example (Vampire: proof)
Initially, the Bundesbank opposed the
introduction of the euro but was compelled
to accept it in light of the political pressure
of the capitalist politicians who supportedits introduction.
-----------------------------------------------------
The introduction of the euro has been opposed.
RTE-2 489 (TRUE)
© J
oh
an
Bos
Ap
ril 2
00
6
Background Knowledge
• Many examples in the RTE dataset require additional knowledge– Lexical knowledge– Linguistic Knowledge – World knowledge
• Generate Background Knowledge for T&H in first order logic
• Give this to the theorem prover: (BK & T’) H’
© J
oh
an
Bos
Ap
ril 2
00
6
Lexical Knowledge
• We use WordNet as a start to get additional knowledge
• All of WordNet is too much, so we create MiniWordNets– Based on hyponym relations– Remove redundant information– Conversion in first order logic
© J
oh
an
Bos
Ap
ril 2
00
6
Linguistic Knowledge
• Manually coded rules – Possessives– Active/passive alternation– Noun noun compound interpretation
© J
oh
an
Bos
Ap
ril 2
00
6
Linguistic & World Knowledge
• Manually coded 115 rules – Spatial knowledge– Causes of death– Winning prizes or awards– Family relations– Diseases– Producers– Employment– Ownership
© J
oh
an
Bos
Ap
ril 2
00
6
Knowledge at work
• Background Knowledge:x(soar(x)rise(x))
Crude oil prices soared to record levels.
-----------------------------------------------------
Crude oil prices rise.
RTE 1952 (TRUE)
© J
oh
an
Bos
Ap
ril 2
00
6
Troubles with theorem proving
• Theorem provers are extremely precise
• They won’t tell you when there is “almost” a proof
• Even if there is a little background knowledge missing, Vampire will say:
NO
© J
oh
an
Bos
Ap
ril 2
00
6
Vampire: no proof
RTE 1049 (TRUE)
Four Venezuelan firefighters who were traveling
to a training course in Texas were killed when their
sport utility vehicle drifted onto the shoulder of a
Highway and struck a parked truck.
----------------------------------------------------------------
Four firefighters were killed in a car accident.
© J
oh
an
Bos
Ap
ril 2
00
6
Using Model Building
• Need a robust way of inference• Use model builders
– Paradox (Claessen & Sorensson 2003)– Mace (McCune)
• Produce minimal model by iteration of domain size
• Use size of models to determine entailment– Compare size of model of T and T&H– If the difference is small, then it is likely that T
entails H
© J
oh
an
Bos
Ap
ril 2
00
6
Using Model Building
• Given a textual entailment pair T/H withtext T and hypothesis H:– Produce DRSs for T and H– Translate these DRSs into FOL– Generate Background Knowledge– Give this to the Model Builder:
i) BK & T’
ii) BK & T’ & H’
• If the models for i) and ii) are similar in size, then T entails H
© J
oh
an
Bos
Ap
ril 2
00
6
Features for Classifier
• Features from deep analysis:– proof (yes/no)– inconsistent (yes/no)– domain size, model size– domain size difference, abs and relative– model size difference, abs and relative
• Combine this with features from shallow approach
• Machine learning took WEKA
© J
oh
an
Bos
Ap
ril 2
00
6
RTE2 Results
Shallow Deep
IE 0.51 0.55
IR 0.66 0.64
QA 0.57 0.53
SUM 0.74 0.71
all 0.62 0.61
© J
oh
an
Bos
Ap
ril 2
00
6
Conclusions
• Why relatively low results?– Recall for feature proof is low– Most proofs are also found by word
overlap– Same for small domain size differences
• Not only bad news– Deep analysis more consistent across
different datasets
© J
oh
an
Bos
Ap
ril 2
00
6
Future Stuff
• Error analysis!– Difficult, dataset not focussed– Many different sources of errors– Prepare more focussed datasets for system
development?
• Use better techniques for usingnumeric features
• Improve linguistic analysis• More background knowledge!
