1
RTE2 2006April 10, 2006
An approach based on Logic Forms and WordNet relationships to Textual Entailment performance
O. Ferrández, R. M. Terol, R. Muñoz, P. Martínez-Barco and M. Palomar
{ofe,rafamt,rafael,patricio,mpalomar}@dlsi.ua.es
GPLSI – Natural Language Processing and Information Systems Group
2
Index
• System Architecture• Derivation of the Logic Forms• Computation of Similarity Measures
between Logic Forms
• Result Analysis
• Conclusions and Future Work
3
Text
Derivation of theLogic Forms
System Architecture
Hypothesis
LF HypothesisLF Text
Computation of similarity measures between Logic Forms
Entailment?
YES NO
score
4
Derivation of the Logic Forms
• The Logic Forms are derived through an analysis of dependency relationships between the words of the sentence
• Employs a set of rules that infer several aspects such as the assert, its type, its identifier and the relationships between the different asserts in the logic form
• The Logic Forms are based on the logic form format defined in the eXtended WordNet
5
• As an example“A shark attacked a human being”
Dependency tree
Logic Formshark:NN(x1) attack:VB(e1,x1,x3)
human:NN(x2) NNC(x3,x2,x4) being:NN(x4)
a:Det
shark:N
attack:V
human:U
being:N
S
detLex-mod
obj
det
a:Det
Derivation of the Logic Forms
6
Computation of Similarity Measures between LF
• The method• Focused on the entailment between the verbs
(verbs generally govern the meaning of sentences)
• Firstly analyses the relation between the verbs of the two logic forms derived from the text and the hypothesis
• Secondly, if there is a relation between the verbs, then the method will analyse the similarity relations between all predicates which depending on the two verbs
7
simWeight = 0Tvb = obtainVerbs(T)Hvb = obtainVerbs(H)for i = 0 ... size(Tvb) do
for j = 0 ... size(Hvb) doif calcSim(Tvb(i),Hvb(j)) > 0 then
simWeight += calcSim(Tvb(i),Hvb(j))Telem = obtainElem(Tvb(i))Helem = obtainElem(Hvb(j))simWeight +=
calcSim(Telem,Helem)end if
end forend forif simWeight > threshold then return TRUEelse return FALSEend if
Computation of Similarity Measures between LF
8
• In order to obtain the similarity between the predicates of the logic forms (calcSim(x,y)), two approaches have been implemented• Based on WordNet relations• Based on Lin’s measure
• A Word Sense Disambiguation module was not employed• The first 50% of the WordNet senses were
taken into account
• The threshold, which determines if the text entails the hypothesis, has been obtained empirically using the development data
Computation of Similarity Measures between LF
9
• Based on WordNet relations• For which concept (word#sense):
• Obtaining the relations among other concepts through the synsets
• Each relation has an associated weight• Synonymy (0.9), Hypernymy (0.8),
Hyponymy and Entailment (0.7), Meronymy and Holonymy (0.5)
• The length of the path that relates the two different concepts must be lower or equal than 4 synsets
Computation of Similarity Measures between LF
10
• Based on WordNet relations• The weight of the path between two different
concepts is calculated as the product of the weights associated to the relations connecting the intermediate synsets
• This weight indicates the relation between two concepts
Computation of Similarity Measures between LF
11
• Based on WordNet relations
• Example
cable_car#n#3 subway#n#3 0.5*0.8*0.7=0.28
cable_car#n#1 railway#n#3
funicular#n#3 subway#n#3
holonymy
Hypernymy Hyponymy0.5
0.8
0.7
Computation of Similarity Measures between LF
12
• Based on Lin’s measure• Lin’s similarity measure as implemented in
WordNet::Similarity• Open source software package
• Lin’s similarity measure augments the information content of the least common subsumer (LCS is the most specific concept that two concepts share as an ancestor) of the two concepts with the sum of the information content of the concepts
Computation of Similarity Measures between LF
13
• T: Five US soldiers were killed in the capital and insurgents blasted polling stations across the country
kill:VB
NNC
five:NNNNC
us:NN soldier:NN
Øin:IN
capital:NN
and:CC
blast:VB
insurgent:NN NNC
polling:U ...
An example
14
• H: Five US soldiers were killed
kill:VB
NNC
five:NN
NNC
us:NN soldier:NN
Ø
An example
15
kill#v kill#v 1
Verbs of H kill Verbs of T kill, blast
kill
NNC
fiveNNC
us soldier
Øin
capital
and
blast
insurgents NNC
polling ...
kill
NNC
fiveNNC
us soldier
Ø
Relation ~ 1
kill#v blast#v 0,34 ...
Accumulating weights > Threshold ENTAILMENT
An example
16
• Results (RTE2 dev&test)
• Run1 (Lin’s measure) • Development data
• Accuracy: 0.5462• Test data
• Accuracy: 0.5563• Average Precision: 0.6089
• Run2 (WN relations) • Development data
• Accuracy: 0.5273• Test data
• Accuracy: 0.5475• Average Precision: 0.5743
Result Analysis
17
• The empirical threshold of the development data
• A value of 0.24
Result Analysis
18
Conclusions and Future Work
• Our system derives the logic forms for the text/hypothesis pair and computes the similarity between them
• The similarity is computed using two different approaches:• Lin’s similarity measure• WordNet relation-based similarity
• Our system provides a score showing the semantic similarity between two logic forms
19
Conclusions and Future Work
• The run using Lin’s similarity measure achieves better results than the approach based onWordNet relations, both when tested on development, as well as test data
• This slight loss of accuracy is due to the fact that our WordNet relations approach attempts to establish an objective semantic comparison between the logic forms rather than an entailment relation
20
Conclusions and Future Work
• As a future work:
• Performing a deeper study about the most suitable WordNet relations for recognising textual entailment. Perhaps only hypernymy, synonymy and entailment relations between the text and the hypothesis would be more suitable for the entailment phenomenon
• Testing how other natural language processing tools can help in detecting textual entailment. For example, using a Named Entity Recognizer could help in detecting entailment between two segment of text
21
RTE2 2006April 10, 2006
Thank you very much
O. Ferrández, R. M. Terol, R. Muñoz, P. Martínez-Barco and M. Palomar
{ofe,rafamt,rafael,patricio,mpalomar}@dlsi.ua.es
GPLSI – Natural Language Processing and Information Systems Group