1

RTE2 2006April 10, 2006

An approach based on Logic Forms and WordNet relationships to Textual Entailment performance

O. Ferrández, R. M. Terol, R. Muñoz, P. Martínez-Barco and M. Palomar

{ofe,rafamt,rafael,patricio,mpalomar}@dlsi.ua.es

GPLSI – Natural Language Processing and Information Systems Group

2

Index

• System Architecture• Derivation of the Logic Forms• Computation of Similarity Measures

between Logic Forms

• Result Analysis

• Conclusions and Future Work

3

Text

Derivation of theLogic Forms

System Architecture

Hypothesis

LF HypothesisLF Text

Computation of similarity measures between Logic Forms

Entailment?

YES NO

score

4

Derivation of the Logic Forms

• The Logic Forms are derived through an analysis of dependency relationships between the words of the sentence

• Employs a set of rules that infer several aspects such as the assert, its type, its identifier and the relationships between the different asserts in the logic form

• The Logic Forms are based on the logic form format defined in the eXtended WordNet

5

• As an example“A shark attacked a human being”

Dependency tree

Logic Formshark:NN(x1) attack:VB(e1,x1,x3)

human:NN(x2) NNC(x3,x2,x4) being:NN(x4)

a:Det

shark:N

attack:V

human:U

being:N

S

detLex-mod

obj

det

a:Det

Derivation of the Logic Forms

6

Computation of Similarity Measures between LF

• The method• Focused on the entailment between the verbs

(verbs generally govern the meaning of sentences)

• Firstly analyses the relation between the verbs of the two logic forms derived from the text and the hypothesis

• Secondly, if there is a relation between the verbs, then the method will analyse the similarity relations between all predicates which depending on the two verbs

7

simWeight = 0Tvb = obtainVerbs(T)Hvb = obtainVerbs(H)for i = 0 ... size(Tvb) do

for j = 0 ... size(Hvb) doif calcSim(Tvb(i),Hvb(j)) > 0 then

simWeight += calcSim(Tvb(i),Hvb(j))Telem = obtainElem(Tvb(i))Helem = obtainElem(Hvb(j))simWeight +=

calcSim(Telem,Helem)end if

end forend forif simWeight > threshold then return TRUEelse return FALSEend if

Computation of Similarity Measures between LF

8

• In order to obtain the similarity between the predicates of the logic forms (calcSim(x,y)), two approaches have been implemented• Based on WordNet relations• Based on Lin’s measure

• A Word Sense Disambiguation module was not employed• The first 50% of the WordNet senses were

taken into account

• The threshold, which determines if the text entails the hypothesis, has been obtained empirically using the development data

Computation of Similarity Measures between LF

9

• Based on WordNet relations• For which concept (word#sense):

• Obtaining the relations among other concepts through the synsets

• Each relation has an associated weight• Synonymy (0.9), Hypernymy (0.8),

Hyponymy and Entailment (0.7), Meronymy and Holonymy (0.5)

• The length of the path that relates the two different concepts must be lower or equal than 4 synsets

Computation of Similarity Measures between LF

10

• Based on WordNet relations• The weight of the path between two different

concepts is calculated as the product of the weights associated to the relations connecting the intermediate synsets

• This weight indicates the relation between two concepts

Computation of Similarity Measures between LF

11

• Based on WordNet relations

• Example

cable_car#n#3 subway#n#3 0.5*0.8*0.7=0.28

cable_car#n#1 railway#n#3

funicular#n#3 subway#n#3

holonymy

Hypernymy Hyponymy0.5

0.8

0.7

Computation of Similarity Measures between LF

12

• Based on Lin’s measure• Lin’s similarity measure as implemented in

WordNet::Similarity• Open source software package

• Lin’s similarity measure augments the information content of the least common subsumer (LCS is the most specific concept that two concepts share as an ancestor) of the two concepts with the sum of the information content of the concepts

Computation of Similarity Measures between LF

13

• T: Five US soldiers were killed in the capital and insurgents blasted polling stations across the country

kill:VB

NNC

five:NNNNC

us:NN soldier:NN

Øin:IN

capital:NN

and:CC

blast:VB

insurgent:NN NNC

polling:U ...

An example

14

• H: Five US soldiers were killed

kill:VB

NNC

five:NN

NNC

us:NN soldier:NN

Ø

An example

15

kill#v kill#v 1

Verbs of H kill Verbs of T kill, blast

kill

NNC

fiveNNC

us soldier

Øin

capital

and

blast

insurgents NNC

polling ...

kill

NNC

fiveNNC

us soldier

Ø

Relation ~ 1

kill#v blast#v 0,34 ...

Accumulating weights > Threshold ENTAILMENT

An example

16

• Results (RTE2 dev&test)

• Run1 (Lin’s measure) • Development data

• Accuracy: 0.5462• Test data

• Accuracy: 0.5563• Average Precision: 0.6089

• Run2 (WN relations) • Development data

• Accuracy: 0.5273• Test data

• Accuracy: 0.5475• Average Precision: 0.5743

Result Analysis

17

• The empirical threshold of the development data

• A value of 0.24

Result Analysis

18

Conclusions and Future Work

• Our system derives the logic forms for the text/hypothesis pair and computes the similarity between them

• The similarity is computed using two different approaches:• Lin’s similarity measure• WordNet relation-based similarity

• Our system provides a score showing the semantic similarity between two logic forms

19

Conclusions and Future Work

• The run using Lin’s similarity measure achieves better results than the approach based onWordNet relations, both when tested on development, as well as test data

• This slight loss of accuracy is due to the fact that our WordNet relations approach attempts to establish an objective semantic comparison between the logic forms rather than an entailment relation

20

Conclusions and Future Work

• As a future work:

• Performing a deeper study about the most suitable WordNet relations for recognising textual entailment. Perhaps only hypernymy, synonymy and entailment relations between the text and the hypothesis would be more suitable for the entailment phenomenon

• Testing how other natural language processing tools can help in detecting textual entailment. For example, using a Named Entity Recognizer could help in detecting entailment between two segment of text

21

RTE2 2006April 10, 2006

Thank you very much

O. Ferrández, R. M. Terol, R. Muñoz, P. Martínez-Barco and M. Palomar

{ofe,rafamt,rafael,patricio,mpalomar}@dlsi.ua.es

GPLSI – Natural Language Processing and Information Systems Group