RTE2 2006 April 10, 2006

download RTE2 2006 April 10, 2006

of 21

  • date post

    28-Jan-2016
  • Category

    Documents

  • view

    39
  • download

    0

Embed Size (px)

description

RTE2 2006 April 10, 2006. An approach based on Logic Forms and WordNet relationships to Textual Entailment performance. O. Ferrández, R. M. Terol, R. Muñoz, P. Martínez-Barco and M. Palomar {ofe,rafamt,rafael,patricio,mpalomar}@dlsi.ua.es - PowerPoint PPT Presentation

Transcript of RTE2 2006 April 10, 2006

  • RTE2 2006April 10, 2006An approach based on Logic Forms and WordNet relationships to Textual Entailment performanceO. Ferrndez, R. M. Terol, R. Muoz, P. Martnez-Barco and M. Palomar{ofe,rafamt,rafael,patricio,mpalomar}@dlsi.ua.esGPLSI Natural Language Processing and Information Systems Group

  • IndexSystem ArchitectureDerivation of the Logic FormsComputation of Similarity Measures between Logic Forms

    Result Analysis

    Conclusions and Future Work

  • TextDerivation of theLogic FormsSystem ArchitectureHypothesisLF HypothesisLF TextComputation of similarity measures between Logic FormsEntailment?YESNOscore

  • Derivation of the Logic Forms

    The Logic Forms are derived through an analysis of dependency relationships between the words of the sentence

    Employs a set of rules that infer several aspects such as the assert, its type, its identifier and the relationships between the different asserts in the logic form

    The Logic Forms are based on the logic form format defined in the eXtended WordNet

  • As an exampleA shark attacked a human being

    Dependency tree

    Logic Formshark:NN(x1) attack:VB(e1,x1,x3) human:NN(x2) NNC(x3,x2,x4) being:NN(x4)a:Detshark:Nattack:Vhuman:Ubeing:NSdetLex-modobjdeta:DetDerivation of the Logic Forms

  • Computation of Similarity Measures between LF

    The methodFocused on the entailment between the verbs (verbs generally govern the meaning of sentences)Firstly analyses the relation between the verbs of the two logic forms derived from the text and the hypothesisSecondly, if there is a relation between the verbs, then the method will analyse the similarity relations between all predicates which depending on the two verbs

  • simWeight = 0Tvb = obtainVerbs(T)Hvb = obtainVerbs(H)for i = 0 ... size(Tvb) dofor j = 0 ... size(Hvb) doif calcSim(Tvb(i),Hvb(j)) > 0 thensimWeight += calcSim(Tvb(i),Hvb(j))Telem = obtainElem(Tvb(i))Helem = obtainElem(Hvb(j))simWeight += calcSim(Telem,Helem)end ifend forend forif simWeight > threshold then return TRUEelse return FALSEend ifComputation of Similarity Measures between LF

  • In order to obtain the similarity between the predicates of the logic forms (calcSim(x,y)), two approaches have been implementedBased on WordNet relationsBased on Lins measure

    A Word Sense Disambiguation module was not employedThe first 50% of the WordNet senses were taken into account

    The threshold, which determines if the text entails the hypothesis, has been obtained empirically using the development dataComputation of Similarity Measures between LF

  • Based on WordNet relationsFor which concept (word#sense):Obtaining the relations among other concepts through the synsetsEach relation has an associated weightSynonymy (0.9), Hypernymy (0.8), Hyponymy and Entailment (0.7), Meronymy and Holonymy (0.5)

    The length of the path that relates the two different concepts must be lower or equal than 4 synsetsComputation of Similarity Measures between LF

  • Based on WordNet relationsThe weight of the path between two different concepts is calculated as the product of the weights associated to the relations connecting the intermediate synsets

    This weight indicates the relation between two conceptsComputation of Similarity Measures between LF

  • Based on WordNet relations

    Example

    cable_car#n#3 subway#n#3 0.5*0.8*0.7=0.28

    cable_car#n#1railway#n#3funicular#n#3subway#n#3holonymyHypernymyHyponymy0.50.80.7Computation of Similarity Measures between LF

  • Based on Lins measureLins similarity measure as implemented in WordNet::SimilarityOpen source software package

    Lins similarity measure augments the information content of the least common subsumer (LCS is the most specific concept that two concepts share as an ancestor) of the two concepts with the sum of the information content of the conceptsComputation of Similarity Measures between LF

  • T: Five US soldiers were killed in the capital and insurgents blasted polling stations across the countrykill:VBNNCfive:NNNNCus:NNsoldier:NNin:INcapital:NNand:CCblast:VBinsurgent:NNNNCpolling:U...An example

  • H: Five US soldiers were killedAn example

  • kill#v kill#v 1

    Verbs of H killVerbs of T kill, blastRelation ~ 1kill#v blast#v 0,34...Accumulating weights > Threshold ENTAILMENTAn example

  • Results (RTE2 dev&test)

    Run1 (Lins measure) Development dataAccuracy: 0.5462Test dataAccuracy: 0.5563Average Precision: 0.6089

    Run2 (WN relations) Development dataAccuracy: 0.5273Test dataAccuracy: 0.5475Average Precision: 0.5743

    Result Analysis

  • The empirical threshold of the development data

    A value of 0.24Result Analysis

  • Conclusions and Future WorkOur system derives the logic forms for the text/hypothesis pair and computes the similarity between them

    The similarity is computed using two different approaches: Lins similarity measureWordNet relation-based similarity

    Our system provides a score showing the semantic similarity between two logic forms

  • Conclusions and Future WorkThe run using Lins similarity measure achieves better results than the approach based onWordNet relations, both when tested on development, as well as test data

    This slight loss of accuracy is due to the fact that our WordNet relations approach attempts to establish an objective semantic comparison between the logic forms rather than an entailment relation

  • Conclusions and Future WorkAs a future work:

    Performing a deeper study about the most suitable WordNet relations for recognising textual entailment. Perhaps only hypernymy, synonymy and entailment relations between the text and the hypothesis would be more suitable for the entailment phenomenon

    Testing how other natural language processing tools can help in detecting textual entailment. For example, using a Named Entity Recognizer could help in detecting entailment between two segment of text

  • RTE2 2006April 10, 2006Thank you very muchO. Ferrndez, R. M. Terol, R. Muoz, P. Martnez-Barco and M. Palomar{ofe,rafamt,rafael,patricio,mpalomar}@dlsi.ua.esGPLSI Natural Language Processing and Information Systems Group