ISCOL 2011 – Bar Ilan University /151 A Probabilistic Model for Lexical Entailment Eyal Shnarch,...

ISCOL 2011 – Bar Ilan University /151

A Probabilistic Model for Lexical Entailment

Eyal Shnarch, Jacob Goldberger, Ido Dagan

Bar Ilan University


Textual Entailment is a common task

Obama gave a speech last night in

the Israeli lobby conference ...

Obama gave a speech last night in

the Israeli lobby conference ...

In his speech at the American Israel Public

Affairs Committee yesterday, the president

challenged …

In his speech at the American Israel Public

Affairs Committee yesterday, the president

challenged … Barack Obama’s AIPAC

address ...Barack Obama’s AIPAC

address ...AIPAC

Israeli lobby

American Israel Public Affairs Committee

address

speech

Barack Obama the president

Obama


Textual Entailment

AIPAC Israeli lobby speech address


The president’s car got stuck in Ireland, surrounded by many peopleThe president’s car got stuck in Ireland, surrounded by many people

Obama’s Cadillac got stuck in Dublin in a large Irish crowdObama’s Cadillac got stuck in Dublin in a large Irish crowd

social group

social group

Modeling entailment at the lexical level


rule2

rule1



social group

social group

Terminology

rule

lexical resource

chain




social group

social group

Goals

p( ) p( ) p( ) Distinguish resources’ reliability levels

Consider transitive chains length

Consider multiple evidence


Probabilistic model for Lexical Entailment

t1 tmti

h1 hnhj

t’

AND

y

OR

chain

… …

……

validity probability of the resource which produces r

(ACL 2011 short paper)


Results on RTE are nice, but…

F1 %Model

RTE 6 RTE 5

33.8 30.5 Avg. of all systems

38.5 36.2 Base Prob.

47.6 44.4 Best lexical system

48.0 45.6 Best full system


Extension 1: relaxing with noisy-AND


Better results on RTE with extension 1

F1 %Model

RTE 6 RTE 5



43.1 44.6 Base Prob. + noisy-AND




Extension 2: considering coverage


Same (better) results on RTE with extension 2

F1 %Model

RTE 6 RTE 5




44.7 42.8 Base Prob. + coverage normalization




Putting it all together is best

F1 %Model

RTE 6 RTE 5




44.7 42.8 Base Prob. + coverage normalization

45.6 48.3 Full Prob. model (noisy-AND + coverage norm)



Negative result: F1 usually decreases when allowing chains


Future work

• Better model for transitivity

• noisy-AND for chains too

• Verify rule application in a specific context

• Test with other application data sets• passage retrieval for QA

• Integrate into a full entailment system


Summary

• Learn for each lexical resource an individual

reliability value

• Consider multiple evidence and chain length

• Probabilistic method to relax the strict AND

demand

• Taking into account the number of covered

terms when modeling entailment probability

A first probabilistic model:

noisy-

ISCOL 2011 – Bar Ilan University /151 A Probabilistic Model for Lexical Entailment Eyal Shnarch,...

Documents

Transcript of ISCOL 2011 – Bar Ilan University /151 A Probabilistic Model for Lexical Entailment Eyal Shnarch,...