1 Textual Entailment: A Perspective on Applied Text Understanding Ido DaganBar-Ilan University,...
-
Upload
augusta-burns -
Category
Documents
-
view
218 -
download
0
Transcript of 1 Textual Entailment: A Perspective on Applied Text Understanding Ido DaganBar-Ilan University,...
![Page 1: 1 Textual Entailment: A Perspective on Applied Text Understanding Ido DaganBar-Ilan University, Israel Joint works with: Oren Glickman, Idan Szpektor,](https://reader035.fdocuments.in/reader035/viewer/2022062518/56649e995503460f94b9c4ca/html5/thumbnails/1.jpg)
1
Textual Entailment:A Perspective on Applied
Text Understanding
Ido Dagan Bar-Ilan University, Israel
Joint works with:Oren Glickman, Idan Szpektor, Roy Bar Haim
Bar Ilan University, IsraelMaayan Geffet Hebrew University, IsraelHristo Tanev, Bernardo Magnini, Alberto Lavelli, Lorenza Romano ITC-irst, ItalyBonaventura Coppola and Milen Kouylekov
University of Trento and ITC-irst, Italy
![Page 2: 1 Textual Entailment: A Perspective on Applied Text Understanding Ido DaganBar-Ilan University, Israel Joint works with: Oren Glickman, Idan Szpektor,](https://reader035.fdocuments.in/reader035/viewer/2022062518/56649e995503460f94b9c4ca/html5/thumbnails/2.jpg)
2
Talk Focus: A Framework for “Applied Semantics”
• The textual entailment task – what and why?• Empirical evaluation – PASCAL RTE Challenge• Problem scope, decomposition and analysis• Different perspective on semantic inference• Probabilistic framework
• Cf. syntax, MT – clear task, methodology and community
![Page 3: 1 Textual Entailment: A Perspective on Applied Text Understanding Ido DaganBar-Ilan University, Israel Joint works with: Oren Glickman, Idan Szpektor,](https://reader035.fdocuments.in/reader035/viewer/2022062518/56649e995503460f94b9c4ca/html5/thumbnails/3.jpg)
3
Natural Language and Meaning
Meaning
Language
Ambiguity
Variability
![Page 4: 1 Textual Entailment: A Perspective on Applied Text Understanding Ido DaganBar-Ilan University, Israel Joint works with: Oren Glickman, Idan Szpektor,](https://reader035.fdocuments.in/reader035/viewer/2022062518/56649e995503460f94b9c4ca/html5/thumbnails/4.jpg)
4
Variability of Semantic Expression
Dow ends up
Dow climbs 255
The Dow Jones Industrial Average closed up 255
Stock market hits a record high
Dow gains 255 pointsAll major stock markets surged
![Page 5: 1 Textual Entailment: A Perspective on Applied Text Understanding Ido DaganBar-Ilan University, Israel Joint works with: Oren Glickman, Idan Szpektor,](https://reader035.fdocuments.in/reader035/viewer/2022062518/56649e995503460f94b9c4ca/html5/thumbnails/5.jpg)
5
Variability Recognition –Major Inference in Applications
Information Retrieval (IR)
Question Answering (QA)
Multi Document Summarization (MDS)
Information Extraction (IE)
![Page 6: 1 Textual Entailment: A Perspective on Applied Text Understanding Ido DaganBar-Ilan University, Israel Joint works with: Oren Glickman, Idan Szpektor,](https://reader035.fdocuments.in/reader035/viewer/2022062518/56649e995503460f94b9c4ca/html5/thumbnails/6.jpg)
6
Typical Application Inference
Overture’s acquisition by Yahoo
Yahoo bought Overture
Question Expected answer formWho bought Overture? >> X bought Overture
• Similar for IE: X buy Y
• Similar for “semantic” IR: t: Overture was bought …
• Summarization (multi-document) – identify redundant info
• MT evaluation (and recent proposals for MT?)
text hypothesized answer
![Page 7: 1 Textual Entailment: A Perspective on Applied Text Understanding Ido DaganBar-Ilan University, Israel Joint works with: Oren Glickman, Idan Szpektor,](https://reader035.fdocuments.in/reader035/viewer/2022062518/56649e995503460f94b9c4ca/html5/thumbnails/7.jpg)
7
KRAQ'05 Workshop - KNOWLEDGE and REASONING for ANSWERING QUESTIONS
(IJCAI-05)
CFP:– Reasoning aspects:
* information fusion, * search criteria expansion models * summarization and intensional answers, * reasoning under uncertainty or with incomplete
knowledge,– Knowledge representation and integration:
* levels of knowledge involved (e.g. ontologies, domain knowledge),
* knowledge extraction models and techniques to optimize response accuracy,
* coherence and integration.
![Page 8: 1 Textual Entailment: A Perspective on Applied Text Understanding Ido DaganBar-Ilan University, Israel Joint works with: Oren Glickman, Idan Szpektor,](https://reader035.fdocuments.in/reader035/viewer/2022062518/56649e995503460f94b9c4ca/html5/thumbnails/8.jpg)
8
Inference for Textual Question Answering Workshop (AAAI-05)
CFP:• abductions, default reasoning, inference with epistemic logic
or description logic • inference methods for QA need to be robust, cover all
ambiguities of language • available knowledge sources that can be used for inference
… but similar needs for other applications – can we address a uniform empirical task?
![Page 9: 1 Textual Entailment: A Perspective on Applied Text Understanding Ido DaganBar-Ilan University, Israel Joint works with: Oren Glickman, Idan Szpektor,](https://reader035.fdocuments.in/reader035/viewer/2022062518/56649e995503460f94b9c4ca/html5/thumbnails/9.jpg)
9
Applied Textual Entailment: Abstract Semantic Variability Inference
• QA: “Where was John Wayne Born?” – Answer: Iowa
Text (t): The birthplace of John Wayne is in Iowa
Hypothesis (h): John Wayne was born in Iowa
inference
![Page 10: 1 Textual Entailment: A Perspective on Applied Text Understanding Ido DaganBar-Ilan University, Israel Joint works with: Oren Glickman, Idan Szpektor,](https://reader035.fdocuments.in/reader035/viewer/2022062518/56649e995503460f94b9c4ca/html5/thumbnails/10.jpg)
10
The Generic Entailment Task
Text (t): The birthplace of John Wayne is in Iowa
Hypothesis (h): John Wayne was born in Iowa
inference
Given the text t, can we infer that h is (most likely) true?
![Page 11: 1 Textual Entailment: A Perspective on Applied Text Understanding Ido DaganBar-Ilan University, Israel Joint works with: Oren Glickman, Idan Szpektor,](https://reader035.fdocuments.in/reader035/viewer/2022062518/56649e995503460f94b9c4ca/html5/thumbnails/11.jpg)
11
Classical Entailment Definition
• Chierchia & McConnell-Ginet (2001):A text t entails a hypothesis h if h is true in every circumstance (possible world) in which t is true
• Strict entailment - doesn't account for some uncertainty allowed in applications
![Page 12: 1 Textual Entailment: A Perspective on Applied Text Understanding Ido DaganBar-Ilan University, Israel Joint works with: Oren Glickman, Idan Szpektor,](https://reader035.fdocuments.in/reader035/viewer/2022062518/56649e995503460f94b9c4ca/html5/thumbnails/12.jpg)
12
“Almost certain” Entailments
t: The technological triumph known as GPS … was incubated in the mind of Ivan Getting.
h: Ivan Getting invented the GPS.
t: According to the Encyclopedia Britannica, Indonesia is the largest archipelagic nation in the world, consisting of 13,670 islands.
h: 13,670 islands make up Indonesia.
![Page 13: 1 Textual Entailment: A Perspective on Applied Text Understanding Ido DaganBar-Ilan University, Israel Joint works with: Oren Glickman, Idan Szpektor,](https://reader035.fdocuments.in/reader035/viewer/2022062518/56649e995503460f94b9c4ca/html5/thumbnails/13.jpg)
13
Textual Entailment ≈ Human Reading Comprehension
• From a children’s English learning book(Sela and Greenberg):
• Reference Text: “…The Bermuda Triangle lies in the Atlantic Ocean, off the coast of Florida. …”
• Hypothesis (True/False?): The Bermuda Triangle is near the United States
???
![Page 14: 1 Textual Entailment: A Perspective on Applied Text Understanding Ido DaganBar-Ilan University, Israel Joint works with: Oren Glickman, Idan Szpektor,](https://reader035.fdocuments.in/reader035/viewer/2022062518/56649e995503460f94b9c4ca/html5/thumbnails/14.jpg)
14
Reading Comprehension QA
By Canadian Broadcasting Corporation
T: The school has turned its one-time metal shop – lost to budget cuts almost two years ago - into a money-making professional fitness club.
Q: When did the metal shop close?
A: Almost two years ago
![Page 15: 1 Textual Entailment: A Perspective on Applied Text Understanding Ido DaganBar-Ilan University, Israel Joint works with: Oren Glickman, Idan Szpektor,](https://reader035.fdocuments.in/reader035/viewer/2022062518/56649e995503460f94b9c4ca/html5/thumbnails/15.jpg)
15
Recognizing Textual Entailment (RTE) Challenge
PASCAL NOE Challenge2004-5
Ido Dagan, Oren glickman Bar-Ilan University, IsraelBernardo Magnini ITC-irst, Trento, Italy
![Page 16: 1 Textual Entailment: A Perspective on Applied Text Understanding Ido DaganBar-Ilan University, Israel Joint works with: Oren Glickman, Idan Szpektor,](https://reader035.fdocuments.in/reader035/viewer/2022062518/56649e995503460f94b9c4ca/html5/thumbnails/16.jpg)
16
Generic Dataset by Application Use
• QA
• IE
• Similar for “semantic” IR: Overture was acquired by Yahoo
• Comparable documents (summarization)
• MT evaluation
• Reading comprehension
• Paraphrase acquisition
![Page 17: 1 Textual Entailment: A Perspective on Applied Text Understanding Ido DaganBar-Ilan University, Israel Joint works with: Oren Glickman, Idan Szpektor,](https://reader035.fdocuments.in/reader035/viewer/2022062518/56649e995503460f94b9c4ca/html5/thumbnails/17.jpg)
17
Some Examples
TEXTHYPOTHESISTASKENTAIL-
MENT
1iTunes software has seen lower sales in Europe.
Strong sales for iTunes in Europe.
IRFalse
2Cavern Club sessions paid the Beatles £15 evenings and £5 lunchtime.
The Beatles perform at Cavern Club at lunchtime.
IRTrue
3
:…a shootout at the Guadalajara airport in May, 1993, that killed Cardinal Juan Jesus Posadas Ocampo and six others.
Cardinal Juan Jesus Posadas Ocampo died in 1993.
QATrue
• 567 development examples, 800 test examples
![Page 18: 1 Textual Entailment: A Perspective on Applied Text Understanding Ido DaganBar-Ilan University, Israel Joint works with: Oren Glickman, Idan Szpektor,](https://reader035.fdocuments.in/reader035/viewer/2022062518/56649e995503460f94b9c4ca/html5/thumbnails/18.jpg)
18
Dataset Characteristics
• Examples selected and annotated manually– Using automatic systems where available
• Balanced True/False split
• True – certain or highly probable entailment– Filtering controversial examples
• Example distribution?
• Mode –explorative rather than competitive
![Page 19: 1 Textual Entailment: A Perspective on Applied Text Understanding Ido DaganBar-Ilan University, Israel Joint works with: Oren Glickman, Idan Szpektor,](https://reader035.fdocuments.in/reader035/viewer/2022062518/56649e995503460f94b9c4ca/html5/thumbnails/19.jpg)
19
Arthur Bernstein Competition
“… Competition, even a piano competition, is legitimate … as long as it is just an anecdotal side effect of the musical culture scene, and doesn’t threat to overtake the center stage”
Haaretz News Paper
Culture Section, April 1st, 2005
![Page 20: 1 Textual Entailment: A Perspective on Applied Text Understanding Ido DaganBar-Ilan University, Israel Joint works with: Oren Glickman, Idan Szpektor,](https://reader035.fdocuments.in/reader035/viewer/2022062518/56649e995503460f94b9c4ca/html5/thumbnails/20.jpg)
20
Submissions
• 17 participating groups– 26 system submissions – Microsoft Research: manual analysis of dataset
at lexical-syntactic matching level
![Page 21: 1 Textual Entailment: A Perspective on Applied Text Understanding Ido DaganBar-Ilan University, Israel Joint works with: Oren Glickman, Idan Szpektor,](https://reader035.fdocuments.in/reader035/viewer/2022062518/56649e995503460f94b9c4ca/html5/thumbnails/21.jpg)
21
Broad Range of System Types
• Knowledge sources and inferences– Direct t-h matching:
• Word overlap / Syntactic tree matching
– Lexical relations:• WordNet & statistical (corpus based)
– Theorem Provers / Logical inference • Adding a fuzzy scoring mechanism
• Supervised / unsupervised learning methods
![Page 22: 1 Textual Entailment: A Perspective on Applied Text Understanding Ido DaganBar-Ilan University, Israel Joint works with: Oren Glickman, Idan Szpektor,](https://reader035.fdocuments.in/reader035/viewer/2022062518/56649e995503460f94b9c4ca/html5/thumbnails/22.jpg)
22
![Page 23: 1 Textual Entailment: A Perspective on Applied Text Understanding Ido DaganBar-Ilan University, Israel Joint works with: Oren Glickman, Idan Szpektor,](https://reader035.fdocuments.in/reader035/viewer/2022062518/56649e995503460f94b9c4ca/html5/thumbnails/23.jpg)
23
Accuracy
0.4
0.5
0.6
MIT
RE
Bar
Ila
nU
NE
DD
ublin
Edi
nbur
gh-
Dub
linS
tanf
ord
UIU
CIR
ST
IRS
TU
NE
DE
dinb
urgh
-A
mst
erda
mS
tanf
ord
LCC
Am
ster
dam
accuracy
0.01 sig
0.05 sig
![Page 24: 1 Textual Entailment: A Perspective on Applied Text Understanding Ido DaganBar-Ilan University, Israel Joint works with: Oren Glickman, Idan Szpektor,](https://reader035.fdocuments.in/reader035/viewer/2022062518/56649e995503460f94b9c4ca/html5/thumbnails/24.jpg)
24
Where are we?
![Page 25: 1 Textual Entailment: A Perspective on Applied Text Understanding Ido DaganBar-Ilan University, Israel Joint works with: Oren Glickman, Idan Szpektor,](https://reader035.fdocuments.in/reader035/viewer/2022062518/56649e995503460f94b9c4ca/html5/thumbnails/25.jpg)
25
What’s next – RTE-2• Organizers:
– Bar Ilan, CELCT (Trento), MITRE, MS-Research
• Main dataset: utilizing real systems outputs – QA, IE, IR, summarization
• Human performance dataset– Reading comprehension, human QA (planned)
• Schedule (RTE website):– October – development set– February – results submission (test set January)– April 10 – PASCAL workshop in Venice!
• right after EACL
![Page 26: 1 Textual Entailment: A Perspective on Applied Text Understanding Ido DaganBar-Ilan University, Israel Joint works with: Oren Glickman, Idan Szpektor,](https://reader035.fdocuments.in/reader035/viewer/2022062518/56649e995503460f94b9c4ca/html5/thumbnails/26.jpg)
26
Other Evaluation Modes
• Entailment subtasks evaluations– Lexical, lexical-syntactic, alignment…
• “Seek” mode:– Input: h and corpus– Output: All entailing t’s in corpus– Captures nicely information seeking needs, but
requires post-run annotation (like TREC)
• Contribution to specific applications
![Page 27: 1 Textual Entailment: A Perspective on Applied Text Understanding Ido DaganBar-Ilan University, Israel Joint works with: Oren Glickman, Idan Szpektor,](https://reader035.fdocuments.in/reader035/viewer/2022062518/56649e995503460f94b9c4ca/html5/thumbnails/27.jpg)
27
Empirical Modeling of Meaning Equivalence and Entailment
ACL-05 Workshop
Roy Bar-Haim Idan Szpektor
Oren GlickmanBar-Ilan University
Decomposition ofEntailment Levels
![Page 28: 1 Textual Entailment: A Perspective on Applied Text Understanding Ido DaganBar-Ilan University, Israel Joint works with: Oren Glickman, Idan Szpektor,](https://reader035.fdocuments.in/reader035/viewer/2022062518/56649e995503460f94b9c4ca/html5/thumbnails/28.jpg)
28
Why?
• Entailment Modeling is Complex!!– Was apparent at RTE1
• How can we decompose it, for– Better analysis and sub-task modeling– Piecewise evaluation
• Avoid “this is the performance of my complex system…” methodology
![Page 29: 1 Textual Entailment: A Perspective on Applied Text Understanding Ido DaganBar-Ilan University, Israel Joint works with: Oren Glickman, Idan Szpektor,](https://reader035.fdocuments.in/reader035/viewer/2022062518/56649e995503460f94b9c4ca/html5/thumbnails/29.jpg)
29
Combination of Inference Types
TThe oddest thing about the UAE is that only 500,000 of the 2 million people living in the country are UAE citizens.
HThe population of the United Arab Emirates is 2 million.
T H
![Page 30: 1 Textual Entailment: A Perspective on Applied Text Understanding Ido DaganBar-Ilan University, Israel Joint works with: Oren Glickman, Idan Szpektor,](https://reader035.fdocuments.in/reader035/viewer/2022062518/56649e995503460f94b9c4ca/html5/thumbnails/30.jpg)
30
Combination of Inference Types
The oddest thing about the UAE is that only 500,000 of the 2 million people living in the country are UAE citizens.
The oddest thing about the UAE is that only 500,000 of the 2 million people living in the UAE are UAE citizens.
2 million people live in UAE.
The population of the UAE is 2 million.
The population of the United Arab Emirates is 2 million
Co-reference
Syntactic trans.
paraphrasing
Lexical world knowledge
Diverse inference types, different levels of representation
T
H
![Page 31: 1 Textual Entailment: A Perspective on Applied Text Understanding Ido DaganBar-Ilan University, Israel Joint works with: Oren Glickman, Idan Szpektor,](https://reader035.fdocuments.in/reader035/viewer/2022062518/56649e995503460f94b9c4ca/html5/thumbnails/31.jpg)
31
Defining Intermediate Models
• Lexical
• Lexical-syntactic
![Page 32: 1 Textual Entailment: A Perspective on Applied Text Understanding Ido DaganBar-Ilan University, Israel Joint works with: Oren Glickman, Idan Szpektor,](https://reader035.fdocuments.in/reader035/viewer/2022062518/56649e995503460f94b9c4ca/html5/thumbnails/32.jpg)
32
Lexical Model• T and H are represented as bag of terms
• T L H if
– for each term u H there exists a term v T such that v L u
• v L u if
– they share the same lemma and POS
OR– they are connected by a chain of lexical
transformations
![Page 33: 1 Textual Entailment: A Perspective on Applied Text Understanding Ido DaganBar-Ilan University, Israel Joint works with: Oren Glickman, Idan Szpektor,](https://reader035.fdocuments.in/reader035/viewer/2022062518/56649e995503460f94b9c4ca/html5/thumbnails/33.jpg)
33
Lexical Transformations
• We assume perfect word sense disambiguation
Morphological derivations
acquisition acquire
terrorist terror
Ontological relations
Synonyms (buy acquire)
Hypernyms (produce make)
Meronym (executive company)
Lexical world knowledge
Bill Gates Microsoft’s founder
kill die
![Page 34: 1 Textual Entailment: A Perspective on Applied Text Understanding Ido DaganBar-Ilan University, Israel Joint works with: Oren Glickman, Idan Szpektor,](https://reader035.fdocuments.in/reader035/viewer/2022062518/56649e995503460f94b9c4ca/html5/thumbnails/34.jpg)
34
Lexical Entailment - Examples• #1952 from RTE1 (TH)
TTLLHH
TCrude oil prices soared to record levels
HCrude oil prices rise
?
![Page 35: 1 Textual Entailment: A Perspective on Applied Text Understanding Ido DaganBar-Ilan University, Israel Joint works with: Oren Glickman, Idan Szpektor,](https://reader035.fdocuments.in/reader035/viewer/2022062518/56649e995503460f94b9c4ca/html5/thumbnails/35.jpg)
35
Lexical Entailment - Examples• #1361 from RTE1 (TH)
TCrude oil prices soared to record levels
HCrude oil prices rise.
![Page 36: 1 Textual Entailment: A Perspective on Applied Text Understanding Ido DaganBar-Ilan University, Israel Joint works with: Oren Glickman, Idan Szpektor,](https://reader035.fdocuments.in/reader035/viewer/2022062518/56649e995503460f94b9c4ca/html5/thumbnails/36.jpg)
36
Lexical Entailment - Examples• #1361 from RTE1 (TH)
TCrude oil prices soared to record levels
HCrude oil prices rise
Synonym
![Page 37: 1 Textual Entailment: A Perspective on Applied Text Understanding Ido DaganBar-Ilan University, Israel Joint works with: Oren Glickman, Idan Szpektor,](https://reader035.fdocuments.in/reader035/viewer/2022062518/56649e995503460f94b9c4ca/html5/thumbnails/37.jpg)
37
Lexical Entailment - Examples• #1952 from RTE1 (TH)
TCrude oil prices soared to record levels
HCrude oil prices rise
Synonym
TTLLH H
![Page 38: 1 Textual Entailment: A Perspective on Applied Text Understanding Ido DaganBar-Ilan University, Israel Joint works with: Oren Glickman, Idan Szpektor,](https://reader035.fdocuments.in/reader035/viewer/2022062518/56649e995503460f94b9c4ca/html5/thumbnails/38.jpg)
38
Lexical Entailment - Examples• #2127 from RTE1 (TH)
TA coyote was shot after biting girl in park
HA girl was shot in a park
TTLLHH?
![Page 39: 1 Textual Entailment: A Perspective on Applied Text Understanding Ido DaganBar-Ilan University, Israel Joint works with: Oren Glickman, Idan Szpektor,](https://reader035.fdocuments.in/reader035/viewer/2022062518/56649e995503460f94b9c4ca/html5/thumbnails/39.jpg)
39
Lexical Entailment - Examples• #2127 from RTE1 (TH)
T A coyote was shot after biting girl in Vanier Park
Hgirl was shot in a park A
TTLLH H
![Page 40: 1 Textual Entailment: A Perspective on Applied Text Understanding Ido DaganBar-Ilan University, Israel Joint works with: Oren Glickman, Idan Szpektor,](https://reader035.fdocuments.in/reader035/viewer/2022062518/56649e995503460f94b9c4ca/html5/thumbnails/40.jpg)
40
Lexical-Syntactic Model• T and H are represented by syntactic
dependency relations
• T LS H if the relations within H can be matched by the relations in T
• The coverage can be obtained through a sequence of lexical-syntactic transformations
![Page 41: 1 Textual Entailment: A Perspective on Applied Text Understanding Ido DaganBar-Ilan University, Israel Joint works with: Oren Glickman, Idan Szpektor,](https://reader035.fdocuments.in/reader035/viewer/2022062518/56649e995503460f94b9c4ca/html5/thumbnails/41.jpg)
41
Lexical-Syntactic Transformations
• We assume perfect disambiguation and reference resolution
Lexical•Synonyms, hypernyms, etc. (as before)
Syntactic •Active/Passive
•Apposition
do not change lexical elements
Lexical-synt. Entailment Paraphrases
•X take in Y Y join X
•X is Y man by birth X was born in Y
change both lexical elements and structure
Co-referenceThe country UAE
![Page 42: 1 Textual Entailment: A Perspective on Applied Text Understanding Ido DaganBar-Ilan University, Israel Joint works with: Oren Glickman, Idan Szpektor,](https://reader035.fdocuments.in/reader035/viewer/2022062518/56649e995503460f94b9c4ca/html5/thumbnails/42.jpg)
42
Lexical-Syntactic Entailment - Examples
• #1361 from RTE1 (TH)
TCrude oil prices soared to record levels
HCrude oil prices rise
subj
subj
TTLSLSH H
![Page 43: 1 Textual Entailment: A Perspective on Applied Text Understanding Ido DaganBar-Ilan University, Israel Joint works with: Oren Glickman, Idan Szpektor,](https://reader035.fdocuments.in/reader035/viewer/2022062518/56649e995503460f94b9c4ca/html5/thumbnails/43.jpg)
43
Lexical-Syntactic Entailment - Examples
• #2127 from RTE1 (TH)
T A Coyote was shot after biting girl in Vanier Park
H A girl was shot in a park
TTLSLSH H
subj
subj
![Page 44: 1 Textual Entailment: A Perspective on Applied Text Understanding Ido DaganBar-Ilan University, Israel Joint works with: Oren Glickman, Idan Szpektor,](https://reader035.fdocuments.in/reader035/viewer/2022062518/56649e995503460f94b9c4ca/html5/thumbnails/44.jpg)
44
Beyond Lexical-Syntactic Models TThe SPD got just 21.5% of the vote in the European
Parliament elections, while the conservative opposition parties polled 44.5%
HThe SPD was defeated by the opposition parties.
• Future work…
![Page 45: 1 Textual Entailment: A Perspective on Applied Text Understanding Ido DaganBar-Ilan University, Israel Joint works with: Oren Glickman, Idan Szpektor,](https://reader035.fdocuments.in/reader035/viewer/2022062518/56649e995503460f94b9c4ca/html5/thumbnails/45.jpg)
45
Empirical Analysis
![Page 46: 1 Textual Entailment: A Perspective on Applied Text Understanding Ido DaganBar-Ilan University, Israel Joint works with: Oren Glickman, Idan Szpektor,](https://reader035.fdocuments.in/reader035/viewer/2022062518/56649e995503460f94b9c4ca/html5/thumbnails/46.jpg)
46
Annotation
• 240 T-H pairs of RTE1 dataset• T L H ; T LS H
• High annotator agreement (authors)
Entailment ModelAgreementKappa
Lexical89.6%0.78
Lexical-Syntactic88.8%0.73
• Kappa: “substantial agreement”
![Page 47: 1 Textual Entailment: A Perspective on Applied Text Understanding Ido DaganBar-Ilan University, Israel Joint works with: Oren Glickman, Idan Szpektor,](https://reader035.fdocuments.in/reader035/viewer/2022062518/56649e995503460f94b9c4ca/html5/thumbnails/47.jpg)
47
Model evaluation results
• Low precision for Lexical model Lexical match fails to predict entailment
• High precision for Lexical Syntactic model Checking syntactic relations is crucial
• Medium recall for both levels Higher levels of inference are missing
ModelRecallPrecisionF1
Lexical44%59%0.50
Lexical Syntactic50%86%0.63
![Page 48: 1 Textual Entailment: A Perspective on Applied Text Understanding Ido DaganBar-Ilan University, Israel Joint works with: Oren Glickman, Idan Szpektor,](https://reader035.fdocuments.in/reader035/viewer/2022062518/56649e995503460f94b9c4ca/html5/thumbnails/48.jpg)
48
contribution of individual components RTE 1 positive examples
Inference typefR%
Synonym1914%16%
Morphological1610%14%
Lexical world knowledge128%10%
Hypernym74%6%
Meronym11%1%
Entailment paraphrases3726%31%
Syntactic Transformations2217%19%
Co-reference105%8%
Lexi
cal
Lex-
Syn
![Page 49: 1 Textual Entailment: A Perspective on Applied Text Understanding Ido DaganBar-Ilan University, Israel Joint works with: Oren Glickman, Idan Szpektor,](https://reader035.fdocuments.in/reader035/viewer/2022062518/56649e995503460f94b9c4ca/html5/thumbnails/49.jpg)
49
Summary (1)
• Annotating and analaysing entailment components
• Guide research on entailment
• Opens new research problems and redirects old ones
![Page 50: 1 Textual Entailment: A Perspective on Applied Text Understanding Ido DaganBar-Ilan University, Israel Joint works with: Oren Glickman, Idan Szpektor,](https://reader035.fdocuments.in/reader035/viewer/2022062518/56649e995503460f94b9c4ca/html5/thumbnails/50.jpg)
50
Summary (2)
• Allows better evaluation of systems– Performance of individual components
• Future work – expand analysis to additional levels of representation and inferences– Identify the exciting semantic phenomena …
![Page 51: 1 Textual Entailment: A Perspective on Applied Text Understanding Ido DaganBar-Ilan University, Israel Joint works with: Oren Glickman, Idan Szpektor,](https://reader035.fdocuments.in/reader035/viewer/2022062518/56649e995503460f94b9c4ca/html5/thumbnails/51.jpg)
51
A Different Perspective on Semantic Inference
![Page 52: 1 Textual Entailment: A Perspective on Applied Text Understanding Ido DaganBar-Ilan University, Israel Joint works with: Oren Glickman, Idan Szpektor,](https://reader035.fdocuments.in/reader035/viewer/2022062518/56649e995503460f94b9c4ca/html5/thumbnails/52.jpg)
52
Text Mapping vs. Interpretation
• Focus on the entailment relation as a (directed) mapping between language expressions– Identify the contextual constraints for mappings
• Vs. interpret language into meaning representations (explicitly stipulated senses, logical form, etc.)– Can still be a mean, rather than goal
• How far (faster) can we get?– Cf. MT – direct, transfer, interlingua
![Page 53: 1 Textual Entailment: A Perspective on Applied Text Understanding Ido DaganBar-Ilan University, Israel Joint works with: Oren Glickman, Idan Szpektor,](https://reader035.fdocuments.in/reader035/viewer/2022062518/56649e995503460f94b9c4ca/html5/thumbnails/53.jpg)
53
Making sense of (implicit) senses
• What is the RIGHT set of senses?– Any concrete set is problematic/subjective– … but WSD forces you to choose one
• A lexical entailment perspective:– Instead of identifying an explicitly stipulated
sense of a word occurrence …– identify whether a word occurrence (i.e. its
implicit sense) entails another word occurrence, in context
![Page 54: 1 Textual Entailment: A Perspective on Applied Text Understanding Ido DaganBar-Ilan University, Israel Joint works with: Oren Glickman, Idan Szpektor,](https://reader035.fdocuments.in/reader035/viewer/2022062518/56649e995503460f94b9c4ca/html5/thumbnails/54.jpg)
54
That’s what applications need• Lexical matching: recognize sense equivalence
T1: IKEA announced a new comfort chair
Q: announcement of new models of chairs
T2: MIT announced a new CS chair position
T1: IKEA announced a new comfort chair
Q: announcement of new models of furniture
T2: MIT announced a new CS chair position
• Lexical expansion: Recognize sense entailment
![Page 55: 1 Textual Entailment: A Perspective on Applied Text Understanding Ido DaganBar-Ilan University, Israel Joint works with: Oren Glickman, Idan Szpektor,](https://reader035.fdocuments.in/reader035/viewer/2022062518/56649e995503460f94b9c4ca/html5/thumbnails/55.jpg)
55
Bottom Line
• Address semantic inference as text mapping, rather than interpretation
• From applications perspective - interpretation may be a mean, not the goal– we shouldn’t create artificial problems, which
might be harder than those we need to solve
![Page 56: 1 Textual Entailment: A Perspective on Applied Text Understanding Ido DaganBar-Ilan University, Israel Joint works with: Oren Glickman, Idan Szpektor,](https://reader035.fdocuments.in/reader035/viewer/2022062518/56649e995503460f94b9c4ca/html5/thumbnails/56.jpg)
56
Probabilistic Framework forTextual Entailment
Oren Glickman, Ido Dagan,Moshe Koppel and Jacob Goldberger
Bar Ilan UniversityACL-05 Workshop, AAAI-05
![Page 57: 1 Textual Entailment: A Perspective on Applied Text Understanding Ido DaganBar-Ilan University, Israel Joint works with: Oren Glickman, Idan Szpektor,](https://reader035.fdocuments.in/reader035/viewer/2022062518/56649e995503460f94b9c4ca/html5/thumbnails/57.jpg)
57
Motivation
• Approach entailment uncertainty by principled probabilistic models– Following success of statistical MT, parsing,
language modeling etc.– Integrating inferences and knowledge sources– Vs. ad-hoc scoring
• Need to define concrete probability space– Generative model
![Page 58: 1 Textual Entailment: A Perspective on Applied Text Understanding Ido DaganBar-Ilan University, Israel Joint works with: Oren Glickman, Idan Szpektor,](https://reader035.fdocuments.in/reader035/viewer/2022062518/56649e995503460f94b9c4ca/html5/thumbnails/58.jpg)
58
Notation
• t -- a text (t T)
• h -- a hypothesis (h H)– propositional statements which can be assigned
a truth value
• w: H → {true, false} -- a possible world– truth assignment for every hypothesis
![Page 59: 1 Textual Entailment: A Perspective on Applied Text Understanding Ido DaganBar-Ilan University, Israel Joint works with: Oren Glickman, Idan Szpektor,](https://reader035.fdocuments.in/reader035/viewer/2022062518/56649e995503460f94b9c4ca/html5/thumbnails/59.jpg)
59
A Generative Model
We assume a probabilistic generative model:– generation event of <t,w>:
a text along with a (hidden) possible world
– based on a joint probability distribution
John was born in France
(t)
John Speaks French 1John was born in Paris 1John likes fois gras 0John is married to Alice 1
…(w)
Hidden Possible World
(w)
![Page 60: 1 Textual Entailment: A Perspective on Applied Text Understanding Ido DaganBar-Ilan University, Israel Joint works with: Oren Glickman, Idan Szpektor,](https://reader035.fdocuments.in/reader035/viewer/2022062518/56649e995503460f94b9c4ca/html5/thumbnails/60.jpg)
60
Probabilities
• For a given text t and hypothesis h, we consider the following probabilities:– P(Trh=1)
• Probability that h is assigned a truth value of 1 in a generated <t,w> pair
– P(Trh=1| t) • Probability that h is assigned a truth value of 1 given
that the corresponding text is t
![Page 61: 1 Textual Entailment: A Perspective on Applied Text Understanding Ido DaganBar-Ilan University, Israel Joint works with: Oren Glickman, Idan Szpektor,](https://reader035.fdocuments.in/reader035/viewer/2022062518/56649e995503460f94b9c4ca/html5/thumbnails/61.jpg)
61
Probabilistic Textual Entailment
Definition: • t probabilistically entails h if:
– P(Trh = 1| t) > P(Trh = 1)• t increases the likelihood of h being true • Positive PMI – t provides information on h’s truth
• P(Trh = 1| t): entailment confidence– The relevant entailment score for applications– In practice: high confidence required
![Page 62: 1 Textual Entailment: A Perspective on Applied Text Understanding Ido DaganBar-Ilan University, Israel Joint works with: Oren Glickman, Idan Szpektor,](https://reader035.fdocuments.in/reader035/viewer/2022062518/56649e995503460f94b9c4ca/html5/thumbnails/62.jpg)
62
Setting Properties (1)
• Logical vs. Textual Entailment– Logical entailment: proposition proposition– Textual entailment: text text
• Conditioning on generation of texts rather than on propositional values– David’s father was born in Italy David was born in Italy
• Possible ambiguities of the texts are taken into account– Play baseball with a bat play baseball with an animal
![Page 63: 1 Textual Entailment: A Perspective on Applied Text Understanding Ido DaganBar-Ilan University, Israel Joint works with: Oren Glickman, Idan Szpektor,](https://reader035.fdocuments.in/reader035/viewer/2022062518/56649e995503460f94b9c4ca/html5/thumbnails/63.jpg)
63
Setting Properties (2)
• We do not distinguish between inferences that are based on – language semantics: e.g. murdering killing– vs. domain or world knowledge:
• e.g. live in Paris live in France
• Setting accounts for all causes of uncertainty
![Page 64: 1 Textual Entailment: A Perspective on Applied Text Understanding Ido DaganBar-Ilan University, Israel Joint works with: Oren Glickman, Idan Szpektor,](https://reader035.fdocuments.in/reader035/viewer/2022062518/56649e995503460f94b9c4ca/html5/thumbnails/64.jpg)
64
Setting Properties (3)
• for a given text t and hypothesis h
h P(Trh=1|t) ≠ 1
• But rather:
–P(Trh=1|t) + P(Trh=0 |t) = 1
• Vs. generative language models (cf. speech, MT, LM for IR)
![Page 65: 1 Textual Entailment: A Perspective on Applied Text Understanding Ido DaganBar-Ilan University, Israel Joint works with: Oren Glickman, Idan Szpektor,](https://reader035.fdocuments.in/reader035/viewer/2022062518/56649e995503460f94b9c4ca/html5/thumbnails/65.jpg)
65
Having a probability space
• we can now define concrete probabilistic models for various entailment phenomena
![Page 66: 1 Textual Entailment: A Perspective on Applied Text Understanding Ido DaganBar-Ilan University, Israel Joint works with: Oren Glickman, Idan Szpektor,](https://reader035.fdocuments.in/reader035/viewer/2022062518/56649e995503460f94b9c4ca/html5/thumbnails/66.jpg)
66
Initial Lexical Models• Alignment-based (ACL-05 Workshop)
– The probability that a term in h is entailed by a particular term in t
• Bayesian classification (AAAI-05)– The probability that a term in h is entailed by
(fits in) the entire text of t– An unsupervised text categorization setting
(with EM) – each term is a category
• Demonstrate directions for probabilistic modeling and unsupervised estimation
![Page 67: 1 Textual Entailment: A Perspective on Applied Text Understanding Ido DaganBar-Ilan University, Israel Joint works with: Oren Glickman, Idan Szpektor,](https://reader035.fdocuments.in/reader035/viewer/2022062518/56649e995503460f94b9c4ca/html5/thumbnails/67.jpg)
67
Additional Work:Acquiring Entailment Relations
• Lexical (Geffet and Dagan, 2004/2005)– A clear goal for distributional similarity
– Obtain characteristic features via bootstrapping
– Test characteristic feature inclusion (vs. overlap)
• Lexical Syntactic – TEASE (Szpektor et al. 2004)– Deduce entailment from joint anchor sets
– Initial prospects for unsupervised IE
• Next: obtain probabilities for these entailment “rules”
![Page 68: 1 Textual Entailment: A Perspective on Applied Text Understanding Ido DaganBar-Ilan University, Israel Joint works with: Oren Glickman, Idan Szpektor,](https://reader035.fdocuments.in/reader035/viewer/2022062518/56649e995503460f94b9c4ca/html5/thumbnails/68.jpg)
68
Conclusions: Textual entailment…
• Provides a framework for semantic inference– Application-independent abstraction– Text mapping rather than interpretation
• Raises interesting problems to work on• Amenable for empirical evaluation and
decomposition• May be modeled in principled probabilistic
terms
Thank you!
![Page 69: 1 Textual Entailment: A Perspective on Applied Text Understanding Ido DaganBar-Ilan University, Israel Joint works with: Oren Glickman, Idan Szpektor,](https://reader035.fdocuments.in/reader035/viewer/2022062518/56649e995503460f94b9c4ca/html5/thumbnails/69.jpg)
69
Textual Entailment ReferencesWorkshops · PASCAL Challenges Workshop for Recognizing Textual Entailment, 2005
http://www.cs.biu.ac.il/~glikmao/rte05/index.htmlNote: see 2nd RTE Challenge at http://www.cs.biu.ac.il/~barhair/RTE2/
· ACL 2005 Workshop on Empirical Modeling of Semantic Equivalence and Entailment, 2005http://acl.ldc.upenn.edu/W/W05/#W05-1200
Papers from recent conferences and workshopsJ. Bos & K. Markert. 2005. Recognising Textual Entailment with Logical Inference. Proceedings
of EMNLP 2005.R. Braz, R. Girju, V. Punyakanok, D. Roth, and M. Sammons. 2005. An Inference Model for
Semantic Entailment in Natural Language. Twentieth National Conference on Artificial Intelligence (AAAI-05)
R. Braz, R. Girju, V. Punyakanok, D. Roth, and M. Sammons. 2005. Knowledge Representation for Semantic Entailment and Question-Answering. IJCAI-05 Workshop on Knowledge and Reasoning for Answering Questions.
C. Corley, A. Csomai and R. Mihalcea. Text Semantic Similarity, with Applications. RANLP-05.I. Dagan and O. Glickman. 2004. Probabilistic textual entailment: Generic applied modeling of
language variability. In PASCAL Workshop on Learning Methods for Text Understanding and Mining, Grenoble.
![Page 70: 1 Textual Entailment: A Perspective on Applied Text Understanding Ido DaganBar-Ilan University, Israel Joint works with: Oren Glickman, Idan Szpektor,](https://reader035.fdocuments.in/reader035/viewer/2022062518/56649e995503460f94b9c4ca/html5/thumbnails/70.jpg)
70
Textual Entailment References (2)M. Geffet and I. Dagan. Feature Vector Quality and Distributional Similarity. Proceedings of
The 20th International Conference on Computational Linguistics (COLING), 2004.M. Geffet and I. Dagan. 2005. "The Distributional Inclusion Hypotheses and Lexical
Entailment", ACL 2005, Michigan, USA. O. Glickman, I. Dagan and M. Koppel. 2005. A Probabilistic Classification Approach for
Lexical Textual Entailment, Twentieth National Conference on Artificial Intelligence (AAAI-05)
A. Haghighi, A. Y. Ng, and C. D. Manning. 2005. Robust Textual Inference via Graph Matching. HLT-EMNLP 2005.
M. Kouylekov and B. Magnini. 2005. Tree Edit Distance for Textual Entailment. RANLP 2005.R. Raina, A. Y. Ng, and C. Manning. 2005. Robust textual inference via learning and abductive
reasoning. Twentieth National Conference on Artificial Intelligence (AAAI-05) V. Rus, A. Graesser and K. Desai. 2005. Lexico-Syntactic Subsumption for Textual Entailment.
RANLP 2005.M. Tatu and D. Moldovan. 2005. A Semantic Approach to Recognizing Textual Entailment.
HLT-EMNLP 2005.We would be glad to receive more references on textual entailment. Please send them to