Recognizing Partial Textual Entailment
description
Transcript of Recognizing Partial Textual Entailment
1
Recognizing Partial Textual Entailment
Torsten Zesch Iryna GurevychUbiquitous Knowledge Processing Lab
Technische Universität Darmstadt
Omer Levy Ido DaganNatural Language Processing Lab
Bar-Ilan University
2
Textual Entailment
3
T: muscles move bones
4
T: muscles move bones
H: muscles generate movement
5
T: muscles move bones
H: muscles generate movement
6
T: muscles move bones
H: the muscles’ main job is to move bones
?
7
T: muscles move bones
H: the muscles’ main job is to move bones
8
Complete Textual Entailment
9
Complete Textual Entailment
10
Complete Textual Entailment. Partial .
11
Complete Textual Entailment. Partial .
(Nielsen et al, 2009)
12
T: muscles move bones
H: the muscles’ main job is to move bones
13
T: muscles move bones
(main job of muscles) (muscles move) (bones are moved) H: the muscles’ main job is to move bones
14
T: muscles move bones
(main job of muscles) (muscles move) (bones are moved) H: the muscles’ main job is to move bones
15
T: muscles move bones
(main job of muscles) (muscles move) (bones are moved) H: the muscles’ main job is to move bones
Student Response Analysis
16
T: muscles move bones
(main job of muscles) (muscles move) (bones are moved) H: the muscles’ main job is to move bones
Student Response Analysis
Summarization
Information Extraction
17
T: muscles move bones
(main job of muscles) (muscles move) (bones are moved) H: the muscles’ main job is to move bones
Facets
18
T: muscles move bones
(main job of muscles) (muscles move) (bones are moved) H: the muscles’ main job is to move bones (main job, muscles) (muscles, move) (move, bones)
Facets
19
T: muscles move bones
(main job of muscles) (muscles move) (bones are moved) H: the muscles’ main job is to move bones (main job, muscles) (muscles, move) (move, bones)
Facets
20
T: muscles move bones
(main job of muscles) (muscles move) (bones are moved) H: the muscles’ main job is to move bones (main job, muscles) (muscles, move) (move, bones)
Facets
21
Facet: a pair of terms and their implied semantic relation.
22
Facet: a pair of terms and their implied semantic relation.
(move, bones)
23
Facet: a pair of terms and their implied semantic relation.
(move, bones)
… is to move bones
theme
24
Recognizing Faceted Entailment
25
T: muscles move bones
H: the muscles’ main job is to move bones
Recognizing Faceted Entailment
26
T: muscles move bones
H: the muscles’ main job is to move bones
Recognizing Faceted Entailment
27
T: muscles move bones
H: the muscles’ main job is to move bones
Recognizing Faceted Entailment
28
T: muscles move bones
H: the muscles’ main job is to move bones
Recognizing Faceted Entailment
29
What did we do?
30
Approach: Leverage Existing Methods
31
Approach: Leverage Existing Methods
32
Exact Match P: 96%, R: 30%
penicillin cures pneumonia (penicillin, cure) Lexical Inference (penicillin, cure) pneumonia is treated by penicillin
Lexical-Syntactic Inference
Approach: Leverage Existing Methods
33
Exact Match P: 96%, R: 30%
penicillin cures pneumonia (penicillin, cure) Lexical Inference (penicillin, cure) pneumonia is treated by penicillin
Lexical-Syntactic Inference
Approach: Leverage Existing Methods
34
Exact Matchpenicillin cures pneumonia (penicillin, cure) Lexical Inference WordNet
pneumonia is treated by penicillin (penicillin, cure) Lexical-Syntactic Inference
Approach: Leverage Existing Methods
35
Exact Matchpenicillin cures pneumonia (penicillin, cure) Lexical Inference WordNet
pneumonia is treated by penicillin (penicillin, cure) Lexical-Syntactic Inference
Approach: Leverage Existing Methods
36
Exact Matchpenicillin cures pneumonia (penicillin, cure) Lexical Inferencepneumonia is treated by penicillin (penicillin, cure) Lexical-Syntactic Inference
Approach: Leverage Existing Methods
penicillin cures pneumonia
pneumonia is treated by penicillin
37
Bar-Ilan University Textual Entailment Engine
(Stern and Dagan, 2011)
38
BIUTEE
39
BIUTEE
pneumonia
cures
penicillin
T H
40
BIUTEE
pneumonia
cures
penicillin
penicillin
treated
pneumonia is by
T H
41
BIUTEE
pneumonia
cures
penicillin
penicillin
treated
pneumonia is by
T H
42
pneumonia
cures
penicillin
43
pneumonia
cures
penicillin
pneumonia
treats
penicillin
cure treat(lexical)
44
pneumonia
cures
penicillin
penicillin
treated
pneumonia is by
pneumonia
treats
penicillin
cure treat(lexical)
active passive(syntactic)
45
Facets to Dependencies
pneumonia is by
treated
penicillin
46
Facets to Dependencies
penicillin
pneumonia is by
treated
47
Baseline
BaseLex
BaseSyn
Majority
Decision Mechanisms
Exact Match
Exact Match OR Lexical Inference
Exact Match OR Syntactic Inference
Lexical Inference
Exact Match OR AND
Syntactic Inference
48
Baseline
BaseLex
BaseSyn
Majority
Decision Mechanisms
Exact Match
Exact Match OR Lexical Inference
Exact Match OR Syntactic Inference
Lexical Inference
Exact Match OR AND
Syntactic Inference
49
Baseline
BaseLex
BaseSyn
Majority
Decision Mechanisms
Exact Match
Exact Match OR WordNet
Exact Match OR BIUTEE
Lexical Inference
Exact Match OR AND
Syntactic Inference
50
Baseline
BaseLex
BaseSyn
Majority
Decision Mechanisms
Exact Match
Exact Match OR WordNet
Exact Match OR BIUTEE
WordNet
Exact Match OR AND
BIUTEE
51
SemEval 2013 Task 7
58
SciEntsBank (Djikovska et al, 2012)
15 Domains
5,106 Student Responses
16,263 Facets
Dataset
59
SciEntsBank (Djikovska et al, 2012)
15 Domains
5,106 Student Responses
16,263 Facets
Dataset
60
SciEntsBank (Djikovska et al, 2012)
15 Domains
5,106 Student Responses
16,263 Facets
Dataset
61
SciEntsBank (Djikovska et al, 2012)
15 Domains
13,145 Train Facets
16,263 Test Facets
Dataset
62
SciEntsBank (Djikovska et al, 2012)
15 Domains
13,145 Train Facets
16,263 Test Facets
Dataset
63
Unseen Answers 9%
Unseen Questions 12%
Unseen Domains 79%
Scenarios
64
Unseen Answers 9%
Unseen Questions 12%
Unseen Domains 79%
Scenarios
65
Unseen Answers 9%
Unseen Questions 12%
Unseen Domains 79%
Scenarios
66
Unseen Answers 9%
Unseen Questions 12%
Unseen Domains 79%
Scenarios
67
Results
68
Performance (Weighted F1)
Unseen Answers Unseen Questions Unseen Domains0
0.2
0.4
0.6
0.8
1
BaselineBaseLexBaseSynMajority
69
Summary
70
Problem: Need fine-grained entailment
Solution: Partial Textual Entailment
Approach: Leverage existing methods
Results: Significant improvement
71
Problem: Need fine-grained entailment
Solution: Partial Textual Entailment
Approach: Leverage existing methods
Results: Significant improvement
72
Problem: Need fine-grained entailment
Solution: Partial Textual Entailment
Approach: Leverage existing methods
Results: Significant improvement
73
Problem: Need fine-grained entailment
Solution: Partial Textual Entailment
Approach: Leverage existing methods
Results: Significant improvement
76
Partial Textual Entailment is interesting
77
Partial Textual Entailment is useful
78
Partial Textual Entailment is practical
79
Partial Textual Entailment is interesting
80
Questions?