Toward Dependency Path based Entailment
description
Transcript of Toward Dependency Path based Entailment
Toward Dependency Path based Entailment
Rodney Nielsen, Wayne Ward, and James Martin
Dependency Path-based Entailment
DIRT (Lin and Pantel, 2001) Unsupervised method to discover
inference rules “X is author of Y ≈ X wrote Y” “X solved Y ≈ X found a solution to Y”
If two dependency paths tend to link the same sets of words, they hypothesize that their meanings are similar
ML Classification Approach
Features derived from corpus statistics Unigram co-occurrence Surface form bigram co-occurrence Dependency-derived bigram co-occurrence
Mixture of experts: About 18 ML classifiers from Weka toolkit Classify by majority vote or average
probability
Bag of Words Graph MatchingDependency PathBased Entailment
Corpora
7.4M articles, 2.5B words, 347 words/doc Gigaword (Graff, 2003) – 77% of documents Reuters Corpus (Lewis et al., 2004) TIPSTER
Lucene IR engine Two indices
Word surface form Porter stem filter
Stop words = {a, an, the}
Core Features
Core Repeated Features
Product of MLEs
Average of MLEs
Geometric Mean of MLEs
Worst Non-Zero MLE
Entailing Ngrams for the Lowest Non-Zero MLE
Largest Entailing Ngram Count with a Zero MLE
Smallest Entailing Ngram Count with a Non-Zero MLE
Count of Ngrams in h that do not Co-occur with any Ngrams from t
Count of Ngrams in h that do Co-occur with Ngrams in t
Dependency Features
Dependency bigram features
pc
pcpc
dpcvv
vvww
tvvpc n
ntwwP
,
,,,
,max,,
Hypothesis h Text t
rising
cost is
The of
paper
choke
Newspapers on
costs
and
falling
rising paper revenues
Dependency Features
c
cw
wcw tsPtwwPK
tsP ,,,1
, 21
Hypothesis h Text t
rising
cost is
The of
paper
choke
Newspapers on
costs
and
falling
rising paper revenues
Descendent relation statistics
Dependency Features
0,,1
1 , twwPtsP ofpaperof
Hypothesis h Text t
rising
cost is
The of
paper
choke
Newspapers on
costs
and
falling
rising paper revenues
Descendent relation statistics
Dependency Features
tsPtwwPtwwPtsP ,21,,2
10,,2
1 , ofcostofcostthecost
Hypothesis h Text t
rising
cost is
The of
paper
choke
Newspapers on
costs
and
falling
rising paper revenues
Descendent relation statistics
Dependency Features
0,,,21,,2
12
1 , risingiscostrisingcostrising twwPtsPtwwPtsP
Hypothesis h Text t
rising
cost is
The of
paper
choke
Newspapers on
costs
and
falling
rising paper revenues
Descendent relation statistics
Verb Dependency Features
Hypothesis h Text t
rising
cost is
The of
paper
choke
Newspapers on
costs
and
falling
rising paper revenues
Combined verb descendent relation features
Worst verb descendent relation features
Subject Dependency
Features
Combined and worst subject descendent relations
Combined and worst subject-to-verb paths
Hypothesis h Text t
rising
cost is
The of
paper
choke
Newspapers on
costs
and
falling
rising paper revenues
Other Dependency Features
Repeat these same features for: Object pcomp-n Other descendent relations
Results
RTE2 by Task: IE IR QA SUM Overall
Accuracy 55.5 64.0 55.0 70.0 61.1
Average Precision 49.4 73.0 57.3 80.7 65.2
RTE2 Accuracy SUM NonSUM Overall
Test Set 70.0 58.2 61.1
Training Set CV 84.5 62.7 68.1
RTE1 Accuracy CD NonCD Overall
Test Set (Best
submission)83.3
(83.3)56.8
(52.8)61.8
(58.6)
Training Set CV 83.7 56.9 61.6
Feature Analysis
All feature sets are contributing according to cross validation on the training set
Most significant feature set: Unigram stem based word alignment
Most significant core repeated feature: Average MLE
Questions
Mixture of experts classifier using corpus co-occurrence statistics Moving in the direction of DIRT Domain of Interest: Student response analysis in intelligent tutoring systems
RTE2 Task: IE IR QA SUM
All
Accuracy 55.5
64.0
55.0
70.0
61.1
Average Precision
49.4
73.0
57.3
80.7
65.2
Bag of Words Graph MatchingDependency PathBased Entailment
Hypothesis hRTE2 Accuracy SUM NonSUM Overall
Test Set 70.0 58.2 61.1
Training Set CV 84.5 62.7 68.1
Text t
rising
cost is
The of
paper
choke
Newspapers on
costs
and
falling
rising paper revenues
RTE1 Accuracy CD NonCD Overall
Test Set (Best Subm)
83.3 (83.3)
56.8 (52.8)
61.8 (58.6)
Training Set CV 83.7 56.9 61.6
c
cw
wcw tsPtwwPK
tsP ,,,1
, 21
Why Entailment
Intelligent Tutoring Systems Student Interaction Analysis
Are all aspects of the student’s answer entailed by the text and the gold standard answer
Are all aspects of the desired answer entailed by the student’s response
Word Alignment Features
hw v
vw
tvh
v
vw
tv
v
vw
tvw
tvw
n
ntTrP
n
ntw
n
nvTrPtTrP
,
,
,
max|1
max,MLE
max|1max|1
Unigram word alignment
Word Alignment Features Bigram word alignment
Example: <t>Newspapers choke on rising paper costs and
falling revenue.</t><h>The cost of paper is rising.</h>
MLE(cost, t) = ncost of, costs of /ncosts of = 6086/35800 = 0.17
1
11
1
11
1
11
1
11
,
4
,
3
,
2
,
1
1max,MLE
jj
jjji
jj
jjij
ij
ijii
ji
jiii
vv
vvvw
vv
vvwv
wv
wvww
vw
vwww
tvi
n
n
n
n
n
n
n
n
ktw
Word Alignment Features
Average unigram and bigram
Stem-based tokens
Corpora
7.4M articles/docs & 2.5B words, 347 words/doc Gigaword (Graff, 2003) -
5.7M articles, 2.1B words, 375 words/article 77% of documents and 83% of indexed
words Reuters Corpus (Lewis et al., 2004)
0.8M articles, 0.17B words, 213 words/article TIPSTER
0.9M articles, 0.26B words, 291 words/article