Answer Mining by Combining Extraction Techniques with Abductive Reasoning Sanda Harabagiu, Dan...
-
Upload
anastasia-ross -
Category
Documents
-
view
216 -
download
1
description
Transcript of Answer Mining by Combining Extraction Techniques with Abductive Reasoning Sanda Harabagiu, Dan...
![Page 1: Answer Mining by Combining Extraction Techniques with Abductive Reasoning Sanda Harabagiu, Dan Moldovan, Christine Clark, Mitchell Bowden, Jown Williams.](https://reader035.fdocuments.in/reader035/viewer/2022070605/5a4d1acb7f8b9ab05996f9b5/html5/thumbnails/1.jpg)
Answer Mining by Combining Extraction Techniques with Abductiv
e ReasoningSanda Harabagiu, Dan Moldovan, Christine Clark, Mitchell Bowden, Jown Williams and Jer
emy BensleyLCC
TREC 2003 Question Answering Track
![Page 2: Answer Mining by Combining Extraction Techniques with Abductive Reasoning Sanda Harabagiu, Dan Moldovan, Christine Clark, Mitchell Bowden, Jown Williams.](https://reader035.fdocuments.in/reader035/viewer/2022070605/5a4d1acb7f8b9ab05996f9b5/html5/thumbnails/2.jpg)
Abstract• Information Extraction Technique:
– Axiomatic knowledge derived from WordNet for justifying answers extracted from the AQUAINT text collection
• CICERO LITE:– Named entity recognizer– Recognize precisely a large set of entities that ranged
over an extended set of semantic categories• Theorem Prover:
– Produce abductive justifications of the answers when it had access to the axiomatic transformations of the WordNet glosses
![Page 3: Answer Mining by Combining Extraction Techniques with Abductive Reasoning Sanda Harabagiu, Dan Moldovan, Christine Clark, Mitchell Bowden, Jown Williams.](https://reader035.fdocuments.in/reader035/viewer/2022070605/5a4d1acb7f8b9ab05996f9b5/html5/thumbnails/3.jpg)
Introduction
• TREC-2003: Main task & Passage task• Main task:
– Factoids– Lists– Definitions
• Main_task_score = ½ * factoid_score + ¼ * list_score + ¼ *definition_score
![Page 4: Answer Mining by Combining Extraction Techniques with Abductive Reasoning Sanda Harabagiu, Dan Moldovan, Christine Clark, Mitchell Bowden, Jown Williams.](https://reader035.fdocuments.in/reader035/viewer/2022070605/5a4d1acb7f8b9ab05996f9b5/html5/thumbnails/4.jpg)
![Page 5: Answer Mining by Combining Extraction Techniques with Abductive Reasoning Sanda Harabagiu, Dan Moldovan, Christine Clark, Mitchell Bowden, Jown Williams.](https://reader035.fdocuments.in/reader035/viewer/2022070605/5a4d1acb7f8b9ab05996f9b5/html5/thumbnails/5.jpg)
• Factoid questions:– Seek short, fact-based answers– Ex. ”What are pennies made of?”
![Page 6: Answer Mining by Combining Extraction Techniques with Abductive Reasoning Sanda Harabagiu, Dan Moldovan, Christine Clark, Mitchell Bowden, Jown Williams.](https://reader035.fdocuments.in/reader035/viewer/2022070605/5a4d1acb7f8b9ab05996f9b5/html5/thumbnails/6.jpg)
• List questions:– Requests a set of instances of specified types– Ex. “What grapes are used in making wine?”– Final answer set was created form the
participants & assessors– IR = #instances judged correct and distinct /
#answers in the final set– IP = #instances judged correct and distinct /
#instances returned– F = (2 * IP * IR) / (IP + IR)
![Page 7: Answer Mining by Combining Extraction Techniques with Abductive Reasoning Sanda Harabagiu, Dan Moldovan, Christine Clark, Mitchell Bowden, Jown Williams.](https://reader035.fdocuments.in/reader035/viewer/2022070605/5a4d1acb7f8b9ab05996f9b5/html5/thumbnails/7.jpg)
• Definition questions:– Assessor created a list of acceptable info
nuggets, some of which are deemed essential– NR (Nugget Recall) = #essential nuggets
returned in response / #essential nuggets– NP (Nugget Precision)
• Allowance = 100 * #essential and acceptable nuggets returned
• Length = total #non-white space characters in answer strings
![Page 8: Answer Mining by Combining Extraction Techniques with Abductive Reasoning Sanda Harabagiu, Dan Moldovan, Christine Clark, Mitchell Bowden, Jown Williams.](https://reader035.fdocuments.in/reader035/viewer/2022070605/5a4d1acb7f8b9ab05996f9b5/html5/thumbnails/8.jpg)
• Definition questions:– NP = 1, if length < allowance– NP = 1 – (length – allowance) / length,
otherwise– F = (26 * NP * NR) / (25 * NP + NR)
![Page 9: Answer Mining by Combining Extraction Techniques with Abductive Reasoning Sanda Harabagiu, Dan Moldovan, Christine Clark, Mitchell Bowden, Jown Williams.](https://reader035.fdocuments.in/reader035/viewer/2022070605/5a4d1acb7f8b9ab05996f9b5/html5/thumbnails/9.jpg)
• TREC-2003:– Factoids: 413– Lists: 37– Definition: 50
Answer Type Count
Answers to Factoid 383
NIL-answers to Factoid 30
Answer instances in List final set 549
Essential nuggets for Definition 207
Total nuggets for Definition 417
![Page 10: Answer Mining by Combining Extraction Techniques with Abductive Reasoning Sanda Harabagiu, Dan Moldovan, Christine Clark, Mitchell Bowden, Jown Williams.](https://reader035.fdocuments.in/reader035/viewer/2022070605/5a4d1acb7f8b9ab05996f9b5/html5/thumbnails/10.jpg)
The Architecture of the QA System
![Page 11: Answer Mining by Combining Extraction Techniques with Abductive Reasoning Sanda Harabagiu, Dan Moldovan, Christine Clark, Mitchell Bowden, Jown Williams.](https://reader035.fdocuments.in/reader035/viewer/2022070605/5a4d1acb7f8b9ab05996f9b5/html5/thumbnails/11.jpg)
Question Processing
• Factoid or List questions:– Identify the expected answer type encoded as
• Semantic class recognized by CICERO LITE or• In a hierarchy of semantic concepts using the Wor
dNet hierarchies for verbs and nouns– Ex. “What American revolutionary general turn
ed over West Point to the British?”• Expected answer type is PERSON due to the noun general found in the hierarchy of humans in WordNet
![Page 12: Answer Mining by Combining Extraction Techniques with Abductive Reasoning Sanda Harabagiu, Dan Moldovan, Christine Clark, Mitchell Bowden, Jown Williams.](https://reader035.fdocuments.in/reader035/viewer/2022070605/5a4d1acb7f8b9ab05996f9b5/html5/thumbnails/12.jpg)
• Definition questions:– Parsed for detecting the NPs and matched ag
ainst a set of patterns– Ex. “What is Iqra?”
• Matched against the pattern <What is Question-Phrase>
• Associated with the answer pattern <Question-Phrase, which means Answer-Phrase>
![Page 13: Answer Mining by Combining Extraction Techniques with Abductive Reasoning Sanda Harabagiu, Dan Moldovan, Christine Clark, Mitchell Bowden, Jown Williams.](https://reader035.fdocuments.in/reader035/viewer/2022070605/5a4d1acb7f8b9ab05996f9b5/html5/thumbnails/13.jpg)
Document Processing
• Retrieve relevant passages based on the keywords provided by question processing
• Factoid questions:– Ranks the candidate passages
• List questions:– Ranks better passages having multiple occurrences
of concepts of the expected answer type• Definition questions:
– Allows multiple matches of keywords
![Page 14: Answer Mining by Combining Extraction Techniques with Abductive Reasoning Sanda Harabagiu, Dan Moldovan, Christine Clark, Mitchell Bowden, Jown Williams.](https://reader035.fdocuments.in/reader035/viewer/2022070605/5a4d1acb7f8b9ab05996f9b5/html5/thumbnails/14.jpg)
Answer Extraction
• Factoid:– Answers first extracted based on the answer p
hrase provided by CICERO LITE– If the answer is not a named entity, it is justifie
d abductively by using a theorem prover that makes user of axioms derived form WordNet
– Ex. “What apostle was crucified?”
![Page 15: Answer Mining by Combining Extraction Techniques with Abductive Reasoning Sanda Harabagiu, Dan Moldovan, Christine Clark, Mitchell Bowden, Jown Williams.](https://reader035.fdocuments.in/reader035/viewer/2022070605/5a4d1acb7f8b9ab05996f9b5/html5/thumbnails/15.jpg)
• List:– Extracted by using the ranked set of extracted
questions– Then determining a cutoff measure based on
the semantic similarity of answers
![Page 16: Answer Mining by Combining Extraction Techniques with Abductive Reasoning Sanda Harabagiu, Dan Moldovan, Christine Clark, Mitchell Bowden, Jown Williams.](https://reader035.fdocuments.in/reader035/viewer/2022070605/5a4d1acb7f8b9ab05996f9b5/html5/thumbnails/16.jpg)
• Definition– Relies on pattern matching
![Page 17: Answer Mining by Combining Extraction Techniques with Abductive Reasoning Sanda Harabagiu, Dan Moldovan, Christine Clark, Mitchell Bowden, Jown Williams.](https://reader035.fdocuments.in/reader035/viewer/2022070605/5a4d1acb7f8b9ab05996f9b5/html5/thumbnails/17.jpg)
Extracting Answers for Factoid Questions
• 289 correct answers– 234: identified by the CICERO LITE or recogni
zing it from the Answer Type Hierarchy– 65: due to theorem prover reported in Moldov
an et al. 2003• The role of theorem prover is to boost the precision
by filtering out incorrect answers that are not supported by an abductive justification
![Page 18: Answer Mining by Combining Extraction Techniques with Abductive Reasoning Sanda Harabagiu, Dan Moldovan, Christine Clark, Mitchell Bowden, Jown Williams.](https://reader035.fdocuments.in/reader035/viewer/2022070605/5a4d1acb7f8b9ab05996f9b5/html5/thumbnails/18.jpg)
• Ex. “what country does Greenland belong to?”– Answered by “Greenland, which is a territory o
f Denmark”– The gloss of the synset of {territory, dominion,
province} is “a territorial possession controlled by a ruling state”
![Page 19: Answer Mining by Combining Extraction Techniques with Abductive Reasoning Sanda Harabagiu, Dan Moldovan, Christine Clark, Mitchell Bowden, Jown Williams.](https://reader035.fdocuments.in/reader035/viewer/2022070605/5a4d1acb7f8b9ab05996f9b5/html5/thumbnails/19.jpg)
• Ex. “what country does Greenland belong to?”– The logical transformation for this gloss:
• control:v#1(e,x1,x2) & country:n#1(x1) & ruling:a#1(x1) & possession:n#2(x2) & territorial:a#1(x2)
– Refined expression:• process:v#2(e,x1,x2) & COUNTRY:n#1(x1) &
ruling:a#1(x1) & territory:n#2(x2)
![Page 20: Answer Mining by Combining Extraction Techniques with Abductive Reasoning Sanda Harabagiu, Dan Moldovan, Christine Clark, Mitchell Bowden, Jown Williams.](https://reader035.fdocuments.in/reader035/viewer/2022070605/5a4d1acb7f8b9ab05996f9b5/html5/thumbnails/20.jpg)
Extracting Answers for Definition Questions
• 50 definition questions evaluated• 207 essential nuggets• 417 total nuggets• 485 answers extracted by this system
– Two runs: Exact answers & Corresponding sentence-type answers
– Vital matches: 68(exact) & 86(sentence) form 207– 110(exact ) & 114(sentence) from final set 417
![Page 21: Answer Mining by Combining Extraction Techniques with Abductive Reasoning Sanda Harabagiu, Dan Moldovan, Christine Clark, Mitchell Bowden, Jown Williams.](https://reader035.fdocuments.in/reader035/viewer/2022070605/5a4d1acb7f8b9ab05996f9b5/html5/thumbnails/21.jpg)
• 38 patterns– 23 patterns had at least a match for the tested
questions
![Page 22: Answer Mining by Combining Extraction Techniques with Abductive Reasoning Sanda Harabagiu, Dan Moldovan, Christine Clark, Mitchell Bowden, Jown Williams.](https://reader035.fdocuments.in/reader035/viewer/2022070605/5a4d1acb7f8b9ab05996f9b5/html5/thumbnails/22.jpg)
![Page 23: Answer Mining by Combining Extraction Techniques with Abductive Reasoning Sanda Harabagiu, Dan Moldovan, Christine Clark, Mitchell Bowden, Jown Williams.](https://reader035.fdocuments.in/reader035/viewer/2022070605/5a4d1acb7f8b9ab05996f9b5/html5/thumbnails/23.jpg)
Extracting Answers for List Questions
• 37 list questions• A threshold-based cutoff of the answers
extracted• Decided on the threshold value by using
concept similarities between candidate answers
![Page 24: Answer Mining by Combining Extraction Techniques with Abductive Reasoning Sanda Harabagiu, Dan Moldovan, Christine Clark, Mitchell Bowden, Jown Williams.](https://reader035.fdocuments.in/reader035/viewer/2022070605/5a4d1acb7f8b9ab05996f9b5/html5/thumbnails/24.jpg)
• Given N list answers– First computes the
similarity between the first and the last answer
– Similarity of a pair of answers
– Consider a window of three noun or verb concepts to the left and to the right of the exact answer
![Page 25: Answer Mining by Combining Extraction Techniques with Abductive Reasoning Sanda Harabagiu, Dan Moldovan, Christine Clark, Mitchell Bowden, Jown Williams.](https://reader035.fdocuments.in/reader035/viewer/2022070605/5a4d1acb7f8b9ab05996f9b5/html5/thumbnails/25.jpg)
• Given N list answers:– Separate the concepts in nouns and verbs obtaining
– Similarity formula:
![Page 26: Answer Mining by Combining Extraction Techniques with Abductive Reasoning Sanda Harabagiu, Dan Moldovan, Christine Clark, Mitchell Bowden, Jown Williams.](https://reader035.fdocuments.in/reader035/viewer/2022070605/5a4d1acb7f8b9ab05996f9b5/html5/thumbnails/26.jpg)
• Given N list answers:
![Page 27: Answer Mining by Combining Extraction Techniques with Abductive Reasoning Sanda Harabagiu, Dan Moldovan, Christine Clark, Mitchell Bowden, Jown Williams.](https://reader035.fdocuments.in/reader035/viewer/2022070605/5a4d1acb7f8b9ab05996f9b5/html5/thumbnails/27.jpg)
• Given N list answers:
![Page 28: Answer Mining by Combining Extraction Techniques with Abductive Reasoning Sanda Harabagiu, Dan Moldovan, Christine Clark, Mitchell Bowden, Jown Williams.](https://reader035.fdocuments.in/reader035/viewer/2022070605/5a4d1acb7f8b9ab05996f9b5/html5/thumbnails/28.jpg)
• Given N list answers:
![Page 29: Answer Mining by Combining Extraction Techniques with Abductive Reasoning Sanda Harabagiu, Dan Moldovan, Christine Clark, Mitchell Bowden, Jown Williams.](https://reader035.fdocuments.in/reader035/viewer/2022070605/5a4d1acb7f8b9ab05996f9b5/html5/thumbnails/29.jpg)
Performance Evaluation
• Two different runs:– Exact answers & whole sentence containing the
answer
![Page 30: Answer Mining by Combining Extraction Techniques with Abductive Reasoning Sanda Harabagiu, Dan Moldovan, Christine Clark, Mitchell Bowden, Jown Williams.](https://reader035.fdocuments.in/reader035/viewer/2022070605/5a4d1acb7f8b9ab05996f9b5/html5/thumbnails/30.jpg)
Conclusion
• Second submission was slightly higher than first submission
• Definition question gets higher score:– An entire sentence allowed more vital nuggets to be
identified by the assessors• Factoid questions in the main task were slightly
better than in the passage task– Passage might have contained multiple concepts
similar to the answer, and thus produced a more vague evaluation context