Day 14 Information Retrieval Question Answering 1.
-
Upload
gertrude-turner -
Category
Documents
-
view
216 -
download
0
Transcript of Day 14 Information Retrieval Question Answering 1.
1
Day 14
Information RetrievalQuestion Answering
2
TREC
• TREC – Text REtrieval Conference• Administered by the National Institute of
Standards (NIST)• Annual competition held annually since 1992.• First conference included leading text retrieval
groups at UMass, City University London, Cornell, and a smattering of industry groups.
3
The Aquaint Corpus
• The corpus used by TREC from which answers are drawn.
• LDC2002T31 (on patas)– Newswire from three sources:
• Xinhua News Service (People's Republic of China) • New York Times News Service• Associated Press Worldstream News Service
– Not current: years 1996-2000 for Xinhua, 1998-2000 for NYT and AP
– For TREC competition: assumed current
4
TREC QA track
• Three types of questions in TREC QA track:– Factoid– List– Other
• All clustered into topics
5
TREC Question File
6
Question Answering (QA)
• Uses IR and IE techniques (and more…)• Questions posed in Natural Language
– Who was Genghis Khan?– What songs did Barry Manilow compose?– What countries fly the F-16?– When was James Dean born?– What does Park Jae-sang sing?
• Answers retrieved from a collection of documents (or a database, or the Web)
7
Park Jae-sang
8
Park Jae-sang
9
Designing a QA System
• Start with a question:– Who won the Nobel Peace Prize in 1991?
• Assume you have a search engine API at your disposal
• Need to return the answer:– Aung San Suu Kyi– Aung San Suu Kyi won the Nobel Peace Prize in 1991
• What do you do?
10
Designing a QA System
• Assume:– The search engine returns
• Snippets, and,• Documents
11
Designing a QA System
12
Designing a QA System
• Assume:– The search engine returns
• Snippets, and,• Documents
– Documents • Are in English• Contain passages of interest• Not all documents will have the answer
13
Designing a QA System
But many foreign investors remain sceptical, and western governments are withholding aid because of the Slorc's dismal human rights record and the continued detention of Ms Aung San Suu Kyi, the opposition leader who won the Nobel Peace Prize in 1991.
The military junta took power in 1988 as pro-democracy demonstrations were sweeping the country. It held elections in 1990, but has ignored their result. It has kept the 1991 Nobel peace prize winner, Aung San Suu Kyi - leader of the opposition party which won a landslide victory in the poll - under house arrest since July 1989.
The regime, which is also engaged in a battle with insurgents near its eastern border with Thailand, ignored a 1990 election victory by an opposition party and is detaining its leader, Ms Aung San Suu Kyi, who was awarded the 1991 Nobel Peace Prize. According to the British Red Cross, 5,000 or more refugees, mainly the elderly and women and children, are crossing into Bangladesh each day.
Who won the Nobel Peace Prize in 1991?
14
Designing a QA System
• Assume:– The search engine returns
• Snippets, and,• Documents
– Documents • Are in English• Contain passages of interest• Not all documents will have the answer
– The sky’s the limit wrt tools, resources, time, etc.
15
Designing a QA SystemWho won the Nobel Peace Prize in 1991?
16
Designing a QA System
But many foreign investors remain sceptical, and western governments are withholding aid because of the Slorc's dismal human rights record and the continued detention of Ms Aung San Suu Kyi, the opposition leader who won the Nobel Peace Prize in 1991.
The military junta took power in 1988 as pro-democracy demonstrations were sweeping the country. It held elections in 1990, but has ignored their result. It has kept the 1991 Nobel peace prize winner, Aung San Suu Kyi - leader of the opposition party which won a landslide victory in the poll - under house arrest since July 1989.
The regime, which is also engaged in a battle with insurgents near its eastern border with Thailand, ignored a 1990 election victory by an opposition party and is detaining its leader, Ms Aung San Suu Kyi, who was awarded the 1991 Nobel Peace Prize. According to the British Red Cross, 5,000 or more refugees, mainly the elderly and women and children, are crossing into Bangladesh each day.
Who won the Nobel Peace Prize in 1991?
17
An Example
But many foreign investors remain sceptical, and western governments are withholding aid because of the Slorc's dismal human rights record and the continued detention of Ms Aung San Suu Kyi, the opposition leader who won the Nobel Peace Prize in 1991.
The military junta took power in 1988 as pro-democracy demonstrations were sweeping the country. It held elections in 1990, but has ignored their result. It has kept the 1991 Nobel peace prize winner, Aung San Suu Kyi - leader of the opposition party which won a landslide victory in the poll - under house arrest since July 1989.
The regime, which is also engaged in a battle with insurgents near its eastern border with Thailand, ignored a 1990 election victory by an opposition party and is detaining its leader, Ms Aung San Suu Kyi, who was awarded the 1991 Nobel Peace Prize. According to the British Red Cross, 5,000 or more refugees, mainly the elderly and women and children, are crossing into Bangladesh each day.
Who won the Nobel Peace Prize in 1991?
18
Question Answering (QA)
• For a QA system to work, we need to– Find documents that may contain the answer
• Form search engine query from original question
– Find passages within the documents that may contain the answer
• What is “type” of answer?• Determine what kind of answer is expected (query
classification)
– Extract the answer from the relevant passage(s)• Repeated occurrences may reinforce
– Return the answer
A Generic QA Framework
DocumentCollection
Search EngineTop n
documents
DocumentProcessing
Questions Questions
Answers
• Passage extractor needed too
20
The UWCLMAQA System
21
Steps for the UWCLMAQA System
• Query Analysis• Query Processing (some additional steps)
• Document Selection• Passage Extraction & Ranking• Answer Extraction
• “Unit” evaluation done at each step
22
Query Analysis
• Grouped questions into types • Purpose: Determine what the answer will look like• Categorized by enhanced UIUC scheme:
– Abbreviation– Description– Entity– Human– Location – Country, State, City– Numeric – Date, Measure
UIUC: http://l2r.cs.uiuc.edu/~cogcomp/Data/QA/QC/definition.html
23
Alternative Strategy: Query Analysis and Rewrite
• Intuition: The user’s question is often syntactically quite close to sentences that contain the answer
– Where is the Louvre Museum located? • The Louvre Museum is located in Paris
– Who created the character of Scrooge?• Charles Dickens created the character of Scrooge.
24
Alternative Strategy:Query Analysis and Rewrite
• Hand-craft category-specific transformation rules
e.g.:
“Where is the Louvre Museum located?” “is the Louvre Museum located” “the is Louvre Museum located” “the Louvre is Museum located” “the Louvre Museum is located” “the Louvre Museum located is”
• Search for all permutations
25
Query Processing
• Basic process:– Extracted question– Appended topic– “Web boosted” query– Threw against Lucene
26
Query Processing
• Web boosting strategy– Supplied question and topic to Google API– Results were
• Stop-worded, query terms removed• Ranked by frequency
– 5 most frequent terms added to Lucene query
27
Document Selection
• Lucene returned top 1,000 documents• Took top 3 for Factoid, Top 25 for List• (Hook for reranking provided, but not implemented.)• Our doc retrieval performance for 2005 Qs:
– F-measure - .3517 n=3, .3620 n=1– Mean 2005: .2958– Max (LCC) 2005: .7920
28
Passage Extraction & Ranking
• From top documents, extracted relevant paragraphs
• Paragraphs ranked by tf/idf:– tf = 1+log(word frequency in paragraph)– idf = log(total doc count/# docs containing word)– total doc count = # docs by day by news source– tf/idf score normalized by paragraph length
29
Passage Ranking
• tf/idf multiplied by count of query terms in paragraph (giving them more weight)
• 10 paragraphs returned for factoids• 45 paragraphs returned for lists
30
Passage Extraction & Ranking
But many foreign investors remain sceptical, and western governments are withholding aid because of the Slorc's dismal human rights record and the continued detention of Ms Aung San Suu Kyi, the opposition leader who won the Nobel Peace Prize in 1991.
The military junta took power in 1988 as pro-democracy demonstrations were sweeping the country. It held elections in 1990, but has ignored their result. It has kept the 1991 Nobel peace prize winner, Aung San Suu Kyi - leader of the opposition party which won a landslide victory in the poll - under house arrest since July 1989.
The regime, which is also engaged in a battle with insurgents near its eastern border with Thailand, ignored a 1990 election victory by an opposition party and is detaining its leader, Ms Aung San Suu Kyi, who was awarded the 1991 Nobel Peace Prize. According to the British Red Cross, 5,000 or more refugees, mainly the elderly and women and children, are crossing into Bangladesh each day.
Who won the Nobel Peace Prize in 1991?
31
Answer Extraction
• Most factoids need NP answer (e.g., most are NEs, such as countries, cities, dates, people’s names, company names, …)– All NPs considered as possible answers
• For passages– Used Lingua::Stem to find sentences (sentence breaking)– POS tagged (Stanford POS Tagger)– Chunked using the fnTBL Chunker (ID NP-chunks)
• Prior query classification used to identify kind of NP answer expected
• Some other heuristics (e.g., most likely place NP would occur)
32
An Example
But many foreign investors remain sceptical, and western governments are withholding aid because of the Slorc's dismal human rights record and the continued detention of Ms Aung San Suu Kyi, the opposition leader who won the Nobel Peace Prize in 1991.
The military junta took power in 1988 as pro-democracy demonstrations were sweeping the country. It held elections in 1990, but has ignored their result. It has kept the 1991 Nobel peace prize winner, Aung San Suu Kyi - leader of the opposition party which won a landslide victory in the poll - under house arrest since July 1989.
The regime, which is also engaged in a battle with insurgents near its eastern border with Thailand, ignored a 1990 election victory by an opposition party and is detaining its leader, Ms Aung San Suu Kyi, who was awarded the 1991 Nobel Peace Prize. According to the British Red Cross, 5,000 or more refugees, mainly the elderly and women and children, are crossing into Bangladesh each day.
Who won the Nobel Peace Prize in 1991?
33
Answer Extraction
• For lists:– Question topic appeared the most important– Heavily weighted topic terms for Lucene– Similar process to Factoids (tagging, chunking) for
finding answers– Cut-off determined by 2005 data
34
Answer Extraction
• For others:– Anything left over that might be answer bearing– Top 15 returned
35
How’d we do?
• Before answering the question:– Mean Reciprocal Rank (MRR)
36
Mean Reciprocal Rank (MRR)
• Assumes: test set of questions with human-labeled answers
• Assumes: system returns short ranked list of answers or passages with answers
• Answers scored with the sum of the reciprocal rank of the correct answers over total returned answers
(for N questions)
37
How’d we do?
• Factoid– UWCLMAQA: .112 and .109– Median: .186, Best: .578, Worst: .040
• List– UWCLMAQA: .051 and .046– Median: .087, Best: .433, Worst: .000
• Other– UWCLMAQA: .164 and .153– Median: .125, Best: .250, Worst: .000
38
39
40
Full List of Tools Used• SGML::Parser::OpenSP
http://search.cpan.org/~bjoern/SGML-Parser-OpenSP-0.98/• OpenSP
http://openjade.sourceforge.net/• UIUC Question Classification
http://l2r.cs.uiuc.edu/~cogcomp/Data/QA/QC/• Lucene
http://lucene.apache.org/java/docs/index.html• SAX (Simple API for XML)
http://www.saxproject.org/• Maxent Toolkit
http://homepages.inf.ed.ac.uk/s0450736/maxent_toolkit.html• PyGoogle
http://pygoogle.sourceforge.net/• SOAPy
http://soapy.sourceforge.net/• Google API
http://www.google.com/apis/• Lingua::Stem
http://search.cpan.org/~snowhare/Lingua-Stem-0.82/lib/Lingua/Stem/En.pm• Lingua::Sentence
http://search.cpan.org/~shlomoy/Lingua-EN-Sentence-0.25/lib/Lingua/EN/Sentence.pm• Stanford POS Tagger
http://nlp.stanford.edu/software/tagger.shtml• fnTBL Chunker
http://nlp.cs.jhu.edu/~rflorian/fntbl/• Lingpipe
http://www.alias-i.com/lingpipe/• LevenshteinXS.pm
http://search.cpan.org/~jgoldberg/Text-LevenshteinXS-0.03/