Question Answering - Cleveland State...
Transcript of Question Answering - Cleveland State...
![Page 1: Question Answering - Cleveland State Universitycis.csuohio.edu/~sschung/CIS660/QALectureNotesStanford.pdf · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms](https://reader033.fdocuments.in/reader033/viewer/2022042222/5ec905a2ae99265e9e4c1fa9/html5/thumbnails/1.jpg)
Question Answering
What is Ques+on Answering?
![Page 2: Question Answering - Cleveland State Universitycis.csuohio.edu/~sschung/CIS660/QALectureNotesStanford.pdf · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms](https://reader033.fdocuments.in/reader033/viewer/2022042222/5ec905a2ae99265e9e4c1fa9/html5/thumbnails/2.jpg)
Dan Jurafsky
2
Ques%on Answering
What do worms eat?
worms
eat
what
worms
eat
grass
Worms eat grass
worms
eat
grass
Grass is eaten by wormsbirds
eat
worms
Birds eat worms
horses
eat
grass
Horses with worms eat grass
with
worms
!"#$%&'( )&*#'%+,-.'$/#0$(
One of the oldest NLP tasks (punched card systems in 1961) Simmons, Klein, McConlogue. 1964. Indexing and Dependency Logic for Answering English Ques+ons. American Documenta+on 15:30, 196-‐204
![Page 3: Question Answering - Cleveland State Universitycis.csuohio.edu/~sschung/CIS660/QALectureNotesStanford.pdf · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms](https://reader033.fdocuments.in/reader033/viewer/2022042222/5ec905a2ae99265e9e4c1fa9/html5/thumbnails/3.jpg)
Dan Jurafsky
Ques%on Answering: IBM’s Watson • Won Jeopardy on February 16, 2011!
3
WILLIAM WILKINSON’S “AN ACCOUNT OF THE PRINCIPALITIES OF
WALLACHIA AND MOLDOVIA” INSPIRED THIS AUTHOR’S
MOST FAMOUS NOVEL
Bram Stoker
![Page 4: Question Answering - Cleveland State Universitycis.csuohio.edu/~sschung/CIS660/QALectureNotesStanford.pdf · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms](https://reader033.fdocuments.in/reader033/viewer/2022042222/5ec905a2ae99265e9e4c1fa9/html5/thumbnails/4.jpg)
Dan Jurafsky
Apple’s Siri
4
![Page 5: Question Answering - Cleveland State Universitycis.csuohio.edu/~sschung/CIS660/QALectureNotesStanford.pdf · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms](https://reader033.fdocuments.in/reader033/viewer/2022042222/5ec905a2ae99265e9e4c1fa9/html5/thumbnails/5.jpg)
Dan Jurafsky
Wolfram Alpha
5
![Page 6: Question Answering - Cleveland State Universitycis.csuohio.edu/~sschung/CIS660/QALectureNotesStanford.pdf · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms](https://reader033.fdocuments.in/reader033/viewer/2022042222/5ec905a2ae99265e9e4c1fa9/html5/thumbnails/6.jpg)
Dan Jurafsky
6
Types of Ques%ons in Modern Systems
• Factoid ques+ons • Who wrote “The Universal Declara4on of Human Rights”? • How many calories are there in two slices of apple pie? • What is the average age of the onset of au4sm? • Where is Apple Computer based?
• Complex (narra+ve) ques+ons: • In children with an acute febrile illness, what is the efficacy of acetaminophen in reducing fever?
• What do scholars think about Jefferson’s posi4on on dealing with pirates?
![Page 7: Question Answering - Cleveland State Universitycis.csuohio.edu/~sschung/CIS660/QALectureNotesStanford.pdf · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms](https://reader033.fdocuments.in/reader033/viewer/2022042222/5ec905a2ae99265e9e4c1fa9/html5/thumbnails/7.jpg)
Dan Jurafsky
Commercial systems: mainly factoid ques%ons
Where is the Louvre Museum located? In Paris, France
What’s the abbrevia+on for limited partnership?
L.P.
What are the names of Odin’s ravens? Huginn and Muninn
What currency is used in China? The yuan
What kind of nuts are used in marzipan? almonds
What instrument does Max Roach play? drums
What is the telephone number for Stanford University?
650-‐723-‐2300
![Page 8: Question Answering - Cleveland State Universitycis.csuohio.edu/~sschung/CIS660/QALectureNotesStanford.pdf · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms](https://reader033.fdocuments.in/reader033/viewer/2022042222/5ec905a2ae99265e9e4c1fa9/html5/thumbnails/8.jpg)
Dan Jurafsky
Paradigms for QA
• IR-‐based approaches • TREC; IBM Watson; Google
• Knowledge-‐based and Hybrid approaches • IBM Watson; Apple Siri; Wolfram Alpha; True Knowledge Evi
8
![Page 9: Question Answering - Cleveland State Universitycis.csuohio.edu/~sschung/CIS660/QALectureNotesStanford.pdf · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms](https://reader033.fdocuments.in/reader033/viewer/2022042222/5ec905a2ae99265e9e4c1fa9/html5/thumbnails/9.jpg)
Dan Jurafsky
Many ques%ons can already be answered by web search
• a
9
![Page 10: Question Answering - Cleveland State Universitycis.csuohio.edu/~sschung/CIS660/QALectureNotesStanford.pdf · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms](https://reader033.fdocuments.in/reader033/viewer/2022042222/5ec905a2ae99265e9e4c1fa9/html5/thumbnails/10.jpg)
Dan Jurafsky
IR-‐based Ques%on Answering
• a
10
![Page 11: Question Answering - Cleveland State Universitycis.csuohio.edu/~sschung/CIS660/QALectureNotesStanford.pdf · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms](https://reader033.fdocuments.in/reader033/viewer/2022042222/5ec905a2ae99265e9e4c1fa9/html5/thumbnails/11.jpg)
Dan Jurafsky
11
IR-‐based Factoid QA
DocumentDocumentDocument
DocumentDocume
ntDocumentDocume
ntDocument
Question Processing
PassageRetrieval
Query Formulation
Answer Type Detection
Question
Passage Retrieval
Document Retrieval
Answer Processing
Answer
passages
Indexing
RelevantDocs
DocumentDocumentDocument
![Page 12: Question Answering - Cleveland State Universitycis.csuohio.edu/~sschung/CIS660/QALectureNotesStanford.pdf · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms](https://reader033.fdocuments.in/reader033/viewer/2022042222/5ec905a2ae99265e9e4c1fa9/html5/thumbnails/12.jpg)
Dan Jurafsky
IR-‐based Factoid QA • QUESTION PROCESSING
• Detect ques+on type, answer type, focus, rela+ons • Formulate queries to send to a search engine
• PASSAGE RETRIEVAL • Retrieve ranked documents • Break into suitable passages and rerank
• ANSWER PROCESSING • Extract candidate answers • Rank candidates
• using evidence from the text and external sources
![Page 13: Question Answering - Cleveland State Universitycis.csuohio.edu/~sschung/CIS660/QALectureNotesStanford.pdf · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms](https://reader033.fdocuments.in/reader033/viewer/2022042222/5ec905a2ae99265e9e4c1fa9/html5/thumbnails/13.jpg)
Dan Jurafsky
Knowledge-‐based approaches (Siri)
• Build a seman+c representa+on of the query • Times, dates, loca+ons, en++es, numeric quan++es
• Map from this seman+cs to query structured data or resources • Geospa+al databases • Ontologies (Wikipedia infoboxes, dbPedia, WordNet, Yago) • Restaurant review sources and reserva+on services • Scien+fic databases
13
![Page 14: Question Answering - Cleveland State Universitycis.csuohio.edu/~sschung/CIS660/QALectureNotesStanford.pdf · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms](https://reader033.fdocuments.in/reader033/viewer/2022042222/5ec905a2ae99265e9e4c1fa9/html5/thumbnails/14.jpg)
Dan Jurafsky
Hybrid approaches (IBM Watson)
• Build a shallow seman+c representa+on of the query • Generate answer candidates using IR methods
• Augmented with ontologies and semi-‐structured data
• Score each candidate using richer knowledge sources • Geospa+al databases • Temporal reasoning • Taxonomical classifica+on
14
![Page 15: Question Answering - Cleveland State Universitycis.csuohio.edu/~sschung/CIS660/QALectureNotesStanford.pdf · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms](https://reader033.fdocuments.in/reader033/viewer/2022042222/5ec905a2ae99265e9e4c1fa9/html5/thumbnails/15.jpg)
Question Answering
What is Ques+on Answering?
![Page 16: Question Answering - Cleveland State Universitycis.csuohio.edu/~sschung/CIS660/QALectureNotesStanford.pdf · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms](https://reader033.fdocuments.in/reader033/viewer/2022042222/5ec905a2ae99265e9e4c1fa9/html5/thumbnails/16.jpg)
Question Answering
Answer Types and Query Formula+on
![Page 17: Question Answering - Cleveland State Universitycis.csuohio.edu/~sschung/CIS660/QALectureNotesStanford.pdf · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms](https://reader033.fdocuments.in/reader033/viewer/2022042222/5ec905a2ae99265e9e4c1fa9/html5/thumbnails/17.jpg)
Dan Jurafsky
Factoid Q/A
17
DocumentDocumentDocument
DocumentDocume
ntDocumentDocume
ntDocument
Question Processing
PassageRetrieval
Query Formulation
Answer Type Detection
Question
Passage Retrieval
Document Retrieval
Answer Processing
Answer
passages
Indexing
RelevantDocs
DocumentDocumentDocument
![Page 18: Question Answering - Cleveland State Universitycis.csuohio.edu/~sschung/CIS660/QALectureNotesStanford.pdf · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms](https://reader033.fdocuments.in/reader033/viewer/2022042222/5ec905a2ae99265e9e4c1fa9/html5/thumbnails/18.jpg)
Dan Jurafsky
Ques%on Processing Things to extract from the ques%on
• Answer Type Detec+on • Decide the named en%ty type (person, place) of the answer
• Query Formula+on • Choose query keywords for the IR system
• Ques+on Type classifica+on • Is this a defini+on ques+on, a math ques+on, a list ques+on?
• Focus Detec+on • Find the ques+on words that are replaced by the answer
• Rela+on Extrac+on • Find rela+ons between en++es in the ques+on 18
![Page 19: Question Answering - Cleveland State Universitycis.csuohio.edu/~sschung/CIS660/QALectureNotesStanford.pdf · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms](https://reader033.fdocuments.in/reader033/viewer/2022042222/5ec905a2ae99265e9e4c1fa9/html5/thumbnails/19.jpg)
Dan Jurafsky
Question Processing They’re the two states you could be reentering if you’re crossing Florida’s northern border
• Answer Type: US state • Query: two states, border, Florida, north • Focus: the two states • Rela+ons: borders(Florida, ?x, north)
19
![Page 20: Question Answering - Cleveland State Universitycis.csuohio.edu/~sschung/CIS660/QALectureNotesStanford.pdf · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms](https://reader033.fdocuments.in/reader033/viewer/2022042222/5ec905a2ae99265e9e4c1fa9/html5/thumbnails/20.jpg)
Dan Jurafsky
Answer Type Detec%on: Named En%%es
• Who founded Virgin Airlines? • PERSON
• What Canadian city has the largest popula4on? • CITY.
![Page 21: Question Answering - Cleveland State Universitycis.csuohio.edu/~sschung/CIS660/QALectureNotesStanford.pdf · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms](https://reader033.fdocuments.in/reader033/viewer/2022042222/5ec905a2ae99265e9e4c1fa9/html5/thumbnails/21.jpg)
Dan Jurafsky
Answer Type Taxonomy
• 6 coarse classes • ABBEVIATION, ENTITY, DESCRIPTION, HUMAN, LOCATION, NUMERIC
• 50 finer classes • LOCATION: city, country, mountain… • HUMAN: group, individual, +tle, descrip+on • ENTITY: animal, body, color, currency…
21
Xin Li, Dan Roth. 2002. Learning Ques+on Classifiers. COLING'02
![Page 22: Question Answering - Cleveland State Universitycis.csuohio.edu/~sschung/CIS660/QALectureNotesStanford.pdf · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms](https://reader033.fdocuments.in/reader033/viewer/2022042222/5ec905a2ae99265e9e4c1fa9/html5/thumbnails/22.jpg)
Dan Jurafsky
22
Part of Li & Roth’s Answer Type Taxonomy
LOCATION
NUMERIC
ENTITY HUMAN
ABBREVIATIONDESCRIPTION
country city state
datepercent
money
sizedistance
individual
title
group
food
currency
animal
definition
reason expression
abbreviation
![Page 23: Question Answering - Cleveland State Universitycis.csuohio.edu/~sschung/CIS660/QALectureNotesStanford.pdf · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms](https://reader033.fdocuments.in/reader033/viewer/2022042222/5ec905a2ae99265e9e4c1fa9/html5/thumbnails/23.jpg)
Dan Jurafsky
23
Answer Types
![Page 24: Question Answering - Cleveland State Universitycis.csuohio.edu/~sschung/CIS660/QALectureNotesStanford.pdf · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms](https://reader033.fdocuments.in/reader033/viewer/2022042222/5ec905a2ae99265e9e4c1fa9/html5/thumbnails/24.jpg)
Dan Jurafsky
24
More Answer Types
![Page 25: Question Answering - Cleveland State Universitycis.csuohio.edu/~sschung/CIS660/QALectureNotesStanford.pdf · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms](https://reader033.fdocuments.in/reader033/viewer/2022042222/5ec905a2ae99265e9e4c1fa9/html5/thumbnails/25.jpg)
Dan Jurafsky
Answer types in Jeopardy
• 2500 answer types in 20,000 Jeopardy ques+on sample • The most frequent 200 answer types cover < 50% of data • The 40 most frequent Jeopardy answer types he, country, city, man, film, state, she, author, group, here, company, president, capital, star, novel, character, woman, river, island, king, song, part, series, sport, singer, actor, play, team, show, actress, animal, presiden+al, composer, musical, na+on, book, +tle, leader, game
25
Ferrucci et al. 2010. Building Watson: An Overview of the DeepQA Project. AI Magazine. Fall 2010. 59-‐79.
![Page 26: Question Answering - Cleveland State Universitycis.csuohio.edu/~sschung/CIS660/QALectureNotesStanford.pdf · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms](https://reader033.fdocuments.in/reader033/viewer/2022042222/5ec905a2ae99265e9e4c1fa9/html5/thumbnails/26.jpg)
Dan Jurafsky
Answer Type Detec%on
• Hand-‐wrioen rules • Machine Learning • Hybrids
![Page 27: Question Answering - Cleveland State Universitycis.csuohio.edu/~sschung/CIS660/QALectureNotesStanford.pdf · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms](https://reader033.fdocuments.in/reader033/viewer/2022042222/5ec905a2ae99265e9e4c1fa9/html5/thumbnails/27.jpg)
Dan Jurafsky
Answer Type Detec%on
• Regular expression-‐based rules can get some cases: • Who {is|was|are|were} PERSON • PERSON (YEAR – YEAR)
• Other rules use the ques%on headword: (the headword of the first noun phrase ater the wh-‐word)
• Which city in China has the largest number of foreign financial companies?
• What is the state flower of California?
![Page 28: Question Answering - Cleveland State Universitycis.csuohio.edu/~sschung/CIS660/QALectureNotesStanford.pdf · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms](https://reader033.fdocuments.in/reader033/viewer/2022042222/5ec905a2ae99265e9e4c1fa9/html5/thumbnails/28.jpg)
Dan Jurafsky
Answer Type Detec%on
• Most oten, we treat the problem as machine learning classifica+on • Define a taxonomy of ques+on types • Annotate training data for each ques+on type • Train classifiers for each ques+on class using a rich set of features. • features include those hand-‐wrioen rules!
28
![Page 29: Question Answering - Cleveland State Universitycis.csuohio.edu/~sschung/CIS660/QALectureNotesStanford.pdf · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms](https://reader033.fdocuments.in/reader033/viewer/2022042222/5ec905a2ae99265e9e4c1fa9/html5/thumbnails/29.jpg)
Dan Jurafsky
Features for Answer Type Detec%on
• Ques+on words and phrases • Part-‐of-‐speech tags • Parse features (headwords) • Named En++es • Seman+cally related words
29
![Page 30: Question Answering - Cleveland State Universitycis.csuohio.edu/~sschung/CIS660/QALectureNotesStanford.pdf · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms](https://reader033.fdocuments.in/reader033/viewer/2022042222/5ec905a2ae99265e9e4c1fa9/html5/thumbnails/30.jpg)
Dan Jurafsky
Factoid Q/A
30
DocumentDocumentDocument
DocumentDocume
ntDocumentDocume
ntDocument
Question Processing
PassageRetrieval
Query Formulation
Answer Type Detection
Question
Passage Retrieval
Document Retrieval
Answer Processing
Answer
passages
Indexing
RelevantDocs
DocumentDocumentDocument
![Page 31: Question Answering - Cleveland State Universitycis.csuohio.edu/~sschung/CIS660/QALectureNotesStanford.pdf · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms](https://reader033.fdocuments.in/reader033/viewer/2022042222/5ec905a2ae99265e9e4c1fa9/html5/thumbnails/31.jpg)
Dan Jurafsky
Keyword Selec%on Algorithm
1. Select all non-‐stop words in quota+ons 2. Select all NNP words in recognized named en++es 3. Select all complex nominals with their adjec+val modifiers 4. Select all other complex nominals 5. Select all nouns with their adjec+val modifiers 6. Select all other nouns 7. Select all verbs 8. Select all adverbs 9. Select the QFW word (skipped in all previous steps) 10. Select all other words
Dan Moldovan, Sanda Harabagiu, Marius Paca, Rada Mihalcea, Richard Goodrum, Roxana Girju and Vasile Rus. 1999. Proceedings of TREC-‐8.
![Page 32: Question Answering - Cleveland State Universitycis.csuohio.edu/~sschung/CIS660/QALectureNotesStanford.pdf · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms](https://reader033.fdocuments.in/reader033/viewer/2022042222/5ec905a2ae99265e9e4c1fa9/html5/thumbnails/32.jpg)
Dan Jurafsky
Choosing keywords from the query
32
Who coined the term “cyberspace” in his novel “Neuromancer”?
1 1
4 4
7
cyberspace/1 Neuromancer/1 term/4 novel/4 coined/7
Slide from Mihai Surdeanu
![Page 33: Question Answering - Cleveland State Universitycis.csuohio.edu/~sschung/CIS660/QALectureNotesStanford.pdf · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms](https://reader033.fdocuments.in/reader033/viewer/2022042222/5ec905a2ae99265e9e4c1fa9/html5/thumbnails/33.jpg)
Question Answering
Answer Types and Query Formula+on
![Page 34: Question Answering - Cleveland State Universitycis.csuohio.edu/~sschung/CIS660/QALectureNotesStanford.pdf · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms](https://reader033.fdocuments.in/reader033/viewer/2022042222/5ec905a2ae99265e9e4c1fa9/html5/thumbnails/34.jpg)
Question Answering
Passage Retrieval and Answer Extrac+on
![Page 35: Question Answering - Cleveland State Universitycis.csuohio.edu/~sschung/CIS660/QALectureNotesStanford.pdf · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms](https://reader033.fdocuments.in/reader033/viewer/2022042222/5ec905a2ae99265e9e4c1fa9/html5/thumbnails/35.jpg)
Dan Jurafsky
Factoid Q/A
35
DocumentDocumentDocument
DocumentDocume
ntDocumentDocume
ntDocument
Question Processing
PassageRetrieval
Query Formulation
Answer Type Detection
Question
Passage Retrieval
Document Retrieval
Answer Processing
Answer
passages
Indexing
RelevantDocs
DocumentDocumentDocument
![Page 36: Question Answering - Cleveland State Universitycis.csuohio.edu/~sschung/CIS660/QALectureNotesStanford.pdf · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms](https://reader033.fdocuments.in/reader033/viewer/2022042222/5ec905a2ae99265e9e4c1fa9/html5/thumbnails/36.jpg)
Dan Jurafsky
36
Passage Retrieval
• Step 1: IR engine retrieves documents using query terms • Step 2: Segment the documents into shorter units
• something like paragraphs
• Step 3: Passage ranking • Use answer type to help rerank passages
![Page 37: Question Answering - Cleveland State Universitycis.csuohio.edu/~sschung/CIS660/QALectureNotesStanford.pdf · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms](https://reader033.fdocuments.in/reader033/viewer/2022042222/5ec905a2ae99265e9e4c1fa9/html5/thumbnails/37.jpg)
Dan Jurafsky
Features for Passage Ranking
• Number of Named En++es of the right type in passage • Number of query words in passage • Number of ques+on N-‐grams also in passage • Proximity of query keywords to each other in passage • Longest sequence of ques+on words • Rank of the document containing passage
Either in rule-‐based classifiers or with supervised machine learning
![Page 38: Question Answering - Cleveland State Universitycis.csuohio.edu/~sschung/CIS660/QALectureNotesStanford.pdf · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms](https://reader033.fdocuments.in/reader033/viewer/2022042222/5ec905a2ae99265e9e4c1fa9/html5/thumbnails/38.jpg)
Dan Jurafsky
Factoid Q/A
38
DocumentDocumentDocument
DocumentDocume
ntDocumentDocume
ntDocument
Question Processing
PassageRetrieval
Query Formulation
Answer Type Detection
Question
Passage Retrieval
Document Retrieval
Answer Processing
Answer
passages
Indexing
RelevantDocs
DocumentDocumentDocument
![Page 39: Question Answering - Cleveland State Universitycis.csuohio.edu/~sschung/CIS660/QALectureNotesStanford.pdf · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms](https://reader033.fdocuments.in/reader033/viewer/2022042222/5ec905a2ae99265e9e4c1fa9/html5/thumbnails/39.jpg)
Dan Jurafsky
Answer Extrac%on
• Run an answer-‐type named-‐en+ty tagger on the passages • Each answer type requires a named-‐en+ty tagger that detects it • If answer type is CITY, tagger has to tag CITY • Can be full NER, simple regular expressions, or hybrid
• Return the string with the right type: • Who is the prime minister of India (PERSON) Manmohan Singh, Prime Minister of India, had told left leaders that the deal would not be renegotiated.!
• How tall is Mt. Everest? (LENGTH) The official height of Mount Everest is 29035 feet!
![Page 40: Question Answering - Cleveland State Universitycis.csuohio.edu/~sschung/CIS660/QALectureNotesStanford.pdf · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms](https://reader033.fdocuments.in/reader033/viewer/2022042222/5ec905a2ae99265e9e4c1fa9/html5/thumbnails/40.jpg)
Dan Jurafsky
Ranking Candidate Answers
• But what if there are mul+ple candidate answers!
Q: Who was Queen Victoria’s second son?!• Answer Type: Person
• Passage: The Marie biscuit is named ater Marie Alexandrovna, the daughter of Czar Alexander II of Russia and wife of Alfred, the second son of Queen Victoria and Prince Albert
![Page 41: Question Answering - Cleveland State Universitycis.csuohio.edu/~sschung/CIS660/QALectureNotesStanford.pdf · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms](https://reader033.fdocuments.in/reader033/viewer/2022042222/5ec905a2ae99265e9e4c1fa9/html5/thumbnails/41.jpg)
Dan Jurafsky
Ranking Candidate Answers
• But what if there are mul+ple candidate answers!
Q: Who was Queen Victoria’s second son?!• Answer Type: Person
• Passage: The Marie biscuit is named ater Marie Alexandrovna, the daughter of Czar Alexander II of Russia and wife of Alfred, the second son of Queen Victoria and Prince Albert
![Page 42: Question Answering - Cleveland State Universitycis.csuohio.edu/~sschung/CIS660/QALectureNotesStanford.pdf · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms](https://reader033.fdocuments.in/reader033/viewer/2022042222/5ec905a2ae99265e9e4c1fa9/html5/thumbnails/42.jpg)
Dan Jurafsky
Use machine learning: Features for ranking candidate answers
Answer type match: Candidate contains a phrase with the correct answer type. PaZern match: Regular expression paoern matches the candidate. Ques%on keywords: # of ques+on keywords in the candidate. Keyword distance: Distance in words between the candidate and query keywords Novelty factor: A word in the candidate is not in the query. Apposi%on features: The candidate is an apposi+ve to ques+on terms Punctua%on loca%on: The candidate is immediately followed by a comma, period, quota+on marks, semicolon, or exclama+on mark. Sequences of ques%on terms: The length of the longest sequence of ques+on terms that occurs in the candidate answer.
![Page 43: Question Answering - Cleveland State Universitycis.csuohio.edu/~sschung/CIS660/QALectureNotesStanford.pdf · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms](https://reader033.fdocuments.in/reader033/viewer/2022042222/5ec905a2ae99265e9e4c1fa9/html5/thumbnails/43.jpg)
Dan Jurafsky
Candidate Answer scoring in IBM Watson
• Each candidate answer gets scores from >50 components • (from unstructured text, semi-‐structured text, triple stores)
• logical form (parse) match between ques+on and candidate • passage source reliability • geospa+al loca+on • California is ”southwest of Montana”
• temporal rela+onships • taxonomic classifica+on 43
![Page 44: Question Answering - Cleveland State Universitycis.csuohio.edu/~sschung/CIS660/QALectureNotesStanford.pdf · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms](https://reader033.fdocuments.in/reader033/viewer/2022042222/5ec905a2ae99265e9e4c1fa9/html5/thumbnails/44.jpg)
Dan Jurafsky
44
Common Evalua%on Metrics
1. Accuracy (does answer match gold-‐labeled answer?) 2. Mean Reciprocal Rank
• Return a ranked list of N candidate answers. • Score is 1/Rank of the first right answer.
MRR =
1rankii=1
N
!
N
![Page 45: Question Answering - Cleveland State Universitycis.csuohio.edu/~sschung/CIS660/QALectureNotesStanford.pdf · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms](https://reader033.fdocuments.in/reader033/viewer/2022042222/5ec905a2ae99265e9e4c1fa9/html5/thumbnails/45.jpg)
Question Answering
Passage Retrieval and Answer Extrac+on
![Page 46: Question Answering - Cleveland State Universitycis.csuohio.edu/~sschung/CIS660/QALectureNotesStanford.pdf · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms](https://reader033.fdocuments.in/reader033/viewer/2022042222/5ec905a2ae99265e9e4c1fa9/html5/thumbnails/46.jpg)
Question Answering
Using Knowledge in QA
![Page 47: Question Answering - Cleveland State Universitycis.csuohio.edu/~sschung/CIS660/QALectureNotesStanford.pdf · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms](https://reader033.fdocuments.in/reader033/viewer/2022042222/5ec905a2ae99265e9e4c1fa9/html5/thumbnails/47.jpg)
Dan Jurafsky
Rela%on Extrac%on
• Answers: Databases of Rela+ons • born-‐in(“Emma Goldman”, “June 27 1869”) • author-‐of(“Cao Xue Qin”, “Dream of the Red Chamber”) • Draw from Wikipedia infoboxes, DBpedia, FreeBase, etc.
• Ques+ons: Extrac+ng Rela+ons in Ques+ons Whose granddaughter starred in E.T.?
(acted-in ?x “E.T.”)! (granddaughter-of ?x ?y)!47
![Page 48: Question Answering - Cleveland State Universitycis.csuohio.edu/~sschung/CIS660/QALectureNotesStanford.pdf · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms](https://reader033.fdocuments.in/reader033/viewer/2022042222/5ec905a2ae99265e9e4c1fa9/html5/thumbnails/48.jpg)
Dan Jurafsky
Temporal Reasoning
• Rela+on databases • (and obituaries, biographical dic+onaries, etc.)
• IBM Watson ”In 1594 he took a job as a tax collector in Andalusia” Candidates: • Thoreau is a bad answer (born in 1817) • Cervantes is possible (was alive in 1594)
48
![Page 49: Question Answering - Cleveland State Universitycis.csuohio.edu/~sschung/CIS660/QALectureNotesStanford.pdf · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms](https://reader033.fdocuments.in/reader033/viewer/2022042222/5ec905a2ae99265e9e4c1fa9/html5/thumbnails/49.jpg)
Dan Jurafsky
Geospa%al knowledge (containment, direc%onality, borders)
• Beijing is a good answer for ”Asian city” • California is ”southwest of Montana” • geonames.org:
49
![Page 50: Question Answering - Cleveland State Universitycis.csuohio.edu/~sschung/CIS660/QALectureNotesStanford.pdf · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms](https://reader033.fdocuments.in/reader033/viewer/2022042222/5ec905a2ae99265e9e4c1fa9/html5/thumbnails/50.jpg)
Dan Jurafsky
Context and Conversa%on in Virtual Assistants like Siri
• Coreference helps resolve ambigui+es U: “Book a table at Il Fornaio at 7:00 with my mom” U: “Also send her an email reminder”
• Clarifica+on ques+ons: U: “Chicago pizza” S: “Did you mean pizza restaurants in Chicago or Chicago-‐style pizza?”
50
![Page 51: Question Answering - Cleveland State Universitycis.csuohio.edu/~sschung/CIS660/QALectureNotesStanford.pdf · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms](https://reader033.fdocuments.in/reader033/viewer/2022042222/5ec905a2ae99265e9e4c1fa9/html5/thumbnails/51.jpg)
Question Answering
Using Knowledge in QA
![Page 52: Question Answering - Cleveland State Universitycis.csuohio.edu/~sschung/CIS660/QALectureNotesStanford.pdf · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms](https://reader033.fdocuments.in/reader033/viewer/2022042222/5ec905a2ae99265e9e4c1fa9/html5/thumbnails/52.jpg)
Question Answering
Advanced: Answering Complex Ques+ons
![Page 53: Question Answering - Cleveland State Universitycis.csuohio.edu/~sschung/CIS660/QALectureNotesStanford.pdf · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms](https://reader033.fdocuments.in/reader033/viewer/2022042222/5ec905a2ae99265e9e4c1fa9/html5/thumbnails/53.jpg)
Dan Jurafsky
Answering harder ques%ons Q: What is water spinach? A: Water spinach (ipomoea aqua+ca) is a semi-‐aqua+c leafy green plant with long hollow stems and spear-‐ or heart-‐shaped leaves, widely grown throughout Asia as a leaf vegetable. The leaves and stems are oten eaten s+r-‐fried flavored with salt or in soups. Other common names include morning glory vegetable, kangkong (Malay), rau muong (Viet.), ong choi (Cant.), and kong xin cai (Mand.). It is not related to spinach, but is closely related to sweet potato and convolvulus.
![Page 54: Question Answering - Cleveland State Universitycis.csuohio.edu/~sschung/CIS660/QALectureNotesStanford.pdf · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms](https://reader033.fdocuments.in/reader033/viewer/2022042222/5ec905a2ae99265e9e4c1fa9/html5/thumbnails/54.jpg)
Dan Jurafsky
Answering harder ques%on Q: In children with an acute febrile illness, what is the efficacy of single medica+on therapy with acetaminophen or ibuprofen in reducing fever? A: Ibuprofen provided greater temperature decrement and longer dura+on of an+pyresis than acetaminophen when the two drugs were administered in approximately equal doses. (PubMedID: 1621668, Evidence Strength: A)
![Page 55: Question Answering - Cleveland State Universitycis.csuohio.edu/~sschung/CIS660/QALectureNotesStanford.pdf · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms](https://reader033.fdocuments.in/reader033/viewer/2022042222/5ec905a2ae99265e9e4c1fa9/html5/thumbnails/55.jpg)
Dan Jurafsky
Answering harder ques%ons via query-‐focused summariza%on
• The (booom-‐up) snippet method • Find a set of relevant documents • Extract informa+ve sentences from the documents (using |-‐idf, MMR) • Order and modify the sentences into an answer
• The (top-‐down) informa+on extrac+on method • build specific answerers for different ques+on types: • defini+on ques+ons, • biography ques+ons, • certain medical ques+ons
![Page 56: Question Answering - Cleveland State Universitycis.csuohio.edu/~sschung/CIS660/QALectureNotesStanford.pdf · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms](https://reader033.fdocuments.in/reader033/viewer/2022042222/5ec905a2ae99265e9e4c1fa9/html5/thumbnails/56.jpg)
Dan Jurafsky
The Informa%on Extrac%on method
• a good biography of a person contains: • a person’s birth/death, fame factor, educa%on, na%onality and so on
• a good defini%on contains: • genus or hypernym • The Hajj is a type of ritual
• a medical answer about a drug’s use contains: • the problem (the medical condi+on), • the interven%on (the drug or procedure), and • the outcome (the result of the study).
![Page 57: Question Answering - Cleveland State Universitycis.csuohio.edu/~sschung/CIS660/QALectureNotesStanford.pdf · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms](https://reader033.fdocuments.in/reader033/viewer/2022042222/5ec905a2ae99265e9e4c1fa9/html5/thumbnails/57.jpg)
Dan Jurafsky
Informa%on that should be in the answer for 3 kinds of ques%ons
![Page 58: Question Answering - Cleveland State Universitycis.csuohio.edu/~sschung/CIS660/QALectureNotesStanford.pdf · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms](https://reader033.fdocuments.in/reader033/viewer/2022042222/5ec905a2ae99265e9e4c1fa9/html5/thumbnails/58.jpg)
Dan Jurafsky
Document Retrieval
11 Web documents1127 total sentences
Predicate Identification
Data-Driven Analysis
383 Non-Specific Definitional sentences
Sentence clusters, Importance ordering
DefinitionCreation
9 Genus-Species SentencesThe Hajj, or pilgrimage to Makkah (Mecca), is the central duty of Islam.The Hajj is a milestone event in a Muslim's life.The hajj is one of five pillars that make up the foundation of Islam....
The Hajj, or pilgrimage to Makkah [Mecca], is the central duty of Islam. More than two million Muslims are expected to take the Hajj this year. Muslims must perform the hajj at least once in their lifetime if physically and financially able. The Hajj is a milestone event in a Muslim's life. The annual hajj begins in the twelfth month of the Islamic year (which is lunar, not solar, so that hajj and Ramadan fall sometimes in summer, sometimes in winter). The Hajj is a week-long pilgrimage that begins in the 12th month of the Islamic lunar calendar. Another ceremony, which was not connected with the rites of the Ka'ba before the rise of Islam, is the Hajj, the annual pilgrimage to 'Arafat, about two miles east of Mecca, toward Mina!
"What is the Hajj?" (Ndocs=20, Len=8)
Architecture for complex ques%on answering: defini%on ques%ons S. Blair-‐Goldensohn, K. McKeown and A. Schlaikjer. 2004.
Answering Defini+on Ques+ons: A Hyrbid Approach.
![Page 59: Question Answering - Cleveland State Universitycis.csuohio.edu/~sschung/CIS660/QALectureNotesStanford.pdf · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms](https://reader033.fdocuments.in/reader033/viewer/2022042222/5ec905a2ae99265e9e4c1fa9/html5/thumbnails/59.jpg)
Question Answering
Advanced: Answering Complex Ques+ons