1 SIMS 290-2: Applied Natural Language Processing Marti Hearst Sept 15, 2004.
-
date post
21-Dec-2015 -
Category
Documents
-
view
215 -
download
2
Transcript of 1 SIMS 290-2: Applied Natural Language Processing Marti Hearst Sept 15, 2004.
![Page 1: 1 SIMS 290-2: Applied Natural Language Processing Marti Hearst Sept 15, 2004.](https://reader030.fdocuments.in/reader030/viewer/2022032704/56649d6d5503460f94a4d671/html5/thumbnails/1.jpg)
1
SIMS 290-2: Applied Natural Language Processing
Marti HearstSept 15, 2004
![Page 2: 1 SIMS 290-2: Applied Natural Language Processing Marti Hearst Sept 15, 2004.](https://reader030.fdocuments.in/reader030/viewer/2022032704/56649d6d5503460f94a4d671/html5/thumbnails/2.jpg)
2
Class Pace and ScheduleNeed a foundation before you can do anything interesting.
Tokenizing, Tagging, Regex’sText Classification Principles and TechniquesTraining vs. Testing, processing corpora
Through (approximately) the 6th week, keep doing exercises from the NLTK tutorials to build that foundation.
2 more homeworksI’m trying to make them bite-sized pieces
7th – 10th Group Miniproject on Enron CorpusWill involve classification or Information ExtractionDifferent groups will do different thingsMay have a homework within this timeframe
11th – 15th Another MiniprojectEither on Enron project or your choicesI will suggest ideas; you can propose them tooMay also have 1-2 other homeworks in this timeframe
![Page 3: 1 SIMS 290-2: Applied Natural Language Processing Marti Hearst Sept 15, 2004.](https://reader030.fdocuments.in/reader030/viewer/2022032704/56649d6d5503460f94a4d671/html5/thumbnails/3.jpg)
3
Language Modeling
An fundamental concept in NLPMain idea:
For a given language, some words are more likely than others to follow each other, orYou can predict (with some degree of accuracy) the probability that a given word will follow another word.
Illustration:Distributions of words in class-participation exercise.
![Page 4: 1 SIMS 290-2: Applied Natural Language Processing Marti Hearst Sept 15, 2004.](https://reader030.fdocuments.in/reader030/viewer/2022032704/56649d6d5503460f94a4d671/html5/thumbnails/4.jpg)
4Adapted from slide by Bonnie Dorr
Next Word Prediction
From a NY Times story...Stocks ...Stocks plunged this ….Stocks plunged this morning, despite a cut in interest ratesStocks plunged this morning, despite a cut in interest rates by the Federal Reserve, as Wall ...Stocks plunged this morning, despite a cut in interest rates by the Federal Reserve, as Wall Street began
![Page 5: 1 SIMS 290-2: Applied Natural Language Processing Marti Hearst Sept 15, 2004.](https://reader030.fdocuments.in/reader030/viewer/2022032704/56649d6d5503460f94a4d671/html5/thumbnails/5.jpg)
5Adapted from slide by Bonnie Dorr
Stocks plunged this morning, despite a cut in interest rates by the Federal Reserve, as Wall Street began trading for the first time since last …Stocks plunged this morning, despite a cut in interest rates by the Federal Reserve, as Wall Street began trading for the first time since last Tuesday's terrorist attacks.
![Page 6: 1 SIMS 290-2: Applied Natural Language Processing Marti Hearst Sept 15, 2004.](https://reader030.fdocuments.in/reader030/viewer/2022032704/56649d6d5503460f94a4d671/html5/thumbnails/6.jpg)
6Adapted from slide by Bonnie Dorr
Human Word Prediction
Clearly, at least some of us have the ability to predict future words in an utterance.How?
Domain knowledgeSyntactic knowledgeLexical knowledge
![Page 7: 1 SIMS 290-2: Applied Natural Language Processing Marti Hearst Sept 15, 2004.](https://reader030.fdocuments.in/reader030/viewer/2022032704/56649d6d5503460f94a4d671/html5/thumbnails/7.jpg)
7Adapted from slide by Bonnie Dorr
Claim
A useful part of the knowledge needed to allow word prediction can be captured using simple statistical techniquesIn particular, we'll rely on the notion of the probability of a sequence (a phrase, a sentence)
![Page 8: 1 SIMS 290-2: Applied Natural Language Processing Marti Hearst Sept 15, 2004.](https://reader030.fdocuments.in/reader030/viewer/2022032704/56649d6d5503460f94a4d671/html5/thumbnails/8.jpg)
8Adapted from slide by Bonnie Dorr
Applications
Why do we want to predict a word, given some preceding words?
Rank the likelihood of sequences containing various alternative hypotheses, e.g. for ASRTheatre owners say popcorn/unicorn sales have doubled...Assess the likelihood/goodness of a sentence
– for text generation or machine translation.
The doctor recommended a cat scan.El doctor recommendó una exploración del gato.
![Page 9: 1 SIMS 290-2: Applied Natural Language Processing Marti Hearst Sept 15, 2004.](https://reader030.fdocuments.in/reader030/viewer/2022032704/56649d6d5503460f94a4d671/html5/thumbnails/9.jpg)
9Adapted from slide by Bonnie Dorr
N-Gram Models of Language
Use the previous N-1 words in a sequence to predict the next wordLanguage Model (LM)
unigrams, bigrams, trigrams,…
How do we train these models?Very large corpora
![Page 10: 1 SIMS 290-2: Applied Natural Language Processing Marti Hearst Sept 15, 2004.](https://reader030.fdocuments.in/reader030/viewer/2022032704/56649d6d5503460f94a4d671/html5/thumbnails/10.jpg)
10Adapted from slide by Bonnie Dorr
Simple N-Grams
Assume a language has V word types in its lexicon, how likely is word x to follow word y?
Simplest model of word probability: 1/VAlternative 1: estimate likelihood of x occurring in new text based on its general frequency of occurrence estimated from a corpus (unigram probability)
popcorn is more likely to occur than unicorn
Alternative 2: condition the likelihood of x occurring in the context of previous words (bigrams, trigrams,…)
mythical unicorn is more likely than mythical popcorn
![Page 11: 1 SIMS 290-2: Applied Natural Language Processing Marti Hearst Sept 15, 2004.](https://reader030.fdocuments.in/reader030/viewer/2022032704/56649d6d5503460f94a4d671/html5/thumbnails/11.jpg)
11
A Word on Notation
P(unicorn)Read this as “The probability of seeing the token unicorn”Unigram tagger uses this.
P(unicorn|mythical)Called the Conditional Probability.Read this as “The probability of seeing the token unicorn given that you’ve seen the token mythicalBigram tagger uses this.Related to the conditional frequency distributions that we’ve been working with.
![Page 12: 1 SIMS 290-2: Applied Natural Language Processing Marti Hearst Sept 15, 2004.](https://reader030.fdocuments.in/reader030/viewer/2022032704/56649d6d5503460f94a4d671/html5/thumbnails/12.jpg)
12Adapted from slide by Bonnie Dorr
Computing the Probability of a Word Sequence
Compute the product of component conditional probabilities?
P(the mythical unicorn) = P(the) P(mythical|the) P(unicorn|the mythical)
The longer the sequence, the less likely we are to find it in a training corpus
P(Most biologists and folklore specialists believe that in fact the mythical unicorn horns derived from the narwhal)
Solution: approximate using n-grams
![Page 13: 1 SIMS 290-2: Applied Natural Language Processing Marti Hearst Sept 15, 2004.](https://reader030.fdocuments.in/reader030/viewer/2022032704/56649d6d5503460f94a4d671/html5/thumbnails/13.jpg)
13Adapted from slide by Bonnie Dorr
Bigram Model
Approximate by P(unicorn|the mythical) by P(unicorn|mythical)
Markov assumption: The probability of a word depends only on the probability of a limited history
Generalization: The probability of a word depends only on the probability of the n previous words
– trigrams, 4-grams, …– the higher n is, the more data needed to train– backoff models
)11|( nn wwP )|( 1nn wwP
![Page 14: 1 SIMS 290-2: Applied Natural Language Processing Marti Hearst Sept 15, 2004.](https://reader030.fdocuments.in/reader030/viewer/2022032704/56649d6d5503460f94a4d671/html5/thumbnails/14.jpg)
14Adapted from slide by Bonnie Dorr
Using N-Grams
For N-gram models
P(wn-1,wn) = P(wn | wn-1) P(wn-1)
By the Chain Rule we can decompose a joint probability, e.g. P(w1,w2,w3)
P(w1,w2, ...,wn) = P(w1|w2,w3,...,wn) P(w2|w3, ...,wn) … P(wn-
1|wn) P(wn)
For bigrams then, the probability of a sequence is just the product of the conditional probabilities of its bigrams
P(the,mythical,unicorn) = P(unicorn|mythical)P(mythical|the) P(the|<start>)
n
kkkn wwPwP
111 )|()(
![Page 15: 1 SIMS 290-2: Applied Natural Language Processing Marti Hearst Sept 15, 2004.](https://reader030.fdocuments.in/reader030/viewer/2022032704/56649d6d5503460f94a4d671/html5/thumbnails/15.jpg)
15Adapted from slide by Bonnie Dorr
Training and Testing
N-Gram probabilities come from a training corpus
overly narrow corpus: probabilities don't generalizeoverly general corpus: probabilities don't reflect task or domain
A separate test corpus is used to evaluate the model, typically using standard metrics
held out test set; development test setcross validationresults tested for statistical significance
![Page 16: 1 SIMS 290-2: Applied Natural Language Processing Marti Hearst Sept 15, 2004.](https://reader030.fdocuments.in/reader030/viewer/2022032704/56649d6d5503460f94a4d671/html5/thumbnails/16.jpg)
16Adapted from slide by Bonnie Dorr
A Simple Example
From BeRP: The Berkeley Restaurant ProjectA testbed for a Speech Recognition projectSystem prompts user for information in order to fill in slots in a restaurant database.
– Type of food, hours open, how expensiveAfter getting lots of input, can compute how likely it is that someone will say X given that they already said Y.
P(I want to each Chinese food) = P(I | <start>) P(want | I) P(to | want) P(eat | to) P(Chinese | eat) P(food | Chinese)
![Page 17: 1 SIMS 290-2: Applied Natural Language Processing Marti Hearst Sept 15, 2004.](https://reader030.fdocuments.in/reader030/viewer/2022032704/56649d6d5503460f94a4d671/html5/thumbnails/17.jpg)
17Adapted from slide by Bonnie Dorr
A Bigram Grammar Fragment from BeRP
.001Eat British.03Eat today
.007Eat dessert.04Eat Indian
.01Eat tomorrow.04Eat a
.02Eat Mexican.04Eat at
.02Eat Chinese.05Eat dinner
.02Eat in.06Eat lunch
.03Eat breakfast.06Eat some
.03Eat Thai.16Eat on
![Page 18: 1 SIMS 290-2: Applied Natural Language Processing Marti Hearst Sept 15, 2004.](https://reader030.fdocuments.in/reader030/viewer/2022032704/56649d6d5503460f94a4d671/html5/thumbnails/18.jpg)
18Adapted from slide by Bonnie Dorr
.01British lunch.05Want a
.01British cuisine.65Want to
.15British restaurant.04I have
.60British food.08I don’t
.02To be.29I would
.09To spend.32I want
.14To have.02<start> I’m
.26To eat.04<start> Tell
.01Want Thai.06<start> I’d
.04Want some.25<start> I
![Page 19: 1 SIMS 290-2: Applied Natural Language Processing Marti Hearst Sept 15, 2004.](https://reader030.fdocuments.in/reader030/viewer/2022032704/56649d6d5503460f94a4d671/html5/thumbnails/19.jpg)
19Adapted from slide by Bonnie Dorr
P(I want to eat British food) = P(I|<start>) P(want|I) P(to|want) P(eat|to) P(British|eat) P(food|British) = .25*.32*.65*.26*.001*.60 = .000080vs. I want to eat Chinese food = .00015Probabilities seem to capture “syntactic'' facts, “world knowledge''
eat is often followed by an NPBritish food is not too popular
N-gram models can be trained by counting and normalization
![Page 20: 1 SIMS 290-2: Applied Natural Language Processing Marti Hearst Sept 15, 2004.](https://reader030.fdocuments.in/reader030/viewer/2022032704/56649d6d5503460f94a4d671/html5/thumbnails/20.jpg)
20Adapted from slide by Bonnie Dorr
What do we learn about the language?
What's being captured with ...P(want | I) = .32 P(to | want) = .65P(eat | to) = .26 P(food | Chinese) = .56P(lunch | eat) = .055
What about...P(I | I) = .0023P(I | want) = .0025P(I | food) = .013
![Page 21: 1 SIMS 290-2: Applied Natural Language Processing Marti Hearst Sept 15, 2004.](https://reader030.fdocuments.in/reader030/viewer/2022032704/56649d6d5503460f94a4d671/html5/thumbnails/21.jpg)
21Modified from Massio Poesio's lecture
Tagging with lexical frequencies
Secretariat/NNP is/VBZ expected/VBN to/TO race/VB tomorrow/NNPeople/NNS continue/VBP to/TO inquire/VB the/DT reason/NN for/IN the/DT race/NN for/IN outer/JJ space/NNProblem: assign a tag to race given its lexical frequencySolution: we choose the tag that has the greater
P(race|VB) Probability of “race” given “VB” on prior wordP(race|NN) Probability of “race” given “NN” on prior word
Actual estimate from the Switchboard corpus:P(race|NN) = .00041P(race|VB) = .00003
![Page 22: 1 SIMS 290-2: Applied Natural Language Processing Marti Hearst Sept 15, 2004.](https://reader030.fdocuments.in/reader030/viewer/2022032704/56649d6d5503460f94a4d671/html5/thumbnails/22.jpg)
22Modified from Diane Litman's version of Steve Bird's notes
Combining Taggers
Use more accurate algorithms when we can, backoff to wider coverage when needed.
Try tagging the token with the 1st order tagger. If the 1st order tagger is unable to find a tag for the token, try finding a tag with the 0th order tagger. If the 0th order tagger is also unable to find a tag, use the NN_CD_Tagger to find a tag.
![Page 23: 1 SIMS 290-2: Applied Natural Language Processing Marti Hearst Sept 15, 2004.](https://reader030.fdocuments.in/reader030/viewer/2022032704/56649d6d5503460f94a4d671/html5/thumbnails/23.jpg)
23Modified from Diane Litman's version of Steve Bird's notes
BackoffTagger class>>> train_toks =
TaggedTokenizer().tokenize(tagged_text_str)
# Construct the taggers >>> tagger1 = NthOrderTagger(1,
SUBTOKENS=‘WORDS’) >>> tagger2 = UnigramTagger() # 0th order>>> tagger3 = NN_CD_Tagger()
# Train the taggers >>> for tok in train_toks:
tagger1.train(tok) tagger2.train(tok)
![Page 24: 1 SIMS 290-2: Applied Natural Language Processing Marti Hearst Sept 15, 2004.](https://reader030.fdocuments.in/reader030/viewer/2022032704/56649d6d5503460f94a4d671/html5/thumbnails/24.jpg)
24Modified from Diane Litman's version of Steve Bird's notes
Backoff (continued)
# Combine the taggers (in order, by specificity) > tagger = BackoffTagger([tagger1, tagger2, tagger3])
# Use the combined tagger> accuracy = tagger_accuracy(tagger, unseen_tokens)
![Page 25: 1 SIMS 290-2: Applied Natural Language Processing Marti Hearst Sept 15, 2004.](https://reader030.fdocuments.in/reader030/viewer/2022032704/56649d6d5503460f94a4d671/html5/thumbnails/25.jpg)
25Modified from Diane Litman's version of Steve Bird's notes
Rule-Based Tagger
The Linguistic ComplaintWhere is the linguistic knowledge of a tagger?Just a massive table of numbersAren’t there any linguistic insights that could emerge from the data?Could thus use handcrafted sets of rules to tag input sentences, for example, if input follows a determiner tag it as a noun.
![Page 26: 1 SIMS 290-2: Applied Natural Language Processing Marti Hearst Sept 15, 2004.](https://reader030.fdocuments.in/reader030/viewer/2022032704/56649d6d5503460f94a4d671/html5/thumbnails/26.jpg)
26Slide modified from Massimo Poesio's
The Brill tagger
An example of TRANSFORMATION-BASED LEARNING Very popular (freely available, works fairly well)A SUPERVISED method: requires a tagged corpusBasic idea: do a quick job first (using frequency), then revise it using contextual rules
![Page 27: 1 SIMS 290-2: Applied Natural Language Processing Marti Hearst Sept 15, 2004.](https://reader030.fdocuments.in/reader030/viewer/2022032704/56649d6d5503460f94a4d671/html5/thumbnails/27.jpg)
27
Brill Tagging: In more detail
Start with simple (less accurate) rules…learn better ones from tagged corpus
Tag each word initially with most likely POSExamine set of transformations to see which improves tagging decisions compared to tagged corpus Re-tag corpus using best transformationRepeat until, e.g., performance doesn’t improveResult: tagging procedure (ordered list of transformations) which can be applied to new, untagged text
![Page 28: 1 SIMS 290-2: Applied Natural Language Processing Marti Hearst Sept 15, 2004.](https://reader030.fdocuments.in/reader030/viewer/2022032704/56649d6d5503460f94a4d671/html5/thumbnails/28.jpg)
28Slide modified from Massimo Poesio's
An example
Examples:They are expected to race tomorrow.The race for outer space.
Tagging algorithm:1. Tag all uses of “race” as NN (most likely tag in the Brown
corpus)• They are expected to race/NN tomorrow• the race/NN for outer space
2. Use a transformation rule to replace the tag NN with VB for all uses of “race” preceded by the tag TO:• They are expected to race/VB tomorrow• the race/NN for outer space
![Page 29: 1 SIMS 290-2: Applied Natural Language Processing Marti Hearst Sept 15, 2004.](https://reader030.fdocuments.in/reader030/viewer/2022032704/56649d6d5503460f94a4d671/html5/thumbnails/29.jpg)
29
First 20 Transformation Rules
From: Transformation-Based Error-Driven Learning and Natural Language Processing: A Case Study in Part of Speech
Tagging Eric Brill. Computational Linguistics. December, 1995.
![Page 30: 1 SIMS 290-2: Applied Natural Language Processing Marti Hearst Sept 15, 2004.](https://reader030.fdocuments.in/reader030/viewer/2022032704/56649d6d5503460f94a4d671/html5/thumbnails/30.jpg)
30
Transformation Rules for Tagging Unknown Words
From: Transformation-Based Error-Driven Learning and Natural Language Processing: A Case Study in Part of Speech
Tagging Eric Brill. Computational Linguistics. December, 1995.
![Page 31: 1 SIMS 290-2: Applied Natural Language Processing Marti Hearst Sept 15, 2004.](https://reader030.fdocuments.in/reader030/viewer/2022032704/56649d6d5503460f94a4d671/html5/thumbnails/31.jpg)
31Adapted from Massio Peosio's
Additional issues
Most of the difference in performance between POS algorithms depends on their treatment of UNKNOWN WORDS
Class-based N-grams
![Page 32: 1 SIMS 290-2: Applied Natural Language Processing Marti Hearst Sept 15, 2004.](https://reader030.fdocuments.in/reader030/viewer/2022032704/56649d6d5503460f94a4d671/html5/thumbnails/32.jpg)
32Modified from Diane Litman's version of Steve Bird's notes
Evaluating a Tagger
Tagged tokens – the original dataUntag (exclude) the dataTag the data with your own taggerCompare the original and new tags
Iterate over the two lists checking for identity and countingAccuracy = fraction correct
![Page 33: 1 SIMS 290-2: Applied Natural Language Processing Marti Hearst Sept 15, 2004.](https://reader030.fdocuments.in/reader030/viewer/2022032704/56649d6d5503460f94a4d671/html5/thumbnails/33.jpg)
33
Assessing the Errors
Why the tuple method? Dictionaries cannot be indexedby lists, so convert lists to tuples.
exclude returns a new token containing only the properties that are not named in the given list.
![Page 34: 1 SIMS 290-2: Applied Natural Language Processing Marti Hearst Sept 15, 2004.](https://reader030.fdocuments.in/reader030/viewer/2022032704/56649d6d5503460f94a4d671/html5/thumbnails/34.jpg)
34
Assessing the Errors
![Page 35: 1 SIMS 290-2: Applied Natural Language Processing Marti Hearst Sept 15, 2004.](https://reader030.fdocuments.in/reader030/viewer/2022032704/56649d6d5503460f94a4d671/html5/thumbnails/35.jpg)
35
Upcoming
First assignment due 8pm tonightTurn in on course Assignments page
For next week:Read the Chunking tutorial.(The pdf version has the missing images)http://nltk.sourceforge.net/tutorial/chunking.pdfWe’ll have an assignment getting practice with this.