Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string...
Transcript of Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string...
![Page 1: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/1.jpg)
1/225 NLP Hal Daumé III, [email protected]
NaturalLanguage
Processing(for smart people who know nothing
about natural language processing)
Hal Daume III | UMD | [email protected] | @haldaume3Machine Learning Summer School 2012 @ UC Santa Cruz
Title cred: Persi Diaconis
![Page 2: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/2.jpg)
2/225 NLP Hal Daumé III, [email protected]
What is NLP?
� Fundamental goal: deep understanding of text� Not just string processing or keyword matching
� End systems that we want to build� Simple: Spelling correction, text categorization, etc.
� Complex: Speech recognition, machine translation, information extraction, dialog interfaces, question answering
� Unknown: human-level comprehension (might be more than just NLP)
Slide cred: Dan Klein
![Page 3: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/3.jpg)
3/225 NLP Hal Daumé III, [email protected]
Topology of the Field
Human Language Technologies
Automatic Speech
Recognition
Information
Retrieval
Computational
Linguistics
Natural Language
Processing
ICASSP
??? ACL
SIGIR
![Page 4: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/4.jpg)
4/225 NLP Hal Daumé III, [email protected]
Topology of the Field
Human Language Technologies
Automatic Speech
Recognition
Information
Retrieval
Computational
Linguistics
Natural Language
Processing
NLP
Machine Translation
Summarization
Question Answering
Information Extraction
Parsing
“Understanding”
Generation
![Page 5: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/5.jpg)
5/225 NLP Hal Daumé III, [email protected]
What can I teach you in 6 hours...
� a bag is just language of not words
� NLP through the eyes of machine translation
� Machine learning for complex structured prediction problems (and then some)
� How to learn language by interacting with the world
(no math, no machine learning)
(a little math, a little machine learning)
(lotsa math, lotsa machine learning)
(some math, some machine learning)
![Page 6: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/6.jpg)
6/225 NLP Hal Daumé III, [email protected]
Speech
� Automatic speech recognition (ASR)� Audio in, text out� SOTA: 0.3% error for digits, 10% dictation, 50%+ TV
� Text to speech (TTS)� Text in, audio out� SOTA: completely intelligible (sometimes unnatural)
�Speech Lab�
Slide cred: Dan Klein
![Page 7: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/7.jpg)
7/225 NLP Hal Daumé III, [email protected]
Question answering
� More than search� Can be really easy:
�What is the capital of Wyoming?�
� �Can also be hard � Can be open ended:
�What are the main issues in the global warming debate?�
� SOTA: Can do factoids, can't do much else
Slide cred: Dan Klein
![Page 8: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/8.jpg)
8/225 NLP Hal Daumé III, [email protected]
Information Extraction� Unstructured text to database entries
� SOTA: 40% for hard things, 80% accuracy for multi-sentence templates, 90%+ for single easy fields
New York Times Co. named Russell T. Lewis, 45, president and general
manager of its flagship New York Times newspaper, responsible for all
business-side activities. He was executive vice president and deputy
general manager. He succeeds Lance R. Primis, who in September was
named president and chief operating officer of the parent.
startpresident and CEONew York Times Co.Lance R. Primis
endexecutive vice
presidentNew York Times
newspaperRussell T. Lewis
startpresident and
general managerNew York Times
newspaperRussell T. Lewis
StatePostCompanyPerson
Slide cred: Dan Klein
![Page 9: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/9.jpg)
9/225 NLP Hal Daumé III, [email protected]
Summarization� Condensing documents
� Single or multiple
� Extractive or abstractive
� Indicative or informative
� Etc...
� Very context (and user) dependent
� Involves analysis and generation
� SOTA: who knows...
Slide cred: Dan Klein
![Page 11: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/11.jpg)
11/225 NLP Hal Daumé III, [email protected]
Language comprehension
But man is not destined to vanish. He can be killed, but he cannot be destroyed, because his soul is deathless and his spirit is irrepressible. Therefore, though the situation seems dark in the context of the confrontation between the superpowers, the silver lining is provided by amazing phenomenon that the very nations which have spent incalculable resources and energy for the production of deadly weapons are desperately trying to find out how they might never be used. They threaten each other, intimidate each other and go to the brink, but before the total hour arrives they withdraw from the brink.
The main point from the author's view is:
1. Man's soul and spirit can not be destroyed by superpowers.
2. Man's destiny is not fully clear or visible. 3. Man's soul and spirit are immortal. 4. Man's safety is assured by the delicate balance
of power in terms of nuclear weapons. 5. Human society will survive despite the
serious threat of total annihilation.
![Page 12: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/12.jpg)
12/225 NLP Hal Daumé III, [email protected]
Why is language hard?
� Ambiguity abounds (some headlines)� Iraqi Head Seeks Arms� Teacher Strikes Idle Kids� Kids Make Nutritious Snacks� Stolen Painting Found by Tree� Local HS Dropouts Cut in Half� Enraged Cow Injures Farmer with Ax� Hospitals are Sued by 7 Foot Doctors� Ban on Nude Dancing on Governor's Desk
� Why are these funny?
Slide cred: Dan Klein
![Page 13: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/13.jpg)
13/225 NLP Hal Daumé III, [email protected]
Why is language hard?
� Ambiguity abounds (some headlines)� Iraqi Head Seeks Arms� Teacher Strikes Idle Kids� Kids Make Nutritious Snacks� Stolen Painting Found by Tree� Local HS Dropouts Cut in Half� Enraged Cow Injures Farmer with Ax� Hospitals are Sued by 7 Foot Doctors� Ban on Nude Dancing on Governor's Desk
� Why are these funny?
Slide cred: Dan Klein
![Page 14: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/14.jpg)
14/225 NLP Hal Daumé III, [email protected]
Why (else) is language hard?
� Data sparsity� NYT: 500M+ words of newswire text
� Brown Corpus: 1M words of tagged �balanced� text
� Penn Treebank: 1M words of parsed WSJ text
� Canadian Hansard: 10M+ words of parallel French/English text
� The Web: *B+ words of who knows what...
Brain teaser:You've already seen N tokens of text.This contained a bunch of word types.What is the probability that the N+1st word will be new?
Slide cred: Dan Klein
![Page 15: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/15.jpg)
15/225 NLP Hal Daumé III, [email protected]
Why (else) is language hard?
00.10.20.30.40.50.60.70.80.9
1
0 200000 400000 600000 800000 1000000
Fraction Seen
Number of Words
Unigrams
Bigrams
![Page 16: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/16.jpg)
16/225 NLP Hal Daumé III, [email protected]
Structural ambiguity
Time flies like an arrow.
I saw the Grand Canyon flying to New York.
I saw the man on the hill with the telescope.
I ate spaghetti with....
.... meatballs
.... a fork
Slide cred: Ellen Riloff
![Page 17: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/17.jpg)
17/225 NLP Hal Daumé III, [email protected]
Structural ambiguity
Hurricane Emily howled toward Mexico's Caribbean coast on Sundaypacking 135 mph winds and torrential rain and causing panic inCancun, where frightened tourists squeezed into musty shelters.
SOTA: 90% accuracy for mid-1980s WSJ text
Slide cred: Dan Klein
![Page 18: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/18.jpg)
18/225 NLP Hal Daumé III, [email protected]
Syntax doesn't nail down semantics
� Every morning someone'salarm clock wakes me up
� Every person on this islandspeaks two languages
� Please come see me next Friday
Slide cred: Dan Klein
![Page 19: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/19.jpg)
19/225 NLP Hal Daumé III, [email protected]
Discourse: coreference
President John F. Kennedy was assassinated.
The President was shot yesterday.
Relatives said that John was a good father.
JFK was the youngest president in history.
His family will bury him tomorrow.
Friends of the Massachusetts native will hold
a candlelight service in Mr. Kennedy's
home town.
SOTA: 70-80% accuracy for newswire text
Slide cred: Ellen Riloff
![Page 20: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/20.jpg)
20/225 NLP Hal Daumé III, [email protected]
Discourse structure
� I only like traveling to Europe.
So I submitted a paper to ACL.
� I only like traveling to Europe.
Nevertheless I submitted a paper to ACL.
![Page 21: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/21.jpg)
21/225 NLP Hal Daumé III, [email protected]
Pragmatics
Rules of conversation:
- Can you tell me what time it is?
- Could you pass the salt?
Speech acts:
- I bet you $50 that the Jazz will win.
- Will you marry me?
Slide cred: Ellen Riloff
![Page 22: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/22.jpg)
22/225 NLP Hal Daumé III, [email protected]
Pragmatics
Scene 1: Penn Station, NY
(Hal) Long Beach?
(Passerby) Downstairs, LIRR Station.
S2: Ticket Counter, LIRR (Hal) Long Beach?
(Clerk) $4.50.
Scene 3: Info. Booth, LIRR Station
(Hal) Long Beach?
(Clerk) 4:19, Track 17.
Scene 4: On train, vicinity of Forest Hills
(Hal) Long Beach?
(Conductor) Change at Jamaica.
Scene 5: Next train, near Lynbrook
(Hal) Long Beach?
(Conductor) Right after Island Park.
Slide cred: Kathy McKeown
![Page 23: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/23.jpg)
23/225 NLP Hal Daumé III, [email protected]
Translate Centauri Arcturan�
1a. ok-voon ororok sprok .1b. at-voon bichat dat .
7a. lalok farok ororok lalok sprok izok enemok 7b. wat jjat bichat wat dat vat eneat .
2a. ok-drubel ok-voon anok plok sprok .2b. at-drubel at-voon pippat rrat dat .
8a. lalok brok anok plok nok .8b. iat lat pippat rrat nnat .
3a. erok sprok izok hihok ghirok .3b. totat dat arrat vat hilat .
9a. wiwok nok izok kantok ok-yurp .9b. totat nnat quat oloat at-yurp .
4a. ok-voon anok drok brok jok .4b. at-voon krat pippat sat lat .
10a. lalok mok nok yorok ghirok clok .10b. wat nnat gat mat bat hilat .
5a. wiwok farok izok stok .5b. totat jjat quat cat .
11a. lalok nok crrrok hihok yorok zanzanok .11b. wat nnat arrat mat zanzanat .
6a. lalok sprok izok jok stok .6b. wat dat krat quat cat .
12a. lalok rarok nok izok hihok mok .12b. wat nnat forat arrat vat gat .
Your assignment, translate this Centauri sentence to Arcturan:farok crrrok hihok yorok clok kantok ok-yurp
Slide cred: Kevin Knight
![Page 24: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/24.jpg)
24/225 NLP Hal Daumé III, [email protected]
A Bit of History1940s Computations begins, AI hot, Turing test
Machine translation = Code-breaking?
1950s Cold war continues
1960s Chomsky and statistics, ALPAC report
1970s Dry spell
1980s Statistics makes significant advances in speech
1990s Web arrives
Statistical revolution in machine translation, parsing, IE, etc
Serious �corpus� work, increasing focus on evaluation
2000s Focus on optimizing loss functions, reranking
Even more focus on evaluation, automatic metrics
Huge process in machine translation
Gigantic corpora become available, scaling
New challenges
![Page 25: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/25.jpg)
25/225 NLP Hal Daumé III, [email protected]
Data-driven NLP
� I could hand-code that:� rope is a way of committing suicide� meatballs are themes and forks are means� canyons don't fly� some bugs fly and others are planted
� Or I could try to learn/infer it from data:� (committing suicide with Google)� (flying canyons with Google)
![Page 26: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/26.jpg)
26/225 NLP Hal Daumé III, [email protected]
What is Nearby NLP?� Computational Linguistics
� Using computational methods to learn more about how language works
� We end up doing this and using it
� Cognitive Science� Figuring out how the human brain works� Includes the bits that do language� Humans: the only working NLP prototype!
� Speech?� Mapping audio signals to text� Traditionally separate from NLP, converging?� Two components: acoustic models and language
models� Language models in the domain of stat NLP
Slide cred: Dan Klein
![Page 27: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/27.jpg)
27/225 NLP Hal Daumé III, [email protected]
English is an outlier...
uyuyorum I am sleeping uyuorsun you are sleepinguyuyor he/she/it is sleeping uyuyoruz we are sleepinguyuyorsunuz you are sleeping uyuyorlar they are sleepinguyuduk we slept uyumaliyiz we must sleepuyuman your sleeping uyumadan without sleeping
uyudukca as long as (somebody) sleepsuyurken while (somebody) is sleepinguyuyunca when (somebody) sleepsuyutmak to cause somebody to sleepuyuturmak to cause somebody to cause another to sleepuyutturtturmak to cause somebody to cause another to cause
yet another to sleep
![Page 28: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/28.jpg)
28/225 NLP Hal Daumé III, [email protected]
Classical MT (1970s and 1980s)
SourceText
SourceLanguageAnalysis
SourceLexicon
TargetText
TargetLanguageGeneration
TargetLexicon
KnowledgeBase
Transfer/Interlingua
Representation
TransferRules
![Page 29: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/29.jpg)
29/225 NLP Hal Daumé III, [email protected]
How Much Analysis?
Source Words Target Words
SourceMorphology
Source Syntax
Source Semantics
Interlingua
TargetMorphology
Target Syntax
Target Semantics
Anal
ysis
Gen
eration
Direct
Classical
Not practicalfor open domain
Becoming Popular
![Page 30: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/30.jpg)
30/225 NLP Hal Daumé III, [email protected]
Syntactic Transfer
� Now, just get a bunch of linguists to sit down and write rules and grammars
The student will see the man
D N AX V D N
NP NP
SUB A V OB
S
Der Student wird den Mann seben
D N AX D N V
NP NP
SUB A OB V
S
![Page 31: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/31.jpg)
31/225 NLP Hal Daumé III, [email protected]
While We're Busy Writing Grammars
Claude Shannon
InformationSource
NoisyChannel
CorruptedOutput
p(s) p(o | s) p(s | o) � p(s) * p(o | s)
�Imagined�Words
Speech Process Acoustic Signal
Need: p(word sequence) and p(signal | word sequence)
![Page 32: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/32.jpg)
32/225 NLP Hal Daumé III, [email protected]
Acoustic Modeling: p(a | w)
Signal:
Transcription: the man ate
Key notion: acoustic-word alignments
Chicken-and-eggproblem that wecan solve usingExpectationMaximization
![Page 33: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/33.jpg)
33/225 NLP Hal Daumé III, [email protected]
Language Modeling: p(w)
� Sentence = sequence of symbols from alphabet
p �w1,w2, ... , w I � = �i=1
I
p �wi �w1, ... ,w i�1�
�i=1
I
p �wi �wi�k , ... , wi�1�
The beloved n-gramlanguage model
In practice, probabilitiesare estimated from a largecorpus, but are �smoothed�intelligently to avoid zeroprobability n-grams.
Language modeling isoften the art of goodsmoothing.
See [Goodman 1998]
![Page 34: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/34.jpg)
34/225 NLP Hal Daumé III, [email protected]
Speech Rec = Machine Translation?� Peter F. Brown� Stephen A. Della Pietra� Vincent J. Della Pietra� Robert Mercer� The Mathematics of Statistical Machine Translation:
Parameter Estimation� Computational Linguistics 19 (2), June 1993
� Maybe most important paper in NLP in last 20 years
�Brown 93�
![Page 35: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/35.jpg)
35/225 NLP Hal Daumé III, [email protected]
Centauri/Arcturan [Knight 97]
1a. ok-voon ororok sprok .
1b. at-voon bichat dat .
7a. lalok farok ororok lalok sprok izok enemok .
7b. wat jjat bichat wat dat vat eneat .
2a. ok-drubel ok-voon anok plok sprok .
2b. at-drubel at-voon pippat rrat dat .
8a. lalok brok anok plok nok .
8b. iat lat pippat rrat nnat .
3a. erok sprok izok hihok ghirok .
3b. totat dat arrat vat hilat .
9a. wiwok nok izok kantok ok-yurp .
9b. totat nnat quat oloat at-yurp .
4a. ok-voon anok drok brok jok .
4b. at-voon krat pippat sat lat .
10a. lalok mok nok yorok ghirok clok .
10b. wat nnat gat mat bat hilat .
5a. wiwok farok izok stok .
5b. totat jjat quat cat .
11a. lalok nok crrrok hihok yorok zanzanok .
11b. wat nnat arrat mat zanzanat .
6a. lalok sprok izok jok stok .
6b. wat dat krat quat cat .
12a. lalok rarok nok izok hihok mok .
12b. wat nnat forat arrat vat gat .
Your assignment, translate this to Arcturan: farok crrrok hihok yorok clok kantok ok-yurp
![Page 36: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/36.jpg)
36/225 NLP Hal Daumé III, [email protected]
Centauri/Arcturan [Knight 97]
1a. ok-voon ororok sprok .
1b. at-voon bichat dat .
7a. lalok farok ororok lalok sprok izok enemok .
7b. wat jjat bichat wat dat vat eneat .
2a. ok-drubel ok-voon anok plok sprok .
2b. at-drubel at-voon pippat rrat dat .
8a. lalok brok anok plok nok .
8b. iat lat pippat rrat nnat .
3a. erok sprok izok hihok ghirok .
3b. totat dat arrat vat hilat .
9a. wiwok nok izok kantok ok-yurp .
9b. totat nnat quat oloat at-yurp .
4a. ok-voon anok drok brok jok .
4b. at-voon krat pippat sat lat .
10a. lalok mok nok yorok ghirok clok .
10b. wat nnat gat mat bat hilat .
5a. wiwok farok izok stok .
5b. totat jjat quat cat .
11a. lalok nok crrrok hihok yorok zanzanok .
11b. wat nnat arrat mat zanzanat .
6a. lalok sprok izok jok stok .
6b. wat dat krat quat cat .
12a. lalok rarok nok izok hihok mok .
12b. wat nnat forat arrat vat gat .
Your assignment, translate this to Arcturan: farok crrrok hihok yorok clok kantok ok-yurp
![Page 37: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/37.jpg)
37/225 NLP Hal Daumé III, [email protected]
Centauri/Arcturan [Knight 97]
1a. ok-voon ororok sprok .
1b. at-voon bichat dat .
7a. lalok farok ororok lalok sprok izok enemok .
7b. wat jjat bichat wat dat vat eneat .
2a. ok-drubel ok-voon anok plok sprok .
2b. at-drubel at-voon pippat rrat dat .
8a. lalok brok anok plok nok .
8b. iat lat pippat rrat nnat .
3a. erok sprok izok hihok ghirok .
3b. totat dat arrat vat hilat .
9a. wiwok nok izok kantok ok-yurp .
9b. totat nnat quat oloat at-yurp .
4a. ok-voon anok drok brok jok .
4b. at-voon krat pippat sat lat .
10a. lalok mok nok yorok ghirok clok .
10b. wat nnat gat mat bat hilat .
5a. wiwok farok izok stok .
5b. totat jjat quat cat .
11a. lalok nok crrrok hihok yorok zanzanok .
11b. wat nnat arrat mat zanzanat .
6a. lalok sprok izok jok stok .
6b. wat dat krat quat cat .
12a. lalok rarok nok izok hihok mok .
12b. wat nnat forat arrat vat gat .
Your assignment, translate this to Arcturan: farok crrrok hihok yorok clok kantok ok-yurp
![Page 38: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/38.jpg)
38/225 NLP Hal Daumé III, [email protected]
Centauri/Arcturan [Knight 97]
1a. ok-voon ororok sprok .
1b. at-voon bichat dat .
7a. lalok farok ororok lalok sprok izok enemok .
7b. wat jjat bichat wat dat vat eneat .
2a. ok-drubel ok-voon anok plok sprok .
2b. at-drubel at-voon pippat rrat dat .
8a. lalok brok anok plok nok .
8b. iat lat pippat rrat nnat .
3a. erok sprok izok hihok ghirok .
3b. totat dat arrat vat hilat .
9a. wiwok nok izok kantok ok-yurp .
9b. totat nnat quat oloat at-yurp .
4a. ok-voon anok drok brok jok .
4b. at-voon krat pippat sat lat .
10a. lalok mok nok yorok ghirok clok .
10b. wat nnat gat mat bat hilat .
5a. wiwok farok izok stok .
5b. totat jjat quat cat .
11a. lalok nok crrrok hihok yorok zanzanok .
11b. wat nnat arrat mat zanzanat .
6a. lalok sprok izok jok stok .
6b. wat dat krat quat cat .
12a. lalok rarok nok izok hihok mok .
12b. wat nnat forat arrat vat gat .
Your assignment, translate this to Arcturan: farok crrrok hihok yorok clok kantok ok-yurp
![Page 39: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/39.jpg)
39/225 NLP Hal Daumé III, [email protected]
Centauri/Arcturan [Knight 97]
1a. ok-voon ororok sprok .
1b. at-voon bichat dat .
7a. lalok farok ororok lalok sprok izok enemok .
7b. wat jjat bichat wat dat vat eneat .
2a. ok-drubel ok-voon anok plok sprok .
2b. at-drubel at-voon pippat rrat dat .
8a. lalok brok anok plok nok .
8b. iat lat pippat rrat nnat .
3a. erok sprok izok hihok ghirok .
3b. totat dat arrat vat hilat .
9a. wiwok nok izok kantok ok-yurp .
9b. totat nnat quat oloat at-yurp .
4a. ok-voon anok drok brok jok .
4b. at-voon krat pippat sat lat .
10a. lalok mok nok yorok ghirok clok .
10b. wat nnat gat mat bat hilat .
5a. wiwok farok izok stok .
5b. totat jjat quat cat .
11a. lalok nok crrrok hihok yorok zanzanok .
11b. wat nnat arrat mat zanzanat .
6a. lalok sprok izok jok stok .
6b. wat dat krat quat cat .
12a. lalok rarok nok izok hihok mok .
12b. wat nnat forat arrat vat gat .
Your assignment, translate this to Arcturan: farok crrrok hihok yorok clok kantok ok-yurp
![Page 40: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/40.jpg)
40/225 NLP Hal Daumé III, [email protected]
Centauri/Arcturan [Knight 97]
1a. ok-voon ororok sprok .
1b. at-voon bichat dat .
7a. lalok farok ororok lalok sprok izok enemok .
7b. wat jjat bichat wat dat vat eneat .
2a. ok-drubel ok-voon anok plok sprok .
2b. at-drubel at-voon pippat rrat dat .
8a. lalok brok anok plok nok .
8b. iat lat pippat rrat nnat .
3a. erok sprok izok hihok ghirok .
3b. totat dat arrat vat hilat .
9a. wiwok nok izok kantok ok-yurp .
9b. totat nnat quat oloat at-yurp .
4a. ok-voon anok drok brok jok .
4b. at-voon krat pippat sat lat .
10a. lalok mok nok yorok ghirok clok .
10b. wat nnat gat mat bat hilat .
5a. wiwok farok izok stok .
5b. totat jjat quat cat .
11a. lalok nok crrrok hihok yorok zanzanok .
11b. wat nnat arrat mat zanzanat .
6a. lalok sprok izok jok stok .
6b. wat dat krat quat cat .
12a. lalok rarok nok izok hihok mok .
12b. wat nnat forat arrat vat gat .
Your assignment, translate this to Arcturan: farok crrrok hihok yorok clok kantok ok-yurp
![Page 41: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/41.jpg)
41/225 NLP Hal Daumé III, [email protected]
Centauri/Arcturan [Knight 97]
1a. ok-voon ororok sprok .
1b. at-voon bichat dat .
7a. lalok farok ororok lalok sprok izok enemok .
7b. wat jjat bichat wat dat vat eneat .
2a. ok-drubel ok-voon anok plok sprok .
2b. at-drubel at-voon pippat rrat dat .
8a. lalok brok anok plok nok .
8b. iat lat pippat rrat nnat .
3a. erok sprok izok hihok ghirok .
3b. totat dat arrat vat hilat .
9a. wiwok nok izok kantok ok-yurp .
9b. totat nnat quat oloat at-yurp .
4a. ok-voon anok drok brok jok .
4b. at-voon krat pippat sat lat .
10a. lalok mok nok yorok ghirok clok .
10b. wat nnat gat mat bat hilat .
5a. wiwok farok izok stok .
5b. totat jjat quat cat .
11a. lalok nok crrrok hihok yorok zanzanok .
11b. wat nnat arrat mat zanzanat .
6a. lalok sprok izok jok stok .
6b. wat dat krat quat cat .
12a. lalok rarok nok izok hihok mok .
12b. wat nnat forat arrat vat gat .
Your assignment, translate this to Arcturan: farok crrrok hihok yorok clok kantok ok-yurp
![Page 42: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/42.jpg)
42/225 NLP Hal Daumé III, [email protected]
Centauri/Arcturan [Knight 97]
1a. ok-voon ororok sprok .
1b. at-voon bichat dat .
7a. lalok farok ororok lalok sprok izok enemok .
7b. wat jjat bichat wat dat vat eneat .
2a. ok-drubel ok-voon anok plok sprok .
2b. at-drubel at-voon pippat rrat dat .
8a. lalok brok anok plok nok .
8b. iat lat pippat rrat nnat .
3a. erok sprok izok hihok ghirok .
3b. totat dat arrat vat hilat .
9a. wiwok nok izok kantok ok-yurp .
9b. totat nnat quat oloat at-yurp .
4a. ok-voon anok drok brok jok .
4b. at-voon krat pippat sat lat .
10a. lalok mok nok yorok ghirok clok .
10b. wat nnat gat mat bat hilat .
5a. wiwok farok izok stok .
5b. totat jjat quat cat .
11a. lalok nok crrrok hihok yorok zanzanok .
11b. wat nnat arrat mat zanzanat .
6a. lalok sprok izok jok stok .
6b. wat dat krat quat cat .
12a. lalok rarok nok izok hihok mok .
12b. wat nnat forat arrat vat gat .
???
Your assignment, translate this to Arcturan: farok crrrok hihok yorok clok kantok ok-yurp
![Page 43: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/43.jpg)
43/225 NLP Hal Daumé III, [email protected]
Centauri/Arcturan [Knight 97]
1a. ok-voon ororok sprok .
1b. at-voon bichat dat .
7a. lalok farok ororok lalok sprok izok enemok .
7b. wat jjat bichat wat dat vat eneat .
2a. ok-drubel ok-voon anok plok sprok .
2b. at-drubel at-voon pippat rrat dat .
8a. lalok brok anok plok nok .
8b. iat lat pippat rrat nnat .
3a. erok sprok izok hihok ghirok .
3b. totat dat arrat vat hilat .
9a. wiwok nok izok kantok ok-yurp .
9b. totat nnat quat oloat at-yurp .
4a. ok-voon anok drok brok jok .
4b. at-voon krat pippat sat lat .
10a. lalok mok nok yorok ghirok clok .
10b. wat nnat gat mat bat hilat .
5a. wiwok farok izok stok .
5b. totat jjat quat cat .
11a. lalok nok crrrok hihok yorok zanzanok .
11b. wat nnat arrat mat zanzanat .
6a. lalok sprok izok jok stok .
6b. wat dat krat quat cat .
12a. lalok rarok nok izok hihok mok .
12b. wat nnat forat arrat vat gat .
Your assignment, translate this to Arcturan: farok crrrok hihok yorok clok kantok ok-yurp
process ofelimination
![Page 44: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/44.jpg)
44/225 NLP Hal Daumé III, [email protected]
Centauri/Arcturan [Knight 97]
1a. ok-voon ororok sprok .
1b. at-voon bichat dat .
7a. lalok farok ororok lalok sprok izok enemok .
7b. wat jjat bichat wat dat vat eneat .
2a. ok-drubel ok-voon anok plok sprok .
2b. at-drubel at-voon pippat rrat dat .
8a. lalok brok anok plok nok .
8b. iat lat pippat rrat nnat .
3a. erok sprok izok hihok ghirok .
3b. totat dat arrat vat hilat .
9a. wiwok nok izok kantok ok-yurp .
9b. totat nnat quat oloat at-yurp .
4a. ok-voon anok drok brok jok .
4b. at-voon krat pippat sat lat .
10a. lalok mok nok yorok ghirok clok .
10b. wat nnat gat mat bat hilat .
5a. wiwok farok izok stok .
5b. totat jjat quat cat .
11a. lalok nok crrrok hihok yorok zanzanok .
11b. wat nnat arrat mat zanzanat .
6a. lalok sprok izok jok stok .
6b. wat dat krat quat cat .
12a. lalok rarok nok izok hihok mok .
12b. wat nnat forat arrat vat gat .
cognate?
Your assignment, translate this to Arcturan: farok crrrok hihok yorok clok kantok ok-yurp
![Page 45: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/45.jpg)
45/225 NLP Hal Daumé III, [email protected]
Centauri/Arcturan [Knight 97]
1a. ok-voon ororok sprok .
1b. at-voon bichat dat .
7a. lalok farok ororok lalok sprok izok enemok .
7b. wat jjat bichat wat dat vat eneat .
2a. ok-drubel ok-voon anok plok sprok .
2b. at-drubel at-voon pippat rrat dat .
8a. lalok brok anok plok nok .
8b. iat lat pippat rrat nnat .
3a. erok sprok izok hihok ghirok .
3b. totat dat arrat vat hilat .
9a. wiwok nok izok kantok ok-yurp .
9b. totat nnat quat oloat at-yurp .
4a. ok-voon anok drok brok jok .
4b. at-voon krat pippat sat lat .
10a. lalok mok nok yorok ghirok clok .
10b. wat nnat gat mat bat hilat .
5a. wiwok farok izok stok .
5b. totat jjat quat cat .
11a. lalok nok crrrok hihok yorok zanzanok .
11b. wat nnat arrat mat zanzanat .
6a. lalok sprok izok jok stok .
6b. wat dat krat quat cat .
12a. lalok rarok nok izok hihok mok .
12b. wat nnat forat arrat vat gat .
cognate?
Your assignment, translate this to Arcturan: farok crrrok hihok yorok clok kantok ok-yurp
Zero fertility!
![Page 46: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/46.jpg)
46/225 NLP Hal Daumé III, [email protected]
Unsupervised EM Training
� la maison �.. la maison bleue �... la fleur �
� the house �.. the blue house ...� the flower �
All P(french-word | english-word) equally likely
![Page 47: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/47.jpg)
47/225 NLP Hal Daumé III, [email protected]
Unsupervised EM Training
� la maison ..� la maison bleue ...� la fleur �
� the house ..� the blue house ...� the flower �
�la� and �the� observed to co-occur frequently,so P(la | the) is increased.
![Page 48: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/48.jpg)
48/225 NLP Hal Daumé III, [email protected]
Unsupervised EM Training
� la maison ..� la maison bleue �... la fleur �
� the house ..� the blue house ...� the flower �
�maison� co-occurs with both �the� and �house�, butP(maison | house) can be raised without limit, to 1.0,
while P(maison | the) is limited because of �la�
(pigeonhole principle)
![Page 49: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/49.jpg)
49/225 NLP Hal Daumé III, [email protected]
Unsupervised EM Training
� la maison �.. la maison bleue ...� la fleur �
� the house ..� the blue house �... the flower �
settling down after another iteration
![Page 50: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/50.jpg)
50/225 NLP Hal Daumé III, [email protected]
Unsupervised EM Training
� la maison �.. la maison bleue �... la fleur �
� the house �.. the blue house ...� the flower �
Inherent hidden structure revealed by EM training!
� �A Statistical MT Tutorial Workbook� (Knight, 1999). Promises free beer.
� �The Mathematics of Statistical Machine Translation� (Brown et al, 1993)
� Software: GIZA++
![Page 51: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/51.jpg)
51/225 NLP Hal Daumé III, [email protected]
Mary did not slap the green witch
Mary not slap slap slap the green witch n(3|slap)
Maria no daba una botefada a la bruja verde
d(j|i)
Mary not slap slap slap NULL the green witch
P NULL
Maria no daba una botefada a la verde bruja
t(la|the)
Use the EM algorithm for training the parameters
The IBM model: �Brown93�
![Page 52: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/52.jpg)
52/225 NLP Hal Daumé III, [email protected]
Ready-to-use Data
1994 1996 1998 2000 2002 20040
20
40
60
80
100
120
140
160
180
French-English
Chinese-English
Arabic-English
Mill
ions o
f W
ord
s (
Englis
h s
ide)
![Page 53: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/53.jpg)
53/225 NLP Hal Daumé III, [email protected]
Decoding for Machine Translation
EnglishSource
TranslationModel
FrenchOutput
p(e) p(f | e) p(e | f) � p(e) * p(f | e)
Decoding: e = argmaxe
p�e� p� f � e�
Problem in NP-hard; use search:
Beam Search
Greedy SearchInteger Programming
A* Search
![Page 54: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/54.jpg)
54/225 NLP Hal Daumé III, [email protected]
insistent Wednesday may recurred her trips to Libya tomorrow for flying
Cairo 6-4 ( AFP ) - an official announced today in the Egyptian lines company for flying Tuesday is a company " insistent for flying " may resumed a consideration of a day Wednesday tomorrow her trips to Libya of Security Council decision trace international the imposed ban comment .
And said the official " the institution sent a speech to Ministry of Foreign Affairs of lifting on Libya air , a situation her receiving replying are so a trip will pull to Libya a morning Wednesday " .
Egyptair Has Tomorrow to Resume Its Flights to Libya
Cairo 4-6 (AFP) - said an official at the Egyptian Aviation Company today that the company egyptair may resume as of tomorrow, Wednesday its flights to Libya after the International Security Council resolution to the suspension of the embargo imposed on Libya.
" The official said that the company had sent a letter to the Ministry of Foreign Affairs, information on the lifting of the air embargo on Libya, where it had received a response, the first take off a trip to Libya on Wednesday morning ".
20022002 20032003
Progress in machine translation
![Page 55: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/55.jpg)
55/225 NLP Hal Daumé III, [email protected]
Phrase-Based Translation
Aus
tral
ia
Nor
th
dipl
omat
ic
rela
tionsha
ve
that
coun
trie
s
one
of
�� � �� � � � �� ���
Australia is withNorthKorea
isdiplomaticrelations
one of the few
countries
Australia isdiplomaticrelations
withNorthKorea
isone of the few
countries
[Koehn, Och and Marcu, NAACL03]
![Page 56: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/56.jpg)
56/225 NLP Hal Daumé III, [email protected]
Training Phrase-Based MT Systems
Mar
ia no
dab
a
una
bote
fada a la
bru
ja
verd
e
witchgreen
theslapnotdid
Mary
[Koehn, Och and Marcu, NAACL03]
![Page 57: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/57.jpg)
57/225 NLP Hal Daumé III, [email protected]
Decoding Phrase-Based MTMaria no daba una botefada a la bruja verde
Mary did not slap the green witch
� Each step induces a cost attributed to:� Language model probability: p(slap | did not)� T-table probability: p(the | a la) and p(a la | the)� Distortion probability: p(skip 1) [for a la --> verde]� Length penalty� ...
[Koehn, Och and Marcu, NAACL03]
![Page 58: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/58.jpg)
58/225 NLP Hal Daumé III, [email protected]
Hierarchical Phrase-Based MT
�� � �� � � � �� ���
fewcountries
NorthKorea
diplomaticrelationshave with �isAustralia
Australia isNorthKorea
diplomaticrelations
fewcountries
� � �
[Chiang, ACL05]
![Page 59: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/59.jpg)
59/225 NLP Hal Daumé III, [email protected]
Hierarchical Phrase-Based MT
fewcountries
NorthKorea
diplomaticrelationshave with �isAustralia
fewcountries
NorthKorea
diplomaticrelationshave withthatisAustralia
one ofthe
isAustraliafew
countriesNorthKorea
diplomaticrelationshave withthat �
[Chiang, ACL05]
![Page 60: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/60.jpg)
60/225 NLP Hal Daumé III, [email protected]
Syntax for MT
I ate
lunch
P
R
O
VB
D
N
N
N
P
N
P
V
P
S
tabeta
ate
VB
D
hirugohan
lunch
N
N
N
P
watashi
I
P
R
O
N
P
V
P
x yy wo x
S
x yx wa y
I
P
R
O
N
P
ate
lunch
VB
D
N
N
N
P
V
P
wa
ate
lunch
VB
D
N
N
N
P
V
P
watashi wa
atelunc
h
VB
DN
N
N
P
watashi wa wo
watashi wa hirugohan wo tabeta
Kevin Knight, Daniel Marcu, Ignacio Thayer, Jonathan Graehl, Jon May, Steve DeNeefe
![Page 61: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/61.jpg)
61/225 NLP Hal Daumé III, [email protected]
Syntax for MT� Decoding:
� Tree-to-tree/string automata
� CKY parsing algorithm
� Currently only unigram LM and trigram reranking
� No syntax-based LM
� Rule learning:� Parsed English
corpus� Aligned data
(GIZA++)� Extract rules and
assign probabilities
Time
BLE
U
Phrase-based MT
Syntax-based MT
![Page 62: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/62.jpg)
62/225 NLP Hal Daumé III, [email protected]
Traditional NLP
� The process of building computational models for understanding natural language
� INPUT: natural language text (or speech)
� OUTPUT: representation of the meaning of the text
� Sometimes called natural language understanding (NLU)
Big Question: What does �understand� mean?
![Page 66: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/66.jpg)
Title66 Hal Daumé III ([email protected])
07/19/12
NLP as transduction
Task Input Output
MachineTranslation
Ces deux principes se tiennent à
la croisée de la philosophie, de
la politique, de l’économie, de la
sociologie et du droit.
Both principles lie at the
crossroads of philosophy,
politics, economics,
sociology, and law.
Document Summarization
SyntacticAnalysis
The man ate a big sandwich.
...many more...
Argentina was still obsessed with theFalkland Islands even in 1994, 12 years after its defeat in the 74-day warwith Britain. The country's overriding foreign policy aim continued to be winning sovereignty over the islands.
The Falkland islands war, in 1982, was fought between Britain and Argentina.
The man ate a big sandwich
![Page 67: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/67.jpg)
Title67 Hal Daumé III ([email protected])
Structured prediction 101
Learn a function mapping inputs to complex outputs:
I can can a can
Pro Md Vb Dt Nn
Pro Md Vb Dt Vb
Pro Md Vb Dt Md
Sequence Labeling
Pro Md Nn Dt Nn
Pro Md Nn Dt Vb
Pro Md Nn Dt Md
Pro Md Md Dt Nn
Pro Md Md Dt Vb
Parsing
Barack
Obama
McCain
John
he
the President
Input Space Decoding Output Space
Mary did not slap the green witch .
Mary no dio una bofetada a la bruja verda .
Coreference ResolutionMachine Translation
f : X � Y
![Page 68: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/68.jpg)
Title68 Hal Daumé III ([email protected])
Structured prediction 101
Learn a function mapping inputs to complex outputs:
I can can a can
Input Space Decoding Output Space
f : X � Y
![Page 69: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/69.jpg)
Title69 Hal Daumé III ([email protected])
07/19/12
Why is structure important?
� Correlations among outputs� Determiners often precede nouns
� Sentences usually have verbs
� Global coherence� It just doesn't make sense to have three determiners next to
each other
� My objective (aka �loss function�) forces it� Translations should have good sequences of words
� Summaries should be coherent
![Page 70: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/70.jpg)
Title70 Hal Daumé III ([email protected])
Outline: Part II
� Learning to Search� Incremental parsing
� Learning to queue
� Refresher on Markov Decision Processes
� Inverse Reinforcement Learning� Determining rewards given policies
� Maximum margin planning
� Learning by Demonstration� Searn
� Dagger
� Discussion
![Page 72: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/72.jpg)
Title72 Hal Daumé III ([email protected])
What does it mean to learn?
� Informally:� to predict the future based on the past
� Slightly-less-informally:� to take labeled examples and construct a function that will
label them as a human would
� Formally:� Given:
� A fixed unknown distribution D over X*Y
� A loss function over Y*Y
� A finite sample of (x,y) pairs drawn i.i.d. from D
� Construct a function f that has low expected loss with respect to D
![Page 73: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/73.jpg)
Title73 Hal Daumé III ([email protected])
Feature extractors
� A feature extractor � maps examples to vectors
� Feature vectors in NLP are frequently sparse
Dear Sir.
First, I must solicit
your confidence in
this transaction,
this is by virture of
its nature as being
utterly confidencial
and top secret. �
W=dear : 1
W=sir : 1
W=this : 2
...
W=wish : 0
...
MISSPELLED : 2
NAMELESS : 1
ALL_CAPS : 0
NUM_URLS : 0
...
f : X � Y�
![Page 74: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/74.jpg)
Title74 Hal Daumé III ([email protected])
Linear models for binary classification
f1
f2
� Decision boundaryis the set of �uncertain�points
� Linear decisionboundaries arecharacterized byweight vectors
BIAS : -3
free : 4
money : 2
the : 0
...
BIAS : 1
free : 1
money : 1
the : 0
...
�free
money�
x �(x) w i wi �i(x)
![Page 75: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/75.jpg)
Title75 Hal Daumé III ([email protected])
The perceptron
� Inputs = feature values� Params = weights� Sum is the response
� If the response is:� Positive, output +1� Negative, output -1
� When training,update on errors:
S
�1
�2
�3
w1
w2
w3
>0?
75
�Error� when:w=w� y�� x �y w���x ��0
![Page 76: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/76.jpg)
Title76 Hal Daumé III ([email protected])
Why does that update work?
� When , update:wnew=w
old� y��x �y wold���x ��0
wold
wnew
y wnew��x �= y �wold� y��x � ��� x �
= y wold��x �� yy��x ���x �
<0 >0
![Page 77: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/77.jpg)
Title77 Hal Daumé III ([email protected])
Support vector machines
� Explicitly optimizethe margin
� Enforce thatall training pointsare correctlyclassified
f1
f2
maxw
margin s.t. all points arecorrectly classified
maxw
margin s.t.
minw
s.t.
yn w���xn��1 , �n
�w�2 yn w���xn��1 , �n
![Page 78: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/78.jpg)
Title78 Hal Daumé III ([email protected])
Support vector machines with slack
� Explicitly optimizethe margin
� Allow some �noisy�points to bemisclassified f1
f2
minw,�
s.t.
1
2�w�2�C
n�n
yn w���xn�� �n �1 , �n
�n�0 , � n
![Page 79: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/79.jpg)
Title79 Hal Daumé III ([email protected])
Batch versus stochastic optimization
� Batch = read in all the data, then process it
� Stochastic = (roughly) process a bit at a time
minw,�
s.t.
� For n=1..N:
� If
�
1
2�w�2�C
n�n
yn w���xn���n�1
, � n
�n�0 , � n
w=w� yn�� xn�
yn w���xn��0
![Page 80: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/80.jpg)
Title80 Hal Daumé III ([email protected])
Stochastically optimized SVMs
� For n=1..N:
� If
�
�
SOMEMATH
� For n=1..N:
� If
�
SVM Objective
w=w� yn�� xn�
yn w���xn��0yn w���xn��1
Implementation Note:
Weight shrinkage is SLOW.Implement it lazily, at thecost of double storage.
w=�1� 1
CN �ww=w� yn�� xn�
![Page 82: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/82.jpg)
Title82 Hal Daumé III ([email protected])
Perceptron with multiple classes
� Store separate weightvector for each classw1, w2, ..., wK
w1��(x)biggest
w2��(x)biggest
w3��(x)biggest
� For n=1..N:
� Predict:
� If :
y=argmaxk wk���xn�
w y=w y���xn�
y� yn
wyn
=wyn
���xn�
Why does thisdo the right thing??!
![Page 83: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/83.jpg)
Title83 Hal Daumé III ([email protected])
Perceptron with multiple classes v2
� Originally:
w1 w2 w3
w
� For n=1..N:
� Predict:
� If :
� For n=1..N:
� Predict:
� If :
y=argmaxk wk���xn�
w y=w y���xn�
y� yn
wyn
=wyn
���xn�
y=argmaxk w���xn , k �
y� yn
w=w���xn, y �
��� xn, y
n�
![Page 84: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/84.jpg)
Title84 Hal Daumé III ([email protected])
Perceptron with multiple classes v2
� Originally:
w1 w2 w3
w
� For n=1..N:
� Predict:
� If :
� For n=1..N:
� Predict:
� If :
spam_BIAS : 1
spam_free : 1
spam_money : 1
spam_the : 0
...
�free
money�
x �(x,1) �(x,2)
ham_BIAS : 1
ham_free : 1
ham_money : 1
ham_the : 0
...
y=argmaxk wk���xn�
w y=w y���xn�
y� yn
wyn
=wyn
���xn�
y=argmaxk w���xn , k �
y� yn
w=w���xn, y �
��� xn, y
n�
![Page 85: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/85.jpg)
Title85 Hal Daumé III ([email protected])
Features for structured prediction
I can can a can
Pro Md Vb Dt Nn
��x , y �=
I_Pro : 1
can_Md : 1
can_Vb : 1
a_Dt : 1
can_Nn : 1
...
<s>-Pro : 1Pro-Md : 1Md-Vb : 1Vb-Dt : 1Dt-Nn : 1Nn-</s> : 1
has_verb : 1has_nn_lft : 0has_n_lft : 1has_nn_rgt : 1has_n_rgt : 1...
� Allowed to encode anything you want
� Output features, Markov features, other features
![Page 86: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/86.jpg)
Title86 Hal Daumé III ([email protected])
Structured perceptron
� For n=1..N:
� Predict:
� If :
� For n=1..N:
� Predict:
� If :
Enumerationover 1..K
Enumerationover all outputs
y=argmaxk w���xn , k �
y� yn
w=w���xn, y �
��� xn, y
n�
y=argmaxk w���xn , k �
y� yn
w=w���xn, y �
��� xn, y
n�
[Co
llins, E
MN
LP
02
]
![Page 87: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/87.jpg)
Title87 Hal Daumé III ([email protected])
Argmax for sequences
� If we only have output and Markov features, we can use Viterbi algorithm:
(plus some work to account for boundary conditions)
w�[I_Pro] w�[can_Pro] w�[can_Pro]
w�[I_Md] w�[can_Md] w�[can_Md]
w�[I_Vb] w�[can_Vb] w�[can_Vb]
I can can
w�[Pro-Pro]
w�[Pro-Md]
w�[Pro-Vb]
![Page 88: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/88.jpg)
Title88 Hal Daumé III ([email protected])
Structured perceptron as ranking
� For n=1..N:
� Run Viterbi:
� If :
� When does this make an update?
I can can a can
Pro Md Vb Dt Nn
Pro Md Vb Dt Vb
Pro Md Vb Dt Md
Pro Md Nn Dt Nn
Pro Md Nn Dt Md
Pro Md Md Dt Nn
Pro Md Md Dt Vb
y=argmaxk w���xn , k �y� yn w=w���x
n, y ����x
n, y
n�
![Page 89: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/89.jpg)
Title89 Hal Daumé III ([email protected])
From perceptron to margins
minw,�
s.t.
MaximizeMargin
MinimizeErrors
Each point is correctlyclassified, modulo �
s.t.
Each true output is morehighly ranked, modulo �
Responsefor truth
Responsefor other
1
2�w�2�C
n�n
yn w���xn���n�1
, � n
minw,�
1
2�w�2�C
n�n , y
w���xn , yn�
�w���xn , y �
��n�1 ,�n , y� yn
[Ta
ska
r+a
l, JM
LR
05
; Tsh
och
and
aritis
, JM
LR
05
]
![Page 90: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/90.jpg)
Title90 Hal Daumé III ([email protected])
From perceptron to margins
minw,�
s.t.
Each true output is morehighly ranked, modulo �
Responsefor truth
Responsefor other
1
2�w�2�C
n�n , y
w���xn , yn�
�w���xn , y �
��n�1 ,�n , y� yn
[Ta
ska
r+a
l, JM
LR
05
; Tsh
och
and
aritis
, JM
LR
05
]
��x1,y
1�
��x1,y�
��x1,y�
��x1,y�
��x1,y�
��x2,y
2�
��x2,y �
��x2,y �
��x2,y �
![Page 91: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/91.jpg)
Title91 Hal Daumé III ([email protected])
Ranking margins
� Some errors are worse than others...
I can can a can
Pro Md Vb Dt Nn
Pro Md Vb Dt Vb
Pro Md Vb Dt Md
Pro Md Nn Dt Nn
Pro Md Nn Dt Md
Pro Md Md Dt Nn
Pro Md Md Dt Vb
Margin of one
[Ta
ska
r+a
l, JM
LR
05
; Tsh
och
and
aritis
, JM
LR
05
]
![Page 92: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/92.jpg)
Title92 Hal Daumé III ([email protected])
Accounting for a loss function
� Some errors are worse than others...
I can can a can
Pro Md Vb Dt Nn
Margin of l(y,y')
[Ta
ska
r+a
l, JM
LR
05
; Tsh
och
and
aritis
, JM
LR
05
]
Pro Md Vb Dt Vb
Pro Md Vb Nn Md
Nn Md Vb Nn Md
Pro Md Md Dt NnPro Md Nn Dt Nn
Nn Md Vb Dt MdPro Vb Nn Dt Nn
Nn Md Vb Md MdPro Vb Nn Vb Nn
![Page 93: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/93.jpg)
Title93 Hal Daumé III ([email protected])
Accounting for a loss function
w���xn , yn��w���xn , y���n
[Ta
ska
r+a
l, JM
LR
05
; Tsh
och
and
aritis
, JM
LR
05
]
��x1,y
1�
��x1,y�
��x1,y�
��x1,y�
��x1,y�
��x2,y
2�
��x2,y �
��x2,y �
��x2,y �
w���xn , yn��w���xn , y ���n
�1 �l � yn , y �
is equivalent to
� y , w���xn , yn��w���xn , y ���n�l � yn , y �
max y w���xn , yn��w���xn , y ���n�l � yn , y �
![Page 94: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/94.jpg)
Title94 Hal Daumé III ([email protected])
Augmented argmax for sequences
� Add �loss� to each wrong node!
w�[I_Pro] w�[can_Pro] w�[can_Pro]
w�[I_Md] w�[can_Md] w�[can_Md]
w�[I_Vb] w�[can_Vb] w�[can_Vb]
I can can
w�[Pro-Pro]
w�[Pro-Md]
w�[Pro-Vb]
+1
+1 +1
+1
+1
+1
[Ta
ska
r+a
l, JM
LR
05
; Tsh
och
and
aritis
, JM
LR
05
]
What are weassuming here??!
![Page 95: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/95.jpg)
Title95 Hal Daumé III ([email protected])
Stochastically optimizing Markov nets
SOMEMATH
M3N Objective
� For n=1..N:
� Viterbi:
� If :
� For n=1..N:
� Augmented Viterbi:
� If :
�
y=argmaxk w���xn , k �y� yn
w=w���xn, y �
��� xn, y
n�
y=argmaxk w���xn , k �
�l � yn , k �y� yn
w=w���xn, y �
��� xn, y
n�
w=�1� 1
CN �w
[Ratliff+
al, A
ISta
ts07
]
![Page 96: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/96.jpg)
Title96 Hal Daumé III ([email protected])
07/19/12
Stacking
� Structured models: accurate but slow
� Independent models: less accurate but fast
� Stacking: multiple independent models
Output Labels
Input Features
Output Labels
Input Features
Output Labels
Input Features
Output Labels 2
![Page 97: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/97.jpg)
Title97 Hal Daumé III ([email protected])
07/19/12
Training a stacked model
� Train independentclassifier f1 on
input features
� Train independentclassifier f2 on
input features + f1's output
� Danger: overfitting!
� Solution: cross-validation
Output Labels
Input Features
Output Labels 2
![Page 98: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/98.jpg)
Title98 Hal Daumé III ([email protected])
07/19/12
Do we really need structure?
� Structured models: accurate but slow
� Independent models: less accurate but fast
� Goal: transfer power to get fast+accurate
� Questions: are independent models...
� � expressive enough? (approximation error)
� � easy to learn? (estimation error)
Output Labels
Input Features
Output Labels
Input Features
Output Labels
Input Features
[Lia
ng
+D
+K
lein
, ICM
L0
8]
![Page 99: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/99.jpg)
Title99 Hal Daumé III ([email protected])
07/19/12
�Compiling� structure out[L
iang
+D
+K
lein
, ICM
L0
8]
LabeledData
CRF(f1) POS: 95.0%NER: 75.3%
f1 = words/prefixes/suffixes/forms
IRL(f1) POS: 91.7%NER: 69.1%
![Page 100: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/100.jpg)
Title100 Hal Daumé III ([email protected])
07/19/12
�Compiling� structure out[L
iang
+D
+K
lein
, ICM
L0
8]
LabeledData
Auto-LabeledData
CRF(f1) POS: 95.0%NER: 75.3%
f1 = words/prefixes/suffixes/forms
f2 = f1 applied to a larger window
IRL(f1) POS: 91.7%NER: 69.1%
IRL(f2) POS: 94.4%NER: 66.2%
CompIRL(f2) POS: 95.0%NER: 72.7%
![Page 101: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/101.jpg)
Title101 Hal Daumé III ([email protected])
07/19/12
Decomposition of errors[L
iang
+D
+K
lein
, ICM
L0
8]
CRF(f1): pC
� � �
IRL(f�): pA*
marginalized CRF(f1): pMC
IRL(f2): p1*
coherence
nonlinearities
global information
Theorem:KL(pC || p1*) = KL(pC || pMC) + KL(pMC || pA*) + KL(pA* || p1*)
Sum of MI on edgesPOS=.003 (95.0% 95.0%)�NER=.009 (76.3% 76.0%)�
Train a truncated CRFNER: 76.0% 72.7%�
Train a marginalized CRFNER: 76.0% 76.0%�
![Page 102: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/102.jpg)
Title102 Hal Daumé III ([email protected])
07/19/12
Structure compilation results
4 14 24 44 84 16476
78
80
82
84
86
88
Structured
Independent
2 4 8 16 32 6466
68
70
72
74
76
78
2 4 8 16 32 6493
94
95
96
Part of speech Named Entity Parsing
[Lia
ng
+D
+K
lein
, ICM
L0
8]
![Page 104: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/104.jpg)
Title104 Hal Daumé III ([email protected])
07/19/12
Argmax is hard!
� Classic formulation of structured prediction:
something we learnto make �good� x,y pairsscore highly
� At test time:
� Combinatorial optimization problem� Efficient only in very limiting cases
� Solved by heuristic search: beam + A* + local search
f �x� = argmaxy�Y
score �x , y �
score �x , y� =
![Page 105: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/105.jpg)
Title105 Hal Daumé III ([email protected])
07/19/12
Argmax is hard!
� Classic formulation of structured prediction:
something we learnto make �good� x,y pairsscore highly
� At test time:
� Combinatorial optimization problem� Efficient only in very limiting cases
� Solved by heuristic search: beam + A* + local search
f �x� = argmaxy�Y
score �x , y �
score �x , y� =
Order these words: bart better I madonna say than ,
[Soricut, PhD Thesis, USC 2007]
![Page 106: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/106.jpg)
Title106 Hal Daumé III ([email protected])
07/19/12
Argmax is hard!
� Classic formulation of structured prediction:
something we learnto make �good� x,y pairsscore highly
� At test time:
� Combinatorial optimization problem� Efficient only in very limiting cases
� Solved by heuristic search: beam + A* + local search
f �x� = argmaxy�Y
score �x , y �
score �x , y� =
Order these words: bart better I madonna say than ,
Best search (32.3): I say better than bart madonna ,
Original (41.6): better bart than madonna , I say
[Soricut, PhD Thesis, USC 2007]
![Page 107: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/107.jpg)
Title107 Hal Daumé III ([email protected])
07/19/12
Argmax is hard!
� Classic formulation of structured prediction:
something we learnto make �good� x,y pairsscore highly
� At test time:
� Combinatorial optimization problem� Efficient only in very limiting cases
� Solved by heuristic search: beam + A* + local search
f �x� = argmaxy�Y
score �x , y �
score �x , y� =
Order these words: bart better I madonna say than ,
Best search (32.3): I say better than bart madonna ,
Original (41.6): better bart than madonna , I say
Best search (51.6): and so could really be a neural
apparently thought things as
dissimilar firing two identical
[Soricut, PhD Thesis, USC 2007]
![Page 108: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/108.jpg)
Title108 Hal Daumé III ([email protected])
07/19/12
Argmax is hard!
� Classic formulation of structured prediction:
something we learnto make �good� x,y pairsscore highly
� At test time:
� Combinatorial optimization problem� Efficient only in very limiting cases
� Solved by heuristic search: beam + A* + local search
f �x� = argmaxy�Y
score �x , y �
score �x , y� =
Order these words: bart better I madonna say than ,
Best search (32.3): I say better than bart madonna ,
Original (41.6): better bart than madonna , I say
Best search (51.6): and so could really be a neural
apparently thought things as
dissimilar firing two identical
Original (64.3): could two things so apparently
dissimilar as a thought and neural
firing really be identical
[Soricut, PhD Thesis, USC 2007]
![Page 109: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/109.jpg)
Title109 Hal Daumé III ([email protected])
Incremental parsing, early 90s style
I / Pro can / Md can / Vb a / Dt can / Nn
NP NP
VP
S
UnaryLeft
Left Up
Right
Right
Right
Left
Train a classifierto make decisions
[Ma
ge
rman
, AC
L9
5]
![Page 110: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/110.jpg)
Title110 Hal Daumé III ([email protected])
Incremental parsing, mid 2000s style
I / Pro can / Md can / Vb a / Dt can / Nn
NP
VP
S
NP
Train a classifierto make decisions
[Co
llsn
s+
Ro
ark
, AC
L0
4]
![Page 111: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/111.jpg)
Title111 Hal Daumé III ([email protected])
Learning to beam-search
I / Pro can / Md can / Vb a / Dt can / Nn
NP
VP
S
NP
� For n=1..N:� Viterbi:
� If :
Beam-search
y=argmaxk w���xn , k �y� yn
w=w���xn, y �
���xn, y
n�
[Co
llins+
Ro
ark
, AC
L0
4]
![Page 112: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/112.jpg)
Title112 Hal Daumé III ([email protected])
Learning to beam-search
I / Pro can / Md can / Vb a / Dt can / Nn
NP
VP
S
NP
� For n=1..N:� Viterbi:
� If :y=argmaxk w���xn , k �
y� yn
w=w���xn, y �
���xn, y
n�
Beam-search
� For n=1..N:� Run beam search until
truth falls out of beam� Update weights
immediately!
[Co
llins+
Ro
ark
, AC
L0
4]
![Page 113: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/113.jpg)
Title113 Hal Daumé III ([email protected])
Learning to beam-search
I / Pro can / Md can / Vb a / Dt can / Nn
NP
VP
S
NP
� For n=1..N:� Viterbi:
� If :
Beam-search
� For n=1..N:� Run beam search until
truth falls out of beam� Update weights
immediately!
� For n=1..N:� Run beam search until
truth falls out of beam� Update weights
immediately!� Restart at truth
[D+
Ma
rcu
, ICM
L0
5; X
u+
al, J
ML
R0
9]
![Page 115: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/115.jpg)
Title115 Hal Daumé III ([email protected])
Generic Search Formulation
� Search Problem:� Search space
� Operators
� Goal-test function
� Path-cost function
� Search Variable:� Enqueue function
� nodes := MakeQueue(S0)
� while nodes is not empty� node :=
RemoveFront(nodes)
� if node is a goal state return node
� next := Operators(node)
� nodes := Enqueue(nodes, next)
� fail
Varying the Enqueue function can give us DFS, BFS, beam search, A* search, etc...
[D+
Ma
rcu
, ICM
L0
5; X
u+
al, J
ML
R0
9]
![Page 116: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/116.jpg)
Title116 Hal Daumé III ([email protected])
Online Learning Framework (LaSO)
� nodes := MakeQueue(S0)
� while nodes is not empty� node := RemoveFront(nodes)
� if none of {node} � nodes is y-good or node is a goal & not y-good
� sibs := siblings(node, y)
� w := update(w, x, sibs, {node} � nodes)
� nodes := MakeQueue(sibs)
� else� if node is a goal state return w
� next := Operators(node)
� nodes := Enqueue(nodes, next)
Monotonicity: for anynode, we can tell if it
can lead to the correct solution or not
If we erred... Where should we have gone?
Update our weights based on
the good and the bad choicesContinue search...
[D+
Ma
rcu
, ICM
L0
5; X
u+
al, J
ML
R0
9]
![Page 117: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/117.jpg)
Title117 Hal Daumé III ([email protected])
Search-based Margin
� The margin is the amount by which we are correct:
� Note that the margin and hence linear separability is also a function of the search algorithm!
�uT��x , g
1�
�uT��x , g
2�
�uT��x ,b
1�
�uT��x ,b
2�
�u
�
[D+
Ma
rcu
, ICM
L0
5; X
u+
al, J
ML
R0
9]
![Page 118: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/118.jpg)
Title118 Hal Daumé III ([email protected])
Syntactic chunking Results
Training Time (minutes)
F-
Score
[Collins 2002]
[Zhang+Damerau+Johnson
2002]; timing unknown
33 min
22 min
24 min
4 min
[D+
Ma
rcu
, ICM
L0
5; X
u+
al, J
ML
R0
9]
![Page 119: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/119.jpg)
Title119 Hal Daumé III ([email protected])
Tagging+chunking results
Training Time (hours) [log scale]
Join
t ta
ggin
g/c
hunkin
g
accura
cy
[Sutton+McCallum 2004]
23 min
7 min3 min
1 min
[D+
Ma
rcu
, ICM
L0
5; X
u+
al, J
ML
R0
9]
![Page 120: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/120.jpg)
Title120 Hal Daumé III ([email protected])
Variations on a beam
� Observation:
� We needn't use the same beam size for training and decoding
� Varying these values independently yields:
1 5 10 25 501 93.9 92.8 91.9 91.3 90.9
5 90.5 94.3 94.4 94.1 94.1
10 89.5 94.3 94.4 94.2 94.2
25 88.7 94.2 94.5 94.3 94.3
50 88.4 94.2 94.4 94.2 94.4
Decoding Beam
Train
ing
Beam
[D+
Ma
rcu
, ICM
L0
5; X
u+
al, J
ML
R0
9]
![Page 121: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/121.jpg)
Title121 Hal Daumé III ([email protected])
What if our model sucks?
� Sometimes our model cannot produce the �correct� output� canonical example: machine translation
ModelModelOutputsOutputs
GoodGoodOutputsOutputs
Reference
[Och
, AC
L0
3; L
ian
g+
al, A
CL
06
]
CurrentHypothesis
Bestachievableoutput
�Local� update
�Bold� update
N-best listor �optimaldecoding�
or ...
![Page 122: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/122.jpg)
Title122 Hal Daumé III ([email protected])
Local versus bold updating...
Monotonic Distortion32.5
33
33.5
34
34.5
35
35.5
Machine Translation Performance (Bleu)
Bold
Local
Pharoah
![Page 124: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/124.jpg)
Title124 Hal Daumé III ([email protected])
Reinforcement learning
� Basic idea:� Receive feedback in the form of rewards� Agent�s utility is defined by the reward function� Must learn to act to maximize expected rewards� Change the rewards, change the learned behavior
� Examples:� Playing a game, reward at the end for outcome� Vacuuming, reward for each piece of dirt picked up� Driving a taxi, reward for each passenger delivered
![Page 125: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/125.jpg)
Title125 Hal Daumé III ([email protected])
Markov decision processes
What are the values (expected future rewards) of states and actions?
5
1
3
2
4
0.6
0.4
0.1
0.9
1.0
Q(s,a1)* = 30
Q(s,a2)* = 23
Q(s,a3)* = 17
V(s)* = 30
![Page 126: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/126.jpg)
Title126 Hal Daumé III ([email protected])
Markov Decision Processes
� An MDP is defined by:� A set of states s � S� A set of actions a � A� A transition function T(s,a,s�)
� Prob that a from s leads to s�� i.e., P(s� | s,a)� Also called the model
� A reward function R(s, a, s�) � Sometimes just R(s) or R(s�)
� A start state (or distribution)� Maybe a terminal state
� MDPs are a family of non-deterministic search problems
� Total utility is one of:
or t
rt t
�tr t
![Page 127: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/127.jpg)
Title127 Hal Daumé III ([email protected])
Solving MDPs
� In deterministic single-agent search problem, want an optimal plan, or sequence of actions, from start to a goal
� In an MDP, we want an optimal policy p(s)
� A policy gives an action for each state� Optimal policy maximizes expected if followed� Defines a reflex agent
Optimal policy when R(s, a, s�) = -0.04 for all non-terminals s
![Page 128: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/128.jpg)
Title128 Hal Daumé III ([email protected])
Example Optimal Policies
R(s) = -2.0R(s) = -0.4
R(s) = -0.03R(s) = -0.01
![Page 129: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/129.jpg)
Title129 Hal Daumé III ([email protected])
Optimal Utilities
� Fundamental operation: compute the optimal utilities of states s (all at once)
� Why? Optimal values define optimal policies!
� Define the utility of a state s:V*(s) = expected return starting in
s and acting optimally
� Define the utility of a q-state (s,a):Q*(s,a) = expected return starting
in s, taking action a and thereafter acting optimally
� Define the optimal policy:�*(s) = optimal action from state s
a
s
s, a
s,a,s�
s�
![Page 130: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/130.jpg)
Title130 Hal Daumé III ([email protected])
The Bellman Equations
� Definition of utility leads to a simple one-step lookahead relationship amongst optimal utility values:
Optimal rewards = maximize over first action and then follow optimal policy
� Formally:
a
s
s, a
s,a,s�
s�
![Page 131: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/131.jpg)
Title131 Hal Daumé III ([email protected])
Solving MDPs / memoized recursion
� Recurrences:
� Cache all function call results so you never repeat work
� What happened to the evaluation function?
![Page 132: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/132.jpg)
Title132 Hal Daumé III ([email protected])
Q-Value Iteration
� Value iteration: iterate approx optimal values� Start with V0
*(s) = 0, which we know is right (why?)� Given Vi
*, calculate the values for all states for depth i+1:
� But Q-values are more useful!� Start with Q0
*(s,a) = 0, which we know is right (why?)� Given Qi
*, calculate the q-values for all q-states for depth i+1:
![Page 133: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/133.jpg)
Title133 Hal Daumé III ([email protected])
RL = Unknown MDPs
� If we knew the MDP (i.e., the reward function and transition function):
� Value iteration leads to optimal values
� Will always converge to the truth
� Reinforcement learning is what we do when we do not know the MDP
� All we observe is a trajectory
� (s1,a1,r1, s2,a2,r2, s3,a3,r3, �)
� Many algorithms exist for this problem; see Sutton+Barto's excellent book!
![Page 134: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/134.jpg)
Title134 Hal Daumé III ([email protected])
Q-Learning
� Learn Q*(s,a) values� Receive a sample (s,a,s�,r)
� Consider your old estimate:
� Consider your new sample estimate:
� Incorporate the new estimate into a running average:
![Page 135: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/135.jpg)
Title135 Hal Daumé III ([email protected])
Exploration / Exploitation
� Several schemes for forcing exploration� Simplest: random actions (e greedy)
� Every time step, flip a coin
� With probability e, act randomly
� With probability 1-e, act according to current policy
� Problems with random actions?� You do explore the space, but keep thrashing around once
learning is done
� One solution: lower e over time� Another solution: exploration functions
![Page 136: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/136.jpg)
Title136 Hal Daumé III ([email protected])
Q-Learning
� In realistic situations, we cannot possibly learn about every single state!
� Too many states to visit them all in training� Too many states to hold the q-tables in memory
� Instead, we want to generalize:� Learn about some small number of training states from
experience� Generalize that experience to new, similar states:
� Very simple stochastic updates:
![Page 137: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/137.jpg)
Title137 Hal Daumé III ([email protected])
InverseReinforcement Learning (aka Inverse Optimal Control)
![Page 138: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/138.jpg)
Title138 Hal Daumé III ([email protected])
Inverse RL: Task
� Given:
� measurements of an agent's behavior over time, in a variety of circumstances
� if needed, measurements of the sensory inputs to that agent
� if available, a model of the environment.
� Determine: the reward function being optimized
� Proposed by [Kalman68]
� First solution, by [Boyd94]
![Page 139: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/139.jpg)
Title139 Hal Daumé III ([email protected])
Why inverse RL?
� Computational models for animal learning
� �In examining animal and human behavior we must consider the reward function as an unknown to be ascertained through empirical investigation.�
� Agent construction� �An agent designer [...] may only have a very rough idea of
the reward function whose optimization would generate 'desirable' behavior.�
� eg., �Driving well�
� Multi-agent systems and mechanism design� learning opponents� reward functions that guide their actions
to devise strategies against them
![Page 140: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/140.jpg)
Title140 Hal Daumé III ([email protected])
IRL from Sample Trajectories[N
g+
Ru
sse
ll, ICM
L0
0]
� Optimal policy available through sampled trajectories (eg., driving a car)
� Want to find Reward function that makes this policy look as good as possible
� Write so the reward is linear
and be the value of the starting state
How good does the�optimal policy� look?
maxw
k=1
K
f �V w
��
�s0��V w
�k �s0��
Rw �s�=w��s�
V w
� �s0�
How good does thesome other policy look?
Warning: need to becareful to avoidtrivial solutions!
![Page 141: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/141.jpg)
Title141 Hal Daumé III ([email protected])
Apprenticeship Learning via IRL
� For t = 1,2,�
� Inverse RL step:
Estimate expert�s reward function R(s)= wTf(s) such that under R(s) the expert performs better than all
previously found policies {pi}.
� RL step:
Compute optimal policy pt for the estimated reward w
[Ab
be
el+
Ng
, ICM
L0
4]
![Page 142: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/142.jpg)
Title142 Hal Daumé III ([email protected])
Car Driving Experiment
� No explicit reward function at all!
� Expert demonstrates proper policy via 2 min. of driving time on simulator (1200 data points).
� 5 different �driver types� tried.
� Features: which lane the car is in, distance to closest car in current lane.
� Algorithm run for 30 iterations, policy hand-picked.
� Movie Time! (Expert left, IRL right) [Ab
be
el+
Ng
, ICM
L0
4]
![Page 145: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/145.jpg)
Title145 Hal Daumé III ([email protected])
Distribution over trajectories:P( )�
Match the reward of observed behavior:
�� P( ) f� � = fdem
Maximizing the causal entropy over trajectories given stochastic outcomes: max H(P( )||O)�
(Condition on random uncontrolled outcomes, but only after they happen)
As uniform as possible
Maxent IRL[Z
ieb
art+
al, A
AA
I08
]
![Page 146: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/146.jpg)
Title146 Hal Daumé III ([email protected])
LengthSpeedRoad TypeLanes
AccidentsConstructionCongestionTime of day
25 Taxi Drivers
Over 100,000 miles
Data collection[Z
ieb
art+
al, A
AA
I08
]
![Page 148: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/148.jpg)
Title148 Hal Daumé III ([email protected])
Planning as structured prediction[R
atliff+
al, N
IPS
05
]
![Page 149: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/149.jpg)
Title149 Hal Daumé III ([email protected])
Maximum margin planning
� Let �(s,a) denote the probability of reaching q-state (s,a) under current model w
maxw
margin s.t. planner run with wyields human output
minw
s.t.��s , a�w��� xn , s , a�
� ��s , a�w���xn , s , a��1
, �n , s , a
1
2�w�2
[Ratliff+
al, N
IPS
05
]
Q-state visitationfrequency by human
Q-state visitationfrequency by planner
All trajectories,and all q-states
![Page 150: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/150.jpg)
Title150 Hal Daumé III ([email protected])
Optimizing MMP
SOMEMATH
MMP Objective
� For n=1..N:
� Augmented planning:Run A* on current (augmented) cost mapto get q-state visitation frequencies
� Update:
� Shrink:
w=w� s
a
[ ��s , a����s , a� ]�� xn, s ,a�
w=�1� 1
CN �w
��s , a�
[Ratliff+
al, N
IPS
05
]
![Page 152: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/152.jpg)
Title152 Hal Daumé III ([email protected])
Parsing via inverse optimal control
� State space = all partial parse trees over the full sentence labeled �S�
� Actions: take a partial parse and split it anywhere in the middle
� Transitions: obvious
� Terminal states: when there are no actions left
� Reward: parse score at completion
[Ne
u+
Szep
evari, M
LJ0
9]
![Page 153: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/153.jpg)
Title153 Hal Daumé III ([email protected])
Parsing via inverse optimal control
Small Medium Large60
65
70
75
80
85
90
Maximum Likelihood
Projection Perceptron Appren-ticeship Learning
Maximum Margin
Maximum Entropy
Policy Matching
[Ne
u+
Szep
evari, M
LJ0
9]
![Page 155: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/155.jpg)
Title155 Hal Daumé III ([email protected])
07/19/12
Integrating search and learning Input: Le homme mange l' croissant.
Output: The man ate a croissant.
Hyp: The man ate
Cov: Le homme mange
l' croissant.
Hyp: The man ate a
Cov: Le homme mange
l' croissant.
Hyp: The man ate a
Cov: Le homme mange
l' croissant.
Hyp: The man ate happy
Cov: Le homme mange
l' croissant.
Hyp: The man ate a fox
Cov: Le homme mange
l' croissant.
Hyp: The man ate a croissant
Cov: Le homme mange
l' croissant.
Classifier 'h'
[D+
Ma
rcu
, ICM
L0
5; D
+L
an
gfo
rd+
Ma
rcu
, ML
J0
9]
![Page 156: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/156.jpg)
Title156 Hal Daumé III ([email protected])
07/19/12
Reducing search to classification
� Natural chicken and egg problem:� Want h to get low expected future loss
� � on future decisions made by h
� � and starting from states visited by h
� Iterative solution Input: Le homme mange l' croissant.
Output: The man ate a croissant.
Hyp: The man ate
Cov: Le homme mange
l' croissant.
Hyp: The man ate a
Cov: Le homme mange
l' croissant.
Hyp: The man ate a
Cov: Le homme mange
l' croissant.
Hyp: The man ate happy
Cov: Le homme mange
l' croissant.
Hyp: The man ate a fox
Cov: Le homme mange
l' croissant.
Hyp: The man ate a croissant
Cov: Le homme mange
l' croissant.
h(t-1)
h(t)
Loss = 0Loss = 0
Loss = 1.8
Loss = 1.2
Loss = 0Loss = 0.5
Loss = 0Loss = 0
[D+
La
ng
ford
+M
arc
u, M
LJ0
9]
![Page 157: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/157.jpg)
Title157 Hal Daumé III ([email protected])
Reduction for Structured Prediction
� Idea: view structured prediction in light of search
A
N
D
V
R
A
N
D
V
R
A
N
D
V
R
Each step here looks like it could be represented as a weighted multi-class problem.
Can we formalize this idea?
I sing merrily
Loss function: L([N V R], [N V R]) = 0 L([N V R], [N V V]) = 1/3 ...
[D+
La
ng
ford
+M
arc
u, M
LJ0
9]
![Page 158: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/158.jpg)
Title158 Hal Daumé III ([email protected])
Reducing Structured Prediction
A
N
D
V
R
A
N
D
V
R
A
N
D
V
R
I sing merrily
Key Assumption: Optimal Policy for training data
Weak!
Given: input, true output and state;Return: best successor state
Desired: good policy on test data (i.e., given only input string)
[D+
La
ng
ford
+M
arc
u, M
LJ0
9]
![Page 159: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/159.jpg)
Title159 Hal Daumé III ([email protected])
How to Learn in Search
Idea: Train based only on optimal path (ala MEMM)
Better Idea: Train based only on optimal policy, then train based on optimal policy + a little learned policy then train based on optimal policy + a little more learned policy then ... eventually only use learned policy
[D+
La
ng
ford
+M
arc
u, M
LJ0
9]
![Page 160: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/160.jpg)
Title160 Hal Daumé III ([email protected])
How to Learn in Search
I sing merrily
� Translating DSP into Searn(DSP, loss, �):
� Draw x ~ DSP
� Run � on x, to get a path
� Pick position uniformly on path
� Generate example with costs given by expected (wrt �) completion costs for �loss�
sing
c=0
c=1
c=2
c=1
c=1
[D+
La
ng
ford
+M
arc
u, M
LJ0
9]
![Page 161: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/161.jpg)
Title161 Hal Daumé III ([email protected])
Searn
Algorithm: Searn-Learn(A, DSP, loss, �*, �)1: Initialize: � = �*2: while not converged do3: Sample: D ~ Searn(DSP, loss, �)4: Learn: h � A(D)5: Update: � � (1-�) � + � h6: end while7: return � without �*
Theorem: L(�) � L(�*) + 2 avg(loss) T ln T + c(1+ln T) / T
Ingredients for Searn:Input space (X) and output space (Y), data from XLoss function (loss(y, y')) and features�Optimal� policy �*(x, y0)
[D+
La
ng
ford
+M
arc
u, M
LJ0
9]
![Page 162: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/162.jpg)
Title162 Hal Daumé III ([email protected])
07/19/12
But what about demonstrations?
� What did we assume before?
� We can have a human (or system) demonstrate, thus giving us an optimal policy
Key Assumption: Optimal Policy for training data
Given: input, true output and state;Return: best successor state
![Page 163: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/163.jpg)
Title163 Hal Daumé III ([email protected])
Steering in [-1,1]
Input: Output:
Resized to 25x19 pixels (1425 features)
3d racing game (TuxKart)[R
oss+
Go
rdo
n+
Ba
gn
ell, A
ISta
ts11
]
![Page 164: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/164.jpg)
Title164 Hal Daumé III ([email protected])
07/19/12
DAgger: Dataset Aggregation
��**
� Collect trajectories from expert �*
� Dataset D0 = { ( s, �*(s) ) | s ~ �* }
� Train �1 on D0
� Collect new trajectories from �1
� But let the expert steer!
� Dataset D1 = { ( s, �*(s) ) | s ~ �1 }
� Train �2 on D0 � D1
� In general:
� Dn = { ( s, �*(s) ) | s ~ �n }
� Train �n on � i<n Di
��11
��22
If N = T log T,
L(�n) < T �N + O(1)
for some n
[Ro
ss+
Go
rdo
n+
Ba
gn
ell, A
ISta
ts11
]
![Page 165: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/165.jpg)
Title165 Hal Daumé III ([email protected])
Steering in [-1,1]
Input:Output:
Resized to 25x19 pixels (1425 features)
Experiments: Racing Game[R
oss+
Go
rdo
n+
Ba
gn
ell, A
ISta
ts11
]
![Page 166: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/166.jpg)
Title166 Hal Daumé III ([email protected])
Bette
r
Average falls per lap[R
oss+
Go
rdo
n+
Ba
gn
ell, A
ISta
ts11
]
![Page 167: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/167.jpg)
Title167 Hal Daumé III ([email protected])
Jump in {0,1}Right in {0,1}Left in {0,1}Speed in {0,1}
Extracted 27K+ binary featuresfrom last 4 observations(14 binary features for every cell)
Output:Input:
From Mario AI competition 2009
Super Mario Bros.[R
oss+
Go
rdo
n+
Ba
gn
ell, A
ISta
ts11
]
![Page 168: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/168.jpg)
Title168 Hal Daumé III ([email protected])
Training (expert)[R
oss+
Go
rdo
n+
Ba
gn
ell, A
ISta
ts11
]
![Page 169: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/169.jpg)
Title169 Hal Daumé III ([email protected])
Test-time execution (classifier)[R
oss+
Go
rdo
n+
Ba
gn
ell, A
ISta
ts11
]
![Page 170: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/170.jpg)
Title170 Hal Daumé III ([email protected])
Test-time execution (Dagger)[R
oss+
Go
rdo
n+
Ba
gn
ell, A
ISta
ts11
]
![Page 171: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/171.jpg)
Title171 Hal Daumé III ([email protected])
Bett
er
Average distance per stage[R
oss+
Go
rdo
n+
Ba
gn
ell, A
ISta
ts11
]
![Page 172: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/172.jpg)
Title172 Hal Daumé III ([email protected])
07/19/12
Perceptron vs. LaSO vs. Searn
Incremental perceptron
LaSO
Searn / DAgger
Un-learnable decision
![Page 174: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/174.jpg)
Title174 Hal Daumé III ([email protected])
Relationship between SP and IRL
� Formally, they're (nearly) the same problem� See humans performing some task
� Define some loss function
� Try to mimic the humans
� Difference is in philosophy:� (I)RL has little notion of beam search or dynamic
programming
� SP doesn't think about separating reward estimation from solving the prediction problem
� (I)RL has to deal with stochastiticity in MDPs
![Page 175: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/175.jpg)
Title175 Hal Daumé III ([email protected])
Important Concepts
� Search and loss-augmented search for margin-based methods
� Bold versus local updates for approximate search
� Training on-path versus off-path
� Stochastic versus deterministic worlds
� Q-states / values
� Learning reward functions vs. matching behavior
![Page 176: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/176.jpg)
176/225 NLP Hal Daumé III, [email protected]
Language comprehension
But man is not destined to vanish. He can be killed, but he cannot be destroyed, because his soul is deathless and his spirit is irrepressible. Therefore, though the situation seems dark in the context of the confrontation between the superpowers, the silver lining is provided by amazing phenomenon that the very nations which have spent incalculable resources and energy for the production of deadly weapons are desperately trying to find out how they might never be used. They threaten each other, intimidate each other and go to the brink, but before the total hour arrives they withdraw from the brink.
The main point from the author's view is:
1. Man's soul and spirit can not be destroyed by superpowers.
2. Man's destiny is not fully clear or visible. 3. Man's soul and spirit are immortal. 4. Man's safety is assured by the delicate balance
of power in terms of nuclear weapons. 5. Human society will survive despite the
serious threat of total annihilation.
![Page 177: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/177.jpg)
177/225 NLP Hal Daumé III, [email protected]
Language
The World
Cognitive scientistscall this �grounding�
"Understanding� isbeing able to interactwith the world basedon language inputs
Language and the world
![Page 179: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/179.jpg)
179/225 NLP Hal Daumé III, [email protected]
WalkthroughsYou will appear on screen, and you will see that you have no weapon in hand whatsoever! Walk north to enter the open cave above you, and an Old Man will be inside. He addresses you with the following, insightful sentence: "It is dangerous to go alone! Take this." At this point you should step forward to grasp the Wooden Sword (used with the A Button), and then head back into the overworld.
![Page 180: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/180.jpg)
180/225 NLP Hal Daumé III, [email protected]
WalkthroughsYou will appear on screen, and you will see that you have no weapon in hand whatsoever! Walk north to enter the open cave above you, and an Old Man will be inside. He addresses you with the following, insightful sentence: "It is dangerous to go alone! Take this." At this point you should step forward to grasp the Wooden Sword (used with the A Button), and then head back into the overworld.
Go to the small opening in the rock face on your right to arrive at H9, where you should kill the four Red Octoroks before going through the opening on the right. H10 has 5 Blue Tektites waiting for you, so be sure to kill them off before leaving the screen through the right as these fellows are an excellent source of Rupees, Fairies, and Hearts. H11 has four Blue Tektites, and after going right to ...
![Page 181: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/181.jpg)
181/225 NLP Hal Daumé III, [email protected]
Why is this so hard?
You will appear on screen, and you will see that you have no weapon in hand whatsoever!
Walk north to enter the open cave above you, and an Old Man will be inside.
He addresses you with the following, insightful sentence: "It is dangerous to go alone! Take this."
At this point you should step forward to grasp the Wooden Sword (used with the A Button), and then head back into the overworld.
![Page 182: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/182.jpg)
182/225 NLP Hal Daumé III, [email protected]
Walk north to enter the cave...1111111001111111
1111213001111111
1113000001111111
1130000001111111
1300000004111111
0000000000000000
5600000000000011
1100000000000011
1100000000000011
1111111111111111
1111111111111111
a=8 b=8 c=0 d=0
e=0 f=3 g=3 h=8
i=6 j=0 k=0
L L L U U U UL L U L U U UU U U L L U U
L U L U L U UU L L U U L U
![Page 183: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/183.jpg)
Title183 Hal Daumé III ([email protected])
![Page 184: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/184.jpg)
Title184 Hal Daumé III ([email protected])
![Page 185: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/185.jpg)
Title185 Hal Daumé III ([email protected])
![Page 186: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/186.jpg)
Title186 Hal Daumé III ([email protected])
![Page 187: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/187.jpg)
Title187 Hal Daumé III ([email protected])
![Page 188: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/188.jpg)
Title188 Hal Daumé III ([email protected])
![Page 189: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/189.jpg)
Title189 Hal Daumé III ([email protected])
![Page 190: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/190.jpg)
Title190 Hal Daumé III ([email protected])
![Page 191: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/191.jpg)
Title191 Hal Daumé III ([email protected])
![Page 192: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/192.jpg)
Title192 Hal Daumé III ([email protected])
![Page 193: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/193.jpg)
Title193 Hal Daumé III ([email protected])
![Page 194: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/194.jpg)
Title194 Hal Daumé III ([email protected])
![Page 195: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/195.jpg)
Title195 Hal Daumé III ([email protected])
![Page 196: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/196.jpg)
Title196 Hal Daumé III ([email protected])
![Page 197: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/197.jpg)
Title197 Hal Daumé III ([email protected])
![Page 198: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/198.jpg)
Title198 Hal Daumé III ([email protected])
![Page 199: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/199.jpg)
Title199 Hal Daumé III ([email protected])
![Page 200: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/200.jpg)
Title200 Hal Daumé III ([email protected])
![Page 201: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/201.jpg)
Title201 Hal Daumé III ([email protected])
![Page 202: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/202.jpg)
Title202 Hal Daumé III ([email protected])
![Page 203: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/203.jpg)
Title203 Hal Daumé III ([email protected])
![Page 204: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/204.jpg)
Title204 Hal Daumé III ([email protected])
![Page 205: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/205.jpg)
Title205 Hal Daumé III ([email protected])
![Page 206: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/206.jpg)
Title206 Hal Daumé III ([email protected])
![Page 207: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/207.jpg)
Title207 Hal Daumé III ([email protected])
![Page 208: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/208.jpg)
Title208 Hal Daumé III ([email protected])
![Page 209: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/209.jpg)
Title209 Hal Daumé III ([email protected])
![Page 210: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/210.jpg)
Title210 Hal Daumé III ([email protected])
![Page 211: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/211.jpg)
Title211 Hal Daumé III ([email protected])
![Page 212: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/212.jpg)
Title212 Hal Daumé III ([email protected])
![Page 213: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/213.jpg)
Title213 Hal Daumé III ([email protected])
![Page 214: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/214.jpg)
Title214 Hal Daumé III ([email protected])
![Page 215: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/215.jpg)
Title215 Hal Daumé III ([email protected])
![Page 216: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/216.jpg)
Title216 Hal Daumé III ([email protected])
Hal's Wager
� Give me a structured prediction problem where:� Annotations are at the lexical level
� Humans can do the annotation with reasonable agreement
� You give me a few thousand labeled sentences
� Then I can learn reasonably well...� ...using one of the algorithms we talked about
� Why do I say this?� Lots of positive experience
� I'm an optimist
� I want your counter-examples!
![Page 217: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/217.jpg)
Title217 Hal Daumé III ([email protected])
Open problems
� How to do SP when argmax is intractable....� Bad: simple algorithms diverge [Kulesza+Pereira, NIPS07]
� Good: some work well [Finley+Joachims, ICML08]
� And you can make it fast! [Meshi+al, ICML10]
� How to do SP with delayed feedback (credit assignment)
� Kinda just works sometimes [D, ICML09; Chang+al, ICML10]
� Generic RL also works [Branavan+al, ACL09; Liang+al, ACL09]
� What role does structure actually play?� Little: only constraints outputs [Punyakanok+al, IJCAI05]
� Little: only introduces non-linearities [Liang+al, ICML08]
� Role of experts?� what if your expert isn't actually optimal?
� what if you have more than one expert?
� what if you only have trajectories, not the expert?
![Page 218: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/218.jpg)
Title218 Hal Daumé III ([email protected])
Things I have no idea how to solve...
all : (a Bool) [a] Bool� � �
Applied to a predicate and a list, returns `True' if all elementsof the list satisfy the predicate, and `False' otherwise.
all p [] = Trueall p (x:xs) = if p x then all p xs else False
%module main:MyPrelude %data main:MyPrelude.MyList aadj = {main:MyPrelude.Nil; main:MyPrelude.Cons aadj ((main:MyPrelude.MyList aadj))}; %rec {main:MyPrelude.myzuall :: %forall tadA . (tadA -> ghczmprim:GHCziBool.Bool) -> (main:MyPrelude.MyList tadA) -> ghczmprim:GHCziBool.Bool = \ @ tadA (padk::tadA -> ghczmprim:GHCziBool.Bool) (dsddE::(main:MyPrelude.MyList tadA)) -> %case ghczmprim:GHCziBool.Bool dsddE %of (wildB1::(main:MyPrelude.MyList tadA)) {main:MyPrelude.Nil -> ghczmprim:GHCziBool.True; main:MyPrelude.Cons (xadm::tadA) (xsadn::(main:MyPrelude.MyList tadA)) -> %case ghczmprim:GHCziBool.Bool (padk xadm) %of (wild1Xc::ghczmprim:GHCziBool.Bool) {ghczmprim:GHCziBool.False -> ghczmprim:GHCziBool.False; ghczmprim:GHCziBool.True -> main:MyPrelude.myzuall @ tadA padk xsadn}}};
![Page 219: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/219.jpg)
Title219 Hal Daumé III ([email protected])
Things I have no idea how to solve...
(s1) A father had a family of sons who were perpetually quarreling among themselves. (s2) When he failed to healtheir disputes by his exhortations, he determined to give them a practical illustration of the evils of disunion; and forthis purpose he one day told them to bring him a bundle ofsticks. (s3) When they had done so, he placed the faggotinto the hands of each of them in succession, and orderedthem to break it in pieces. (s4) They tried with all theirstrength, and were not able to do it. (s5) He next openedthe faggot, took the sticks separately, one by one, andagain put them into his sons' hands, upon which they brokethem easily. (s6) He then addressed them in these words:�My sons, if you are of one mind, and unite to assist eachother, you will be as this faggot, uninjured by all the attemptsof your enemies; but if you are divided among yourselves,you will be broken as easily as these sticks.''
![Page 220: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/220.jpg)
Title220 Hal Daumé III ([email protected])
Software
� Sequence labeling� Mallet http://mallet.cs.umass.edu
� CRF++ http://crfpp.sourceforge.net
� Search-based structured prediction� LaSO http://hal3.name/TagChunk
� Searn http://hal3.name/searn
� Higher-level �feature template� approaches� Alchemy http://alchemy.cs.washington.edu
� Factorie http://code.google.com/p/factorie
![Page 221: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/221.jpg)
Title221 Hal Daumé III ([email protected])
Summary
� Structured prediction is easy if you can do argmax search (esp. loss-augmented!)
� Label-bias can kill you, so iterate (Searn/Dagger)
� Stochastic worlds modeled by MDPs
� IRL is all about learning reward functions
� IRL has fewer assumptions� More general
� Less likely to work on easy problems
� We're a long way from a complete solution
� Hal's wager: we can learn pretty much anything
Thanks! Questions?
![Page 223: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/223.jpg)
Title223 Hal Daumé III ([email protected])
References
See also:http://www.cs.utah.edu/~suresh/mediawiki/index.php/MLRGhttp://braque.cc/ShowChannel?handle=P5BVAC34
![Page 224: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/224.jpg)
Title224 Hal Daumé III ([email protected])
Stuff we talked about explicitly� Apprenticeship learning via inverse reinforcement learning, P. Abbeel and A. Ng. ICML, 2004.
� Incremental parsing with the Perceptron algorithm. M. Collins and B. Roark. ACL 2004.
� Discriminative Training Methods for Hidden Markov Models: Theory and Experiments with Perceptron Algorithms. M. Collins. EMNLP 2002.
� Search-based Structured Prediction. H. Daumé III, J. Langford and D. Marcu. Machine Learning, 2009.� Learning as Search Optimization: Approximate Large Margin Methods for Structured Prediction. H. Daumé III and D.
Marcu. ICML, 2005.
� An End-to-end Discriminative Approach to Machine Translation. P. Liang, A. Bouchard-Côté, D. Klein, B. Taskar. ACL 2006.
� Statistical Decision-Tree Models for Parsing. D. Magerman. ACL 1995.
� Training Parsers by Inverse Reinforcement Learning. G. Neu and Cs. Szepesvári. Machine Learning 77, 2009.
� Algorithms for inverse reinforcement learning, A. Ng and A. Russell. ICML, 2000.
� (Online) Subgradient Methods for Structured Prediction. N. Ratliff, J. Bagnell, and M. Zinkevich. AIStats 2007.
� Maximum margin planning. N. Ratliff, J. Bagnell and M. Zinkevich. ICML, 2006.� Learning to search: Functional gradient techniques for imitation learning. N. Ratliff, D. Silver, and J. Bagnell.
Autonomous Robots, Vol. 27, No. 1, July, 2009.
� Reduction of Imitation Learning to No-Regret Online Learning. S. Ross, G. Gordon and J. Bagnell. AIStats 2011.� Max-Margin Markov Networks. B. Taskar, C. Guestrin, V. Chatalbashev and D. Koller. JMLR 2005.� Large Margin Methods for Structured and Interdependent Output Variables. I. Tsochantaridis, T. Joachims, T.
Hofmann, and Y. Altun. JMLR 2005.� Learning Linear Ranking Functions for Beam Search with Application to Planning. Y. Xu, A. Fern, and S. Yoon.
JMLR 2009.� Maximum Entropy Inverse Reinforcement Learning. B. Ziebart, A. Maas, J. Bagnell, and A. Dey. AAAI 2008.
![Page 225: Natural Language Processinglegacydirs.umiacs.umd.edu/~hal/tmp/mlss.pdf · Not just string processing or keyword matching End systems that we want to build Simple: Spelling correction,](https://reader033.fdocuments.in/reader033/viewer/2022041819/5e9fbafb9db13a337b7a1b63/html5/thumbnails/225.jpg)
Title225 Hal Daumé III ([email protected])
Other good stuff� Reinforcement learning for mapping instructions to actions. S.R.K. Branavan, H. Chen, L. Zettlemoyer and R.
Barzilay. ACL, 2009.
� Driving semantic parsing from the world's response. J. Clarke, D. Goldwasser, M.-W. Chang, D. Roth. CoNLL 2010.
� New Ranking Algorithms for Parsing and Tagging: Kernels over Discrete Structures, and the Voted Perceptron. M.Collins and N. Duffy. ACL 2002.
� Unsupervised Search-based Structured Prediction. H. Daumé III. ICML 2009.
� Training structural SVMs when exact inference is intractable. T. Finley and T. Joachims. ICML, 2008.
� Structured learning with approximate inference. A. Kulesza and F. Pereira. NIPS, 2007.
� Conditional random fields: Probabilistic models for segmenting and labeling sequence data. J. Lafferty, A. McCallum, F. Pereira. ICML 2001.
� Structure compilation: trading structure for features. P. Liang, H. Daume, D. Klein. ICML 2008.
� Learning semantic correspondences with less supervision. P. Liang, M. Jordan and D. Klein. ACL, 2009.
� Generalization Bounds and Consistency for Structured Labeling. D. McAllester. In Predicting Structured Data, 2007.
� Maximum entropy Markov models for information extraction and segmentation. A. McCallum, D. Freitag, F. Pereira. ICML 2000.
� FACTORIE: Efficient Probabilistic Programming for Relational Factor Graphs via Imperative Declarations of Structure, Inference and Learning. A. McCallum, K. Rohanemanesh, M. Wick, K. Schultz, S. Singh. NIPS Workshop on Probabilistic Programming, 2008
� Learning efficiently with approximate inference via dual losses. O. Meshi, D. Sontag, T. Jaakkola, A. Globerson. ICML 2010.
� Learning and inference over constrained output. V. Punyakanok, D. Roth, W. Yih, D. Zimak. IJCAI, 2005.
� Boosting Structured Prediction for Imitation Learning. N. Ratliff, D. Bradley, J. Bagnell, and J. Chestnutt. NIPS 2007.
� Efficient Reductions for Imitation Learning. S. Ross and J. Bagnell. AISTATS, 2010.
� Kernel Dependency Estimation. J. Weston, O. Chapelle, A. Elisseeff, B. Schoelkopf and V. Vapnik. NIPS 2002.