Speech & NLP (Fall 2014): Parsing with Context-Free Grammars and Probabilistic Context-Free Grammars
-
Upload
vladimir-kulyukin -
Category
Science
-
view
100 -
download
1
description
Transcript of Speech & NLP (Fall 2014): Parsing with Context-Free Grammars and Probabilistic Context-Free Grammars
![Page 1: Speech & NLP (Fall 2014): Parsing with Context-Free Grammars and Probabilistic Context-Free Grammars](https://reader030.fdocuments.in/reader030/viewer/2022013102/548f9777b47959f2248b50a0/html5/thumbnails/1.jpg)
Speech & NLP
Parsing with Context-Free Grammars&
Probabilistic Context-Free Grammars
Vladimir Kulyukin
ww.vkedco.blogspot.com
![Page 2: Speech & NLP (Fall 2014): Parsing with Context-Free Grammars and Probabilistic Context-Free Grammars](https://reader030.fdocuments.in/reader030/viewer/2022013102/548f9777b47959f2248b50a0/html5/thumbnails/2.jpg)
Outline
● Background● Chart Parsing with Context-Free Grammars: The Early Algorithm
● Probabilistic Context-Free Grammars● Part-of-Speech Tagging
![Page 3: Speech & NLP (Fall 2014): Parsing with Context-Free Grammars and Probabilistic Context-Free Grammars](https://reader030.fdocuments.in/reader030/viewer/2022013102/548f9777b47959f2248b50a0/html5/thumbnails/3.jpg)
Parts of Speech & Syntactic Categories
![Page 4: Speech & NLP (Fall 2014): Parsing with Context-Free Grammars and Probabilistic Context-Free Grammars](https://reader030.fdocuments.in/reader030/viewer/2022013102/548f9777b47959f2248b50a0/html5/thumbnails/4.jpg)
Part of Speech vs. Syntactic Category
In NLP, every wordform has a part of speech (e.g., NOUN, PREPOSITION, VERB, DET, etc.)
Syntactic categories are higher level in that they are composed of parts of speech (e.g. NP DET NOMINAL)
In context-free grammars for formal languages a part of speech is any variable V that has a production V → t, where t is a terminal
![Page 5: Speech & NLP (Fall 2014): Parsing with Context-Free Grammars and Probabilistic Context-Free Grammars](https://reader030.fdocuments.in/reader030/viewer/2022013102/548f9777b47959f2248b50a0/html5/thumbnails/5.jpg)
NL Example
S ::= NP VPS ::= AUX NP VPS ::= VPNP ::= DET NOMINALNOMINAL ::= NOUNNOMINAL ::= NOUN NOMINALNP ::= ProperNounVP ::= VERBVP ::= VERB NP
DET ::= this | that | aNOUN ::= book | flight VERB ::= book | includeAUX ::= doesPREP ::= from | to | onProperNoun ::= Houston
Suppose a CFG G has the following productions:
S, NP, VP, NOMINAL are syntactic categories;DET, NOUN, VERB, AUX, PREP, ProperNoun are parts of speech.
![Page 6: Speech & NLP (Fall 2014): Parsing with Context-Free Grammars and Probabilistic Context-Free Grammars](https://reader030.fdocuments.in/reader030/viewer/2022013102/548f9777b47959f2248b50a0/html5/thumbnails/6.jpg)
Formal Language Example
E ::= E MINUS EE ::= E TIMES EE ::= E EQUALS EE ::= E LESS EE ::= LP E RPE ::= AE ::= BE ::= C
MINUS ::= -TIMES ::= *EQUALS ::= =LESS ::= <LP ::= (RP ::= )A ::= aB ::= bC ::= c
Suppose a CFG G has the following productions:
E is the only syntactic category in G;MINUS, TIMES, EQUALS, LESS, LP, RP, A, B, C are parts of speech.
![Page 7: Speech & NLP (Fall 2014): Parsing with Context-Free Grammars and Probabilistic Context-Free Grammars](https://reader030.fdocuments.in/reader030/viewer/2022013102/548f9777b47959f2248b50a0/html5/thumbnails/7.jpg)
Early Algorithm
![Page 8: Speech & NLP (Fall 2014): Parsing with Context-Free Grammars and Probabilistic Context-Free Grammars](https://reader030.fdocuments.in/reader030/viewer/2022013102/548f9777b47959f2248b50a0/html5/thumbnails/8.jpg)
General Remarks
The Early Algorithm (EA) uses a dynamic programming approach to find all possible parse trees of a given input
EA makes a single left-to-right pass that gradually fills an array called a chart (this class of algorithms is known as chart parsers)
Each found parse tree can be retrieved from the chart, which makes this algorithm well suited for retrieving partial inputs
EA can be used as a prediction management framework
![Page 9: Speech & NLP (Fall 2014): Parsing with Context-Free Grammars and Probabilistic Context-Free Grammars](https://reader030.fdocuments.in/reader030/viewer/2022013102/548f9777b47959f2248b50a0/html5/thumbnails/9.jpg)
Helper Data Structures & Functions
Production { LHS; // left-hand side variable RHS; // sequence of right-hand side variables and terminals}// Suppose that A is a syntactic or part-of-speech category and G is a CFG.FindProductions(A, G) returns a set of grammar productions {(A λ) | where A is the LHS and λ is the RHS of a production in G}
PartOfSpeech(wordform) returns a set of part-of-speech symbols for wordform (e.g., PartOfSpeech(“make”) returns { NOUN, VERB }
![Page 10: Speech & NLP (Fall 2014): Parsing with Context-Free Grammars and Probabilistic Context-Free Grammars](https://reader030.fdocuments.in/reader030/viewer/2022013102/548f9777b47959f2248b50a0/html5/thumbnails/10.jpg)
NL Example
S ::= NP VPS ::= AUX NP VPS ::= VPNP ::= DET NOMINALNOMINAL ::= NOUNNOMINAL ::= NOUN NOMINALNP ::= ProperNounVP ::= VERBVP ::= VERB NP
DET ::= this | that | aNOUN ::= book | flight VERB ::= book | includeAUX ::= doesPREP ::= from | to | onProperNoun ::= Houston
Suppose a CFG G has the following productions:
FindProductions(S, G) returns { S ::= NP VP, S ::= AUX NP VP, S ::= VP };PartOfSpeech(book) returns { NOUN, VERB}
![Page 11: Speech & NLP (Fall 2014): Parsing with Context-Free Grammars and Probabilistic Context-Free Grammars](https://reader030.fdocuments.in/reader030/viewer/2022013102/548f9777b47959f2248b50a0/html5/thumbnails/11.jpg)
Formal Language Example
E ::= E MINUS EE ::= E TIMES EE ::= E EQUALS EE ::= E LESS EE ::= LP E RPE ::= AE ::= BE ::= C
MINUS ::= -TIMES ::= *EQUALS ::= =LESS ::= <LP ::= (RP ::= )A ::= aB ::= bC ::= c
Suppose a CFG G has the following productions:
FindProductions(E, G) returns { E ::= E MINUS E, E ::= E TIMES E, E ::= E EQUALS E, E ::= E LESS E, E ::= LP E RP, E ::= A, E ::= B, E ::= C};PartOfSpeech(a) returns { A };PartOfSpeech(<) returns { LESS }
![Page 12: Speech & NLP (Fall 2014): Parsing with Context-Free Grammars and Probabilistic Context-Free Grammars](https://reader030.fdocuments.in/reader030/viewer/2022013102/548f9777b47959f2248b50a0/html5/thumbnails/12.jpg)
Parser States (aka Dotted Rules)
![Page 13: Speech & NLP (Fall 2014): Parsing with Context-Free Grammars and Probabilistic Context-Free Grammars](https://reader030.fdocuments.in/reader030/viewer/2022013102/548f9777b47959f2248b50a0/html5/thumbnails/13.jpg)
Input & Input Positions
“MAKE” “A” “LEFT”
0 1 2 3
INPUT: “MAKE A LEFT”
![Page 14: Speech & NLP (Fall 2014): Parsing with Context-Free Grammars and Probabilistic Context-Free Grammars](https://reader030.fdocuments.in/reader030/viewer/2022013102/548f9777b47959f2248b50a0/html5/thumbnails/14.jpg)
Relationship b/w Input & Parser States
0 i
Current state of production
N
(B E F * G, i, j)
(A B C D *, 0, N)
… … …j
Input Start Input End
![Page 15: Speech & NLP (Fall 2014): Parsing with Context-Free Grammars and Probabilistic Context-Free Grammars](https://reader030.fdocuments.in/reader030/viewer/2022013102/548f9777b47959f2248b50a0/html5/thumbnails/15.jpg)
Input & Parser State: Examples
“MAKE” “A” “LEFT”0 1 2 3
(NP DET * NOMINAL, 1, 2)
(VP Verb NP *, 0, 3)
(S *VP, 0, 0)
![Page 16: Speech & NLP (Fall 2014): Parsing with Context-Free Grammars and Probabilistic Context-Free Grammars](https://reader030.fdocuments.in/reader030/viewer/2022013102/548f9777b47959f2248b50a0/html5/thumbnails/16.jpg)
Predictions, Scans, & Completions
![Page 17: Speech & NLP (Fall 2014): Parsing with Context-Free Grammars and Probabilistic Context-Free Grammars](https://reader030.fdocuments.in/reader030/viewer/2022013102/548f9777b47959f2248b50a0/html5/thumbnails/17.jpg)
PredictionPrediction is the creation of new parser states (aka dotted rules) that predict what is expected to be seen in the input
If a parser state’s production’s right-hand side has the dot to the left of a variable (i.e., non-terminal) that is not a part-of-speech category, prediction is applicable
Prediction generates new parser states that start and end at the same input position as the parser state from which they were predicted
![Page 18: Speech & NLP (Fall 2014): Parsing with Context-Free Grammars and Probabilistic Context-Free Grammars](https://reader030.fdocuments.in/reader030/viewer/2022013102/548f9777b47959f2248b50a0/html5/thumbnails/18.jpg)
Prediction Example
● Suppose a CFG grammar G contains the following productions: S → VP, VP → Verb, VP → Verb NP
● Suppose that there is a parser state (S → * VP, 0, 0)● Since VP is not a part-of-speech category, prediction applies● Prediction generates two new parser states from the old parser
state (S → * VP, 0, 0):● 1) (VP → * Verb, 0, 0)● 2) (VP → * Verb NP, 0, 0)
![Page 19: Speech & NLP (Fall 2014): Parsing with Context-Free Grammars and Probabilistic Context-Free Grammars](https://reader030.fdocuments.in/reader030/viewer/2022013102/548f9777b47959f2248b50a0/html5/thumbnails/19.jpg)
Prediction Pseudocode
// G is a grammar, chart[] is an array of parser statesPredict((A → α *X β, start, end), G, chart[]) { Productions = FindProductions(X, G);
find each production (X → λ) in Productions { // Insert a new parser state ((X → *λ, end, end) // into chart[end] AddToChart((X → *λ, end, end), chart[end]); }}
![Page 20: Speech & NLP (Fall 2014): Parsing with Context-Free Grammars and Probabilistic Context-Free Grammars](https://reader030.fdocuments.in/reader030/viewer/2022013102/548f9777b47959f2248b50a0/html5/thumbnails/20.jpg)
Example
“a” “*” “b”0 1 2 3
INPUT: “a * b”
E ::= E MINUS EE ::= E TIMES EE ::= E EQUALS EE ::= E LESS EE ::= LP E RPE ::= AE ::= BE ::= C
MINUS ::= -TIMES ::= *EQUALS ::= =LESS ::= <LP ::= (RP ::= )A ::= aB ::= bC ::= c
Suppose a CFG G has these productions:
![Page 21: Speech & NLP (Fall 2014): Parsing with Context-Free Grammars and Probabilistic Context-Free Grammars](https://reader030.fdocuments.in/reader030/viewer/2022013102/548f9777b47959f2248b50a0/html5/thumbnails/21.jpg)
Example
INPUT: “a * b”
E ::= E MINUS EE ::= E TIMES EE ::= E EQUALS EE ::= E LESS EE ::= LP E RPE ::= AE ::= BE ::= C
MINUS ::= -TIMES ::= *EQUALS ::= =LESS ::= <LP ::= (RP ::= )A ::= aB ::= bC ::= c
Suppose a CFG G has these productions:
1. ((λ *E), 0, 0) // dummy prediction to start everything going
![Page 22: Speech & NLP (Fall 2014): Parsing with Context-Free Grammars and Probabilistic Context-Free Grammars](https://reader030.fdocuments.in/reader030/viewer/2022013102/548f9777b47959f2248b50a0/html5/thumbnails/22.jpg)
Example
INPUT: “a * b”
E ::= E MINUS EE ::= E TIMES EE ::= E EQUALS EE ::= E LESS EE ::= LP E RPE ::= AE ::= BE ::= C
MINUS ::= -TIMES ::= *EQUALS ::= =LESS ::= <LP ::= (RP ::= )A ::= aB ::= bC ::= c
Suppose a CFG G has these productions:
1. ((λ *E), 0, 0) // dummy prediction to start everything going2. ((E *E MINUS E), 0, 0) // prediction from E ::= E MINUS E
![Page 23: Speech & NLP (Fall 2014): Parsing with Context-Free Grammars and Probabilistic Context-Free Grammars](https://reader030.fdocuments.in/reader030/viewer/2022013102/548f9777b47959f2248b50a0/html5/thumbnails/23.jpg)
Example
INPUT: “a * b”
E ::= E MINUS EE ::= E TIMES EE ::= E EQUALS EE ::= E LESS EE ::= LP E RPE ::= AE ::= BE ::= C
MINUS ::= -TIMES ::= *EQUALS ::= =LESS ::= <LP ::= (RP ::= )A ::= aB ::= bC ::= c
Suppose a CFG G has these productions:
1. ((λ *E), 0, 0) // dummy prediction to start everything going2. ((E *E MINUS E), 0, 0) // prediction from E ::= E MINUS E3. ((E *E TIMES E), 0, 0) // prediction from E ::= E TIMES E
![Page 24: Speech & NLP (Fall 2014): Parsing with Context-Free Grammars and Probabilistic Context-Free Grammars](https://reader030.fdocuments.in/reader030/viewer/2022013102/548f9777b47959f2248b50a0/html5/thumbnails/24.jpg)
Example
INPUT: “a * b”
E ::= E MINUS EE ::= E TIMES EE ::= E EQUALS EE ::= E LESS EE ::= LP E RPE ::= AE ::= BE ::= C
MINUS ::= -TIMES ::= *EQUALS ::= =LESS ::= <LP ::= (RP ::= )A ::= aB ::= bC ::= c
Suppose a CFG G has these productions:
1. ((λ *E), 0, 0) // dummy prediction to start everything going2. ((E *E MINUS E), 0, 0) // prediction from E ::= E MINUS E3. ((E *E TIMES E), 0, 0) // prediction from E ::= E TIMES E4. ((E *E EQUALS E), 0, 0) // prediction from E ::= E EQUALS E
![Page 25: Speech & NLP (Fall 2014): Parsing with Context-Free Grammars and Probabilistic Context-Free Grammars](https://reader030.fdocuments.in/reader030/viewer/2022013102/548f9777b47959f2248b50a0/html5/thumbnails/25.jpg)
Example
INPUT: “a * b”
E ::= E MINUS EE ::= E TIMES EE ::= E EQUALS EE ::= E LESS EE ::= LP E RPE ::= AE ::= BE ::= C
MINUS ::= -TIMES ::= *EQUALS ::= =LESS ::= <LP ::= (RP ::= )A ::= aB ::= bC ::= c
Suppose a CFG G has these productions:
1. ((λ *E), 0, 0) // dummy prediction to start everything going2. ((E *E MINUS E), 0, 0) // prediction from E ::= E MINUS E3. ((E *E TIMES E), 0, 0) // prediction from E ::= E TIMES E4. ((E *E EQUALS E), 0, 0) // prediction from E ::= E EQUALS E5. ((E *E LESS E), 0, 0) // prediction from E ::= LESS E
![Page 26: Speech & NLP (Fall 2014): Parsing with Context-Free Grammars and Probabilistic Context-Free Grammars](https://reader030.fdocuments.in/reader030/viewer/2022013102/548f9777b47959f2248b50a0/html5/thumbnails/26.jpg)
Example
INPUT: “a * b”
E ::= E MINUS EE ::= E TIMES EE ::= E EQUALS EE ::= E LESS EE ::= LP E RPE ::= AE ::= BE ::= C
MINUS ::= -TIMES ::= *EQUALS ::= =LESS ::= <LP ::= (RP ::= )A ::= aB ::= bC ::= c
Suppose a CFG G has these productions:
1. ((λ *E), 0, 0) // dummy prediction to start everything going2. ((E *E MINUS E), 0, 0) // prediction from E ::= E MINUS E3. ((E *E TIMES E), 0, 0) // prediction from E ::= E TIMES E4. ((E *E EQUALS E), 0, 0) // prediction from E ::= E EQUALS E5. ((E *E LESS E), 0, 0) // prediction from E ::= LESS E6. ((E *LP E RP), 0, 0) // prediction from E ::= LP E RP
![Page 27: Speech & NLP (Fall 2014): Parsing with Context-Free Grammars and Probabilistic Context-Free Grammars](https://reader030.fdocuments.in/reader030/viewer/2022013102/548f9777b47959f2248b50a0/html5/thumbnails/27.jpg)
Example
INPUT: “a * b”
E ::= E MINUS EE ::= E TIMES EE ::= E EQUALS EE ::= E LESS EE ::= LP E RPE ::= AE ::= BE ::= C
MINUS ::= -TIMES ::= *EQUALS ::= =LESS ::= <LP ::= (RP ::= )A ::= aB ::= bC ::= c
Suppose a CFG G has these productions:
1. ((λ *E), 0, 0) // dummy prediction to start everything going2. ((E *E MINUS E), 0, 0) // prediction from E ::= E MINUS E3. ((E *E TIMES E), 0, 0) // prediction from E ::= E TIMES E4. ((E *E EQUALS E), 0, 0) // prediction from E ::= E EQUALS E5. ((E *E LESS E), 0, 0) // prediction from E ::= LESS E6. ((E *LP E RP), 0, 0) // prediction from E ::= LP E RP7. ((E *A), 0, 0) // prediction from E ::= A
![Page 28: Speech & NLP (Fall 2014): Parsing with Context-Free Grammars and Probabilistic Context-Free Grammars](https://reader030.fdocuments.in/reader030/viewer/2022013102/548f9777b47959f2248b50a0/html5/thumbnails/28.jpg)
Example
INPUT: “a * b”
E ::= E MINUS EE ::= E TIMES EE ::= E EQUALS EE ::= E LESS EE ::= LP E RPE ::= AE ::= BE ::= C
MINUS ::= -TIMES ::= *EQUALS ::= =LESS ::= <LP ::= (RP ::= )A ::= aB ::= bC ::= c
Suppose a CFG G has these productions:
1. ((λ *E), 0, 0) // dummy prediction to start everything going2. ((E *E MINUS E), 0, 0) // prediction from E ::= E MINUS E3. ((E *E TIMES E), 0, 0) // prediction from E ::= E TIMES E4. ((E *E EQUALS E), 0, 0) // prediction from E ::= E EQUALS E5. ((E *E LESS E), 0, 0) // prediction from E ::= LESS E6. ((E *LP E RP), 0, 0) // prediction from E ::= LP E RP7. ((E *A), 0, 0) // prediction from E ::= A8. ((E *B), 0, 0) // prediction from E ::= B
![Page 29: Speech & NLP (Fall 2014): Parsing with Context-Free Grammars and Probabilistic Context-Free Grammars](https://reader030.fdocuments.in/reader030/viewer/2022013102/548f9777b47959f2248b50a0/html5/thumbnails/29.jpg)
Example
INPUT: “a * b”
E ::= E MINUS EE ::= E TIMES EE ::= E EQUALS EE ::= E LESS EE ::= LP E RPE ::= AE ::= BE ::= C
MINUS ::= -TIMES ::= *EQUALS ::= =LESS ::= <LP ::= (RP ::= )A ::= aB ::= bC ::= c
Suppose a CFG G has these productions:
1. ((λ *E), 0, 0) // dummy prediction to start everything going2. ((E *E MINUS E), 0, 0) // prediction from E ::= E MINUS E3. ((E *E TIMES E), 0, 0) // prediction from E ::= E TIMES E4. ((E *E EQUALS E), 0, 0) // prediction from E ::= E EQUALS E5. ((E *E LESS E), 0, 0) // prediction from E ::= LESS E6. ((E *LP E RP), 0, 0) // prediction from E ::= LP E RP7. ((E *A), 0, 0) // prediction from E ::= A8. ((E *B), 0, 0) // prediction from E ::= B9. ((E *C), 0, 0) // prediction from E ::= C
![Page 30: Speech & NLP (Fall 2014): Parsing with Context-Free Grammars and Probabilistic Context-Free Grammars](https://reader030.fdocuments.in/reader030/viewer/2022013102/548f9777b47959f2248b50a0/html5/thumbnails/30.jpg)
Scanning Scanning advances the input pointer exactly one wordform to
the right (or left as the case may be) Scanning applies to any parser state whose production has a
dot (any other symbol can be used) to the left of a part-of-speech category
Scanning creates a new parser state (NPS) from an old parser state (OPS) by 1) moving the dot in the OPR’s production’s RHS one symbol right; 2) incrementing the OPS’s input end position by 1, and 3) placing the NPS to the chart entry that follows the OPS’s chart entry
![Page 31: Speech & NLP (Fall 2014): Parsing with Context-Free Grammars and Probabilistic Context-Free Grammars](https://reader030.fdocuments.in/reader030/viewer/2022013102/548f9777b47959f2248b50a0/html5/thumbnails/31.jpg)
Scanning Example
Suppose the current wordform at position 0 in the input is MAKE
Suppose an old parser state in chart[0] is (VP → * VERB NP, 0, 0)
Since VERB is a part-of-speech category and MAKE is a VERB, scanning applies and generates a new parser state (VP → VERB * NP, 0, 1) that is placed in chart[1]
![Page 32: Speech & NLP (Fall 2014): Parsing with Context-Free Grammars and Probabilistic Context-Free Grammars](https://reader030.fdocuments.in/reader030/viewer/2022013102/548f9777b47959f2248b50a0/html5/thumbnails/32.jpg)
Scanning Pseudocode
// α, β are possibly empty sequences of variables and// terminalsScan((A → α *X β, start, end), input[], G, chart[]) { POSList = PartsOfSpeech(input[end]); if ( X is in POSList ) { AddToChart((X → input[end]*, end, end+1), chart[end+1]);}
![Page 33: Speech & NLP (Fall 2014): Parsing with Context-Free Grammars and Probabilistic Context-Free Grammars](https://reader030.fdocuments.in/reader030/viewer/2022013102/548f9777b47959f2248b50a0/html5/thumbnails/33.jpg)
Example INPUT: “a * b”
E ::= E MINUS EE ::= E TIMES EE ::= E EQUALS EE ::= E LESS EE ::= LP E RPE ::= AE ::= BE ::= C
MINUS ::= -TIMES ::= *EQUALS ::= =LESS ::= <LP ::= (RP ::= )A ::= aB ::= bC ::= c
CFG G
1. ((λ *E), 0, 0) // dummy prediction to start everything going2. ((E *E MINUS E), 0, 0) // prediction from E ::= E MINUS E3. ((E *E TIMES E), 0, 0) // prediction from E ::= E TIMES E4. ((E *E EQUALS E), 0, 0) // prediction from E ::= E EQUALS E5. ((E *E LESS E), 0, 0) // prediction from E ::= LESS E6. ((E *LP E RP), 0, 0) // prediction from E ::= LP E RP7. ((E *A), 0, 0) // prediction from E ::= A8. ((E *B), 0, 0) // prediction from E ::= B9. ((E *C), 0, 0) // prediction from E ::= C10. ((A a*), 0, 1) // scanning
Scanning is applicable in state ((E *A), 0, 0), because A is a part-of-speech and the input at position 0 is a.
![Page 34: Speech & NLP (Fall 2014): Parsing with Context-Free Grammars and Probabilistic Context-Free Grammars](https://reader030.fdocuments.in/reader030/viewer/2022013102/548f9777b47959f2248b50a0/html5/thumbnails/34.jpg)
Completion Completion is the process that indicates the completion of a
production’s RHS Completion is applied to any parser state whose dot has
reached the end of its production’s RHS (to the right of the rightmost symbol)
In other words, completion signals that the parser has completed a production (LHS → RHS) over a specific portion of the input
Completion triggers the process of finding all previously created parser states that wait for a specific category (LHS) at the completed parser state’s start position
![Page 35: Speech & NLP (Fall 2014): Parsing with Context-Free Grammars and Probabilistic Context-Free Grammars](https://reader030.fdocuments.in/reader030/viewer/2022013102/548f9777b47959f2248b50a0/html5/thumbnails/35.jpg)
Completion Example● Suppose that there are two parser states:● 1) (NP → DET NOMINAL *, 1, 3)● 2) (VP → VERB * NP, 0, 1)● Since the dot has reached the end of the 1st production’s RHS, completion applies (this
parser state is called completed parser state)● Completion finds all states that end at position 1 and look for NP; in this case, (VP →
VERB * NP, 0, 1) is found● For each found old parser state (OPS – old parser state):
● a new parser state (NPS) is created by advancing the OPS’s dot one symbol to the right and setting the NPS’s end position to the completed parser state’s end position; in this case, (VP → VERB NP *, 0, 3)
● NPS is placed in chart[completed parser state’s end position]
![Page 36: Speech & NLP (Fall 2014): Parsing with Context-Free Grammars and Probabilistic Context-Free Grammars](https://reader030.fdocuments.in/reader030/viewer/2022013102/548f9777b47959f2248b50a0/html5/thumbnails/36.jpg)
Completion Pseudocode
// α, β, λ are sequences of variables and// terminals, G is a grammarComplete((A λ *, compStart, compEnd), input[], G, chart[]) { for each old parser state ((X α *A β), oldStart, compStart) { AddToChart((X α A * β, oldStart, compEnd), chart[compEnd]);}
![Page 37: Speech & NLP (Fall 2014): Parsing with Context-Free Grammars and Probabilistic Context-Free Grammars](https://reader030.fdocuments.in/reader030/viewer/2022013102/548f9777b47959f2248b50a0/html5/thumbnails/37.jpg)
EA PseudocodeEarlyParse(input[], G, chart[]) { AddToChart((λ *S, 0, 0), char[0]); for i from 0 to length(input[]) { for each parser state PS in chart[i] { if ( isIncomplete(PS) && !isPartOfSpeech(nextCat(PS)) ) { Predict(PS, G, chart[]); } else if ( isIncomplete(PS) && isPartOfSpeech(nextCat(PS) ) { Scan(PS, input[], G, chart[]); } else { Complete(PS, input[], G, chart[]); }}}}
![Page 38: Speech & NLP (Fall 2014): Parsing with Context-Free Grammars and Probabilistic Context-Free Grammars](https://reader030.fdocuments.in/reader030/viewer/2022013102/548f9777b47959f2248b50a0/html5/thumbnails/38.jpg)
ExampleINPUT: “a * b”
E ::= E MINUS EE ::= E TIMES EE ::= E EQUALS EE ::= E LESS EE ::= LP E RPE ::= AE ::= BE ::= C
MINUS ::= -TIMES ::= *EQUALS ::= =LESS ::= <LP ::= (RP ::= )A ::= aB ::= bC ::= c
CFG G1) ((λ *E), 0, 0) // dummy prediction to start everything2) ((E *E MINUS E), 0, 0) // prediction3) ((E *E TIMES E), 0, 0) // prediction4) ((E *E EQUALS E), 0, 0) // prediction5) ((E *E LESS E), 0, 0) // prediction6) ((LP *LP E RP), 0, 0) // prediction7) ((E *A), 0, 0) // prediction8) ((E *B), 0, 0) // prediction9) ((E *C), 0, 0) // prediction10) ((A a *, 0, 1) // scanning11) ((E A*, 0, 1) // completion of parser state 712) ((λ E *, 0, 1) // completion of parser state 113) ((E E * MINUS E), 0, 1) // completion of parser state 214) ((E E *TIMES E), 0, 1) // completion of parser state 315) ((E E *EQUALS E), 0, 1) // completion of parser state 416) ((E E *LESS E , 0, 1) // completion of parser state 5
![Page 40: Speech & NLP (Fall 2014): Parsing with Context-Free Grammars and Probabilistic Context-Free Grammars](https://reader030.fdocuments.in/reader030/viewer/2022013102/548f9777b47959f2248b50a0/html5/thumbnails/40.jpg)
CFGSymbol.java public class CFGSymbol { String mSymbolName;
public CFGSymbol() { mSymbolName = ""; }
public CFGSymbol(String n) { mSymbolName = new String(n); }
public CFGSymbol(CFGSymbol s) { mSymbolName = new String(s.mSymbolName); }
public String toString() { return mSymbolName; }
public boolean isEqual(CFGSymbol s) { if ( s == null ) return false; else return this.mSymbolName.equals(s.mSymbolName); }}
![Page 41: Speech & NLP (Fall 2014): Parsing with Context-Free Grammars and Probabilistic Context-Free Grammars](https://reader030.fdocuments.in/reader030/viewer/2022013102/548f9777b47959f2248b50a0/html5/thumbnails/41.jpg)
CFProduction.javapublic class CFProduction { int mID; CFGSymbol mLHS; // left-hand side ArrayList<CFGSymbol> mRHS; // right-hand side // production ID defaults to 0 public CFProduction(CFGSymbol lhs, ArrayList<CFGSymbol> rhs) { this.mID = 0; this.mLHS = new CFGSymbol(lhs); this.mRHS = new ArrayList<CFGSymbol>(); Iterator<CFGSymbol> iter = rhs.iterator(); while ( iter.hasNext() ) { this.mRHS.add(new CFGSymbol(iter.next())); } }
public CFProduction(int id, CFGSymbol lhs, ArrayList<CFGSymbol> rhs) { this(lhs, rhs); this.mID = id; }}
![Page 42: Speech & NLP (Fall 2014): Parsing with Context-Free Grammars and Probabilistic Context-Free Grammars](https://reader030.fdocuments.in/reader030/viewer/2022013102/548f9777b47959f2248b50a0/html5/thumbnails/42.jpg)
CFGrammar.java
public class CFGrammar {
protected ArrayList<String> mVariables; // sorted array of variables protected ArrayList<String> mTerminals; // sorted array of terminals protected TreeMap<String, ArrayList<CFProduction>> mProductions; // variable to // productions map where variable is lhs protected CFGSymbol mStartSymbol; protected ArrayList<CFProduction> mIdToProductionMap; // given an id, find a rule. protected ArrayList<String> mPosVars; // sorted lists of parts of speech. protected TreeMap<String, ArrayList<CFGSymbol>> mTerminalsToPosVarsMap;
…}
![Page 43: Speech & NLP (Fall 2014): Parsing with Context-Free Grammars and Probabilistic Context-Free Grammars](https://reader030.fdocuments.in/reader030/viewer/2022013102/548f9777b47959f2248b50a0/html5/thumbnails/43.jpg)
CFGrammar.java: Compiling CFG from File
public CFGrammar() { mVariables = new ArrayList<String>(); mTerminals = new ArrayList<String>(); mProductions = new TreeMap<String, ArrayList<CFProduction>>(); mIdToProductionMap = new ArrayList<CFProduction>(); mPosVars = new ArrayList<String>(); mTerminalsToPosVarsMap = new TreeMap<String, ArrayList<CFGSymbol>>(); mStartSymbol = null; }
public CFGrammar(String grammarFilePath) { this(); compileCFGGrammar(grammarFilePath); }
![Page 44: Speech & NLP (Fall 2014): Parsing with Context-Free Grammars and Probabilistic Context-Free Grammars](https://reader030.fdocuments.in/reader030/viewer/2022013102/548f9777b47959f2248b50a0/html5/thumbnails/44.jpg)
RecognizerState.java public class RecognizerState { int mRuleNum; // number of a cfg rule that this state tracks. int mDotPos; // dot position in the rhs of the rule. int mInputFromPos; // where in the input the rule starts. int mUptoPos; // where in the input the rule ends. CFProduction mTrackedRule; // the actual CFG rule being tracked; the rule's number == ruleNum.
public CFGSymbol nextCat() { if ( mDotPos < mTrackedRule.mRHS.size() ) return mTrackedRule.mRHS.get(mDotPos); else return null; }
public boolean isComplete() { return mDotPos == mTrackedRule.mRHS.size(); }
public int getDotPos() { return mDotPos; } public int getFromPos() { return mInputFromPos; } public int getUptoPos() { return mUptoPos; } public CFProduction getCFGRule() { return mTrackedRule; } public int getRuleNum() { return mRuleNum; }
…}
![Page 45: Speech & NLP (Fall 2014): Parsing with Context-Free Grammars and Probabilistic Context-Free Grammars](https://reader030.fdocuments.in/reader030/viewer/2022013102/548f9777b47959f2248b50a0/html5/thumbnails/45.jpg)
ParserState.java public class ParserState extends RecognizerState { static int mCount = 0; int mID = 0; // this is the unique id of this ParserState ArrayList<ParserState> mParentParserStates = null; // IDs of parent parser states String mOrigin = "N"; // it can be N (None), S (Scanner), P (Predictor), C (Completer)
public ParserState(int ruleNum, int dotPos, int fromPos, int uptoPos, CFProduction r) { super(ruleNum, dotPos, fromPos, uptoPos, r); mID = mCount; mCount = mCount + 1; mParentParserStates = new ArrayList<ParserState>(); }
String getID() { return "PS" + mID; } void addPreviousState(ParserState ps) { mParentParserStates.add(ps); } // add all previous states of ps into the this state. void addPreviousStatesOf(ParserState ps) { if ( ps.mParentParserStates.isEmpty() ) return; Iterator<ParserState> iter = ps.mParentParserStates.iterator(); while ( iter.hasNext() ) { addPreviousState(iter.next()); } } void setOrigin(String origin) { mOrigin = origin; }
![Page 46: Speech & NLP (Fall 2014): Parsing with Context-Free Grammars and Probabilistic Context-Free Grammars](https://reader030.fdocuments.in/reader030/viewer/2022013102/548f9777b47959f2248b50a0/html5/thumbnails/46.jpg)
Retrieval of Parse Trees
Each state is augmented with an additional member variable to its parents, i.e., the completed parser states that generated it
The parse tree retrieval algorithm begins with each parser state that spans the entire input and completes a production whose LHS is S
The algorithm recursively retrieves each sub-tree by for the parents and then linking them to the tree of the current parser state
![Page 47: Speech & NLP (Fall 2014): Parsing with Context-Free Grammars and Probabilistic Context-Free Grammars](https://reader030.fdocuments.in/reader030/viewer/2022013102/548f9777b47959f2248b50a0/html5/thumbnails/47.jpg)
Probabilistic Context-Free Grammars
![Page 48: Speech & NLP (Fall 2014): Parsing with Context-Free Grammars and Probabilistic Context-Free Grammars](https://reader030.fdocuments.in/reader030/viewer/2022013102/548f9777b47959f2248b50a0/html5/thumbnails/48.jpg)
Probabilistic CFGs
PCFGs extend regular CFGs Productions are assigned probabilities The probability of a derivation is the product of all applied
productions in that derivation The production probabilities are typically learned through
various machine learning approaches applied to very large tree banks (e.g., the Penn State Tree Bank Project at www.cis.upenn.edu/~treebank )
PCFGs have been used in NLP, RNA analysis, and programming language design
![Page 49: Speech & NLP (Fall 2014): Parsing with Context-Free Grammars and Probabilistic Context-Free Grammars](https://reader030.fdocuments.in/reader030/viewer/2022013102/548f9777b47959f2248b50a0/html5/thumbnails/49.jpg)
Formal Definition
.each toiesprobabilit assignshat function t a is .5
symbol;start theis .4
; and where
, form theof sproduction ofset a is 3.
symbols; terminalofset a is 2.
symbols; terminal-non ofset a is N .1
,,,, tuple-5 a isG PCFG A
*
PD
S
NNA
AP
DSPN
![Page 50: Speech & NLP (Fall 2014): Parsing with Context-Free Grammars and Probabilistic Context-Free Grammars](https://reader030.fdocuments.in/reader030/viewer/2022013102/548f9777b47959f2248b50a0/html5/thumbnails/50.jpg)
Production Probabilities
.|
or as expand will lnontermina that the
y probabilit lconditiona theis p where,
form theof isPCFG ain productionEach
AAP
A
pA
![Page 51: Speech & NLP (Fall 2014): Parsing with Context-Free Grammars and Probabilistic Context-Free Grammars](https://reader030.fdocuments.in/reader030/viewer/2022013102/548f9777b47959f2248b50a0/html5/thumbnails/51.jpg)
Production Probabilities
[.30]
[.30]
.40 can
[.15]
]10[.
[.75]
example,For 1. toup summust
terminal-non a of expansions possible All
doAux
doesAux
Aux
VPS
NP VPAuxS
VPNPS
![Page 52: Speech & NLP (Fall 2014): Parsing with Context-Free Grammars and Probabilistic Context-Free Grammars](https://reader030.fdocuments.in/reader030/viewer/2022013102/548f9777b47959f2248b50a0/html5/thumbnails/52.jpg)
PCFG & Disambiguation
Given a PCFG G, we can assign a probability to each parse tree of a given sentence
The parse tree with the largest probability is chosen as the parse of the sentence
PCFG-based disambiguation approaches typically take the winner-takes-all approach: the parse with the highest probability always wins
![Page 53: Speech & NLP (Fall 2014): Parsing with Context-Free Grammars and Probabilistic Context-Free Grammars](https://reader030.fdocuments.in/reader030/viewer/2022013102/548f9777b47959f2248b50a0/html5/thumbnails/53.jpg)
Probability of a Parse Tree
rule. that ofy probabilit theis
and it, expand toused rule theis ,
in node a is where,,
Then . treeparse specific a be Let
nrp
nrT
nnrpSTP
T
Tn
![Page 54: Speech & NLP (Fall 2014): Parsing with Context-Free Grammars and Probabilistic Context-Free Grammars](https://reader030.fdocuments.in/reader030/viewer/2022013102/548f9777b47959f2248b50a0/html5/thumbnails/54.jpg)
Probability of a Parse TreeS
Aux NP VP
Can you
Pronoun
V NP NP
TWA flights
Noun
NomPNoun
book
S AUX NP VP→ [.15]AUX Can→ [.40]NP Pronoun→ [.40]VP V NP NP→ [.05]V book→ [.30]NP PNoun→ [.35]PNoun TWA→ [.40]NP Nom→ [.05]Nom Noun→ [.75]Noun flights→ [.50]
PCFG Fragment
flightsNounpNounNompNomNPpTWAPNounp
PNounNPpbookVpVVPpp
NPpCanAuxpVPNPAuxSpSTP
NP NP you Pronound
Pronoun ,
![Page 55: Speech & NLP (Fall 2014): Parsing with Context-Free Grammars and Probabilistic Context-Free Grammars](https://reader030.fdocuments.in/reader030/viewer/2022013102/548f9777b47959f2248b50a0/html5/thumbnails/55.jpg)
Picking Most Probable Parse Tree
.|
because istion simplifica second The 2.
s.' allfor same the
is because istion simplificafirst The 1.
.maxarg,maxarg,
maxarg
.Sfor treesparse ofset a be Let
max
T PTSPT P
T, SP
T
SP
TPSTPSP
STPT
ST
T(S)TT(S)TT(S)T
![Page 56: Speech & NLP (Fall 2014): Parsing with Context-Free Grammars and Probabilistic Context-Free Grammars](https://reader030.fdocuments.in/reader030/viewer/2022013102/548f9777b47959f2248b50a0/html5/thumbnails/56.jpg)
Part-of-Speech Tagging With
Apache Open NL
![Page 57: Speech & NLP (Fall 2014): Parsing with Context-Free Grammars and Probabilistic Context-Free Grammars](https://reader030.fdocuments.in/reader030/viewer/2022013102/548f9777b47959f2248b50a0/html5/thumbnails/57.jpg)
Installing Apache Open NL
Download & unzip the JAR files from http://opennlp.apache.org/cgi-bin/download.cgi
Create a Java project and add the JARs to your project For example, you can include the following JARs: jwnl-1.3.3.jar opennlp-maxent-3.0.3.jar opennlp-tools-1.5.3.jar opennlp-uima-1.5.3.jar Download the language model files from
http://opennlp.sourceforge.net/models-1.5/ Place the model files into a directory
![Page 58: Speech & NLP (Fall 2014): Parsing with Context-Free Grammars and Probabilistic Context-Free Grammars](https://reader030.fdocuments.in/reader030/viewer/2022013102/548f9777b47959f2248b50a0/html5/thumbnails/58.jpg)
Tokenization of Input
Apache Open NL provides a set of tools that can be used to: - Split text into tokens or sentences
- Do part-of-speech tagging and use the output as the input to a parser - The categories used in Open NL POS tagging are at http://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html
![Page 59: Speech & NLP (Fall 2014): Parsing with Context-Free Grammars and Probabilistic Context-Free Grammars](https://reader030.fdocuments.in/reader030/viewer/2022013102/548f9777b47959f2248b50a0/html5/thumbnails/59.jpg)
PCFG & Dependency Parsing with Stanford Parser
![Page 60: Speech & NLP (Fall 2014): Parsing with Context-Free Grammars and Probabilistic Context-Free Grammars](https://reader030.fdocuments.in/reader030/viewer/2022013102/548f9777b47959f2248b50a0/html5/thumbnails/60.jpg)
Stanford Parser
Stanford is an example of a probabilistic parser Probabilistic parsers require extensive training, but, once the
training is done, they work robustly Even when they fail (independence assumption & lexical
dependency), they give you partially completed parser trees that can be put to productive use
Stanford Parser does both PCFG and dependency parsing
![Page 61: Speech & NLP (Fall 2014): Parsing with Context-Free Grammars and Probabilistic Context-Free Grammars](https://reader030.fdocuments.in/reader030/viewer/2022013102/548f9777b47959f2248b50a0/html5/thumbnails/61.jpg)
References & Reading Suggestions
Early, J. (1968). An Efficient Context-Free Parsing Algorithm. Ph.D. thesis, CMU, Pittsburgh, PA
Ch. 12 in Jurafsky & Martin. Speech & Language Processing. Prentice Hall Hopcroft and Ullman. Introduction to Automata Theory, Languages, and
Computation, Narosa Publishing House URL for Stanford Parser: http://nlp.stanford.edu/software/lex-parser.shtml