Natural Language Models and Interfacesivan-titov.org/teaching/nlmi/nlmi-syntax-2.pdf · 2017. 10....

Natural Language Models and Interfaces��Part B, lecture 2

Ivan Titov

Institute for Logic, Language and Computation

}  Parsing algorithms for CFGs

}  Recap, Chomsky Normal Form (CNF)

}  A dynamic programming algorithm for parsing (CKY)

}  Extension of CKY to support unary inner rules

}  Parsing for PCFGs

}  Extension of CKY for parsing with PCFGs

}  Parser evaluation (if we have time)

After this lecture you should be able to start working on the assignment step 2.1

Parsing

}  Parsing is search through the space of all possible parses

}  e.g., we may want either any parse, all parses or the highest scoring parse (if PCFG):

}  Bottom-up:

}  One starts from words and attempt to construct the full tree

}  Top-down

}  Start from the start symbol and attempt to expand to get the sentence

argmax

T2G(x)P (T )

Set of all trees given by the grammar for the sentence x

The probability by the PCFG model

CKY algorithm (aka CYK)

}  Cocke-Kasami-Younger algorithm

}  Independently discovered in late 60s / early 70s

}  An efficient bottom up parsing algorithm for (P)CFGs

}  can be used both for the recognition and parsing problems

}  Very important in NLP (and beyond)

}  We will start with the non-probabilistic version

Constraints on the grammar

}  The basic CKY algorithm supports only rules in the Chomsky Normal Form (CNF):

}  Any CFG can be converted to an equivalent CNF

}  Equivalent means that they define the same language

}  However (syntactic) trees will look differently

}  It is possible to address it but defining such transformations that allows for easy reverse transformation

Unary preterminal rules (generation of words given PoS tags , , … )

Binary inner rules (e.g., , )

C ! C1C2

Makes linguists unhappy

N ! telescope D ! the

S ! NP V P NP ! D N

Transformation to CNF form

}  What one need to do to convert to CNF form

}  Get rid of empty (aka epsilon) productions:

}  Get rid of unary rules:

}  N-ary rules:

C ! ✏

Generally not a problem as there are not empty production in the standard (postporcessed) treebanks

C ! C1

Not a problem, as our CKY algorithm will support unary rules

Crucial to process them, as required for efficient parsing

C ! C1 C2 . . . Cn (n > 2)

Transformation to CNF form: binarization

}  Consider

}  How do we get a set of binary rules which are equivalent?

}  A more systematic way to refer to new non-terminals

publishing

NP ! DT NNP V BG NN

NP ! DT X

X ! NNP Y

Y ! V BG NN

NP ! DT @NP |DT

@NP |DT ! NNP @NP |DT NNP

@NP |DT NNP ! V BG NN

}  Instead of binarizing tules we can binarize trees on preprocessing:

publishing

@NP-> DT

@NP-> DT NNP

publishing

Can be easily reversed on postprocessing

Also known as lossless Markovization in the context of PCFGs

}  Instead of binarizing tules we can binarize trees on preprocessing:

publishing

@NP-> DT

@NP-> DT NNP

publishing

@NP-> DT NNP VBG

This is exactly the transformation used in the code of the assignment

}  We a given

}  a grammar

}  a sequence of words

}  Our goal is to produce a parse tree for

}  We need an easy way to refer to substrings of

CKY: Parsing task

indices refer to fenceposts

w = (w1, w2, . . . , wn)

G = (V,⌃, R, S)

[Some illustrations and slides in this lecture are from Marco Kuhlmann]

span (i, j) refers to words between fenceposts i and j

start symbol

Key problems

}  Recognition problem: does the sentence belong to the language defined by CFG?

}  Is there a derivation which yields the sentence?

}  Parsing problem: what is a derivation (tree) corresponding the sentence?

}  Probabilistic parsing: what is the most probable tree for the sentence?

Parsing one word

C ! wi

Parsing one word

C ! wi

Parsing one word

C ! wi

Parsing longer spans

C ! C1 C2

C1, C2,midCheck through all

C ! C1 C2

C1, C2,midCheck through all

Signatures

}  Applications of rules is independent of inner structure of a parse tree

}  We only need to know the corresponding span and the root label of the tree

}  Its signature [min,max,C]

Also known as an edge

CKY idea

}  Compute for every span a set of admissible labels (may be empty for some spans)

}  Start from small trees (single words) and proceed to larger ones

}  When done, check if S is among admissible labels for the whole sentence, if yes – the sentence belong to the language

}  That is if a tree with signature [0, n, S] exists

}  Unary rules?

CKY in action S ! NP V P

V P ! V P NP

V P ! M V

V P ! V

NP ! N

NP ! N NP

N ! can

N ! lead

N ! poison

M ! can

M ! must

V ! poison

V ! lead

lead can poison

0 1 2 3

min = 0

min = 1

min = 2

max = 1 max = 2 max = 3

V P ! V P NP

V P ! M V

V P ! V

NP ! N

NP ! N NP

N ! can

N ! lead

N ! poison

M ! can

M ! must

V ! poison

V ! lead

lead can poison

0 1 2 3

Chart (aka parsing triangle)

lead can poison

3 min = 1

min = 2

min = 3

V P ! V P NP

V P ! M V

V P ! V

NP ! N

NP ! N NP

N ! can

N ! lead

N ! poison

M ! can

M ! must

V ! poison

V ! lead

lead can poison

0 1 2 3

lead can poison

3 min = 1

min = 2

min = 3

V P ! V P NP

V P ! M V

V P ! V

NP ! N

NP ! N NP

N ! can

N ! lead

N ! poison

M ! can

M ! must

V ! poison

V ! lead

lead can poison

0 1 2 3

lead can poison

3 min = 1

min = 2

min = 3

V P ! V P NP

V P ! M V

V P ! V

NP ! N

NP ! N NP

N ! can

N ! lead

N ! poison

M ! can

M ! must

V ! poison

V ! lead

lead can poison

0 1 2 3

min = 0

min = 1

min = 2

max = 1 max = 2 max = 3

V P ! V P NP

V P ! M V

V P ! V

NP ! N

NP ! N NP

N ! can

N ! lead

N ! poison

M ! can

M ! must

V ! poison

V ! lead

lead can poison

0 1 2 3

min = 0

min = 1

min = 2

max = 1 max = 2 max = 3

V P ! V P NP

V P ! M V

V P ! V

NP ! N

NP ! N NP

N ! can

N ! lead

N ! poison

M ! can

M ! must

V ! poison

V ! lead

lead can poison

0 1 2 3

min = 0

min = 1

min = 2

max = 1 max = 2 max = 3

V P ! V P NP

V P ! M V

V P ! V

NP ! N

NP ! N NP

N ! can

N ! lead

N ! poison

M ! can

M ! must

V ! poison

V ! lead

lead can poison

0 1 2 3

min = 0

min = 1

min = 2

max = 1 max = 2 max = 3

V P ! V P NP

V P ! M V

V P ! V

NP ! N

NP ! N NP

N ! can

N ! lead

N ! poison

M ! can

M ! must

V ! poison

V ! lead

lead can poison

0 1 2 3

min = 0

min = 1

min = 2

max = 1 max = 2 max = 3

NP, V P

V P ! V P NP

V P ! M V

V P ! V

NP ! N

NP ! N NP

N ! can

N ! lead

N ! poison

M ! can

M ! must

V ! poison

V ! lead

lead can poison

0 1 2 3

min = 0

min = 1

min = 2

max = 1 max = 2 max = 3

NP, V P

V P ! V P NP

V P ! M V

V P ! V

NP ! N

NP ! N NP

N ! can

N ! lead

N ! poison

M ! can

M ! must

V ! poison

V ! lead

lead can poison

0 1 2 3

min = 0

min = 1

min = 2

max = 1 max = 2 max = 3

NP, V P

V P ! V P NP

V P ! M V

V P ! V

NP ! N

NP ! N NP

N ! can

N ! lead

N ! poison

M ! can

M ! must

V ! poison

V ! lead

lead can poison

0 1 2 3

min = 0

min = 1

min = 2

max = 1 max = 2 max = 3

N, V NP

NP, V P

V P ! V P NP

V P ! M V

V P ! V

NP ! N

NP ! N NP

N ! can

N ! lead

N ! poison

M ! can

M ! must

V ! poison

V ! lead

lead can poison

0 1 2 3

min = 0

min = 1

min = 2

max = 1 max = 2 max = 3

N, V NP

NP, V P

V P ! V P NP

V P ! M V

V P ! V

NP ! N

NP ! N NP

N ! can

N ! lead

N ! poison

M ! can

M ! must

V ! poison

V ! lead

lead can poison

0 1 2 3

Check about unary rules: no unary rules here

min = 0

min = 1

min = 2

max = 1 max = 2 max = 3

N, V NP

NP, V P

V P ! V P NP

V P ! M V

V P ! V

NP ! N

NP ! N NP

N ! can

N ! lead

N ! poison

M ! can

M ! must

V ! poison

V ! lead

lead can poison

0 1 2 3

min = 0

min = 1

min = 2

max = 1 max = 2 max = 3

N, V NP

NP, V P

S, V P,

V P ! V P NP

V P ! M V

V P ! V

NP ! N

NP ! N NP

N ! can

N ! lead

N ! poison

M ! can

M ! must

V ! poison

V ! lead

lead can poison

0 1 2 3

min = 0

min = 1

min = 2

max = 1 max = 2 max = 3

N, V NP

NP, V P

S, V P,

V P ! V P NP

V P ! M V

V P ! V

NP ! N

NP ! N NP

N ! can

N ! lead

N ! poison

M ! can

M ! must

V ! poison

V ! lead

lead can poison

0 1 2 3

Check about unary rules: no unary rules here

min = 0

min = 1

min = 2

max = 1 max = 2 max = 3

N, V NP

NP, V P

S, V P,

V P ! V P NP

V P ! M V

V P ! V

NP ! N

NP ! N NP

N ! can

N ! lead

N ! poison

M ! can

M ! must

V ! poison

V ! lead

lead can poison

0 1 2 3

min = 0

min = 1

min = 2

max = 1 max = 2 max = 3

N, V NP

NP, V P

S, V P,

V P ! V P NP

V P ! M V

V P ! V

NP ! N

NP ! N NP

N ! can

N ! lead

N ! poison

M ! can

M ! must

V ! poison

V ! lead

lead can poison

0 1 2 3

min = 0

min = 1

min = 2

max = 1 max = 2 max = 3

N, V NP

NP, V P

S, V P,

6S, NP

CKY in action

lead can poison

0 1 2 3

S ! NP V P

V P ! V P NP

V P ! M V

V P ! V

NP ! N

NP ! N NP

N ! can

N ! lead

N ! poison

M ! can

M ! must

V ! poison

V ! lead

mid = 1

min = 0

min = 1

min = 2

max = 1 max = 2 max = 3

N, V NP

NP, V P

S, V P,

6S, NP

V P ! V P NP

V P ! M V

V P ! V

NP ! N

NP ! N NP

N ! can

N ! lead

N ! poison

M ! can

M ! must

V ! poison

V ! lead

lead can poison

0 1 2 3

mid = 2

min = 0

min = 1

min = 2

max = 1 max = 2 max = 3

N, V NP

NP, V P

S, V P,

6S, NP

V P ! V P NP

V P ! M V

V P ! V

NP ! N

NP ! N NP

N ! can

N ! lead

N ! poison

M ! can

M ! must

V ! poison

V ! lead

lead can poison

0 1 2 3

Apparently the sentence is ambiguous for the grammar: (as the grammar overgenerates)

mid = 2

Ambiguity

poison

No subject-verb agreement, and poison used as an intransitive verb

Apparently the sentence is ambiguous for the grammar: (as the grammar overgenerates)

CKY more formally

}  Chart can be represented by a Boolean array chart[min][max][label]

}  Relevant entries have

}  if the signature (min, max, C) is already added to the chart; otherwise.

Here we assume that labels (C) are integer indices

chart[min][max][C] = true

chart[min][max][C]

0 < min < max n

min = 0

min = 1

min = 2

max = 1 max = 2 max = 3

N, V NP

NP, V P

S, V P,

S,V P,

6In the assignment code we use a class Chart but its access methods are similar

Implementation: preterminal rules

Implementation: binary rules

Implementation: unary rules

But we forgot something!

Unary closure

}  What if the grammar contained 2 rules:

}  But C can be derived from A by a chain of rules:

}  One could support chains in the algorithm but it is easier to extend the grammar, to get the reflexive transitive closure

A ! B ! C

B ! C)

Convenient for programming reasons in the PCFG case

Implementation: skeleton

Algorithm analysis

}  Time complexity?

}  , where is the number of rules in the grammar

}  There exist algorithms with better asymptotical time complexity but the `constant' makes them slower in practice (in general)

✓(n3|R|) |R|

A few seconds for sentences under < 20 words for a non-optimized parser

Practical time complexity

}  Time complexity? (for the PCFG version)

⇠ n3.6

[Plot by Dan Klein]

}  Parser evaluation (if we have time)

Probabilistic parsing

}  We discussed the recognition problem:

}  check if a sentence is parsable with a CFG

}  Now we consider parsing with PCFGs

}  Recognition with PCFGs: what is the probability of the most probable parse tree?

}  Parsing with PCFGs: What is the most probable parse tree?

Distribution over trees

}  Let us denote by the set of derivations for the sentence

}  The probability distribution defines the scoring over the trees

}  Finding the best parse for the sentence according to PCFG:

G(x) x

T 2 G(x)

P (T )

argmax

T2G(x)P (T )

CKY with PCFGs

}  Chart is represented by a double array chart[min][max][label]

}  It stores probabilities for the most probable subtree with a given signature

}  will store the probability of the most probable full parse tree

chart[min][max][C]

chart[0][n][S]

Intuition

C ! C1 C2

For every choose , and mid such that

is maximal, where and are left and right subtrees.

C1 C2C

P (T1)⇥ P (T2)⇥ P (C ! C1C2)

Implementation: preterminal rules

Implementation: binary rules

Unary (reflexive transitive) closure

B ! C)

0.2⇥ 0.1

Note that this is not a PCFG anymore as the rules do not sum to 1 for each parent

)A ! B

1.e� 5

. . .. . .

What about loops, like: ? A ! B ! A ! C

The fact that the rule is composite needs to be stored to recover the true tree

Recovery of the tree

}  For each signature we store backpointers to the elements from which it was built (e.g., rule and, for binary rules, midpoint)

}  start recovering from [0, n, S]

}  Be careful with unary rules

}  Basically you can assume that you always used an unary rule from the closure (but it could be the trivial one )

Speeding up the algorithm (approximate search)

}  Basic pruning (roughly):

}  For every span (i,j) store only labels which have the probability at most N times smaller than the probability of the most probable label for this span

}  Check not all rules but only rules yielding subtree labels having non-zero probability

}  Coarse-to-fine pruning

}  Parse with a smaller (simpler) grammar, and precompute (posterior) probabilities for each spans, and use only the ones with non-negligible probability from the previous grammar

}  Parser evaluation

Parser evaluation

}  Intrinsic evaluation:

}  Automatic: evaluate against annotation provided by human experts (gold standard) according to some predefined measure

}  Manual: … according to human judgment

}  Extrinsic evaluation: score syntactic representation by comparing how well a system using this representation performs on some task

}  E.g., use syntactic representation as input for a semantic analyzer and compare results of the analyzer using syntax predicted by different parsers.

Though has many drawbacks it is easier and allows to track state-of-the-art across many years

Standard evaluation setting in parsing

}  Automatic intrinsic evaluation is used: parsers are evaluated against gold standard by provided by linguists

}  There is a standard split into the parts:

}  training set: used for estimation of model parameters

}  development set: used for tuning the model (initial experiments)

}  test set: final experiments to compare against previous work

Automatic evaluation of constituent parsers

}  Exact match: percentage of trees predicted correctly

}  Bracket score: scores how well individual phrases (and their boundaries) are identified

}  Crossing brackets: percentage of phrases boundaries crossing

}  Dependency metrics: scores dependency structure corresponding to the constituent tree (percentage of correctly identified heads)

The most standard measure; we will focus on it

Brackets scores

}  The most standard score is bracket score

}  It regards a tree as a collection of brackets:

}  The set of brackets predicted by a parser is compared against the set of brackets in the tree annotated by a linguist

}  Precision, recall and F1 are used as scores

[min,max,C]

Subtree signatures for CKY

Bracketing notation

}  The same tree as a bracketed sequence

(NP (PN My) (N Dog) )

(VP (V ate)

(NP (D a ) (N sausage) )

sausage

Brackets scores

Harmonic mean of precision and recall

number of brackets the parser and annotation agree on

number of brackets predicted by the parser

Re =number of brackets the parser and annotation agree on

number of brackets in annotation

F1 =2⇥ Pr ⇥Re

Pr +Re

Preview: F1 bracket score

Treebank PCFG Unlexicalized PCFG (Klein and Manning, 2003)

Lexicalized PCFG (Collins, 1999)

Automatically Induced PCFG (Petrov et al.,

The best results reported (as of

We will introduce these models next time

Natural Language Models and Interfacesivan-titov.org/teaching/nlmi/nlmi-syntax-2.pdf · 2017. 10....

Documents

Transcript of Natural Language Models and Interfacesivan-titov.org/teaching/nlmi/nlmi-syntax-2.pdf · 2017. 10....

Foundations of Natural Lanugage Processing Lecture 12 CKY ...

u.cs.biu.ac.ilu.cs.biu.ac.il/~89-680/parsing-algorithms.pdf · Title: parsing-algorithms

Statistical Parsing - Stanford University · 1 Statistical Parsing Christopher Manning CS224N Example of uniform cost search vs. CKY parsing: The grammar, lexicon, and sentence •

Syntactic Analysis Operator-Precedence Parsing Recursive-Descent Parsing

October 2008csa3180: Setence Parsing Algorithms 1 1 CSA350: NLP Algorithms Sentence Parsing I The Parsing Problem Parsing as Search Top Down/Bottom Up.

jkVªh; cky LokLF; dk;ZØe ¼vkjch,lds½103.94.204.36/WebContent/RBSK/RBSKJobAidsinHindi.pdf · 2015. 10. 19. · 7 lkexzh dh rkfydk çLrkouk iii ;ksxnkudrkZvksa dh lwph 5& 6 1 cky

CKY and Earley Algorithms Chapter 13

Lecture 9: The CKY parsing algorithm

CKY Parsing Ling 571 Deep Processing Techniques for NLP January 12, 2011.

Chart Parsing and Probabilistic Parsing - SourceForgenltk.sourceforge.net/doc/en/advanced-parsing.pdfChart Parsing and Probabilistic Parsing 9.1 Introduction ... Furthermore, it is

Parsing Expression Grammar and Packrat Parsing (Survey)

Statistical Constituency Parsing · Based on slides from Sameer Singh,DanJurafsky, Noah Smith, Slav Petrov, and everyone else they copied from. Recap: CKY Algorithm CKY uses a two-dimensional

Parsing: Top-Down vs. Bottom-Up Parsing Algorithms Partial Parsing

Efficient Parallel CKY Parsing on GPUs Youngmin Yi (University of Seoul) Chao-Yue Lai (UC Berkeley) Slav Petrov (Google Research) Kurt Keutzer (UC Berkeley)

CKY algorithm / PCFGs

EVALB, Improving CKY Parsing, Hw3courses.washington.edu/ling571/ling571_fall_2010/slides/... · 2010-01-28 · EVALB, Improving CKY Parsing, Hw3 Scott Farrar CLMA, University of Washington

Math Notes cky

L U CKY BRE AK - pdmsview.shawinc.com

ling571 class3 cnf cky flat - University of Washingtoncourses.washington.edu/ling571/ling571_WIN2016/slides/...(CNF) ! CKY parsing requires grammars in CNF ! Chomsky Normal Form !

Top Down Parsing, Predictive Parsing