Context-free Grammarbrenocon/anlp2017/lectures/11-cfg.pdf · • (10.3) *The cat that the dog is...

Context-free Grammar

CS 690N, Spring 2017Advanced Natural Language Processing

http://people.cs.umass.edu/~brenocon/anlp2017/

Brendan O’ConnorCollege of Information and Computer Sciences

University of Massachusetts Amherst



• Syntax: how do words structurally combine to form sentences and meaning?

• Representations

• Constituents

• [the big dogs] chase cats

• [colorless green clouds] chase cats

• Dependencies

• The dog chased the cat.

• My dog, a big old one, chased the cat.

• Idea of a grammar (G): global template for how sentences / utterances / phrases w are formed, via latent syntactic structure y

• Linguistics: what do G and P(w,y | G) look like?

• Generation: score with, or sample from, P(w, y | G)

• Parsing: predict P(y | w, G)

2

Is language context-free?

3 [Examples from Eisenstein (2017)]

https://github.com/jacobeisenstein/gt-nlp-class/blob/master/notes/eisenstein-nlp-notes-snapshot.pdf


Is language context-free?• Regular language: repetition of repeated structures





• e.g. Justeson and Katz (1995)’s noun phrase pattern:(Noun | Adj)* Noun (Prep Det? (Noun | Adj)* Noun)*






• Context-free: hierarchical recursion







• Center-embedding: classic theoretical argument for CFG vs. regular languages








• (10.1) The cat is fat.









• (10.2) The cat that the dog chased is fat.










• (10.3) *The cat that the dog is fat.











• (10.4) The cat that the dog that the monkey kissed chased is fat.












• (10.5) *The cat that the dog that the monkey chased is fat.












• (10.5) *The cat that the dog that the monkey chased is fat.

• Competence vs. Performance?




Hierarchical view of syntax

• “a Sentence made of Noun Phrase followed by a Verb Phrase”

4

!"#$%&'$!(") *+#+,%'- (" '(./,%,$!0+ #1"$,2 -,#.+$ $-!# '-,33+"4+ 51 #++6!"4 $( '-,%,'$+%!7+-&.," 3,"4&,4+# !" $+%.# (8 &"!0+%#,3 #1"$,'$!'/%(/+%$!+#9 :-!'- .,1 %+83+'$ $-+ '-!3;<# !"",$+6"(:3+;4+9 ,"; "("=&"!0+%#,3 '3&#$+%# (8 #1"$,'$!'/%(/+%$!+# $-,$ /,$$+%" $(4+$-+% ,'%(## 3,"4&,4+#9,"; -+"'+ .,1 5+ 3+,%"+; ,# , 4%(&/) >-&#9 $-+#$&;1 (8 '(./,%,$!0+ #1"$,2 ,"; $-+ #$&;1 (8 3,"=4&,4+ 3+,%"!"4 ,%+ '3(#+31 %+3,$+;) ?!"" !""#$%"%&&#"' (")*%+&#, -+#..#+@

!"#$%&'#(%)* +! *,#(%-(.-(/'+0,

$1234565 .7817169

A3.(#$ ,33 ,''(&"$# (8 $-+ ;!#'%+$+ !"8!"!$1 /%(/+%$1(8 ",$&%,3 3,"4&,4+ #1"$,2 #$,%$ 8%(. $-+ "($!("$-,$ #+"$+"'+# '("#!#$ (8 .(%+ $-," B&#$ #+C&+"'+#(8 :(%;#) D" $-+ .!";# (8 #/+,6+%# ,"; 3!#$+"+%#9#+"$+"'+# ,%+ -!+%,%'-!',331 #$%&'$&%+; %+/%+#+"$,=$!("#9 !" :-!'- :(%;# ,%+ 4%(&/+; $(4+$-+% $( 8(%./-%,#+#9 :-!'- !" $&%" '(.5!"+ $( 8(%. 3,%4+%/-%,#+#) E(% +2,./3+9 , .!"!.,3 #+"$+"'+ (8 F"4=3!#-9 #&'- ,# GH(-" ,%%!0+;<9 '("$,!"# , #&5B+'$ ,"; ,/%+;!',$+9 5&$ $-+ %(3+# (8 #&5B+'$ ,"; /%+;!',$+.,1 5+ %+/3,'+; 51 /-%,#+# (8 ,%5!$%,%1 '(./3+2=!$1) I1 %+/%+#+"$!"4 /(##!53+ #&5B+'$# ,"; /%+;!=',$+# ,# #$%# &'()*"* ?JK#@ ,"; +"(, &'()*"* ?LK#@%+#/+'$!0+319 $-+ #$%&'$&%+ (8 .,"1 /(##!53+ #+"=$+"'+# ?M@ '," 5+ ',/$&%+;) >-!# 5,#!' G$+./3,$+<8(% #+"$+"'+# (8 F"43!#- '," 5+ +2/%+##+; ,# , $%++#$%&'$&%+9 ,# !" ?N,@9 (% ,# , /-%,#+ #$%&'$&%+ %&3+9 ,#!" ?N5@)

Sa.

b. S NP VP

NP VPJohnthe manthe elderly janitor

arrivedate an applelooked at his watch

(1)

{ {{ {H&#$ ,# %&3+# 3!6+ M ! JK LK /%(0!;+ $+./3,$+#

8(% #+"$+"'+#9 $+./3,$+# '," ,3#( 5+ #/+'!8!+; 8(%$-+ !"$+%",3 #$%&'$&%+ (8 "(&" /-%,#+#9 0+%5/-%,#+#9 ,"; .,"1 ($-+% /-%,#+=$1/+#) F0+" ,#.,33 "&.5+% (8 /-%,#+ #$%&'$&%+ %&3+# ,"; ,#.,33 3+2!'(" '," 4+"+%,$+ 3,%4+ "&.5+%# (8 #+"=$+"'+#) O!$- ("31 $-+ 8!0+ /-%,#+ #$%&'$&%+ %&3+# !"?P@ ,"; , QR=:(%; 3+2!'(" ?'("#!#$!"4 (8 NR "(&"#9NR ;+$+%.!"+%#9 ,"; NR 0+%5#@ NPP9NRR ;!88+%+"$ #+"=$+"'+# '," 5+ 4+"+%,$+;)

M ! JK LK

LK ! LJK

LK ! L

JK ! S+$ JK

JK ! J "P#

*&3+# $-,$ ,33(: , /-%,#+ $( 5+ +.5+;;+; !"#!;+,"($-+% /-%,#+ (8 $-+ #,.+ $1/+ ,%+ 6"(:" ,# ("-%(.*/+" %&3+#) T((%;!",$!(" ?Q@9 .(;!8!',$!(" ?U@9 ,";#+"$+"$!,3 '(./3+.+"$,$!(" ?V@ ,33 !"0(30+ %+'&%=#!(") >-+1 '," $-&# 5+ !"0(6+; ,%5!$%,%!31 .,"1$!.+# !" , #!"43+ #+"$+"'+) M&'- %&3+# !"'%+,#+ $-++2/%+##!0+ /(:+% (8 $-+ 4%,..,% 8%(. .+%+31 0,#$$( '3+,%31 !"8!"!$+) >-+%+ ,%+ (50!(&# /%,'$!',3 3!.!=$,$!("# (" $-+ 3+"4$- ,"; '(./3+2!$1 (8 ",$&%,331(''&%%!"4 #+"$+"'+#9 5&$ #&'- 3!.!$,$!("# ,%+ $1/!'=,331 ,$$%!5&$+; $( !";+/+";+"$ 3!.!$,$!("# (" ,$$+"=$!(" ,"; .+.(%1)

JK ! JK T("B JK

LK ! LK T("B LK

T("B ! )#0 "Q#

LK ! LK KK

JK ! JK KK "U#

LK ! L M$

M$ ! T(./ M

T(./ ! 1')1 "V#

A3$-(&4- $-+ %&3+# 3!#$+; !" ?N@W?V@ 8,33 8,% #-(%$(8 $-+ +2/%+##!0+ /(:+% (8 F"43!#-9 +0+" $-!##.,33 8%,4.+"$ #-(:# -(: ",$&%,3 3,"4&,4+#1"$,2 &#+# 8!"!$+ .+,"# $( 4+"+%,$+ !"8!"!$+31.,"1 #+"$+"'+#) ?!"" /0+#&% 1$+23$2+% #"' 456#+70%8+9@

&:61;<617= *64>36>452? -:72616>5739

>-+ #1"$,'$!'!,"<# $((35(2 !"'3&;+# , "&.5+% (8#$%&'$&%,3 $+#$# $-,$ '," 5+ &#+; ,# ,!;# !" ;!,4"(#=!"4 #+"$+"'+ #$%&'$&%+#X 8(% +2,./3+9 -$#*1/1%"#1* (8#+"$+"'+# '," 4+"+%,331 5+ '("B&"'$# !" '((%;!",$+#$%&'$&%+#9 ,# !# #-(:" 8(% JK# ,"; LK# !" ?Y,9 5@)Z$-+% $+#$# $-,$ #-(: $-+ '("#$!$&+"'1 (8 LK# !"='3&;+ #&5#$!$&$!(" (8 $-+ +2/%+##!(" G;( #(< 8(% , LK?[,@9 ,"; 8%("$!"4 (8 $-+ LK $( , '3,&#+=!"!$!,3 /(#=!$!(" ?[5@)

,! O,33,'+ 8+$'-+;%JK $-+ '-++#+& ,"; %JK $-+'%,'6+%#&

5! O,33,'+%LK #3!'+; $-+ '-++#+& ,"; %LK (/+"+;$-+ '%,'6+%#& "Y#

QPR !2#1)3

[From Phillips (2003)]

http://people.cs.umass.edu/~brenocon/inlp2016/readings/phillips2003-syntax.pdf

http://people.cs.umass.edu/~brenocon/inlp2016/readings/phillips2003-syntax.pdf

Is language context-free?

• Practical examples where nesting seems like a useful explanation

• The processor has 10 million times fewer transistors on it than todays typical micro- processors, runs much more slowly, and operates at five times the voltage...

• S → NN VP VP → VP3S | VPN3S | . . . VP3S → VP3S, VP3S, and VP3S | VBZ | VBZ NP | . . .




• A context-free grammar is a 4-tuple:

6

180 CHAPTER 10. CONTEXT-FREE GRAMMARS

• pushdown automata define context-free languages;

• Turing machines define recursively-enumerable languages.

In the Chomsky hierarchy, context-free languages (CFLs) are a strict generalization ofregular languages.

regular languages context-free languages

regular expressions context-free grammars (CFGs)finite-state machines pushdown automatapaths derivations

Context-free grammars define CFLs. They are sets of permissible productions whichallow you to derive strings composed of surface symbols. An important feature of CFGsis recursion, in which a nonterminal can be derived from itself.

More formally, a CFG is a tuple hN, ⌃, R, Si:

N a set of non-terminals⌃ a set of terminals (distinct from N )R a set of productions, each of the form A ! �,

where A 2 N and � 2 (⌃ [ N)⇤

S a designated start symbol

Context free grammars provide rules for generating strings.

• The left-hand side (LHS) of each production is a non-terminal 2 N

• The right-hand side (RHS) of each production is a sequence of terminals or non-terminals, {n, �}⇤, n 2 N, � 2 ⌃.

A derivation t is a sequence of steps from S to a surface string w 2 ⌃⇤, which is theyield of the derivation. A derivation can be viewed as trees or as bracketings, as shownin Figure 11.4.

If there is some derivation t in grammar G such that w is the yield of t, then w is inthe language defined by the grammar. Equivalently, for grammar G, we can write that|T

G

(w)| � 1. When there are multiple derivations of w in grammar G, this is a case ofderivational ambiguity; if any such w exists, then we can say that the grammar itself isambiguous.

(c) Jacob Eisenstein 2014-2017. Work in progress.

• Derivation: sequence of rewrite steps from S to a string (sequence of terminals, i.e. words)

• Yield: the final string

• A CFG is a “boolean language model”

• A probabilistic CFG is a probabilistic language model:

• Every production rule has a probability; defines prob dist. over strings.

• Regular language <=> RegEx <=> paths in finite state machine

• Context-free language <=> CFG <=> derivations in pushdown automaton

Example

• All useful grammars are ambiguous: multiple derivations with same yield

• [Parse tree representations: Nested parens or non-terminal spans]

7

10.2. CONTEXT-FREE LANGUAGES 181

S

VP

NP

PP

NP

NNS

chopsticks

IN

with

NP

NN

sushi

VBZ

eats

NP

PRP

She

(S(NP(PRP She)(VP(VBZ eats)(NP(NP(NN sushi))(PP (INwith)(NP(NNS chopsticks)))))))

S

VP

PP

NP

NNS

chopsticks

IN

with

NP

NN

sushi

VBZ

eats

NP

PRP

She

(S(NP(PRP She)(VP(VBZ eats)(NP(NN sushi))(PP(INwith)(NP(NNS chopsticks))))))

Figure 10.1: Two derivations of the same sentence, shown as both parse trees and brack-etings

(c) Jacob Eisenstein 2014-2017. Work in progress. [Examples from Eisenstein (2017)]



Example

• All useful grammars are ambiguous: multiple derivations with same yield

• [Parse tree representations: Nested parens or non-terminal spans]

7


S

VP

NP

PP

NP

NNS

chopsticks

IN

with

NP

NN

sushi

VBZ

eats

NP

PRP

She


S

VP

PP

NP

NNS

chopsticks

IN

with

NP

NN

sushi

VBZ

eats

NP

PRP

She





S

VP

NP

PP

NP

NNS

chopsticks

IN

with

NP

NN

sushi

VBZ

eats

NP

PRP

She


S

VP

PP

NP

NNS

chopsticks

IN

with

NP

NN

sushi

VBZ

eats

NP

PRP

She




[Examples from Eisenstein (2017)]



Constituents• Constituent tree/parse is one representation of sentence’s syntax.

What should be considered a constituent, or constituents of the same category?

• Substitution tests

• Pronoun substitution

• Coordination tests

• Simple grammar of English

• Must balance overgeneration versus undergeneration

• Noun phrases

• NP modification: adjectives, PPs

• Verb phrases

• Coordination...

8

Parsing with a CFG

• Task: given text and a CFG, answer:

• Does there exist at least one parse?

• Enumerate parses (backpointers)

• Cocke-Kasami-Younger algorithm

• Bottom-up dynamic programming:Find possible nonterminals for short spans of sentence, then possible combinations for higher spans

• Requires converting CFG to Chomsky Normal Form(a.k.a. binarization)

9

CKY

0:1

0:2

0:3

2:3

GrammarAdj -> yummyNP -> foodsNP -> storeNP -> NP NPNP -> Adj NP

1:2

1:3

Adj NP NP

10

For cell [i,j] (loop through them bottom-up) For possible splitpoint k=(i+1)..(j-1): For every B in [i,k] and C in [k,j], If exists rule A -> B C, add A to cell [i,j] (Recognizer) ... or ...

yummy foods store

Recognizer: per span, record list of possible nonterminals

Parser: per span, record possible ways the nonterminal was constructed.add (A,B,C, k) to cell [i,j] (Parser)

0 1 2 3

CKY

0:1

0:2

0:3

2:3


1:2

1:3

Adj NP NP

NP

10


yummy foods store



0 1 2 3

CKY

0:1

0:2

0:3

2:3


1:2

1:3

Adj NP NP

NP NP

10


yummy foods store



0 1 2 3

CKY

11

0:1


yummy foods store

0:2

0:3

2:3


1:2

1:3

Adj NP NP

NP NP



0 1 2 3

CKY

12

0:1


yummy foods store

0:2

0:3

2:3


1:2

1:3

Adj NP NP

NP NP



0 1 2 3

CKY

12

0:1


yummy foods store

0:2

0:3

2:3


1:2

1:3

Adj NP NP

NP NP



0 1 2 3

NP

CKY

12

0:1


yummy foods store

0:2

0:3

2:3


1:2

1:3

Adj NP NP

NP NP



0 1 2 3

NPNP

Context-free Grammarbrenocon/anlp2017/lectures/11-cfg.pdf · • (10.3) *The cat that the dog is...

Documents

Transcript of Context-free Grammarbrenocon/anlp2017/lectures/11-cfg.pdf · • (10.3) *The cat that the dog is...