6863J Natural Language Processing Lecture 8: Not an Earley finish Instructor: Robert C Berwick...

6•863J Natural Language ProcessingLecture 8: Not an Earley finish

Instructor: Robert C• Berwickberwick@csail•mit•edu

6•863J/9•611J SP05

The Menu Bar• Administrivia:Agenda:

Earley’s algorithmTime complexityParsing strategies: Earley algorithmWhat do people do?

6•863J/9•611J SP05

Example Grammar

S NP VP NP NameNP Det N N pjsNP NP PP N elephantVP V NP V shotVP VP PP P inPP P NP Det an

Det myName I

I shot an elephant in my pjs Name V Det N P Det N

6•863J/9•611J SP05

Marxist analysis ---- Rules -----Start -> SS -> NP VPNP -> Det NNP -> NP PPNP -> NamePP -> P NPVP -> VP PPVP -> V NPVP -> V---- Lexicon -----an Detmy Detshot Vshot Nmy Detelephant Npajamas NI Namein P

I shot an elephant in my pajamas

6•863J/9•611J SP05

State set simulation of nondeterministic machine

• State Set S0 = set of all states (‘edges’) we can be in after reading 0 words/pos

• March along constructing each state set Si+1 from previous state set state set Si

6•863J/9•611J SP05

State-set constructionInitialize: S0initial state set= initial state edge [Start Sn] -closure of this set under predict, complete Loop: For word i=1,…,n Si+1 computed from Si

(using scan, predict, complete)

scan; then predict, complete

Final: Is a final edge in Sn? [Start Sn]Sn ?

6•863J/9•611J SP05

The basic state representation: an item triple

[NP Det • N, 0, 2]

Dotted ruleStart position(left edge)

Progress position(right edge)

6•863J/9•611J SP05

Another way to view it

[NP Det N• 0 2]

N

guy

Scan

Det

theScan

[NP • Det N 0 0] [NP Det • N 0 1] [NP Det N • 0 2]

6•863J/9•611J SP05

Earley Parser

6•863J/9•611J SP05

The chart represents ambiguity by multiple back links in matrix

1 2 3 4 5 6 70 NP S S S

1 v, n VP VP

2 d NP PP

3 n

4 p PP

5 d NP

6 n

Startposof edge

End position of edge

6•863J/9•611J SP05

[S •NP VP, 0,0]

Example: Top-down init w/ chart

0 I 1 2 shot 3 an 4 elephant 5 in 6 my 7 pjs

We are constructing State set S0 -

6•863J/9•611J SP05

[NP • Name, 0, 0]

In picture form

State set S0 now done

[S •NP VP, 0, 0]

[NP • D N, 0, 0] (from td rule, or predict)

[NP • NP PP, 0, 0]+ all POS expansions


6•863J/9•611J SP05

Construct S1 from S0: Scan to next word…follow the bouncing dot…

I shot an elephant in my pjs

[S •NP VP, 0, 0][NP • Det N, 0, 0][NP • Name, 0, 0]

[NP • NP PP, 0, 0]

[NP Name •, 0, 1]

6•863J/9•611J SP05

The Fundamental Rule(“complete”) Applies…

• As time goes by…• Actually, as NP goes by…• We can also extend the length of all the

other edges that had an NP with a dot before them…

• That is,

6•863J/9•611J SP05

In picture form

State set S1 now done

[S NP • VP, 0, 1]

[NP NP • PP, 0, 1]


[NP Name •, 0, 1]

[VP • V NP, 1, 1][VP • VP PP, 1, 1][PP • P NP, 1, 1]

completecompletepredictpredictpredict

6•863J/9•611J SP05

Scan Verb - extend edge


NP N •

S NP • VP

[VP V • NP, 1,2]

VP • VP PP

What next? … Predict NP, add all edgescorresponding to expansion of NP

6•863J/9•611J SP05

Picture: Complete combines edges (The “fundamental rule”)

NP D N •

NP • NP PP


NP N •

S NP • VP

VP V NP •

VP VP • PP

S NP VP •

6•863J/9•611J SP05

State set construction – cols in chart

State Set 0

0 Start • S

0 S • NP VP

0 NP • Name

0 NP • NP PP

0 NP • Det N

State Set i

1 V •

1 VP V • NP

2 NP • Det N

2 NP • NP PP

2 NP • Name

State Set n

6 N •

5 NP Det N •

4 PP P NP •

5 NP NP • PP

2 NP NP PP •

1 VP VP PP •

7 PP • P NP

1 VP V NP •

2 NP NP • PP

0 S NP VP •

1 VP VP • PP

7 P •

0 Start S •

… …

6•863J/9•611J SP05

How does each of the 3 ops change Si?

• Scan: (jump over a token)

• Before: [A tk, i-1] in State Set Si-1 & word i= t

• Result: Add [A t k, i] to State Set Si

• (Do this for all items [triples] in State Set Si-1 )

• Predict (Push): (encounter nonterminal)

• Before: [A Bk, i-1] , B= a nonterminal, in Si

then• After: Add all new triples of form [B , i, i] to State

Set Si

• Complete(Pop): (finish w/ nonterminal)

• Before: If Si contains triple in form [B k, i] then

• After: go to state set Sk and for all rules of form [A

Bk, i-1], add triples [A B k, i] to state set Si

6•863J/9•611J SP05

The main dealinput x = x1 …… xn

• S0 = {[Start •S, 0, 0]}• For 0 i n do:

Process each item s Si in order by applying to it the single applicable operation among:(a) Predictor (adds new items to Si )

(b) Completer (adds new items to Si )

(c) Scanner (adds new items to Si+1 )• If Si+1 = , reject the input• If i= n and S n = {[Start S • , 0, n],…} then

Accept then input; else reject

6•863J/9•611J SP05

Earley’s Algorithm: Predictor

• Predictor(AB, [i,j])

A B

A

B

i j B

B

Bi

For each rule: Add:

ABE

BCD

Input Rule

Example

6•863J/9•611J SP05

Predictor (wishor)

• Predict (Push): • Before: [A B k, i] , B=nonterminal, in

Si then• After: Add all new edges of form [B ,

i+1, i+1] to State Set Si+1

• Cries out its need for a phrase of type B

6•863J/9•611J SP05

Earley’s Algorithm: Scanner

• Scanner(AB, [i,j])

A B

A

B

i j

AB

A

B

i j+1

w

Bw

For each rule: Add edge:

Aw Aw

Input Rule

Example

6•863J/9•611J SP05

Scan – formally (“Find a word/POS”)

• Scan: (jump over a token)

• Before: [A t k, i] in State Set Si & word i= t

• Result: Add [A t k, i+1] to State Set Si+1

6•863J/9•611J SP05

Earley’s Algorithm: Completer

• Completer(B, [i,j])

AB

A

k j

For each edgeB

B

i j A B

A

B

k i

B

Add:

DEA DEA

ABC

Input Rule

Example

6•863J/9•611J SP05

More precisely

• Complete(Pop): (finish w/ phrase)• Before: If Si contains e in form [B k, i] then

go to back to state set Sk and for all rules of form

[A B j, k], add edges E’ [A B j, i] to state set Si

6•863J/9•611J SP05

“The fundamental rule”: glues smaller trees into larger ones

VP V NP•

NP d n • VP V • NP

=

shot start pos= 1, len 1

an elephantstart= 2, len=2

1 2 2 4

start pos= 1, len 3 1 4

6•863J/9•611J SP05

Earley’s Algorithm: Rules

Aw Aw

Scanner

ABE

BCD

Predictor

DEA DEA

ABC

Completer

SAB

Initialization

6•863J/9•611J SP05

State set construction – cols in chart

State Set 0

0 Start • S

0 S • NP VP

0 NP • Name

0 NP • NP PP

0 NP • Det N

State Set i

1 V •

1 VP V • NP

2 NP • Det N

2 NP • NP PP

2 NP • Name

State Set n

6 N •

5 NP Det N •

4 PP P NP •

5 NP NP • PP

2 NP NP PP •

1 VP VP PP •

7 PP • P NP

1 VP V NP •

2 NP NP • PP

0 S NP VP •

1 VP VP • PP

7 P •

0 Start S •

… …

6•863J/9•611J SP05

Take note of the start-stop indices

NP

Initially: S [i, k]

i k

the dogk k+2k+1

jPredictStart: k

S

Predict: NP [k, k]

Scan: NP [k, k+1]

Scan: NP [k, k+2]

Complete: NP [k, k+2]

After NP: S [i, k+2]

6•863J/9•611J SP05

Indices: the left-hand edge ‘you are here’

• Predict: does not increment - NP[k,k]

• Scan: does increment, by 1, the left-hand edge: NP[k,k] NP[k,k+1] NP[k,k+2]

• Complete: increments left-hand edge of item in a previous State Set: S[j,k] S[j, k+2]

6•863J/9•611J SP05

Example Grammar

S NP VP NP NameNP Det N N pjsNP NP PP N elephantVP V NP V shotVP VP PP P inPP P NP Det an

Det myName I

I shot an elephant in my pjs Name V Det N P Det N

0

0 Start • SInitialize

Remember this stands for (0, Start • S)

0

0 Start • S

0 S • NP VPpredict the kind of S we are looking for

Remember this stands for (0, S • NP VP)

0

0 START • S

0 S • NP VP

0 NP • Det N

0 NP • NP PP

0 NP • I

predict the kind of NP we are looking for(actually we’ll look for 3 kinds: any of the 3 will do)

0

0 START • S

0 S • NP VP

0 NP • Det N

0 NP • NP PP

0 NP • I

0 Det • my

0 Det • an

predict the kind of Det we are looking for (2 kinds)

0

0 START • S

0 S • NP VP

0 NP • Det N

0 NP • NP PP

0 NP • I

0 Det • my

0 Det • an

predict the kind of NP we’re looking for but we were already looking for these sodon’t add duplicates! Note that this happenedwhen we were processing a left-recursive rule•

0 I 1

0 START • S 0 NP I •

0 S • NP VP

0 NP • Det N

0 NP • NP PP

0 NP • I

0 Det • my

0 Det • an

scan: the desired word is in the input!

0 I 1


0 S • NP VP

0 NP • Det N

0 NP • NP PP

0 NP • I

0 Det • my

0 Det • an

scan: failure

0 I 1


0 S • NP VP

0 NP • Det N

0 NP • NP PP

0 NP • I

0 Det • my

0 Det • an scan: failure

0 I 1


0 S • NP VP 0 S NP • VP

0 NP • Det N 0 NP NP • PP

0 NP • NP PP

0 NP • I

0 Det • my

0 Det • an

attach the newly created NP(which starts at 0) to its customers (incomplete constituents that end at 0and have NP after the dot)

0 I 1




0 NP • NP PP 1 VP • V NP

0 NP • I 1 VP • VP PP

0 Det • my

0 Det • an

predict

0 I 1






0 Det • my 1 PP • P NP

0 Det • an

predict

0 I 1







0 Det • an 1 V • shot

predict

0 I 1








1 P • in

predict

0 I 1 shot 2

0 START • S 0 NP I • 1 V shot •







1 P • in

scan: success!

0 I 1 shot 2








1 P • in scan: failure

0 I 1 shot 2


0 S • NP VP 0 S NP • VP 1 VP V • NP






1 P • with

complete

0 I 1 shot 2



0 NP • Det N 0 NP NP • PP 2 NP • Det N

0 NP • NP PP 1 VP • V NP 2 NP • NP PP

0 NP • I 1 VP • VP PP 2 NP • I



1 P • with

predict

0 I 1 shot 2






0 Det • my 1 PP • P NP 2 Det • my

0 Det • an 1 V • shot 2 Det • an

1 P • with

predict (these next few stepsshould look familiar)

0 I 1 shot 2








1 P • with

predict

0 I 1 shot 2








1 P • with

scan (this time we fail sinceI is not the next word)

0 I 1 shot 2 an 3

0 START • S 0 NP I • 1 V shot • 2 Det an •







1 P • with

scan: success!

0 I 1 shot 2 an 3








1 P • in

0 I 1 shot 2 an 3


0 S • NP VP 0 S NP • VP 1 VP V • NP 2 NP Det • N






1 P • in

0 I 1 shot 2 an 3



0 NP • Det N 0 NP NP • PP 2 NP • Det N 3 N • eleph

0 NP • NP PP 1 VP • V NP 2 NP • NP PP 3 N • pjs




1 P • in

0 I 1 shot 2 an 3 eleph 4

0 START • S 0 NP I • 1 V shot • 2 Det an • 3 N eleph •






0 Det • a 1 V • shot 2 Det • a

1 P • in



0 S • NP VP 0 S NP • VP 1 VP V • NP 2 NP Det • N 2 NP Det N •






1 P • in

complete




0 NP • Det N 0 NP NP • PP 2 NP • Det N 3 N • eleph 1 VP V NP •

0 NP • NP PP 1 VP • V NP 2 NP • NP PP 3 N • pjs 2 NP NP • PP




1 P • in

complete(again!)






0 NP • I 1 VP • VP PP 2 NP • I 0 S NP VP •

0 Det • my 1 PP • P NP 2 Det • my 1 VP VP • PP


1 P • in

complete(again!)








0 Det • an 1 V • shot 2 Det • an 4 PP • P NP

1 P • in









1 P • in 0 START S •

complete(again!)










4 P • in

0 I 1 shot 2 an 3 eleph 4 in 5

0 START • S 0 NP I • 1 V shot • 2 Det an • 3 N eleph • 4 P in •








4 P • in



0 S • NP VP 0 S NP • VP 1 VP V • NP 2 NP Det • N 2 NP Det N • 4 PP P • NP







4 P • in




0 NP • Det N 0 NP NP • PP 2 NP • Det N 3 N • eleph 1 VP V NP • 5 NP • Det N

0 NP • NP PP 1 VP • V NP 2 NP • NP PP 3 N • pjs 2 NP NP • PP 5 NP • NP PP

0 NP • I 1 VP • VP PP 2 NP • I 0 S NP VP • 5 NP • I




4 P • in







0 Det • my 1 PP • P NP 2 Det • my 1 VP VP • PP 5 Det • my

0 Det • an 1 V • shot 2 Det • an 4 PP • P NP 5 Det • an


4 P • in


0 START • S 0 NP I • 1 V shot • 2 Det an• 3 N eleph • 4 P in •








4 P • in










4 P • in

0 I 1 shot 2 an 3 eleph 4 in 5 my 6

0 START • S 0 NP I • 1 V shot • 2 Det an • 3 N eleph • 4 P in • 5 Det my •






0 Det • a 1 V • shot 2 Det • an 4 PP • P NP 5 Det • an


4 P • in


0 START • S 0 NP I • 1 V shot • 2 Det my • 3 N eleph • 4 P in • 5 Det my •

0 S • NP VP 0 S NP • VP 1 VP V • NP 2 NP Det • N 2 NP Det N • 4 PP P • NP 5 NP Det • N







4 P • in


0 START • S 0 NP I • 1 V shot • 2 Det an • 3 N eleph • 4 P in • 5 Det my •


0 NP • Det N 0 NP NP • PP 2 NP • Det N 3 N • eleph 1 VP V NP • 5 NP • Det N 6 N • eleph

0 NP • NP PP 1 VP • V NP 2 NP • NP PP 3 N • pjs 2 NP NP • PP 5 NP • NP PP 6 N • pjs





4 P • in

0 I 1 shot 2 an 3 eleph 4 in 5 my 6 pjs 7

0 START • S 0 NP I • 1 V shot • 2 Det an • 3 N eleph • 4 P in • 5 Det my • 6 N pjs •






0 Det • a 1 V • shot 2 Det • an 4 PP • P NP 5 Det • a


4 P • in



0 S • NP VP 0 S NP • VP 1 VP V • NP 2 NP Det • N 2 NP Det N • 4 PP P • NP 5 NP Det • N 5 NP Det N •







4 P • in



0 S • NP VP 0 S NP • VP 1 VP V • NP 2 NP Det • N 2 NP Det N • 4 PP P • NP 5 NP Det • N 5 NP Det N •

0 NP • Det N 0 NP NP • PP 2 NP • Det N 3 N • eleph 1 VP V NP • 5 NP • Det N 6 N • eleph 4 PP P NP •

0 NP • NP PP 1 VP • V NP 2 NP • NP PP 3 N • pjs 2 NP NP • PP 5 NP • NP PP 6 N • pjs 5 NP NP • PP





4 P • in

0 I 1 shot 2 an 3 eleph 4 in my pjs 7

0 START • S 0 NP I • 1 V shot • 2 Det an • 3 N eleph • … 6 N pjs •

0 S • NP VP 0 S NP • VP 1 VP V • NP 2 NP Det • N 2 NP Det N • 5 NP Det N •

0 NP • Det N 0 NP NP • PP 2 NP • Det N 3 N • eleph 1 VP V NP • 4 PP P NP •

0 NP • NP PP 1 VP • V NP 2 NP • NP PP 3 N • pjs 2 NP NP • PP 5 NP NP • PP

0 NP • I 1 VP • VP PP 2 NP • I 0 S NP VP • 2 NP NP PP •

0 Det • my 1 PP • P NP 2 Det • my 1 VP VP • PP 1 VP VP PP •



4 P • in

0 I 1 shot 2 my 3 eleph 4 in my pjs 7







0 Det • an 1 V • shot 2 Det • an 4 PP • P NP 7 PP • P NP


4 P • in









1 P • in 0 START S • 1 VP V NP •

4 P • in 2 NP NP • PP











0 S NP VP •

1 VP VP • PP











0 S NP VP •

1 VP VP • PP

7 P • in











0 S NP VP •

1 VP VP • PP

7 P • in

0 START S •

0 Name 1 shot 2 an 3 eleph 4 in my pjs 7

0 Start • S 0 NP Name • 1 V shot • 2 Det my • 3 N eleph • … 6 N pjs •




0 NP • Name 1 VP • VP PP 2 NP • Name 0 S NP VP • 2 NP NP PP •

0 Det • the 1 PP • P NP 2 Det • the 1 VP VP • PP 1 VP VP PP •


1 P • in 0 Start S • 1 VP V NP •


0 S NP VP •

1 VP VP • PP

7 P • in

0 Start S •


0 Start • S 0 NP Name • 1 V shot • 2 Det an • 3 N eleph • …6 N pjs •









0 S NP VP •

1 VP VP • PP

7 P • in

0 Start S •











0 S NP VP •

1 VP VP • PP

7 P • in

0 Start S •

Complexity Max # of state sets = O(n)











0 S NP VP •

1 VP VP • PP

7 P • in

0 Start S •

Complexity Max size of state set (column)

|G|n

O(|G| n)











0 S NP VP •

1 VP VP • PP

7 P • in

0 Start S •

Max time to process one item in set

Any possibleitem in set i, soO(|G|n)

6•863J/9•611J SP05

Grand total

n x |G| n x |G| n = O(|G|2 n3)

# state sets x # items/state set x max time/item

6•863J/9•611J SP05

Left Recursion Kills Pure Top-Down Parsing …

VP

6•863J/9•611J SP05


VP

VP PP

6•863J/9•611J SP05


VP PP

VP

VP PP

6•863J/9•611J SP05


VP PP

VP

VP PP

VP PP

makes new hypothesesad infinitum before we’veseen the PPs at all

hypotheses try to predictin advance how many PP’s will arrive in input

6•863J/9•611J SP05

… but Earley’s Alg is Okay!

VP

PPVP

1 VP . VP PP

(in column 1)

6•863J/9•611J SP05


VP

V NP

1 VP V NP .

shot an eleph

VP

PPVP

1 VP . VP PP

(in column 1)

(in column 4)

6•863J/9•611J SP05


VP

V NP

VP

PPVP

V NP

complete

shot an eleph

1 VP VP . PP

VP

PPVP

1 VP . VP PP

(in column 1)

(in column 4)

6•863J/9•611J SP05


VP

V NP

VP

PPVP

V NPshot an eleph

in a pjs

1 VP VP PP .

VP

PPVP

1 VP . VP PP

(in column 1)

(in column 7)

6•863J/9•611J SP05


VP

V NP

VP

PPVP

V NPshot an elephant

in my pjs

1 VP VP PP .

VP

PPVP

1 VP . VP PP

can be reused(in column 1)

(in column 7)

6•863J/9•611J SP05


VP

V NP

VP

PPVP

V NP

VP

PP

1 VP VP . PP

shot an eleph

In my pjs

VP

PPVP

1 VP . VP PP


(in column 7)

complete

6•863J/9•611J SP05


VP

V NP

VP

PPVP

V NP

VP

PP

shot an eleph

in my pjs

in his bed

1 VP VP PP .

VP

PPVP

1 VP . VP PP


(in column 10)

6•863J/9•611J SP05


VP

V NP

VP

PPVP

V NP

VP

PP

shot an eleph

in my pjs

in his bed

1 VP VP PP .

VP

PPVP

1 VP . VP PP

can be reused again(in column 1)

(in column 10)

6•863J/9•611J SP05


VP

V NP

VP

PPVP

V NP

VP

PPVP

1 VP . VP PP

can be reused again

VP

PP

VP

PP

1 VP VP . PP

shot an eleph

in my pjs

in his bed

(in column 1)

(in column 10)

complete

6•863J/9•611J SP05

How this algorithm is clever• Handling left-recursion• Tail-recursion trick• Example: John’s brother’s book• NP NP NP | NP Noun | Noun ’s• Note how this loops on endless call to

NPNP

NP

NP

’s

NPNoun

bookJohn NP

’s brother

NP …but predict cutsoff after 1 round!

6•863J/9•611J SP05

Note tail recursion• Stshot set S0 :• Add triples: [NPNoun, 0, 0]

[NPNPNP, 0, 0] predict: ?[NPNPNP, 0, 0] … No

need!Duplicates!

Note tail recursion: the call returns to itself – so noneed to ‘keep’ return addresses in stack!

6•863J/9•611J SP05

The edge loops to itself:

[NPNPNP, 0, 0]


0 START . S 0 NP I . 1 V shot . 2 Det an . 3 N eleph … 6 N pjs .

0 S . NP VP 0 S NP . VP 1 VP V . NP 2 NP Det . N 2 NP Det N . 5 NP Det N .

0 NP . Det N 0 NP NP . PP 2 NP . Det N 3 N . eleph 1 VP V NP . 4 PP P NP .

0 NP . NP PP 1 VP . V NP 2 NP . NP PP 3 N . pjs 2 NP NP . PP 5 NP NP . PP

0 NP . I 1 VP . VP PP 2 NP . I 0 S NP VP . 2 NP NP PP .

0 Det . my 1 PP . P NP 2 Det . my 1 VP VP . PP 1 VP VP PP .

0 Det . an 1 V . shot 2 Det . an 4 PP . P NP 7 PP . P NP

1 P . in 0 START S . 1 VP V NP .

4 P . in 2 NP NP . PP

0 S NP VP .

1 VP VP . PP

7 P . in

0 START S .

completed a VP in col 4col 1 lets us use it in a VP PP structure


0 START . S 0 NP I . 1 V shot . 2 Det an . 3 N pjs. … 6 N pjs .

0 S . NP VP 0 S NP . VP 1 VP V . NP 2 NP Det . N 2 NP Det N . 5 NP Det N .

0 NP . Det N 0 NP NP . PP 2 NP . Det N 3 N . eleph 1 VP V NP . 4 PP P NP .

0 NP . NP PP 1 VP . V NP 2 NP . NP PP 3 N . pjs 2 NP NP . PP 5 NP NP . PP

0 NP . I 1 VP . VP PP 2 NP . I 0 S NP VP . 2 NP NP PP .

0 Det . my 1 PP . P NP 2 Det . my 1 VP VP . PP 1 VP VP PP .

0 Det . an 1 V . shot 2 Det . an 4 PP . P NP 7 PP . P NP

1 P . in 0 START S . 1 VP V NP .

4 P . in 2 NP NP . PP

0 S NP VP .

1 VP VP . PP

7 P . in

0 START S .

completed that VP = VP PP in col 7col 1 would let us use it in a VP PP structurecan reuse col 1 as often as we need

6•863J/9•611J SP05

What about parsing?

• If we retrieve all parses explicitly, then exponential

• We have to pack possible paths into the space

• Number the items, let’s see how this can work


0 Start • S 1 0 NP Name • 8 1 V shot • 16 2 Det an • 23 3 N eleph • 27 … 6 N pjs • 40

0 S • NP VP 2 0 S NP • VP 9 1 VP V • NP 17 2 NP Det • N 24 2 NP Det N • 28 5 NP Det N • 41

0 NP • Det N 3 0 NP NP • PP 10 2 NP • Det N 18 3 N • eleph 25 1 VP V NP • 29 4 PP P NP • 42

0 NP • NP PP 4 1 VP • V NP 11 2 NP • NP PP 19 3 N • pjs 26 2 NP NP • PP 30 5 NP NP • PP 43

0 NP • Name 5 1 VP • VP PP 12 2 NP • Name 20 0 S NP VP • 31 2 NP NP PP • 44

0 Det • the 6 1 PP • P NP 13 2 Det • the 21 1 VP VP • PP 32 1 VP VP PP • 45

0 Det • an7 1 V • shot 14 2 Det • an 22 4 PP • P NP 33 7 PP • P NP 46

1 P • an 15 0 Start S • 34 1 VP V NP • 47

4 P • in 35 2 NP NP • PP 48

0 S NP VP • 49

1 VP VP • PP 50

7 P • in 51

0 Start S • 52

6•863J/9•611J SP05

This implicitly shares structure:packed, shared forest

NP [VP VP]

[S, S]

Name

I

V

shot NP

Det

an

N

PP

P NP

Det N

eleph my pjsin

NP

VP

6•863J/9•611J SP05

+ and - for Earley algorithm• Note that Earley algorithm already does

some top-down filtering: any item (state, or triple) added to a given State Set must be compatible with some derivation on the left, e.g.,Swi where wi is the sentence seen so far

wi

S

*

6•863J/9•611J SP05

+ and – for Earley algorithm

• Manipulates explicit representation of grammar – transparent

• Can add simple method to deal with optional elements in the rule, e.g., NPName (PP) or Kleene *, etc.

6•863J/9•611J SP05

What are the weak points?

• Explicit representation of rules: wastes time building them.

• Better to ‘compile’ items and their connections ahead of time: compute links between items in advance

• Example: if sentence starts an NP, and NP expands as Det N, go directly from S to Det N. (this is what LR parsing method uses)

• Does filtering on left but not on the right

6•863J/9•611J SP05

Weak points of Earley, cont’d• Example: What did John eat• The Earley parser will build all possible

expansions Swh, Sdid, etc.• This suggests that we define the following

lookahead filter for nonterminals A:FIRST(A)= {x|A x}, x= 1 tokene.g., FIRST(S)= who, did, the, etc.

• Apply by throwing out triples that whose first item doesn’t match this

• Reduces state set size by a factor of 1.5-1.8

6•863J/9•611J SP05

Use of FIRST calculation

• Example: John ate the ice-cream on the table

• Using FIRST, we throw out the expansions of NP that start with Det, etc. – and there are typically hundreds of these

6•863J/9•611J SP05

Anything else?

• If anything, Earley parsing is too good – it gets all the parses, even ones that people do not

• We shall see how to deal with this, using probabilities on rules; and

• Other parsing methods• But first, what do people do?• Consider the examples

6•863J/9•611J SP05

Precomputing predictor• Predictor(AB, [i,j]):

For every grammar rule B:add a rule B at [j,j]

• Note that this depends only on B and the grammar• We can save time by pre-computing predictor

For every grammar rule B:PREDICT[B].append(B)

Repeat until nothing new is added: For every B:

For every CD in PREDICT[B]:For every grammar rule D:

PREDICT[B].append(D)

6•863J/9•611J SP05

Worst case time for Earley algorithm

• Is the cubic bound ever reached in artificial or natural languages?

• Here is the artificial ‘worst case’ - # of parses arbitrarily large with sentence length; infinite ambiguity

• Here is the grammar: SSS, SSa{a, aa, aaa, aaaa,…}

• # of binary trees with n leaves= 1,1,2,5,14,42,132,429,1430,4862,16796,…=

6•863J/9•611J SP05

This does indeed get costly -Verb NP PP example

30201000

2

4

6

8

10

12

NP–PP ambiguity sentences NP P NP•••

with Earley's algorithm

length,words

Tim

e, s

econ

dstime, secs•

6•863J/9•611J SP05

Anything else?

• If anything, Earley parsing is too good – it gets all the parses, even ones that people do not

• We shall see how to deal with this, using probabilities on rules; and

• Other parsing methods• But first, what do people do?• Consider the examples

6•863J/9•611J SP05

Wouldn’t matter so much – but it does seem to match what people do

• Both left- and right- branching ‘structures’ seem to be readily parseable by people without any sort of memory load (all other things being equal)

• John’s brother’s mother’s aunt…•• I believed that Mary shot that Fred knew

that Bill shoveled snow

6•863J/9•611J SP05

Pictures of this••

John’s brother’s bookshot that Fred knew that Bill shoveled snow

6•863J/9•611J SP05

So what’s hard for people to process?

The rat died

the cat chased

the dog bit “center-embedded”

6•863J/9•611J SP05

Why is this hard?

• Model: people have to “hold onto” open predicates (definition: open if verb+arguments have not yet been put together)

• In the preceding example, we have to hold onto a stack of Subjects (the rat, the cat, the dog…) before the corresponding verbs are seen

• This even shows up in unexpected places – speech intonational pattern actually seems to transduce center-embedded structures into left- or right- branching ones

6•863J/9•611J SP05

Chomsky & Miller, 1959-63 analysis

the dog that chased the cat that bit the rat

6•863J/9•611J SP05

Parsing vs• intonational contours

Syntactic structure is center-embedded:

NP

NP

the rat

S

thatchased

NP VP

NP

the cat

proS

thatbit

NP VP

NPthe ratpro

NP

6•863J/9•611J SP05

But the intonational structure follows this:

NP

NP

the rat

S

thatNP

chased

VP

pro

NP

the catNP

Suggests 2-stage parser (proposed by C&M):Stage 1: parse into ‘flat’ structureStage 2: make 2nd pass & rearrange hierarchically

6•863J/9•611J SP05

Also hints at how to do semantic interpretation – akin to syntax-driven translation

• Recall from compilers: if we complete the right-hand side of a rule, we can now fire off any associsated semantic action (because we now have the item and all its ‘arguments’

• This amounts to getting left-most complete subtree at each point to interpret

• Example:VPV NP , e•g•, “shot the ice-cream”Can now ‘interpret’ thispair syntactic, ‘semantic’ rule: VPV NP, apply VP(NP)

6•863J/9•611J SP05

One more search space enumeration that will be of some value

• Left-corner parsing• Looks bottom-up in serial fashion for the

first symbol (left-corner) of a phrase; and then tries to confirm the rest of the phrase top-down

• Tries to combine best features of b-u and t-d

• Clearly geared to the way a particular language (eg English) is set up

6•863J/9•611J SP05

A picture of left-corner parsing

S

NP

the

VP

Noun

SNP VP

NPthe Noun

12

find

predict

shot

VPshot NP

6•863J/9•611J SP05

This works well

• In a language like English:• A head-first language (function-

argument)• What about German, Dutch, Japanese?• dat het mesije van Holland houdt• “that the girl from Holland liked”• These are head-final languages

6•863J/9•611J SP05

Parsing as search: different search patterns different parsing methods different traversals thru

Top-down Bottom-up

Serial Recursive descent

Shift-reduce (LR)

Parallel(simulate ndet machine)

Earley CKY(Cocke-Kasami-Younger)

6•863J/9•611J SP05

A pictureRecursive descent

Earley 1

2 2

3

6•863J/9•611J SP05

And a few moreshift-reduce

CKY

2

13

41

2

1

6•863J/9•611J SP05

What about constructing grammars?

6•863J/9•611J SP05

Grammars for natural languages

• Where do the rules come from?• Roughly: read them off of parse trees…• A “rule-based”, construction-based

point of view• Take ‘surface’ phrase patterns (mostly)• But we still want to map to an

underlying ‘logical’ form• How do we start out?

6•863J/9•611J SP05

Reading rules from parse trees…

SNP VPVPV NPNPDet NNP N*

Can’t we get a computer to do this?

6•863J/9•611J SP05

Key elements – part 1

• Establish basic phrase types: S, VP, NP, PP, …

• Where do these come from???

6•863J/9•611J SP05

What kinds of phrases are there?

• Noun phrases, verb phrases, adjectival phrases (“green with envy”), adverbial phrases (“quickly up the hill”), prepositional phrases (“off the wall”), etc.

• In general: grounded on lexical items• Shows us the constraints on context-

free rules for natural grammars• Example:

6•863J/9•611J SP05

Phrase types are constrained by lexical projection

Verb Phrase Verb Noun Phrase “is-a” (“kick the ball”)

Prepositional Phrase Preposition Noun Phrase(“on the table”)

Adjective Phrase Adjective Prep. Phrase

(“green with envy”)Etc. … what is the pattern?

6•863J/9•611J SP05

Function-argument relation

XP X arguments, where X= Noun, Verb, Preposition, Adjective (all lexical

categories in the language)Like function-argument structure(so-called “Xbar theory”)Constrains what grammar rules cannot be:Verb Phrase Noun Noun Phraseor evenVerb Phrase Noun Phrase Verb Noun

Phrase

6•863J/9•611J SP05

English is function-argument form

function

at

args

green

soldthe stock

a bargain price

with envy

the over-priced stock

6•863J/9•611J SP05

Other languages are the mirror-inverse: arg-function

at

green

soldthe stock

a bargain price

with envy

the over-priced stockThis is like Japanese

6•863J/9•611J SP05

Key elements – part 2

• Establish verb subcategories• What are these?

• Different verbs take different # arguments• 0, 1, 2 arguments (‘complements’)• Poirot thought; Poirot thought the gun; Poirot

thought the gun was the cause.• Some verbs take certain sentence complements:• I know who John shot/? I think who John shot

propositional types: • Embedded questions: I wonder whether…• Embedded proposition: I think that John shot

Mary

6•863J/9•611J SP05

Key elements

• Subtlety to this• Believe, know, think, wonder,…

• ? I believe why John likes ice-cream• I know why John likes ice-cream• I believe that John likes ice-cream• I believe (that) John likes ice-cream

• # args, type: Verb subcategories• How many subcategories are there?• What is the structure?

6•863J/9•611J SP05

Idea for phrases

• They are based on ‘projections’ of words (lexical items) – imagine features ‘percolating’ up

know [V +proposition]

XP [ ]V +proposition

6•863J/9•611J SP05

Heads of phrases

know [V +proposition]

V +proposition

6•863J/9•611J SP05

The parse structure for ‘embedded’ sentences

I believe (that) John likes ice-cream

S

NP VP

I

V

believe

that J. likes ice-cream

6•863J/9•611J SP05

New phrase type: S-bar

NP VP

I

V

believe

S

Sbar

that J. likes ice-cream

6•863J/9•611J SP05

Sbar VP

V

believe

Sbar

that

Comp S

J. likes ice-cream

6•863J/9•611J SP05

Sbar VP

V

believe

Sbar

Comp S

J. likes ice-cream

6•863J/9•611J SP05

In fact, true for all sentences…

Comp S

S

J. likes ice-cream

SbarJohn likes ice-cream

Why?

6•863J/9•611J SP05

What rules will we need?

• (U do it..)

6•863J/9•611J SP05

Verb types - continued

• What about:Clinton admires honesty/Honesty

admires ClintonHow do we encode these in a CFG?Should we encode them?• Colorless green ideas sleep furiously• Revolutionary new ideas appear

infrequently

6•863J/9•611J SP05

Features

6•863J/9•611J SP05

The trouble with tribbles

morphology of a single word: Verb[head=thrill, tense=present, num=sing, person=3,…] thrills

projection of features up to a bigger phrase VP[head=, tense=, num=…] V[head=, tense=, num=…] NP

provided is in the set TRANSITIVE-VERBS

agreement between sister phrases:S[head=, tense=] NP[num=,…] VP[head=, tense=,

num=…]

provided is in the set TRANSITIVE-VERBS

6•863J/9•611J SP05

3 Common Ways to Use Features

NPVerb

VPNP

S

A roller coaster thrills every teenager

Verb[head=thrill, tense=present, num=sing, person=3,…]

thrillsVP[head=, tense=, num=…] V[head=, tense=,

num=…] NPS[head=, tense=] NP[num=,…] VP[head=, tense=,

num=…]

num=singnum=sing

(comprehensionperspective)

num=sing

thrills

6•863J/9•611J SP05

CFG Solution

• Encode constraints into the non-terminals• Noun/verb agreement

S SgSS PlSSgS SgNP SgVPSgNP SgDet SgNom

• Verb subcategories:IntransVP IntransVTransVP TransV NP

6•863J/9•611J SP05

Problems with this – how much info?

6•863J/9•611J SP05

Agreement gets complex…

POS

SUBPOS

GENDER

NUMBER

CASE

POSSG

POSSNPERSON

TENSEDCOMP

NEG

VOICE

VAR

–Czech: AGFS3----1A----

Lots of features(tense, number, person, gaps, vowels, commas, wh, etc., etc....)

Sorry, that’s just how language is … You know too much to write it down easily!

He

gone

has

6•863J/9•611J SP05

Other sentence types

• Questions: • Will John eat ice-cream?• Did John eat ice-cream?

• How do we encode this?

6•863J/9•611J SP05

`Empty’ elements or categories• Where surface phrase is displaced from its canonical

syntactic position• Examples:

• The ice-cream was eaten vs.• John ate the ice-cream• What did John eat?• What did Bill say that that John thought the cat

ate?• For What x, did Bill say… the cat ate x• Bush is too stubborn to talk to• Bush is too stubborn [x to talk to Bush]• Bush is too stubborn to talk to the Pope• Bush is too stubborn [Bush to talk to the Pope]

6•863J/9•611J SP05

More interesting clause types• Apparently “long distance” effects:

‘displacement’ of phrases from their ‘base’ positions

1. So-called ‘wh-movement’:What did John eat ?

2. Topicalization (actually the same)On this day, it snowed two feet.

3. Other cases: so-called ‘passive’:The eggplant was eaten by John

• How to handle this?

6•863J/9•611J SP05

We can think of this as ‘fillers’ and ‘gaps’

• Filler= the displaced item• Gap = the place where it belongs, as argument• Fillers can be NPs, PPs, S’s• Gaps are invisible- so hard to parse! (we have

to guess)• Can be complex:

Which book did you file__ without__ reading__ ?

Which violins are these sonatas difficult to play__ on ___

6863J Natural Language Processing Lecture 8: Not an Earley finish Instructor: Robert C Berwick...

Documents

Transcript of 6863J Natural Language Processing Lecture 8: Not an Earley finish Instructor: Robert C Berwick...