Parser Evalua+on
Transcript of Parser Evalua+on
ParserEvalua+on
ParserEvalua+on
‣ Viewaparseasasetoflabeledbrackets/cons+tuents
S(0,3)
NP(0,1)
PRP(0,1)(butstandardevalua+ondoesnotcountPOStags)
VP(1,3),VBD(1,2),NP(2,3),PRP(2,3)
S
NPVP
She saw it
VBD PRPPRP
0123
NP
ParserEvalua+onS(0,3),NP(0,1),VP(1,3),NP(2,3),PRP(0,1),VBD(1,2),PRP(2,3)
S
NP
She saw it
NN PRPPRP
0123
NP
S(0,3),NP(0,2),NP(2,3),PRP(0,1),NN(1,2),PRP(2,3)
‣ Precision:numberofcorrectpredic+ons/numberofpredic+ons =2/3
‣ Recall:numberofcorrectpredic+ons/numberofgolds =2/4
‣ F1:harmonicmeanofprecisionandrecall=(1/2*((2/4)-1+(2/3)-1))-1
=0.57(closertomin)
S
NPVP
She saw it
VBD PRPPRP
0123
NP
Results
‣ StandarddatasetforEnglish:PennTreebank(Marcusetal.,1993)
‣ “Vanilla”PCFG:~71F1
‣ BestPCFGsforEnglish:~90F1
‣ Otherlanguages:resultsvarywidelydependingonannota+on+complexityofthegrammar
‣ State-of-the-artdiscrimina+vemodels(usingunlabeleddata):95F1
GrammarPreprocessing
Binariza+on‣ Toparseefficiently,weneedourPCFGstobeatmostbinary(notCNF)
VP
VBD NP PP PP
sold thebook toher for$3
P(VP→VBDNPPPPP)=0.2
VP
VBD VP-[NPPPPP]
NP
PP
VP-[PPPP]
PP
‣ Solu+on:transformthetrees.Introduceintermediatespecialsymbolsthatrewritedeterminis+cally
P(VP→VBZPP)=0.1
…
P(VP→VBDVP-[NPPPPP])=0.2P(VP-[NPPPPP]→ NPVP-[PPPP])=1.0P(VP-[PPPP]→ PPPP)=1.0
PCFGIndependenceAssump+ons
11%9%
6%
NP PP DT NN PRP
9% 9%
21%
NP PP DT NN PRP
7%4%
23%
NP PP DT NN PRP
All NPs NPs under S NPs under VP
‣ Languageisnotcontext-free:NPsindifferentcontextsrewritedifferently
‣ [They]NPreceived[thepackageofbooks]NP
Ver+calMarkoviza+on
S^ROOT
NP^S VP^S
She saw it
VBD^VP PRP^VPPRP^NP
S
NP VP
She saw it
VBD PRPPRP
Basictree(v=1) v=2Markoviza+on
‣Whyisthisagoodidea?
AnnotatedTree
KleinandManning(2003)
‣ 75F1withbasicPCFG=>86.3F1withthishighlycustomizedPCFG(SOTAwas90F1atthe+me,butwithmorecomplexmethods)
‣ Augmentthegrammar:determinis+callytransformsymbolstobe“lesscontextfree”(binariza+onnotshownhere)
LexicalizedParsers§ Add“headwords”to
eachphrasalnode§ Syntac4cvs.seman4c
heads§ Headshipnotin(most)
treebanks§ Usuallyuseheadrules,
e.g.:§ NP:
§ TakeleFmostNP§ TakerightmostN*§ TakerightmostJJ§ Takerightchild
§ VP:§ TakeleFmostVB*§ TakeleFmostVP§ TakeleFchild
‣ Annotateeachgrammarsymbolwithits“headword”:mostimportantwordofthatcons+tuent
‣ Rulesforiden+fyingheadwords(e.g.,thelastwordofanNPbeforeapreposi+onistypicallythehead)
‣ CollinsandCharniak(late90s):~89F1withthese
DependencyParsing
LexicalizedParsing
S(ran)
NP(dog)
VP(ran)
PP(to)
NP(house)
DT(the) NN(house)TO(to)VBD(ran)DT(the) NN(dog)the housetoranthe dog
DependencyParsing
DT NNTOVBDDT NNthe housetoranthe dog
‣ Dependencysyntax:syntac+cstructureisdefinedbythesearcs‣ Head(parent,governor)connectedtodependent(child,modifier)‣ EachwordhasexactlyoneparentexceptfortheROOTsymbol,dependenciesmustformadirectedacyclicgraph
ROOT
‣ POStagssameasbefore,usuallyrunataggerfirstaspreprocessing
Whyaretheydefinedthisway?‣ Cons+tuencytests:‣ Subs+tu+onbyproform:thedogdidso[rantothehouse],he[thedog]rantothehouse
‣ Cleqing(Itwas[tothehouse]thatthedogran…)
‣ Dependency:verbistherootoftheclause,everythingelsefollowsfromthat
‣ Nono+onofaVP!
DependencyParsing
DT
NN
TO
VBD
DT
NN
the
house
to
ran
the
dog
‣ S+llano+onofhierarchy!Subtreesoqenalignwithcons+tuents
DependencyParsing
DT NNTOVBDDT NNthe housetoranthe dog
‣ Canlabeldependenciesaccordingtosyntac+cfunc+on
det
‣Majorsourceofambiguityisinthestructure,sowefocusonthatmore(labelingseparatelywithaclassifierworkspretywell)
nsubj
pobj
detprep
Dependencyvs.Cons+tuency:PPAtachment
‣ Cons+tuency:severalruleproduc+onsneedtochange
thechildrenatethecakewithaspoon
‣ Dependency:oneword(with)assignedadifferentparent
Dependencyvs.Cons+tuency:PPAtachment
‣Morepredicate-argumentfocusedviewofsyntax
‣ “What’sthemainverbofthesentence?Whatisitssubjectandobject?”—easiertoanswerunderdependencyparsing
‣ corenlp.run:spoonischildinsteadofwith.Thisisjustadifferentformalism
‣ Cons+tuency:ternaryruleNP->NPCCNP
Dependencyvs.Cons+tuency:Coordina+on
dogsinhousesandcats
‣ Dependency:firstitemisthehead
Dependencyvs.Cons+tuency:Coordina+on
dogsinhousesandcats
‣ Coordina+onisdecomposedacrossafewarcsasopposedtobeingasingleruleproduc+onasincons+tuency
‣ Canalsochooseandtobethehead‣ Inbothcases,headworddoesn’treallyrepresentthephrase—cons+tuencyrepresenta+onmakesmoresense
[dogsinhouses]andcats dogsin[housesandcats]
Shiq-ReduceParsing(seenotes)