Administrivia CS388: Natural Language Processing Lecture ...
Transcript of Administrivia CS388: Natural Language Processing Lecture ...
CS388:NaturalLanguageProcessingLecture13:Seman8csI
GregDurrett
SlidesadaptedfromDanKlein,UCBerkeley
Administrivia‣Mini2due*today*at5pm
Recall:Dependencies
DT NNTOVBDDT NNthe housetoranthe dog
‣ Dependencysyntax:syntac8cstructureisdefinedbydependencies‣Head(parent,governor)connectedtodependent(child,modifier)‣ EachwordhasexactlyoneparentexceptfortheROOTsymbol‣Dependenciesmustformadirectedacyclicgraph
ROOT
Recall:ShiX-ReduceParsing
IatesomespagheZbolognese
ROOT
‣ State:Stack:[ROOTIate]Buffer:[somespagheZbolognese]
‣ LeX-arc(reduceopera8on):Letdenotethestack�‣ “Poptwoelements,addanarc,putthembackonthestack”
‣ State:Stack:[ROOTate]Buffer:[somespagheZbolognese]
I
�|w�2, w�1 ! �|w�1 w�1w�2 isnowachildof,
Wherearewenow?‣ Earlyintheclass:sentencesarejustsequencesofwords
‣Whyisthisuseful?Whatdoesthisallowustodo?
‣Nowwecanunderstandthemintermsoftreestructuresaswell
‣ We’regoingtoseehowparsingcanbeasteppingstonetowardsmoreformalrepresenta8onsoflanguagemeaning
Today‣ First-orderlogic
‣ CCGparsingfordatabasequeries
‣ Composi8onalseman8cswithfirst-orderlogic
‣ Lambda-DCSforques8onanswering
First-OrderLogic
First-orderLogic
‣ singsisapredicate(withoneargument),func8onf:en8ty=>true/false
‣ Powerfullogicformalismincludingthingslikeen88es,rela8ons,andquan8fica8ons
‣ [[sings]]=denota,on,setofen88eswhichsing(sortoflikeexecu8ngthispredicateontheworld—we’llcomebacktothis)
‣ Proposi8ons:leta=Itisday,b=Itisnight‣ a∨b=eitheraistrueorbistrue,a=>¬b=aimpliesnotb
‣ Morecomplexstatements:“LadyGagasings”
‣ sings(LadyGaga)=trueorfalse,havetoexecutethisagainstsomedatabase(calledaworld)
Quan8fica8on
‣ ∀xsings(x)∨dances(x)=>performs(x)
‣ ∃y∀xfriend(x,y)
‣ Universalquan8fica8on:“forall”operator
‣ Existen8alquan8fica8on:“thereexists”operator
‣ ∀x∃yfriend(x,y)‣ Sourceofambiguity!“Everyoneisfriendswithsomeone”
“Everyonewhosingsordancesperforms”
“Someonesings”‣ ∃xsings(x)
LogicinNLP‣ Ques8onanswering:
‣ Informa8onextrac8on: LadyGagaandEminemarebothmusicians
∀xmusician(x)=>performer(x)
musician(LadyGaga)∧musician(Eminem)
Then:performer(LadyGaga)∧performer(Eminem)
WhoarealltheAmericansingersnamedAmy?
λx.na8onality(x,USA)∧sings(x)∧firstName(x,Amy)
‣ Func8onthatmapsfromxtotrue/false,likefilter.Executethisontheworldtoanswertheques8on
‣ Cannowdoreasoning.Maybeknow:
‣ Lambdacalculus:powerfulsystemforexpressingthesefunc8ons
Composi8onalSeman8cswithFirst-OrderLogic
Truth-Condi8onalSeman8csId Name Alias Birthdate Sings?
e470 StefaniGermanooa LadyGaga 3/28/1986 Te728 MarshallMathers Eminem 10/17/1972 T
‣ Databasecontainingen88es,predicates,etc.
‣ Truth-condi8onalseman8cs:sentenceexpressessomethingabouttheworldwhichiseithertrueorfalse
NP VP
NNP NNP
S
VBPLadyGaga sings
‣ Denota8on:evalua8onofsomeexpressionagainstthisdatabase
‣[[LadyGaga]] = e470
denota8onofthisstringisanen8ty
‣[[sings(e470)]] = True
denota8onofthisexpressionisT/F
ParsestoLogicalForms
NP VP
NNP NNP
S
VBPLadyGaga sings
e470
λy. sings(y)takesoneargument(y,theen8ty)andreturnsalogicalformsings(y)
λy. sings(y)
sings(e470)
‣Wecanusethesyntac8cparseasabridgetothelambda-calculusrepresenta8on,buildupalogicalformcomposi,onally
func8onapplica8on:applythistoe470ID
ParsestoLogicalForms
NP
VPNNP NNP
S
VBPLadyGaga
sings
e470
λy. sings(y)
sings(e470) ∧ dances(e470)
VP
CC VP
VBPdancesλy. dances(y)
and
VP:λy.a(y)∧b(y)->VP:λy.a(y)CCVP:λy.b(y)
λy. sings(y) ∧ dances(y)
‣ Generalrules:S:f(x)->NP:xVP:f
ParsestoLogicalForms
NP
NNP NNP
S
VBDLadyGaga was
e470
λx.λy. born(y,x)
born(e470,3/28/1986)
VP
NP
March28,1986born
λy. born(y, 3/28/1986)
VBN
VP
λy. born(y, 3/28/1986)
‣Howtohandletense:shouldweindicatethatthishappenedinthepast?‣ Func8ontakestwoarguments:firstx(date),theny(en8ty)
3/28/1986
Trickythings‣ Adverbs/temporality:LadyGagasangwellyesterday
∃e. type(e,sing) ∧ agent(e,e470) ∧ manner(e,well) ∧ time(e,…)‣ “Neo-Davidsonian”viewofevents:thingswithmanyproper8es:
‣ Quan8fica8on:Everyoneisfriendswithsomeone
‣ Generic:Catseatmice(allcatseatmice?mostcats?somecats?)
∀x ∃y friend(x,y)∃y ∀x friend(x,y)(differentfriends)(onefriend)
‣ Samesyntac8cparseforboth!Sosyntaxdoesn'tresolveallambigui8es
sings(Lady Gaga, time=yesterday, manner=well)
‣ Indefinite:Amyateawaffle ∃w. waffle(w) ∧ ate(Amy,w)
QAfromParsing
LadyGagaborn
NP VP
WHADVP
VBD
SQ
WRB
NNP NNP VBN
SBARQ
When was
λx. born(e470,x)‣ Executethisfunc8onagainstaknowledgebasetoanswertheques8on
‣ Trickytoparseduetowh-movement…wouldbeeasierifwesaidLadyGagawasbornwhen
Seman8cParsing‣ Forques8onanswering,syntac8cparsingdoesn’ttellyoueverythingyouwanttoknow,butindicatestherightstructure
‣ Solu8on:seman,cparsing:manyformsofthistaskdependingonseman8cformalisms
‣ Twotoday:CCG(lookslikewhatwe’vebeendoing)andlambda-DCS
CCGParsing
CombinatoryCategorialGrammar‣ Steedman+Szabolcsi1980s:formalismbridgingsyntaxandseman8cs
‣ Syntac8ccategories(forthislecture):S,NP,“slash”categories
‣ S\NP:“ifIcombinewithanNPonmyleXside,Iformasentence”—verb
NP S\NP
Eminem singse728 λy. sings(y)
Ssings(e728)
‣ Parallelderiva8onsofsyntac8cparseandlambdacalculusexpression
‣ Whenyouapplythis,therehastobeaparallelinstanceoffunc8onapplica8onontheseman8csside
CombinatoryCategorialGrammar‣ Steedman+Szabolcsi1980s:formalismbridgingsyntaxandseman8cs‣ Syntac8ccategories(forthislecture):S,NP,“slash”categories‣ S\NP:“ifIcombinewithanNPonmyleXside,Iformasentence”—verb‣ (S\NP)/NP:“IneedanNPonmyrightandthenonmyleX”—verbwithadirectobject
NP S\NP
Eminem singse728 λy. sings(y)
Ssings(e728)
NP (S\NP)/NP
Oklahoma borderse101
Texase89NP
λx.λy borders(y,x)
S\NPλy borders(y,e89)
Sborders(e101,e89)
CCGParsing
ZeolemoyerandCollins(2005)
‣ “What”isaverycomplextype:needsanounandneedsaS\NPtoformasentence.S\NPisbasicallyaverbphrase(borderTexas)
‣ Lexiconishighlyambiguous—allthechallengeofCCGparsingisinpickingtherightlexiconentries
CCGParsing
Slidecredit:DanKlein
‣ “to”needsanNP(des8na8on)andN(parent)
BuildingCCGParsers
ZeolemoyerandCollins(2005)
‣Model:log-linearmodeloverderiva8onswithfeaturesonrules:
P (d|x) / expw
>
X
r2d
f(r, x)
!
‣ CanparsewithavariantofCKYEminem sings
NP S\NPe728 λy. sings(y)
Ssings(e728)
f
f
f =Indicator(S\NP->sings)
=Indicator(S->NPS\NP)
BuildingCCGParsers
ZeolemoyerandCollins(2005)
‣ Trainingdatalookslikepairsofsentencesandlogicalforms
WhatstatesborderTexas λx. state(x) ∧ borders(x, e89)
‣ TexascorrespondstoNP|e89inthelogicalform(easytofigureout)
(S/(S\NP))/N|λf.λg.λx. f(x) ∧ g(x)‣ Whatcorrespondsto
‣ Howdoweinferthatwithoutbeingtoldit?
‣ Problem:wedon’tknowthederiva8on
Lexicon
WhatstatesborderTexas λx. state(x) ∧ borders(x, e89)
‣ Anysubstringcanparsetoanyoftheseinthelexicon
‣ Chunksinferredfromthelogicformbasedonrules:
‣ GENLEX:takessentenceSandlogicalformL.BreakuplogicalformintochunksC(L),assumeanysubstringofSmightmaptoanychunk
‣ Texas->NP:e89iscorrect‣ borderTexas->NP:e89‣ WhatstatesborderTexas->NP:e89… ZeolemoyerandCollins(2005)
‣ NP:e89 ‣ (S\NP)/NP:λx.λy. borders(x,y)
GENLEX
‣ Verycomplexandhand-engineeredwayoftakinglambdacalculusexpressionsand“backsolving”forthederiva8on
ZeolemoyerandCollins(2005)
Learning
ZeolemoyerandCollins(2005)
‣ Itera8veprocedureliketheEMalgorithm:es8mate“best”parsesthatderiveeachlogicalform,retraintheparserusingtheseparseswithsupervisedlearning
‣ We’lltalkaboutasimplerformofthisinafewslides
Applica8ons
‣ GeoQuery:answeringques8onsaboutstates(~80%accuracy)
‣ Jobs:answeringques8onsaboutjobpos8ngs(~80%accuracy)
‣ ATIS:flightsearch
‣ CandowellonallofthesetasksifyouhandcraXsystemsanduseplentyoftrainingdata:thesedomainsaren’tthatrich
‣ WhataboutbroaderQA?
Lambda-DCS
Lambda-DCS
Liangetal.(2011),Liang(2013)
‣ Dependency-basedcomposi8onalseman8cs—originalversionwaslesspowerfulthanlambdacalculus,lambda-DCSisaspowerful
‣ DesignedinthecontextofbuildingaQAsystemfromFreebase
‣ Freebase:setofen88esandrela8ons
AliceSmith
BobCooper
Seaole
March15,1961 WashingtonDateOfBirth PlaceOfBirth
PlaceOfBirth
CapitalOf
‣ [[PlaceOfBirth]]=setofpairsof(person,loca8on)
Lambda-DCS
Liangetal.(2011),Liang(2013)
Lambda-DCS Lambdacalculus
Seattle λx. x = SeattlePlaceOfBirth λx.λy. PlaceOfBirth(x,y)
PlaceOfBirth.Seattle λx. PlaceOfBirth(x,Seattle)
‣ LookslikeatreefragmentoverFreebase
SeaolePlaceOfBirth???
Profession.Scientist ∧ PlaceOfBirth.Seattle
λx. Profession(x,Scientist) ∧ PlaceOfBirth(x,Seattle)
Lambda-DCS
Liangetal.(2011),Liang(2013)
AliceSmith
BobCooper
Seaole
March15,1961 WashingtonDateOfBirth
PlaceOfBirth
PlaceOfBirth
CapitalOf
Profession
Scien8st
Profession.Scientist ∧ PlaceOfBirth.Seattle“listofscien8stsborninSeaole”
‣ ExecutethisfragmentagainstFreebase,returnsAliceSmith(andothers)
???
SeaolePlaceOfBirthProfession
Scien8st
ParsingintoLambda-DCS
Berantetal.(2013)
‣ Buildingthelexicon:moresophis8catedprocessthanGENLEX,butcanhandlethousandsofpredicates
‣ Log-linearmodelwithfeaturesonrules: P (d|x) / expw
>
X
r2d
f(r, x)
!
‣ Deriva8ondonsentencex:
‣ SimilartoCRFparsers
‣ Nomoreexplicitsyntax inthesederiva8onslikewehadinCCG
ParsingwithLambda-DCS
Berantetal.(2013)
‣ Learnjustfromques8on-answerpairs:maximizethelikelihoodoftherightdenota8onywiththederiva8ondmarginalizedout
Foreachexample:Runbeamsearchtogetasetofderiva8ons
Letd*=highest-scoringderiva8oninthebeamwithcorrectdenota,onDoastructuredperceptronupdatetowardsd*awayfromd
Letd=highest-scoringderiva8oninthebeam
sumoverderiva8onsdsuchthatthedenota8onofdonknowledgebaseKisyi
Learning
Berantetal.(2013)
‣Onlyasmallnumberofques8onsareevenreachablebybeamsearchini8ally(butsomeques8onsareveryeasysoevenatotallyuntrainedmodelcananswerthem)‣Duringtraining,moreandmore“good”deriva8onssurfaceandwillresultinmodelupdates
‣ Eachver8calsliceisthebeamforoneexample.Green=correctdenota8on