Andnowforsomethingcompletelydifferent

AlgorithmsforNLP(11-711)Fall2017

FormalLanguageTheoryInonelecture

RobertFrederking

NowforSomethingCompletelyDifferent

• Wewilllookatgrammarsfroma“mathematical”pointofview

• ButDiscreteMath(logic)– Norealnumbers– Symbolicdiscretestructures,proofs

• Thisisthesourceofmanycommonalgorithms/models

• Interestedincomplexity/powerofdifferentformalmodelsofcomputation– Relatedtoasymptoticcomplexitytheory

Twomainclassesofmodels

• Automata–Machines,likeFinite-StateAutomata

• Grammars– Rulesets,likewehavebeenusingtoparse

• Wewilllookateachclassofmodel,goingfromsimplertomorecomplex/powerful

• Wecanformallyprovecomplexity-classrelationsbetweentheseformalmodels

Simplestlevel:FSA/Regularsets

Finite-StateAutomata(FSAs)

• Simplestformalautomata• We’veseenthesewithnumbersonthemasHMMs,etc.

(fromWikipedia)

Formaldefinitionofautomata

• Afinitesetofstates,Q• Afinitealphabetofinputsymbols,Σ• Aninitial(start)state,Q0∈Q• Asetoffinalstates,Fi∈Q• Atransitionfunction,δ:QxΣ →Q

• ThisrigorouslydefinestheFSAsweusuallyjustdrawascirclesandarrows

RegularGrammars

• Left-linearorright-lineargrammars• Left-lineartemplate:

A→ Bx orA→x• Right-lineartemplate:

A→ xB orA→x

• Example:S→aA |bB |ε ,A→aS ,B→bbS

FormalDefinitionofaGrammar

• Vocabularyofterminalsymbols,Σ (e.g.,a)• Setofnonterminalsymbols,N(e.g.,A)• Specialstartsymbol,S∈ N• Productionrules,suchasA→aB• Restrictionsontherulesdeterminewhatkindofgrammaryouhave

• AformalgrammarGdefinesaformallanguage,L(G),thesetofstringsitgenerates

RegularExpressions

• Forregulargrammars,there’sasimplerwaytowriteexpressions:regularexpressions:

Terminalsymbols(r+s)(r•s)r*ε

• Forexample:(aa+bbb)*

Amazingfact#1:FSAsareequivalenttoRGs

• Proof:twoconstructiveproofs:– 1:givenanarbitraryFSA,constructthecorrespondingRegularGrammar(andprovethatitwillonlyproducethestringstheFSAwould)

– 2:givenanarbitraryRegularGrammar,constructthecorrespondingFSA(andprovethatitwillonlyproducethestringsthegrammarwould)

DFSAs,NDFSAs

• DeterministicorNon-deterministic– Isδ functionambiguousornot?

– ForFSAs,weaklyequivalent

Intersecting,etc.,FSAs

• WecaninvestigatewhathappensafterperformingdifferentoperationsonFSAs:– Union– Intersection– Concatenation– Negation– otheroperations:determinizing andminimizingFSAs

Provingalanguageisnot regular

• So,whatkindsoflanguagesarenot regular?

• Informally,aFSAcanonlyremember afinitenumberofspecific things.Soalanguagerequiringanunboundedmemorywon’tberegular.

Provingalanguageisnot regular

• So,whatkindsoflanguagesarenot regular?

• Informally,aFSAcanonlyremember afinitenumberofspecific things.Soalanguagerequiringanunboundedmemorywon’tberegular.

• Howaboutanbn? “equalcountofa’s and b’s”

PumpingLemma:argument:

• ConsideramachinewithNstates• NowconsideraninputoflengthN;sincewestartedinQ0,wewillnowbeinthe(N+1)ststatevisited

• Theremust bealoop:wehadtovisitatleast1statetwice;letxbethestringuptotheloop,ythepartintheloop,andzaftertheloop

• SoitmustbeokaytoalsohaveMcopiesofyforanyM(including0copies)

PumpingLemma:formally:

• IfLisaninfiniteregularlanguage,thentherearestringsx,y,andzsuchthaty≠ε andxynz∈ L,foralln≥0.

• xyzbeinginthelanguagerequiresalso:• xz,xyyz,xyyyz,xyyyyz,…, xyyyyyyyyyyz,…

PumpingLemma:figure:

q0 qNqx z

y

ExampleproofthataLisnotregular

• Whataboutanbn?abaabbaaabbbaaaabbbbaaaaabbbbb…

• Wheredoyoudrawthexynz lines?

ExampleproofthataLisnotregular

• Whataboutanbn?Wheredoyoudrawthelines?• Threecases:– y isonlya’s:thenxynz willhavetoomanya’s– y isonlyb’s:thenxynz willhavetoomanyb’s– y isamix:thentherewillbeintersperseda’sandb’s

• Soanbn cannotberegular,sinceitcannotbepumped

Nextlevel:PDA/CFG

Push-DownAutomata(PDAs)

• Let’saddsomeunboundedmemory,butinalimitedfashion

• So,addastack:

• Allowsyoutohandlesomenon-regularlanguages,butnoteverything

Context-FreeGrammars

• Ruletemplate:A→γ whereγ isanysequenceofterminals/non-terminals

• Example:S→aSb|ε• WeusethesealotinNLP– Expressiveenough,nottoocomplextoparse.• Weoftenaddhackstoallownon-CFinformationflow.

– Itjustreallyfeelsliketherightlevelofanalysis.• (Moreonthislater.)

AmazingFact#2:PDAsandCFGsareequivalent

• SamekindofproofasforFSAsandRGs,butmorecomplicated

• Aretherenon-CFlanguages?Howaboutanbncn?

Highestlevel:TMs/Unrestrictedgrammars

TuringMachines

• Justletthemachinemoveandwriteonthetape:

• Thissimplechangeproducesgeneral-purposecomputer:Church-TuringHypothesis

TMmadeofLEGOs

UnrestrictedGrammars

• α→β,whereeachcanbeanysequence(αnotempty)

• Thus,thereiscontext intherules:aAb →aabbAb →bbb

• Nosurpriseatthispoint:equivalenttoTMs

Evenmoreamazingfact:Chomskyhierarchy

• Provablethateachofthesefourclassesisapropersubsetofthenextone:

Type0:TMType1:CSGType2:CFGType3:RE

01

* 2 3

Linear-BoundedAutomata/Context-SensitiveGrammars

• TMthatusesspacelinearintheinput• αAβ→αγβ(γ notempty)

• Wemostlyignorethese;theygetnorespect• Correspondtoeachother• Limitedcomparedtofull-blownTM– Butcomplexitycanalreadybeundecidable

ChomskyHierarchy:proofs

• Formofhierarchyproofs:– Foreachclass,youcanprovetherearelanguagesnotintheclass,similartoPumpingLemmaproof

– Youcaneasilyprovethatthelargerclassreallydoescontainalltheonesinthesmallerclass

Intersecting,etc.,Ls

• WecanagaininvestigatewhathappenswithLsinthesevariousclassesunderdifferentoperationsonLs:– Union– Intersection– Concatenation– Negation– otheroperations

Chomskyhierarchy:table

MildlyContext-SensitiveGrammars

• WereallylikeCFGs,butaretheyinfactexpressiveenoughtocaptureallhumangrammar?

• Manyapproachesstartwitha“CFbackbone”,andaddregisters,equations,etc.,thatarenot CF.

• Severalnon-hackextensions(CCG,TAG,etc.)turnouttobeweaklyequivalent!– “Mildlycontextsensitive”– SoCSFsgetevenlessrespect…– AndsomuchfortheChomskyHierarchybeingsuchabigdeal

Tryingtoprovehumanlanguagesarenot CF

• Certainlytrueofsemantics.ButNLsyntax?• Cross-serialdependenciesseemlikeagoodtarget:–Mary,Jane,andJimlikered,green,andblue,respectively.

– Butisthissyntactic?• Surprisinglyhardtoprove

SwissGermandialect!dative-NPaccusative-NPdative-taking-VPaccusative-taking-VP

•JansaitdasmeremHanseshuushalfedaastriiche•JansaysthatweHansthehousehelpedpaint•“JansaysthatwehelpedHanspaintthehouse”•Jansaitdasmerd’chindemHanseshuushaendwelelaahalfeaastriiche•JansaysthatwethechildrenHansthehousehavewantedtolethelppaint•“JansaysthatwehavewantedtoletthechildrenhelpHanspaintthehouse”

(Alittlelike“Thecatthedogthemousescaredchasedlikestunafish”)

IsSwissGermanContext-Free?

Shieber’scomplexargument…

L1=Jansaitdasmer(d’chind)*(emHans)*eshuushaendwele(laa)*(halfe)*aastriiche

L2=SwissGerman

L1∩L2=Jansaitdasmer(d’chind)n(emHans)meshuushaendwele(laa)n(halfe)maastriiche

Whydowecare?(1)

• Mathisfun?• Complexity:– IfyoucanuseaRE,don’tuseaCFG.– BecarefulwithanythingfancierthanaCFG.

• Safety:hardertowritecorrectsystemsonaTuringMachine.

• Beingabletouseaweakerformalismmayhaveexplanatorypower?

Whydowecare?(2)

• Probablyasourceforfuturenewalgorithms• Probablynot howhumansactuallyprocessNL• MightnotmatterasmuchforNLPnowthatweknowaboutrealnumbers?– Butwedon’twantyourfriendsmakingfunofyou

MoreExamples

•Thecatlikestunafish•Thecatthedogchasedlikestunafish•Thecatthedogthemousescaredchasedlikestunafish•Thecatthedogthemousetheelephantsquashedscaredchasedlikestunafish•Thecatthedogthemousetheelephantthefleabitsquashedscaredchasedlikestunafish•Thecatthedogthemousetheelephantthefleathevirusinfectedbitsquashedscaredchasedlikestunafish