Building Resources for Human and Computaonal Language ... · o Projeto PorPopular (Finao et al....
37
Building Resources for Human and Computa6onal Language Processing of Portuguese Sílvio Cordeiro, Carlos Ramisch, Marco Idiart, Rodrigo Wilkens, Leonardo Zilio, Jorge Wagner, Aline Villavicencio Federal University of Rio Grande do Sul (Brazil) Aix Marseille Université, CNRS, LIF UMR 7279 (France) University of Essex (UK)
Transcript of Building Resources for Human and Computaonal Language ... · o Projeto PorPopular (Finao et al....
BuildingResourcesforHumanandComputa6onalLanguageProcessing
ofPortugueseSílvioCordeiro,CarlosRamisch,MarcoIdiart,RodrigoWilkens,
LeonardoZilio,JorgeWagner,AlineVillavicencioFederalUniversityofRioGrandedoSul(Brazil)
AixMarseilleUniversité,CNRS,LIFUMR7279(France)UniversityofEssex(UK)
MWE-awareprocessingwiththemwetoolkit
Mul6wordExpressionsinaNutshell• Acombina6onofwordsthatmustbetreatedasaunitatsomelevel
oflinguis6cprocessing(Calzolarietal.,2002)o CompoundNounso Verb-par6cleconstruc6onso Light-verbconstruc6onso Idioms
Mul6wordExpressionsinaNutshell• Lexical,syntac6c,seman6c,pragma6c,sta6s6cal
idiosyncrasieso Adhoc,wineanddine(KimandBaldwin2010)
• ArbitrarinessandIns6tu6onalisa6ono saltandpepper,?pepperandsalt(Smadja,1993)
• Frequencyo Sameorderofmagnitudeaswordsinmentallexicon(Jackendoff,1997)
• Limitedlexical,syntac6candseman6cvariabilityo kickthebucket/?pail/?container(Sagetal.,2002)
MWEsandNLP• RealunderstandingrequiresMWE-awarecorpusprocessing
1. Corpusprocessing2. MWEdiscoveryfromcorpus(tobuildMWElexica)3. MWErepresenta6on(inlexiconandgrammar)4. MWEtokeniden6fica6onincorpus(toannotateMWEs)5. MWEseman6cprocessing6. MWEintegra6oninapplica6ons
mwetoolkit• LanguageindependentframeworkforMWEprocessing• ExtractsMWEfromcorpora• AnnotatescorporawithMWEs• CalculatesAMs• Pre-processesMWEsincorporaforDSMconstruc6on• ImportsDSMs(word2vec,glove,PPMI)• Providesfunc6onsforvectorcombina6ons• Calculatescomposi6onality• Evaluatesagainstgoldstandard
Project CAPES-COFECUB (France-Brazil)
Overviewofthemwetoolkit
MWETypeDiscovery• Candidateextrac6on
o Pafern-basedheuris6cs(e.g.noun-noun,verb-par6cle…)
MWE-awarecorpusprocessing• MWEtokeniden6fica6on
o Pafern-basedheuris6cso Con6guous/gappyiden6fica6ono Shortest/longest/allmatchdistanceso Projec6ngextractedMWEtypesbackinsourcecorpus
Annota6onOp6ons• CorpusbutnoMWEList:
o GenerateMWElistfromcorpusandprojec6ngthemback• CorpusàMWElistàAnnotatedCorpus
• CorpusandMWEListo Annota6onbasedonexternallistsofMWEs
• Corpus+MWElistàAnnotatedCorpus
MWEseman6cprocessing• MeaningofMWEmaynotbeunderstoodfrommeaningof
individualwordso brickwallisawallmadeofbricks,o cheeseknifeisnotaknifemadeofcheeseàknifeforcu@ngcheese(Girjuetal.,2005).o Loansharkisnotasharkforloanbutapersonwhooffersloansatextremelyhighinterestrates
Howtodetectcomposi6onality?
• Distribu6onalSeman6cModels(DSMs)o Posi6onwordsinmul6dimensionalseman6cspace
• Eachword/MWErepresentedasavectorintheseman6cspaceo Proximityinspaceindicatesseman6crelatedness
Cloud nine
Access road
Compositionality Idiomaticity
Grandfather clock
Howtodetectcomposi6onality?• CosinesimilaritybetweentheMWEvectorandthesumofthe
vectorsofthecomponentwordso Thecloservectorsarethemorecomposi6onaltheyare(Reddyetal.2011)o cos(w1w2vector,w1vector+w2vector)
Distribu6onalSeman6cModels• Techniquesandtoolsforconstruc6ngDSMs
o Dissect(Dinuetal.,2013),Miniman6cs(Ramischetal.2013),word2vec(Mikolovetal.,2013)andGlove4(Penningtonetal.,2014).
Minimantics
word2vec
dissect
LexVec (Lexical Vectors)
GoldStandardsforEvalua6on• Rolleretal.(2013)244Germancompounds
o around30judgmentsbycrowdsourcingscalefrom1to7
• Farahmandetal.(2015)1,042Englishcompoundso 4expertsjudgesbinaryscalefornon-composi6onalityandconven6onality
• Reddyetal.(2011)90Englishcompoundso around30judgmentsbycrowdsourcingscalefrom0to5
DSMsandComposi6onality• Datasetofnominalcompoundswithhumanjudgmentsabout
literality/composi6onalityo 180compoundsforEnglish,FrenchandPortugueseo Resourcefreelyavailable
• hfp://pageperso.lif.univ-mrs.fr/~carlos.ramisch/?page=downloads/compounds&lang=en
DSMsandComposi6onality• DatasetofLexicalSubs6tu6onofNominalCompoundsin
Portuguese(LexSubNC)o 180compoundsforPortugueseo Resourcefreelyavailable
• hfp://pageperso.lif.univ-mrs.fr/~carlos.ramisch/?page=downloads/compounds&lang=en
Collec6ngHumanJudgments• Judgmentswithlikertscale(0to5)
o Forcompoundo Forw1andw2separately
• AgreementforPortugueseo Forsubsetofannotators
• α=.52forhead,• α=.36formodifier• α=.42forcompound
o Sameannotatoraqer1month:0.59forcompound
Collec6ngHumanJudgments-Agreement
• Greateragreementbetweenscoreforcompoundandhead(ormodifier)forextremeso totallyidioma6candfullycomposi6onal
• ForPTandFRcompoundscoredeterminedbyscoreoftheleastliteralword
Agreement• Most/leastvaria6oninscores(average±σscore)
Themodels• WaCkyCorpora(Baronietal.,2009):
o ukWaCforEnglish(∼2billiontokens)o frWaC(∼1.6billiontokens)forFrencho brWaC(∼2.3billiontokens)forPortuguese(WagnerFilhoetal.2016)o Pre-processing
• surface+:theoriginalcorpus• surface:withstopwordremoval.• lemma:stopwordremovalandlemma6za6on;• lemmaPOS:stopwordremoval,lemma6za6onandPOS-tagging
o ContextWindowsize:1,4and8o Dimensionsize:250,500,750
• DSMso PPMImodels–posi6vePMI(Miniman6cs)o GloVe(Penningtonetal.2014)o Word2vec(Mikolovetal2013)Skipgram,CBOWo LexVec
French
Portuguese
English
ResourcesforTextSimplifica6onforPortuguese
RodrigoWilkens,LeonardoZilio,MarcoIdiart,JorgeWagnerFilho,EduardoFerreira,LuisMollmann,BiancaPasqualini,AlineVillavicencio
FederalUniversityofRioGrandedoSul(Brazil)
TextSimplifica6on(TS)• TSqualitydependentonresources
o English:• Corpora:
o SimpleEnglishWikipediaparallelcorpus,withalignmentsbetweentheSimpleandtheStandardEnglishWikipedia
o PennTreebank,Bri6shNa6onalCorpus,ukWaC• ResourcesforLexicalSubs6tu6ons:WordNet,Roget,Moby• GoldStandardSubs6tu6onLists:SemEvalLexicalSubs6tu6onTask
o Portuguese:• Corpora
o PorSimples(Aluísioetal)• Thesauri
o WordNet.Pt,OntoPT,
TextSimplifica6on• Twomaintasks(Shardlow,2014):
o lexicalsimplifica6on(LS),• replacingcomplexexpressionswithsimplersynonyms,
o syntac6csimplifica6on(SS)• changethestructureofasentencebyusingsimplersyntac6cconstruc6ons
(Siddharthan,2002).o TSwithMTtechniquesformonolingualtransla6on
• learningalignmentsbetweensimpleandstandardsentences
GeneralCorpora• WaCky(Baronietal.2009)
o ukWaC(Baronietal.2009)o brWaC(Boosetal.2014,WagnerFilhoetal.2018)
• Crawling,frommediumfrequencycontentwordsasseedso LinguatecaCorporaFrequencyList
• Cleaningo HTMLandboilerplatestripping,usingdensitymetricsand
shallowtextfeatures• Near-duplicatedetec6onandremoval
o pairwisecomparisonofalldocuments
SimpleCorpora• ForEnglish,
o SimpleEnglishWikipediaalignedwithEnglishWikipedia(CosterandKauchak,2011)
• ForPortugueseo ColeçãoÉSóoComeço(Wilkensetal.2014)
• 5booksmanuallysimplifiedbylinguists.o Caselietal.2009
• manuallyannotatedcorpusofsyntac6candlexicalsimplifica6onso WikiJunior
• illustratedbooksforchildrenupto12yearsold.o ProjetoPorPopular(Finafoetal.2012)
• Tabloidsforlowliteracyreaders
SimpleCorpora• WikilivrosReadabilityCorpus(WRC)
o BooklibraryfromWikilivros• L1:33booksfrom1stto9thgrades• L2:65booksfrom10thto12thgrades• L3:21booksforcollegeeduca6on
• ReadabilityAssessedWaC(RAW)o readabilityassessmentmodule(WagnerFilhoetal.2016)
• intermediatemoduleofreadabilityassessment• severalreadabilityfeaturesusedasfeaturesforclassifier
o 129,000sentencesfromL1,• 13.5wordspersentence
o 236,000sentencesfromL2• 15.2wordspersentence
o 96,000sentencesfromL3,• 17.4wordspersentence
Mul6wordExpressions(MWEs)
• ForEnglish• NomLex,WordNet• Verb-Par6cleConstruc6ons(Baldwin2005)• CompoundNouns(Nakov2010,Reddyetal2011,Yazdanietal2015,Ramischet
al2016)
• ForPortuguese• LightVerbConstruc6ons
o Duranetal.2011• CompoundNouns
o NounPreposi6onNounsfromEuroparl(Zilioetal.2016)• Parsing-based(FIPS)combinedwithSta6s6cal(PMI)
o NounAdjec6ve(Cordeiroetal.2016)
SimpleWordsLists• Manuallycreatedlists
o ForEnglish• Oxford3000
o ForPortuguese• 3,853wordsfromOxford3000transla6oncomplementedwithmostfrequent
wordsincorpora(Finafoetal.2013)
LexicalSubs6tu6on• Manualresources
o ForEnglish• WordNet(Fellbaum1998),Roget,Moby
o ForPortuguese• Onto.PT(Oliveiraetal.2010),• OpenWN-PT(Paivaetal.2012),• Mul6Wordnet(Brancoetal.),• WordNet.PT(Marrafa,2002),• WordNet.Br(DiasdaSilvaetal.,2008)
LexicalSubs6tu6on• Distribu6onalSeman6cModels
o GloVe,word2vec,Miniman6cs,Dissect,LexVec
• QualityEvalua6ono ForEnglish
• WordNet-BasedSynonymyTest(WBST)(Freitagetal.2005)• Wordsimilarityandanalogytasks
o ForPortuguese• BabelNet-BasedSeman6cGoldStandard(B2SG)(Wilkensetal.2016)
o synonymy,antonymyandhypernymyfornounsandverbs
Seman6cRoleLabeling(SRL)• Forwordsubs6tu6onincontext
o ForEnglish• FrameNet(Bakeretal.1998),PropBank(Kingsburyetal.2002)
o ForPortuguese• PropBank.Br(DuranandAluísio2012),VerbNet.Br(Scarton2013),andFrameNet
Brasil(Salomao,2009).• VerbLexPor(Zilioetal.2016)
o Cardiologypapersvs.Newspaperar6cles• 15,281annotatedarguments(4,192inCARDand11,089inDG)
ResourcesinNumbers
ConclusionandFutureWork• EnglishandPortuguese
o Differenceinmagnitudeandavailabilityofmanuallyconstructedresources
• Alterna6ve:languageindependentmethodso Extrapolatefrommanuallycreatedresourceso Corpora
• brWaC(WagnerFilhoetal.2016)• ReadabilityAssessedWac(WagnerFilhoetal.2016,WagnerFilhoetal.2018)
o Distribu6onalSeman6cModels• LexVec(Salleetal.2016)
o Goldstandards• NCComposi6onalityDataset(Cordeiroetal.2016)• NCLexSub(Cordeiroetal.2017)• BabelNet-BasedSeman6cGoldStandard(B2SG)(Wilkensetal.2016)
Acknowledgments• Thisworkhasbeenfundedbythe
• BrazilianAgencyCNPq(482520/2012-4and312114/2015-0)• Projects“SimplificaçãoTextualdeExpressõesComplexas”,sponsoredbySamsung
EletrônicadaAmazôniaLtda.underthetermsofBrazilianfederallawNo.8.248/91,and• FrenchAgenceNa6onalepourlaRecherchethroughprojectsPARSEME-FR(ANR-14-
CERA-0001)andORFEO(ANR-12-CORP-0005),andby• French-Braziliancoopera6onprojectsCAMELEON(CAPES-COFECUB707/11)andAIM-
WEST(FAPERGS-INRIA1706-2551/13-7).
BuildingResourcesforHumanandComputa6onalLanguageProcessing
ofPortugueseSílvioCordeiro,CarlosRamisch,MarcoIdiart,RodrigoWilkens,
LeonardoZilio,JorgeWagner,AlineVillavicencioFederalUniversityofRioGrandedoSul(Brazil)
AixMarseilleUniversité,CNRS,LIFUMR7279(France)UniversityofEssex(UK)
