Multilinguality to the Rescue

Manaal Faruqui & Chris DyerLanguage Technologies Institute

SCS, CMU

Multilinguality

Using more than one language at a time

Image source: https://buffy.eecs.berkeley.edu/PHP/resabs/images/2006//101268-1.png

Multilinguality

Why ?

Bank

Images: http://www.realestategolfodulce.com/ , http://thetrustadvisor.com/

बैंक

तट

Cross lingual Word Sense Disambiguation(Diab and Resnik, 2002)

Multilinguality

Why ?

Bilingual Word Clustering (Faruqui & Dyer, 2013)

Multilinguality

Why ?

Bilingual Word Clustering (Faruqui & Dyer, 2013)

Multilinguality

Using data from other languages

Direct Indirect

Assume foreign = original language

Extract information from foreign language

Direct Information Transfer

NLP System

Language 1 data

Language 2 data

Output

Direct Information Transfer

Why would it work ?

• Works for specific tasks like NER• Many NEs retain their “orthographic” form• Across languages that use the same “alphabet”

• English, German, French, Spanish• Hindi, Marathi, Bihari

• Specially proper nouns• Names of Locations

• USA, London, New York, Pittsburgh• Names of People

• Obama, William, Roger

Barack Obama hat 2012 mit dieser Strategie die Präsidentschaftswahlen gewonnen.

The Obama administration has poured billions of dollars into expanding the reach of the Internet.

Pour finir, en défendant les bonus et en tentant de faire dérailler les nouvelles règles prudentielles, ce démocrate s'est mis à dos Barack Obama.

Direct Information Transfer

... sagte Jimmy Wales dem Wall Street Journal in einem Interview in Hongkong.

Mads Refslund, executive chef at Acme, forages in the overgrown spaces and hidden markets of Hongkong for regional delicacies.

Les sacs de luxe, nouvelle monnaie d'échange à Hongkong.

Direct Information Transfer

Semantic Generalization

Deutschland (100)Ostdeutschland (5)

Westdeutschland (0)LOC

Direct Information Transfer

How?

NER System

Language 1Training data Language 2

Word Clusters

NE-tagged Text

Input

EvaluationTools

• Stanford NER for training (Finkel and Manning, 2009)• In-built functionality to use word clusters for

generalization

• Word clustering software (distributional + morphological) (Clark., 2003)

Data

• NER training data• German, English: CoNLL 2003• Dutch, Spanish: CoNLL 2002

• Generalization data• WMT-2012 news commentary: 200 million tokens• English, German, French, Spanish, Czech

Results

Results

Results

Improvement in F1 scores by NE type

Quick Takeaways

• Multilingual data can be put to use for monolingual benefits

• The amount of help depends on how similar the two languages are “orthographically”

Indirect Information Transfer

NLP System

Language 1 data

Language 2 data

Output

+

Vector Space Word Models

Image: http://www.emeraldinsight.com

Vector Space Models

Image: http://d1avok0lzls2w.cloudfront.net/

Vector Space Models

Monolingual Word Vectors 1

Monolingual Word Vectors 2

+

Better Monolingual Word Vectors 1 ??

Indirect Information Transfer

+ = Canonical Correlation Analysis

n n

k

d2

nn

d1

k

+

wxd1

k

wy d2

k

x yn n

d2d1

Canonical Correlation Analysis

* *

nn

k k

Indirect Information Transfer

Word Vectors in Language 1

Word Vectors in Language 2

Obtain 1-to-1 mapping using word alignments

Word Vectors in Language 1

Word Vectors in Language 2

+

Word Vectors in Language 1

Word Vectors in Language 2

Experiments

Task: Word Pair Reranking• Rank a list of word pairs according to semantic similarity

Datasets• WS-353: 353 word pairs• RG-65: 65 noun pairs

Truncation• Maybe the correlation introduces noise • Keep only the top k% of correlated dimensions

EvaluationTools

• Word vectors: RNNLM Toolkit (Mikolov, 2009)• Word alignments: cdec (Dyer et al, 2013)• CCA: Matlab Toolkit

Data

• Word vector monolingual training data• WMT news commentary: 2011, 2012• English, French, Spanish, German

• Word alignment data• WMT news commentary 2010, 09, 08. 07, 06• {French, Spanish, German} - English

Results

Results

Original English Vectors

German Projected on English

Conclusion

• Word vector quality can be improved using multilingual data• At least for lexical semantic tasks

• The amount of help provided by these languages depend on how similar they are to each other

• A task like NER can use data from multiple languages in a simple framework

Thank You!