Linguistic Regularities in Sparse and Explicit Word Representations
description
Transcript of Linguistic Regularities in Sparse and Explicit Word Representations
![Page 1: Linguistic Regularities in Sparse and Explicit Word Representations](https://reader031.fdocuments.in/reader031/viewer/2022012914/568163c9550346895dd50224/html5/thumbnails/1.jpg)
Linguistic Regularities in Sparse and Explicit
Word RepresentationsOmer Levy Yoav Goldberg
Bar-Ilan UniversityIsrael
![Page 2: Linguistic Regularities in Sparse and Explicit Word Representations](https://reader031.fdocuments.in/reader031/viewer/2022012914/568163c9550346895dd50224/html5/thumbnails/2.jpg)
Papers in ACL 2014*
Neural Networks & Word Embed-
dings
Other Topics
* Sampling error: +/- 100%
![Page 3: Linguistic Regularities in Sparse and Explicit Word Representations](https://reader031.fdocuments.in/reader031/viewer/2022012914/568163c9550346895dd50224/html5/thumbnails/3.jpg)
Neural Embeddings• Dense vectors• Each dimension is a latent feature• Common software package: word2vec
• “Magic”king man woman queen
(analogies)
![Page 4: Linguistic Regularities in Sparse and Explicit Word Representations](https://reader031.fdocuments.in/reader031/viewer/2022012914/568163c9550346895dd50224/html5/thumbnails/4.jpg)
Representing words as vectors is not new!
![Page 5: Linguistic Regularities in Sparse and Explicit Word Representations](https://reader031.fdocuments.in/reader031/viewer/2022012914/568163c9550346895dd50224/html5/thumbnails/5.jpg)
Explicit Representations (Distributional)• Sparse vectors• Each dimension is an explicit context• Common association metric: PMI, PPMI
• Does the same “magic” work for explicit representations too?• Baroni et al. (2014) showed that embeddings outperform explicit, but…
![Page 6: Linguistic Regularities in Sparse and Explicit Word Representations](https://reader031.fdocuments.in/reader031/viewer/2022012914/568163c9550346895dd50224/html5/thumbnails/6.jpg)
Questions• Are analogies unique to neural embeddings?Compare neural embeddings with explicit representations
• Why does vector arithmetic reveal analogies?Unravel the mystery behind neural embeddings and their “magic”
![Page 7: Linguistic Regularities in Sparse and Explicit Word Representations](https://reader031.fdocuments.in/reader031/viewer/2022012914/568163c9550346895dd50224/html5/thumbnails/7.jpg)
Background
![Page 8: Linguistic Regularities in Sparse and Explicit Word Representations](https://reader031.fdocuments.in/reader031/viewer/2022012914/568163c9550346895dd50224/html5/thumbnails/8.jpg)
Mikolov et al. (2013a,b,c)• Neural embeddings have interesting geometries
![Page 9: Linguistic Regularities in Sparse and Explicit Word Representations](https://reader031.fdocuments.in/reader031/viewer/2022012914/568163c9550346895dd50224/html5/thumbnails/9.jpg)
![Page 10: Linguistic Regularities in Sparse and Explicit Word Representations](https://reader031.fdocuments.in/reader031/viewer/2022012914/568163c9550346895dd50224/html5/thumbnails/10.jpg)
Mikolov et al. (2013a,b,c)• Neural embeddings have interesting geometries
• These patterns capture “relational similarities”
• Can be used to solve analogies:man is to woman as king is to queen
![Page 11: Linguistic Regularities in Sparse and Explicit Word Representations](https://reader031.fdocuments.in/reader031/viewer/2022012914/568163c9550346895dd50224/html5/thumbnails/11.jpg)
Mikolov et al. (2013a,b,c)• Neural embeddings have interesting geometries
• These patterns capture “relational similarities”
• Can be used to solve analogies: is to as is to
• Can be recovered by “simple” vector arithmetic:
![Page 12: Linguistic Regularities in Sparse and Explicit Word Representations](https://reader031.fdocuments.in/reader031/viewer/2022012914/568163c9550346895dd50224/html5/thumbnails/12.jpg)
Mikolov et al. (2013a,b,c)• Neural embeddings have interesting geometries
• These patterns capture “relational similarities”
• Can be used to solve analogies: is to as is to
• With simple vector arithmetic:
![Page 13: Linguistic Regularities in Sparse and Explicit Word Representations](https://reader031.fdocuments.in/reader031/viewer/2022012914/568163c9550346895dd50224/html5/thumbnails/13.jpg)
𝑎−𝑎∗=𝑏−𝑏∗
Mikolov et al. (2013a,b,c)
![Page 14: Linguistic Regularities in Sparse and Explicit Word Representations](https://reader031.fdocuments.in/reader031/viewer/2022012914/568163c9550346895dd50224/html5/thumbnails/14.jpg)
𝑏−𝑎+𝑎∗=𝑏∗
Mikolov et al. (2013a,b,c)
![Page 15: Linguistic Regularities in Sparse and Explicit Word Representations](https://reader031.fdocuments.in/reader031/viewer/2022012914/568163c9550346895dd50224/html5/thumbnails/15.jpg)
king man woman queen
Mikolov et al. (2013a,b,c)
𝑏𝑎𝑎∗𝑏∗
![Page 16: Linguistic Regularities in Sparse and Explicit Word Representations](https://reader031.fdocuments.in/reader031/viewer/2022012914/568163c9550346895dd50224/html5/thumbnails/16.jpg)
Tokyo Japan France Paris
Mikolov et al. (2013a,b,c)
𝑏𝑎𝑎∗𝑏∗
![Page 17: Linguistic Regularities in Sparse and Explicit Word Representations](https://reader031.fdocuments.in/reader031/viewer/2022012914/568163c9550346895dd50224/html5/thumbnails/17.jpg)
best good strong strongest
Mikolov et al. (2013a,b,c)
𝑏𝑎𝑎∗𝑏∗
![Page 18: Linguistic Regularities in Sparse and Explicit Word Representations](https://reader031.fdocuments.in/reader031/viewer/2022012914/568163c9550346895dd50224/html5/thumbnails/18.jpg)
best good strong strongest
Mikolov et al. (2013a,b,c)
vectors in
𝑏𝑎𝑎∗𝑏∗
![Page 19: Linguistic Regularities in Sparse and Explicit Word Representations](https://reader031.fdocuments.in/reader031/viewer/2022012914/568163c9550346895dd50224/html5/thumbnails/19.jpg)
Are analogies unique to neural embeddings?
![Page 20: Linguistic Regularities in Sparse and Explicit Word Representations](https://reader031.fdocuments.in/reader031/viewer/2022012914/568163c9550346895dd50224/html5/thumbnails/20.jpg)
• Experiment: compare embeddings to explicit representations
Are analogies unique to neural embeddings?
![Page 21: Linguistic Regularities in Sparse and Explicit Word Representations](https://reader031.fdocuments.in/reader031/viewer/2022012914/568163c9550346895dd50224/html5/thumbnails/21.jpg)
Are analogies unique to neural embeddings?• Experiment: compare embeddings to explicit representations
![Page 22: Linguistic Regularities in Sparse and Explicit Word Representations](https://reader031.fdocuments.in/reader031/viewer/2022012914/568163c9550346895dd50224/html5/thumbnails/22.jpg)
Are analogies unique to neural embeddings?• Experiment: compare embeddings to explicit representations
• Learn different representations from the same corpus:
![Page 23: Linguistic Regularities in Sparse and Explicit Word Representations](https://reader031.fdocuments.in/reader031/viewer/2022012914/568163c9550346895dd50224/html5/thumbnails/23.jpg)
Are analogies unique to neural embeddings?• Experiment: compare embeddings to explicit representations
• Learn different representations from the same corpus:
• Evaluate with the same recovery method:
![Page 24: Linguistic Regularities in Sparse and Explicit Word Representations](https://reader031.fdocuments.in/reader031/viewer/2022012914/568163c9550346895dd50224/html5/thumbnails/24.jpg)
Analogy Datasets• 4 words per analogy: is to as is to
• Given 3 words: is to as is to
• Guess the best suiting from the entire vocabulary • Excluding the question words
• MSR: 8000 syntactic analogies• Google: 19,000 syntactic and semantic analogies
![Page 25: Linguistic Regularities in Sparse and Explicit Word Representations](https://reader031.fdocuments.in/reader031/viewer/2022012914/568163c9550346895dd50224/html5/thumbnails/25.jpg)
Embedding vs Explicit (Round 1)
![Page 26: Linguistic Regularities in Sparse and Explicit Word Representations](https://reader031.fdocuments.in/reader031/viewer/2022012914/568163c9550346895dd50224/html5/thumbnails/26.jpg)
Embedding vs Explicit (Round 1)
MSR Google0%
10%
20%
30%
40%
50%
60%
70%
Embedding54%
Embedding63%
Explicit29%
Explicit45%
Accu
racy
Many analogies recovered by explicit, but many more by embedding.
![Page 27: Linguistic Regularities in Sparse and Explicit Word Representations](https://reader031.fdocuments.in/reader031/viewer/2022012914/568163c9550346895dd50224/html5/thumbnails/27.jpg)
Why does vector arithmetic reveal analogies?
![Page 28: Linguistic Regularities in Sparse and Explicit Word Representations](https://reader031.fdocuments.in/reader031/viewer/2022012914/568163c9550346895dd50224/html5/thumbnails/28.jpg)
Why does vector arithmetic reveal analogies?• We wish to find the closest to • This is done with cosine similarity:
Problem: one similarity might dominate the rest.
![Page 29: Linguistic Regularities in Sparse and Explicit Word Representations](https://reader031.fdocuments.in/reader031/viewer/2022012914/568163c9550346895dd50224/html5/thumbnails/29.jpg)
Why does vector arithmetic reveal analogies?• We wish to find the closest to
![Page 30: Linguistic Regularities in Sparse and Explicit Word Representations](https://reader031.fdocuments.in/reader031/viewer/2022012914/568163c9550346895dd50224/html5/thumbnails/30.jpg)
Why does vector arithmetic reveal analogies?• We wish to find the closest to • This is done with cosine similarity:
![Page 31: Linguistic Regularities in Sparse and Explicit Word Representations](https://reader031.fdocuments.in/reader031/viewer/2022012914/568163c9550346895dd50224/html5/thumbnails/31.jpg)
Why does vector arithmetic reveal analogies?• We wish to find the closest to • This is done with cosine similarity:
![Page 32: Linguistic Regularities in Sparse and Explicit Word Representations](https://reader031.fdocuments.in/reader031/viewer/2022012914/568163c9550346895dd50224/html5/thumbnails/32.jpg)
Why does vector arithmetic reveal analogies?• We wish to find the closest to • This is done with cosine similarity:
vector arithmetic similarity arithmetic
![Page 33: Linguistic Regularities in Sparse and Explicit Word Representations](https://reader031.fdocuments.in/reader031/viewer/2022012914/568163c9550346895dd50224/html5/thumbnails/33.jpg)
Why does vector arithmetic reveal analogies?• We wish to find the closest to • This is done with cosine similarity:
vector arithmetic similarity arithmetic
![Page 34: Linguistic Regularities in Sparse and Explicit Word Representations](https://reader031.fdocuments.in/reader031/viewer/2022012914/568163c9550346895dd50224/html5/thumbnails/34.jpg)
Why does vector arithmetic reveal analogies?• We wish to find the closest to • This is done with cosine similarity:
vector arithmetic similarity arithmetic
![Page 35: Linguistic Regularities in Sparse and Explicit Word Representations](https://reader031.fdocuments.in/reader031/viewer/2022012914/568163c9550346895dd50224/html5/thumbnails/35.jpg)
Why does vector arithmetic reveal analogies?• We wish to find the closest to • This is done with cosine similarity:
vector arithmetic similarity arithmetic
royal? female?
![Page 36: Linguistic Regularities in Sparse and Explicit Word Representations](https://reader031.fdocuments.in/reader031/viewer/2022012914/568163c9550346895dd50224/html5/thumbnails/36.jpg)
What does each similarity term mean?• Observe the joint features with explicit representations!
uncrowned Elizabethmajesty Katherinesecond impregnate
… …
![Page 37: Linguistic Regularities in Sparse and Explicit Word Representations](https://reader031.fdocuments.in/reader031/viewer/2022012914/568163c9550346895dd50224/html5/thumbnails/37.jpg)
Can we do better?
![Page 38: Linguistic Regularities in Sparse and Explicit Word Representations](https://reader031.fdocuments.in/reader031/viewer/2022012914/568163c9550346895dd50224/html5/thumbnails/38.jpg)
Let’s look at some mistakes…
![Page 39: Linguistic Regularities in Sparse and Explicit Word Representations](https://reader031.fdocuments.in/reader031/viewer/2022012914/568163c9550346895dd50224/html5/thumbnails/39.jpg)
Let’s look at some mistakes…
England London Baghdad ?
![Page 40: Linguistic Regularities in Sparse and Explicit Word Representations](https://reader031.fdocuments.in/reader031/viewer/2022012914/568163c9550346895dd50224/html5/thumbnails/40.jpg)
Let’s look at some mistakes…
England London Baghdad Iraq
![Page 41: Linguistic Regularities in Sparse and Explicit Word Representations](https://reader031.fdocuments.in/reader031/viewer/2022012914/568163c9550346895dd50224/html5/thumbnails/41.jpg)
Let’s look at some mistakes…
England London Baghdad Mosul?
![Page 42: Linguistic Regularities in Sparse and Explicit Word Representations](https://reader031.fdocuments.in/reader031/viewer/2022012914/568163c9550346895dd50224/html5/thumbnails/42.jpg)
The Additive Objective
![Page 43: Linguistic Regularities in Sparse and Explicit Word Representations](https://reader031.fdocuments.in/reader031/viewer/2022012914/568163c9550346895dd50224/html5/thumbnails/43.jpg)
The Additive Objective
![Page 44: Linguistic Regularities in Sparse and Explicit Word Representations](https://reader031.fdocuments.in/reader031/viewer/2022012914/568163c9550346895dd50224/html5/thumbnails/44.jpg)
The Additive Objective
![Page 45: Linguistic Regularities in Sparse and Explicit Word Representations](https://reader031.fdocuments.in/reader031/viewer/2022012914/568163c9550346895dd50224/html5/thumbnails/45.jpg)
The Additive Objective
![Page 46: Linguistic Regularities in Sparse and Explicit Word Representations](https://reader031.fdocuments.in/reader031/viewer/2022012914/568163c9550346895dd50224/html5/thumbnails/46.jpg)
The Additive Objective
![Page 47: Linguistic Regularities in Sparse and Explicit Word Representations](https://reader031.fdocuments.in/reader031/viewer/2022012914/568163c9550346895dd50224/html5/thumbnails/47.jpg)
The Additive Objective
• Problem: one similarity might dominate the rest• Much more prevalent in explicit representation• Might explain why explicit underperformed
![Page 48: Linguistic Regularities in Sparse and Explicit Word Representations](https://reader031.fdocuments.in/reader031/viewer/2022012914/568163c9550346895dd50224/html5/thumbnails/48.jpg)
How can we do better?
![Page 49: Linguistic Regularities in Sparse and Explicit Word Representations](https://reader031.fdocuments.in/reader031/viewer/2022012914/568163c9550346895dd50224/html5/thumbnails/49.jpg)
How can we do better?• Instead of adding similarities, multiply them!
![Page 50: Linguistic Regularities in Sparse and Explicit Word Representations](https://reader031.fdocuments.in/reader031/viewer/2022012914/568163c9550346895dd50224/html5/thumbnails/50.jpg)
How can we do better?• Instead of adding similarities, multiply them!
![Page 51: Linguistic Regularities in Sparse and Explicit Word Representations](https://reader031.fdocuments.in/reader031/viewer/2022012914/568163c9550346895dd50224/html5/thumbnails/51.jpg)
How can we do better?• Instead of adding similarities, multiply them!
![Page 52: Linguistic Regularities in Sparse and Explicit Word Representations](https://reader031.fdocuments.in/reader031/viewer/2022012914/568163c9550346895dd50224/html5/thumbnails/52.jpg)
Embedding vs Explicit (Round 2)
![Page 53: Linguistic Regularities in Sparse and Explicit Word Representations](https://reader031.fdocuments.in/reader031/viewer/2022012914/568163c9550346895dd50224/html5/thumbnails/53.jpg)
Multiplication > Addition
MSR Google MSR GoogleEmbedding Explicit
0%
10%
20%
30%
40%
50%
60%
70%
80%
Add54%
Add63%
Add29%
Add45%
Mul59%
Mul67% Mul
57%
Mul68%Ac
cura
cy
![Page 54: Linguistic Regularities in Sparse and Explicit Word Representations](https://reader031.fdocuments.in/reader031/viewer/2022012914/568163c9550346895dd50224/html5/thumbnails/54.jpg)
Explicit is on-par with Embedding
MSR Google0%
10%
20%
30%
40%
50%
60%
70%
80%
Embedding59%
Embedding67%Explicit
57%
Explicit68%Ac
cura
cy
![Page 55: Linguistic Regularities in Sparse and Explicit Word Representations](https://reader031.fdocuments.in/reader031/viewer/2022012914/568163c9550346895dd50224/html5/thumbnails/55.jpg)
Explicit is on-par with Embedding• Embeddings are not “magical”
• Embedding-based similarities have a more uniform distribution
• The additive objective performs better on smoother distributions
• The multiplicative objective overcomes this issue
![Page 56: Linguistic Regularities in Sparse and Explicit Word Representations](https://reader031.fdocuments.in/reader031/viewer/2022012914/568163c9550346895dd50224/html5/thumbnails/56.jpg)
Conclusion• Are analogies unique to neural embeddings?No! They occur in sparse and explicit representations as well.
• Why does vector arithmetic reveal analogies?Because vector arithmetic is equivalent to similarity arithmetic.
• Can we do better?Yes! The multiplicative objective is significantly better.
![Page 57: Linguistic Regularities in Sparse and Explicit Word Representations](https://reader031.fdocuments.in/reader031/viewer/2022012914/568163c9550346895dd50224/html5/thumbnails/57.jpg)
More Results and Analyses (in the paper)• Evaluation on closed-vocabulary analogy questions (SemEval 2012)
• Experiments with a third objective function (PairDirection)
• Do different representations reveal the same analogies?
• Error analysis
• A feature-level interpretation of how word similarity reveals analogies
![Page 58: Linguistic Regularities in Sparse and Explicit Word Representations](https://reader031.fdocuments.in/reader031/viewer/2022012914/568163c9550346895dd50224/html5/thumbnails/58.jpg)
Thanks for listening )