Neural Network Language Models and word2vec

Tambet Matiisen

8.10.2014

Sources

• Yoshua Bengio. Neural net language models.

• Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient Estimation of Word Representations in Vector Space.

• Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. Distributed Representations of Words and Phrases and their Compositionality.

• Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. Linguistic Regularities in Continuous Space Word Representations.

• Tomas Mikolov, Quoc V. Le and Ilya Sutskever. Exploiting Similarities among Languages for Machine Translation.

Language models

• A language model captures the statistical characteristics of sequences of words in a natural language, typically allowing one to make probabilistic predictions of the next word given preceding ones.

• E.g. the standard “trigram” method:

)(

)(),|(

12

1212

tt

tttttt

wwcount

wwwcountwwwP

Neural network language models

• A neural network language model is a language model based on neural networks, exploiting their ability to learn distributed representations.

• A distributed representation of a word is a vector of activations of neurons (real values) which characterizes the meaning of the word.

• A distributed representation is opposed to a local representation, in which only one neuron (or very few) is active at each time.

Softmax output layer (one unit per next word)

Hidden layer to predict output from features of the input words

Learned distributed representation of word t-2

Learned distributed representation of word t-1

Sparse representation of word t-2

Sparse representation of word t-1

NNLM architecture

V nodes

H nodes

D nodes D nodes

V nodes V nodes

HxV weights

HxD weights HxD weights

VxD weights (shared)

word2vec

• An efficient implementation of the continuous bag-of-words and skip-gram architectures for computing vector representations of words.

• The word vectors can be used to significantly improve and simplify many NLP applications.

CBOW architecture

Predicts current word given the context.

sparse representation

weights = distributed representation NB! Shared!

softmax

Skip-gram architecture

Predicts the surrounding words given the current word

sparse representation

weights = distributed representation

softmax

softmax

softmax

softmax

output weights

Linguistic regularities

The word vector space implicitly encodes many regularities among words, i.e. vector(KINGS) – vector(KING) +

vector(QUEEN) is close to vector(QUEENS)

Semantic-Syntactic Word Relationship test set

Accuracy

days

minutes

hours

From words to phrases

• Find words that appear frequently together and infrequently in other contexts.

• The bigrams with score above the chosen threshold are then used as phrases.

• The δ is used as a discounting coefficient and prevents too many phrases consisting of very infrequent words to be formed.

)()(

)(),(

ji

ji

jiwcountwcount

wwcountwwscore

Examples - analogy

Examples – distance (rare words)

Examples – addition

Parameters

• Architecture: skip-gram (slower, better for infrequent words) vs CBOW (fast)

• The training algorithm: hierarchical softmax (better for infrequent words) vs negative sampling (better for frequent words, better with low dimensional vectors)

• Sub-sampling of frequent words: can improve both accuracy and speed for large data sets (useful values are in range 1e-3 to 1e-5)

• Dimensionality of the word vectors: usually more is better, but not always

• Context (window) size: for skip-gram usually around 10, for CBOW around 5

Machine translation using distributed representations

1. Build monolingual models of languages using large amounts of text.

2. Use a small bilingual dictionary to learn a linear projection between the languages.

3. Translate a word by projecting its vector representation from the source language space to the target language space.

4. Output the most similar word vector from target language space as the translation.

English vs Spanish

Translation accuracy

English Spanish English Vietnamese

How is this related to neuroscience?

How to calculate similarity matrix

import sys

import gensim

if len(sys.argv) < 2:

print "Usage: matrix.py <vectorfile> <wordfile>"

sys.exit(1)

model = gensim.models.Word2Vec.load_word2vec_format(sys.argv[1], binary=True)

with open(sys.argv[2]) as f:

words = f.read().splitlines()

for w1 in words:

s = ""

for w2 in words:

if s != "": s += ","

s += str(model.similarity(w1, w2))

print s

Discovery of structural form - animals

Discovery of structural form - cities