Word Embeddings - Princeton University · Word2vec • Input: a large text corpora, V, d •...

36
Word Embeddings Fall 2019 COS 484: Natural Language Processing (Slides adapted from Chris Manning, Dan Jurafsky)

Transcript of Word Embeddings - Princeton University · Word2vec • Input: a large text corpora, V, d •...

Page 1: Word Embeddings - Princeton University · Word2vec • Input: a large text corpora, V, d • Output: • V: a pre-defined vocabulary • d: dimension of word vectors (e.g. 300) •

Word Embeddings

Fall 2019

COS 484: Natural Language Processing

(Slides adapted from Chris Manning, Dan Jurafsky)

Page 2: Word Embeddings - Princeton University · Word2vec • Input: a large text corpora, V, d • Output: • V: a pre-defined vocabulary • d: dimension of word vectors (e.g. 300) •

How to represent words?

N-gram language models

It is 76 F and ___. [0.0001, 0.1, 0, 0, 0.002, …, 0.3, …, 0]

!P(w ∣ it is 76 F and)

red

Text classification

I like this movie. 👍

I don’t like this movie. 👎

[0, 1, 0, 0, 0, …, 1, …, 1][0, 1, 0, 1, 0, …, 1, …, 1]

!P(y = 1 ∣ x) = σ(θ⊺w + b)

!w(1)

!w(2)

sunny

don’t

Page 3: Word Embeddings - Princeton University · Word2vec • Input: a large text corpora, V, d • Output: • V: a pre-defined vocabulary • d: dimension of word vectors (e.g. 300) •

Representing words as discrete symbols

In traditional NLP, we regard words as discrete symbols: hotel, conference, motel — a localist representation

Words can be represented by one-hot vectors:

one 1, the rest 0’s

Vector dimension = number of words in vocabulary (e.g., 500,000)

hotel = [0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0] motel = [0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0]

There is no way to encode similarity of words in these vectors!

Page 4: Word Embeddings - Princeton University · Word2vec • Input: a large text corpora, V, d • Output: • V: a pre-defined vocabulary • d: dimension of word vectors (e.g. 300) •

Representing words by their context

Distributional hypothesis: words that occur in similar contexts tend to have similar meanings

J.R.Firth 1957

• “You shall know a word by the company it keeps”

• One of the most successful ideas of modern statistical NLP!

These context words will represent banking.

Page 5: Word Embeddings - Princeton University · Word2vec • Input: a large text corpora, V, d • Output: • V: a pre-defined vocabulary • d: dimension of word vectors (e.g. 300) •

Distributional hypothesis

“tejuino” C1: A bottle of ___ is on the table.

C2: Everybody likes ___.

C3: Don’t have ___ before you drive.

C4: We make ___ out of corn.

Page 6: Word Embeddings - Princeton University · Word2vec • Input: a large text corpora, V, d • Output: • V: a pre-defined vocabulary • d: dimension of word vectors (e.g. 300) •

Distributional hypothesis

C1 C2 C3 C4

tejuino 1 1 1 1

loud 0 0 0 0

motor-oil 1 0 0 0

tortillas 0 1 0 1

choices 0 1 0 0

wine 1 1 1 0

C1: A bottle of ___ is on the table.

C2: Everybody likes ___.

C3: Don’t have ___ before you drive.

C4: We make ___ out of corn.

“words that occur in similar contexts tend to have similar meanings”

Page 7: Word Embeddings - Princeton University · Word2vec • Input: a large text corpora, V, d • Output: • V: a pre-defined vocabulary • d: dimension of word vectors (e.g. 300) •

Words as vectors• We’ll build a new model of meaning focusing on similarity

• Each word is a vector • Similar words are “nearby in space”

• word-word co-occurrence matrix:

• A first solution: we can just use context vectors to represent the meaning of words!

Page 8: Word Embeddings - Princeton University · Word2vec • Input: a large text corpora, V, d • Output: • V: a pre-defined vocabulary • d: dimension of word vectors (e.g. 300) •

Words as vectors

cos(u,v) =u · v

kukkvk<latexit sha1_base64="LwUwX9BK80Y7mx5ulvUsqXb08A0=">AAACSHicbVDLSsNAFJ3UV62vqEs3g0WoICURQTdC0Y3LCvYBTQiT6aQdOsmEmUmhpPk8Ny7d+Q1uXCjizklbqbYeGDj3nHu5d44fMyqVZb0YhZXVtfWN4mZpa3tnd8/cP2hKnghMGpgzLto+koTRiDQUVYy0Y0FQ6DPS8ge3ud8aEiEpjx7UKCZuiHoRDShGSkue6WEuK06IVN8P0iQ7gz98mJ3Ca+gEAuF07kMHd7n61ZSlznhuO2M4L4e6zDyzbFWtCeAysWekDGaoe+az0+U4CUmkMENSdmwrVm6KhKKYkazkJJLECA9Qj3Q0jVBIpJtOgsjgiVa6MOBCv0jBifp7IkWhlKPQ1535kXLRy8X/vE6igis3pVGcKBLh6aIgYVBxmKcKu1QQrNhIE4QF1bdC3Ec6O6WzL+kQ7MUvL5PmedW2qvb9Rbl2M4ujCI7AMagAG1yCGrgDddAAGDyCV/AOPown4834NL6mrQVjNnMI/qBQ+AbQvrTU</latexit><latexit sha1_base64="LwUwX9BK80Y7mx5ulvUsqXb08A0=">AAACSHicbVDLSsNAFJ3UV62vqEs3g0WoICURQTdC0Y3LCvYBTQiT6aQdOsmEmUmhpPk8Ny7d+Q1uXCjizklbqbYeGDj3nHu5d44fMyqVZb0YhZXVtfWN4mZpa3tnd8/cP2hKnghMGpgzLto+koTRiDQUVYy0Y0FQ6DPS8ge3ud8aEiEpjx7UKCZuiHoRDShGSkue6WEuK06IVN8P0iQ7gz98mJ3Ca+gEAuF07kMHd7n61ZSlznhuO2M4L4e6zDyzbFWtCeAysWekDGaoe+az0+U4CUmkMENSdmwrVm6KhKKYkazkJJLECA9Qj3Q0jVBIpJtOgsjgiVa6MOBCv0jBifp7IkWhlKPQ1535kXLRy8X/vE6igis3pVGcKBLh6aIgYVBxmKcKu1QQrNhIE4QF1bdC3Ec6O6WzL+kQ7MUvL5PmedW2qvb9Rbl2M4ujCI7AMagAG1yCGrgDddAAGDyCV/AOPown4834NL6mrQVjNnMI/qBQ+AbQvrTU</latexit><latexit sha1_base64="LwUwX9BK80Y7mx5ulvUsqXb08A0=">AAACSHicbVDLSsNAFJ3UV62vqEs3g0WoICURQTdC0Y3LCvYBTQiT6aQdOsmEmUmhpPk8Ny7d+Q1uXCjizklbqbYeGDj3nHu5d44fMyqVZb0YhZXVtfWN4mZpa3tnd8/cP2hKnghMGpgzLto+koTRiDQUVYy0Y0FQ6DPS8ge3ud8aEiEpjx7UKCZuiHoRDShGSkue6WEuK06IVN8P0iQ7gz98mJ3Ca+gEAuF07kMHd7n61ZSlznhuO2M4L4e6zDyzbFWtCeAysWekDGaoe+az0+U4CUmkMENSdmwrVm6KhKKYkazkJJLECA9Qj3Q0jVBIpJtOgsjgiVa6MOBCv0jBifp7IkWhlKPQ1535kXLRy8X/vE6igis3pVGcKBLh6aIgYVBxmKcKu1QQrNhIE4QF1bdC3Ec6O6WzL+kQ7MUvL5PmedW2qvb9Rbl2M4ujCI7AMagAG1yCGrgDddAAGDyCV/AOPown4834NL6mrQVjNnMI/qBQ+AbQvrTU</latexit><latexit sha1_base64="LwUwX9BK80Y7mx5ulvUsqXb08A0=">AAACSHicbVDLSsNAFJ3UV62vqEs3g0WoICURQTdC0Y3LCvYBTQiT6aQdOsmEmUmhpPk8Ny7d+Q1uXCjizklbqbYeGDj3nHu5d44fMyqVZb0YhZXVtfWN4mZpa3tnd8/cP2hKnghMGpgzLto+koTRiDQUVYy0Y0FQ6DPS8ge3ud8aEiEpjx7UKCZuiHoRDShGSkue6WEuK06IVN8P0iQ7gz98mJ3Ca+gEAuF07kMHd7n61ZSlznhuO2M4L4e6zDyzbFWtCeAysWekDGaoe+az0+U4CUmkMENSdmwrVm6KhKKYkazkJJLECA9Qj3Q0jVBIpJtOgsjgiVa6MOBCv0jBifp7IkWhlKPQ1535kXLRy8X/vE6igis3pVGcKBLh6aIgYVBxmKcKu1QQrNhIE4QF1bdC3Ec6O6WzL+kQ7MUvL5PmedW2qvb9Rbl2M4ujCI7AMagAG1yCGrgDddAAGDyCV/AOPown4834NL6mrQVjNnMI/qBQ+AbQvrTU</latexit>

cos(u,v) =

PVi=1 uiviqPV

i=1 u2i

qPVi=1 v

2i

<latexit sha1_base64="Ex14HCNmlweatqm/BBdVuIIyGZI=">AAACX3icbZFLSwMxFIUz47tWHXUlboJFqCBlpgi6EUQ3LivYKnTaIZNmajDzMI+BEvIn3Qlu/Cdm2graeiFwON+9JPckLhgV0vc/HHdldW19Y3Ortl3f2d3z9g96Ilccky7OWc6fYyQIoxnpSioZeS44QWnMyFP8elfxp5JwQfPsUU4KMkjROKMJxUhaK/JKnItmmCL5EidamXP4o0tzBq9hmHCEdShUGml6HZih7hmtIgrLiBpjwRuXS3jYruC/rJwxE3kNv+VPCy6LYC4aYF6dyHsPRzlWKckkZkiIfuAXcqARlxQzYmqhEqRA+BWNSd/KDKVEDPQ0HwNPrTOCSc7tySScur8nNEqFmKSx7ay2F4usMv9jfSWTq4GmWaEkyfDsokQxKHNYhQ1HlBMs2cQKhDm1b4X4BdlIpf2Smg0hWFx5WfTarcBvBQ8XjZvbeRyb4BicgCYIwCW4AfegA7oAg0/HdbaduvPlbri7rjdrdZ35zCH4U+7RNyRRuQc=</latexit><latexit sha1_base64="Ex14HCNmlweatqm/BBdVuIIyGZI=">AAACX3icbZFLSwMxFIUz47tWHXUlboJFqCBlpgi6EUQ3LivYKnTaIZNmajDzMI+BEvIn3Qlu/Cdm2graeiFwON+9JPckLhgV0vc/HHdldW19Y3Ortl3f2d3z9g96Ilccky7OWc6fYyQIoxnpSioZeS44QWnMyFP8elfxp5JwQfPsUU4KMkjROKMJxUhaK/JKnItmmCL5EidamXP4o0tzBq9hmHCEdShUGml6HZih7hmtIgrLiBpjwRuXS3jYruC/rJwxE3kNv+VPCy6LYC4aYF6dyHsPRzlWKckkZkiIfuAXcqARlxQzYmqhEqRA+BWNSd/KDKVEDPQ0HwNPrTOCSc7tySScur8nNEqFmKSx7ay2F4usMv9jfSWTq4GmWaEkyfDsokQxKHNYhQ1HlBMs2cQKhDm1b4X4BdlIpf2Smg0hWFx5WfTarcBvBQ8XjZvbeRyb4BicgCYIwCW4AfegA7oAg0/HdbaduvPlbri7rjdrdZ35zCH4U+7RNyRRuQc=</latexit><latexit sha1_base64="Ex14HCNmlweatqm/BBdVuIIyGZI=">AAACX3icbZFLSwMxFIUz47tWHXUlboJFqCBlpgi6EUQ3LivYKnTaIZNmajDzMI+BEvIn3Qlu/Cdm2graeiFwON+9JPckLhgV0vc/HHdldW19Y3Ortl3f2d3z9g96Ilccky7OWc6fYyQIoxnpSioZeS44QWnMyFP8elfxp5JwQfPsUU4KMkjROKMJxUhaK/JKnItmmCL5EidamXP4o0tzBq9hmHCEdShUGml6HZih7hmtIgrLiBpjwRuXS3jYruC/rJwxE3kNv+VPCy6LYC4aYF6dyHsPRzlWKckkZkiIfuAXcqARlxQzYmqhEqRA+BWNSd/KDKVEDPQ0HwNPrTOCSc7tySScur8nNEqFmKSx7ay2F4usMv9jfSWTq4GmWaEkyfDsokQxKHNYhQ1HlBMs2cQKhDm1b4X4BdlIpf2Smg0hWFx5WfTarcBvBQ8XjZvbeRyb4BicgCYIwCW4AfegA7oAg0/HdbaduvPlbri7rjdrdZ35zCH4U+7RNyRRuQc=</latexit><latexit sha1_base64="Ex14HCNmlweatqm/BBdVuIIyGZI=">AAACX3icbZFLSwMxFIUz47tWHXUlboJFqCBlpgi6EUQ3LivYKnTaIZNmajDzMI+BEvIn3Qlu/Cdm2graeiFwON+9JPckLhgV0vc/HHdldW19Y3Ortl3f2d3z9g96Ilccky7OWc6fYyQIoxnpSioZeS44QWnMyFP8elfxp5JwQfPsUU4KMkjROKMJxUhaK/JKnItmmCL5EidamXP4o0tzBq9hmHCEdShUGml6HZih7hmtIgrLiBpjwRuXS3jYruC/rJwxE3kNv+VPCy6LYC4aYF6dyHsPRzlWKckkZkiIfuAXcqARlxQzYmqhEqRA+BWNSd/KDKVEDPQ0HwNPrTOCSc7tySScur8nNEqFmKSx7ay2F4usMv9jfSWTq4GmWaEkyfDsokQxKHNYhQ1HlBMs2cQKhDm1b4X4BdlIpf2Smg0hWFx5WfTarcBvBQ8XjZvbeRyb4BicgCYIwCW4AfegA7oAg0/HdbaduvPlbri7rjdrdZ35zCH4U+7RNyRRuQc=</latexit>

What is the range of ! ?cos( ⋅ )

Page 9: Word Embeddings - Princeton University · Word2vec • Input: a large text corpora, V, d • Output: • V: a pre-defined vocabulary • d: dimension of word vectors (e.g. 300) •

Words as vectors

Problem: using raw frequency counts is not always very good.. • Solution: let’s weight the counts! • PPMI = Positive Pointwise Mutual Information

Page 10: Word Embeddings - Princeton University · Word2vec • Input: a large text corpora, V, d • Output: • V: a pre-defined vocabulary • d: dimension of word vectors (e.g. 300) •

Sparse vs dense vectors

• Still, the vectors we get from word-word occurrence matrix are sparse (most are 0’s) & long (vocabulary size)

• Alternative: we want to represent words as short (50-300 dimensional) & dense (real-valued) vectors

• The focus of this lecture • The basis of all the modern NLP systems

Page 11: Word Embeddings - Princeton University · Word2vec • Input: a large text corpora, V, d • Output: • V: a pre-defined vocabulary • d: dimension of word vectors (e.g. 300) •

Dense vectors

employees =

0

BBBBBBBBBBBB@

0.2860.792�0.177�0.10710.109�0.5420.3490.2710.487

1

CCCCCCCCCCCCA

<latexit sha1_base64="rlyV9z4BFdXN5ATSBwwve48t4Vs=">AAACa3icbZHPT9swFMedjA3oNijjAGI7WKuQdlmVhELaAxKCC0cmrYDUVJXjvhYLx4nsF0QV9bI/cTf+Ay78DzhNQAP2JEsffd8PP38dZ1IY9Lw7x3239P7D8spq4+Onz2vrzY0v5ybNNYc+T2WqL2NmQAoFfRQo4TLTwJJYwkV8fVLmL25AG5Gq3zjLYJiwqRITwRlaadT8EyHcYgFJJtMZgJnTQxrFMBWqyBKGWtzOqdcOugc0iiyEvaCEn17bD8Mn8hbkl9Srtf1OUDXsdXoVBKFfQadry0GNn+ePmi2v7S2CvgW/hhap42zU/BuNU54noJBLZszA9zIcFkyj4BLmjSg3kDF+zaYwsKhYAmZYLLya012rjOkk1fYopAv1346CJcbMkthW2v2uzOtcKf4vN8hx0h0WQmU5guLVRZNcUkxpaTwdCw0c5cwC41rYXSm/YppxtN/TsCb4r5/8Fs6Ddmnzr07r6Li2Y4V8Jd/JD+KTkByRU3JG+oSTe2fN2XK2nQd3091xv1WlrlP3bJIX4e4+Ani5r64=</latexit><latexit sha1_base64="rlyV9z4BFdXN5ATSBwwve48t4Vs=">AAACa3icbZHPT9swFMedjA3oNijjAGI7WKuQdlmVhELaAxKCC0cmrYDUVJXjvhYLx4nsF0QV9bI/cTf+Ay78DzhNQAP2JEsffd8PP38dZ1IY9Lw7x3239P7D8spq4+Onz2vrzY0v5ybNNYc+T2WqL2NmQAoFfRQo4TLTwJJYwkV8fVLmL25AG5Gq3zjLYJiwqRITwRlaadT8EyHcYgFJJtMZgJnTQxrFMBWqyBKGWtzOqdcOugc0iiyEvaCEn17bD8Mn8hbkl9Srtf1OUDXsdXoVBKFfQadry0GNn+ePmi2v7S2CvgW/hhap42zU/BuNU54noJBLZszA9zIcFkyj4BLmjSg3kDF+zaYwsKhYAmZYLLya012rjOkk1fYopAv1346CJcbMkthW2v2uzOtcKf4vN8hx0h0WQmU5guLVRZNcUkxpaTwdCw0c5cwC41rYXSm/YppxtN/TsCb4r5/8Fs6Ddmnzr07r6Li2Y4V8Jd/JD+KTkByRU3JG+oSTe2fN2XK2nQd3091xv1WlrlP3bJIX4e4+Ani5r64=</latexit><latexit sha1_base64="rlyV9z4BFdXN5ATSBwwve48t4Vs=">AAACa3icbZHPT9swFMedjA3oNijjAGI7WKuQdlmVhELaAxKCC0cmrYDUVJXjvhYLx4nsF0QV9bI/cTf+Ay78DzhNQAP2JEsffd8PP38dZ1IY9Lw7x3239P7D8spq4+Onz2vrzY0v5ybNNYc+T2WqL2NmQAoFfRQo4TLTwJJYwkV8fVLmL25AG5Gq3zjLYJiwqRITwRlaadT8EyHcYgFJJtMZgJnTQxrFMBWqyBKGWtzOqdcOugc0iiyEvaCEn17bD8Mn8hbkl9Srtf1OUDXsdXoVBKFfQadry0GNn+ePmi2v7S2CvgW/hhap42zU/BuNU54noJBLZszA9zIcFkyj4BLmjSg3kDF+zaYwsKhYAmZYLLya012rjOkk1fYopAv1346CJcbMkthW2v2uzOtcKf4vN8hx0h0WQmU5guLVRZNcUkxpaTwdCw0c5cwC41rYXSm/YppxtN/TsCb4r5/8Fs6Ddmnzr07r6Li2Y4V8Jd/JD+KTkByRU3JG+oSTe2fN2XK2nQd3091xv1WlrlP3bJIX4e4+Ani5r64=</latexit><latexit sha1_base64="rlyV9z4BFdXN5ATSBwwve48t4Vs=">AAACa3icbZHPT9swFMedjA3oNijjAGI7WKuQdlmVhELaAxKCC0cmrYDUVJXjvhYLx4nsF0QV9bI/cTf+Ay78DzhNQAP2JEsffd8PP38dZ1IY9Lw7x3239P7D8spq4+Onz2vrzY0v5ybNNYc+T2WqL2NmQAoFfRQo4TLTwJJYwkV8fVLmL25AG5Gq3zjLYJiwqRITwRlaadT8EyHcYgFJJtMZgJnTQxrFMBWqyBKGWtzOqdcOugc0iiyEvaCEn17bD8Mn8hbkl9Srtf1OUDXsdXoVBKFfQadry0GNn+ePmi2v7S2CvgW/hhap42zU/BuNU54noJBLZszA9zIcFkyj4BLmjSg3kDF+zaYwsKhYAmZYLLya012rjOkk1fYopAv1346CJcbMkthW2v2uzOtcKf4vN8hx0h0WQmU5guLVRZNcUkxpaTwdCw0c5cwC41rYXSm/YppxtN/TsCb4r5/8Fs6Ddmnzr07r6Li2Y4V8Jd/JD+KTkByRU3JG+oSTe2fN2XK2nQd3091xv1WlrlP3bJIX4e4+Ani5r64=</latexit>

Page 12: Word Embeddings - Princeton University · Word2vec • Input: a large text corpora, V, d • Output: • V: a pre-defined vocabulary • d: dimension of word vectors (e.g. 300) •

Why dense vectors?

• Short vectors are easier to use as features in ML systems • Dense vectors may generalize better than storing explicit counts • They do better at capturing synonymy

• ! co-occurs with “car”, ! co-occurs with “automobile”w1 w2

• Different methods for getting dense vectors: • Singular value decomposition (SVD) • word2vec and friends: “learn” the vectors!

Page 13: Word Embeddings - Princeton University · Word2vec • Input: a large text corpora, V, d • Output: • V: a pre-defined vocabulary • d: dimension of word vectors (e.g. 300) •

Word2vec and friends

(Mikolov et al, 2013): Distributed Representations of Words and Phrases and their Compositionality

Page 14: Word Embeddings - Princeton University · Word2vec • Input: a large text corpora, V, d • Output: • V: a pre-defined vocabulary • d: dimension of word vectors (e.g. 300) •

Word2vec

• Input: a large text corpora, V, d

• Output:

• V: a pre-defined vocabulary • d: dimension of word vectors (e.g. 300) • Text corpora:

• Wikipedia + Gigaword 5: 6B • Twitter: 27B

• Common Crawl: 840B

vcat =

0

BB@

�0.2240.130�0.2900.276

1

CCA

<latexit sha1_base64="ZS11t+SATcIQYaaJ4VZuEjXjz0Y=">AAACOXicbZDPShxBEMZrjIk65s8aj3poIoFcsvRsQlSIIHjxuIKrws6y9PTWro09PUN3jbgM8wx5m1zyFt4ELx4U8ZoXSM+uiFE/aPj4VRVd9SW5Vo44vwhmXs2+fjM3vxAuvn33/kNj6eOByworsSMzndmjRDjUymCHFGk8yi2KNNF4mJzs1PXDU7ROZWafxjn2UjEyaqikII/6jfZpv4wJz6j0pKq2wjjBkTJlngqy6qwKv/Jmq/WdxXHIm9E3XpsabfIpaq3/CGM0g4eBfmONN/lE7LmJ7s3a9uqvmAFAu984jweZLFI0JLVwrhvxnHqlsKSkxiqMC4e5kCdihF1vjUjR9crJ5RX77MmADTPrnyE2oY8nSpE6N04T3+n3O3ZPazV8qdYtaLjRK5XJC0Ijpx8NC80oY3WMbKAsStJjb4S0yu/K5LGwQpIPO/QhRE9Pfm4OWs3Ih7rn0/gJU83DCnyCLxDBOmzDLrShAxJ+wyVcw03wJ7gKboO7aetMcD+zDP8p+PsPf3eqbQ==</latexit><latexit sha1_base64="X7JObiHYNXwbsISLOmkjXbSsJws=">AAACOXicbZBNSyNBEIZ7/Fh1dHejHvXQrCx42dCTFT9AQfDiMYJRIRNCT6cSG3t6hu4aMQzzG/w3Xrz5E7wJXjwo4lW825OIrLovNLw8VUVXvVGqpEXGbryR0bHxbxOTU/70zPcfPyuzcwc2yYyAhkhUYo4ibkFJDQ2UqOAoNcDjSMFhdLJT1g9PwViZ6H3sp9CKeU/LrhQcHWpX6qftPEQ4w9yRotjywwh6UudpzNHIs8L/w6q12goNQ59Vg7+sNCXaYENUW1v1Q9Cd94F2ZYlV2UD0qwnezNL24nm4/HJ1Xm9XrsNOIrIYNArFrW0GLMVWzg1KoaDww8xCysUJ70HTWc1jsK18cHlBfzvSod3EuKeRDui/EzmPre3Hket0+x3bz7US/q/WzLC73sqlTjMELYYfdTNFMaFljLQjDQhUfWe4MNLtSsUxN1ygC9t3IQSfT/5qDmrVwIW659LYJENNkgXyiyyTgKyRbbJL6qRBBLkgt+SePHiX3p336D0NW0e8t5l58kHe8yuUTqy7</latexit><latexit sha1_base64="X7JObiHYNXwbsISLOmkjXbSsJws=">AAACOXicbZBNSyNBEIZ7/Fh1dHejHvXQrCx42dCTFT9AQfDiMYJRIRNCT6cSG3t6hu4aMQzzG/w3Xrz5E7wJXjwo4lW825OIrLovNLw8VUVXvVGqpEXGbryR0bHxbxOTU/70zPcfPyuzcwc2yYyAhkhUYo4ibkFJDQ2UqOAoNcDjSMFhdLJT1g9PwViZ6H3sp9CKeU/LrhQcHWpX6qftPEQ4w9yRotjywwh6UudpzNHIs8L/w6q12goNQ59Vg7+sNCXaYENUW1v1Q9Cd94F2ZYlV2UD0qwnezNL24nm4/HJ1Xm9XrsNOIrIYNArFrW0GLMVWzg1KoaDww8xCysUJ70HTWc1jsK18cHlBfzvSod3EuKeRDui/EzmPre3Hket0+x3bz7US/q/WzLC73sqlTjMELYYfdTNFMaFljLQjDQhUfWe4MNLtSsUxN1ygC9t3IQSfT/5qDmrVwIW659LYJENNkgXyiyyTgKyRbbJL6qRBBLkgt+SePHiX3p336D0NW0e8t5l58kHe8yuUTqy7</latexit><latexit sha1_base64="yUhkDlYwUUEoQ+3MeiaCkTTY5/M=">AAACOXicbZBNSyNBEIZ71PVj3NWsHr00BsHLhp4oxgUFwYvHCEaFTAg9nUps7OkZumvEMMzf8uK/8CZ48bCLePUP2JME8euFhpenquiqN0qVtMjYvTc1PfNjdm5+wV/8+WtpufJ75dQmmRHQEolKzHnELSipoYUSFZynBngcKTiLLg/L+tkVGCsTfYLDFDoxH2jZl4KjQ91K86qbhwjXmDtSFPt+GMFA6jyNORp5Xfh/WK1e36Zh6LNasMVKU6K/bIzqjR0/BN17G+hWqqzGRqJfTTAxVTJRs1u5C3uJyGLQKBS3th2wFDs5NyiFgsIPMwspF5d8AG1nNY/BdvLR5QXdcKRH+4lxTyMd0fcTOY+tHcaR63T7XdjPtRJ+V2tn2N/t5FKnGYIW44/6maKY0DJG2pMGBKqhM1wY6Xal4oIbLtCF7bsQgs8nfzWn9VrgQj1m1YO9SRzzZI2sk00SkAY5IEekSVpEkBvyQP6R/96t9+g9ec/j1ilvMrNKPsh7eQWaI6kG</latexit>

vdog =

0

BB@

�0.1240.430�0.2000.329

1

CCA

<latexit sha1_base64="7A8Mq63LMTMc+l3UeNcP10h1a/Y=">AAACOXicbVBNSxxBFHyjJjGTDzd6zKVRhFyy9KyCBiIIuXgRNsRVYWdZenrero09PUP3G3EZ5m958V94C3jQgyJe8wfSuyuSqAUNRdUr+r1KCq0ccf47mJmde/X6zfzb8N37Dx8XGp8W911eWokdmevcHibCoVYGO6RI42FhUWSJxoPk+MfYPzhB61Ru9mhUYC8TQ6MGSgryUr/RPulXMeEpVWk+rOutME5wqExVZIKsOq3Dr7wZtdZZHIe8ub7Gx8RLLc6n0lrrWxijSR8D/cYKb/IJ2HMSPZCVbbb76woA2v3GRZzmsszQkNTCuW7EC+pVwpKSGuswLh0WQh6LIXY9NSJD16sml9ds1SspG+TWP0Nsov6bqETm3ChL/KTf78g99cbiS163pMFmr1KmKAmNnH40KDWjnI1rZKmyKEmPPBHSKr8rk0fCCkm+7NCXED09+TnZbzUj3+5P38Z3mGIePsMyfIEINmAbdqANHZBwBpdwA7fBeXAd3AX309GZ4CGzBP8h+PMXGHGq4A==</latexit><latexit sha1_base64="uVvmVkONvo7ZgSwrHlzj6B+jPNQ=">AAACOXicbVDPaxNBGJ1tba1rf2zr0ctgKHhpmE0CtdBCQRAvQkSTFrJLmJ39kgyZnV1mvg0Ny/5bXvwvvAlePCjixYP/gJOkiDZ9MPB473vM972kUNIiY5+9jc0HW9sPdx75j3f39g+Cw6O+zUsjoCdylZvrhFtQUkMPJSq4LgzwLFFwlUxfLvyrGRgrc/0e5wXEGR9rOZKCo5OGQXc2rCKEG6zSfFzXF36UwFjqqsg4GnlT+yesGbY6NIp81uy02YI4qcXYSmq3zvwIdPo3MAwarMmWoOskvCWNS/rm3a9p/Ko7DD5FaS7KDDQKxa0dhKzAuOIGpVBQ+1FpoeBiyscwcFTzDGxcLS+v6bFTUjrKjXsa6VL9N1HxzNp5lrhJt9/E3vUW4n3eoMTRi7iSuigRtFh9NCoVxZwuaqSpNCBQzR3hwki3KxUTbrhAV7bvSgjvnrxO+q1m6Np969o4JyvskKfkGXlOQnJKLslr0iU9IsgH8oV8I9+9j95X74f3czW64d1mnpD/4P3+A5WBq/0=</latexit><latexit sha1_base64="uVvmVkONvo7ZgSwrHlzj6B+jPNQ=">AAACOXicbVDPaxNBGJ1tba1rf2zr0ctgKHhpmE0CtdBCQRAvQkSTFrJLmJ39kgyZnV1mvg0Ny/5bXvwvvAlePCjixYP/gJOkiDZ9MPB473vM972kUNIiY5+9jc0HW9sPdx75j3f39g+Cw6O+zUsjoCdylZvrhFtQUkMPJSq4LgzwLFFwlUxfLvyrGRgrc/0e5wXEGR9rOZKCo5OGQXc2rCKEG6zSfFzXF36UwFjqqsg4GnlT+yesGbY6NIp81uy02YI4qcXYSmq3zvwIdPo3MAwarMmWoOskvCWNS/rm3a9p/Ko7DD5FaS7KDDQKxa0dhKzAuOIGpVBQ+1FpoeBiyscwcFTzDGxcLS+v6bFTUjrKjXsa6VL9N1HxzNp5lrhJt9/E3vUW4n3eoMTRi7iSuigRtFh9NCoVxZwuaqSpNCBQzR3hwki3KxUTbrhAV7bvSgjvnrxO+q1m6Np969o4JyvskKfkGXlOQnJKLslr0iU9IsgH8oV8I9+9j95X74f3czW64d1mnpD/4P3+A5WBq/0=</latexit><latexit sha1_base64="QWAMEHLqCoErtEma6Wi/777uLqM=">AAACOXicbVDLSiNBFK12fMT2lRmXbgqD4MZQHQUVHAi4cRnBqJAOobr6JhZWVzdVt8XQ9G/NZv5idgNuXCji1h+w8kB8HSg4nHMPde+JMiUtMvbfm/kxOze/UFn0l5ZXVteqP3+d2zQ3AtoiVam5jLgFJTW0UaKCy8wATyIFF9H18ci/uAFjZarPcJhBN+EDLftScHRSr9q66RUhwi0WcTooy99+GMFA6iJLOBp5W/o7rB409mgY+qy+t8tGxEkNxibSbuPQD0HHb4FetcbqbAz6lQRTUiNTtHrVf2GcijwBjUJxazsBy7BbcINSKCj9MLeQcXHNB9BxVPMEbLcYX17SLafEtJ8a9zTSsfo+UfDE2mESuUm335X97I3E77xOjv2DbiF1liNoMfmonyuKKR3VSGNpQKAaOsKFkW5XKq644QJd2b4rIfh88ldy3qgHrt1TVmseTeuokA2ySbZJQPZJk5yQFmkTQf6QO/JAHr2/3r335D1PRme8aWadfID38gqQ96kA</latexit>

vthe =

0

BB@

0.2340.2660.239�0.199

1

CCA

<latexit sha1_base64="odYGt+syjpaXyhzBR2lH0qeQ+vM=">AAACOHicbZBBSxtBFMffWtuma1tje6yHoSJ4adjVogYUBC/etGBUyIYwO3lJBmdnl5m3krDsZ+in6cWP0VvpxYMiXv0EnU2CWPUPAz/+7z3mvX+cKWkpCP54c6/mX795W3vnL7z/8HGxvvTpxKa5EdgSqUrNWcwtKqmxRZIUnmUGeRIrPI3P96v66QUaK1N9TOMMOwkfaNmXgpOzuvXDi24REY6ooCGW5a4fxTiQusgSTkaOSj9orG98Z1FUwebmDDaaFXwLGmGz6Ueoew/93fpK0AgmYs8hnMHK3vLPiAHAUbf+O+qlIk9Qk1Dc2nYYZNQpuCEpFJZ+lFvMuDjnA2w71DxB2ykmh5ds1Tk91k+Ne5rYxH08UfDE2nESu06339A+rVXmS7V2Tv3tTiF1lhNqMf2onytGKatSZD1pUJAaO+DCSLcrE0NuuCCXte9CCJ+e/BxO1huhS/CHS2MHpqrBF/gKaxDCFuzBARxBCwT8gr9wDTfepXfl3Xp309Y5bzbzGf6Td/8PLj6qUQ==</latexit><latexit sha1_base64="kC++ZoeREatr7v57EgIo3XrJSss=">AAACOHicbZDPattAEMZXSds46p84ybE9LA0FX2okOzgxJGDIpbe6UDsBy5jVemwvWa3E7sjYCD2DnyaXHvsIuYVeekgpvRZ678o2pXX6wcKPb2bYmS9MpDDoeXfO1vajx092Srvu02fPX+yV9w+6Jk41hw6PZayvQmZACgUdFCjhKtHAolDCZXh9UdQvp6CNiNVHnCfQj9hYiZHgDK01KL+fDrIAYYYZTiDPz90ghLFQWRIx1GKWu161Vj+mQVBAo7GGerOAt17VbzbdANTwT/+gfORVvaXoQ/DXcNR6tQgqvz4v2oPybTCMeRqBQi6ZMT3fS7CfMY2CS8jdIDWQMH7NxtCzqFgEpp8tD8/pG+sM6SjW9imkS/fviYxFxsyj0Hba/SZms1aY/6v1Uhyd9jOhkhRB8dVHo1RSjGmRIh0KDRzl3ALjWthdKZ8wzTjarF0bgr958kPo1qq+TfCDTeOMrFQiL8lrUiE+OSEt8o60SYdwckO+kHvyzfnkfHW+Oz9WrVvOeuaQ/CPn529DFayf</latexit><latexit sha1_base64="kC++ZoeREatr7v57EgIo3XrJSss=">AAACOHicbZDPattAEMZXSds46p84ybE9LA0FX2okOzgxJGDIpbe6UDsBy5jVemwvWa3E7sjYCD2DnyaXHvsIuYVeekgpvRZ678o2pXX6wcKPb2bYmS9MpDDoeXfO1vajx092Srvu02fPX+yV9w+6Jk41hw6PZayvQmZACgUdFCjhKtHAolDCZXh9UdQvp6CNiNVHnCfQj9hYiZHgDK01KL+fDrIAYYYZTiDPz90ghLFQWRIx1GKWu161Vj+mQVBAo7GGerOAt17VbzbdANTwT/+gfORVvaXoQ/DXcNR6tQgqvz4v2oPybTCMeRqBQi6ZMT3fS7CfMY2CS8jdIDWQMH7NxtCzqFgEpp8tD8/pG+sM6SjW9imkS/fviYxFxsyj0Hba/SZms1aY/6v1Uhyd9jOhkhRB8dVHo1RSjGmRIh0KDRzl3ALjWthdKZ8wzTjarF0bgr958kPo1qq+TfCDTeOMrFQiL8lrUiE+OSEt8o60SYdwckO+kHvyzfnkfHW+Oz9WrVvOeuaQ/CPn529DFayf</latexit><latexit sha1_base64="nKGHRyDYykkoHfjg5UdytxCiaVE=">AAACOHicbZBNSyNBEIZ7/Hb8iu7RS7NB8GKYUVEDCsJe9qbCRoVMCD2dStKkp2forgkJw/wsL/4Mb8tePLiIV3+BPXEQv15oeHiriq56w0QKg57315manpmdm19YdJeWV1bXKusblyZONYcGj2Wsr0NmQAoFDRQo4TrRwKJQwlU4+FXUr4agjYjVHxwn0IpYT4mu4Ayt1a6cDdtZgDDCDPuQ5yduEEJPqCyJGGoxyl2vtru3T4OggIODEvbqBex4Nb9edwNQnbf+dqXq1byJ6FfwS6iSUuftyl3QiXkagUIumTFN30uwlTGNgkvI3SA1kDA+YD1oWlQsAtPKJofndMs6HdqNtX0K6cR9P5GxyJhxFNpOu1/ffK4V5ne1Zordo1YmVJIiKP76UTeVFGNapEg7QgNHObbAuBZ2V8r7TDOONmvXhuB/PvkrXO7WfJvghVc9PS7jWCCb5CfZJj45JKfkNzknDcLJDflHHsh/59a5dx6dp9fWKaec+UE+yHl+AUjqqOo=</latexit>

vlanguage =

0

BB@

0.290�0.4410.7620.982

1

CCA

<latexit sha1_base64="b4xc+Okp2p3fvc/1+aow+HYTGjA=">AAACPXicbZBNaxsxEIZn03y42yZ1m2MvIibQSxetCU0CKRh66aWQktgJeI3RymNHRKtdpNkQs+wf66X/obfecsmhpfTaa2U7hHy9IHh4ZwbNvGmhlSPOfwZLz5ZXVtcaz8MXL9c3XjVfv+m5vLQSuzLXuT1NhUOtDHZJkcbTwqLIUo0n6fmnWf3kAq1TuTmmaYGDTEyMGispyFvD5vHFsEoIL6nSwkxKMcG6/hgmKU6UqYpMkFWXdcij9j5nSRK+59HOTjwjHu1+aC9gf68dJmhGt/3DZotHfC72GOIbaHXYl6MrADgcNn8ko1yWGRqSWjjXj3lBg0pYUlJjHSalw0LIc79e36MRGbpBNb++ZtveGbFxbv0zxObu3YlKZM5Ns9R3+v3O3MPazHyq1i9pvDeolClKQiMXH41LzShnsyjZSFmUpKcehLTK78rkmbBCkg889CHED09+DL12FPMo/urTOICFGvAWtuAdxLALHfgMh9AFCd/gCn7B7+B7cB38Cf4uWpeCm5lNuKfg3386Saz9</latexit><latexit sha1_base64="hZMi7uSxdZH/uZAh+zd/W8RR0ZM=">AAACPXicbZBNSxxBEIZ7jPFjNLqao5cmInhx6FnED1AQAiEXQYmrws6w9PTWrs329AzdNcsuw/yxXPIfvHnLJYeI6NGrvbsiifpCw8NbVXTVm+RKWmTsxpv6MP1xZnZu3l9Y/LS0XFtZPbdZYQQ0RKYyc5lwC0pqaKBEBZe5AZ4mCi6S3tdR/aIPxspMn+EwhzjlXS07UnB0Vqt21m+VEcIAS8V1t+BdqKpDP0qgK3WZpxyNHFQ+C+r7jEaRv8WC7e1wRCzY3alPYH+v7keg2y/9rdo6C9hY9C2Ez7B+RI9/PPTibyet2nXUzkSRgkahuLXNkOUYl9ygFAoqPyos5Fz03HpNh5qnYONyfH1FN5zTpp3MuKeRjt1/J0qeWjtME9fp9ruyr2sj871as8DOXlxKnRcIWkw+6hSKYkZHUdK2NCBQDR1wYaTblYorbrhAF7jvQghfn/wWzutByILw1KVxQCaaI2vkC9kkIdklR+Q7OSENIshP8pv8JbfeL++Pd+fdT1qnvOeZz+Q/eY9Pt1muGg==</latexit><latexit sha1_base64="hZMi7uSxdZH/uZAh+zd/W8RR0ZM=">AAACPXicbZBNSxxBEIZ7jPFjNLqao5cmInhx6FnED1AQAiEXQYmrws6w9PTWrs329AzdNcsuw/yxXPIfvHnLJYeI6NGrvbsiifpCw8NbVXTVm+RKWmTsxpv6MP1xZnZu3l9Y/LS0XFtZPbdZYQQ0RKYyc5lwC0pqaKBEBZe5AZ4mCi6S3tdR/aIPxspMn+EwhzjlXS07UnB0Vqt21m+VEcIAS8V1t+BdqKpDP0qgK3WZpxyNHFQ+C+r7jEaRv8WC7e1wRCzY3alPYH+v7keg2y/9rdo6C9hY9C2Ez7B+RI9/PPTibyet2nXUzkSRgkahuLXNkOUYl9ygFAoqPyos5Fz03HpNh5qnYONyfH1FN5zTpp3MuKeRjt1/J0qeWjtME9fp9ruyr2sj871as8DOXlxKnRcIWkw+6hSKYkZHUdK2NCBQDR1wYaTblYorbrhAF7jvQghfn/wWzutByILw1KVxQCaaI2vkC9kkIdklR+Q7OSENIshP8pv8JbfeL++Pd+fdT1qnvOeZz+Q/eY9Pt1muGg==</latexit><latexit sha1_base64="QL8ADr7tR9srk1Wpqs+QUhsdqR4=">AAACPXicbZBLSywxEIXT6r1q672OunQTHAQ3Nulh8AEKghuXCo4K08OQztSMwXS6SarFoek/5sb/4M6dGxeKuHVr5oH4OhD4OFVFqk6cKWmRsXtvYnLqz9/pmVl/bv7f/4XK4tKpTXMjoCFSlZrzmFtQUkMDJSo4zwzwJFZwFl8eDOpnV2CsTPUJ9jNoJbynZVcKjs5qV06u2kWEcI2F4rqX8x6U5Z4fxdCTusgSjkZelz4LajuMRpG/wYJ6PRwQC7Y2ayPY2a75EejOR3+7UmUBG4r+hHAMVTLWUbtyF3VSkSegUShubTNkGbYKblAKBaUf5RYyLi7dek2HmidgW8Xw+pKuOadDu6lxTyMdup8nCp5Y209i1+n2u7DfawPzt1ozx+52q5A6yxG0GH3UzRXFlA6ipB1pQKDqO+DCSLcrFRfccIEucN+FEH4/+Sec1oKQBeExq+7vjuOYIStklayTkGyRfXJIjkiDCHJDHsgTefZuvUfvxXsdtU5445ll8kXe2zuyz6sd</latexit>

f : V ! Rd<latexit sha1_base64="v4LU3fRnmQUVApJPSDxJstTKueI=">AAACBnicbVDLSsNAFJ3UV62vqEsRBovgqiQiKK6KblxWsQ9oYplMJu3QyUyYmSgldOXGX3HjQhG3foM7/8ZJm4W2HrhwOOde7r0nSBhV2nG+rdLC4tLySnm1sra+sbllb++0lEglJk0smJCdACnCKCdNTTUjnUQSFAeMtIPhZe6374lUVPBbPUqIH6M+pxHFSBupZ+9H57AFPUn7A42kFA/Qi5EeBEF2M74Le3bVqTkTwHniFqQKCjR69pcXCpzGhGvMkFJd10m0nyGpKWZkXPFSRRKEh6hPuoZyFBPlZ5M3xvDQKCGMhDTFNZyovycyFCs1igPTmd+oZr1c/M/rpjo68zPKk1QTjqeLopRBLWCeCQypJFizkSEIS2puhXiAJMLaJFcxIbizL8+T1nHNdWru9Um1flHEUQZ74AAcARecgjq4Ag3QBBg8gmfwCt6sJ+vFerc+pq0lq5jZBX9gff4AqouYng==</latexit><latexit sha1_base64="v4LU3fRnmQUVApJPSDxJstTKueI=">AAACBnicbVDLSsNAFJ3UV62vqEsRBovgqiQiKK6KblxWsQ9oYplMJu3QyUyYmSgldOXGX3HjQhG3foM7/8ZJm4W2HrhwOOde7r0nSBhV2nG+rdLC4tLySnm1sra+sbllb++0lEglJk0smJCdACnCKCdNTTUjnUQSFAeMtIPhZe6374lUVPBbPUqIH6M+pxHFSBupZ+9H57AFPUn7A42kFA/Qi5EeBEF2M74Le3bVqTkTwHniFqQKCjR69pcXCpzGhGvMkFJd10m0nyGpKWZkXPFSRRKEh6hPuoZyFBPlZ5M3xvDQKCGMhDTFNZyovycyFCs1igPTmd+oZr1c/M/rpjo68zPKk1QTjqeLopRBLWCeCQypJFizkSEIS2puhXiAJMLaJFcxIbizL8+T1nHNdWru9Um1flHEUQZ74AAcARecgjq4Ag3QBBg8gmfwCt6sJ+vFerc+pq0lq5jZBX9gff4AqouYng==</latexit><latexit sha1_base64="v4LU3fRnmQUVApJPSDxJstTKueI=">AAACBnicbVDLSsNAFJ3UV62vqEsRBovgqiQiKK6KblxWsQ9oYplMJu3QyUyYmSgldOXGX3HjQhG3foM7/8ZJm4W2HrhwOOde7r0nSBhV2nG+rdLC4tLySnm1sra+sbllb++0lEglJk0smJCdACnCKCdNTTUjnUQSFAeMtIPhZe6374lUVPBbPUqIH6M+pxHFSBupZ+9H57AFPUn7A42kFA/Qi5EeBEF2M74Le3bVqTkTwHniFqQKCjR69pcXCpzGhGvMkFJd10m0nyGpKWZkXPFSRRKEh6hPuoZyFBPlZ5M3xvDQKCGMhDTFNZyovycyFCs1igPTmd+oZr1c/M/rpjo68zPKk1QTjqeLopRBLWCeCQypJFizkSEIS2puhXiAJMLaJFcxIbizL8+T1nHNdWru9Um1flHEUQZ74AAcARecgjq4Ag3QBBg8gmfwCt6sJ+vFerc+pq0lq5jZBX9gff4AqouYng==</latexit><latexit sha1_base64="v4LU3fRnmQUVApJPSDxJstTKueI=">AAACBnicbVDLSsNAFJ3UV62vqEsRBovgqiQiKK6KblxWsQ9oYplMJu3QyUyYmSgldOXGX3HjQhG3foM7/8ZJm4W2HrhwOOde7r0nSBhV2nG+rdLC4tLySnm1sra+sbllb++0lEglJk0smJCdACnCKCdNTTUjnUQSFAeMtIPhZe6374lUVPBbPUqIH6M+pxHFSBupZ+9H57AFPUn7A42kFA/Qi5EeBEF2M74Le3bVqTkTwHniFqQKCjR69pcXCpzGhGvMkFJd10m0nyGpKWZkXPFSRRKEh6hPuoZyFBPlZ5M3xvDQKCGMhDTFNZyovycyFCs1igPTmd+oZr1c/M/rpjo68zPKk1QTjqeLopRBLWCeCQypJFizkSEIS2puhXiAJMLaJFcxIbizL8+T1nHNdWru9Um1flHEUQZ74AAcARecgjq4Ag3QBBg8gmfwCt6sJ+vFerc+pq0lq5jZBX9gff4AqouYng==</latexit>

Page 15: Word Embeddings - Princeton University · Word2vec • Input: a large text corpora, V, d • Output: • V: a pre-defined vocabulary • d: dimension of word vectors (e.g. 300) •

Word2vec

word = “sweden”

Page 16: Word Embeddings - Princeton University · Word2vec • Input: a large text corpora, V, d • Output: • V: a pre-defined vocabulary • d: dimension of word vectors (e.g. 300) •

Word2vec

Continuous Bag of Words (CBOW) Skip-grams

Page 17: Word Embeddings - Princeton University · Word2vec • Input: a large text corpora, V, d • Output: • V: a pre-defined vocabulary • d: dimension of word vectors (e.g. 300) •

Skip-gram

• The idea: we want to use words to predict their context words • Context: a fixed window of size 2m

Page 18: Word Embeddings - Princeton University · Word2vec • Input: a large text corpora, V, d • Output: • V: a pre-defined vocabulary • d: dimension of word vectors (e.g. 300) •

Skip-gram

Page 19: Word Embeddings - Princeton University · Word2vec • Input: a large text corpora, V, d • Output: • V: a pre-defined vocabulary • d: dimension of word vectors (e.g. 300) •

Skip-gram: objective function

• For each position ! , predict context words within context size m, given center word ! :

t = 1,2,…Twj

L(✓) =TY

t=1

Y

�mjm,j 6=0

P (wt+j | wt; ✓)

<latexit sha1_base64="eVY3k2h9oFNi4RVzO+rdBhuftGs=">AAACTXicbZFNaxsxEIa1zreTJm567EXUBBySmt1SaCEEQnrpoQcX4iTgdRatdhwrkbRbaTbBLPsHcyn01n/RSw4tJURr7yFfA0KP3pnRx6s4k8Ki7//2GnPzC4tLyyvN1bVX6xut15vHNs0Nhz5PZWpOY2ZBCg19FCjhNDPAVCzhJL78UuVPrsBYkeojnGQwVOxci5HgDJ0UtZJQMRxzJotvZSfEMSDbpvs0zEyaRAXuB+VZcVTWy/eKhhJ+0IvZpHYr0o78suh1rl39zkVJQyUSeh3hHq33K6NW2+/606DPIaihTeroRa1fYZLyXIFGLpm1g8DPcFgwg4JLKJthbiFj/JKdw8ChZgrssJi6UdItpyR0lBo3NNKp+rCjYMraiYpdZfV2+zRXiS/lBjmOPg8LobMcQfPZQaNcUkxpZS1NhAGOcuKAcSPcXSkfM8M4ug9oOhOCp09+DscfuoHfDb5/bB8c1nYsk7fkHemQgHwiB+Qr6ZE+4eSG/CF/yT/vp3fr/ffuZqUNr+55Qx5FY+keuray+g==</latexit><latexit sha1_base64="eVY3k2h9oFNi4RVzO+rdBhuftGs=">AAACTXicbZFNaxsxEIa1zreTJm567EXUBBySmt1SaCEEQnrpoQcX4iTgdRatdhwrkbRbaTbBLPsHcyn01n/RSw4tJURr7yFfA0KP3pnRx6s4k8Ki7//2GnPzC4tLyyvN1bVX6xut15vHNs0Nhz5PZWpOY2ZBCg19FCjhNDPAVCzhJL78UuVPrsBYkeojnGQwVOxci5HgDJ0UtZJQMRxzJotvZSfEMSDbpvs0zEyaRAXuB+VZcVTWy/eKhhJ+0IvZpHYr0o78suh1rl39zkVJQyUSeh3hHq33K6NW2+/606DPIaihTeroRa1fYZLyXIFGLpm1g8DPcFgwg4JLKJthbiFj/JKdw8ChZgrssJi6UdItpyR0lBo3NNKp+rCjYMraiYpdZfV2+zRXiS/lBjmOPg8LobMcQfPZQaNcUkxpZS1NhAGOcuKAcSPcXSkfM8M4ug9oOhOCp09+DscfuoHfDb5/bB8c1nYsk7fkHemQgHwiB+Qr6ZE+4eSG/CF/yT/vp3fr/ffuZqUNr+55Qx5FY+keuray+g==</latexit><latexit sha1_base64="eVY3k2h9oFNi4RVzO+rdBhuftGs=">AAACTXicbZFNaxsxEIa1zreTJm567EXUBBySmt1SaCEEQnrpoQcX4iTgdRatdhwrkbRbaTbBLPsHcyn01n/RSw4tJURr7yFfA0KP3pnRx6s4k8Ki7//2GnPzC4tLyyvN1bVX6xut15vHNs0Nhz5PZWpOY2ZBCg19FCjhNDPAVCzhJL78UuVPrsBYkeojnGQwVOxci5HgDJ0UtZJQMRxzJotvZSfEMSDbpvs0zEyaRAXuB+VZcVTWy/eKhhJ+0IvZpHYr0o78suh1rl39zkVJQyUSeh3hHq33K6NW2+/606DPIaihTeroRa1fYZLyXIFGLpm1g8DPcFgwg4JLKJthbiFj/JKdw8ChZgrssJi6UdItpyR0lBo3NNKp+rCjYMraiYpdZfV2+zRXiS/lBjmOPg8LobMcQfPZQaNcUkxpZS1NhAGOcuKAcSPcXSkfM8M4ug9oOhOCp09+DscfuoHfDb5/bB8c1nYsk7fkHemQgHwiB+Qr6ZE+4eSG/CF/yT/vp3fr/ffuZqUNr+55Qx5FY+keuray+g==</latexit><latexit sha1_base64="eVY3k2h9oFNi4RVzO+rdBhuftGs=">AAACTXicbZFNaxsxEIa1zreTJm567EXUBBySmt1SaCEEQnrpoQcX4iTgdRatdhwrkbRbaTbBLPsHcyn01n/RSw4tJURr7yFfA0KP3pnRx6s4k8Ki7//2GnPzC4tLyyvN1bVX6xut15vHNs0Nhz5PZWpOY2ZBCg19FCjhNDPAVCzhJL78UuVPrsBYkeojnGQwVOxci5HgDJ0UtZJQMRxzJotvZSfEMSDbpvs0zEyaRAXuB+VZcVTWy/eKhhJ+0IvZpHYr0o78suh1rl39zkVJQyUSeh3hHq33K6NW2+/606DPIaihTeroRa1fYZLyXIFGLpm1g8DPcFgwg4JLKJthbiFj/JKdw8ChZgrssJi6UdItpyR0lBo3NNKp+rCjYMraiYpdZfV2+zRXiS/lBjmOPg8LobMcQfPZQaNcUkxpZS1NhAGOcuKAcSPcXSkfM8M4ug9oOhOCp09+DscfuoHfDb5/bB8c1nYsk7fkHemQgHwiB+Qr6ZE+4eSG/CF/yT/vp3fr/ffuZqUNr+55Qx5FY+keuray+g==</latexit>

all the parameters to be optimized

• The objective function ! is the (average) negative log likelihood:J(θ)

J(✓) = � 1

TlogL(✓) = � 1

T

TX

t=1

X

�mjm,j 6=0

logP (wt+j | wt; ✓)

<latexit sha1_base64="23utKwn7ZJE6urpMOKPMcw5eqOk=">AAACe3icdVFdaxQxFM2MWuuq7aqPggQXca3tMiOKghSKvoj4sEK3Lexsl0z2zm7aJDMmd5Ql5E/403zzn/gimNkdRFu9EHLuuV/JuXklhcUk+R7FV65e27i+eaNz89btre3unbtHtqwNhxEvZWlOcmZBCg0jFCjhpDLAVC7hOD9/28SPP4OxotSHuKxgothci0JwhoGadr++72e4AGRP6D7dywrDuEu9O/SZLOc0UwwXnEn3wf8vzdZq6nA/9ae/vT1FMwmf6Nn6UrsN0gEl3q3aDvtfQs3TMx8GiBltHP+atgP8tNtLBsnK6GWQtqBHWhtOu9+yWclrBRq5ZNaO06TCiWMGBZfgO1ltoWL8nM1hHKBmCuzErbTz9FFgZrQoTTga6Yr9s8IxZe1S5SGzEcNejDXkv2LjGotXEyd0VSNovh5U1JJiSZtF0JkwwFEuA2DciPBWyhcsCIthXZ0gQnrxy5fB0bNBmgzSj897B29aOTbJffKQ9ElKXpID8o4MyYhw8iN6ED2O+tHPuBfvxLvr1Dhqa+6Rvyx+8QtA7b8+</latexit><latexit sha1_base64="23utKwn7ZJE6urpMOKPMcw5eqOk=">AAACe3icdVFdaxQxFM2MWuuq7aqPggQXca3tMiOKghSKvoj4sEK3Lexsl0z2zm7aJDMmd5Ql5E/403zzn/gimNkdRFu9EHLuuV/JuXklhcUk+R7FV65e27i+eaNz89btre3unbtHtqwNhxEvZWlOcmZBCg0jFCjhpDLAVC7hOD9/28SPP4OxotSHuKxgothci0JwhoGadr++72e4AGRP6D7dywrDuEu9O/SZLOc0UwwXnEn3wf8vzdZq6nA/9ae/vT1FMwmf6Nn6UrsN0gEl3q3aDvtfQs3TMx8GiBltHP+atgP8tNtLBsnK6GWQtqBHWhtOu9+yWclrBRq5ZNaO06TCiWMGBZfgO1ltoWL8nM1hHKBmCuzErbTz9FFgZrQoTTga6Yr9s8IxZe1S5SGzEcNejDXkv2LjGotXEyd0VSNovh5U1JJiSZtF0JkwwFEuA2DciPBWyhcsCIthXZ0gQnrxy5fB0bNBmgzSj897B29aOTbJffKQ9ElKXpID8o4MyYhw8iN6ED2O+tHPuBfvxLvr1Dhqa+6Rvyx+8QtA7b8+</latexit><latexit sha1_base64="23utKwn7ZJE6urpMOKPMcw5eqOk=">AAACe3icdVFdaxQxFM2MWuuq7aqPggQXca3tMiOKghSKvoj4sEK3Lexsl0z2zm7aJDMmd5Ql5E/403zzn/gimNkdRFu9EHLuuV/JuXklhcUk+R7FV65e27i+eaNz89btre3unbtHtqwNhxEvZWlOcmZBCg0jFCjhpDLAVC7hOD9/28SPP4OxotSHuKxgothci0JwhoGadr++72e4AGRP6D7dywrDuEu9O/SZLOc0UwwXnEn3wf8vzdZq6nA/9ae/vT1FMwmf6Nn6UrsN0gEl3q3aDvtfQs3TMx8GiBltHP+atgP8tNtLBsnK6GWQtqBHWhtOu9+yWclrBRq5ZNaO06TCiWMGBZfgO1ltoWL8nM1hHKBmCuzErbTz9FFgZrQoTTga6Yr9s8IxZe1S5SGzEcNejDXkv2LjGotXEyd0VSNovh5U1JJiSZtF0JkwwFEuA2DciPBWyhcsCIthXZ0gQnrxy5fB0bNBmgzSj897B29aOTbJffKQ9ElKXpID8o4MyYhw8iN6ED2O+tHPuBfvxLvr1Dhqa+6Rvyx+8QtA7b8+</latexit><latexit sha1_base64="23utKwn7ZJE6urpMOKPMcw5eqOk=">AAACe3icdVFdaxQxFM2MWuuq7aqPggQXca3tMiOKghSKvoj4sEK3Lexsl0z2zm7aJDMmd5Ql5E/403zzn/gimNkdRFu9EHLuuV/JuXklhcUk+R7FV65e27i+eaNz89btre3unbtHtqwNhxEvZWlOcmZBCg0jFCjhpDLAVC7hOD9/28SPP4OxotSHuKxgothci0JwhoGadr++72e4AGRP6D7dywrDuEu9O/SZLOc0UwwXnEn3wf8vzdZq6nA/9ae/vT1FMwmf6Nn6UrsN0gEl3q3aDvtfQs3TMx8GiBltHP+atgP8tNtLBsnK6GWQtqBHWhtOu9+yWclrBRq5ZNaO06TCiWMGBZfgO1ltoWL8nM1hHKBmCuzErbTz9FFgZrQoTTga6Yr9s8IxZe1S5SGzEcNejDXkv2LjGotXEyd0VSNovh5U1JJiSZtF0JkwwFEuA2DciPBWyhcsCIthXZ0gQnrxy5fB0bNBmgzSj897B29aOTbJffKQ9ElKXpID8o4MyYhw8iN6ED2O+tHPuBfvxLvr1Dhqa+6Rvyx+8QtA7b8+</latexit>

Page 20: Word Embeddings - Princeton University · Word2vec • Input: a large text corpora, V, d • Output: • V: a pre-defined vocabulary • d: dimension of word vectors (e.g. 300) •

How to define ?P(wt+j ∣ wt; θ)

• We have two sets of vectors for each word in the vocabulary

ui 2 Rd<latexit sha1_base64="Qsgo7bHXmdt/5AAiowyqkJ/9E+0=">AAACBnicbVBNS8NAEJ3Ur1q/oh5FWCyCp5KIoMeiF49V7Ae0MWy2m3bpZhN2N0IJOXnxr3jxoIhXf4M3/42btgdtfTDweG+GmXlBwpnSjvNtlZaWV1bXyuuVjc2t7R17d6+l4lQS2iQxj2UnwIpyJmhTM81pJ5EURwGn7WB0VfjtByoVi8WdHifUi/BAsJARrI3k24e9COthEGZp7jPUYwJNhSC7ze/7vl11as4EaJG4M1KFGRq+/dXrxySNqNCEY6W6rpNoL8NSM8JpXumliiaYjPCAdg0VOKLKyyZv5OjYKH0UxtKU0Gii/p7IcKTUOApMZ3GjmvcK8T+vm+rwwsuYSFJNBZkuClOOdIyKTFCfSUo0HxuCiWTmVkSGWGKiTXIVE4I7//IiaZ3WXKfm3pxV65ezOMpwAEdwAi6cQx2uoQFNIPAIz/AKb9aT9WK9Wx/T1pI1m9mHP7A+fwB1FZkZ</latexit><latexit sha1_base64="Qsgo7bHXmdt/5AAiowyqkJ/9E+0=">AAACBnicbVBNS8NAEJ3Ur1q/oh5FWCyCp5KIoMeiF49V7Ae0MWy2m3bpZhN2N0IJOXnxr3jxoIhXf4M3/42btgdtfTDweG+GmXlBwpnSjvNtlZaWV1bXyuuVjc2t7R17d6+l4lQS2iQxj2UnwIpyJmhTM81pJ5EURwGn7WB0VfjtByoVi8WdHifUi/BAsJARrI3k24e9COthEGZp7jPUYwJNhSC7ze/7vl11as4EaJG4M1KFGRq+/dXrxySNqNCEY6W6rpNoL8NSM8JpXumliiaYjPCAdg0VOKLKyyZv5OjYKH0UxtKU0Gii/p7IcKTUOApMZ3GjmvcK8T+vm+rwwsuYSFJNBZkuClOOdIyKTFCfSUo0HxuCiWTmVkSGWGKiTXIVE4I7//IiaZ3WXKfm3pxV65ezOMpwAEdwAi6cQx2uoQFNIPAIz/AKb9aT9WK9Wx/T1pI1m9mHP7A+fwB1FZkZ</latexit><latexit sha1_base64="Qsgo7bHXmdt/5AAiowyqkJ/9E+0=">AAACBnicbVBNS8NAEJ3Ur1q/oh5FWCyCp5KIoMeiF49V7Ae0MWy2m3bpZhN2N0IJOXnxr3jxoIhXf4M3/42btgdtfTDweG+GmXlBwpnSjvNtlZaWV1bXyuuVjc2t7R17d6+l4lQS2iQxj2UnwIpyJmhTM81pJ5EURwGn7WB0VfjtByoVi8WdHifUi/BAsJARrI3k24e9COthEGZp7jPUYwJNhSC7ze/7vl11as4EaJG4M1KFGRq+/dXrxySNqNCEY6W6rpNoL8NSM8JpXumliiaYjPCAdg0VOKLKyyZv5OjYKH0UxtKU0Gii/p7IcKTUOApMZ3GjmvcK8T+vm+rwwsuYSFJNBZkuClOOdIyKTFCfSUo0HxuCiWTmVkSGWGKiTXIVE4I7//IiaZ3WXKfm3pxV65ezOMpwAEdwAi6cQx2uoQFNIPAIz/AKb9aT9WK9Wx/T1pI1m9mHP7A+fwB1FZkZ</latexit><latexit sha1_base64="Qsgo7bHXmdt/5AAiowyqkJ/9E+0=">AAACBnicbVBNS8NAEJ3Ur1q/oh5FWCyCp5KIoMeiF49V7Ae0MWy2m3bpZhN2N0IJOXnxr3jxoIhXf4M3/42btgdtfTDweG+GmXlBwpnSjvNtlZaWV1bXyuuVjc2t7R17d6+l4lQS2iQxj2UnwIpyJmhTM81pJ5EURwGn7WB0VfjtByoVi8WdHifUi/BAsJARrI3k24e9COthEGZp7jPUYwJNhSC7ze/7vl11as4EaJG4M1KFGRq+/dXrxySNqNCEY6W6rpNoL8NSM8JpXumliiaYjPCAdg0VOKLKyyZv5OjYKH0UxtKU0Gii/p7IcKTUOApMZ3GjmvcK8T+vm+rwwsuYSFJNBZkuClOOdIyKTFCfSUo0HxuCiWTmVkSGWGKiTXIVE4I7//IiaZ3WXKfm3pxV65ezOMpwAEdwAi6cQx2uoQFNIPAIz/AKb9aT9WK9Wx/T1pI1m9mHP7A+fwB1FZkZ</latexit>

: embedding for target word i

: embedding for context word i’

Q: Why two sets of vectors?

vi0 2 Rd<latexit sha1_base64="jlnCkKyjEgmmzyrWVfCH8VFvPB4=">AAACCXicbVBNS8NAEJ34WetX1KOXxSJ6KokIeix68VjFfkAby2a7aZduNmF3UyghVy/+FS8eFPHqP/Dmv3HT5qCtDwYe780wM8+POVPacb6tpeWV1bX10kZ5c2t7Z9fe22+qKJGENkjEI9n2saKcCdrQTHPajiXFoc9pyx9d535rTKVikbjXk5h6IR4IFjCCtZF6NuqGWA/9IB1nvZSdZKjLRKH56V320O/ZFafqTIEWiVuQChSo9+yvbj8iSUiFJhwr1XGdWHsplpoRTrNyN1E0xmSEB7RjqMAhVV46/SRDx0bpoyCSpoRGU/X3RIpDpSahbzrzG9W8l4v/eZ1EB5deykScaCrIbFGQcKQjlMeC+kxSovnEEEwkM7ciMsQSE23CK5sQ3PmXF0nzrOo6Vff2vFK7KuIowSEcwSm4cAE1uIE6NIDAIzzDK7xZT9aL9W59zFqXrGLmAP7A+vwBvMiaVw==</latexit><latexit sha1_base64="jlnCkKyjEgmmzyrWVfCH8VFvPB4=">AAACCXicbVBNS8NAEJ34WetX1KOXxSJ6KokIeix68VjFfkAby2a7aZduNmF3UyghVy/+FS8eFPHqP/Dmv3HT5qCtDwYe780wM8+POVPacb6tpeWV1bX10kZ5c2t7Z9fe22+qKJGENkjEI9n2saKcCdrQTHPajiXFoc9pyx9d535rTKVikbjXk5h6IR4IFjCCtZF6NuqGWA/9IB1nvZSdZKjLRKH56V320O/ZFafqTIEWiVuQChSo9+yvbj8iSUiFJhwr1XGdWHsplpoRTrNyN1E0xmSEB7RjqMAhVV46/SRDx0bpoyCSpoRGU/X3RIpDpSahbzrzG9W8l4v/eZ1EB5deykScaCrIbFGQcKQjlMeC+kxSovnEEEwkM7ciMsQSE23CK5sQ3PmXF0nzrOo6Vff2vFK7KuIowSEcwSm4cAE1uIE6NIDAIzzDK7xZT9aL9W59zFqXrGLmAP7A+vwBvMiaVw==</latexit><latexit sha1_base64="jlnCkKyjEgmmzyrWVfCH8VFvPB4=">AAACCXicbVBNS8NAEJ34WetX1KOXxSJ6KokIeix68VjFfkAby2a7aZduNmF3UyghVy/+FS8eFPHqP/Dmv3HT5qCtDwYe780wM8+POVPacb6tpeWV1bX10kZ5c2t7Z9fe22+qKJGENkjEI9n2saKcCdrQTHPajiXFoc9pyx9d535rTKVikbjXk5h6IR4IFjCCtZF6NuqGWA/9IB1nvZSdZKjLRKH56V320O/ZFafqTIEWiVuQChSo9+yvbj8iSUiFJhwr1XGdWHsplpoRTrNyN1E0xmSEB7RjqMAhVV46/SRDx0bpoyCSpoRGU/X3RIpDpSahbzrzG9W8l4v/eZ1EB5deykScaCrIbFGQcKQjlMeC+kxSovnEEEwkM7ciMsQSE23CK5sQ3PmXF0nzrOo6Vff2vFK7KuIowSEcwSm4cAE1uIE6NIDAIzzDK7xZT9aL9W59zFqXrGLmAP7A+vwBvMiaVw==</latexit><latexit sha1_base64="jlnCkKyjEgmmzyrWVfCH8VFvPB4=">AAACCXicbVBNS8NAEJ34WetX1KOXxSJ6KokIeix68VjFfkAby2a7aZduNmF3UyghVy/+FS8eFPHqP/Dmv3HT5qCtDwYe780wM8+POVPacb6tpeWV1bX10kZ5c2t7Z9fe22+qKJGENkjEI9n2saKcCdrQTHPajiXFoc9pyx9d535rTKVikbjXk5h6IR4IFjCCtZF6NuqGWA/9IB1nvZSdZKjLRKH56V320O/ZFafqTIEWiVuQChSo9+yvbj8iSUiFJhwr1XGdWHsplpoRTrNyN1E0xmSEB7RjqMAhVV46/SRDx0bpoyCSpoRGU/X3RIpDpSahbzrzG9W8l4v/eZ1EB5deykScaCrIbFGQcKQjlMeC+kxSovnEEEwkM7ciMsQSE23CK5sQ3PmXF0nzrOo6Vff2vFK7KuIowSEcwSm4cAE1uIE6NIDAIzzDK7xZT9aL9W59zFqXrGLmAP7A+vwBvMiaVw==</latexit>

• Use inner product to measure how likely word i appears with context word i’, the larger the better

“softmax” we learned last time!

ui · vi0<latexit sha1_base64="RzTZ0bVG1tX3m7GXesoGab/HjRI=">AAACC3icbVBNS8NAEN3Ur1q/oh69LC2ip5KIoMeiF48VbCu0IWw2m3bpZjfsbgol5O7Fv+LFgyJe/QPe/Ddu2gja+mDg8d4MM/OChFGlHefLqqysrq1vVDdrW9s7u3v2/kFXiVRi0sGCCXkfIEUY5aSjqWbkPpEExQEjvWB8Xfi9CZGKCn6npwnxYjTkNKIYaSP5dn0QIz0KoizNfQoHOBQa/kiT3M/oSe7bDafpzACXiVuSBijR9u3PQShwGhOuMUNK9V0n0V6GpKaYkbw2SBVJEB6jIekbylFMlJfNfsnhsVFCGAlpims4U39PZChWahoHprM4Uy16hfif1091dOlllCepJhzPF0Upg1rAIhgYUkmwZlNDEJbU3ArxCEmEtYmvZkJwF19eJt2zpus03dvzRuuqjKMKjkAdnAIXXIAWuAFt0AEYPIAn8AJerUfr2Xqz3uetFaucOQR/YH18A6ZPm2s=</latexit><latexit sha1_base64="RzTZ0bVG1tX3m7GXesoGab/HjRI=">AAACC3icbVBNS8NAEN3Ur1q/oh69LC2ip5KIoMeiF48VbCu0IWw2m3bpZjfsbgol5O7Fv+LFgyJe/QPe/Ddu2gja+mDg8d4MM/OChFGlHefLqqysrq1vVDdrW9s7u3v2/kFXiVRi0sGCCXkfIEUY5aSjqWbkPpEExQEjvWB8Xfi9CZGKCn6npwnxYjTkNKIYaSP5dn0QIz0KoizNfQoHOBQa/kiT3M/oSe7bDafpzACXiVuSBijR9u3PQShwGhOuMUNK9V0n0V6GpKaYkbw2SBVJEB6jIekbylFMlJfNfsnhsVFCGAlpims4U39PZChWahoHprM4Uy16hfif1091dOlllCepJhzPF0Upg1rAIhgYUkmwZlNDEJbU3ArxCEmEtYmvZkJwF19eJt2zpus03dvzRuuqjKMKjkAdnAIXXIAWuAFt0AEYPIAn8AJerUfr2Xqz3uetFaucOQR/YH18A6ZPm2s=</latexit><latexit sha1_base64="RzTZ0bVG1tX3m7GXesoGab/HjRI=">AAACC3icbVBNS8NAEN3Ur1q/oh69LC2ip5KIoMeiF48VbCu0IWw2m3bpZjfsbgol5O7Fv+LFgyJe/QPe/Ddu2gja+mDg8d4MM/OChFGlHefLqqysrq1vVDdrW9s7u3v2/kFXiVRi0sGCCXkfIEUY5aSjqWbkPpEExQEjvWB8Xfi9CZGKCn6npwnxYjTkNKIYaSP5dn0QIz0KoizNfQoHOBQa/kiT3M/oSe7bDafpzACXiVuSBijR9u3PQShwGhOuMUNK9V0n0V6GpKaYkbw2SBVJEB6jIekbylFMlJfNfsnhsVFCGAlpims4U39PZChWahoHprM4Uy16hfif1091dOlllCepJhzPF0Upg1rAIhgYUkmwZlNDEJbU3ArxCEmEtYmvZkJwF19eJt2zpus03dvzRuuqjKMKjkAdnAIXXIAWuAFt0AEYPIAn8AJerUfr2Xqz3uetFaucOQR/YH18A6ZPm2s=</latexit><latexit sha1_base64="RzTZ0bVG1tX3m7GXesoGab/HjRI=">AAACC3icbVBNS8NAEN3Ur1q/oh69LC2ip5KIoMeiF48VbCu0IWw2m3bpZjfsbgol5O7Fv+LFgyJe/QPe/Ddu2gja+mDg8d4MM/OChFGlHefLqqysrq1vVDdrW9s7u3v2/kFXiVRi0sGCCXkfIEUY5aSjqWbkPpEExQEjvWB8Xfi9CZGKCn6npwnxYjTkNKIYaSP5dn0QIz0KoizNfQoHOBQa/kiT3M/oSe7bDafpzACXiVuSBijR9u3PQShwGhOuMUNK9V0n0V6GpKaYkbw2SBVJEB6jIekbylFMlJfNfsnhsVFCGAlpims4U39PZChWahoHprM4Uy16hfif1091dOlllCepJhzPF0Upg1rAIhgYUkmwZlNDEJbU3ArxCEmEtYmvZkJwF19eJt2zpus03dvzRuuqjKMKjkAdnAIXXIAWuAFt0AEYPIAn8AJerUfr2Xqz3uetFaucOQR/YH18A6ZPm2s=</latexit>

P (wt+j | wt) =exp(uwt · vwt+j )Pk2V exp(uwt · vk)

<latexit sha1_base64="YxU1x4J5AlDT3J/Dp+p53Qpgi+U=">AAACcnicjVFda9swFJXdbc3SfaQdfdlg0xYGCRvBHoPupRC6lz1msKSFOBhZllstkmyk67ZB6Af07/Wtv6Iv+wFVPA+6dg87IDicc+/V1VFWCW4giq6CcOPBw0ebncfdrSdPnz3vbe/MTFlryqa0FKU+yohhgis2BQ6CHVWaEZkJdpgtv679w1OmDS/VD1hVbCHJseIFpwS8lPYuJoOz1MKHnw4nkuf4LIUh3sdJoQm1CTuvBokkcJIVtnap9a6vo3kJ+I982sjNBDd0NjG1TO0SJ1zhmfvfCcuhc2mvH42iBvg+iVvSRy0mae8yyUtaS6aACmLMPI4qWFiigVPBXDepDasIXZJjNvdUEcnMwjaROfzeKzkuSu2PAtyotzsskcasZOYr12uau95a/Jc3r6H4srBcVTUwRX9fVNQCQ4nX+eOca0ZBrDwhVHO/K6YnxKcN/pe6PoT47pPvk9mnURyN4u+f++ODNo4OeoXeoQGK0R4ao29ogqaIoutgN3gdvAl+hS/Dt2GbXRi0PS/QXwg/3gCAcL/l</latexit><latexit sha1_base64="YxU1x4J5AlDT3J/Dp+p53Qpgi+U=">AAACcnicjVFda9swFJXdbc3SfaQdfdlg0xYGCRvBHoPupRC6lz1msKSFOBhZllstkmyk67ZB6Af07/Wtv6Iv+wFVPA+6dg87IDicc+/V1VFWCW4giq6CcOPBw0ebncfdrSdPnz3vbe/MTFlryqa0FKU+yohhgis2BQ6CHVWaEZkJdpgtv679w1OmDS/VD1hVbCHJseIFpwS8lPYuJoOz1MKHnw4nkuf4LIUh3sdJoQm1CTuvBokkcJIVtnap9a6vo3kJ+I982sjNBDd0NjG1TO0SJ1zhmfvfCcuhc2mvH42iBvg+iVvSRy0mae8yyUtaS6aACmLMPI4qWFiigVPBXDepDasIXZJjNvdUEcnMwjaROfzeKzkuSu2PAtyotzsskcasZOYr12uau95a/Jc3r6H4srBcVTUwRX9fVNQCQ4nX+eOca0ZBrDwhVHO/K6YnxKcN/pe6PoT47pPvk9mnURyN4u+f++ODNo4OeoXeoQGK0R4ao29ogqaIoutgN3gdvAl+hS/Dt2GbXRi0PS/QXwg/3gCAcL/l</latexit><latexit sha1_base64="YxU1x4J5AlDT3J/Dp+p53Qpgi+U=">AAACcnicjVFda9swFJXdbc3SfaQdfdlg0xYGCRvBHoPupRC6lz1msKSFOBhZllstkmyk67ZB6Af07/Wtv6Iv+wFVPA+6dg87IDicc+/V1VFWCW4giq6CcOPBw0ebncfdrSdPnz3vbe/MTFlryqa0FKU+yohhgis2BQ6CHVWaEZkJdpgtv679w1OmDS/VD1hVbCHJseIFpwS8lPYuJoOz1MKHnw4nkuf4LIUh3sdJoQm1CTuvBokkcJIVtnap9a6vo3kJ+I982sjNBDd0NjG1TO0SJ1zhmfvfCcuhc2mvH42iBvg+iVvSRy0mae8yyUtaS6aACmLMPI4qWFiigVPBXDepDasIXZJjNvdUEcnMwjaROfzeKzkuSu2PAtyotzsskcasZOYr12uau95a/Jc3r6H4srBcVTUwRX9fVNQCQ4nX+eOca0ZBrDwhVHO/K6YnxKcN/pe6PoT47pPvk9mnURyN4u+f++ODNo4OeoXeoQGK0R4ao29ogqaIoutgN3gdvAl+hS/Dt2GbXRi0PS/QXwg/3gCAcL/l</latexit><latexit sha1_base64="hP+6LrUf2d3tZaldqaQQvEKMXyw=">AAAB2XicbZDNSgMxFIXv1L86Vq1rN8EiuCozbnQpuHFZwbZCO5RM5k4bmskMyR2hDH0BF25EfC93vo3pz0JbDwQ+zknIvSculLQUBN9ebWd3b/+gfugfNfzjk9Nmo2fz0gjsilzl5jnmFpXU2CVJCp8LgzyLFfbj6f0i77+gsTLXTzQrMMr4WMtUCk7O6oyaraAdLMW2IVxDC9YaNb+GSS7KDDUJxa0dhEFBUcUNSaFw7g9LiwUXUz7GgUPNM7RRtRxzzi6dk7A0N+5oYkv394uKZ9bOstjdzDhN7Ga2MP/LBiWlt1EldVESarH6KC0Vo5wtdmaJNChIzRxwYaSblYkJN1yQa8Z3HYSbG29D77odBu3wMYA6nMMFXEEIN3AHD9CBLghI4BXevYn35n2suqp569LO4I+8zx84xIo4</latexit><latexit sha1_base64="e+z+9fOroxs43UnXStWJ/Sb1g8o=">AAACZ3icjVFdSxwxFM1MrdXV2rXgiwUbuxR2EZaZvtQXQfDFxy10V2FnGTKZjMZNMkNyR11CfoB/zzd/hS/+ALPjCvXjoQcCh3Nu7r05ySrBDUTRXRB+WPq4/GlltbW2/nnjS3tzfWTKWlM2pKUo9WlGDBNcsSFwEOy00ozITLCTbHo0908umTa8VH9hVrGJJGeKF5wS8FLavhl0r1ILexcOJ5Ln+CqFHj7ASaEJtQm7rrqJJHCeFbZ2qfWur6N5CfhZvmzkpoPrOZuYWqZ2ihOu8Mj9b4dpz7m03Yn6UQP8lsQL0kELDNL2bZKXtJZMARXEmHEcVTCxRAOngrlWUhtWETolZ2zsqSKSmYltInP4p1dyXJTaHwW4Uf+9YYk0ZiYzXzlf07z25uJ73riGYn9iuapqYIo+DSpqgaHE8/xxzjWjIGaeEKq53xXTc+LTBv9LLR9C/PrJb8noVz+O+vGfCK2gb+gH6qIY/UaH6BgN0BBRdB9sBTvB9+Ah3A53n+IKg0VuX9ELhJ1HMpa/Bw==</latexit><latexit sha1_base64="e+z+9fOroxs43UnXStWJ/Sb1g8o=">AAACZ3icjVFdSxwxFM1MrdXV2rXgiwUbuxR2EZaZvtQXQfDFxy10V2FnGTKZjMZNMkNyR11CfoB/zzd/hS/+ALPjCvXjoQcCh3Nu7r05ySrBDUTRXRB+WPq4/GlltbW2/nnjS3tzfWTKWlM2pKUo9WlGDBNcsSFwEOy00ozITLCTbHo0908umTa8VH9hVrGJJGeKF5wS8FLavhl0r1ILexcOJ5Ln+CqFHj7ASaEJtQm7rrqJJHCeFbZ2qfWur6N5CfhZvmzkpoPrOZuYWqZ2ihOu8Mj9b4dpz7m03Yn6UQP8lsQL0kELDNL2bZKXtJZMARXEmHEcVTCxRAOngrlWUhtWETolZ2zsqSKSmYltInP4p1dyXJTaHwW4Uf+9YYk0ZiYzXzlf07z25uJ73riGYn9iuapqYIo+DSpqgaHE8/xxzjWjIGaeEKq53xXTc+LTBv9LLR9C/PrJb8noVz+O+vGfCK2gb+gH6qIY/UaH6BgN0BBRdB9sBTvB9+Ah3A53n+IKg0VuX9ELhJ1HMpa/Bw==</latexit><latexit sha1_base64="JwMv/sOfzaSR0cvFGLscrahs/c0=">AAACcnicjVHRTtswFHXCxlg3oGPihUmbtwqpFVOV7AVeJiH2ssdOWgtSU0WO44Cp7UT2DVBZ/gB+jze+ghc+YG7IpA32sCNZOjrn3uvr46wS3EAU3QbhyrPnqy/WXnZevV7f2Oy+2ZqYstaUjWkpSn2SEcMEV2wMHAQ7qTQjMhPsOJt/W/rHF0wbXqqfsKjYTJJTxQtOCXgp7V6P+pephb1zhxPJc3yZwgB/xUmhCbUJu6r6iSRwlhW2dqn1rq+jeQn4t3zRyM0EN3A2MbVM7RwnXOGJ+98J84FzabcXDaMG+CmJW9JDLUZp9ybJS1pLpoAKYsw0jiqYWaKBU8FcJ6kNqwidk1M29VQRyczMNpE5vOuVHBel9kcBbtQ/OyyRxixk5iuXa5rH3lL8lzetoTiYWa6qGpiiDxcVtcBQ4mX+OOeaURALTwjV3O+K6RnxaYP/pY4PIX785Kdk8mUYR8P4R9Q7PGrjWEPv0CfURzHaR4foOxqhMaLoLtgO3gcfgvtwJ/wYttmFQdvzFv2F8PMvfzC/4Q==</latexit><latexit sha1_base64="YxU1x4J5AlDT3J/Dp+p53Qpgi+U=">AAACcnicjVFda9swFJXdbc3SfaQdfdlg0xYGCRvBHoPupRC6lz1msKSFOBhZllstkmyk67ZB6Af07/Wtv6Iv+wFVPA+6dg87IDicc+/V1VFWCW4giq6CcOPBw0ebncfdrSdPnz3vbe/MTFlryqa0FKU+yohhgis2BQ6CHVWaEZkJdpgtv679w1OmDS/VD1hVbCHJseIFpwS8lPYuJoOz1MKHnw4nkuf4LIUh3sdJoQm1CTuvBokkcJIVtnap9a6vo3kJ+I982sjNBDd0NjG1TO0SJ1zhmfvfCcuhc2mvH42iBvg+iVvSRy0mae8yyUtaS6aACmLMPI4qWFiigVPBXDepDasIXZJjNvdUEcnMwjaROfzeKzkuSu2PAtyotzsskcasZOYr12uau95a/Jc3r6H4srBcVTUwRX9fVNQCQ4nX+eOca0ZBrDwhVHO/K6YnxKcN/pe6PoT47pPvk9mnURyN4u+f++ODNo4OeoXeoQGK0R4ao29ogqaIoutgN3gdvAl+hS/Dt2GbXRi0PS/QXwg/3gCAcL/l</latexit><latexit sha1_base64="YxU1x4J5AlDT3J/Dp+p53Qpgi+U=">AAACcnicjVFda9swFJXdbc3SfaQdfdlg0xYGCRvBHoPupRC6lz1msKSFOBhZllstkmyk67ZB6Af07/Wtv6Iv+wFVPA+6dg87IDicc+/V1VFWCW4giq6CcOPBw0ebncfdrSdPnz3vbe/MTFlryqa0FKU+yohhgis2BQ6CHVWaEZkJdpgtv679w1OmDS/VD1hVbCHJseIFpwS8lPYuJoOz1MKHnw4nkuf4LIUh3sdJoQm1CTuvBokkcJIVtnap9a6vo3kJ+I982sjNBDd0NjG1TO0SJ1zhmfvfCcuhc2mvH42iBvg+iVvSRy0mae8yyUtaS6aACmLMPI4qWFiigVPBXDepDasIXZJjNvdUEcnMwjaROfzeKzkuSu2PAtyotzsskcasZOYr12uau95a/Jc3r6H4srBcVTUwRX9fVNQCQ4nX+eOca0ZBrDwhVHO/K6YnxKcN/pe6PoT47pPvk9mnURyN4u+f++ODNo4OeoXeoQGK0R4ao29ogqaIoutgN3gdvAl+hS/Dt2GbXRi0PS/QXwg/3gCAcL/l</latexit><latexit sha1_base64="YxU1x4J5AlDT3J/Dp+p53Qpgi+U=">AAACcnicjVFda9swFJXdbc3SfaQdfdlg0xYGCRvBHoPupRC6lz1msKSFOBhZllstkmyk67ZB6Af07/Wtv6Iv+wFVPA+6dg87IDicc+/V1VFWCW4giq6CcOPBw0ebncfdrSdPnz3vbe/MTFlryqa0FKU+yohhgis2BQ6CHVWaEZkJdpgtv679w1OmDS/VD1hVbCHJseIFpwS8lPYuJoOz1MKHnw4nkuf4LIUh3sdJoQm1CTuvBokkcJIVtnap9a6vo3kJ+I982sjNBDd0NjG1TO0SJ1zhmfvfCcuhc2mvH42iBvg+iVvSRy0mae8yyUtaS6aACmLMPI4qWFiigVPBXDepDasIXZJjNvdUEcnMwjaROfzeKzkuSu2PAtyotzsskcasZOYr12uau95a/Jc3r6H4srBcVTUwRX9fVNQCQ4nX+eOca0ZBrDwhVHO/K6YnxKcN/pe6PoT47pPvk9mnURyN4u+f++ODNo4OeoXeoQGK0R4ao29ogqaIoutgN3gdvAl+hS/Dt2GbXRi0PS/QXwg/3gCAcL/l</latexit><latexit sha1_base64="YxU1x4J5AlDT3J/Dp+p53Qpgi+U=">AAACcnicjVFda9swFJXdbc3SfaQdfdlg0xYGCRvBHoPupRC6lz1msKSFOBhZllstkmyk67ZB6Af07/Wtv6Iv+wFVPA+6dg87IDicc+/V1VFWCW4giq6CcOPBw0ebncfdrSdPnz3vbe/MTFlryqa0FKU+yohhgis2BQ6CHVWaEZkJdpgtv679w1OmDS/VD1hVbCHJseIFpwS8lPYuJoOz1MKHnw4nkuf4LIUh3sdJoQm1CTuvBokkcJIVtnap9a6vo3kJ+I982sjNBDd0NjG1TO0SJ1zhmfvfCcuhc2mvH42iBvg+iVvSRy0mae8yyUtaS6aACmLMPI4qWFiigVPBXDepDasIXZJjNvdUEcnMwjaROfzeKzkuSu2PAtyotzsskcasZOYr12uau95a/Jc3r6H4srBcVTUwRX9fVNQCQ4nX+eOca0ZBrDwhVHO/K6YnxKcN/pe6PoT47pPvk9mnURyN4u+f++ODNo4OeoXeoQGK0R4ao29ogqaIoutgN3gdvAl+hS/Dt2GbXRi0PS/QXwg/3gCAcL/l</latexit><latexit sha1_base64="YxU1x4J5AlDT3J/Dp+p53Qpgi+U=">AAACcnicjVFda9swFJXdbc3SfaQdfdlg0xYGCRvBHoPupRC6lz1msKSFOBhZllstkmyk67ZB6Af07/Wtv6Iv+wFVPA+6dg87IDicc+/V1VFWCW4giq6CcOPBw0ebncfdrSdPnz3vbe/MTFlryqa0FKU+yohhgis2BQ6CHVWaEZkJdpgtv679w1OmDS/VD1hVbCHJseIFpwS8lPYuJoOz1MKHnw4nkuf4LIUh3sdJoQm1CTuvBokkcJIVtnap9a6vo3kJ+I982sjNBDd0NjG1TO0SJ1zhmfvfCcuhc2mvH42iBvg+iVvSRy0mae8yyUtaS6aACmLMPI4qWFiigVPBXDepDasIXZJjNvdUEcnMwjaROfzeKzkuSu2PAtyotzsskcasZOYr12uau95a/Jc3r6H4srBcVTUwRX9fVNQCQ4nX+eOca0ZBrDwhVHO/K6YnxKcN/pe6PoT47pPvk9mnURyN4u+f++ODNo4OeoXeoQGK0R4ao29ogqaIoutgN3gdvAl+hS/Dt2GbXRi0PS/QXwg/3gCAcL/l</latexit><latexit sha1_base64="YxU1x4J5AlDT3J/Dp+p53Qpgi+U=">AAACcnicjVFda9swFJXdbc3SfaQdfdlg0xYGCRvBHoPupRC6lz1msKSFOBhZllstkmyk67ZB6Af07/Wtv6Iv+wFVPA+6dg87IDicc+/V1VFWCW4giq6CcOPBw0ebncfdrSdPnz3vbe/MTFlryqa0FKU+yohhgis2BQ6CHVWaEZkJdpgtv679w1OmDS/VD1hVbCHJseIFpwS8lPYuJoOz1MKHnw4nkuf4LIUh3sdJoQm1CTuvBokkcJIVtnap9a6vo3kJ+I982sjNBDd0NjG1TO0SJ1zhmfvfCcuhc2mvH42iBvg+iVvSRy0mae8yyUtaS6aACmLMPI4qWFiigVPBXDepDasIXZJjNvdUEcnMwjaROfzeKzkuSu2PAtyotzsskcasZOYr12uau95a/Jc3r6H4srBcVTUwRX9fVNQCQ4nX+eOca0ZBrDwhVHO/K6YnxKcN/pe6PoT47pPvk9mnURyN4u+f++ODNo4OeoXeoQGK0R4ao29ogqaIoutgN3gdvAl+hS/Dt2GbXRi0PS/QXwg/3gCAcL/l</latexit>

are all the parameters in this model!✓ = {{uk}, {vk}}<latexit sha1_base64="uE6wEg+cbVDNn7T6D276YV5+N9k=">AAACGHicbVDLSsNAFJ3UV62vqks3g0VwITURQTdC0Y3LCvYBTQiT6aQdOnkwc1MoIZ/hxl9x40IRt935N07TgNp6hoHDOfdy7z1eLLgC0/wySiura+sb5c3K1vbO7l51/6CtokRS1qKRiGTXI4oJHrIWcBCsG0tGAk+wjje6m/mdMZOKR+EjTGLmBGQQcp9TAlpyq+c2DBkQfIPtVL+AwNDz0yRzR3Z2hn+Uca7YmVutmXUzB14mVkFqqEDTrU7tfkSTgIVABVGqZ5kxOCmRwKlgWcVOFIsJHZEB62kakoApJ80Py/CJVvrYj6T+IeBc/d2RkkCpSeDpytmeatGbif95vQT8ayflYZwAC+l8kJ8IDBGepYT7XDIKYqIJoZLrXTEdEkko6CwrOgRr8eRl0r6oW2bderisNW6LOMroCB2jU2ShK9RA96iJWoiiJ/SC3tC78Wy8Gh/G57y0ZBQ9h+gPjOk3ue6g1w==</latexit><latexit sha1_base64="uE6wEg+cbVDNn7T6D276YV5+N9k=">AAACGHicbVDLSsNAFJ3UV62vqks3g0VwITURQTdC0Y3LCvYBTQiT6aQdOnkwc1MoIZ/hxl9x40IRt935N07TgNp6hoHDOfdy7z1eLLgC0/wySiura+sb5c3K1vbO7l51/6CtokRS1qKRiGTXI4oJHrIWcBCsG0tGAk+wjje6m/mdMZOKR+EjTGLmBGQQcp9TAlpyq+c2DBkQfIPtVL+AwNDz0yRzR3Z2hn+Uca7YmVutmXUzB14mVkFqqEDTrU7tfkSTgIVABVGqZ5kxOCmRwKlgWcVOFIsJHZEB62kakoApJ80Py/CJVvrYj6T+IeBc/d2RkkCpSeDpytmeatGbif95vQT8ayflYZwAC+l8kJ8IDBGepYT7XDIKYqIJoZLrXTEdEkko6CwrOgRr8eRl0r6oW2bderisNW6LOMroCB2jU2ShK9RA96iJWoiiJ/SC3tC78Wy8Gh/G57y0ZBQ9h+gPjOk3ue6g1w==</latexit><latexit sha1_base64="uE6wEg+cbVDNn7T6D276YV5+N9k=">AAACGHicbVDLSsNAFJ3UV62vqks3g0VwITURQTdC0Y3LCvYBTQiT6aQdOnkwc1MoIZ/hxl9x40IRt935N07TgNp6hoHDOfdy7z1eLLgC0/wySiura+sb5c3K1vbO7l51/6CtokRS1qKRiGTXI4oJHrIWcBCsG0tGAk+wjje6m/mdMZOKR+EjTGLmBGQQcp9TAlpyq+c2DBkQfIPtVL+AwNDz0yRzR3Z2hn+Uca7YmVutmXUzB14mVkFqqEDTrU7tfkSTgIVABVGqZ5kxOCmRwKlgWcVOFIsJHZEB62kakoApJ80Py/CJVvrYj6T+IeBc/d2RkkCpSeDpytmeatGbif95vQT8ayflYZwAC+l8kJ8IDBGepYT7XDIKYqIJoZLrXTEdEkko6CwrOgRr8eRl0r6oW2bderisNW6LOMroCB2jU2ShK9RA96iJWoiiJ/SC3tC78Wy8Gh/G57y0ZBQ9h+gPjOk3ue6g1w==</latexit><latexit sha1_base64="uE6wEg+cbVDNn7T6D276YV5+N9k=">AAACGHicbVDLSsNAFJ3UV62vqks3g0VwITURQTdC0Y3LCvYBTQiT6aQdOnkwc1MoIZ/hxl9x40IRt935N07TgNp6hoHDOfdy7z1eLLgC0/wySiura+sb5c3K1vbO7l51/6CtokRS1qKRiGTXI4oJHrIWcBCsG0tGAk+wjje6m/mdMZOKR+EjTGLmBGQQcp9TAlpyq+c2DBkQfIPtVL+AwNDz0yRzR3Z2hn+Uca7YmVutmXUzB14mVkFqqEDTrU7tfkSTgIVABVGqZ5kxOCmRwKlgWcVOFIsJHZEB62kakoApJ80Py/CJVvrYj6T+IeBc/d2RkkCpSeDpytmeatGbif95vQT8ayflYZwAC+l8kJ8IDBGepYT7XDIKYqIJoZLrXTEdEkko6CwrOgRr8eRl0r6oW2bderisNW6LOMroCB2jU2ShK9RA96iJWoiiJ/SC3tC78Wy8Gh/G57y0ZBQ9h+gPjOk3ue6g1w==</latexit>

Page 21: Word Embeddings - Princeton University · Word2vec • Input: a large text corpora, V, d • Output: • V: a pre-defined vocabulary • d: dimension of word vectors (e.g. 300) •

How to train the model

Calculating all the gradients together!

Q: How many parameters are in total?

J(✓) = � 1

TlogL(✓) = � 1

T

TX

t=1

X

�mjm,j 6=0

logP (wt+j | wt; ✓)

<latexit sha1_base64="23utKwn7ZJE6urpMOKPMcw5eqOk=">AAACe3icdVFdaxQxFM2MWuuq7aqPggQXca3tMiOKghSKvoj4sEK3Lexsl0z2zm7aJDMmd5Ql5E/403zzn/gimNkdRFu9EHLuuV/JuXklhcUk+R7FV65e27i+eaNz89btre3unbtHtqwNhxEvZWlOcmZBCg0jFCjhpDLAVC7hOD9/28SPP4OxotSHuKxgothci0JwhoGadr++72e4AGRP6D7dywrDuEu9O/SZLOc0UwwXnEn3wf8vzdZq6nA/9ae/vT1FMwmf6Nn6UrsN0gEl3q3aDvtfQs3TMx8GiBltHP+atgP8tNtLBsnK6GWQtqBHWhtOu9+yWclrBRq5ZNaO06TCiWMGBZfgO1ltoWL8nM1hHKBmCuzErbTz9FFgZrQoTTga6Yr9s8IxZe1S5SGzEcNejDXkv2LjGotXEyd0VSNovh5U1JJiSZtF0JkwwFEuA2DciPBWyhcsCIthXZ0gQnrxy5fB0bNBmgzSj897B29aOTbJffKQ9ElKXpID8o4MyYhw8iN6ED2O+tHPuBfvxLvr1Dhqa+6Rvyx+8QtA7b8+</latexit><latexit sha1_base64="23utKwn7ZJE6urpMOKPMcw5eqOk=">AAACe3icdVFdaxQxFM2MWuuq7aqPggQXca3tMiOKghSKvoj4sEK3Lexsl0z2zm7aJDMmd5Ql5E/403zzn/gimNkdRFu9EHLuuV/JuXklhcUk+R7FV65e27i+eaNz89btre3unbtHtqwNhxEvZWlOcmZBCg0jFCjhpDLAVC7hOD9/28SPP4OxotSHuKxgothci0JwhoGadr++72e4AGRP6D7dywrDuEu9O/SZLOc0UwwXnEn3wf8vzdZq6nA/9ae/vT1FMwmf6Nn6UrsN0gEl3q3aDvtfQs3TMx8GiBltHP+atgP8tNtLBsnK6GWQtqBHWhtOu9+yWclrBRq5ZNaO06TCiWMGBZfgO1ltoWL8nM1hHKBmCuzErbTz9FFgZrQoTTga6Yr9s8IxZe1S5SGzEcNejDXkv2LjGotXEyd0VSNovh5U1JJiSZtF0JkwwFEuA2DciPBWyhcsCIthXZ0gQnrxy5fB0bNBmgzSj897B29aOTbJffKQ9ElKXpID8o4MyYhw8iN6ED2O+tHPuBfvxLvr1Dhqa+6Rvyx+8QtA7b8+</latexit><latexit sha1_base64="23utKwn7ZJE6urpMOKPMcw5eqOk=">AAACe3icdVFdaxQxFM2MWuuq7aqPggQXca3tMiOKghSKvoj4sEK3Lexsl0z2zm7aJDMmd5Ql5E/403zzn/gimNkdRFu9EHLuuV/JuXklhcUk+R7FV65e27i+eaNz89btre3unbtHtqwNhxEvZWlOcmZBCg0jFCjhpDLAVC7hOD9/28SPP4OxotSHuKxgothci0JwhoGadr++72e4AGRP6D7dywrDuEu9O/SZLOc0UwwXnEn3wf8vzdZq6nA/9ae/vT1FMwmf6Nn6UrsN0gEl3q3aDvtfQs3TMx8GiBltHP+atgP8tNtLBsnK6GWQtqBHWhtOu9+yWclrBRq5ZNaO06TCiWMGBZfgO1ltoWL8nM1hHKBmCuzErbTz9FFgZrQoTTga6Yr9s8IxZe1S5SGzEcNejDXkv2LjGotXEyd0VSNovh5U1JJiSZtF0JkwwFEuA2DciPBWyhcsCIthXZ0gQnrxy5fB0bNBmgzSj897B29aOTbJffKQ9ElKXpID8o4MyYhw8iN6ED2O+tHPuBfvxLvr1Dhqa+6Rvyx+8QtA7b8+</latexit><latexit sha1_base64="23utKwn7ZJE6urpMOKPMcw5eqOk=">AAACe3icdVFdaxQxFM2MWuuq7aqPggQXca3tMiOKghSKvoj4sEK3Lexsl0z2zm7aJDMmd5Ql5E/403zzn/gimNkdRFu9EHLuuV/JuXklhcUk+R7FV65e27i+eaNz89btre3unbtHtqwNhxEvZWlOcmZBCg0jFCjhpDLAVC7hOD9/28SPP4OxotSHuKxgothci0JwhoGadr++72e4AGRP6D7dywrDuEu9O/SZLOc0UwwXnEn3wf8vzdZq6nA/9ae/vT1FMwmf6Nn6UrsN0gEl3q3aDvtfQs3TMx8GiBltHP+atgP8tNtLBsnK6GWQtqBHWhtOu9+yWclrBRq5ZNaO06TCiWMGBZfgO1ltoWL8nM1hHKBmCuzErbTz9FFgZrQoTTga6Yr9s8IxZe1S5SGzEcNejDXkv2LjGotXEyd0VSNovh5U1JJiSZtF0JkwwFEuA2DciPBWyhcsCIthXZ0gQnrxy5fB0bNBmgzSj897B29aOTbJffKQ9ElKXpID8o4MyYhw8iN6ED2O+tHPuBfvxLvr1Dhqa+6Rvyx+8QtA7b8+</latexit>

J(✓) = � 1

TlogL(✓) = � 1

T

TX

t=1

X

�mjm,j 6=0

logP (wt+j | wt; ✓)

<latexit sha1_base64="23utKwn7ZJE6urpMOKPMcw5eqOk=">AAACe3icdVFdaxQxFM2MWuuq7aqPggQXca3tMiOKghSKvoj4sEK3Lexsl0z2zm7aJDMmd5Ql5E/403zzn/gimNkdRFu9EHLuuV/JuXklhcUk+R7FV65e27i+eaNz89btre3unbtHtqwNhxEvZWlOcmZBCg0jFCjhpDLAVC7hOD9/28SPP4OxotSHuKxgothci0JwhoGadr++72e4AGRP6D7dywrDuEu9O/SZLOc0UwwXnEn3wf8vzdZq6nA/9ae/vT1FMwmf6Nn6UrsN0gEl3q3aDvtfQs3TMx8GiBltHP+atgP8tNtLBsnK6GWQtqBHWhtOu9+yWclrBRq5ZNaO06TCiWMGBZfgO1ltoWL8nM1hHKBmCuzErbTz9FFgZrQoTTga6Yr9s8IxZe1S5SGzEcNejDXkv2LjGotXEyd0VSNovh5U1JJiSZtF0JkwwFEuA2DciPBWyhcsCIthXZ0gQnrxy5fB0bNBmgzSj897B29aOTbJffKQ9ElKXpID8o4MyYhw8iN6ED2O+tHPuBfvxLvr1Dhqa+6Rvyx+8QtA7b8+</latexit><latexit sha1_base64="23utKwn7ZJE6urpMOKPMcw5eqOk=">AAACe3icdVFdaxQxFM2MWuuq7aqPggQXca3tMiOKghSKvoj4sEK3Lexsl0z2zm7aJDMmd5Ql5E/403zzn/gimNkdRFu9EHLuuV/JuXklhcUk+R7FV65e27i+eaNz89btre3unbtHtqwNhxEvZWlOcmZBCg0jFCjhpDLAVC7hOD9/28SPP4OxotSHuKxgothci0JwhoGadr++72e4AGRP6D7dywrDuEu9O/SZLOc0UwwXnEn3wf8vzdZq6nA/9ae/vT1FMwmf6Nn6UrsN0gEl3q3aDvtfQs3TMx8GiBltHP+atgP8tNtLBsnK6GWQtqBHWhtOu9+yWclrBRq5ZNaO06TCiWMGBZfgO1ltoWL8nM1hHKBmCuzErbTz9FFgZrQoTTga6Yr9s8IxZe1S5SGzEcNejDXkv2LjGotXEyd0VSNovh5U1JJiSZtF0JkwwFEuA2DciPBWyhcsCIthXZ0gQnrxy5fB0bNBmgzSj897B29aOTbJffKQ9ElKXpID8o4MyYhw8iN6ED2O+tHPuBfvxLvr1Dhqa+6Rvyx+8QtA7b8+</latexit><latexit sha1_base64="23utKwn7ZJE6urpMOKPMcw5eqOk=">AAACe3icdVFdaxQxFM2MWuuq7aqPggQXca3tMiOKghSKvoj4sEK3Lexsl0z2zm7aJDMmd5Ql5E/403zzn/gimNkdRFu9EHLuuV/JuXklhcUk+R7FV65e27i+eaNz89btre3unbtHtqwNhxEvZWlOcmZBCg0jFCjhpDLAVC7hOD9/28SPP4OxotSHuKxgothci0JwhoGadr++72e4AGRP6D7dywrDuEu9O/SZLOc0UwwXnEn3wf8vzdZq6nA/9ae/vT1FMwmf6Nn6UrsN0gEl3q3aDvtfQs3TMx8GiBltHP+atgP8tNtLBsnK6GWQtqBHWhtOu9+yWclrBRq5ZNaO06TCiWMGBZfgO1ltoWL8nM1hHKBmCuzErbTz9FFgZrQoTTga6Yr9s8IxZe1S5SGzEcNejDXkv2LjGotXEyd0VSNovh5U1JJiSZtF0JkwwFEuA2DciPBWyhcsCIthXZ0gQnrxy5fB0bNBmgzSj897B29aOTbJffKQ9ElKXpID8o4MyYhw8iN6ED2O+tHPuBfvxLvr1Dhqa+6Rvyx+8QtA7b8+</latexit><latexit sha1_base64="23utKwn7ZJE6urpMOKPMcw5eqOk=">AAACe3icdVFdaxQxFM2MWuuq7aqPggQXca3tMiOKghSKvoj4sEK3Lexsl0z2zm7aJDMmd5Ql5E/403zzn/gimNkdRFu9EHLuuV/JuXklhcUk+R7FV65e27i+eaNz89btre3unbtHtqwNhxEvZWlOcmZBCg0jFCjhpDLAVC7hOD9/28SPP4OxotSHuKxgothci0JwhoGadr++72e4AGRP6D7dywrDuEu9O/SZLOc0UwwXnEn3wf8vzdZq6nA/9ae/vT1FMwmf6Nn6UrsN0gEl3q3aDvtfQs3TMx8GiBltHP+atgP8tNtLBsnK6GWQtqBHWhtOu9+yWclrBRq5ZNaO06TCiWMGBZfgO1ltoWL8nM1hHKBmCuzErbTz9FFgZrQoTTga6Yr9s8IxZe1S5SGzEcNejDXkv2LjGotXEyd0VSNovh5U1JJiSZtF0JkwwFEuA2DciPBWyhcsCIthXZ0gQnrxy5fB0bNBmgzSj897B29aOTbJffKQ9ElKXpID8o4MyYhw8iN6ED2O+tHPuBfvxLvr1Dhqa+6Rvyx+8QtA7b8+</latexit>

r✓J(✓) =?<latexit sha1_base64="oFtCj5NE4VIa6vcNKQlly3hbvtM=">AAACA3icbZDLSsNAFIYnXmu9Vd3pJliEuimJCLoRi27EVQV7gSaUk+mkHTqZhJkToYSCG1/FjQtF3PoS7nwbp5eFtv4w8PGfczhz/iARXKPjfFsLi0vLK6u5tfz6xubWdmFnt67jVFFWo7GIVTMAzQSXrIYcBWsmikEUCNYI+tejeuOBKc1jeY+DhPkRdCUPOQU0Vruw70kIBLQzD3sMYXhbmsDxxWW7UHTKzlj2PLhTKJKpqu3Cl9eJaRoxiVSA1i3XSdDPQCGngg3zXqpZArQPXdYyKCFi2s/GNwztI+N07DBW5km0x+7viQwirQdRYDojwJ6erY3M/2qtFMNzP+MySZFJOlkUpsLG2B4FYne4YhTFwABQxc1fbdoDBRRNbHkTgjt78jzUT8quU3bvTouVq2kcOXJADkmJuOSMVMgNqZIaoeSRPJNX8mY9WS/Wu/UxaV2wpjN75I+szx9vapdb</latexit><latexit sha1_base64="oFtCj5NE4VIa6vcNKQlly3hbvtM=">AAACA3icbZDLSsNAFIYnXmu9Vd3pJliEuimJCLoRi27EVQV7gSaUk+mkHTqZhJkToYSCG1/FjQtF3PoS7nwbp5eFtv4w8PGfczhz/iARXKPjfFsLi0vLK6u5tfz6xubWdmFnt67jVFFWo7GIVTMAzQSXrIYcBWsmikEUCNYI+tejeuOBKc1jeY+DhPkRdCUPOQU0Vruw70kIBLQzD3sMYXhbmsDxxWW7UHTKzlj2PLhTKJKpqu3Cl9eJaRoxiVSA1i3XSdDPQCGngg3zXqpZArQPXdYyKCFi2s/GNwztI+N07DBW5km0x+7viQwirQdRYDojwJ6erY3M/2qtFMNzP+MySZFJOlkUpsLG2B4FYne4YhTFwABQxc1fbdoDBRRNbHkTgjt78jzUT8quU3bvTouVq2kcOXJADkmJuOSMVMgNqZIaoeSRPJNX8mY9WS/Wu/UxaV2wpjN75I+szx9vapdb</latexit><latexit sha1_base64="oFtCj5NE4VIa6vcNKQlly3hbvtM=">AAACA3icbZDLSsNAFIYnXmu9Vd3pJliEuimJCLoRi27EVQV7gSaUk+mkHTqZhJkToYSCG1/FjQtF3PoS7nwbp5eFtv4w8PGfczhz/iARXKPjfFsLi0vLK6u5tfz6xubWdmFnt67jVFFWo7GIVTMAzQSXrIYcBWsmikEUCNYI+tejeuOBKc1jeY+DhPkRdCUPOQU0Vruw70kIBLQzD3sMYXhbmsDxxWW7UHTKzlj2PLhTKJKpqu3Cl9eJaRoxiVSA1i3XSdDPQCGngg3zXqpZArQPXdYyKCFi2s/GNwztI+N07DBW5km0x+7viQwirQdRYDojwJ6erY3M/2qtFMNzP+MySZFJOlkUpsLG2B4FYne4YhTFwABQxc1fbdoDBRRNbHkTgjt78jzUT8quU3bvTouVq2kcOXJADkmJuOSMVMgNqZIaoeSRPJNX8mY9WS/Wu/UxaV2wpjN75I+szx9vapdb</latexit><latexit sha1_base64="oFtCj5NE4VIa6vcNKQlly3hbvtM=">AAACA3icbZDLSsNAFIYnXmu9Vd3pJliEuimJCLoRi27EVQV7gSaUk+mkHTqZhJkToYSCG1/FjQtF3PoS7nwbp5eFtv4w8PGfczhz/iARXKPjfFsLi0vLK6u5tfz6xubWdmFnt67jVFFWo7GIVTMAzQSXrIYcBWsmikEUCNYI+tejeuOBKc1jeY+DhPkRdCUPOQU0Vruw70kIBLQzD3sMYXhbmsDxxWW7UHTKzlj2PLhTKJKpqu3Cl9eJaRoxiVSA1i3XSdDPQCGngg3zXqpZArQPXdYyKCFi2s/GNwztI+N07DBW5km0x+7viQwirQdRYDojwJ6erY3M/2qtFMNzP+MySZFJOlkUpsLG2B4FYne4YhTFwABQxc1fbdoDBRRNbHkTgjt78jzUT8quU3bvTouVq2kcOXJADkmJuOSMVMgNqZIaoeSRPJNX8mY9WS/Wu/UxaV2wpjN75I+szx9vapdb</latexit>

We can apply stochastic gradient descent (SGD)!

Let’s walk through the math..

✓(t+1) = ✓(t) � ⌘r✓J(✓)<latexit sha1_base64="2xbrEJR+XVhUcysjVyGPSHic0HY=">AAACJnicbVDLSgNBEJz1GeMr6tHLYBAiYtgVQS+C6EU8RTBRyK5L72RihszOLjO9QljyNV78FS8eIiLe/BQnD/BZMFBd1U1PV5RKYdB1352p6ZnZufnCQnFxaXlltbS23jBJphmvs0Qm+iYCw6VQvI4CJb9JNYc4kvw66p4N/et7ro1I1BX2Uh7EcKdEWzBAK4WlYx87HOE2r+Cut9Onx/RLsOUe9W1BfQWRhDAfe/2LypjshKWyW3VHoH+JNyFlMkEtLA38VsKymCtkEoxpem6KQQ4aBZO8X/Qzw1NgXbjjTUsVxNwE+ejMPt22Sou2E22fQjpSv0/kEBvTiyPbGQN2zG9vKP7nNTNsHwW5UGmGXLHxonYmKSZ0mBltCc0Zyp4lwLSwf6WsAxoY2mSLNgTv98l/SWO/6rlV7/KgfHI6iaNANskWqRCPHJITck5qpE4YeSBPZEBenEfn2Xl13satU85kZoP8gPPxCRNypFM=</latexit><latexit sha1_base64="2xbrEJR+XVhUcysjVyGPSHic0HY=">AAACJnicbVDLSgNBEJz1GeMr6tHLYBAiYtgVQS+C6EU8RTBRyK5L72RihszOLjO9QljyNV78FS8eIiLe/BQnD/BZMFBd1U1PV5RKYdB1352p6ZnZufnCQnFxaXlltbS23jBJphmvs0Qm+iYCw6VQvI4CJb9JNYc4kvw66p4N/et7ro1I1BX2Uh7EcKdEWzBAK4WlYx87HOE2r+Cut9Onx/RLsOUe9W1BfQWRhDAfe/2LypjshKWyW3VHoH+JNyFlMkEtLA38VsKymCtkEoxpem6KQQ4aBZO8X/Qzw1NgXbjjTUsVxNwE+ejMPt22Sou2E22fQjpSv0/kEBvTiyPbGQN2zG9vKP7nNTNsHwW5UGmGXLHxonYmKSZ0mBltCc0Zyp4lwLSwf6WsAxoY2mSLNgTv98l/SWO/6rlV7/KgfHI6iaNANskWqRCPHJITck5qpE4YeSBPZEBenEfn2Xl13satU85kZoP8gPPxCRNypFM=</latexit><latexit sha1_base64="2xbrEJR+XVhUcysjVyGPSHic0HY=">AAACJnicbVDLSgNBEJz1GeMr6tHLYBAiYtgVQS+C6EU8RTBRyK5L72RihszOLjO9QljyNV78FS8eIiLe/BQnD/BZMFBd1U1PV5RKYdB1352p6ZnZufnCQnFxaXlltbS23jBJphmvs0Qm+iYCw6VQvI4CJb9JNYc4kvw66p4N/et7ro1I1BX2Uh7EcKdEWzBAK4WlYx87HOE2r+Cut9Onx/RLsOUe9W1BfQWRhDAfe/2LypjshKWyW3VHoH+JNyFlMkEtLA38VsKymCtkEoxpem6KQQ4aBZO8X/Qzw1NgXbjjTUsVxNwE+ejMPt22Sou2E22fQjpSv0/kEBvTiyPbGQN2zG9vKP7nNTNsHwW5UGmGXLHxonYmKSZ0mBltCc0Zyp4lwLSwf6WsAxoY2mSLNgTv98l/SWO/6rlV7/KgfHI6iaNANskWqRCPHJITck5qpE4YeSBPZEBenEfn2Xl13satU85kZoP8gPPxCRNypFM=</latexit><latexit sha1_base64="2xbrEJR+XVhUcysjVyGPSHic0HY=">AAACJnicbVDLSgNBEJz1GeMr6tHLYBAiYtgVQS+C6EU8RTBRyK5L72RihszOLjO9QljyNV78FS8eIiLe/BQnD/BZMFBd1U1PV5RKYdB1352p6ZnZufnCQnFxaXlltbS23jBJphmvs0Qm+iYCw6VQvI4CJb9JNYc4kvw66p4N/et7ro1I1BX2Uh7EcKdEWzBAK4WlYx87HOE2r+Cut9Onx/RLsOUe9W1BfQWRhDAfe/2LypjshKWyW3VHoH+JNyFlMkEtLA38VsKymCtkEoxpem6KQQ4aBZO8X/Qzw1NgXbjjTUsVxNwE+ejMPt22Sou2E22fQjpSv0/kEBvTiyPbGQN2zG9vKP7nNTNsHwW5UGmGXLHxonYmKSZ0mBltCc0Zyp4lwLSwf6WsAxoY2mSLNgTv98l/SWO/6rlV7/KgfHI6iaNANskWqRCPHJITck5qpE4YeSBPZEBenEfn2Xl13satU85kZoP8gPPxCRNypFM=</latexit>

✓ = {{uk}, {vk}}<latexit sha1_base64="uE6wEg+cbVDNn7T6D276YV5+N9k=">AAACGHicbVDLSsNAFJ3UV62vqks3g0VwITURQTdC0Y3LCvYBTQiT6aQdOnkwc1MoIZ/hxl9x40IRt935N07TgNp6hoHDOfdy7z1eLLgC0/wySiura+sb5c3K1vbO7l51/6CtokRS1qKRiGTXI4oJHrIWcBCsG0tGAk+wjje6m/mdMZOKR+EjTGLmBGQQcp9TAlpyq+c2DBkQfIPtVL+AwNDz0yRzR3Z2hn+Uca7YmVutmXUzB14mVkFqqEDTrU7tfkSTgIVABVGqZ5kxOCmRwKlgWcVOFIsJHZEB62kakoApJ80Py/CJVvrYj6T+IeBc/d2RkkCpSeDpytmeatGbif95vQT8ayflYZwAC+l8kJ8IDBGepYT7XDIKYqIJoZLrXTEdEkko6CwrOgRr8eRl0r6oW2bderisNW6LOMroCB2jU2ShK9RA96iJWoiiJ/SC3tC78Wy8Gh/G57y0ZBQ9h+gPjOk3ue6g1w==</latexit><latexit sha1_base64="uE6wEg+cbVDNn7T6D276YV5+N9k=">AAACGHicbVDLSsNAFJ3UV62vqks3g0VwITURQTdC0Y3LCvYBTQiT6aQdOnkwc1MoIZ/hxl9x40IRt935N07TgNp6hoHDOfdy7z1eLLgC0/wySiura+sb5c3K1vbO7l51/6CtokRS1qKRiGTXI4oJHrIWcBCsG0tGAk+wjje6m/mdMZOKR+EjTGLmBGQQcp9TAlpyq+c2DBkQfIPtVL+AwNDz0yRzR3Z2hn+Uca7YmVutmXUzB14mVkFqqEDTrU7tfkSTgIVABVGqZ5kxOCmRwKlgWcVOFIsJHZEB62kakoApJ80Py/CJVvrYj6T+IeBc/d2RkkCpSeDpytmeatGbif95vQT8ayflYZwAC+l8kJ8IDBGepYT7XDIKYqIJoZLrXTEdEkko6CwrOgRr8eRl0r6oW2bderisNW6LOMroCB2jU2ShK9RA96iJWoiiJ/SC3tC78Wy8Gh/G57y0ZBQ9h+gPjOk3ue6g1w==</latexit><latexit sha1_base64="uE6wEg+cbVDNn7T6D276YV5+N9k=">AAACGHicbVDLSsNAFJ3UV62vqks3g0VwITURQTdC0Y3LCvYBTQiT6aQdOnkwc1MoIZ/hxl9x40IRt935N07TgNp6hoHDOfdy7z1eLLgC0/wySiura+sb5c3K1vbO7l51/6CtokRS1qKRiGTXI4oJHrIWcBCsG0tGAk+wjje6m/mdMZOKR+EjTGLmBGQQcp9TAlpyq+c2DBkQfIPtVL+AwNDz0yRzR3Z2hn+Uca7YmVutmXUzB14mVkFqqEDTrU7tfkSTgIVABVGqZ5kxOCmRwKlgWcVOFIsJHZEB62kakoApJ80Py/CJVvrYj6T+IeBc/d2RkkCpSeDpytmeatGbif95vQT8ayflYZwAC+l8kJ8IDBGepYT7XDIKYqIJoZLrXTEdEkko6CwrOgRr8eRl0r6oW2bderisNW6LOMroCB2jU2ShK9RA96iJWoiiJ/SC3tC78Wy8Gh/G57y0ZBQ9h+gPjOk3ue6g1w==</latexit><latexit sha1_base64="uE6wEg+cbVDNn7T6D276YV5+N9k=">AAACGHicbVDLSsNAFJ3UV62vqks3g0VwITURQTdC0Y3LCvYBTQiT6aQdOnkwc1MoIZ/hxl9x40IRt935N07TgNp6hoHDOfdy7z1eLLgC0/wySiura+sb5c3K1vbO7l51/6CtokRS1qKRiGTXI4oJHrIWcBCsG0tGAk+wjje6m/mdMZOKR+EjTGLmBGQQcp9TAlpyq+c2DBkQfIPtVL+AwNDz0yRzR3Z2hn+Uca7YmVutmXUzB14mVkFqqEDTrU7tfkSTgIVABVGqZ5kxOCmRwKlgWcVOFIsJHZEB62kakoApJ80Py/CJVvrYj6T+IeBc/d2RkkCpSeDpytmeatGbif95vQT8ayflYZwAC+l8kJ8IDBGepYT7XDIKYqIJoZLrXTEdEkko6CwrOgRr8eRl0r6oW2bderisNW6LOMroCB2jU2ShK9RA96iJWoiiJ/SC3tC78Wy8Gh/G57y0ZBQ9h+gPjOk3ue6g1w==</latexit>

Page 22: Word Embeddings - Princeton University · Word2vec • Input: a large text corpora, V, d • Output: • V: a pre-defined vocabulary • d: dimension of word vectors (e.g. 300) •

Warm-up

f(x) = exp(x)<latexit sha1_base64="F2Q/ji6x1DLqPqoZ1t1sz9mjuL8=">AAAB9XicbZDLSgNBEEVr4ivGV9Slm8YgxE2YEUE3QtCNywjmAckYejo1SZOeB909mjDkP9y4UMSt/+LOv7GTzEITLzQcblVR1deLBVfatr+t3Mrq2vpGfrOwtb2zu1fcP2ioKJEM6ywSkWx5VKHgIdY11wJbsUQaeAKb3vBmWm8+olQ8Cu/1OEY3oP2Q+5xRbawHvzw6JVekg6PYULdYsiv2TGQZnAxKkKnWLX51ehFLAgw1E1SptmPH2k2p1JwJnBQ6icKYsiHtY9tgSANUbjq7ekJOjNMjfiTNCzWZub8nUhooNQ480xlQPVCLtan5X62daP/STXkYJxpDNl/kJ4LoiEwjID0ukWkxNkCZ5OZWwgZUUqZNUAUTgrP45WVonFUcu+LcnZeq11kceTiCYyiDAxdQhVuoQR0YSHiGV3iznqwX6936mLfmrGzmEP7I+vwBaEqRJA==</latexit><latexit sha1_base64="F2Q/ji6x1DLqPqoZ1t1sz9mjuL8=">AAAB9XicbZDLSgNBEEVr4ivGV9Slm8YgxE2YEUE3QtCNywjmAckYejo1SZOeB909mjDkP9y4UMSt/+LOv7GTzEITLzQcblVR1deLBVfatr+t3Mrq2vpGfrOwtb2zu1fcP2ioKJEM6ywSkWx5VKHgIdY11wJbsUQaeAKb3vBmWm8+olQ8Cu/1OEY3oP2Q+5xRbawHvzw6JVekg6PYULdYsiv2TGQZnAxKkKnWLX51ehFLAgw1E1SptmPH2k2p1JwJnBQ6icKYsiHtY9tgSANUbjq7ekJOjNMjfiTNCzWZub8nUhooNQ480xlQPVCLtan5X62daP/STXkYJxpDNl/kJ4LoiEwjID0ukWkxNkCZ5OZWwgZUUqZNUAUTgrP45WVonFUcu+LcnZeq11kceTiCYyiDAxdQhVuoQR0YSHiGV3iznqwX6936mLfmrGzmEP7I+vwBaEqRJA==</latexit><latexit sha1_base64="F2Q/ji6x1DLqPqoZ1t1sz9mjuL8=">AAAB9XicbZDLSgNBEEVr4ivGV9Slm8YgxE2YEUE3QtCNywjmAckYejo1SZOeB909mjDkP9y4UMSt/+LOv7GTzEITLzQcblVR1deLBVfatr+t3Mrq2vpGfrOwtb2zu1fcP2ioKJEM6ywSkWx5VKHgIdY11wJbsUQaeAKb3vBmWm8+olQ8Cu/1OEY3oP2Q+5xRbawHvzw6JVekg6PYULdYsiv2TGQZnAxKkKnWLX51ehFLAgw1E1SptmPH2k2p1JwJnBQ6icKYsiHtY9tgSANUbjq7ekJOjNMjfiTNCzWZub8nUhooNQ480xlQPVCLtan5X62daP/STXkYJxpDNl/kJ4LoiEwjID0ukWkxNkCZ5OZWwgZUUqZNUAUTgrP45WVonFUcu+LcnZeq11kceTiCYyiDAxdQhVuoQR0YSHiGV3iznqwX6936mLfmrGzmEP7I+vwBaEqRJA==</latexit><latexit sha1_base64="F2Q/ji6x1DLqPqoZ1t1sz9mjuL8=">AAAB9XicbZDLSgNBEEVr4ivGV9Slm8YgxE2YEUE3QtCNywjmAckYejo1SZOeB909mjDkP9y4UMSt/+LOv7GTzEITLzQcblVR1deLBVfatr+t3Mrq2vpGfrOwtb2zu1fcP2ioKJEM6ywSkWx5VKHgIdY11wJbsUQaeAKb3vBmWm8+olQ8Cu/1OEY3oP2Q+5xRbawHvzw6JVekg6PYULdYsiv2TGQZnAxKkKnWLX51ehFLAgw1E1SptmPH2k2p1JwJnBQ6icKYsiHtY9tgSANUbjq7ekJOjNMjfiTNCzWZub8nUhooNQ480xlQPVCLtan5X62daP/STXkYJxpDNl/kJ4LoiEwjID0ukWkxNkCZ5OZWwgZUUqZNUAUTgrP45WVonFUcu+LcnZeq11kceTiCYyiDAxdQhVuoQR0YSHiGV3iznqwX6936mLfmrGzmEP7I+vwBaEqRJA==</latexit>

df

dx=

<latexit sha1_base64="h5ZKeLyvOAKrQ28BVy2eVsxqRMI=">AAAB+XicbVBNS8NAEJ3Ur1q/oh69LBbBU0lE0ItQ9OKxgv2ANpTNZtMu3WzC7qZYQv6JFw+KePWfePPfuGlz0NYHA4/3ZpiZ5yecKe0431ZlbX1jc6u6XdvZ3ds/sA+POipOJaFtEvNY9nysKGeCtjXTnPYSSXHkc9r1J3eF351SqVgsHvUsoV6ER4KFjGBtpKFtD0KJSRaEeRY85egGDe2603DmQKvELUkdSrSG9tcgiEkaUaEJx0r1XSfRXoalZoTTvDZIFU0wmeAR7RsqcESVl80vz9GZUQIUxtKU0Giu/p7IcKTULPJNZ4T1WC17hfif1091eO1lTCSppoIsFoUpRzpGRQwoYJISzWeGYCKZuRWRMTZRaBNWzYTgLr+8SjoXDddpuA+X9eZtGUcVTuAUzsGFK2jCPbSgDQSm8Ayv8GZl1ov1bn0sWitWOXMMf2B9/gAs4ZNW</latexit><latexit sha1_base64="h5ZKeLyvOAKrQ28BVy2eVsxqRMI=">AAAB+XicbVBNS8NAEJ3Ur1q/oh69LBbBU0lE0ItQ9OKxgv2ANpTNZtMu3WzC7qZYQv6JFw+KePWfePPfuGlz0NYHA4/3ZpiZ5yecKe0431ZlbX1jc6u6XdvZ3ds/sA+POipOJaFtEvNY9nysKGeCtjXTnPYSSXHkc9r1J3eF351SqVgsHvUsoV6ER4KFjGBtpKFtD0KJSRaEeRY85egGDe2603DmQKvELUkdSrSG9tcgiEkaUaEJx0r1XSfRXoalZoTTvDZIFU0wmeAR7RsqcESVl80vz9GZUQIUxtKU0Giu/p7IcKTULPJNZ4T1WC17hfif1091eO1lTCSppoIsFoUpRzpGRQwoYJISzWeGYCKZuRWRMTZRaBNWzYTgLr+8SjoXDddpuA+X9eZtGUcVTuAUzsGFK2jCPbSgDQSm8Ayv8GZl1ov1bn0sWitWOXMMf2B9/gAs4ZNW</latexit><latexit sha1_base64="h5ZKeLyvOAKrQ28BVy2eVsxqRMI=">AAAB+XicbVBNS8NAEJ3Ur1q/oh69LBbBU0lE0ItQ9OKxgv2ANpTNZtMu3WzC7qZYQv6JFw+KePWfePPfuGlz0NYHA4/3ZpiZ5yecKe0431ZlbX1jc6u6XdvZ3ds/sA+POipOJaFtEvNY9nysKGeCtjXTnPYSSXHkc9r1J3eF351SqVgsHvUsoV6ER4KFjGBtpKFtD0KJSRaEeRY85egGDe2603DmQKvELUkdSrSG9tcgiEkaUaEJx0r1XSfRXoalZoTTvDZIFU0wmeAR7RsqcESVl80vz9GZUQIUxtKU0Giu/p7IcKTULPJNZ4T1WC17hfif1091eO1lTCSppoIsFoUpRzpGRQwoYJISzWeGYCKZuRWRMTZRaBNWzYTgLr+8SjoXDddpuA+X9eZtGUcVTuAUzsGFK2jCPbSgDQSm8Ayv8GZl1ov1bn0sWitWOXMMf2B9/gAs4ZNW</latexit><latexit sha1_base64="h5ZKeLyvOAKrQ28BVy2eVsxqRMI=">AAAB+XicbVBNS8NAEJ3Ur1q/oh69LBbBU0lE0ItQ9OKxgv2ANpTNZtMu3WzC7qZYQv6JFw+KePWfePPfuGlz0NYHA4/3ZpiZ5yecKe0431ZlbX1jc6u6XdvZ3ds/sA+POipOJaFtEvNY9nysKGeCtjXTnPYSSXHkc9r1J3eF351SqVgsHvUsoV6ER4KFjGBtpKFtD0KJSRaEeRY85egGDe2603DmQKvELUkdSrSG9tcgiEkaUaEJx0r1XSfRXoalZoTTvDZIFU0wmeAR7RsqcESVl80vz9GZUQIUxtKU0Giu/p7IcKTULPJNZ4T1WC17hfif1091eO1lTCSppoIsFoUpRzpGRQwoYJISzWeGYCKZuRWRMTZRaBNWzYTgLr+8SjoXDddpuA+X9eZtGUcVTuAUzsGFK2jCPbSgDQSm8Ayv8GZl1ov1bn0sWitWOXMMf2B9/gAs4ZNW</latexit>

f(x) = log(x)<latexit sha1_base64="3m6FmaA25KkL1Rsjbj7cjS/qDt0=">AAAB9XicbZDLSgMxFIbP1Futt6pLN8Ei1E2ZkYJuhKIblxXsBdqxZNJMG5pJhiSjlqHv4caFIm59F3e+jWk7C239IfDxn3M4J38Qc6aN6347uZXVtfWN/GZha3tnd6+4f9DUMlGENojkUrUDrClngjYMM5y2Y0VxFHDaCkbX03rrgSrNpLgz45j6ER4IFjKCjbXuw/LTKbpEXS4HlnrFkltxZ0LL4GVQgkz1XvGr25ckiagwhGOtO54bGz/FyjDC6aTQTTSNMRnhAe1YFDii2k9nV0/QiXX6KJTKPmHQzP09keJI63EU2M4Im6FerE3N/2qdxIQXfspEnBgqyHxRmHBkJJpGgPpMUWL42AImitlbERlihYmxQRVsCN7il5eheVbx3Ip3Wy3VrrI48nAEx1AGD86hBjdQhwYQUPAMr/DmPDovzrvzMW/NOdnMIfyR8/kDV4KRGQ==</latexit><latexit sha1_base64="3m6FmaA25KkL1Rsjbj7cjS/qDt0=">AAAB9XicbZDLSgMxFIbP1Futt6pLN8Ei1E2ZkYJuhKIblxXsBdqxZNJMG5pJhiSjlqHv4caFIm59F3e+jWk7C239IfDxn3M4J38Qc6aN6347uZXVtfWN/GZha3tnd6+4f9DUMlGENojkUrUDrClngjYMM5y2Y0VxFHDaCkbX03rrgSrNpLgz45j6ER4IFjKCjbXuw/LTKbpEXS4HlnrFkltxZ0LL4GVQgkz1XvGr25ckiagwhGOtO54bGz/FyjDC6aTQTTSNMRnhAe1YFDii2k9nV0/QiXX6KJTKPmHQzP09keJI63EU2M4Im6FerE3N/2qdxIQXfspEnBgqyHxRmHBkJJpGgPpMUWL42AImitlbERlihYmxQRVsCN7il5eheVbx3Ip3Wy3VrrI48nAEx1AGD86hBjdQhwYQUPAMr/DmPDovzrvzMW/NOdnMIfyR8/kDV4KRGQ==</latexit><latexit sha1_base64="3m6FmaA25KkL1Rsjbj7cjS/qDt0=">AAAB9XicbZDLSgMxFIbP1Futt6pLN8Ei1E2ZkYJuhKIblxXsBdqxZNJMG5pJhiSjlqHv4caFIm59F3e+jWk7C239IfDxn3M4J38Qc6aN6347uZXVtfWN/GZha3tnd6+4f9DUMlGENojkUrUDrClngjYMM5y2Y0VxFHDaCkbX03rrgSrNpLgz45j6ER4IFjKCjbXuw/LTKbpEXS4HlnrFkltxZ0LL4GVQgkz1XvGr25ckiagwhGOtO54bGz/FyjDC6aTQTTSNMRnhAe1YFDii2k9nV0/QiXX6KJTKPmHQzP09keJI63EU2M4Im6FerE3N/2qdxIQXfspEnBgqyHxRmHBkJJpGgPpMUWL42AImitlbERlihYmxQRVsCN7il5eheVbx3Ip3Wy3VrrI48nAEx1AGD86hBjdQhwYQUPAMr/DmPDovzrvzMW/NOdnMIfyR8/kDV4KRGQ==</latexit><latexit sha1_base64="3m6FmaA25KkL1Rsjbj7cjS/qDt0=">AAAB9XicbZDLSgMxFIbP1Futt6pLN8Ei1E2ZkYJuhKIblxXsBdqxZNJMG5pJhiSjlqHv4caFIm59F3e+jWk7C239IfDxn3M4J38Qc6aN6347uZXVtfWN/GZha3tnd6+4f9DUMlGENojkUrUDrClngjYMM5y2Y0VxFHDaCkbX03rrgSrNpLgz45j6ER4IFjKCjbXuw/LTKbpEXS4HlnrFkltxZ0LL4GVQgkz1XvGr25ckiagwhGOtO54bGz/FyjDC6aTQTTSNMRnhAe1YFDii2k9nV0/QiXX6KJTKPmHQzP09keJI63EU2M4Im6FerE3N/2qdxIQXfspEnBgqyHxRmHBkJJpGgPpMUWL42AImitlbERlihYmxQRVsCN7il5eheVbx3Ip3Wy3VrrI48nAEx1AGD86hBjdQhwYQUPAMr/DmPDovzrvzMW/NOdnMIfyR8/kDV4KRGQ==</latexit>

df

dx=

<latexit sha1_base64="h5ZKeLyvOAKrQ28BVy2eVsxqRMI=">AAAB+XicbVBNS8NAEJ3Ur1q/oh69LBbBU0lE0ItQ9OKxgv2ANpTNZtMu3WzC7qZYQv6JFw+KePWfePPfuGlz0NYHA4/3ZpiZ5yecKe0431ZlbX1jc6u6XdvZ3ds/sA+POipOJaFtEvNY9nysKGeCtjXTnPYSSXHkc9r1J3eF351SqVgsHvUsoV6ER4KFjGBtpKFtD0KJSRaEeRY85egGDe2603DmQKvELUkdSrSG9tcgiEkaUaEJx0r1XSfRXoalZoTTvDZIFU0wmeAR7RsqcESVl80vz9GZUQIUxtKU0Giu/p7IcKTULPJNZ4T1WC17hfif1091eO1lTCSppoIsFoUpRzpGRQwoYJISzWeGYCKZuRWRMTZRaBNWzYTgLr+8SjoXDddpuA+X9eZtGUcVTuAUzsGFK2jCPbSgDQSm8Ayv8GZl1ov1bn0sWitWOXMMf2B9/gAs4ZNW</latexit><latexit sha1_base64="h5ZKeLyvOAKrQ28BVy2eVsxqRMI=">AAAB+XicbVBNS8NAEJ3Ur1q/oh69LBbBU0lE0ItQ9OKxgv2ANpTNZtMu3WzC7qZYQv6JFw+KePWfePPfuGlz0NYHA4/3ZpiZ5yecKe0431ZlbX1jc6u6XdvZ3ds/sA+POipOJaFtEvNY9nysKGeCtjXTnPYSSXHkc9r1J3eF351SqVgsHvUsoV6ER4KFjGBtpKFtD0KJSRaEeRY85egGDe2603DmQKvELUkdSrSG9tcgiEkaUaEJx0r1XSfRXoalZoTTvDZIFU0wmeAR7RsqcESVl80vz9GZUQIUxtKU0Giu/p7IcKTULPJNZ4T1WC17hfif1091eO1lTCSppoIsFoUpRzpGRQwoYJISzWeGYCKZuRWRMTZRaBNWzYTgLr+8SjoXDddpuA+X9eZtGUcVTuAUzsGFK2jCPbSgDQSm8Ayv8GZl1ov1bn0sWitWOXMMf2B9/gAs4ZNW</latexit><latexit sha1_base64="h5ZKeLyvOAKrQ28BVy2eVsxqRMI=">AAAB+XicbVBNS8NAEJ3Ur1q/oh69LBbBU0lE0ItQ9OKxgv2ANpTNZtMu3WzC7qZYQv6JFw+KePWfePPfuGlz0NYHA4/3ZpiZ5yecKe0431ZlbX1jc6u6XdvZ3ds/sA+POipOJaFtEvNY9nysKGeCtjXTnPYSSXHkc9r1J3eF351SqVgsHvUsoV6ER4KFjGBtpKFtD0KJSRaEeRY85egGDe2603DmQKvELUkdSrSG9tcgiEkaUaEJx0r1XSfRXoalZoTTvDZIFU0wmeAR7RsqcESVl80vz9GZUQIUxtKU0Giu/p7IcKTULPJNZ4T1WC17hfif1091eO1lTCSppoIsFoUpRzpGRQwoYJISzWeGYCKZuRWRMTZRaBNWzYTgLr+8SjoXDddpuA+X9eZtGUcVTuAUzsGFK2jCPbSgDQSm8Ayv8GZl1ov1bn0sWitWOXMMf2B9/gAs4ZNW</latexit><latexit sha1_base64="h5ZKeLyvOAKrQ28BVy2eVsxqRMI=">AAAB+XicbVBNS8NAEJ3Ur1q/oh69LBbBU0lE0ItQ9OKxgv2ANpTNZtMu3WzC7qZYQv6JFw+KePWfePPfuGlz0NYHA4/3ZpiZ5yecKe0431ZlbX1jc6u6XdvZ3ds/sA+POipOJaFtEvNY9nysKGeCtjXTnPYSSXHkc9r1J3eF351SqVgsHvUsoV6ER4KFjGBtpKFtD0KJSRaEeRY85egGDe2603DmQKvELUkdSrSG9tcgiEkaUaEJx0r1XSfRXoalZoTTvDZIFU0wmeAR7RsqcESVl80vz9GZUQIUxtKU0Giu/p7IcKTULPJNZ4T1WC17hfif1091eO1lTCSppoIsFoUpRzpGRQwoYJISzWeGYCKZuRWRMTZRaBNWzYTgLr+8SjoXDddpuA+X9eZtGUcVTuAUzsGFK2jCPbSgDQSm8Ayv8GZl1ov1bn0sWitWOXMMf2B9/gAs4ZNW</latexit>

1

x<latexit sha1_base64="TH5NZEjext0lzaAh2MLrjaohrl0=">AAAB8nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaUDbbTbt0swm7E7GE/AwvHhTx6q/x5r9x2+agrQ8GHu/NMDMvSKQw6LrfTmltfWNzq7xd2dnd2z+oHh61TZxqxlsslrHuBtRwKRRvoUDJu4nmNAok7wST25nfeeTaiFg94DThfkRHSoSCUbRSrx9qyjIvz57yQbXm1t05yCrxClKDAs1B9as/jFkacYVMUmN6npugn1GNgkmeV/qp4QllEzriPUsVjbjxs/nJOTmzypCEsbalkMzV3xMZjYyZRoHtjCiOzbI3E//zeimG134mVJIiV2yxKEwlwZjM/idDoTlDObWEMi3srYSNqU0BbUoVG4K3/PIqaV/UPbfu3V/WGjdFHGU4gVM4Bw+uoAF30IQWMIjhGV7hzUHnxXl3PhatJaeYOYY/cD5/AKOekXk=</latexit><latexit sha1_base64="TH5NZEjext0lzaAh2MLrjaohrl0=">AAAB8nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaUDbbTbt0swm7E7GE/AwvHhTx6q/x5r9x2+agrQ8GHu/NMDMvSKQw6LrfTmltfWNzq7xd2dnd2z+oHh61TZxqxlsslrHuBtRwKRRvoUDJu4nmNAok7wST25nfeeTaiFg94DThfkRHSoSCUbRSrx9qyjIvz57yQbXm1t05yCrxClKDAs1B9as/jFkacYVMUmN6npugn1GNgkmeV/qp4QllEzriPUsVjbjxs/nJOTmzypCEsbalkMzV3xMZjYyZRoHtjCiOzbI3E//zeimG134mVJIiV2yxKEwlwZjM/idDoTlDObWEMi3srYSNqU0BbUoVG4K3/PIqaV/UPbfu3V/WGjdFHGU4gVM4Bw+uoAF30IQWMIjhGV7hzUHnxXl3PhatJaeYOYY/cD5/AKOekXk=</latexit><latexit sha1_base64="TH5NZEjext0lzaAh2MLrjaohrl0=">AAAB8nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaUDbbTbt0swm7E7GE/AwvHhTx6q/x5r9x2+agrQ8GHu/NMDMvSKQw6LrfTmltfWNzq7xd2dnd2z+oHh61TZxqxlsslrHuBtRwKRRvoUDJu4nmNAok7wST25nfeeTaiFg94DThfkRHSoSCUbRSrx9qyjIvz57yQbXm1t05yCrxClKDAs1B9as/jFkacYVMUmN6npugn1GNgkmeV/qp4QllEzriPUsVjbjxs/nJOTmzypCEsbalkMzV3xMZjYyZRoHtjCiOzbI3E//zeimG134mVJIiV2yxKEwlwZjM/idDoTlDObWEMi3srYSNqU0BbUoVG4K3/PIqaV/UPbfu3V/WGjdFHGU4gVM4Bw+uoAF30IQWMIjhGV7hzUHnxXl3PhatJaeYOYY/cD5/AKOekXk=</latexit><latexit sha1_base64="TH5NZEjext0lzaAh2MLrjaohrl0=">AAAB8nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaUDbbTbt0swm7E7GE/AwvHhTx6q/x5r9x2+agrQ8GHu/NMDMvSKQw6LrfTmltfWNzq7xd2dnd2z+oHh61TZxqxlsslrHuBtRwKRRvoUDJu4nmNAok7wST25nfeeTaiFg94DThfkRHSoSCUbRSrx9qyjIvz57yQbXm1t05yCrxClKDAs1B9as/jFkacYVMUmN6npugn1GNgkmeV/qp4QllEzriPUsVjbjxs/nJOTmzypCEsbalkMzV3xMZjYyZRoHtjCiOzbI3E//zeimG134mVJIiV2yxKEwlwZjM/idDoTlDObWEMi3srYSNqU0BbUoVG4K3/PIqaV/UPbfu3V/WGjdFHGU4gVM4Bw+uoAF30IQWMIjhGV7hzUHnxXl3PhatJaeYOYY/cD5/AKOekXk=</latexit>

a<latexit sha1_base64="YWJIstkf+pdgisEWpHWEe8hMiDM=">AAAB8XicbVDLSsNAFL2pr1pfVZduBovgqiQi6LLoxmUF+8A2lMl00g6dTMLMjVBC/8KNC0Xc+jfu/BsnbRbaemDgcM69zLknSKQw6LrfTmltfWNzq7xd2dnd2z+oHh61TZxqxlsslrHuBtRwKRRvoUDJu4nmNAok7wST29zvPHFtRKwecJpwP6IjJULBKFrpsR9RHAdhRmeDas2tu3OQVeIVpAYFmoPqV38YszTiCpmkxvQ8N0E/oxoFk3xW6aeGJ5RN6Ij3LFU04sbP5oln5MwqQxLG2j6FZK7+3shoZMw0CuxkntAse7n4n9dLMbz2M6GSFLlii4/CVBKMSX4+GQrNGcqpJZRpYbMSNqaaMrQlVWwJ3vLJq6R9Uffcund/WWvcFHWU4QRO4Rw8uIIG3EETWsBAwTO8wptjnBfn3flYjJacYucY/sD5/AHaRpEF</latexit><latexit sha1_base64="YWJIstkf+pdgisEWpHWEe8hMiDM=">AAAB8XicbVDLSsNAFL2pr1pfVZduBovgqiQi6LLoxmUF+8A2lMl00g6dTMLMjVBC/8KNC0Xc+jfu/BsnbRbaemDgcM69zLknSKQw6LrfTmltfWNzq7xd2dnd2z+oHh61TZxqxlsslrHuBtRwKRRvoUDJu4nmNAok7wST29zvPHFtRKwecJpwP6IjJULBKFrpsR9RHAdhRmeDas2tu3OQVeIVpAYFmoPqV38YszTiCpmkxvQ8N0E/oxoFk3xW6aeGJ5RN6Ij3LFU04sbP5oln5MwqQxLG2j6FZK7+3shoZMw0CuxkntAse7n4n9dLMbz2M6GSFLlii4/CVBKMSX4+GQrNGcqpJZRpYbMSNqaaMrQlVWwJ3vLJq6R9Uffcund/WWvcFHWU4QRO4Rw8uIIG3EETWsBAwTO8wptjnBfn3flYjJacYucY/sD5/AHaRpEF</latexit><latexit sha1_base64="YWJIstkf+pdgisEWpHWEe8hMiDM=">AAAB8XicbVDLSsNAFL2pr1pfVZduBovgqiQi6LLoxmUF+8A2lMl00g6dTMLMjVBC/8KNC0Xc+jfu/BsnbRbaemDgcM69zLknSKQw6LrfTmltfWNzq7xd2dnd2z+oHh61TZxqxlsslrHuBtRwKRRvoUDJu4nmNAok7wST29zvPHFtRKwecJpwP6IjJULBKFrpsR9RHAdhRmeDas2tu3OQVeIVpAYFmoPqV38YszTiCpmkxvQ8N0E/oxoFk3xW6aeGJ5RN6Ij3LFU04sbP5oln5MwqQxLG2j6FZK7+3shoZMw0CuxkntAse7n4n9dLMbz2M6GSFLlii4/CVBKMSX4+GQrNGcqpJZRpYbMSNqaaMrQlVWwJ3vLJq6R9Uffcund/WWvcFHWU4QRO4Rw8uIIG3EETWsBAwTO8wptjnBfn3flYjJacYucY/sD5/AHaRpEF</latexit><latexit sha1_base64="YWJIstkf+pdgisEWpHWEe8hMiDM=">AAAB8XicbVDLSsNAFL2pr1pfVZduBovgqiQi6LLoxmUF+8A2lMl00g6dTMLMjVBC/8KNC0Xc+jfu/BsnbRbaemDgcM69zLknSKQw6LrfTmltfWNzq7xd2dnd2z+oHh61TZxqxlsslrHuBtRwKRRvoUDJu4nmNAok7wST29zvPHFtRKwecJpwP6IjJULBKFrpsR9RHAdhRmeDas2tu3OQVeIVpAYFmoPqV38YszTiCpmkxvQ8N0E/oxoFk3xW6aeGJ5RN6Ij3LFU04sbP5oln5MwqQxLG2j6FZK7+3shoZMw0CuxkntAse7n4n9dLMbz2M6GSFLlii4/CVBKMSX4+GQrNGcqpJZRpYbMSNqaaMrQlVWwJ3vLJq6R9Uffcund/WWvcFHWU4QRO4Rw8uIIG3EETWsBAwTO8wptjnBfn3flYjJacYucY/sD5/AHaRpEF</latexit>

f(x) = f1(f2(x))<latexit sha1_base64="UQyNu+ga8UtYHEVSjuzYlakaBow=">AAAB+3icbVDLSsNAFL3xWesr1qWbwSK0m5IUQTdC0Y3LCvYBbQiT6aQdOpmEmYm0hP6KGxeKuPVH3Pk3TtsstPXAhTPn3Mvce4KEM6Ud59va2Nza3tkt7BX3Dw6Pju2TUlvFqSS0RWIey26AFeVM0JZmmtNuIimOAk47wfhu7neeqFQsFo96mlAvwkPBQkawNpJvl8LKpIpuUOi7ldCvm0fVt8tOzVkArRM3J2XI0fTtr/4gJmlEhSYcK9VznUR7GZaaEU5nxX6qaILJGA9pz1CBI6q8bLH7DF0YZYDCWJoSGi3U3xMZjpSaRoHpjLAeqVVvLv7n9VIdXnsZE0mqqSDLj8KUIx2jeRBowCQlmk8NwUQysysiIywx0SauognBXT15nbTrNdepuQ+X5cZtHkcBzuAcKuDCFTTgHprQAgITeIZXeLNm1ov1bn0sWzesfOYU/sD6/AFldJIS</latexit><latexit sha1_base64="UQyNu+ga8UtYHEVSjuzYlakaBow=">AAAB+3icbVDLSsNAFL3xWesr1qWbwSK0m5IUQTdC0Y3LCvYBbQiT6aQdOpmEmYm0hP6KGxeKuPVH3Pk3TtsstPXAhTPn3Mvce4KEM6Ud59va2Nza3tkt7BX3Dw6Pju2TUlvFqSS0RWIey26AFeVM0JZmmtNuIimOAk47wfhu7neeqFQsFo96mlAvwkPBQkawNpJvl8LKpIpuUOi7ldCvm0fVt8tOzVkArRM3J2XI0fTtr/4gJmlEhSYcK9VznUR7GZaaEU5nxX6qaILJGA9pz1CBI6q8bLH7DF0YZYDCWJoSGi3U3xMZjpSaRoHpjLAeqVVvLv7n9VIdXnsZE0mqqSDLj8KUIx2jeRBowCQlmk8NwUQysysiIywx0SauognBXT15nbTrNdepuQ+X5cZtHkcBzuAcKuDCFTTgHprQAgITeIZXeLNm1ov1bn0sWzesfOYU/sD6/AFldJIS</latexit><latexit sha1_base64="UQyNu+ga8UtYHEVSjuzYlakaBow=">AAAB+3icbVDLSsNAFL3xWesr1qWbwSK0m5IUQTdC0Y3LCvYBbQiT6aQdOpmEmYm0hP6KGxeKuPVH3Pk3TtsstPXAhTPn3Mvce4KEM6Ud59va2Nza3tkt7BX3Dw6Pju2TUlvFqSS0RWIey26AFeVM0JZmmtNuIimOAk47wfhu7neeqFQsFo96mlAvwkPBQkawNpJvl8LKpIpuUOi7ldCvm0fVt8tOzVkArRM3J2XI0fTtr/4gJmlEhSYcK9VznUR7GZaaEU5nxX6qaILJGA9pz1CBI6q8bLH7DF0YZYDCWJoSGi3U3xMZjpSaRoHpjLAeqVVvLv7n9VIdXnsZE0mqqSDLj8KUIx2jeRBowCQlmk8NwUQysysiIywx0SauognBXT15nbTrNdepuQ+X5cZtHkcBzuAcKuDCFTTgHprQAgITeIZXeLNm1ov1bn0sWzesfOYU/sD6/AFldJIS</latexit><latexit sha1_base64="UQyNu+ga8UtYHEVSjuzYlakaBow=">AAAB+3icbVDLSsNAFL3xWesr1qWbwSK0m5IUQTdC0Y3LCvYBbQiT6aQdOpmEmYm0hP6KGxeKuPVH3Pk3TtsstPXAhTPn3Mvce4KEM6Ud59va2Nza3tkt7BX3Dw6Pju2TUlvFqSS0RWIey26AFeVM0JZmmtNuIimOAk47wfhu7neeqFQsFo96mlAvwkPBQkawNpJvl8LKpIpuUOi7ldCvm0fVt8tOzVkArRM3J2XI0fTtr/4gJmlEhSYcK9VznUR7GZaaEU5nxX6qaILJGA9pz1CBI6q8bLH7DF0YZYDCWJoSGi3U3xMZjpSaRoHpjLAeqVVvLv7n9VIdXnsZE0mqqSDLj8KUIx2jeRBowCQlmk8NwUQysysiIywx0SauognBXT15nbTrNdepuQ+X5cZtHkcBzuAcKuDCFTTgHprQAgITeIZXeLNm1ov1bn0sWzesfOYU/sD6/AFldJIS</latexit>

exp(x)<latexit sha1_base64="tpobf1ys+qWre7ORxkaLDyz7vVU=">AAAB7nicbVBNS8NAEJ34WetX1aOXxSLUS0lE0GPRi8cK9gPaUDbbSbt0swm7G2kJ/RFePCji1d/jzX/jts1BWx8MPN6bYWZekAiujet+O2vrG5tb24Wd4u7e/sFh6ei4qeNUMWywWMSqHVCNgktsGG4EthOFNAoEtoLR3cxvPaHSPJaPZpKgH9GB5CFn1Fip1cVxUhlf9Eplt+rOQVaJl5My5Kj3Sl/dfszSCKVhgmrd8dzE+BlVhjOB02I31ZhQNqID7FgqaYTaz+bnTsm5VfokjJUtachc/T2R0UjrSRTYzoiaoV72ZuJ/Xic14Y2fcZmkBiVbLApTQUxMZr+TPlfIjJhYQpni9lbChlRRZmxCRRuCt/zyKmleVj236j1clWu3eRwFOIUzqIAH11CDe6hDAxiM4Ble4c1JnBfn3flYtK45+cwJ/IHz+QPK7I8y</latexit><latexit sha1_base64="tpobf1ys+qWre7ORxkaLDyz7vVU=">AAAB7nicbVBNS8NAEJ34WetX1aOXxSLUS0lE0GPRi8cK9gPaUDbbSbt0swm7G2kJ/RFePCji1d/jzX/jts1BWx8MPN6bYWZekAiujet+O2vrG5tb24Wd4u7e/sFh6ei4qeNUMWywWMSqHVCNgktsGG4EthOFNAoEtoLR3cxvPaHSPJaPZpKgH9GB5CFn1Fip1cVxUhlf9Eplt+rOQVaJl5My5Kj3Sl/dfszSCKVhgmrd8dzE+BlVhjOB02I31ZhQNqID7FgqaYTaz+bnTsm5VfokjJUtachc/T2R0UjrSRTYzoiaoV72ZuJ/Xic14Y2fcZmkBiVbLApTQUxMZr+TPlfIjJhYQpni9lbChlRRZmxCRRuCt/zyKmleVj236j1clWu3eRwFOIUzqIAH11CDe6hDAxiM4Ble4c1JnBfn3flYtK45+cwJ/IHz+QPK7I8y</latexit><latexit sha1_base64="tpobf1ys+qWre7ORxkaLDyz7vVU=">AAAB7nicbVBNS8NAEJ34WetX1aOXxSLUS0lE0GPRi8cK9gPaUDbbSbt0swm7G2kJ/RFePCji1d/jzX/jts1BWx8MPN6bYWZekAiujet+O2vrG5tb24Wd4u7e/sFh6ei4qeNUMWywWMSqHVCNgktsGG4EthOFNAoEtoLR3cxvPaHSPJaPZpKgH9GB5CFn1Fip1cVxUhlf9Eplt+rOQVaJl5My5Kj3Sl/dfszSCKVhgmrd8dzE+BlVhjOB02I31ZhQNqID7FgqaYTaz+bnTsm5VfokjJUtachc/T2R0UjrSRTYzoiaoV72ZuJ/Xic14Y2fcZmkBiVbLApTQUxMZr+TPlfIjJhYQpni9lbChlRRZmxCRRuCt/zyKmleVj236j1clWu3eRwFOIUzqIAH11CDe6hDAxiM4Ble4c1JnBfn3flYtK45+cwJ/IHz+QPK7I8y</latexit><latexit sha1_base64="tpobf1ys+qWre7ORxkaLDyz7vVU=">AAAB7nicbVBNS8NAEJ34WetX1aOXxSLUS0lE0GPRi8cK9gPaUDbbSbt0swm7G2kJ/RFePCji1d/jzX/jts1BWx8MPN6bYWZekAiujet+O2vrG5tb24Wd4u7e/sFh6ei4qeNUMWywWMSqHVCNgktsGG4EthOFNAoEtoLR3cxvPaHSPJaPZpKgH9GB5CFn1Fip1cVxUhlf9Eplt+rOQVaJl5My5Kj3Sl/dfszSCKVhgmrd8dzE+BlVhjOB02I31ZhQNqID7FgqaYTaz+bnTsm5VfokjJUtachc/T2R0UjrSRTYzoiaoV72ZuJ/Xic14Y2fcZmkBiVbLApTQUxMZr+TPlfIjJhYQpni9lbChlRRZmxCRRuCt/zyKmleVj236j1clWu3eRwFOIUzqIAH11CDe6hDAxiM4Ble4c1JnBfn3flYtK45+cwJ/IHz+QPK7I8y</latexit>

df1(z)

dz

df2(x)

dx<latexit sha1_base64="vJFFXvrFxkPqtRMpq3Ovj3XweH0=">AAACEnicbZDLSsNAFIYn9VbrLerSzWAR2k1JiqDLohuXFewF2hAm00k7dDIJMxNpG/IMbnwVNy4UcevKnW/jNA2orT8M/HznHM6c34sYlcqyvozC2vrG5lZxu7Szu7d/YB4etWUYC0xaOGSh6HpIEkY5aSmqGOlGgqDAY6Tjja/n9c49EZKG/E5NI+IEaMipTzFSGrlmte8LhJMB9F27Mqum2s1S+APrlUkGJ6lrlq2alQmuGjs3ZZCr6Zqf/UGI44BwhRmSsmdbkXISJBTFjKSlfixJhPAYDUlPW44CIp0kOymFZ5ro/aHQjyuY0d8TCQqknAae7gyQGsnl2hz+V+vFyr90EsqjWBGOF4v8mEEVwnk+cEAFwYpNtUFYUP1XiEdIx6F0iiUdgr188qpp12u2VbNvz8uNqzyOIjgBp6ACbHABGuAGNEELYPAAnsALeDUejWfjzXhftBaMfOYY/JHx8Q1+pZy4</latexit><latexit sha1_base64="vJFFXvrFxkPqtRMpq3Ovj3XweH0=">AAACEnicbZDLSsNAFIYn9VbrLerSzWAR2k1JiqDLohuXFewF2hAm00k7dDIJMxNpG/IMbnwVNy4UcevKnW/jNA2orT8M/HznHM6c34sYlcqyvozC2vrG5lZxu7Szu7d/YB4etWUYC0xaOGSh6HpIEkY5aSmqGOlGgqDAY6Tjja/n9c49EZKG/E5NI+IEaMipTzFSGrlmte8LhJMB9F27Mqum2s1S+APrlUkGJ6lrlq2alQmuGjs3ZZCr6Zqf/UGI44BwhRmSsmdbkXISJBTFjKSlfixJhPAYDUlPW44CIp0kOymFZ5ro/aHQjyuY0d8TCQqknAae7gyQGsnl2hz+V+vFyr90EsqjWBGOF4v8mEEVwnk+cEAFwYpNtUFYUP1XiEdIx6F0iiUdgr188qpp12u2VbNvz8uNqzyOIjgBp6ACbHABGuAGNEELYPAAnsALeDUejWfjzXhftBaMfOYY/JHx8Q1+pZy4</latexit><latexit sha1_base64="vJFFXvrFxkPqtRMpq3Ovj3XweH0=">AAACEnicbZDLSsNAFIYn9VbrLerSzWAR2k1JiqDLohuXFewF2hAm00k7dDIJMxNpG/IMbnwVNy4UcevKnW/jNA2orT8M/HznHM6c34sYlcqyvozC2vrG5lZxu7Szu7d/YB4etWUYC0xaOGSh6HpIEkY5aSmqGOlGgqDAY6Tjja/n9c49EZKG/E5NI+IEaMipTzFSGrlmte8LhJMB9F27Mqum2s1S+APrlUkGJ6lrlq2alQmuGjs3ZZCr6Zqf/UGI44BwhRmSsmdbkXISJBTFjKSlfixJhPAYDUlPW44CIp0kOymFZ5ro/aHQjyuY0d8TCQqknAae7gyQGsnl2hz+V+vFyr90EsqjWBGOF4v8mEEVwnk+cEAFwYpNtUFYUP1XiEdIx6F0iiUdgr188qpp12u2VbNvz8uNqzyOIjgBp6ACbHABGuAGNEELYPAAnsALeDUejWfjzXhftBaMfOYY/JHx8Q1+pZy4</latexit><latexit sha1_base64="vJFFXvrFxkPqtRMpq3Ovj3XweH0=">AAACEnicbZDLSsNAFIYn9VbrLerSzWAR2k1JiqDLohuXFewF2hAm00k7dDIJMxNpG/IMbnwVNy4UcevKnW/jNA2orT8M/HznHM6c34sYlcqyvozC2vrG5lZxu7Szu7d/YB4etWUYC0xaOGSh6HpIEkY5aSmqGOlGgqDAY6Tjja/n9c49EZKG/E5NI+IEaMipTzFSGrlmte8LhJMB9F27Mqum2s1S+APrlUkGJ6lrlq2alQmuGjs3ZZCr6Zqf/UGI44BwhRmSsmdbkXISJBTFjKSlfixJhPAYDUlPW44CIp0kOymFZ5ro/aHQjyuY0d8TCQqknAae7gyQGsnl2hz+V+vFyr90EsqjWBGOF4v8mEEVwnk+cEAFwYpNtUFYUP1XiEdIx6F0iiUdgr188qpp12u2VbNvz8uNqzyOIjgBp6ACbHABGuAGNEELYPAAnsALeDUejWfjzXhftBaMfOYY/JHx8Q1+pZy4</latexit>

z = f2(x)<latexit sha1_base64="ZdgaFkUBPwxKYGMvfRhZ20lQlgE=">AAAB8XicbVBNSwMxEJ2tX7V+VT16CRahXspuEfQiFL14rGA/sF1KNs22odlkSbJiXfovvHhQxKv/xpv/xrTdg7Y+GHi8N8PMvCDmTBvX/XZyK6tr6xv5zcLW9s7uXnH/oKlloghtEMmlagdYU84EbRhmOG3HiuIo4LQVjK6nfuuBKs2kuDPjmPoRHggWMoKNle6f0CUKe9Xy42mvWHIr7gxomXgZKUGGeq/41e1LkkRUGMKx1h3PjY2fYmUY4XRS6CaaxpiM8IB2LBU4otpPZxdP0IlV+iiUypYwaKb+nkhxpPU4CmxnhM1QL3pT8T+vk5jwwk+ZiBNDBZkvChOOjETT91GfKUoMH1uCiWL2VkSGWGFibEgFG4K3+PIyaVYrnlvxbs9KtassjjwcwTGUwYNzqMEN1KEBBAQ8wyu8Odp5cd6dj3lrzslmDuEPnM8fptmPlQ==</latexit><latexit sha1_base64="ZdgaFkUBPwxKYGMvfRhZ20lQlgE=">AAAB8XicbVBNSwMxEJ2tX7V+VT16CRahXspuEfQiFL14rGA/sF1KNs22odlkSbJiXfovvHhQxKv/xpv/xrTdg7Y+GHi8N8PMvCDmTBvX/XZyK6tr6xv5zcLW9s7uXnH/oKlloghtEMmlagdYU84EbRhmOG3HiuIo4LQVjK6nfuuBKs2kuDPjmPoRHggWMoKNle6f0CUKe9Xy42mvWHIr7gxomXgZKUGGeq/41e1LkkRUGMKx1h3PjY2fYmUY4XRS6CaaxpiM8IB2LBU4otpPZxdP0IlV+iiUypYwaKb+nkhxpPU4CmxnhM1QL3pT8T+vk5jwwk+ZiBNDBZkvChOOjETT91GfKUoMH1uCiWL2VkSGWGFibEgFG4K3+PIyaVYrnlvxbs9KtassjjwcwTGUwYNzqMEN1KEBBAQ8wyu8Odp5cd6dj3lrzslmDuEPnM8fptmPlQ==</latexit><latexit sha1_base64="ZdgaFkUBPwxKYGMvfRhZ20lQlgE=">AAAB8XicbVBNSwMxEJ2tX7V+VT16CRahXspuEfQiFL14rGA/sF1KNs22odlkSbJiXfovvHhQxKv/xpv/xrTdg7Y+GHi8N8PMvCDmTBvX/XZyK6tr6xv5zcLW9s7uXnH/oKlloghtEMmlagdYU84EbRhmOG3HiuIo4LQVjK6nfuuBKs2kuDPjmPoRHggWMoKNle6f0CUKe9Xy42mvWHIr7gxomXgZKUGGeq/41e1LkkRUGMKx1h3PjY2fYmUY4XRS6CaaxpiM8IB2LBU4otpPZxdP0IlV+iiUypYwaKb+nkhxpPU4CmxnhM1QL3pT8T+vk5jwwk+ZiBNDBZkvChOOjETT91GfKUoMH1uCiWL2VkSGWGFibEgFG4K3+PIyaVYrnlvxbs9KtassjjwcwTGUwYNzqMEN1KEBBAQ8wyu8Odp5cd6dj3lrzslmDuEPnM8fptmPlQ==</latexit><latexit sha1_base64="ZdgaFkUBPwxKYGMvfRhZ20lQlgE=">AAAB8XicbVBNSwMxEJ2tX7V+VT16CRahXspuEfQiFL14rGA/sF1KNs22odlkSbJiXfovvHhQxKv/xpv/xrTdg7Y+GHi8N8PMvCDmTBvX/XZyK6tr6xv5zcLW9s7uXnH/oKlloghtEMmlagdYU84EbRhmOG3HiuIo4LQVjK6nfuuBKs2kuDPjmPoRHggWMoKNle6f0CUKe9Xy42mvWHIr7gxomXgZKUGGeq/41e1LkkRUGMKx1h3PjY2fYmUY4XRS6CaaxpiM8IB2LBU4otpPZxdP0IlV+iiUypYwaKb+nkhxpPU4CmxnhM1QL3pT8T+vk5jwwk+ZiBNDBZkvChOOjETT91GfKUoMH1uCiWL2VkSGWGFibEgFG4K3+PIyaVYrnlvxbs9KtassjjwcwTGUwYNzqMEN1KEBBAQ8wyu8Odp5cd6dj3lrzslmDuEPnM8fptmPlQ==</latexit>

df

dx=

<latexit sha1_base64="h5ZKeLyvOAKrQ28BVy2eVsxqRMI=">AAAB+XicbVBNS8NAEJ3Ur1q/oh69LBbBU0lE0ItQ9OKxgv2ANpTNZtMu3WzC7qZYQv6JFw+KePWfePPfuGlz0NYHA4/3ZpiZ5yecKe0431ZlbX1jc6u6XdvZ3ds/sA+POipOJaFtEvNY9nysKGeCtjXTnPYSSXHkc9r1J3eF351SqVgsHvUsoV6ER4KFjGBtpKFtD0KJSRaEeRY85egGDe2603DmQKvELUkdSrSG9tcgiEkaUaEJx0r1XSfRXoalZoTTvDZIFU0wmeAR7RsqcESVl80vz9GZUQIUxtKU0Giu/p7IcKTULPJNZ4T1WC17hfif1091eO1lTCSppoIsFoUpRzpGRQwoYJISzWeGYCKZuRWRMTZRaBNWzYTgLr+8SjoXDddpuA+X9eZtGUcVTuAUzsGFK2jCPbSgDQSm8Ayv8GZl1ov1bn0sWitWOXMMf2B9/gAs4ZNW</latexit><latexit sha1_base64="h5ZKeLyvOAKrQ28BVy2eVsxqRMI=">AAAB+XicbVBNS8NAEJ3Ur1q/oh69LBbBU0lE0ItQ9OKxgv2ANpTNZtMu3WzC7qZYQv6JFw+KePWfePPfuGlz0NYHA4/3ZpiZ5yecKe0431ZlbX1jc6u6XdvZ3ds/sA+POipOJaFtEvNY9nysKGeCtjXTnPYSSXHkc9r1J3eF351SqVgsHvUsoV6ER4KFjGBtpKFtD0KJSRaEeRY85egGDe2603DmQKvELUkdSrSG9tcgiEkaUaEJx0r1XSfRXoalZoTTvDZIFU0wmeAR7RsqcESVl80vz9GZUQIUxtKU0Giu/p7IcKTULPJNZ4T1WC17hfif1091eO1lTCSppoIsFoUpRzpGRQwoYJISzWeGYCKZuRWRMTZRaBNWzYTgLr+8SjoXDddpuA+X9eZtGUcVTuAUzsGFK2jCPbSgDQSm8Ayv8GZl1ov1bn0sWitWOXMMf2B9/gAs4ZNW</latexit><latexit sha1_base64="h5ZKeLyvOAKrQ28BVy2eVsxqRMI=">AAAB+XicbVBNS8NAEJ3Ur1q/oh69LBbBU0lE0ItQ9OKxgv2ANpTNZtMu3WzC7qZYQv6JFw+KePWfePPfuGlz0NYHA4/3ZpiZ5yecKe0431ZlbX1jc6u6XdvZ3ds/sA+POipOJaFtEvNY9nysKGeCtjXTnPYSSXHkc9r1J3eF351SqVgsHvUsoV6ER4KFjGBtpKFtD0KJSRaEeRY85egGDe2603DmQKvELUkdSrSG9tcgiEkaUaEJx0r1XSfRXoalZoTTvDZIFU0wmeAR7RsqcESVl80vz9GZUQIUxtKU0Giu/p7IcKTULPJNZ4T1WC17hfif1091eO1lTCSppoIsFoUpRzpGRQwoYJISzWeGYCKZuRWRMTZRaBNWzYTgLr+8SjoXDddpuA+X9eZtGUcVTuAUzsGFK2jCPbSgDQSm8Ayv8GZl1ov1bn0sWitWOXMMf2B9/gAs4ZNW</latexit><latexit sha1_base64="h5ZKeLyvOAKrQ28BVy2eVsxqRMI=">AAAB+XicbVBNS8NAEJ3Ur1q/oh69LBbBU0lE0ItQ9OKxgv2ANpTNZtMu3WzC7qZYQv6JFw+KePWfePPfuGlz0NYHA4/3ZpiZ5yecKe0431ZlbX1jc6u6XdvZ3ds/sA+POipOJaFtEvNY9nysKGeCtjXTnPYSSXHkc9r1J3eF351SqVgsHvUsoV6ER4KFjGBtpKFtD0KJSRaEeRY85egGDe2603DmQKvELUkdSrSG9tcgiEkaUaEJx0r1XSfRXoalZoTTvDZIFU0wmeAR7RsqcESVl80vz9GZUQIUxtKU0Giu/p7IcKTULPJNZ4T1WC17hfif1091eO1lTCSppoIsFoUpRzpGRQwoYJISzWeGYCKZuRWRMTZRaBNWzYTgLr+8SjoXDddpuA+X9eZtGUcVTuAUzsGFK2jCPbSgDQSm8Ayv8GZl1ov1bn0sWitWOXMMf2B9/gAs4ZNW</latexit>

chain rule:

f(x) = x · a<latexit sha1_base64="bbSHyazU/jOeOwDXwnpe5WIq6Ns=">AAACFHicbVDLSsNAFJ3UV62vqEs3g0WoCCURQTdC0Y3LCvYBTSiTyaQdOsmEmYlYQj7Cjb/ixoUibl2482+ctMFH64GBM+fcy733eDGjUlnWp1FaWFxaXimvVtbWNza3zO2dtuSJwKSFOeOi6yFJGI1IS1HFSDcWBIUeIx1vdJn7nVsiJOXRjRrHxA3RIKIBxUhpqW8eBTUnRGroBelddgjP4c8POtjn6ltAWd+sWnVrAjhP7IJUQYFm3/xwfI6TkEQKMyRlz7Zi5aZIKIoZySpOIkmM8AgNSE/TCIVEuunkqAweaMWHARf6RQpO1N8dKQqlHIeersw3lLNeLv7n9RIVnLkpjeJEkQhPBwUJg4rDPCHoU0GwYmNNEBZU7wrxEAmElc6xokOwZ0+eJ+3jum3V7euTauOiiKMM9sA+qAEbnIIGuAJN0AIY3INH8AxejAfjyXg13qalJaPo2QV/YLx/AXZ6nnY=</latexit><latexit sha1_base64="bbSHyazU/jOeOwDXwnpe5WIq6Ns=">AAACFHicbVDLSsNAFJ3UV62vqEs3g0WoCCURQTdC0Y3LCvYBTSiTyaQdOsmEmYlYQj7Cjb/ixoUibl2482+ctMFH64GBM+fcy733eDGjUlnWp1FaWFxaXimvVtbWNza3zO2dtuSJwKSFOeOi6yFJGI1IS1HFSDcWBIUeIx1vdJn7nVsiJOXRjRrHxA3RIKIBxUhpqW8eBTUnRGroBelddgjP4c8POtjn6ltAWd+sWnVrAjhP7IJUQYFm3/xwfI6TkEQKMyRlz7Zi5aZIKIoZySpOIkmM8AgNSE/TCIVEuunkqAweaMWHARf6RQpO1N8dKQqlHIeersw3lLNeLv7n9RIVnLkpjeJEkQhPBwUJg4rDPCHoU0GwYmNNEBZU7wrxEAmElc6xokOwZ0+eJ+3jum3V7euTauOiiKMM9sA+qAEbnIIGuAJN0AIY3INH8AxejAfjyXg13qalJaPo2QV/YLx/AXZ6nnY=</latexit><latexit sha1_base64="bbSHyazU/jOeOwDXwnpe5WIq6Ns=">AAACFHicbVDLSsNAFJ3UV62vqEs3g0WoCCURQTdC0Y3LCvYBTSiTyaQdOsmEmYlYQj7Cjb/ixoUibl2482+ctMFH64GBM+fcy733eDGjUlnWp1FaWFxaXimvVtbWNza3zO2dtuSJwKSFOeOi6yFJGI1IS1HFSDcWBIUeIx1vdJn7nVsiJOXRjRrHxA3RIKIBxUhpqW8eBTUnRGroBelddgjP4c8POtjn6ltAWd+sWnVrAjhP7IJUQYFm3/xwfI6TkEQKMyRlz7Zi5aZIKIoZySpOIkmM8AgNSE/TCIVEuunkqAweaMWHARf6RQpO1N8dKQqlHIeersw3lLNeLv7n9RIVnLkpjeJEkQhPBwUJg4rDPCHoU0GwYmNNEBZU7wrxEAmElc6xokOwZ0+eJ+3jum3V7euTauOiiKMM9sA+qAEbnIIGuAJN0AIY3INH8AxejAfjyXg13qalJaPo2QV/YLx/AXZ6nnY=</latexit><latexit sha1_base64="bbSHyazU/jOeOwDXwnpe5WIq6Ns=">AAACFHicbVDLSsNAFJ3UV62vqEs3g0WoCCURQTdC0Y3LCvYBTSiTyaQdOsmEmYlYQj7Cjb/ixoUibl2482+ctMFH64GBM+fcy733eDGjUlnWp1FaWFxaXimvVtbWNza3zO2dtuSJwKSFOeOi6yFJGI1IS1HFSDcWBIUeIx1vdJn7nVsiJOXRjRrHxA3RIKIBxUhpqW8eBTUnRGroBelddgjP4c8POtjn6ltAWd+sWnVrAjhP7IJUQYFm3/xwfI6TkEQKMyRlz7Zi5aZIKIoZySpOIkmM8AgNSE/TCIVEuunkqAweaMWHARf6RQpO1N8dKQqlHIeersw3lLNeLv7n9RIVnLkpjeJEkQhPBwUJg4rDPCHoU0GwYmNNEBZU7wrxEAmElc6xokOwZ0+eJ+3jum3V7euTauOiiKMM9sA+qAEbnIIGuAJN0AIY3INH8AxejAfjyXg13qalJaPo2QV/YLx/AXZ6nnY=</latexit>

@f

@x=

<latexit sha1_base64="ChLVskzOTZDfN1yqMFtqglj0AMY=">AAACEnicbVDLSsNAFL3xWesr6tLNYBF0UxIRdCMU3bisYB/QlDKZTtqhk0mYmYgl5Bvc+CtuXCji1pU7/8ZJG1BbDwwczrn3zr3HjzlT2nG+rIXFpeWV1dJaeX1jc2vb3tltqiiRhDZIxCPZ9rGinAna0Exz2o4lxaHPacsfXeV+645KxSJxq8cx7YZ4IFjACNZG6tnHXiAxSb0YS80wR0H2w70Q66EfpPdZhi5Qz644VWcCNE/cglSgQL1nf3r9iCQhFZpwrFTHdWLdTfPhhNOs7CWKxpiM8IB2DBU4pKqbTk7K0KFR+iiIpHlCo4n6uyPFoVLj0DeV+ZZq1svF/7xOooPzbspEnGgqyPSjIOFIRyjPB/WZpETzsSGYSGZ2RWSITUbapFg2IbizJ8+T5knVdaruzWmldlnEUYJ9OIAjcOEManANdWgAgQd4ghd4tR6tZ+vNep+WLlhFzx78gfXxDcXVniA=</latexit><latexit sha1_base64="ChLVskzOTZDfN1yqMFtqglj0AMY=">AAACEnicbVDLSsNAFL3xWesr6tLNYBF0UxIRdCMU3bisYB/QlDKZTtqhk0mYmYgl5Bvc+CtuXCji1pU7/8ZJG1BbDwwczrn3zr3HjzlT2nG+rIXFpeWV1dJaeX1jc2vb3tltqiiRhDZIxCPZ9rGinAna0Exz2o4lxaHPacsfXeV+645KxSJxq8cx7YZ4IFjACNZG6tnHXiAxSb0YS80wR0H2w70Q66EfpPdZhi5Qz644VWcCNE/cglSgQL1nf3r9iCQhFZpwrFTHdWLdTfPhhNOs7CWKxpiM8IB2DBU4pKqbTk7K0KFR+iiIpHlCo4n6uyPFoVLj0DeV+ZZq1svF/7xOooPzbspEnGgqyPSjIOFIRyjPB/WZpETzsSGYSGZ2RWSITUbapFg2IbizJ8+T5knVdaruzWmldlnEUYJ9OIAjcOEManANdWgAgQd4ghd4tR6tZ+vNep+WLlhFzx78gfXxDcXVniA=</latexit><latexit sha1_base64="ChLVskzOTZDfN1yqMFtqglj0AMY=">AAACEnicbVDLSsNAFL3xWesr6tLNYBF0UxIRdCMU3bisYB/QlDKZTtqhk0mYmYgl5Bvc+CtuXCji1pU7/8ZJG1BbDwwczrn3zr3HjzlT2nG+rIXFpeWV1dJaeX1jc2vb3tltqiiRhDZIxCPZ9rGinAna0Exz2o4lxaHPacsfXeV+645KxSJxq8cx7YZ4IFjACNZG6tnHXiAxSb0YS80wR0H2w70Q66EfpPdZhi5Qz644VWcCNE/cglSgQL1nf3r9iCQhFZpwrFTHdWLdTfPhhNOs7CWKxpiM8IB2DBU4pKqbTk7K0KFR+iiIpHlCo4n6uyPFoVLj0DeV+ZZq1svF/7xOooPzbspEnGgqyPSjIOFIRyjPB/WZpETzsSGYSGZ2RWSITUbapFg2IbizJ8+T5knVdaruzWmldlnEUYJ9OIAjcOEManANdWgAgQd4ghd4tR6tZ+vNep+WLlhFzx78gfXxDcXVniA=</latexit><latexit sha1_base64="ChLVskzOTZDfN1yqMFtqglj0AMY=">AAACEnicbVDLSsNAFL3xWesr6tLNYBF0UxIRdCMU3bisYB/QlDKZTtqhk0mYmYgl5Bvc+CtuXCji1pU7/8ZJG1BbDwwczrn3zr3HjzlT2nG+rIXFpeWV1dJaeX1jc2vb3tltqiiRhDZIxCPZ9rGinAna0Exz2o4lxaHPacsfXeV+645KxSJxq8cx7YZ4IFjACNZG6tnHXiAxSb0YS80wR0H2w70Q66EfpPdZhi5Qz644VWcCNE/cglSgQL1nf3r9iCQhFZpwrFTHdWLdTfPhhNOs7CWKxpiM8IB2DBU4pKqbTk7K0KFR+iiIpHlCo4n6uyPFoVLj0DeV+ZZq1svF/7xOooPzbspEnGgqyPSjIOFIRyjPB/WZpETzsSGYSGZ2RWSITUbapFg2IbizJ8+T5knVdaruzWmldlnEUYJ9OIAjcOEManANdWgAgQd4ghd4tR6tZ+vNep+WLlhFzx78gfXxDcXVniA=</latexit>

@f

@x= [

@f

@x1,@f

@x2, . . . ,

@f

@xn]

<latexit sha1_base64="HZ3OPqfmGVCIiKXSn7LF8mBSaow=">AAACfXichVFdS8MwFE3r15xfUx99CY6ByhjtEOaLMPTFxwnuA9pS0izdgmlaklQ2Sv+Fv8w3/4ovmm4DdZN5IHA499yb5NwgYVQqy3o3zI3Nre2d0m55b//g8KhyfNKTcSow6eKYxWIQIEkY5aSrqGJkkAiCooCRfvB8X9T7L0RIGvMnNU2IF6ERpyHFSGnJr7y6oUA4cxMkFEUMhvk3dyOkxkGYTfIc3kJnjXPi23kdrjU0CwMbxkr+Y+S551eqVsOaAa4Se0GqYIGOX3lzhzFOI8IVZkhKx7YS5WXFTMxIXnZTSRKEn9GIOJpyFBHpZbP0cljTyhCGsdCHKzhTf3ZkKJJyGgXaWQQil2uF+FfNSVV442WUJ6kiHM8vClMGVQyLVcAhFQQrNtUEYUH1WyEeIx2N0gsr6xDs5S+vkl6zYVsN+/G62r5bxFECZ+AcXAAbtEAbPIAO6AIMPgxoXBpXxqdZM+tmY241jUXPKfgFs/UFnqbEKQ==</latexit><latexit sha1_base64="HZ3OPqfmGVCIiKXSn7LF8mBSaow=">AAACfXichVFdS8MwFE3r15xfUx99CY6ByhjtEOaLMPTFxwnuA9pS0izdgmlaklQ2Sv+Fv8w3/4ovmm4DdZN5IHA499yb5NwgYVQqy3o3zI3Nre2d0m55b//g8KhyfNKTcSow6eKYxWIQIEkY5aSrqGJkkAiCooCRfvB8X9T7L0RIGvMnNU2IF6ERpyHFSGnJr7y6oUA4cxMkFEUMhvk3dyOkxkGYTfIc3kJnjXPi23kdrjU0CwMbxkr+Y+S551eqVsOaAa4Se0GqYIGOX3lzhzFOI8IVZkhKx7YS5WXFTMxIXnZTSRKEn9GIOJpyFBHpZbP0cljTyhCGsdCHKzhTf3ZkKJJyGgXaWQQil2uF+FfNSVV442WUJ6kiHM8vClMGVQyLVcAhFQQrNtUEYUH1WyEeIx2N0gsr6xDs5S+vkl6zYVsN+/G62r5bxFECZ+AcXAAbtEAbPIAO6AIMPgxoXBpXxqdZM+tmY241jUXPKfgFs/UFnqbEKQ==</latexit><latexit sha1_base64="HZ3OPqfmGVCIiKXSn7LF8mBSaow=">AAACfXichVFdS8MwFE3r15xfUx99CY6ByhjtEOaLMPTFxwnuA9pS0izdgmlaklQ2Sv+Fv8w3/4ovmm4DdZN5IHA499yb5NwgYVQqy3o3zI3Nre2d0m55b//g8KhyfNKTcSow6eKYxWIQIEkY5aSrqGJkkAiCooCRfvB8X9T7L0RIGvMnNU2IF6ERpyHFSGnJr7y6oUA4cxMkFEUMhvk3dyOkxkGYTfIc3kJnjXPi23kdrjU0CwMbxkr+Y+S551eqVsOaAa4Se0GqYIGOX3lzhzFOI8IVZkhKx7YS5WXFTMxIXnZTSRKEn9GIOJpyFBHpZbP0cljTyhCGsdCHKzhTf3ZkKJJyGgXaWQQil2uF+FfNSVV442WUJ6kiHM8vClMGVQyLVcAhFQQrNtUEYUH1WyEeIx2N0gsr6xDs5S+vkl6zYVsN+/G62r5bxFECZ+AcXAAbtEAbPIAO6AIMPgxoXBpXxqdZM+tmY241jUXPKfgFs/UFnqbEKQ==</latexit><latexit sha1_base64="HZ3OPqfmGVCIiKXSn7LF8mBSaow=">AAACfXichVFdS8MwFE3r15xfUx99CY6ByhjtEOaLMPTFxwnuA9pS0izdgmlaklQ2Sv+Fv8w3/4ovmm4DdZN5IHA499yb5NwgYVQqy3o3zI3Nre2d0m55b//g8KhyfNKTcSow6eKYxWIQIEkY5aSrqGJkkAiCooCRfvB8X9T7L0RIGvMnNU2IF6ERpyHFSGnJr7y6oUA4cxMkFEUMhvk3dyOkxkGYTfIc3kJnjXPi23kdrjU0CwMbxkr+Y+S551eqVsOaAa4Se0GqYIGOX3lzhzFOI8IVZkhKx7YS5WXFTMxIXnZTSRKEn9GIOJpyFBHpZbP0cljTyhCGsdCHKzhTf3ZkKJJyGgXaWQQil2uF+FfNSVV442WUJ6kiHM8vClMGVQyLVcAhFQQrNtUEYUH1WyEeIx2N0gsr6xDs5S+vkl6zYVsN+/G62r5bxFECZ+AcXAAbtEAbPIAO6AIMPgxoXBpXxqdZM+tmY241jUXPKfgFs/UFnqbEKQ==</latexit>

Page 23: Word Embeddings - Princeton University · Word2vec • Input: a large text corpora, V, d • Output: • V: a pre-defined vocabulary • d: dimension of word vectors (e.g. 300) •

Computing the gradientsConsider one pair of target/context words (t, c):

y = � log

✓exp(ut · vc)P

k2V exp(ut · vk)

<latexit sha1_base64="I2knH7Ic3NO14x5ahBCqS6KnOWM=">AAACYnicjVFNaxsxENVumo86TeI0x/YgagL2oWa3FJJLIbSXHBOInYBlFq08awtrpUWaDVmW/ZO95ZRLfkhlZw/5OvSBxOPNPGb0lBZKOoyi+yDc+LC5tb3zsbP7aW//oHv4eexMaQWMhFHG3qTcgZIaRihRwU1hgeepgut0+WdVv74F66TRV1gVMM35XMtMCo5eSrpVRX/R70yZOVOQYZ9llouawV3RZznHRZrVZZMgZWJm/N1Kt00iBk3NXJkn9ZIyqem4+R/XctA0zMr5AgdJtxcNozXoWxK3pEdaXCTdv2xmRJmDRqG4c5M4KnBac4tSKGg6rHRQcLHkc5h4qnkOblqvI2rosVdmNDPWH410rT531Dx3rspT37na1r2urcT3apMSs9NpLXVRImjxNCgrFUVDV3nTmbQgUFWecGGl35WKBfcho/+Vjg8hfv3kt2T8YxhHw/jyZ+/sdxvHDvlCvpE+ickJOSPn5IKMiCAPwWawHxwEj2EnPAyPnlrDoPUckRcIv/4DLt+4uw==</latexit><latexit sha1_base64="I2knH7Ic3NO14x5ahBCqS6KnOWM=">AAACYnicjVFNaxsxENVumo86TeI0x/YgagL2oWa3FJJLIbSXHBOInYBlFq08awtrpUWaDVmW/ZO95ZRLfkhlZw/5OvSBxOPNPGb0lBZKOoyi+yDc+LC5tb3zsbP7aW//oHv4eexMaQWMhFHG3qTcgZIaRihRwU1hgeepgut0+WdVv74F66TRV1gVMM35XMtMCo5eSrpVRX/R70yZOVOQYZ9llouawV3RZznHRZrVZZMgZWJm/N1Kt00iBk3NXJkn9ZIyqem4+R/XctA0zMr5AgdJtxcNozXoWxK3pEdaXCTdv2xmRJmDRqG4c5M4KnBac4tSKGg6rHRQcLHkc5h4qnkOblqvI2rosVdmNDPWH410rT531Dx3rspT37na1r2urcT3apMSs9NpLXVRImjxNCgrFUVDV3nTmbQgUFWecGGl35WKBfcho/+Vjg8hfv3kt2T8YxhHw/jyZ+/sdxvHDvlCvpE+ickJOSPn5IKMiCAPwWawHxwEj2EnPAyPnlrDoPUckRcIv/4DLt+4uw==</latexit><latexit sha1_base64="I2knH7Ic3NO14x5ahBCqS6KnOWM=">AAACYnicjVFNaxsxENVumo86TeI0x/YgagL2oWa3FJJLIbSXHBOInYBlFq08awtrpUWaDVmW/ZO95ZRLfkhlZw/5OvSBxOPNPGb0lBZKOoyi+yDc+LC5tb3zsbP7aW//oHv4eexMaQWMhFHG3qTcgZIaRihRwU1hgeepgut0+WdVv74F66TRV1gVMM35XMtMCo5eSrpVRX/R70yZOVOQYZ9llouawV3RZznHRZrVZZMgZWJm/N1Kt00iBk3NXJkn9ZIyqem4+R/XctA0zMr5AgdJtxcNozXoWxK3pEdaXCTdv2xmRJmDRqG4c5M4KnBac4tSKGg6rHRQcLHkc5h4qnkOblqvI2rosVdmNDPWH410rT531Dx3rspT37na1r2urcT3apMSs9NpLXVRImjxNCgrFUVDV3nTmbQgUFWecGGl35WKBfcho/+Vjg8hfv3kt2T8YxhHw/jyZ+/sdxvHDvlCvpE+ickJOSPn5IKMiCAPwWawHxwEj2EnPAyPnlrDoPUckRcIv/4DLt+4uw==</latexit><latexit sha1_base64="I2knH7Ic3NO14x5ahBCqS6KnOWM=">AAACYnicjVFNaxsxENVumo86TeI0x/YgagL2oWa3FJJLIbSXHBOInYBlFq08awtrpUWaDVmW/ZO95ZRLfkhlZw/5OvSBxOPNPGb0lBZKOoyi+yDc+LC5tb3zsbP7aW//oHv4eexMaQWMhFHG3qTcgZIaRihRwU1hgeepgut0+WdVv74F66TRV1gVMM35XMtMCo5eSrpVRX/R70yZOVOQYZ9llouawV3RZznHRZrVZZMgZWJm/N1Kt00iBk3NXJkn9ZIyqem4+R/XctA0zMr5AgdJtxcNozXoWxK3pEdaXCTdv2xmRJmDRqG4c5M4KnBac4tSKGg6rHRQcLHkc5h4qnkOblqvI2rosVdmNDPWH410rT531Dx3rspT37na1r2urcT3apMSs9NpLXVRImjxNCgrFUVDV3nTmbQgUFWecGGl35WKBfcho/+Vjg8hfv3kt2T8YxhHw/jyZ+/sdxvHDvlCvpE+ickJOSPn5IKMiCAPwWawHxwEj2EnPAyPnlrDoPUckRcIv/4DLt+4uw==</latexit>

Make sure you know how to do this!

Page 24: Word Embeddings - Princeton University · Word2vec • Input: a large text corpora, V, d • Output: • V: a pre-defined vocabulary • d: dimension of word vectors (e.g. 300) •

Putting it together

• Input: text corpus, context size m, embedding size d, V

• Initialize ! randomlyui, vi

• Walk through the training corpus and collect training data (t, c):

• Update

• Update

ut ut � ⌘@y

@ut<latexit sha1_base64="hfQBrPcT8kKJCvFStxNWjvVIVqA=">AAACPXicbVA9SwNBEN3z2/gVtbRZDIKN4U4ELUUbS4UkCrlwzG3mdHHvg905JRz3x2z8D3Z2NhaK2Nq6F4PfDxYeb97MzrwwU9KQ6947Y+MTk1PTM7O1ufmFxaX68krHpLkW2BapSvVZCAaVTLBNkhSeZRohDhWehpeHVf30CrWRadKiQYa9GM4TGUkBZKWg3vJjoIswKvIyIO4rjAi0Tq/5D32L+0jA/UiDKPwMNElQfFB+8U97QGUZ1Btu0x2C/yXeiDTYCMdB/c7vpyKPMSGhwJiu52bUK6rZQmFZ83ODGYhLOMeupQnEaHrF8PqSb1ilz6NU25cQH6rfOwqIjRnEoXVWS5rftUr8r9bNKdrrFTLJcsJEfHwU5YpTyqsoeV9qFKQGloDQ0u7KxQXYiMgGXrMheL9P/ks6203PbXonO439g1EcM2yNrbNN5rFdts+O2DFrM8Fu2AN7Ys/OrfPovDivH9YxZ9Szyn7AeXsH/8Sw5Q==</latexit><latexit sha1_base64="hfQBrPcT8kKJCvFStxNWjvVIVqA=">AAACPXicbVA9SwNBEN3z2/gVtbRZDIKN4U4ELUUbS4UkCrlwzG3mdHHvg905JRz3x2z8D3Z2NhaK2Nq6F4PfDxYeb97MzrwwU9KQ6947Y+MTk1PTM7O1ufmFxaX68krHpLkW2BapSvVZCAaVTLBNkhSeZRohDhWehpeHVf30CrWRadKiQYa9GM4TGUkBZKWg3vJjoIswKvIyIO4rjAi0Tq/5D32L+0jA/UiDKPwMNElQfFB+8U97QGUZ1Btu0x2C/yXeiDTYCMdB/c7vpyKPMSGhwJiu52bUK6rZQmFZ83ODGYhLOMeupQnEaHrF8PqSb1ilz6NU25cQH6rfOwqIjRnEoXVWS5rftUr8r9bNKdrrFTLJcsJEfHwU5YpTyqsoeV9qFKQGloDQ0u7KxQXYiMgGXrMheL9P/ks6203PbXonO439g1EcM2yNrbNN5rFdts+O2DFrM8Fu2AN7Ys/OrfPovDivH9YxZ9Szyn7AeXsH/8Sw5Q==</latexit><latexit sha1_base64="hfQBrPcT8kKJCvFStxNWjvVIVqA=">AAACPXicbVA9SwNBEN3z2/gVtbRZDIKN4U4ELUUbS4UkCrlwzG3mdHHvg905JRz3x2z8D3Z2NhaK2Nq6F4PfDxYeb97MzrwwU9KQ6947Y+MTk1PTM7O1ufmFxaX68krHpLkW2BapSvVZCAaVTLBNkhSeZRohDhWehpeHVf30CrWRadKiQYa9GM4TGUkBZKWg3vJjoIswKvIyIO4rjAi0Tq/5D32L+0jA/UiDKPwMNElQfFB+8U97QGUZ1Btu0x2C/yXeiDTYCMdB/c7vpyKPMSGhwJiu52bUK6rZQmFZ83ODGYhLOMeupQnEaHrF8PqSb1ilz6NU25cQH6rfOwqIjRnEoXVWS5rftUr8r9bNKdrrFTLJcsJEfHwU5YpTyqsoeV9qFKQGloDQ0u7KxQXYiMgGXrMheL9P/ks6203PbXonO439g1EcM2yNrbNN5rFdts+O2DFrM8Fu2AN7Ys/OrfPovDivH9YxZ9Szyn7AeXsH/8Sw5Q==</latexit><latexit sha1_base64="hfQBrPcT8kKJCvFStxNWjvVIVqA=">AAACPXicbVA9SwNBEN3z2/gVtbRZDIKN4U4ELUUbS4UkCrlwzG3mdHHvg905JRz3x2z8D3Z2NhaK2Nq6F4PfDxYeb97MzrwwU9KQ6947Y+MTk1PTM7O1ufmFxaX68krHpLkW2BapSvVZCAaVTLBNkhSeZRohDhWehpeHVf30CrWRadKiQYa9GM4TGUkBZKWg3vJjoIswKvIyIO4rjAi0Tq/5D32L+0jA/UiDKPwMNElQfFB+8U97QGUZ1Btu0x2C/yXeiDTYCMdB/c7vpyKPMSGhwJiu52bUK6rZQmFZ83ODGYhLOMeupQnEaHrF8PqSb1ilz6NU25cQH6rfOwqIjRnEoXVWS5rftUr8r9bNKdrrFTLJcsJEfHwU5YpTyqsoeV9qFKQGloDQ0u7KxQXYiMgGXrMheL9P/ks6203PbXonO439g1EcM2yNrbNN5rFdts+O2DFrM8Fu2AN7Ys/OrfPovDivH9YxZ9Szyn7AeXsH/8Sw5Q==</latexit>

vk vk � ⌘@y

@vk, 8k 2 V

<latexit sha1_base64="mMKYQXExdRdMc9t7yzb43xvZOkM=">AAACTnicbVFNSyNBEO3JqhvjV1yPXhqD4EHDjAi7R3Eve1QwUciEoaZTo016eobuGpcwzC/0Invbn7GXPSiiPTH4XdDweO9VV/XrOFfSku//9Rpf5uYXvjYXW0vLK6tr7fVvfZsVRmBPZCoz5zFYVFJjjyQpPM8NQhorPIvHP2v97AqNlZk+pUmOwxQutEykAHJU1MYwBbqMk/KqisY8VJgQGJP95m/4PR4iAQ8TA6IMczAkQfFJ9YKf7dG4qnadMzOgFHdXSs37Ubvjd/1p8Y8gmIEOm9Vx1P4TjjJRpKhJKLB2EPg5Dct6mFBYtcLCYg5iDBc4cFBDinZYTuOo+LZjRtxt4I4mPmVfd5SQWjtJY+est7bvtZr8TBsUlPwYllLnBaEWT4OSQnHKeJ0tH0mDgtTEARBGul25uASXGbkfaLkQgvdP/gj6+93A7wYnB53Do1kcTbbJttgOC9h3dsh+sWPWY4Jds3/slt15N95/7957eLI2vFnPBntTjeYjArK19A==</latexit><latexit sha1_base64="mMKYQXExdRdMc9t7yzb43xvZOkM=">AAACTnicbVFNSyNBEO3JqhvjV1yPXhqD4EHDjAi7R3Eve1QwUciEoaZTo016eobuGpcwzC/0Invbn7GXPSiiPTH4XdDweO9VV/XrOFfSku//9Rpf5uYXvjYXW0vLK6tr7fVvfZsVRmBPZCoz5zFYVFJjjyQpPM8NQhorPIvHP2v97AqNlZk+pUmOwxQutEykAHJU1MYwBbqMk/KqisY8VJgQGJP95m/4PR4iAQ8TA6IMczAkQfFJ9YKf7dG4qnadMzOgFHdXSs37Ubvjd/1p8Y8gmIEOm9Vx1P4TjjJRpKhJKLB2EPg5Dct6mFBYtcLCYg5iDBc4cFBDinZYTuOo+LZjRtxt4I4mPmVfd5SQWjtJY+est7bvtZr8TBsUlPwYllLnBaEWT4OSQnHKeJ0tH0mDgtTEARBGul25uASXGbkfaLkQgvdP/gj6+93A7wYnB53Do1kcTbbJttgOC9h3dsh+sWPWY4Jds3/slt15N95/7957eLI2vFnPBntTjeYjArK19A==</latexit><latexit sha1_base64="mMKYQXExdRdMc9t7yzb43xvZOkM=">AAACTnicbVFNSyNBEO3JqhvjV1yPXhqD4EHDjAi7R3Eve1QwUciEoaZTo016eobuGpcwzC/0Invbn7GXPSiiPTH4XdDweO9VV/XrOFfSku//9Rpf5uYXvjYXW0vLK6tr7fVvfZsVRmBPZCoz5zFYVFJjjyQpPM8NQhorPIvHP2v97AqNlZk+pUmOwxQutEykAHJU1MYwBbqMk/KqisY8VJgQGJP95m/4PR4iAQ8TA6IMczAkQfFJ9YKf7dG4qnadMzOgFHdXSs37Ubvjd/1p8Y8gmIEOm9Vx1P4TjjJRpKhJKLB2EPg5Dct6mFBYtcLCYg5iDBc4cFBDinZYTuOo+LZjRtxt4I4mPmVfd5SQWjtJY+est7bvtZr8TBsUlPwYllLnBaEWT4OSQnHKeJ0tH0mDgtTEARBGul25uASXGbkfaLkQgvdP/gj6+93A7wYnB53Do1kcTbbJttgOC9h3dsh+sWPWY4Jds3/slt15N95/7957eLI2vFnPBntTjeYjArK19A==</latexit><latexit sha1_base64="mMKYQXExdRdMc9t7yzb43xvZOkM=">AAACTnicbVFNSyNBEO3JqhvjV1yPXhqD4EHDjAi7R3Eve1QwUciEoaZTo016eobuGpcwzC/0Invbn7GXPSiiPTH4XdDweO9VV/XrOFfSku//9Rpf5uYXvjYXW0vLK6tr7fVvfZsVRmBPZCoz5zFYVFJjjyQpPM8NQhorPIvHP2v97AqNlZk+pUmOwxQutEykAHJU1MYwBbqMk/KqisY8VJgQGJP95m/4PR4iAQ8TA6IMczAkQfFJ9YKf7dG4qnadMzOgFHdXSs37Ubvjd/1p8Y8gmIEOm9Vx1P4TjjJRpKhJKLB2EPg5Dct6mFBYtcLCYg5iDBc4cFBDinZYTuOo+LZjRtxt4I4mPmVfd5SQWjtJY+est7bvtZr8TBsUlPwYllLnBaEWT4OSQnHKeJ0tH0mDgtTEARBGul25uASXGbkfaLkQgvdP/gj6+93A7wYnB53Do1kcTbbJttgOC9h3dsh+sWPWY4Jds3/slt15N95/7957eLI2vFnPBntTjeYjArK19A==</latexit>

Any issues?

Page 25: Word Embeddings - Princeton University · Word2vec • Input: a large text corpora, V, d • Output: • V: a pre-defined vocabulary • d: dimension of word vectors (e.g. 300) •

Skip-gram with negative sampling (SGNS)

Problem: every time you get one pair of (t, c), you need to use ! with all the words in the vocabulary! It is very computationally expensive.

vk

Negative sampling: instead of considering all the words in V, let’s randomly sample K (5-20) negative examples.

softmax:

NS:

y = � log

✓exp(ut · vc)P

k2V exp(ut · vk)

<latexit sha1_base64="I2knH7Ic3NO14x5ahBCqS6KnOWM=">AAACYnicjVFNaxsxENVumo86TeI0x/YgagL2oWa3FJJLIbSXHBOInYBlFq08awtrpUWaDVmW/ZO95ZRLfkhlZw/5OvSBxOPNPGb0lBZKOoyi+yDc+LC5tb3zsbP7aW//oHv4eexMaQWMhFHG3qTcgZIaRihRwU1hgeepgut0+WdVv74F66TRV1gVMM35XMtMCo5eSrpVRX/R70yZOVOQYZ9llouawV3RZznHRZrVZZMgZWJm/N1Kt00iBk3NXJkn9ZIyqem4+R/XctA0zMr5AgdJtxcNozXoWxK3pEdaXCTdv2xmRJmDRqG4c5M4KnBac4tSKGg6rHRQcLHkc5h4qnkOblqvI2rosVdmNDPWH410rT531Dx3rspT37na1r2urcT3apMSs9NpLXVRImjxNCgrFUVDV3nTmbQgUFWecGGl35WKBfcho/+Vjg8hfv3kt2T8YxhHw/jyZ+/sdxvHDvlCvpE+ickJOSPn5IKMiCAPwWawHxwEj2EnPAyPnlrDoPUckRcIv/4DLt+4uw==</latexit><latexit sha1_base64="I2knH7Ic3NO14x5ahBCqS6KnOWM=">AAACYnicjVFNaxsxENVumo86TeI0x/YgagL2oWa3FJJLIbSXHBOInYBlFq08awtrpUWaDVmW/ZO95ZRLfkhlZw/5OvSBxOPNPGb0lBZKOoyi+yDc+LC5tb3zsbP7aW//oHv4eexMaQWMhFHG3qTcgZIaRihRwU1hgeepgut0+WdVv74F66TRV1gVMM35XMtMCo5eSrpVRX/R70yZOVOQYZ9llouawV3RZznHRZrVZZMgZWJm/N1Kt00iBk3NXJkn9ZIyqem4+R/XctA0zMr5AgdJtxcNozXoWxK3pEdaXCTdv2xmRJmDRqG4c5M4KnBac4tSKGg6rHRQcLHkc5h4qnkOblqvI2rosVdmNDPWH410rT531Dx3rspT37na1r2urcT3apMSs9NpLXVRImjxNCgrFUVDV3nTmbQgUFWecGGl35WKBfcho/+Vjg8hfv3kt2T8YxhHw/jyZ+/sdxvHDvlCvpE+ickJOSPn5IKMiCAPwWawHxwEj2EnPAyPnlrDoPUckRcIv/4DLt+4uw==</latexit><latexit sha1_base64="I2knH7Ic3NO14x5ahBCqS6KnOWM=">AAACYnicjVFNaxsxENVumo86TeI0x/YgagL2oWa3FJJLIbSXHBOInYBlFq08awtrpUWaDVmW/ZO95ZRLfkhlZw/5OvSBxOPNPGb0lBZKOoyi+yDc+LC5tb3zsbP7aW//oHv4eexMaQWMhFHG3qTcgZIaRihRwU1hgeepgut0+WdVv74F66TRV1gVMM35XMtMCo5eSrpVRX/R70yZOVOQYZ9llouawV3RZznHRZrVZZMgZWJm/N1Kt00iBk3NXJkn9ZIyqem4+R/XctA0zMr5AgdJtxcNozXoWxK3pEdaXCTdv2xmRJmDRqG4c5M4KnBac4tSKGg6rHRQcLHkc5h4qnkOblqvI2rosVdmNDPWH410rT531Dx3rspT37na1r2urcT3apMSs9NpLXVRImjxNCgrFUVDV3nTmbQgUFWecGGl35WKBfcho/+Vjg8hfv3kt2T8YxhHw/jyZ+/sdxvHDvlCvpE+ickJOSPn5IKMiCAPwWawHxwEj2EnPAyPnlrDoPUckRcIv/4DLt+4uw==</latexit><latexit sha1_base64="I2knH7Ic3NO14x5ahBCqS6KnOWM=">AAACYnicjVFNaxsxENVumo86TeI0x/YgagL2oWa3FJJLIbSXHBOInYBlFq08awtrpUWaDVmW/ZO95ZRLfkhlZw/5OvSBxOPNPGb0lBZKOoyi+yDc+LC5tb3zsbP7aW//oHv4eexMaQWMhFHG3qTcgZIaRihRwU1hgeepgut0+WdVv74F66TRV1gVMM35XMtMCo5eSrpVRX/R70yZOVOQYZ9llouawV3RZznHRZrVZZMgZWJm/N1Kt00iBk3NXJkn9ZIyqem4+R/XctA0zMr5AgdJtxcNozXoWxK3pEdaXCTdv2xmRJmDRqG4c5M4KnBac4tSKGg6rHRQcLHkc5h4qnkOblqvI2rosVdmNDPWH410rT531Dx3rspT37na1r2urcT3apMSs9NpLXVRImjxNCgrFUVDV3nTmbQgUFWecGGl35WKBfcho/+Vjg8hfv3kt2T8YxhHw/jyZ+/sdxvHDvlCvpE+ickJOSPn5IKMiCAPwWawHxwEj2EnPAyPnlrDoPUckRcIv/4DLt+4uw==</latexit>

Page 26: Word Embeddings - Princeton University · Word2vec • Input: a large text corpora, V, d • Output: • V: a pre-defined vocabulary • d: dimension of word vectors (e.g. 300) •

Skip-gram with negative sampling (SGNS)

�(x) =1

1 + exp(�x)<latexit sha1_base64="Qv4DTd6P1Pmvw3zC7Y/cLIekIGA=">AAACC3icbVDLSgMxFM3UV62vUZduQovQIpaJCLoRim5cVrAP6Awlk2ba0GRmSDLSMnTvxl9x40IRt/6AO//GtJ2Fth64cDjnXu69x485U9pxvq3cyura+kZ+s7C1vbO7Z+8fNFWUSEIbJOKRbPtYUc5C2tBMc9qOJcXC57TlD2+mfuuBSsWi8F6PY+oJ3A9ZwAjWRuraRVexvsDlUQVeQTeQmKRokiJ4Al06isuno8qka5ecqjMDXCYoIyWQod61v9xeRBJBQ004VqqDnFh7KZaaEU4nBTdRNMZkiPu0Y2iIBVVeOvtlAo+N0oNBJE2FGs7U3xMpFkqNhW86BdYDtehNxf+8TqKDSy9lYZxoGpL5oiDhUEdwGgzsMUmJ5mNDMJHM3ArJAJtAtImvYEJAiy8vk+ZZFTlVdHdeql1nceTBESiCMkDgAtTALaiDBiDgETyDV/BmPVkv1rv1MW/NWdnMIfgD6/MH4cOZBg==</latexit><latexit sha1_base64="Qv4DTd6P1Pmvw3zC7Y/cLIekIGA=">AAACC3icbVDLSgMxFM3UV62vUZduQovQIpaJCLoRim5cVrAP6Awlk2ba0GRmSDLSMnTvxl9x40IRt/6AO//GtJ2Fth64cDjnXu69x485U9pxvq3cyura+kZ+s7C1vbO7Z+8fNFWUSEIbJOKRbPtYUc5C2tBMc9qOJcXC57TlD2+mfuuBSsWi8F6PY+oJ3A9ZwAjWRuraRVexvsDlUQVeQTeQmKRokiJ4Al06isuno8qka5ecqjMDXCYoIyWQod61v9xeRBJBQ004VqqDnFh7KZaaEU4nBTdRNMZkiPu0Y2iIBVVeOvtlAo+N0oNBJE2FGs7U3xMpFkqNhW86BdYDtehNxf+8TqKDSy9lYZxoGpL5oiDhUEdwGgzsMUmJ5mNDMJHM3ArJAJtAtImvYEJAiy8vk+ZZFTlVdHdeql1nceTBESiCMkDgAtTALaiDBiDgETyDV/BmPVkv1rv1MW/NWdnMIfgD6/MH4cOZBg==</latexit><latexit sha1_base64="Qv4DTd6P1Pmvw3zC7Y/cLIekIGA=">AAACC3icbVDLSgMxFM3UV62vUZduQovQIpaJCLoRim5cVrAP6Awlk2ba0GRmSDLSMnTvxl9x40IRt/6AO//GtJ2Fth64cDjnXu69x485U9pxvq3cyura+kZ+s7C1vbO7Z+8fNFWUSEIbJOKRbPtYUc5C2tBMc9qOJcXC57TlD2+mfuuBSsWi8F6PY+oJ3A9ZwAjWRuraRVexvsDlUQVeQTeQmKRokiJ4Al06isuno8qka5ecqjMDXCYoIyWQod61v9xeRBJBQ004VqqDnFh7KZaaEU4nBTdRNMZkiPu0Y2iIBVVeOvtlAo+N0oNBJE2FGs7U3xMpFkqNhW86BdYDtehNxf+8TqKDSy9lYZxoGpL5oiDhUEdwGgzsMUmJ5mNDMJHM3ArJAJtAtImvYEJAiy8vk+ZZFTlVdHdeql1nceTBESiCMkDgAtTALaiDBiDgETyDV/BmPVkv1rv1MW/NWdnMIfgD6/MH4cOZBg==</latexit><latexit sha1_base64="Qv4DTd6P1Pmvw3zC7Y/cLIekIGA=">AAACC3icbVDLSgMxFM3UV62vUZduQovQIpaJCLoRim5cVrAP6Awlk2ba0GRmSDLSMnTvxl9x40IRt/6AO//GtJ2Fth64cDjnXu69x485U9pxvq3cyura+kZ+s7C1vbO7Z+8fNFWUSEIbJOKRbPtYUc5C2tBMc9qOJcXC57TlD2+mfuuBSsWi8F6PY+oJ3A9ZwAjWRuraRVexvsDlUQVeQTeQmKRokiJ4Al06isuno8qka5ecqjMDXCYoIyWQod61v9xeRBJBQ004VqqDnFh7KZaaEU4nBTdRNMZkiPu0Y2iIBVVeOvtlAo+N0oNBJE2FGs7U3xMpFkqNhW86BdYDtehNxf+8TqKDSy9lYZxoGpL5oiDhUEdwGgzsMUmJ5mNDMJHM3ArJAJtAtImvYEJAiy8vk+ZZFTlVdHdeql1nceTBESiCMkDgAtTALaiDBiDgETyDV/BmPVkv1rv1MW/NWdnMIfgD6/MH4cOZBg==</latexit>

Same as training a logistic regression for binary classification!

Compute the gradients: assignment 2!

P (D = 1 | t, c) = �(ut · vc)<latexit sha1_base64="+eQ6DdAqXMFHX0OwYlYQ5Tw9T24=">AAACJXicbVDLSsNAFJ34tr6iLt0MFqGClEQEXSiIunBZwVahCWEymbRDZ5Iwc1MooT/jxl9x48Iigit/xWmbhVoPzHDmnHuZe0+YCa7BcT6tufmFxaXlldXK2vrG5pa9vdPSaa4oa9JUpOoxJJoJnrAmcBDsMVOMyFCwh7B3PfYf+kxpnib3MMiYL0kn4TGnBIwU2OeN2g2+wC72JI8wHGF6aJ6e5h1Jap4k0A3jIh8GgD0apeYupf4woIeBXXXqzgR4lrglqaISjcAeeVFKc8kSoIJo3XadDPyCKOBUsGHFyzXLCO2RDmsbmhDJtF9MthziA6NEOE6VOQngifqzoyBS64EMTeV4SP3XG4v/ee0c4jO/4EmWA0vo9KM4FxhSPI4MR1wxCmJgCKGKm1kx7RJFKJhgKyYE9+/Ks6R1XHedunt3Ur28KuNYQXtoH9WQi07RJbpFDdREFD2hF/SGRtaz9Wq9Wx/T0jmr7NlFv2B9fQPKiqMc</latexit><latexit sha1_base64="+eQ6DdAqXMFHX0OwYlYQ5Tw9T24=">AAACJXicbVDLSsNAFJ34tr6iLt0MFqGClEQEXSiIunBZwVahCWEymbRDZ5Iwc1MooT/jxl9x48Iigit/xWmbhVoPzHDmnHuZe0+YCa7BcT6tufmFxaXlldXK2vrG5pa9vdPSaa4oa9JUpOoxJJoJnrAmcBDsMVOMyFCwh7B3PfYf+kxpnib3MMiYL0kn4TGnBIwU2OeN2g2+wC72JI8wHGF6aJ6e5h1Jap4k0A3jIh8GgD0apeYupf4woIeBXXXqzgR4lrglqaISjcAeeVFKc8kSoIJo3XadDPyCKOBUsGHFyzXLCO2RDmsbmhDJtF9MthziA6NEOE6VOQngifqzoyBS64EMTeV4SP3XG4v/ee0c4jO/4EmWA0vo9KM4FxhSPI4MR1wxCmJgCKGKm1kx7RJFKJhgKyYE9+/Ks6R1XHedunt3Ur28KuNYQXtoH9WQi07RJbpFDdREFD2hF/SGRtaz9Wq9Wx/T0jmr7NlFv2B9fQPKiqMc</latexit><latexit sha1_base64="+eQ6DdAqXMFHX0OwYlYQ5Tw9T24=">AAACJXicbVDLSsNAFJ34tr6iLt0MFqGClEQEXSiIunBZwVahCWEymbRDZ5Iwc1MooT/jxl9x48Iigit/xWmbhVoPzHDmnHuZe0+YCa7BcT6tufmFxaXlldXK2vrG5pa9vdPSaa4oa9JUpOoxJJoJnrAmcBDsMVOMyFCwh7B3PfYf+kxpnib3MMiYL0kn4TGnBIwU2OeN2g2+wC72JI8wHGF6aJ6e5h1Jap4k0A3jIh8GgD0apeYupf4woIeBXXXqzgR4lrglqaISjcAeeVFKc8kSoIJo3XadDPyCKOBUsGHFyzXLCO2RDmsbmhDJtF9MthziA6NEOE6VOQngifqzoyBS64EMTeV4SP3XG4v/ee0c4jO/4EmWA0vo9KM4FxhSPI4MR1wxCmJgCKGKm1kx7RJFKJhgKyYE9+/Ks6R1XHedunt3Ur28KuNYQXtoH9WQi07RJbpFDdREFD2hF/SGRtaz9Wq9Wx/T0jmr7NlFv2B9fQPKiqMc</latexit><latexit sha1_base64="+eQ6DdAqXMFHX0OwYlYQ5Tw9T24=">AAACJXicbVDLSsNAFJ34tr6iLt0MFqGClEQEXSiIunBZwVahCWEymbRDZ5Iwc1MooT/jxl9x48Iigit/xWmbhVoPzHDmnHuZe0+YCa7BcT6tufmFxaXlldXK2vrG5pa9vdPSaa4oa9JUpOoxJJoJnrAmcBDsMVOMyFCwh7B3PfYf+kxpnib3MMiYL0kn4TGnBIwU2OeN2g2+wC72JI8wHGF6aJ6e5h1Jap4k0A3jIh8GgD0apeYupf4woIeBXXXqzgR4lrglqaISjcAeeVFKc8kSoIJo3XadDPyCKOBUsGHFyzXLCO2RDmsbmhDJtF9MthziA6NEOE6VOQngifqzoyBS64EMTeV4SP3XG4v/ee0c4jO/4EmWA0vo9KM4FxhSPI4MR1wxCmJgCKGKm1kx7RJFKJhgKyYE9+/Ks6R1XHedunt3Ur28KuNYQXtoH9WQi07RJbpFDdREFD2hF/SGRtaz9Wq9Wx/T0jmr7NlFv2B9fQPKiqMc</latexit>

Page 27: Word Embeddings - Princeton University · Word2vec • Input: a large text corpora, V, d • Output: • V: a pre-defined vocabulary • d: dimension of word vectors (e.g. 300) •

Continuous Bag of Words (CBOW)

L(✓) =TY

t=1

P (wt | {wt+j},�m j m, j 6= 0)<latexit sha1_base64="3+l6Abc63xGDhSVFpwKTAlCK8fU=">AAACRHicbZDLThsxFIY9lFKacknbZTdWo0pBhWgGVWo3SKjddNFFkAggxWHk8ZxJDLZnap8pikbzcGx4AHY8QTddtEJsUZ3LgtuRLH/6/3N07D8plHQYhlfBwrPF50svll82Xq2srq03X785cHlpBfRErnJ7lHAHShrooUQFR4UFrhMFh8npt4l/+Ausk7nZx3EBA82HRmZScPRS3Oz/aDMcAfINukNZYfM0rnAnqo+r/brqMgUZts9ipEzLlLLqzLsfT2pWb9ItTb39k57MLr05IeMpZFYOR7hRx81W2AmnRR9DNIcWmVc3bl6yNBelBoNCcef6UVjgoOIWpVBQN1jpoODilA+h79FwDW5QTUOo6QevpDTLrT8G6VS9O1Fx7dxYJ75Tcxy5h95EfMrrl5h9GVTSFCWCEbNFWako5nSSKE2lBYFq7IELK/1bqRhxywX63Bs+hOjhlx/DwXYnCjvR3qfW7td5HMvkHXlP2iQin8ku+U66pEcEOSe/yV/yL7gI/gTXwc2sdSGYz7wl9yq4/Q8lTK+5</latexit><latexit sha1_base64="3+l6Abc63xGDhSVFpwKTAlCK8fU=">AAACRHicbZDLThsxFIY9lFKacknbZTdWo0pBhWgGVWo3SKjddNFFkAggxWHk8ZxJDLZnap8pikbzcGx4AHY8QTddtEJsUZ3LgtuRLH/6/3N07D8plHQYhlfBwrPF50svll82Xq2srq03X785cHlpBfRErnJ7lHAHShrooUQFR4UFrhMFh8npt4l/+Ausk7nZx3EBA82HRmZScPRS3Oz/aDMcAfINukNZYfM0rnAnqo+r/brqMgUZts9ipEzLlLLqzLsfT2pWb9ItTb39k57MLr05IeMpZFYOR7hRx81W2AmnRR9DNIcWmVc3bl6yNBelBoNCcef6UVjgoOIWpVBQN1jpoODilA+h79FwDW5QTUOo6QevpDTLrT8G6VS9O1Fx7dxYJ75Tcxy5h95EfMrrl5h9GVTSFCWCEbNFWako5nSSKE2lBYFq7IELK/1bqRhxywX63Bs+hOjhlx/DwXYnCjvR3qfW7td5HMvkHXlP2iQin8ku+U66pEcEOSe/yV/yL7gI/gTXwc2sdSGYz7wl9yq4/Q8lTK+5</latexit><latexit sha1_base64="3+l6Abc63xGDhSVFpwKTAlCK8fU=">AAACRHicbZDLThsxFIY9lFKacknbZTdWo0pBhWgGVWo3SKjddNFFkAggxWHk8ZxJDLZnap8pikbzcGx4AHY8QTddtEJsUZ3LgtuRLH/6/3N07D8plHQYhlfBwrPF50svll82Xq2srq03X785cHlpBfRErnJ7lHAHShrooUQFR4UFrhMFh8npt4l/+Ausk7nZx3EBA82HRmZScPRS3Oz/aDMcAfINukNZYfM0rnAnqo+r/brqMgUZts9ipEzLlLLqzLsfT2pWb9ItTb39k57MLr05IeMpZFYOR7hRx81W2AmnRR9DNIcWmVc3bl6yNBelBoNCcef6UVjgoOIWpVBQN1jpoODilA+h79FwDW5QTUOo6QevpDTLrT8G6VS9O1Fx7dxYJ75Tcxy5h95EfMrrl5h9GVTSFCWCEbNFWako5nSSKE2lBYFq7IELK/1bqRhxywX63Bs+hOjhlx/DwXYnCjvR3qfW7td5HMvkHXlP2iQin8ku+U66pEcEOSe/yV/yL7gI/gTXwc2sdSGYz7wl9yq4/Q8lTK+5</latexit><latexit sha1_base64="3+l6Abc63xGDhSVFpwKTAlCK8fU=">AAACRHicbZDLThsxFIY9lFKacknbZTdWo0pBhWgGVWo3SKjddNFFkAggxWHk8ZxJDLZnap8pikbzcGx4AHY8QTddtEJsUZ3LgtuRLH/6/3N07D8plHQYhlfBwrPF50svll82Xq2srq03X785cHlpBfRErnJ7lHAHShrooUQFR4UFrhMFh8npt4l/+Ausk7nZx3EBA82HRmZScPRS3Oz/aDMcAfINukNZYfM0rnAnqo+r/brqMgUZts9ipEzLlLLqzLsfT2pWb9ItTb39k57MLr05IeMpZFYOR7hRx81W2AmnRR9DNIcWmVc3bl6yNBelBoNCcef6UVjgoOIWpVBQN1jpoODilA+h79FwDW5QTUOo6QevpDTLrT8G6VS9O1Fx7dxYJ75Tcxy5h95EfMrrl5h9GVTSFCWCEbNFWako5nSSKE2lBYFq7IELK/1bqRhxywX63Bs+hOjhlx/DwXYnCjvR3qfW7td5HMvkHXlP2iQin8ku+U66pEcEOSe/yV/yL7gI/gTXwc2sdSGYz7wl9yq4/Q8lTK+5</latexit>

v̄t =1

2m

X

�mjm,j 6=0

vt+j

<latexit sha1_base64="u3qE2VmpSoWtPsbLZcSm88TLfQ4=">AAACP3icbZBNSwMxEIazftb6VfXoJVgEQS27IuhFKHrxqGCr0C1LNs1qNMmuyaxQwv4zL/4Fb169eFDEqzezbQ9+DYQ8vDOTzLxxJrgB33/yxsYnJqemKzPV2bn5hcXa0nLbpLmmrEVTkeqLmBgmuGIt4CDYRaYZkbFg5/HNUZk/v2Pa8FSdQT9jXUkuFU84JeCkqNYOY6JtKAlcxYm9K4oI8AEOE02oDQq7IwsbmlxGdlviULBbfD285FZJypFffGuPLGxeF+6VWt1v+IPAfyEYQR2N4iSqPYa9lOaSKaCCGNMJ/Ay6lmjgVLCiGuaGZYTekEvWcaiIZKZrB/sXeN0pPZyk2h0FeKB+77BEGtOXsassJzW/c6X4X66TQ7LftVxlOTBFhx8lucCQ4tJM3OOaURB9B4Rq7mbF9Io478BZXnUmBL9X/gvtnUbgN4LT3XrzcGRHBa2iNbSBArSHmugYnaAWougePaNX9OY9eC/eu/cxLB3zRj0r6Ed4n19v4rBb</latexit><latexit sha1_base64="u3qE2VmpSoWtPsbLZcSm88TLfQ4=">AAACP3icbZBNSwMxEIazftb6VfXoJVgEQS27IuhFKHrxqGCr0C1LNs1qNMmuyaxQwv4zL/4Fb169eFDEqzezbQ9+DYQ8vDOTzLxxJrgB33/yxsYnJqemKzPV2bn5hcXa0nLbpLmmrEVTkeqLmBgmuGIt4CDYRaYZkbFg5/HNUZk/v2Pa8FSdQT9jXUkuFU84JeCkqNYOY6JtKAlcxYm9K4oI8AEOE02oDQq7IwsbmlxGdlviULBbfD285FZJypFffGuPLGxeF+6VWt1v+IPAfyEYQR2N4iSqPYa9lOaSKaCCGNMJ/Ay6lmjgVLCiGuaGZYTekEvWcaiIZKZrB/sXeN0pPZyk2h0FeKB+77BEGtOXsassJzW/c6X4X66TQ7LftVxlOTBFhx8lucCQ4tJM3OOaURB9B4Rq7mbF9Io478BZXnUmBL9X/gvtnUbgN4LT3XrzcGRHBa2iNbSBArSHmugYnaAWougePaNX9OY9eC/eu/cxLB3zRj0r6Ed4n19v4rBb</latexit><latexit sha1_base64="u3qE2VmpSoWtPsbLZcSm88TLfQ4=">AAACP3icbZBNSwMxEIazftb6VfXoJVgEQS27IuhFKHrxqGCr0C1LNs1qNMmuyaxQwv4zL/4Fb169eFDEqzezbQ9+DYQ8vDOTzLxxJrgB33/yxsYnJqemKzPV2bn5hcXa0nLbpLmmrEVTkeqLmBgmuGIt4CDYRaYZkbFg5/HNUZk/v2Pa8FSdQT9jXUkuFU84JeCkqNYOY6JtKAlcxYm9K4oI8AEOE02oDQq7IwsbmlxGdlviULBbfD285FZJypFffGuPLGxeF+6VWt1v+IPAfyEYQR2N4iSqPYa9lOaSKaCCGNMJ/Ay6lmjgVLCiGuaGZYTekEvWcaiIZKZrB/sXeN0pPZyk2h0FeKB+77BEGtOXsassJzW/c6X4X66TQ7LftVxlOTBFhx8lucCQ4tJM3OOaURB9B4Rq7mbF9Io478BZXnUmBL9X/gvtnUbgN4LT3XrzcGRHBa2iNbSBArSHmugYnaAWougePaNX9OY9eC/eu/cxLB3zRj0r6Ed4n19v4rBb</latexit><latexit sha1_base64="u3qE2VmpSoWtPsbLZcSm88TLfQ4=">AAACP3icbZBNSwMxEIazftb6VfXoJVgEQS27IuhFKHrxqGCr0C1LNs1qNMmuyaxQwv4zL/4Fb169eFDEqzezbQ9+DYQ8vDOTzLxxJrgB33/yxsYnJqemKzPV2bn5hcXa0nLbpLmmrEVTkeqLmBgmuGIt4CDYRaYZkbFg5/HNUZk/v2Pa8FSdQT9jXUkuFU84JeCkqNYOY6JtKAlcxYm9K4oI8AEOE02oDQq7IwsbmlxGdlviULBbfD285FZJypFffGuPLGxeF+6VWt1v+IPAfyEYQR2N4iSqPYa9lOaSKaCCGNMJ/Ay6lmjgVLCiGuaGZYTekEvWcaiIZKZrB/sXeN0pPZyk2h0FeKB+77BEGtOXsassJzW/c6X4X66TQ7LftVxlOTBFhx8lucCQ4tJM3OOaURB9B4Rq7mbF9Io478BZXnUmBL9X/gvtnUbgN4LT3XrzcGRHBa2iNbSBArSHmugYnaAWougePaNX9OY9eC/eu/cxLB3zRj0r6Ed4n19v4rBb</latexit>

Page 28: Word Embeddings - Princeton University · Word2vec • Input: a large text corpora, V, d • Output: • V: a pre-defined vocabulary • d: dimension of word vectors (e.g. 300) •

GloVe: Global Vectors

(Pennington et al, 2014): GloVe: Global Vectors for Word Representation

• Let’s take the global co-occurrence statistics: !Xi,j

• Training faster

• Scalable to very large corpora

Page 29: Word Embeddings - Princeton University · Word2vec • Input: a large text corpora, V, d • Output: • V: a pre-defined vocabulary • d: dimension of word vectors (e.g. 300) •

GloVe: Global Vectors

(Pennington et al, 2014): GloVe: Global Vectors for Word Representation

Page 30: Word Embeddings - Princeton University · Word2vec • Input: a large text corpora, V, d • Output: • V: a pre-defined vocabulary • d: dimension of word vectors (e.g. 300) •

FastText: Sub-Word Embeddings

(Bojanowski et al, 2017): Enriching Word Vectors with Subword Information

• More to come! Contextualized word embeddings

• Similar as Skip-gram, but break words into n-grams with n = 3 to 6

where: 3-grams: <wh, whe, her, ere, re>

4-grams: <whe, wher, here, ere>

5-grams: <wher, where, here>

6-grams: <where, where>

• Replace byui · vj<latexit sha1_base64="oX8M9O0Ff2ekfvBmSkoLpI7y8XY=">AAACCHicbVBNS8NAEN3Ur1q/oh49uFgETyURQY9FLx4r2FZoQ9hsNu3azW7Y3RRKyNGLf8WLB0W8+hO8+W/ctBG09cHA470ZZuYFCaNKO86XVVlaXlldq67XNja3tnfs3b2OEqnEpI0FE/IuQIowyklbU83IXSIJigNGusHoqvC7YyIVFfxWTxLixWjAaUQx0kby7cN+jPQwiLI09yns41Bo+CONc//et+tOw5kCLhK3JHVQouXbn/1Q4DQmXGOGlOq5TqK9DElNMSN5rZ8qkiA8QgPSM5SjmCgvmz6Sw2OjhDAS0hTXcKr+nshQrNQkDkxncaOa9wrxP6+X6ujCyyhPUk04ni2KUga1gEUqMKSSYM0mhiAsqbkV4iGSCGuTXc2E4M6/vEg6pw3Xabg3Z/XmZRlHFRyAI3ACXHAOmuAatEAbYPAAnsALeLUerWfrzXqftVascmYf/IH18Q1e4Jov</latexit><latexit sha1_base64="oX8M9O0Ff2ekfvBmSkoLpI7y8XY=">AAACCHicbVBNS8NAEN3Ur1q/oh49uFgETyURQY9FLx4r2FZoQ9hsNu3azW7Y3RRKyNGLf8WLB0W8+hO8+W/ctBG09cHA470ZZuYFCaNKO86XVVlaXlldq67XNja3tnfs3b2OEqnEpI0FE/IuQIowyklbU83IXSIJigNGusHoqvC7YyIVFfxWTxLixWjAaUQx0kby7cN+jPQwiLI09yns41Bo+CONc//et+tOw5kCLhK3JHVQouXbn/1Q4DQmXGOGlOq5TqK9DElNMSN5rZ8qkiA8QgPSM5SjmCgvmz6Sw2OjhDAS0hTXcKr+nshQrNQkDkxncaOa9wrxP6+X6ujCyyhPUk04ni2KUga1gEUqMKSSYM0mhiAsqbkV4iGSCGuTXc2E4M6/vEg6pw3Xabg3Z/XmZRlHFRyAI3ACXHAOmuAatEAbYPAAnsALeLUerWfrzXqftVascmYf/IH18Q1e4Jov</latexit><latexit sha1_base64="oX8M9O0Ff2ekfvBmSkoLpI7y8XY=">AAACCHicbVBNS8NAEN3Ur1q/oh49uFgETyURQY9FLx4r2FZoQ9hsNu3azW7Y3RRKyNGLf8WLB0W8+hO8+W/ctBG09cHA470ZZuYFCaNKO86XVVlaXlldq67XNja3tnfs3b2OEqnEpI0FE/IuQIowyklbU83IXSIJigNGusHoqvC7YyIVFfxWTxLixWjAaUQx0kby7cN+jPQwiLI09yns41Bo+CONc//et+tOw5kCLhK3JHVQouXbn/1Q4DQmXGOGlOq5TqK9DElNMSN5rZ8qkiA8QgPSM5SjmCgvmz6Sw2OjhDAS0hTXcKr+nshQrNQkDkxncaOa9wrxP6+X6ujCyyhPUk04ni2KUga1gEUqMKSSYM0mhiAsqbkV4iGSCGuTXc2E4M6/vEg6pw3Xabg3Z/XmZRlHFRyAI3ACXHAOmuAatEAbYPAAnsALeLUerWfrzXqftVascmYf/IH18Q1e4Jov</latexit><latexit sha1_base64="oX8M9O0Ff2ekfvBmSkoLpI7y8XY=">AAACCHicbVBNS8NAEN3Ur1q/oh49uFgETyURQY9FLx4r2FZoQ9hsNu3azW7Y3RRKyNGLf8WLB0W8+hO8+W/ctBG09cHA470ZZuYFCaNKO86XVVlaXlldq67XNja3tnfs3b2OEqnEpI0FE/IuQIowyklbU83IXSIJigNGusHoqvC7YyIVFfxWTxLixWjAaUQx0kby7cN+jPQwiLI09yns41Bo+CONc//et+tOw5kCLhK3JHVQouXbn/1Q4DQmXGOGlOq5TqK9DElNMSN5rZ8qkiA8QgPSM5SjmCgvmz6Sw2OjhDAS0hTXcKr+nshQrNQkDkxncaOa9wrxP6+X6ujCyyhPUk04ni2KUga1gEUqMKSSYM0mhiAsqbkV4iGSCGuTXc2E4M6/vEg6pw3Xabg3Z/XmZRlHFRyAI3ACXHAOmuAatEAbYPAAnsALeLUerWfrzXqftVascmYf/IH18Q1e4Jov</latexit>

X

g2n-grams(wi)

ug · vj

<latexit sha1_base64="vjRr+MXndBS39D+Os22ZGPURTaY=">AAACKHicbVBNS8NAFNzUr1q/qh69LBahHiyJCHqz6MWjgq1CU8Jmu2nXbjZh96VaQn6OF/+KFxFFvPpL3LYRtHVgYZiZx743fiy4Btv+tApz8wuLS8Xl0srq2vpGeXOrqaNEUdagkYjUrU80E1yyBnAQ7DZWjIS+YDd+/3zk3wyY0jyS1zCMWTskXckDTgkYySufujoJvbSLXS6xdIE9QHrQVSTUWfXe4/uZGxLo+UGaZJ4J0U4E+EcaZN6dV67YNXsMPEucnFRQjkuv/Op2IpqETAIVROuWY8fQTokCTgXLSm6iWUxon3RZy1BJQqbb6fjQDO8ZpYODSJknAY/V3xOp2VsPQ98kRzvqaW8k/ue1EghO2imXcQJM0slHQSIwRHjUGu5wxSiIoSGEKm52xbRHFKFgui2ZEpzpk2dJ87Dm2DXn6qhSP8vrKKIdtIuqyEHHqI4u0CVqIIoe0TN6Q+/Wk/VifVifk2jByme20R9YX9/UuKet</latexit><latexit sha1_base64="vjRr+MXndBS39D+Os22ZGPURTaY=">AAACKHicbVBNS8NAFNzUr1q/qh69LBahHiyJCHqz6MWjgq1CU8Jmu2nXbjZh96VaQn6OF/+KFxFFvPpL3LYRtHVgYZiZx743fiy4Btv+tApz8wuLS8Xl0srq2vpGeXOrqaNEUdagkYjUrU80E1yyBnAQ7DZWjIS+YDd+/3zk3wyY0jyS1zCMWTskXckDTgkYySufujoJvbSLXS6xdIE9QHrQVSTUWfXe4/uZGxLo+UGaZJ4J0U4E+EcaZN6dV67YNXsMPEucnFRQjkuv/Op2IpqETAIVROuWY8fQTokCTgXLSm6iWUxon3RZy1BJQqbb6fjQDO8ZpYODSJknAY/V3xOp2VsPQ98kRzvqaW8k/ue1EghO2imXcQJM0slHQSIwRHjUGu5wxSiIoSGEKm52xbRHFKFgui2ZEpzpk2dJ87Dm2DXn6qhSP8vrKKIdtIuqyEHHqI4u0CVqIIoe0TN6Q+/Wk/VifVifk2jByme20R9YX9/UuKet</latexit><latexit sha1_base64="vjRr+MXndBS39D+Os22ZGPURTaY=">AAACKHicbVBNS8NAFNzUr1q/qh69LBahHiyJCHqz6MWjgq1CU8Jmu2nXbjZh96VaQn6OF/+KFxFFvPpL3LYRtHVgYZiZx743fiy4Btv+tApz8wuLS8Xl0srq2vpGeXOrqaNEUdagkYjUrU80E1yyBnAQ7DZWjIS+YDd+/3zk3wyY0jyS1zCMWTskXckDTgkYySufujoJvbSLXS6xdIE9QHrQVSTUWfXe4/uZGxLo+UGaZJ4J0U4E+EcaZN6dV67YNXsMPEucnFRQjkuv/Op2IpqETAIVROuWY8fQTokCTgXLSm6iWUxon3RZy1BJQqbb6fjQDO8ZpYODSJknAY/V3xOp2VsPQ98kRzvqaW8k/ue1EghO2imXcQJM0slHQSIwRHjUGu5wxSiIoSGEKm52xbRHFKFgui2ZEpzpk2dJ87Dm2DXn6qhSP8vrKKIdtIuqyEHHqI4u0CVqIIoe0TN6Q+/Wk/VifVifk2jByme20R9YX9/UuKet</latexit><latexit sha1_base64="vjRr+MXndBS39D+Os22ZGPURTaY=">AAACKHicbVBNS8NAFNzUr1q/qh69LBahHiyJCHqz6MWjgq1CU8Jmu2nXbjZh96VaQn6OF/+KFxFFvPpL3LYRtHVgYZiZx743fiy4Btv+tApz8wuLS8Xl0srq2vpGeXOrqaNEUdagkYjUrU80E1yyBnAQ7DZWjIS+YDd+/3zk3wyY0jyS1zCMWTskXckDTgkYySufujoJvbSLXS6xdIE9QHrQVSTUWfXe4/uZGxLo+UGaZJ4J0U4E+EcaZN6dV67YNXsMPEucnFRQjkuv/Op2IpqETAIVROuWY8fQTokCTgXLSm6iWUxon3RZy1BJQqbb6fjQDO8ZpYODSJknAY/V3xOp2VsPQ98kRzvqaW8k/ue1EghO2imXcQJM0slHQSIwRHjUGu5wxSiIoSGEKm52xbRHFKFgui2ZEpzpk2dJ87Dm2DXn6qhSP8vrKKIdtIuqyEHHqI4u0CVqIIoe0TN6Q+/Wk/VifVifk2jByme20R9YX9/UuKet</latexit>

Page 31: Word Embeddings - Princeton University · Word2vec • Input: a large text corpora, V, d • Output: • V: a pre-defined vocabulary • d: dimension of word vectors (e.g. 300) •

Trained word embeddings available

• word2vec: https://code.google.com/archive/p/word2vec/

• GloVe: https://nlp.stanford.edu/projects/glove/

• FastText: https://fasttext.cc/

Differ in algorithms, text corpora, dimensions, cased/uncased…

Page 32: Word Embeddings - Princeton University · Word2vec • Input: a large text corpora, V, d • Output: • V: a pre-defined vocabulary • d: dimension of word vectors (e.g. 300) •

Evaluating Word Embeddings

Page 33: Word Embeddings - Princeton University · Word2vec • Input: a large text corpora, V, d • Output: • V: a pre-defined vocabulary • d: dimension of word vectors (e.g. 300) •

Extrinsic evaluation

• Let’s plug these word embeddings into a real NLP system and see whether this improves performance

• Could take a long time but still the most important evaluation metric I

( 0.31−0.28) ( 0.01

−0.91) (1.870.03) (−3.17

−0.18) (1.231.59)

don’t like this movie

ML model

👎

Extrinsic vs intrinsic evaluation

Intrinsic evaluation

• Evaluate on a specific/intermediate subtask

• Fast to compute

• Not clear if it really helps the downstream task

Page 34: Word Embeddings - Princeton University · Word2vec • Input: a large text corpora, V, d • Output: • V: a pre-defined vocabulary • d: dimension of word vectors (e.g. 300) •

Intrinsic evaluation

Word similarity Example dataset: wordsim-353353 pairs of words with human judgement http://www.cs.technion.ac.il/~gabr/resources/data/wordsim353/

Cosine similarity:

Metric: Spearman rank correlation

Page 35: Word Embeddings - Princeton University · Word2vec • Input: a large text corpora, V, d • Output: • V: a pre-defined vocabulary • d: dimension of word vectors (e.g. 300) •

Intrinsic evaluation

Word Similarity

Page 36: Word Embeddings - Princeton University · Word2vec • Input: a large text corpora, V, d • Output: • V: a pre-defined vocabulary • d: dimension of word vectors (e.g. 300) •

Intrinsic evaluation

Word analogy man: woman ! king: ? ≈ argmax

i(cos(ui,ub � ua + uc))

<latexit sha1_base64="JrpgXOIk2wxy6PeogrgqRj0rj24=">AAACQHicbVDLSsNAFJ3UV62vqks3g0VoUUsigi6LblxWsA9oQphMJ+3g5MHMjVhCP82Nn+DOtRsXirh15aTtIrYeGDjn3Hu5d44XC67ANF+NwtLyyupacb20sbm1vVPe3WurKJGUtWgkItn1iGKCh6wFHATrxpKRwBOs491fZ/XOA5OKR+EdjGLmBGQQcp9TAtpyyx2byIEdkEc35WNbMB+qNo1UVVsw9Pw0Gbv8BOeUh0/zkuDjvKQ1W/LBEGpuuWLWzQnwIrFmpIJmaLrlF7sf0SRgIVBBlOpZZgxOSiRwKti4ZCeKxYTekwHraRqSgCknnQQwxkfa6WM/kvqFgCdufiIlgVKjwNOd2a1qvpaZ/9V6CfiXTsrDOAEW0ukiPxEYIpyliftcMgpipAmhkutbMR0SSSjozEs6BGv+y4ukfVa3zLp1e15pXM3iKKIDdIiqyEIXqIFuUBO1EEVP6A19oE/j2Xg3vozvaWvBmM3soz8wfn4BlsGw6A==</latexit><latexit sha1_base64="JrpgXOIk2wxy6PeogrgqRj0rj24=">AAACQHicbVDLSsNAFJ3UV62vqks3g0VoUUsigi6LblxWsA9oQphMJ+3g5MHMjVhCP82Nn+DOtRsXirh15aTtIrYeGDjn3Hu5d44XC67ANF+NwtLyyupacb20sbm1vVPe3WurKJGUtWgkItn1iGKCh6wFHATrxpKRwBOs491fZ/XOA5OKR+EdjGLmBGQQcp9TAtpyyx2byIEdkEc35WNbMB+qNo1UVVsw9Pw0Gbv8BOeUh0/zkuDjvKQ1W/LBEGpuuWLWzQnwIrFmpIJmaLrlF7sf0SRgIVBBlOpZZgxOSiRwKti4ZCeKxYTekwHraRqSgCknnQQwxkfa6WM/kvqFgCdufiIlgVKjwNOd2a1qvpaZ/9V6CfiXTsrDOAEW0ukiPxEYIpyliftcMgpipAmhkutbMR0SSSjozEs6BGv+y4ukfVa3zLp1e15pXM3iKKIDdIiqyEIXqIFuUBO1EEVP6A19oE/j2Xg3vozvaWvBmM3soz8wfn4BlsGw6A==</latexit><latexit sha1_base64="JrpgXOIk2wxy6PeogrgqRj0rj24=">AAACQHicbVDLSsNAFJ3UV62vqks3g0VoUUsigi6LblxWsA9oQphMJ+3g5MHMjVhCP82Nn+DOtRsXirh15aTtIrYeGDjn3Hu5d44XC67ANF+NwtLyyupacb20sbm1vVPe3WurKJGUtWgkItn1iGKCh6wFHATrxpKRwBOs491fZ/XOA5OKR+EdjGLmBGQQcp9TAtpyyx2byIEdkEc35WNbMB+qNo1UVVsw9Pw0Gbv8BOeUh0/zkuDjvKQ1W/LBEGpuuWLWzQnwIrFmpIJmaLrlF7sf0SRgIVBBlOpZZgxOSiRwKti4ZCeKxYTekwHraRqSgCknnQQwxkfa6WM/kvqFgCdufiIlgVKjwNOd2a1qvpaZ/9V6CfiXTsrDOAEW0ukiPxEYIpyliftcMgpipAmhkutbMR0SSSjozEs6BGv+y4ukfVa3zLp1e15pXM3iKKIDdIiqyEIXqIFuUBO1EEVP6A19oE/j2Xg3vozvaWvBmM3soz8wfn4BlsGw6A==</latexit><latexit sha1_base64="JrpgXOIk2wxy6PeogrgqRj0rj24=">AAACQHicbVDLSsNAFJ3UV62vqks3g0VoUUsigi6LblxWsA9oQphMJ+3g5MHMjVhCP82Nn+DOtRsXirh15aTtIrYeGDjn3Hu5d44XC67ANF+NwtLyyupacb20sbm1vVPe3WurKJGUtWgkItn1iGKCh6wFHATrxpKRwBOs491fZ/XOA5OKR+EdjGLmBGQQcp9TAtpyyx2byIEdkEc35WNbMB+qNo1UVVsw9Pw0Gbv8BOeUh0/zkuDjvKQ1W/LBEGpuuWLWzQnwIrFmpIJmaLrlF7sf0SRgIVBBlOpZZgxOSiRwKti4ZCeKxYTekwHraRqSgCknnQQwxkfa6WM/kvqFgCdufiIlgVKjwNOd2a1qvpaZ/9V6CfiXTsrDOAEW0ukiPxEYIpyliftcMgpipAmhkutbMR0SSSjozEs6BGv+y4ukfVa3zLp1e15pXM3iKKIDdIiqyEIXqIFuUBO1EEVP6A19oE/j2Xg3vozvaWvBmM3soz8wfn4BlsGw6A==</latexit>

semantic

Chicago:Illinois Philadelphia: ? bad:worst ! cool: ? ≈

syntactic

http://download.tensorflow.org/data/questions-words.txtMore examples at