Lecture 8: NLP and Word Embeddingsfall97.class.vision/slides/8.pdf · Transfer learning and word...
16
Lecture 8 - SRTTU – A.Akhavan 1 شنبه،۱۹ آبان۱۳۹۷ Lecture 8: NLP and Word Embeddings Alireza Akhavan Pour CLASS.VISION
Transcript of Lecture 8: NLP and Word Embeddingsfall97.class.vision/slides/8.pdf · Transfer learning and word...
Lecture 8 -SRTTU – A.Akhavan 1 ۱۳۹۷آبان ۱۹شنبه،
Lecture 8: NLP and Word Embeddings
Alireza Akhavan Pour
CLASS.VISION
Lecture 8 -SRTTU – A.Akhavan
NLP and Word EmbeddingsWord representation
2 ۱۳۹۷آبان ۱۹شنبه،
V = [a, aaron, ..., zulu, <UNK>]1-hot representation
𝑽 = 𝟏𝟎, 𝟎𝟎𝟎
I want a glass of orange ______ .
I want a glass of apple ______ .
𝑂5391 𝑂9853
𝑗𝑢𝑖𝑐𝑒
?
مشکل؟.فاصله اقلیدسی تمام بردارها یکسان است.کندgeneralizeنمیتواند از روی کلماتی که در آموزش دیده است
Lecture 8 -SRTTU – A.Akhavan
Featurized representation: word embedding
3 ۱۳۹۷آبان ۱۹شنبه،
1-جنسیت 1 -0.95 0.97 0.00 0.01
0.94سلطنتی 0.93 -0.01 0.000.020.01
0.71سن 0.69 0.03 -0.020.020.03
0.02خوراکی 0.00 0.96 -0.97-0.010.01
سایززنده
قیمتفعل
....
300
...
𝒆𝟔𝟐𝟓𝟕
...
𝒆𝟒𝟓𝟔
I want a glass of orange ______ .
I want a glass of apple ______ .
𝑗𝑢𝑖𝑐𝑒
𝑗𝑢𝑖𝑐𝑒
Lecture 8 -SRTTU – A.Akhavan
Visualizing word embeddings
4 ۱۳۹۷آبان ۱۹شنبه،
[van der Maaten and Hinton., 2008.Visualizing data using t-SNE]
Lecture 8 -SRTTU – A.Akhavan
Using word embeddings:Named entity recognition example
5 ۱۳۹۷آبان ۱۹شنبه،
Robert Lin is an apple farmer
Sally Johnson is an orange farmer
Robert Lin is a durian cultivator
Lecture 8 -SRTTU – A.Akhavan
Now if you have tested your model with this sentence "Robert Lin is a durian cultivator“ the network should learn the name even if it hasn't seen the word durian before (during training). That's the power of word representations.
The algorithms that are used to learn word embeddings can examine billions of words of unlabeled text - for example, 100 billion words and learn the representation from them.
6 ۱۳۹۷آبان ۱۹شنبه،
Using word embeddings:Named entity recognition example
Lecture 8 -SRTTU – A.Akhavan
Transfer learning and word embeddings
I. Learn word embeddings from large text corpus (1-100 billion of words).
Or download pre-trained embedding online.
II. Transfer embedding to new task with the smaller training set (say, 100k words).
III. Optional: continue to finetune the word embeddings with new data. You bother doing this if your smaller training set (from step 2) is big enough.
7 ۱۳۹۷آبان ۱۹شنبه،
10.000مثال به جای وکتور . یک ویژگی مثبت دیگر کاهش ابعاد ورودی است.بعدی کار خواهیم کرد300با یک وکتور one-hotبعدی
Lecture 8 -SRTTU – A.Akhavan
Relation to face encoding (Embeddings)
8 ۱۳۹۷آبان ۱۹شنبه،
[Taigman et. al., 2014. DeepFace: Closing the gap to human level performance]
Word embeddings have an interesting relationship to the face recognition task:o In this problem, we encode each face into a vector and then check how similar
are these vectors.o Words encoding and embeddings have a similar meaning here.
In the word embeddings task, we are learning a representation for each word in our vocabulary (unlike in image encoding where we have to map each new image to some n-dimensional vector).
Lecture 8 -SRTTU – A.Akhavan
Properties of word embeddings
• Analogies
9 ۱۳۹۷آبان ۱۹شنبه،
[Mikolov et. al., 2013, Linguistic regularities in continuous space word representations]
Can we conclude this relation: Man ==> Woman King ==> ??
𝒆𝑴𝒂𝒏 𝒆𝑾𝒐𝒎𝒂𝒏 𝒆𝑲𝒊𝒏𝒈 𝒆𝑸𝒖𝒆𝒆𝒏
eMan – eWoman ≈
−2000
eKing – eQ𝐮𝐞𝐞𝐧 ≈
−2000
Lecture 8 -SRTTU – A.Akhavan
Analogies using word vectors
10 ۱۳۹۷آبان ۱۹شنبه،
300 D
man
woman
King
Queen
𝑭𝒊𝒏𝒅 𝒘𝒐𝒓𝒅 𝒘: 𝑎𝑟𝑔max𝑤
൯𝑠𝑖𝑚(𝑒𝑤, 𝑒𝑘𝑖𝑛𝑔 − 𝑒𝑚𝑎𝑛 + 𝑒𝑤𝑜𝑚𝑎𝑛
𝒆𝒘
t-SNE
Lecture 8 -SRTTU – A.Akhavan
Cosine similarity
11 ۱۳۹۷آبان ۱۹شنبه،
𝑠𝑖𝑚 𝑢, 𝑣 =𝑈𝑇𝑉
𝑢 2 𝑣 2
൯𝑠𝑖𝑚(𝑒𝑤, 𝑒𝑘𝑖𝑛𝑔 − 𝑒𝑚𝑎𝑛 + 𝑒𝑤𝑜𝑚𝑎𝑛
𝑉
𝑈𝑉
𝑈
1
-1
0
𝑈 − 𝑉 2
:فاصله اقلیدسی
Ottawa:Canada as Iran:Tehran
Man:Woman as boy:girl
Big:bigger as tall:taller
Yen: Japan as Ruble:Russia
Lecture 8 -SRTTU – A.Akhavan
Embedding matrix
12 ۱۳۹۷آبان ۱۹شنبه،
here … orange example ... <UNK>
-0.2 … -0.67 -0.2 … 0.2
0.7 … 0.3 -0.5 … 0.1
0.85 … 0.25 0.3 … 1
-0.04 … -0.18 0.33 … -0.1
...
0.5 … 1 0.3 … 0.2
300
10.000
6257
0000.:1.:00
6257
10
.00
0
𝑂6257
𝐸
Lecture 8 -SRTTU – A.Akhavan
Embedding matrix
13 ۱۳۹۷آبان ۱۹شنبه،
here … orange example ... <UNK>
-0.2 … -0.67 -0.2 … 0.2
0.7 … 0.3 -0.5 … 0.1
0.85 … 0.25 0.3 … 1
-0.04 … -0.18 0.33 … -0.1
...
0.5 … 1 0.3 … 0.2
300
10.000
6257
0000.:1.:00
6257
10
.00
0
𝑂6257
𝐸
𝑬 . 𝑶𝟔𝟐𝟓𝟕 = 𝒆𝟔𝟐𝟓𝟕
Lecture 8 -SRTTU – A.Akhavan
Embedding matrix
14 ۱۳۹۷آبان ۱۹شنبه،
here … orange example ... <UNK>
-0.2 … -0.67 -0.2 … 0.2
0.7 … 0.3 -0.5 … 0.1
0.85 … 0.25 0.3 … 1
-0.04 … -0.18 0.33 … -0.1
...
0.5 … 1 0.3 … 0.2
𝑬 .𝑶𝟔𝟐𝟓𝟕 = 𝒆𝟔𝟐𝟓𝟕
300x10k
0000.:1.:00
10k x 1 300 x 1
If O6257 is the one hot encoding of the word orange of shape (10000, 1), thennp.dot(E,O6257) = e6257 which shape is (300, 1).
Generally np.dot(E, Oj) = ej (embedding for word j)
Lecture 8 -SRTTU – A.Akhavan
Embedding matrix
15 ۱۳۹۷آبان ۱۹شنبه،
The Embedding layer is best understood as a dictionary mapping integer indices (which stand for specific words) to dense vectors. It takes as input integers, it looks up these integers into an internal dictionary, and it returns the associated vectors. It's effectively a dictionary lookup.
https://keras.io/layers/embeddings/
Lecture 8 -SRTTU – A.Akhavan 16 ۱۳۹۷آبان ۱۹شنبه،
منابع
• https://www.coursera.org/specializations/deep-learning
• https://github.com/fchollet/deep-learning-with-python-notebooks/blob/master/6.1-using-word-embeddings.ipynb
• https://github.com/mbadry1/DeepLearning.ai-Summary/
