A survey and practice of Neural-network-based Textual representation › talks › textzoo.pdf ·...
Transcript of A survey and practice of Neural-network-based Textual representation › talks › textzoo.pdf ·...
![Page 1: A survey and practice of Neural-network-based Textual representation › talks › textzoo.pdf · 2020-04-23 · A survey and practice of Neural-network-based Textual representation](https://reader035.fdocuments.in/reader035/viewer/2022070803/5f033e537e708231d4083f33/html5/thumbnails/1.jpg)
A survey and practice of
Neural-network-based Textual representation
WabyWang,LilianWang,JaredWei,LoringLiu
Department of Social Network Operation,
Social Network Group,
Tencent
Wang B, Wang L, Wei Q, Wang Y, Liu L. TextZoo, a New Benchmark for Reconsidering Text Classification[J]. arXiv preprint arXiv:1802.03656, 2018.
![Page 2: A survey and practice of Neural-network-based Textual representation › talks › textzoo.pdf · 2020-04-23 · A survey and practice of Neural-network-based Textual representation](https://reader035.fdocuments.in/reader035/viewer/2022070803/5f033e537e708231d4083f33/html5/thumbnails/2.jpg)
welcome for any issues and contributions !!!
![Page 3: A survey and practice of Neural-network-based Textual representation › talks › textzoo.pdf · 2020-04-23 · A survey and practice of Neural-network-based Textual representation](https://reader035.fdocuments.in/reader035/viewer/2022070803/5f033e537e708231d4083f33/html5/thumbnails/3.jpg)
find . -name "*.py" -print | xargs wc -l
3256 lines
![Page 4: A survey and practice of Neural-network-based Textual representation › talks › textzoo.pdf · 2020-04-23 · A survey and practice of Neural-network-based Textual representation](https://reader035.fdocuments.in/reader035/viewer/2022070803/5f033e537e708231d4083f33/html5/thumbnails/4.jpg)
TextZOOA new Benchmark to Reconsidering Text Classification
Wang B, Wang L, Wei Q, Wang Y, Liu L. TextZoo, a New Benchmark for Reconsidering Text Classification[J]. arXiv preprint arXiv:1802.03656, 2018.
![Page 5: A survey and practice of Neural-network-based Textual representation › talks › textzoo.pdf · 2020-04-23 · A survey and practice of Neural-network-based Textual representation](https://reader035.fdocuments.in/reader035/viewer/2022070803/5f033e537e708231d4083f33/html5/thumbnails/5.jpg)
Can not do
• Can not directly deploy online• Implementing is easy, while design is what really challenging
• Can not tell you the precise hyper-parameter of your task• A fish or a fishing skill?
• Can not ensure to improve your performance • It depends on the scale of your supervised data
![Page 6: A survey and practice of Neural-network-based Textual representation › talks › textzoo.pdf · 2020-04-23 · A survey and practice of Neural-network-based Textual representation](https://reader035.fdocuments.in/reader035/viewer/2022070803/5f033e537e708231d4083f33/html5/thumbnails/6.jpg)
Highly depends on your data and task
• NLP features extraction Model• TFIDF is enough strong, e.s. long text
• A Few pretrained Model• Glove/Word2vec only for initialization
• No common-known CN embedding
• No pretrained Model
• CV features extraction• SIFT or SIFT-like is not very strong.
• pretrained ResNet from ImageNet
Zero-shot learning can hardly works in NLP, currently
![Page 7: A survey and practice of Neural-network-based Textual representation › talks › textzoo.pdf · 2020-04-23 · A survey and practice of Neural-network-based Textual representation](https://reader035.fdocuments.in/reader035/viewer/2022070803/5f033e537e708231d4083f33/html5/thumbnails/7.jpg)
Can do
• Easy to implement a model after talking• Talking is cheap, 10 lines a model.
• Directly support all the public dataset • Testing model
• Know how to design a DL model for NLP, not only text classification• A fishing skill
![Page 8: A survey and practice of Neural-network-based Textual representation › talks › textzoo.pdf · 2020-04-23 · A survey and practice of Neural-network-based Textual representation](https://reader035.fdocuments.in/reader035/viewer/2022070803/5f033e537e708231d4083f33/html5/thumbnails/8.jpg)
Contents
• Brief Introduction of TextZoo
• Why text classification?
• General Overview of Text Classification
• Overview of Text Classification in Neural Network approach.
• Architecture of TextZoo
• Conclusions
![Page 9: A survey and practice of Neural-network-based Textual representation › talks › textzoo.pdf · 2020-04-23 · A survey and practice of Neural-network-based Textual representation](https://reader035.fdocuments.in/reader035/viewer/2022070803/5f033e537e708231d4083f33/html5/thumbnails/9.jpg)
Contents
• Brief Introduction of TextZoo
• Why text classification?
• General Overview of Text Classification
• Overview of Text Classification in Neural Network approach.
• Architecture of TextZoo
• Conclusions
![Page 10: A survey and practice of Neural-network-based Textual representation › talks › textzoo.pdf · 2020-04-23 · A survey and practice of Neural-network-based Textual representation](https://reader035.fdocuments.in/reader035/viewer/2022070803/5f033e537e708231d4083f33/html5/thumbnails/10.jpg)
TextZoo
• Text Classification• Sentimental
• Topic
• Spam filter
• …
• A benchmark• 20 Dataset
• 20 Models
• PyTorch• Life is short, I use PyTorch(Python)
![Page 11: A survey and practice of Neural-network-based Textual representation › talks › textzoo.pdf · 2020-04-23 · A survey and practice of Neural-network-based Textual representation](https://reader035.fdocuments.in/reader035/viewer/2022070803/5f033e537e708231d4083f33/html5/thumbnails/11.jpg)
Models
✓FasText✓CNN (Kim CNN, Multi-Layer CNN, Multi-perspective CNN, Inception CNN)✓LSTM (BILSTM, StackLSTM, LSTM with Attention )✓Hybrids between CNN and RNN (RCNN, C-LSTM)✓Attention (Self Attention / Quantum Attention)✓Transformer - Attention is all you need✓Capsule✓Quantum-inspired NN➢ConS2S➢Memory Network
![Page 12: A survey and practice of Neural-network-based Textual representation › talks › textzoo.pdf · 2020-04-23 · A survey and practice of Neural-network-based Textual representation](https://reader035.fdocuments.in/reader035/viewer/2022070803/5f033e537e708231d4083f33/html5/thumbnails/12.jpg)
Datasets
• IMDB
• MR
• CR
• MPQA
• SST1
• SST2
• Subj
• TREC
![Page 13: A survey and practice of Neural-network-based Textual representation › talks › textzoo.pdf · 2020-04-23 · A survey and practice of Neural-network-based Textual representation](https://reader035.fdocuments.in/reader035/viewer/2022070803/5f033e537e708231d4083f33/html5/thumbnails/13.jpg)
Contents
• Brief Introduction of TextZoo
• Why text classification?
• General Overview of Text Classification
• Overview of Text Classification in Neural Network approach.
• Architecture of TextZoo
• Conclusions
![Page 14: A survey and practice of Neural-network-based Textual representation › talks › textzoo.pdf · 2020-04-23 · A survey and practice of Neural-network-based Textual representation](https://reader035.fdocuments.in/reader035/viewer/2022070803/5f033e537e708231d4083f33/html5/thumbnails/14.jpg)
Supervised tasks in NLP
![Page 15: A survey and practice of Neural-network-based Textual representation › talks › textzoo.pdf · 2020-04-23 · A survey and practice of Neural-network-based Textual representation](https://reader035.fdocuments.in/reader035/viewer/2022070803/5f033e537e708231d4083f33/html5/thumbnails/15.jpg)
Why text classification?
Text Representation
MLP/CNN/RNNText representation classification
![Page 16: A survey and practice of Neural-network-based Textual representation › talks › textzoo.pdf · 2020-04-23 · A survey and practice of Neural-network-based Textual representation](https://reader035.fdocuments.in/reader035/viewer/2022070803/5f033e537e708231d4083f33/html5/thumbnails/16.jpg)
Why text classification?
Text Representation
MLP/CNN/RNNText representation classification
Text Representation
MLP/CNN/RNNText representation
Matching
![Page 17: A survey and practice of Neural-network-based Textual representation › talks › textzoo.pdf · 2020-04-23 · A survey and practice of Neural-network-based Textual representation](https://reader035.fdocuments.in/reader035/viewer/2022070803/5f033e537e708231d4083f33/html5/thumbnails/17.jpg)
Why text classification?
Text Representation
MLP/CNN/RNNText representation classification
TextMLP/CNN/RNN
![Page 18: A survey and practice of Neural-network-based Textual representation › talks › textzoo.pdf · 2020-04-23 · A survey and practice of Neural-network-based Textual representation](https://reader035.fdocuments.in/reader035/viewer/2022070803/5f033e537e708231d4083f33/html5/thumbnails/18.jpg)
Why text classification?
Text Representation
RNN cellToken_1 representation classification
RNN cellToken_2representation
RNN cellToken_3representation
classification
classification
![Page 19: A survey and practice of Neural-network-based Textual representation › talks › textzoo.pdf · 2020-04-23 · A survey and practice of Neural-network-based Textual representation](https://reader035.fdocuments.in/reader035/viewer/2022070803/5f033e537e708231d4083f33/html5/thumbnails/19.jpg)
Examples for LSTM
https://mp.weixin.qq.com/s/MhRrVW44dDX-PpWNqCWCOw
![Page 20: A survey and practice of Neural-network-based Textual representation › talks › textzoo.pdf · 2020-04-23 · A survey and practice of Neural-network-based Textual representation](https://reader035.fdocuments.in/reader035/viewer/2022070803/5f033e537e708231d4083f33/html5/thumbnails/20.jpg)
Fundamental Demo In Code with PyTorch pseudo code
• Model = LSTM/CNN/Capsule/…
• text,lable = Dataset.nextBatch()
• representation = Model(text)
• Classification = FC(representation) FC : Mapping to label size
• Translation = Decode(representation)
• Matching = Cosine(representation1, representation2)
• Sequential_labelling = FCs(representations )
![Page 21: A survey and practice of Neural-network-based Textual representation › talks › textzoo.pdf · 2020-04-23 · A survey and practice of Neural-network-based Textual representation](https://reader035.fdocuments.in/reader035/viewer/2022070803/5f033e537e708231d4083f33/html5/thumbnails/21.jpg)
Contents
• Brief Introduction of TextZoo
• Why text classification?
• General Overview of Text Classification
• Overview of Text Classification in Neural Network approach.
• Architecture of TextZoo
• Conclusions
![Page 22: A survey and practice of Neural-network-based Textual representation › talks › textzoo.pdf · 2020-04-23 · A survey and practice of Neural-network-based Textual representation](https://reader035.fdocuments.in/reader035/viewer/2022070803/5f033e537e708231d4083f33/html5/thumbnails/22.jpg)
Overview
• Traditional Models• Naïve Bayes
• SVM
• DL Models• ???CNN
• ???RNN
• ???NN
![Page 23: A survey and practice of Neural-network-based Textual representation › talks › textzoo.pdf · 2020-04-23 · A survey and practice of Neural-network-based Textual representation](https://reader035.fdocuments.in/reader035/viewer/2022070803/5f033e537e708231d4083f33/html5/thumbnails/23.jpg)
Traditional Classification
• SVM/Naïve Bayes
• Bag-of-words(N-gram) hypothesis
• Features : • TFIDF (unigram, N-gram)
• POS, parser
• hypernyms, WordNet
• hand-coded rules
• May needs “feature selection”
• Good performance in long text
It performs better than you expected !!
![Page 24: A survey and practice of Neural-network-based Textual representation › talks › textzoo.pdf · 2020-04-23 · A survey and practice of Neural-network-based Textual representation](https://reader035.fdocuments.in/reader035/viewer/2022070803/5f033e537e708231d4083f33/html5/thumbnails/24.jpg)
Contents
• Brief Introduction of TextZoo
• Why text classification?
• General Overview of Text Classification
• Overview of Text Classification in Neural Network approach.
• Architecture of TextZoo
• Conclusions
![Page 25: A survey and practice of Neural-network-based Textual representation › talks › textzoo.pdf · 2020-04-23 · A survey and practice of Neural-network-based Textual representation](https://reader035.fdocuments.in/reader035/viewer/2022070803/5f033e537e708231d4083f33/html5/thumbnails/25.jpg)
Embedding and further DL models
linguistic items with similar distributions have similar meaningsDistributional hypothesis
https://en.wikipedia.org/wiki/Distributional_semantics
![Page 26: A survey and practice of Neural-network-based Textual representation › talks › textzoo.pdf · 2020-04-23 · A survey and practice of Neural-network-based Textual representation](https://reader035.fdocuments.in/reader035/viewer/2022070803/5f033e537e708231d4083f33/html5/thumbnails/26.jpg)
Localist representation
• BMW [1, 0, 0, 0, 0]
• Audi [0, 0, 0, 1, 0]
• Benz [0, 0, 1, 0, 0]
• Polo [0, 0, 0, 1, 0]
http://www.cs.toronto.edu/~bonner/courses/2014s/csc321/lectures/lec5.pdf
[.3, .7, .2, .1, .5]
[.5, .3, .2, .1, .0]
[.2, .0, .31, .03, .01]
[.1, .1, .5, .5, 0.2]
Size color … unknown
![Page 27: A survey and practice of Neural-network-based Textual representation › talks › textzoo.pdf · 2020-04-23 · A survey and practice of Neural-network-based Textual representation](https://reader035.fdocuments.in/reader035/viewer/2022070803/5f033e537e708231d4083f33/html5/thumbnails/27.jpg)
Distributed representation
• BMW [1, 0, 0, 0, 0]
• Audi [0, 0, 0, 1, 0]
• Benz [0, 0, 1, 0, 0]
• Polo [0, 0, 0, 1, 0]
[.3, .7, .2, .1, .5]
[.5, .3, .2, .1, .0]
[.2, .0, .31, .03, .01]
[.1, .1, .5, .5, 0.2]
Size color … unknown
![Page 28: A survey and practice of Neural-network-based Textual representation › talks › textzoo.pdf · 2020-04-23 · A survey and practice of Neural-network-based Textual representation](https://reader035.fdocuments.in/reader035/viewer/2022070803/5f033e537e708231d4083f33/html5/thumbnails/28.jpg)
How to get Distributed representation
• Matrix Factorization• Word-word Matrix
• Document-word Matrix• PLSA
• LDA
• Sample-based Prediction• NNLM
• C & W
• Word2vec
Glove is a combination between these two schools of approaches
Levy, Omer, and Yoav Goldberg. "Neural word embedding as implicit matrix factorization." Advances in neural information processing systems. 2014.
![Page 29: A survey and practice of Neural-network-based Textual representation › talks › textzoo.pdf · 2020-04-23 · A survey and practice of Neural-network-based Textual representation](https://reader035.fdocuments.in/reader035/viewer/2022070803/5f033e537e708231d4083f33/html5/thumbnails/29.jpg)
Why embedding is so hot?
• Only automatically build supervised pairs in unsupervised corpus
• Life is complex. It has both real and imaginary parts
![Page 30: A survey and practice of Neural-network-based Textual representation › talks › textzoo.pdf · 2020-04-23 · A survey and practice of Neural-network-based Textual representation](https://reader035.fdocuments.in/reader035/viewer/2022070803/5f033e537e708231d4083f33/html5/thumbnails/30.jpg)
NNLM
![Page 31: A survey and practice of Neural-network-based Textual representation › talks › textzoo.pdf · 2020-04-23 · A survey and practice of Neural-network-based Textual representation](https://reader035.fdocuments.in/reader035/viewer/2022070803/5f033e537e708231d4083f33/html5/thumbnails/31.jpg)
C&W
![Page 32: A survey and practice of Neural-network-based Textual representation › talks › textzoo.pdf · 2020-04-23 · A survey and practice of Neural-network-based Textual representation](https://reader035.fdocuments.in/reader035/viewer/2022070803/5f033e537e708231d4083f33/html5/thumbnails/32.jpg)
Word2Vec
![Page 33: A survey and practice of Neural-network-based Textual representation › talks › textzoo.pdf · 2020-04-23 · A survey and practice of Neural-network-based Textual representation](https://reader035.fdocuments.in/reader035/viewer/2022070803/5f033e537e708231d4083f33/html5/thumbnails/33.jpg)
State-of-art Embedding
• Word2Vec
• Glove
• Many and many improved version of word embedding
• Improved Word Representation Learning with Sememes
• “Polysemy problem”
• “Antonym problem”
• Complex embedding [We are interested, now]• life is complex, it has both real and imaginary parts
![Page 34: A survey and practice of Neural-network-based Textual representation › talks › textzoo.pdf · 2020-04-23 · A survey and practice of Neural-network-based Textual representation](https://reader035.fdocuments.in/reader035/viewer/2022070803/5f033e537e708231d4083f33/html5/thumbnails/34.jpg)
Which is the most similar word of “Tencent” ?
May be “Baidu” or “pony” ?
Nie Jianyun said in SIGIR 2016 Chinese-Author Workshop, Tsinghua University, Beijing
![Page 35: A survey and practice of Neural-network-based Textual representation › talks › textzoo.pdf · 2020-04-23 · A survey and practice of Neural-network-based Textual representation](https://reader035.fdocuments.in/reader035/viewer/2022070803/5f033e537e708231d4083f33/html5/thumbnails/35.jpg)
Attention!!!
Average Embedding may be a problematic practicefor textual representation, especially in long text.
Should add some supervised signals after embedding to reduce the noise !, like Fastext
Zhang, Xiang, Junbo Zhao, and Yann LeCun. "Character-level convolutional networks for text classification." Advances in neural information processing systems. 2015.
![Page 36: A survey and practice of Neural-network-based Textual representation › talks › textzoo.pdf · 2020-04-23 · A survey and practice of Neural-network-based Textual representation](https://reader035.fdocuments.in/reader035/viewer/2022070803/5f033e537e708231d4083f33/html5/thumbnails/36.jpg)
Embedding is everywhere!!!
• Word2vec
• Doc2vec
• Item2vec
• Everything can be embed!!
Embedding is a kind of approach, while word vector is a typical application of embedding
Wu, Ledell, et al. "StarSpace: Embed All The Things!." arXiv preprint arXiv:1709.03856 (2017).
![Page 37: A survey and practice of Neural-network-based Textual representation › talks › textzoo.pdf · 2020-04-23 · A survey and practice of Neural-network-based Textual representation](https://reader035.fdocuments.in/reader035/viewer/2022070803/5f033e537e708231d4083f33/html5/thumbnails/37.jpg)
How to choose Word Vector
• Word2vec or Glove• Depends on you final performance, not a prior test in linguistic/syntax regulation
• Embedding dim, depends on scale of training dataset.• Larger dataset, bigger dimension, but overfitting.
• If possible, train the embedding on own your data.
Topic-relevant is somehow more important than the data size
![Page 38: A survey and practice of Neural-network-based Textual representation › talks › textzoo.pdf · 2020-04-23 · A survey and practice of Neural-network-based Textual representation](https://reader035.fdocuments.in/reader035/viewer/2022070803/5f033e537e708231d4083f33/html5/thumbnails/38.jpg)
More features in DL
• POS Embedding
• CCG Embedding
• Extract matching Embedding
• Position Embedding
• Embed Every discrete features in Neural Network• If it is continuous, bucket it and make it discrete.
![Page 39: A survey and practice of Neural-network-based Textual representation › talks › textzoo.pdf · 2020-04-23 · A survey and practice of Neural-network-based Textual representation](https://reader035.fdocuments.in/reader035/viewer/2022070803/5f033e537e708231d4083f33/html5/thumbnails/39.jpg)
MLP
![Page 40: A survey and practice of Neural-network-based Textual representation › talks › textzoo.pdf · 2020-04-23 · A survey and practice of Neural-network-based Textual representation](https://reader035.fdocuments.in/reader035/viewer/2022070803/5f033e537e708231d4083f33/html5/thumbnails/40.jpg)
UAT in MLP
Multi-layer Non-linear Mapping - > Universal Approximation Theorem
![Page 41: A survey and practice of Neural-network-based Textual representation › talks › textzoo.pdf · 2020-04-23 · A survey and practice of Neural-network-based Textual representation](https://reader035.fdocuments.in/reader035/viewer/2022070803/5f033e537e708231d4083f33/html5/thumbnails/41.jpg)
A sample of 𝜃(wx+b)
http://neuralnetworksanddeeplearning.com/chap4.html
![Page 42: A survey and practice of Neural-network-based Textual representation › talks › textzoo.pdf · 2020-04-23 · A survey and practice of Neural-network-based Textual representation](https://reader035.fdocuments.in/reader035/viewer/2022070803/5f033e537e708231d4083f33/html5/thumbnails/42.jpg)
An another sample
![Page 43: A survey and practice of Neural-network-based Textual representation › talks › textzoo.pdf · 2020-04-23 · A survey and practice of Neural-network-based Textual representation](https://reader035.fdocuments.in/reader035/viewer/2022070803/5f033e537e708231d4083f33/html5/thumbnails/43.jpg)
CNN
• Basic CNN
• Kalchbrenner N, Grefenstette E, Blunsom P. A convolutional neural network for modelling sentences[J]. arXiv preprint arXiv:1404.2188, 2014
• Kim CNN
• VDCNN
![Page 44: A survey and practice of Neural-network-based Textual representation › talks › textzoo.pdf · 2020-04-23 · A survey and practice of Neural-network-based Textual representation](https://reader035.fdocuments.in/reader035/viewer/2022070803/5f033e537e708231d4083f33/html5/thumbnails/44.jpg)
CNN [Kalchbrenner. et.al ACL 2014]
![Page 45: A survey and practice of Neural-network-based Textual representation › talks › textzoo.pdf · 2020-04-23 · A survey and practice of Neural-network-based Textual representation](https://reader035.fdocuments.in/reader035/viewer/2022070803/5f033e537e708231d4083f33/html5/thumbnails/45.jpg)
CNN [kim EMNLP 2014]
![Page 46: A survey and practice of Neural-network-based Textual representation › talks › textzoo.pdf · 2020-04-23 · A survey and practice of Neural-network-based Textual representation](https://reader035.fdocuments.in/reader035/viewer/2022070803/5f033e537e708231d4083f33/html5/thumbnails/46.jpg)
FASTEX [EACL 2017]
![Page 47: A survey and practice of Neural-network-based Textual representation › talks › textzoo.pdf · 2020-04-23 · A survey and practice of Neural-network-based Textual representation](https://reader035.fdocuments.in/reader035/viewer/2022070803/5f033e537e708231d4083f33/html5/thumbnails/47.jpg)
Why Mr. Lace chooses FasText
• Fast
• Input may a set of keywords instead of a sequential of words• (Group name)
• Label may be inaccurate
• Build more hand-code features would get comparable results
![Page 48: A survey and practice of Neural-network-based Textual representation › talks › textzoo.pdf · 2020-04-23 · A survey and practice of Neural-network-based Textual representation](https://reader035.fdocuments.in/reader035/viewer/2022070803/5f033e537e708231d4083f33/html5/thumbnails/48.jpg)
Very Large CNN [Conneau EACL ]
![Page 49: A survey and practice of Neural-network-based Textual representation › talks › textzoo.pdf · 2020-04-23 · A survey and practice of Neural-network-based Textual representation](https://reader035.fdocuments.in/reader035/viewer/2022070803/5f033e537e708231d4083f33/html5/thumbnails/49.jpg)
Go deeper or not?
• DEEP• Slower
• Overfitting• More Parameters, more data need to feed
• Hard for convergence• Highway network
• Residual Block
• Inception
• Shallow: one-layer• Fast
• Less data, es. Fastext.
![Page 50: A survey and practice of Neural-network-based Textual representation › talks › textzoo.pdf · 2020-04-23 · A survey and practice of Neural-network-based Textual representation](https://reader035.fdocuments.in/reader035/viewer/2022070803/5f033e537e708231d4083f33/html5/thumbnails/50.jpg)
Go deeper or not?
![Page 51: A survey and practice of Neural-network-based Textual representation › talks › textzoo.pdf · 2020-04-23 · A survey and practice of Neural-network-based Textual representation](https://reader035.fdocuments.in/reader035/viewer/2022070803/5f033e537e708231d4083f33/html5/thumbnails/51.jpg)
RNN and its Variant
• RNN
• LSTM
• LSTM + mean
• LSTM + bidirectional
• LSTM + Attention
• LSTM + Stack
• LSTM + Self-Attention
• TreeLSTM
![Page 52: A survey and practice of Neural-network-based Textual representation › talks › textzoo.pdf · 2020-04-23 · A survey and practice of Neural-network-based Textual representation](https://reader035.fdocuments.in/reader035/viewer/2022070803/5f033e537e708231d4083f33/html5/thumbnails/52.jpg)
Bias in RNN
![Page 53: A survey and practice of Neural-network-based Textual representation › talks › textzoo.pdf · 2020-04-23 · A survey and practice of Neural-network-based Textual representation](https://reader035.fdocuments.in/reader035/viewer/2022070803/5f033e537e708231d4083f33/html5/thumbnails/53.jpg)
Bias in RNN
![Page 54: A survey and practice of Neural-network-based Textual representation › talks › textzoo.pdf · 2020-04-23 · A survey and practice of Neural-network-based Textual representation](https://reader035.fdocuments.in/reader035/viewer/2022070803/5f033e537e708231d4083f33/html5/thumbnails/54.jpg)
From RNN to LSTM
http://colah.github.io/posts/2015-08-Understanding-LSTMs/
![Page 55: A survey and practice of Neural-network-based Textual representation › talks › textzoo.pdf · 2020-04-23 · A survey and practice of Neural-network-based Textual representation](https://reader035.fdocuments.in/reader035/viewer/2022070803/5f033e537e708231d4083f33/html5/thumbnails/55.jpg)
LSTM
• How many gates ?
• Difference between cell and the hidden state?
• How many parameters in a LSTM?
![Page 56: A survey and practice of Neural-network-based Textual representation › talks › textzoo.pdf · 2020-04-23 · A survey and practice of Neural-network-based Textual representation](https://reader035.fdocuments.in/reader035/viewer/2022070803/5f033e537e708231d4083f33/html5/thumbnails/56.jpg)
Forget gate
![Page 57: A survey and practice of Neural-network-based Textual representation › talks › textzoo.pdf · 2020-04-23 · A survey and practice of Neural-network-based Textual representation](https://reader035.fdocuments.in/reader035/viewer/2022070803/5f033e537e708231d4083f33/html5/thumbnails/57.jpg)
Input gate
replace tanh with softsign (not softmax) activation for prevent overfitting
https://zhuanlan.zhihu.com/p/21952042
![Page 58: A survey and practice of Neural-network-based Textual representation › talks › textzoo.pdf · 2020-04-23 · A survey and practice of Neural-network-based Textual representation](https://reader035.fdocuments.in/reader035/viewer/2022070803/5f033e537e708231d4083f33/html5/thumbnails/58.jpg)
Forgotten + input
![Page 59: A survey and practice of Neural-network-based Textual representation › talks › textzoo.pdf · 2020-04-23 · A survey and practice of Neural-network-based Textual representation](https://reader035.fdocuments.in/reader035/viewer/2022070803/5f033e537e708231d4083f33/html5/thumbnails/59.jpg)
Output Gate
![Page 60: A survey and practice of Neural-network-based Textual representation › talks › textzoo.pdf · 2020-04-23 · A survey and practice of Neural-network-based Textual representation](https://reader035.fdocuments.in/reader035/viewer/2022070803/5f033e537e708231d4083f33/html5/thumbnails/60.jpg)
LSTM Variants: Peephole connections
![Page 61: A survey and practice of Neural-network-based Textual representation › talks › textzoo.pdf · 2020-04-23 · A survey and practice of Neural-network-based Textual representation](https://reader035.fdocuments.in/reader035/viewer/2022070803/5f033e537e708231d4083f33/html5/thumbnails/61.jpg)
LSTM Variants: coupled forget and input gates
![Page 62: A survey and practice of Neural-network-based Textual representation › talks › textzoo.pdf · 2020-04-23 · A survey and practice of Neural-network-based Textual representation](https://reader035.fdocuments.in/reader035/viewer/2022070803/5f033e537e708231d4083f33/html5/thumbnails/62.jpg)
LSTM Variants: GRU
✓ Hidden = Cell✓ Forget gate + input gate =1
![Page 63: A survey and practice of Neural-network-based Textual representation › talks › textzoo.pdf · 2020-04-23 · A survey and practice of Neural-network-based Textual representation](https://reader035.fdocuments.in/reader035/viewer/2022070803/5f033e537e708231d4083f33/html5/thumbnails/63.jpg)
BiLSTM
![Page 64: A survey and practice of Neural-network-based Textual representation › talks › textzoo.pdf · 2020-04-23 · A survey and practice of Neural-network-based Textual representation](https://reader035.fdocuments.in/reader035/viewer/2022070803/5f033e537e708231d4083f33/html5/thumbnails/64.jpg)
Last or Mean?
![Page 65: A survey and practice of Neural-network-based Textual representation › talks › textzoo.pdf · 2020-04-23 · A survey and practice of Neural-network-based Textual representation](https://reader035.fdocuments.in/reader035/viewer/2022070803/5f033e537e708231d4083f33/html5/thumbnails/65.jpg)
RNN/LSTM with Attention
https://www.jianshu.com/p/4fbc4939509f
![Page 66: A survey and practice of Neural-network-based Textual representation › talks › textzoo.pdf · 2020-04-23 · A survey and practice of Neural-network-based Textual representation](https://reader035.fdocuments.in/reader035/viewer/2022070803/5f033e537e708231d4083f33/html5/thumbnails/66.jpg)
Visualization of Attention in RNN/LSTM
Machine TranslationImage Caption
![Page 67: A survey and practice of Neural-network-based Textual representation › talks › textzoo.pdf · 2020-04-23 · A survey and practice of Neural-network-based Textual representation](https://reader035.fdocuments.in/reader035/viewer/2022070803/5f033e537e708231d4083f33/html5/thumbnails/67.jpg)
Visualization of Attention in RNN/LSTM
Sematic Entailment Speech Recognition
![Page 68: A survey and practice of Neural-network-based Textual representation › talks › textzoo.pdf · 2020-04-23 · A survey and practice of Neural-network-based Textual representation](https://reader035.fdocuments.in/reader035/viewer/2022070803/5f033e537e708231d4083f33/html5/thumbnails/68.jpg)
Deeper LSTM
![Page 69: A survey and practice of Neural-network-based Textual representation › talks › textzoo.pdf · 2020-04-23 · A survey and practice of Neural-network-based Textual representation](https://reader035.fdocuments.in/reader035/viewer/2022070803/5f033e537e708231d4083f33/html5/thumbnails/69.jpg)
Deeper LSTM
Deep is not necessary, but more data!!!
![Page 70: A survey and practice of Neural-network-based Textual representation › talks › textzoo.pdf · 2020-04-23 · A survey and practice of Neural-network-based Textual representation](https://reader035.fdocuments.in/reader035/viewer/2022070803/5f033e537e708231d4083f33/html5/thumbnails/70.jpg)
CNN/RNN
• Comparative Study of CNN and RNN for Natural Language Processing
![Page 71: A survey and practice of Neural-network-based Textual representation › talks › textzoo.pdf · 2020-04-23 · A survey and practice of Neural-network-based Textual representation](https://reader035.fdocuments.in/reader035/viewer/2022070803/5f033e537e708231d4083f33/html5/thumbnails/71.jpg)
RNN vs CNN
![Page 72: A survey and practice of Neural-network-based Textual representation › talks › textzoo.pdf · 2020-04-23 · A survey and practice of Neural-network-based Textual representation](https://reader035.fdocuments.in/reader035/viewer/2022070803/5f033e537e708231d4083f33/html5/thumbnails/72.jpg)
CNN vs RNN vs their Hybrids
http://blog.csdn.net/youngair/article/details/78013352
Dimensional Sentiment Analysis Using a Regional CNN-LSTM Model
![Page 73: A survey and practice of Neural-network-based Textual representation › talks › textzoo.pdf · 2020-04-23 · A survey and practice of Neural-network-based Textual representation](https://reader035.fdocuments.in/reader035/viewer/2022070803/5f033e537e708231d4083f33/html5/thumbnails/73.jpg)
From a Industrial perspective
• Add features.
• Understanding your data : pay more attention on data preparation.
• Parameter adjusting with a robust setting• Oh, overfit
• Model is not very important, especially data is not low-quality. • Models differs slightly in low-quality data.
• Trade-off between performance and efficiency• For example, multi-size kennels is better but slower!
![Page 74: A survey and practice of Neural-network-based Textual representation › talks › textzoo.pdf · 2020-04-23 · A survey and practice of Neural-network-based Textual representation](https://reader035.fdocuments.in/reader035/viewer/2022070803/5f033e537e708231d4083f33/html5/thumbnails/74.jpg)
Related Models
• Do not directly aims at this task, but also aims to build a text representation.• ConvS2S
• Attention is all you need
• Dynamic Memory Network
![Page 75: A survey and practice of Neural-network-based Textual representation › talks › textzoo.pdf · 2020-04-23 · A survey and practice of Neural-network-based Textual representation](https://reader035.fdocuments.in/reader035/viewer/2022070803/5f033e537e708231d4083f33/html5/thumbnails/75.jpg)
Conv S2S
![Page 76: A survey and practice of Neural-network-based Textual representation › talks › textzoo.pdf · 2020-04-23 · A survey and practice of Neural-network-based Textual representation](https://reader035.fdocuments.in/reader035/viewer/2022070803/5f033e537e708231d4083f33/html5/thumbnails/76.jpg)
Attention is all you need
![Page 77: A survey and practice of Neural-network-based Textual representation › talks › textzoo.pdf · 2020-04-23 · A survey and practice of Neural-network-based Textual representation](https://reader035.fdocuments.in/reader035/viewer/2022070803/5f033e537e708231d4083f33/html5/thumbnails/77.jpg)
Self-Attention
![Page 78: A survey and practice of Neural-network-based Textual representation › talks › textzoo.pdf · 2020-04-23 · A survey and practice of Neural-network-based Textual representation](https://reader035.fdocuments.in/reader035/viewer/2022070803/5f033e537e708231d4083f33/html5/thumbnails/78.jpg)
Dynamic Memory Network
![Page 79: A survey and practice of Neural-network-based Textual representation › talks › textzoo.pdf · 2020-04-23 · A survey and practice of Neural-network-based Textual representation](https://reader035.fdocuments.in/reader035/viewer/2022070803/5f033e537e708231d4083f33/html5/thumbnails/79.jpg)
Other models
• Tree-LSTM
• Pointer networks
• Bi-Directional Block Self-Attention for Fast and Memory-Efficient Sequence Modeling (T. Shen et al., ICLR 2018)
• Directional Self-Attention Network
• Recurrent Entity Network
![Page 80: A survey and practice of Neural-network-based Textual representation › talks › textzoo.pdf · 2020-04-23 · A survey and practice of Neural-network-based Textual representation](https://reader035.fdocuments.in/reader035/viewer/2022070803/5f033e537e708231d4083f33/html5/thumbnails/80.jpg)
Char-CNN
Zhang, Xiang, Junbo Zhao, and Yann LeCun. "Character-level convolutional networks for text classification." Advances in neural information processing systems. 2015.
![Page 81: A survey and practice of Neural-network-based Textual representation › talks › textzoo.pdf · 2020-04-23 · A survey and practice of Neural-network-based Textual representation](https://reader035.fdocuments.in/reader035/viewer/2022070803/5f033e537e708231d4083f33/html5/thumbnails/81.jpg)
Component-Enhanced Yanran Li, Wenjie Li, Fei Sun, and Sujian Li. Component-Enhanced Chinese Character Embeddings. Proceedings of EMNLP, 201
Component-Enhanced
![Page 82: A survey and practice of Neural-network-based Textual representation › talks › textzoo.pdf · 2020-04-23 · A survey and practice of Neural-network-based Textual representation](https://reader035.fdocuments.in/reader035/viewer/2022070803/5f033e537e708231d4083f33/html5/thumbnails/82.jpg)
Char-word Hybrids
Combining Word-Level and Character-Level Representations for Relation Classification of Informal Text
![Page 83: A survey and practice of Neural-network-based Textual representation › talks › textzoo.pdf · 2020-04-23 · A survey and practice of Neural-network-based Textual representation](https://reader035.fdocuments.in/reader035/viewer/2022070803/5f033e537e708231d4083f33/html5/thumbnails/83.jpg)
Long text/document classification
• Hierarchical Attention Networks(HAN)
![Page 84: A survey and practice of Neural-network-based Textual representation › talks › textzoo.pdf · 2020-04-23 · A survey and practice of Neural-network-based Textual representation](https://reader035.fdocuments.in/reader035/viewer/2022070803/5f033e537e708231d4083f33/html5/thumbnails/84.jpg)
Multi-task Learning
Pengfei Liu, Xipeng Qiu, Xuanjing Huang, Deep Multi-Task Learning with Shared Memory for Text Classification, In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2016.
![Page 85: A survey and practice of Neural-network-based Textual representation › talks › textzoo.pdf · 2020-04-23 · A survey and practice of Neural-network-based Textual representation](https://reader035.fdocuments.in/reader035/viewer/2022070803/5f033e537e708231d4083f33/html5/thumbnails/85.jpg)
Adversarial Multi-task Learning
Pengfei Liu, Xipeng Qiu, Xuanjing Huang, Adversarial Multi-task Learning for Text Classification, In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL), pp. 1-10, 2017.
![Page 86: A survey and practice of Neural-network-based Textual representation › talks › textzoo.pdf · 2020-04-23 · A survey and practice of Neural-network-based Textual representation](https://reader035.fdocuments.in/reader035/viewer/2022070803/5f033e537e708231d4083f33/html5/thumbnails/86.jpg)
RL for text classfication
• Learning Structured Representation for Text Classification via Reinforcement Learning AAAI 2018 minlieHuang
![Page 87: A survey and practice of Neural-network-based Textual representation › talks › textzoo.pdf · 2020-04-23 · A survey and practice of Neural-network-based Textual representation](https://reader035.fdocuments.in/reader035/viewer/2022070803/5f033e537e708231d4083f33/html5/thumbnails/87.jpg)
Adversarial Training Methods For Semi-supervised Text Classification
• ICLR 2017
![Page 88: A survey and practice of Neural-network-based Textual representation › talks › textzoo.pdf · 2020-04-23 · A survey and practice of Neural-network-based Textual representation](https://reader035.fdocuments.in/reader035/viewer/2022070803/5f033e537e708231d4083f33/html5/thumbnails/88.jpg)
To-do List
• Support more datasets, especially in Chinese
• Support more models
• Fine-tune the result.
• Installable Library with Python (Pip)