A Fast Learning Algorithm for Deep Belief Nets - Department of
APPLIED DEEP LEARNING - IARIAiaria.org/.../DeepLearning_Tutorial_2016.pdf · Complex Deep Nets -...
Transcript of APPLIED DEEP LEARNING - IARIAiaria.org/.../DeepLearning_Tutorial_2016.pdf · Complex Deep Nets -...
![Page 1: APPLIED DEEP LEARNING - IARIAiaria.org/.../DeepLearning_Tutorial_2016.pdf · Complex Deep Nets - only practical choice. DEEP NETS BETTER THAN HUMANS Humans were 5.10% wrong ... SAUCE”](https://reader030.fdocuments.in/reader030/viewer/2022041019/5ecd5c121733ab54aa62555b/html5/thumbnails/1.jpg)
APPLIED DEEPLEARNING
PROFALEXIEIDINGLI
![Page 2: APPLIED DEEP LEARNING - IARIAiaria.org/.../DeepLearning_Tutorial_2016.pdf · Complex Deep Nets - only practical choice. DEEP NETS BETTER THAN HUMANS Humans were 5.10% wrong ... SAUCE”](https://reader030.fdocuments.in/reader030/viewer/2022041019/5ecd5c121733ab54aa62555b/html5/thumbnails/2.jpg)
TECH NEWS
![Page 3: APPLIED DEEP LEARNING - IARIAiaria.org/.../DeepLearning_Tutorial_2016.pdf · Complex Deep Nets - only practical choice. DEEP NETS BETTER THAN HUMANS Humans were 5.10% wrong ... SAUCE”](https://reader030.fdocuments.in/reader030/viewer/2022041019/5ecd5c121733ab54aa62555b/html5/thumbnails/3.jpg)
![Page 4: APPLIED DEEP LEARNING - IARIAiaria.org/.../DeepLearning_Tutorial_2016.pdf · Complex Deep Nets - only practical choice. DEEP NETS BETTER THAN HUMANS Humans were 5.10% wrong ... SAUCE”](https://reader030.fdocuments.in/reader030/viewer/2022041019/5ecd5c121733ab54aa62555b/html5/thumbnails/4.jpg)
TECH NEWS
![Page 5: APPLIED DEEP LEARNING - IARIAiaria.org/.../DeepLearning_Tutorial_2016.pdf · Complex Deep Nets - only practical choice. DEEP NETS BETTER THAN HUMANS Humans were 5.10% wrong ... SAUCE”](https://reader030.fdocuments.in/reader030/viewer/2022041019/5ecd5c121733ab54aa62555b/html5/thumbnails/5.jpg)
HOW TO DO IT?
![Page 6: APPLIED DEEP LEARNING - IARIAiaria.org/.../DeepLearning_Tutorial_2016.pdf · Complex Deep Nets - only practical choice. DEEP NETS BETTER THAN HUMANS Humans were 5.10% wrong ... SAUCE”](https://reader030.fdocuments.in/reader030/viewer/2022041019/5ecd5c121733ab54aa62555b/html5/thumbnails/6.jpg)
TECH NEWS
![Page 7: APPLIED DEEP LEARNING - IARIAiaria.org/.../DeepLearning_Tutorial_2016.pdf · Complex Deep Nets - only practical choice. DEEP NETS BETTER THAN HUMANS Humans were 5.10% wrong ... SAUCE”](https://reader030.fdocuments.in/reader030/viewer/2022041019/5ecd5c121733ab54aa62555b/html5/thumbnails/7.jpg)
APPLICATIONS
![Page 8: APPLIED DEEP LEARNING - IARIAiaria.org/.../DeepLearning_Tutorial_2016.pdf · Complex Deep Nets - only practical choice. DEEP NETS BETTER THAN HUMANS Humans were 5.10% wrong ... SAUCE”](https://reader030.fdocuments.in/reader030/viewer/2022041019/5ecd5c121733ab54aa62555b/html5/thumbnails/8.jpg)
TECH NEWS
![Page 9: APPLIED DEEP LEARNING - IARIAiaria.org/.../DeepLearning_Tutorial_2016.pdf · Complex Deep Nets - only practical choice. DEEP NETS BETTER THAN HUMANS Humans were 5.10% wrong ... SAUCE”](https://reader030.fdocuments.in/reader030/viewer/2022041019/5ecd5c121733ab54aa62555b/html5/thumbnails/9.jpg)
TECH NEWS
![Page 10: APPLIED DEEP LEARNING - IARIAiaria.org/.../DeepLearning_Tutorial_2016.pdf · Complex Deep Nets - only practical choice. DEEP NETS BETTER THAN HUMANS Humans were 5.10% wrong ... SAUCE”](https://reader030.fdocuments.in/reader030/viewer/2022041019/5ecd5c121733ab54aa62555b/html5/thumbnails/10.jpg)
![Page 11: APPLIED DEEP LEARNING - IARIAiaria.org/.../DeepLearning_Tutorial_2016.pdf · Complex Deep Nets - only practical choice. DEEP NETS BETTER THAN HUMANS Humans were 5.10% wrong ... SAUCE”](https://reader030.fdocuments.in/reader030/viewer/2022041019/5ecd5c121733ab54aa62555b/html5/thumbnails/11.jpg)
NEURAL NETWORKS
• Interconnectedsetofnodesandedges• Designedtoperformcomplextasks
• DeepLearningbaseduponneuralnetworks
![Page 12: APPLIED DEEP LEARNING - IARIAiaria.org/.../DeepLearning_Tutorial_2016.pdf · Complex Deep Nets - only practical choice. DEEP NETS BETTER THAN HUMANS Humans were 5.10% wrong ... SAUCE”](https://reader030.fdocuments.in/reader030/viewer/2022041019/5ecd5c121733ab54aa62555b/html5/thumbnails/12.jpg)
CLASSIFICATION
![Page 13: APPLIED DEEP LEARNING - IARIAiaria.org/.../DeepLearning_Tutorial_2016.pdf · Complex Deep Nets - only practical choice. DEEP NETS BETTER THAN HUMANS Humans were 5.10% wrong ... SAUCE”](https://reader030.fdocuments.in/reader030/viewer/2022041019/5ecd5c121733ab54aa62555b/html5/thumbnails/13.jpg)
CLASSIFICATION
![Page 14: APPLIED DEEP LEARNING - IARIAiaria.org/.../DeepLearning_Tutorial_2016.pdf · Complex Deep Nets - only practical choice. DEEP NETS BETTER THAN HUMANS Humans were 5.10% wrong ... SAUCE”](https://reader030.fdocuments.in/reader030/viewer/2022041019/5ecd5c121733ab54aa62555b/html5/thumbnails/14.jpg)
NEURAL NETWORKS
INPUT ENGINE OUTPUT
John
Height
Weight
Body
Temperature
1.7m
98Kgs
38°C
HighScore=Sick!
![Page 15: APPLIED DEEP LEARNING - IARIAiaria.org/.../DeepLearning_Tutorial_2016.pdf · Complex Deep Nets - only practical choice. DEEP NETS BETTER THAN HUMANS Humans were 5.10% wrong ... SAUCE”](https://reader030.fdocuments.in/reader030/viewer/2022041019/5ecd5c121733ab54aa62555b/html5/thumbnails/15.jpg)
NEURAL NETWORKS
INPUTLAYER HIDDENLAYERS OUTPUTLAYER
![Page 16: APPLIED DEEP LEARNING - IARIAiaria.org/.../DeepLearning_Tutorial_2016.pdf · Complex Deep Nets - only practical choice. DEEP NETS BETTER THAN HUMANS Humans were 5.10% wrong ... SAUCE”](https://reader030.fdocuments.in/reader030/viewer/2022041019/5ecd5c121733ab54aa62555b/html5/thumbnails/16.jpg)
NEURAL NETWORKS EXAMPLE
INPUTLAYER HIDDENLAYERS OUTPUTLAYER
![Page 17: APPLIED DEEP LEARNING - IARIAiaria.org/.../DeepLearning_Tutorial_2016.pdf · Complex Deep Nets - only practical choice. DEEP NETS BETTER THAN HUMANS Humans were 5.10% wrong ... SAUCE”](https://reader030.fdocuments.in/reader030/viewer/2022041019/5ecd5c121733ab54aa62555b/html5/thumbnails/17.jpg)
NEURAL NETWORKS EXAMPLE
10
5
6 9
![Page 18: APPLIED DEEP LEARNING - IARIAiaria.org/.../DeepLearning_Tutorial_2016.pdf · Complex Deep Nets - only practical choice. DEEP NETS BETTER THAN HUMANS Humans were 5.10% wrong ... SAUCE”](https://reader030.fdocuments.in/reader030/viewer/2022041019/5ecd5c121733ab54aa62555b/html5/thumbnails/18.jpg)
TRAINING
INPUTLAYER HIDDENLAYERS OUTPUTLAYER
![Page 19: APPLIED DEEP LEARNING - IARIAiaria.org/.../DeepLearning_Tutorial_2016.pdf · Complex Deep Nets - only practical choice. DEEP NETS BETTER THAN HUMANS Humans were 5.10% wrong ... SAUCE”](https://reader030.fdocuments.in/reader030/viewer/2022041019/5ecd5c121733ab54aa62555b/html5/thumbnails/19.jpg)
WHY ARE NEURAL NETS HARD?
INPUTLAYER HIDDENLAYERS OUTPUTLAYER
![Page 20: APPLIED DEEP LEARNING - IARIAiaria.org/.../DeepLearning_Tutorial_2016.pdf · Complex Deep Nets - only practical choice. DEEP NETS BETTER THAN HUMANS Humans were 5.10% wrong ... SAUCE”](https://reader030.fdocuments.in/reader030/viewer/2022041019/5ecd5c121733ab54aa62555b/html5/thumbnails/20.jpg)
BACK PROPAGATION
GradientsinHiddenlayers
Layer30.2500*0.33=0.0825
Layer20.0825*0.25=0.0206
Layer10.0206*0.20=0.004125
…andnumberskeepongettingsmaller
![Page 21: APPLIED DEEP LEARNING - IARIAiaria.org/.../DeepLearning_Tutorial_2016.pdf · Complex Deep Nets - only practical choice. DEEP NETS BETTER THAN HUMANS Humans were 5.10% wrong ... SAUCE”](https://reader030.fdocuments.in/reader030/viewer/2022041019/5ecd5c121733ab54aa62555b/html5/thumbnails/21.jpg)
NEURAL NETWORKS EXAMPLE
• http://playground.tensorflow.org/
![Page 22: APPLIED DEEP LEARNING - IARIAiaria.org/.../DeepLearning_Tutorial_2016.pdf · Complex Deep Nets - only practical choice. DEEP NETS BETTER THAN HUMANS Humans were 5.10% wrong ... SAUCE”](https://reader030.fdocuments.in/reader030/viewer/2022041019/5ecd5c121733ab54aa62555b/html5/thumbnails/22.jpg)
PATTERN COMPLEXITY
Simple UseMLmethodslikeSVM
Moderate DeepNets,SVM
Complex DeepNets-onlypracticalchoice
![Page 23: APPLIED DEEP LEARNING - IARIAiaria.org/.../DeepLearning_Tutorial_2016.pdf · Complex Deep Nets - only practical choice. DEEP NETS BETTER THAN HUMANS Humans were 5.10% wrong ... SAUCE”](https://reader030.fdocuments.in/reader030/viewer/2022041019/5ecd5c121733ab54aa62555b/html5/thumbnails/23.jpg)
DEEP NETS BETTER THAN HUMANSHumanswere5.10%wrong
Microsoftwas4.94%wrong
Googlewas4.82%wrong
Baiduwas4.58%wrong
![Page 24: APPLIED DEEP LEARNING - IARIAiaria.org/.../DeepLearning_Tutorial_2016.pdf · Complex Deep Nets - only practical choice. DEEP NETS BETTER THAN HUMANS Humans were 5.10% wrong ... SAUCE”](https://reader030.fdocuments.in/reader030/viewer/2022041019/5ecd5c121733ab54aa62555b/html5/thumbnails/24.jpg)
HOW DOES IT WORK?
![Page 25: APPLIED DEEP LEARNING - IARIAiaria.org/.../DeepLearning_Tutorial_2016.pdf · Complex Deep Nets - only practical choice. DEEP NETS BETTER THAN HUMANS Humans were 5.10% wrong ... SAUCE”](https://reader030.fdocuments.in/reader030/viewer/2022041019/5ecd5c121733ab54aa62555b/html5/thumbnails/25.jpg)
DEEP ARCHITECTURE
![Page 26: APPLIED DEEP LEARNING - IARIAiaria.org/.../DeepLearning_Tutorial_2016.pdf · Complex Deep Nets - only practical choice. DEEP NETS BETTER THAN HUMANS Humans were 5.10% wrong ... SAUCE”](https://reader030.fdocuments.in/reader030/viewer/2022041019/5ecd5c121733ab54aa62555b/html5/thumbnails/26.jpg)
DEEP ARCHITECTURE
![Page 27: APPLIED DEEP LEARNING - IARIAiaria.org/.../DeepLearning_Tutorial_2016.pdf · Complex Deep Nets - only practical choice. DEEP NETS BETTER THAN HUMANS Humans were 5.10% wrong ... SAUCE”](https://reader030.fdocuments.in/reader030/viewer/2022041019/5ecd5c121733ab54aa62555b/html5/thumbnails/27.jpg)
DEEP ARCHITECTURE
![Page 28: APPLIED DEEP LEARNING - IARIAiaria.org/.../DeepLearning_Tutorial_2016.pdf · Complex Deep Nets - only practical choice. DEEP NETS BETTER THAN HUMANS Humans were 5.10% wrong ... SAUCE”](https://reader030.fdocuments.in/reader030/viewer/2022041019/5ecd5c121733ab54aa62555b/html5/thumbnails/28.jpg)
2006 - 2007
![Page 29: APPLIED DEEP LEARNING - IARIAiaria.org/.../DeepLearning_Tutorial_2016.pdf · Complex Deep Nets - only practical choice. DEEP NETS BETTER THAN HUMANS Humans were 5.10% wrong ... SAUCE”](https://reader030.fdocuments.in/reader030/viewer/2022041019/5ecd5c121733ab54aa62555b/html5/thumbnails/29.jpg)
RESTRICTED BOLTZMANN MACHINE (RBM)
VISIBLE
LAYER
HIDDEN
LAYERS
![Page 30: APPLIED DEEP LEARNING - IARIAiaria.org/.../DeepLearning_Tutorial_2016.pdf · Complex Deep Nets - only practical choice. DEEP NETS BETTER THAN HUMANS Humans were 5.10% wrong ... SAUCE”](https://reader030.fdocuments.in/reader030/viewer/2022041019/5ecd5c121733ab54aa62555b/html5/thumbnails/30.jpg)
DEEP BELIEF NETS (DBNS)
VISIBLE
LAYER
HIDDEN
LAYERS
![Page 31: APPLIED DEEP LEARNING - IARIAiaria.org/.../DeepLearning_Tutorial_2016.pdf · Complex Deep Nets - only practical choice. DEEP NETS BETTER THAN HUMANS Humans were 5.10% wrong ... SAUCE”](https://reader030.fdocuments.in/reader030/viewer/2022041019/5ecd5c121733ab54aa62555b/html5/thumbnails/31.jpg)
AUTOENCODERS
![Page 32: APPLIED DEEP LEARNING - IARIAiaria.org/.../DeepLearning_Tutorial_2016.pdf · Complex Deep Nets - only practical choice. DEEP NETS BETTER THAN HUMANS Humans were 5.10% wrong ... SAUCE”](https://reader030.fdocuments.in/reader030/viewer/2022041019/5ecd5c121733ab54aa62555b/html5/thumbnails/32.jpg)
CONVOLUTIONAL NEURAL NETWORK (CNN)
![Page 33: APPLIED DEEP LEARNING - IARIAiaria.org/.../DeepLearning_Tutorial_2016.pdf · Complex Deep Nets - only practical choice. DEEP NETS BETTER THAN HUMANS Humans were 5.10% wrong ... SAUCE”](https://reader030.fdocuments.in/reader030/viewer/2022041019/5ecd5c121733ab54aa62555b/html5/thumbnails/33.jpg)
CONVOLUTIONAL NEURAL NETWORK (CNN)
![Page 34: APPLIED DEEP LEARNING - IARIAiaria.org/.../DeepLearning_Tutorial_2016.pdf · Complex Deep Nets - only practical choice. DEEP NETS BETTER THAN HUMANS Humans were 5.10% wrong ... SAUCE”](https://reader030.fdocuments.in/reader030/viewer/2022041019/5ecd5c121733ab54aa62555b/html5/thumbnails/34.jpg)
CONVOLUTIONAL NEURAL NETWORK (CNN)
Filter1
Filter2
Filter3
![Page 35: APPLIED DEEP LEARNING - IARIAiaria.org/.../DeepLearning_Tutorial_2016.pdf · Complex Deep Nets - only practical choice. DEEP NETS BETTER THAN HUMANS Humans were 5.10% wrong ... SAUCE”](https://reader030.fdocuments.in/reader030/viewer/2022041019/5ecd5c121733ab54aa62555b/html5/thumbnails/35.jpg)
CONVOLUTIONAL NEURAL NETWORK (CNN)
![Page 36: APPLIED DEEP LEARNING - IARIAiaria.org/.../DeepLearning_Tutorial_2016.pdf · Complex Deep Nets - only practical choice. DEEP NETS BETTER THAN HUMANS Humans were 5.10% wrong ... SAUCE”](https://reader030.fdocuments.in/reader030/viewer/2022041019/5ecd5c121733ab54aa62555b/html5/thumbnails/36.jpg)
CONVOLUTIONAL NEURAL NETWORK (CNN)FILTER1 FILTER2 FILTER3
CONVOLUTIONALLAYER RECTIFIEDLINEARUNIT POOLING
![Page 37: APPLIED DEEP LEARNING - IARIAiaria.org/.../DeepLearning_Tutorial_2016.pdf · Complex Deep Nets - only practical choice. DEEP NETS BETTER THAN HUMANS Humans were 5.10% wrong ... SAUCE”](https://reader030.fdocuments.in/reader030/viewer/2022041019/5ecd5c121733ab54aa62555b/html5/thumbnails/37.jpg)
CONVOLUTIONAL NEURAL NETWORK (CNN)
CONVOLUTIONALLAYER RECTIFIEDLINEARUNIT POOLING FC
![Page 38: APPLIED DEEP LEARNING - IARIAiaria.org/.../DeepLearning_Tutorial_2016.pdf · Complex Deep Nets - only practical choice. DEEP NETS BETTER THAN HUMANS Humans were 5.10% wrong ... SAUCE”](https://reader030.fdocuments.in/reader030/viewer/2022041019/5ecd5c121733ab54aa62555b/html5/thumbnails/38.jpg)
CNN ARCHITECTURE
C
N
N
R
E
L
U
P
O
O
L
C
N
N
R
E
L
U
C
N
N
R
E
L
U
P
O
O
L
C
N
N
R
E
L
U
C
N
N
R
E
L
U
P
O
O
L
C
N
N
R
E
L
U
F
C
Boat
Sea
Duck
Car
![Page 39: APPLIED DEEP LEARNING - IARIAiaria.org/.../DeepLearning_Tutorial_2016.pdf · Complex Deep Nets - only practical choice. DEEP NETS BETTER THAN HUMANS Humans were 5.10% wrong ... SAUCE”](https://reader030.fdocuments.in/reader030/viewer/2022041019/5ecd5c121733ab54aa62555b/html5/thumbnails/39.jpg)
RECURRENT NEURAL NETWORK (RNN)
![Page 40: APPLIED DEEP LEARNING - IARIAiaria.org/.../DeepLearning_Tutorial_2016.pdf · Complex Deep Nets - only practical choice. DEEP NETS BETTER THAN HUMANS Humans were 5.10% wrong ... SAUCE”](https://reader030.fdocuments.in/reader030/viewer/2022041019/5ecd5c121733ab54aa62555b/html5/thumbnails/40.jpg)
RNN STRUCTURE
![Page 41: APPLIED DEEP LEARNING - IARIAiaria.org/.../DeepLearning_Tutorial_2016.pdf · Complex Deep Nets - only practical choice. DEEP NETS BETTER THAN HUMANS Humans were 5.10% wrong ... SAUCE”](https://reader030.fdocuments.in/reader030/viewer/2022041019/5ecd5c121733ab54aa62555b/html5/thumbnails/41.jpg)
RECURSIVE NEURAL TENSOR NETWORKS (RNTNS)
Score
Root
Leaf Leaf
Class
![Page 42: APPLIED DEEP LEARNING - IARIAiaria.org/.../DeepLearning_Tutorial_2016.pdf · Complex Deep Nets - only practical choice. DEEP NETS BETTER THAN HUMANS Humans were 5.10% wrong ... SAUCE”](https://reader030.fdocuments.in/reader030/viewer/2022041019/5ecd5c121733ab54aa62555b/html5/thumbnails/42.jpg)
RECURSION
The Car
Is
Fast
![Page 43: APPLIED DEEP LEARNING - IARIAiaria.org/.../DeepLearning_Tutorial_2016.pdf · Complex Deep Nets - only practical choice. DEEP NETS BETTER THAN HUMANS Humans were 5.10% wrong ... SAUCE”](https://reader030.fdocuments.in/reader030/viewer/2022041019/5ecd5c121733ab54aa62555b/html5/thumbnails/43.jpg)
HOW TO CHOOSE A DNN?
✔ UnlabelledData
✔ FeatureExtraction
✔ UnsupervisedLearning
✔ PatternRecognition
RestrictedBoltzmann
Machine(RBM)
Autoencoder}
![Page 44: APPLIED DEEP LEARNING - IARIAiaria.org/.../DeepLearning_Tutorial_2016.pdf · Complex Deep Nets - only practical choice. DEEP NETS BETTER THAN HUMANS Humans were 5.10% wrong ... SAUCE”](https://reader030.fdocuments.in/reader030/viewer/2022041019/5ecd5c121733ab54aa62555b/html5/thumbnails/44.jpg)
HOW TO CHOOSE A DNN?
✔ LabelledData
✔ TextualProcessing
✔ SentimentAnalysis
✔ Parsing
✔NamedEntity
Recognition
RecursiveNeural
TensorNetwork
(RNTN)
RecurrentNet}
![Page 45: APPLIED DEEP LEARNING - IARIAiaria.org/.../DeepLearning_Tutorial_2016.pdf · Complex Deep Nets - only practical choice. DEEP NETS BETTER THAN HUMANS Humans were 5.10% wrong ... SAUCE”](https://reader030.fdocuments.in/reader030/viewer/2022041019/5ecd5c121733ab54aa62555b/html5/thumbnails/45.jpg)
HOW TO CHOOSE A DNN?
✔Object
Recognition
RecursiveNeural
TensorNetwork
(RNTN)
ConvolutionalNet}
![Page 46: APPLIED DEEP LEARNING - IARIAiaria.org/.../DeepLearning_Tutorial_2016.pdf · Complex Deep Nets - only practical choice. DEEP NETS BETTER THAN HUMANS Humans were 5.10% wrong ... SAUCE”](https://reader030.fdocuments.in/reader030/viewer/2022041019/5ecd5c121733ab54aa62555b/html5/thumbnails/46.jpg)
HOW TO CHOOSE A DNN?
✔Image
Classification
DeepBelieve
Network
(DBN)
ConvolutionalNet}
![Page 47: APPLIED DEEP LEARNING - IARIAiaria.org/.../DeepLearning_Tutorial_2016.pdf · Complex Deep Nets - only practical choice. DEEP NETS BETTER THAN HUMANS Humans were 5.10% wrong ... SAUCE”](https://reader030.fdocuments.in/reader030/viewer/2022041019/5ecd5c121733ab54aa62555b/html5/thumbnails/47.jpg)
HOW TO CHOOSE A DNN?
✔Speech
RecognitionRecurrentNet}
![Page 48: APPLIED DEEP LEARNING - IARIAiaria.org/.../DeepLearning_Tutorial_2016.pdf · Complex Deep Nets - only practical choice. DEEP NETS BETTER THAN HUMANS Humans were 5.10% wrong ... SAUCE”](https://reader030.fdocuments.in/reader030/viewer/2022041019/5ecd5c121733ab54aa62555b/html5/thumbnails/48.jpg)
HOW TO CHOOSE A DNN?
✔ Classification
Multilayer
Perceptron
(MLP)with
Rectified
LinearUnits
(RELU)
DeepBeliefNet
}
![Page 49: APPLIED DEEP LEARNING - IARIAiaria.org/.../DeepLearning_Tutorial_2016.pdf · Complex Deep Nets - only practical choice. DEEP NETS BETTER THAN HUMANS Humans were 5.10% wrong ... SAUCE”](https://reader030.fdocuments.in/reader030/viewer/2022041019/5ecd5c121733ab54aa62555b/html5/thumbnails/49.jpg)
DEEP LEARNING PLATFORMS OR LIBRARIES?
![Page 50: APPLIED DEEP LEARNING - IARIAiaria.org/.../DeepLearning_Tutorial_2016.pdf · Complex Deep Nets - only practical choice. DEEP NETS BETTER THAN HUMANS Humans were 5.10% wrong ... SAUCE”](https://reader030.fdocuments.in/reader030/viewer/2022041019/5ecd5c121733ab54aa62555b/html5/thumbnails/50.jpg)
TENSOR FLOW
“Machine learning is a core, transformative way by which we’re rethinking everything we’re doing” Sundar Pichai, Alphabet CEO
![Page 51: APPLIED DEEP LEARNING - IARIAiaria.org/.../DeepLearning_Tutorial_2016.pdf · Complex Deep Nets - only practical choice. DEEP NETS BETTER THAN HUMANS Humans were 5.10% wrong ... SAUCE”](https://reader030.fdocuments.in/reader030/viewer/2022041019/5ecd5c121733ab54aa62555b/html5/thumbnails/51.jpg)
WHAT ARE COMPUTATIONAL GRAPHS?
• Adirectedgraph
• Definescomputationalstructures
• Functionschainedtogethertoproducedaspecificoutput
• Wecanconstruct,complextransformationondatausingsmall,well-defined
mathematicalfunctions
• CoreofTensorFlowPrograms
![Page 52: APPLIED DEEP LEARNING - IARIAiaria.org/.../DeepLearning_Tutorial_2016.pdf · Complex Deep Nets - only practical choice. DEEP NETS BETTER THAN HUMANS Humans were 5.10% wrong ... SAUCE”](https://reader030.fdocuments.in/reader030/viewer/2022041019/5ecd5c121733ab54aa62555b/html5/thumbnails/52.jpg)
BASIC FUNCTION
SUM
1
2
3
![Page 53: APPLIED DEEP LEARNING - IARIAiaria.org/.../DeepLearning_Tutorial_2016.pdf · Complex Deep Nets - only practical choice. DEEP NETS BETTER THAN HUMANS Humans were 5.10% wrong ... SAUCE”](https://reader030.fdocuments.in/reader030/viewer/2022041019/5ecd5c121733ab54aa62555b/html5/thumbnails/53.jpg)
COMPLEX FUNCTION
SUM
5
3
23SUM
MULINPUT
INPUT
5
5
3
3
15
8
![Page 54: APPLIED DEEP LEARNING - IARIAiaria.org/.../DeepLearning_Tutorial_2016.pdf · Complex Deep Nets - only practical choice. DEEP NETS BETTER THAN HUMANS Humans were 5.10% wrong ... SAUCE”](https://reader030.fdocuments.in/reader030/viewer/2022041019/5ecd5c121733ab54aa62555b/html5/thumbnails/54.jpg)
MORE COMPLEX FUNCTION
SUM
5
3
26SUM
MULINPUT
INPUT
5
5
3
3
15
8
3
![Page 55: APPLIED DEEP LEARNING - IARIAiaria.org/.../DeepLearning_Tutorial_2016.pdf · Complex Deep Nets - only practical choice. DEEP NETS BETTER THAN HUMANS Humans were 5.10% wrong ... SAUCE”](https://reader030.fdocuments.in/reader030/viewer/2022041019/5ecd5c121733ab54aa62555b/html5/thumbnails/55.jpg)
DEPENDENCIES
SUM
5
3
23SUM
MULINPUT
INPUT
5
5
3
3
15
8
![Page 56: APPLIED DEEP LEARNING - IARIAiaria.org/.../DeepLearning_Tutorial_2016.pdf · Complex Deep Nets - only practical choice. DEEP NETS BETTER THAN HUMANS Humans were 5.10% wrong ... SAUCE”](https://reader030.fdocuments.in/reader030/viewer/2022041019/5ecd5c121733ab54aa62555b/html5/thumbnails/56.jpg)
DEPENDENCIES
ADDINPUT MUL
ADDINPUT
![Page 57: APPLIED DEEP LEARNING - IARIAiaria.org/.../DeepLearning_Tutorial_2016.pdf · Complex Deep Nets - only practical choice. DEEP NETS BETTER THAN HUMANS Humans were 5.10% wrong ... SAUCE”](https://reader030.fdocuments.in/reader030/viewer/2022041019/5ecd5c121733ab54aa62555b/html5/thumbnails/57.jpg)
CIRCULAR DEPENDENCIES
SUM
5
3
SUM
MULINPUT
INPUT
5
5
15
![Page 58: APPLIED DEEP LEARNING - IARIAiaria.org/.../DeepLearning_Tutorial_2016.pdf · Complex Deep Nets - only practical choice. DEEP NETS BETTER THAN HUMANS Humans were 5.10% wrong ... SAUCE”](https://reader030.fdocuments.in/reader030/viewer/2022041019/5ecd5c121733ab54aa62555b/html5/thumbnails/58.jpg)
UNROLLING THE GRAPH
![Page 59: APPLIED DEEP LEARNING - IARIAiaria.org/.../DeepLearning_Tutorial_2016.pdf · Complex Deep Nets - only practical choice. DEEP NETS BETTER THAN HUMANS Humans were 5.10% wrong ... SAUCE”](https://reader030.fdocuments.in/reader030/viewer/2022041019/5ecd5c121733ab54aa62555b/html5/thumbnails/59.jpg)
COMPUTATIONAL GRAPHS AND TENSOR FLOW
1.DefineComputationalGraph
2.RuntheGraph(includingthedata)
![Page 60: APPLIED DEEP LEARNING - IARIAiaria.org/.../DeepLearning_Tutorial_2016.pdf · Complex Deep Nets - only practical choice. DEEP NETS BETTER THAN HUMANS Humans were 5.10% wrong ... SAUCE”](https://reader030.fdocuments.in/reader030/viewer/2022041019/5ecd5c121733ab54aa62555b/html5/thumbnails/60.jpg)
MULTIPLICATION FUNCTION
3
3
9MUL
INPUT
INPUT
3
3
![Page 61: APPLIED DEEP LEARNING - IARIAiaria.org/.../DeepLearning_Tutorial_2016.pdf · Complex Deep Nets - only practical choice. DEEP NETS BETTER THAN HUMANS Humans were 5.10% wrong ... SAUCE”](https://reader030.fdocuments.in/reader030/viewer/2022041019/5ecd5c121733ab54aa62555b/html5/thumbnails/61.jpg)
TENSORFLOW - MULTIPLICATION
import tensorflow as tf
a = tf.placeholder("float") b = tf.placeholder("float")
y = tf.mul(a, b)
sess = tf.Session()
print sess.run(y, feed_dict={a: 3, b: 3})
![Page 62: APPLIED DEEP LEARNING - IARIAiaria.org/.../DeepLearning_Tutorial_2016.pdf · Complex Deep Nets - only practical choice. DEEP NETS BETTER THAN HUMANS Humans were 5.10% wrong ... SAUCE”](https://reader030.fdocuments.in/reader030/viewer/2022041019/5ecd5c121733ab54aa62555b/html5/thumbnails/62.jpg)
TENSORFLOW - LINEAR REGRESSION
import numpy as np import matplotlib.pyplot as plt
num_points = 1000 vectors_set = []
for i in xrange(num_points): x1= np.random.normal(0.0, 0.55) y1= x1 * 0.1 + 0.3 + np.random.normal(0.0, 0.03) vectors_set.append([x1, y1])
x_data = [v[0] for v in vectors_set] y_data = [v[1] for v in vectors_set]
plt.plot(x_data, y_data, 'ro', label='Original data') plt.legend() plt.show()
![Page 63: APPLIED DEEP LEARNING - IARIAiaria.org/.../DeepLearning_Tutorial_2016.pdf · Complex Deep Nets - only practical choice. DEEP NETS BETTER THAN HUMANS Humans were 5.10% wrong ... SAUCE”](https://reader030.fdocuments.in/reader030/viewer/2022041019/5ecd5c121733ab54aa62555b/html5/thumbnails/63.jpg)
TENSORFLOW LINEAR REGRESSION
![Page 64: APPLIED DEEP LEARNING - IARIAiaria.org/.../DeepLearning_Tutorial_2016.pdf · Complex Deep Nets - only practical choice. DEEP NETS BETTER THAN HUMANS Humans were 5.10% wrong ... SAUCE”](https://reader030.fdocuments.in/reader030/viewer/2022041019/5ecd5c121733ab54aa62555b/html5/thumbnails/64.jpg)
TENSORFLOW LINEAR REGRESSION
W = tf.Variable(tf.random_uniform([1], -1.0, 1.0)) b = tf.Variable(tf.zeros([1])) y = W * x_data + b
loss = tf.reduce_mean(tf.square(y - y_data))
optimizer = tf.train.GradientDescentOptimizer(0.5) train = optimizer.minimize(loss)
init = tf.initialize_all_variables() sess = tf.Session() sess.run(init)
for step in xrange(8): sess.run(train) print step, sess.run(W), sess.run(b)
plt.plot(x_data, y_data, 'ro') plt.plot(x_data, sess.run(W) * x_data + sess.run(b)) plt.legend() plt.show()
![Page 65: APPLIED DEEP LEARNING - IARIAiaria.org/.../DeepLearning_Tutorial_2016.pdf · Complex Deep Nets - only practical choice. DEEP NETS BETTER THAN HUMANS Humans were 5.10% wrong ... SAUCE”](https://reader030.fdocuments.in/reader030/viewer/2022041019/5ecd5c121733ab54aa62555b/html5/thumbnails/65.jpg)
TENSOR FLOW - MNIST DATASET
![Page 66: APPLIED DEEP LEARNING - IARIAiaria.org/.../DeepLearning_Tutorial_2016.pdf · Complex Deep Nets - only practical choice. DEEP NETS BETTER THAN HUMANS Humans were 5.10% wrong ... SAUCE”](https://reader030.fdocuments.in/reader030/viewer/2022041019/5ecd5c121733ab54aa62555b/html5/thumbnails/66.jpg)
MNIST DATASET USING TENSORFLOW
![Page 67: APPLIED DEEP LEARNING - IARIAiaria.org/.../DeepLearning_Tutorial_2016.pdf · Complex Deep Nets - only practical choice. DEEP NETS BETTER THAN HUMANS Humans were 5.10% wrong ... SAUCE”](https://reader030.fdocuments.in/reader030/viewer/2022041019/5ecd5c121733ab54aa62555b/html5/thumbnails/67.jpg)
MNIST DATASET USING TENSORFLOW
![Page 68: APPLIED DEEP LEARNING - IARIAiaria.org/.../DeepLearning_Tutorial_2016.pdf · Complex Deep Nets - only practical choice. DEEP NETS BETTER THAN HUMANS Humans were 5.10% wrong ... SAUCE”](https://reader030.fdocuments.in/reader030/viewer/2022041019/5ecd5c121733ab54aa62555b/html5/thumbnails/68.jpg)
MNIST DATASET USING TENSORFLOW
![Page 69: APPLIED DEEP LEARNING - IARIAiaria.org/.../DeepLearning_Tutorial_2016.pdf · Complex Deep Nets - only practical choice. DEEP NETS BETTER THAN HUMANS Humans were 5.10% wrong ... SAUCE”](https://reader030.fdocuments.in/reader030/viewer/2022041019/5ecd5c121733ab54aa62555b/html5/thumbnails/69.jpg)
MNIST DATASET USING TENSORFLOW - SETUP
importtensorflowastf
x=tf.placeholder(tf.float32,[None,784])
W=tf.Variable(tf.zeros([784,10]))
b=tf.Variable(tf.zeros([10]))
y=tf.nn.softmax(tf.matmul(x,W)+b)
fromtensorflow.examples.tutorials.mnistimportinput_data
mnist=input_data.read_data_sets("MNIST_data",one_hot=True)
![Page 70: APPLIED DEEP LEARNING - IARIAiaria.org/.../DeepLearning_Tutorial_2016.pdf · Complex Deep Nets - only practical choice. DEEP NETS BETTER THAN HUMANS Humans were 5.10% wrong ... SAUCE”](https://reader030.fdocuments.in/reader030/viewer/2022041019/5ecd5c121733ab54aa62555b/html5/thumbnails/70.jpg)
MNIST DATASET USING TENSORFLOW - TRAINING
y_=tf.placeholder(tf.float32,[None,10])
cross_entropy=tf.reduce_mean(-tf.reduce_sum(y_*tf.log(y),reduction_indices=[1]))
train_step=tf.train.GradientDescentOptimizer(0.5).minimize(cross_entropy)
init=tf.initialize_all_variables()
sess=tf.Session()
sess.run(init)
![Page 71: APPLIED DEEP LEARNING - IARIAiaria.org/.../DeepLearning_Tutorial_2016.pdf · Complex Deep Nets - only practical choice. DEEP NETS BETTER THAN HUMANS Humans were 5.10% wrong ... SAUCE”](https://reader030.fdocuments.in/reader030/viewer/2022041019/5ecd5c121733ab54aa62555b/html5/thumbnails/71.jpg)
MNIST DATASET USING TENSORFLOW - TRAINING SETUP
foriinrange(1000):
batch_xs,batch_ys=mnist.train.next_batch(100)
sess.run(train_step,feed_dict={x:batch_xs,y_:batch_ys})
![Page 72: APPLIED DEEP LEARNING - IARIAiaria.org/.../DeepLearning_Tutorial_2016.pdf · Complex Deep Nets - only practical choice. DEEP NETS BETTER THAN HUMANS Humans were 5.10% wrong ... SAUCE”](https://reader030.fdocuments.in/reader030/viewer/2022041019/5ecd5c121733ab54aa62555b/html5/thumbnails/72.jpg)
MNIST DATASET USING TENSORFLOW - EVALUATION
correct_prediction=tf.equal(tf.argmax(y,1),tf.argmax(y_,1))
accuracy=tf.reduce_mean(tf.cast(correct_prediction,tf.float32))
![Page 73: APPLIED DEEP LEARNING - IARIAiaria.org/.../DeepLearning_Tutorial_2016.pdf · Complex Deep Nets - only practical choice. DEEP NETS BETTER THAN HUMANS Humans were 5.10% wrong ... SAUCE”](https://reader030.fdocuments.in/reader030/viewer/2022041019/5ecd5c121733ab54aa62555b/html5/thumbnails/73.jpg)
WORD EMBEDDINGS
• Wordembeddingsisafunc1onmappingwordsinsomelanguagetoahigh-dimensionalvector• ! :"#$%& →ℝ↑) • ! (cat)=(0.3, 0.7, −2.3, …)• Wisini1alizedrandomlyandthenitlearnsmeaningfulvectorstoperformsometask(E.g.
gramma1calcorrectness)• E.g.learnif5wordsarevalid
• catsatonthemat • catsatsongthemat
• R(W(“cat”),W(“sat”),W(“on”),W(“the”),W(“mat”)) =1• R(W(“cat”),W(“sat”),W(“song”),W(“the”),W(“mat”)) =0
![Page 74: APPLIED DEEP LEARNING - IARIAiaria.org/.../DeepLearning_Tutorial_2016.pdf · Complex Deep Nets - only practical choice. DEEP NETS BETTER THAN HUMANS Humans were 5.10% wrong ... SAUCE”](https://reader030.fdocuments.in/reader030/viewer/2022041019/5ecd5c121733ab54aa62555b/html5/thumbnails/74.jpg)
VISUALISE THE WORD EMBEDDINGS
![Page 75: APPLIED DEEP LEARNING - IARIAiaria.org/.../DeepLearning_Tutorial_2016.pdf · Complex Deep Nets - only practical choice. DEEP NETS BETTER THAN HUMANS Humans were 5.10% wrong ... SAUCE”](https://reader030.fdocuments.in/reader030/viewer/2022041019/5ecd5c121733ab54aa62555b/html5/thumbnails/75.jpg)
EXAMPLEAfewpeopleeatwell
Becomes
Acouplepeopleeatwell
Possiblenumberof5-gramsisenormous
FewandCouplewillbelocatedclosetogetherthusallowingforgeneralizationbetween
Sentencestoa
Classofsimilarsentences
Handlesnotjustsynonymsbutotherclasseslikecolorslike…
Thewallisblue
Thewallisred
![Page 76: APPLIED DEEP LEARNING - IARIAiaria.org/.../DeepLearning_Tutorial_2016.pdf · Complex Deep Nets - only practical choice. DEEP NETS BETTER THAN HUMANS Humans were 5.10% wrong ... SAUCE”](https://reader030.fdocuments.in/reader030/viewer/2022041019/5ecd5c121733ab54aa62555b/html5/thumbnails/76.jpg)
WHAT ELSE CAN WE LEARN?Gender
Man – Woman
Uncle – Aunt
King – Queen
Severalotherrelationships
France – Paris
Big – Bigger
Einstein – Scientist
Berlusconi – Silvio
Japan – Sushi
Messi – Midfielder
![Page 77: APPLIED DEEP LEARNING - IARIAiaria.org/.../DeepLearning_Tutorial_2016.pdf · Complex Deep Nets - only practical choice. DEEP NETS BETTER THAN HUMANS Humans were 5.10% wrong ... SAUCE”](https://reader030.fdocuments.in/reader030/viewer/2022041019/5ecd5c121733ab54aa62555b/html5/thumbnails/77.jpg)
THE USE OF WORD REPRESENTATIONS… HAS BECOME A KEY “SECRET SAUCE” FOR THE SUCCESS OF MANY NLP SYSTEMS IN RECENT YEARS, ACROSS TASKS INCLUDING NAMED ENTITY RECOGNITION, PART-OF-
SPEECH TAGGING, PARSING, AND SEMANTIC ROLE LABELING.
LUONG ET AL. (2013)
![Page 78: APPLIED DEEP LEARNING - IARIAiaria.org/.../DeepLearning_Tutorial_2016.pdf · Complex Deep Nets - only practical choice. DEEP NETS BETTER THAN HUMANS Humans were 5.10% wrong ... SAUCE”](https://reader030.fdocuments.in/reader030/viewer/2022041019/5ecd5c121733ab54aa62555b/html5/thumbnails/78.jpg)
WHAT DOES IT HAS TO DO WITH DEEP LEARNING?
It’satacticwidelyutilisedaswell
LearnagoodrepresentationfortaskA
UseitontaskB
Called
Retraining
Transferlearning
Multi-tasklearning
Usingthisapproach,representationcanlearnfrommorethanonekindofdata
![Page 79: APPLIED DEEP LEARNING - IARIAiaria.org/.../DeepLearning_Tutorial_2016.pdf · Complex Deep Nets - only practical choice. DEEP NETS BETTER THAN HUMANS Humans were 5.10% wrong ... SAUCE”](https://reader030.fdocuments.in/reader030/viewer/2022041019/5ecd5c121733ab54aa62555b/html5/thumbnails/79.jpg)
LEARN TO MAP MULTIPLE KINDS OF DATA
Anothertrickusedindeeplearningistolearnmappingsbetweendatainasinglerepresentation
E.g.EnglishWordsandMandarinWords
Wordswithsimilarmeaningsappearclosetogether
Sodowordswedidn’tknowabout
Sousingwhatwe’veseensofar
IfweknowtwomalematchingwordsinEnglishandChinese
Wecanfindthetwofemaleequivalent
![Page 80: APPLIED DEEP LEARNING - IARIAiaria.org/.../DeepLearning_Tutorial_2016.pdf · Complex Deep Nets - only practical choice. DEEP NETS BETTER THAN HUMANS Humans were 5.10% wrong ... SAUCE”](https://reader030.fdocuments.in/reader030/viewer/2022041019/5ecd5c121733ab54aa62555b/html5/thumbnails/80.jpg)
USING DEEP LEARNING TO GO FURTHER
Sofarwe’vedealtwithwords
Butwhataboutimages?
Imagesofdogs,horsesandautomobilesare
mappednexttotheirwordvector
Whathappenswithnewclassessuchascats?
Theyaremappedclosetothedogvector
Butalittledistantfromit
![Page 81: APPLIED DEEP LEARNING - IARIAiaria.org/.../DeepLearning_Tutorial_2016.pdf · Complex Deep Nets - only practical choice. DEEP NETS BETTER THAN HUMANS Humans were 5.10% wrong ... SAUCE”](https://reader030.fdocuments.in/reader030/viewer/2022041019/5ecd5c121733ab54aa62555b/html5/thumbnails/81.jpg)
![Page 82: APPLIED DEEP LEARNING - IARIAiaria.org/.../DeepLearning_Tutorial_2016.pdf · Complex Deep Nets - only practical choice. DEEP NETS BETTER THAN HUMANS Humans were 5.10% wrong ... SAUCE”](https://reader030.fdocuments.in/reader030/viewer/2022041019/5ecd5c121733ab54aa62555b/html5/thumbnails/82.jpg)
![Page 83: APPLIED DEEP LEARNING - IARIAiaria.org/.../DeepLearning_Tutorial_2016.pdf · Complex Deep Nets - only practical choice. DEEP NETS BETTER THAN HUMANS Humans were 5.10% wrong ... SAUCE”](https://reader030.fdocuments.in/reader030/viewer/2022041019/5ecd5c121733ab54aa62555b/html5/thumbnails/83.jpg)
DEMO - AUTO ENCODERS
http://cs.stanford.edu/people/karpathy/convnetjs/demo/autoencoder.html
![Page 84: APPLIED DEEP LEARNING - IARIAiaria.org/.../DeepLearning_Tutorial_2016.pdf · Complex Deep Nets - only practical choice. DEEP NETS BETTER THAN HUMANS Humans were 5.10% wrong ... SAUCE”](https://reader030.fdocuments.in/reader030/viewer/2022041019/5ecd5c121733ab54aa62555b/html5/thumbnails/84.jpg)
DEMO - CNN MNIST
http://cs.stanford.edu/people/karpathy/convnetjs/demo/mnist.html
![Page 85: APPLIED DEEP LEARNING - IARIAiaria.org/.../DeepLearning_Tutorial_2016.pdf · Complex Deep Nets - only practical choice. DEEP NETS BETTER THAN HUMANS Humans were 5.10% wrong ... SAUCE”](https://reader030.fdocuments.in/reader030/viewer/2022041019/5ecd5c121733ab54aa62555b/html5/thumbnails/85.jpg)
DEMO - CNN
http://cs.stanford.edu/people/karpathy/convnetjs/demo/cifar10.html
![Page 86: APPLIED DEEP LEARNING - IARIAiaria.org/.../DeepLearning_Tutorial_2016.pdf · Complex Deep Nets - only practical choice. DEEP NETS BETTER THAN HUMANS Humans were 5.10% wrong ... SAUCE”](https://reader030.fdocuments.in/reader030/viewer/2022041019/5ecd5c121733ab54aa62555b/html5/thumbnails/86.jpg)
CONCLUSION
USES
Predictstockmarkets
PredictcustomerChurn
Performsentimentanalysis
Decodeancienttextsautomatically
AutonomousVehicles