Fundamentals of Deep (Artificial) Neural Networks (DNN) · Recurrent Neural Networks Motivation...
Transcript of Fundamentals of Deep (Artificial) Neural Networks (DNN) · Recurrent Neural Networks Motivation...
![Page 1: Fundamentals of Deep (Artificial) Neural Networks (DNN) · Recurrent Neural Networks Motivation Feed forward networks accept a fixed-sized vector as input and produce a fixed-sized](https://reader030.fdocuments.in/reader030/viewer/2022040609/5ecd24e35f998d2c8838dd80/html5/thumbnails/1.jpg)
Fundamentals of Deep (Artificial) Neural Networks (DNN)Greg Tsagkatakis
CSD - UOC
ICS - FORTH
![Page 2: Fundamentals of Deep (Artificial) Neural Networks (DNN) · Recurrent Neural Networks Motivation Feed forward networks accept a fixed-sized vector as input and produce a fixed-sized](https://reader030.fdocuments.in/reader030/viewer/2022040609/5ecd24e35f998d2c8838dd80/html5/thumbnails/2.jpg)
Accelerated growth
2
![Page 3: Fundamentals of Deep (Artificial) Neural Networks (DNN) · Recurrent Neural Networks Motivation Feed forward networks accept a fixed-sized vector as input and produce a fixed-sized](https://reader030.fdocuments.in/reader030/viewer/2022040609/5ecd24e35f998d2c8838dd80/html5/thumbnails/3.jpg)
Brief history of DL
3
![Page 4: Fundamentals of Deep (Artificial) Neural Networks (DNN) · Recurrent Neural Networks Motivation Feed forward networks accept a fixed-sized vector as input and produce a fixed-sized](https://reader030.fdocuments.in/reader030/viewer/2022040609/5ecd24e35f998d2c8838dd80/html5/thumbnails/4.jpg)
Why Today?Lots of Data
4
![Page 5: Fundamentals of Deep (Artificial) Neural Networks (DNN) · Recurrent Neural Networks Motivation Feed forward networks accept a fixed-sized vector as input and produce a fixed-sized](https://reader030.fdocuments.in/reader030/viewer/2022040609/5ecd24e35f998d2c8838dd80/html5/thumbnails/5.jpg)
Why Today?Lots of Data
Deeper Learning
5
![Page 6: Fundamentals of Deep (Artificial) Neural Networks (DNN) · Recurrent Neural Networks Motivation Feed forward networks accept a fixed-sized vector as input and produce a fixed-sized](https://reader030.fdocuments.in/reader030/viewer/2022040609/5ecd24e35f998d2c8838dd80/html5/thumbnails/6.jpg)
Why Today?Lots of Data
Deep Learning
More Power
https://blogs.nvidia.com/blog/2016/01/12/accelerating-ai-artificial-intelligence-gpus/https://www.slothparadise.com/what-is-cloud-computing/
6
![Page 7: Fundamentals of Deep (Artificial) Neural Networks (DNN) · Recurrent Neural Networks Motivation Feed forward networks accept a fixed-sized vector as input and produce a fixed-sized](https://reader030.fdocuments.in/reader030/viewer/2022040609/5ecd24e35f998d2c8838dd80/html5/thumbnails/7.jpg)
Apps: Gaming
7
![Page 8: Fundamentals of Deep (Artificial) Neural Networks (DNN) · Recurrent Neural Networks Motivation Feed forward networks accept a fixed-sized vector as input and produce a fixed-sized](https://reader030.fdocuments.in/reader030/viewer/2022040609/5ecd24e35f998d2c8838dd80/html5/thumbnails/8.jpg)
Key components of ANN Architecture (input/hidden/output layers)
8
![Page 9: Fundamentals of Deep (Artificial) Neural Networks (DNN) · Recurrent Neural Networks Motivation Feed forward networks accept a fixed-sized vector as input and produce a fixed-sized](https://reader030.fdocuments.in/reader030/viewer/2022040609/5ecd24e35f998d2c8838dd80/html5/thumbnails/9.jpg)
Key components of ANN Architecture (input/hidden/output layers)
Weights
9
![Page 10: Fundamentals of Deep (Artificial) Neural Networks (DNN) · Recurrent Neural Networks Motivation Feed forward networks accept a fixed-sized vector as input and produce a fixed-sized](https://reader030.fdocuments.in/reader030/viewer/2022040609/5ecd24e35f998d2c8838dd80/html5/thumbnails/10.jpg)
Key components of ANN Architecture (input/hidden/output layers)
Weights
Activations
10
LINEAR RECTIFIED
LINEAR (ReLU)
LOGISTIC /
SIGMOIDAL / TANH
![Page 11: Fundamentals of Deep (Artificial) Neural Networks (DNN) · Recurrent Neural Networks Motivation Feed forward networks accept a fixed-sized vector as input and produce a fixed-sized](https://reader030.fdocuments.in/reader030/viewer/2022040609/5ecd24e35f998d2c8838dd80/html5/thumbnails/11.jpg)
Perceptron: an early attempt
Activation function
Need to tune and
σ
x1
x2
…
b
w1
w2
1
11
![Page 12: Fundamentals of Deep (Artificial) Neural Networks (DNN) · Recurrent Neural Networks Motivation Feed forward networks accept a fixed-sized vector as input and produce a fixed-sized](https://reader030.fdocuments.in/reader030/viewer/2022040609/5ecd24e35f998d2c8838dd80/html5/thumbnails/12.jpg)
Multilayer perceptron
w1
w2
w3
A
B
C
D
E
A neuron is of the form
σ(w.x + b) where σ is
an activation function
We just added a neuron layer!
We just introduced non-linearity!
w1A
w2B
w1D
wAE
wDE
12
![Page 14: Fundamentals of Deep (Artificial) Neural Networks (DNN) · Recurrent Neural Networks Motivation Feed forward networks accept a fixed-sized vector as input and produce a fixed-sized](https://reader030.fdocuments.in/reader030/viewer/2022040609/5ecd24e35f998d2c8838dd80/html5/thumbnails/14.jpg)
Training & TestingTraining: determine weights◦ Supervised: labeled training examples
◦ Unsupervised: no labels available
◦ Reinforcement: examples associated with rewards
Testing (Inference): apply weights to new examples
14
![Page 15: Fundamentals of Deep (Artificial) Neural Networks (DNN) · Recurrent Neural Networks Motivation Feed forward networks accept a fixed-sized vector as input and produce a fixed-sized](https://reader030.fdocuments.in/reader030/viewer/2022040609/5ecd24e35f998d2c8838dd80/html5/thumbnails/15.jpg)
Training DNN1. Get batch of data
2. Forward through the network -> estimate loss
3. Backpropagate error
4. Update weights based on gradient
Errors
15
![Page 16: Fundamentals of Deep (Artificial) Neural Networks (DNN) · Recurrent Neural Networks Motivation Feed forward networks accept a fixed-sized vector as input and produce a fixed-sized](https://reader030.fdocuments.in/reader030/viewer/2022040609/5ecd24e35f998d2c8838dd80/html5/thumbnails/16.jpg)
BackPropagationChain Rule in Gradient Descent: Invented in 1969 by Bryson and Ho
Defining a loss/cost function
Assume a function
Types of Loss function
•Hinge
•Exponential
•Logistic
16
![Page 17: Fundamentals of Deep (Artificial) Neural Networks (DNN) · Recurrent Neural Networks Motivation Feed forward networks accept a fixed-sized vector as input and produce a fixed-sized](https://reader030.fdocuments.in/reader030/viewer/2022040609/5ecd24e35f998d2c8838dd80/html5/thumbnails/17.jpg)
Gradient DescentMinimize function J w.r.t. parameters θ
Gradient
Chain rule
17
New weights Gradient
Old weights Learning rate
![Page 18: Fundamentals of Deep (Artificial) Neural Networks (DNN) · Recurrent Neural Networks Motivation Feed forward networks accept a fixed-sized vector as input and produce a fixed-sized](https://reader030.fdocuments.in/reader030/viewer/2022040609/5ecd24e35f998d2c8838dd80/html5/thumbnails/18.jpg)
BackProp
18
Chain Rule:
Given:
…
![Page 19: Fundamentals of Deep (Artificial) Neural Networks (DNN) · Recurrent Neural Networks Motivation Feed forward networks accept a fixed-sized vector as input and produce a fixed-sized](https://reader030.fdocuments.in/reader030/viewer/2022040609/5ecd24e35f998d2c8838dd80/html5/thumbnails/19.jpg)
BackProp
Chain rule:
◦ Single variable
◦ Multiple variables
19
g fxy=g(x)
z=f(y)=f(g(x))
![Page 20: Fundamentals of Deep (Artificial) Neural Networks (DNN) · Recurrent Neural Networks Motivation Feed forward networks accept a fixed-sized vector as input and produce a fixed-sized](https://reader030.fdocuments.in/reader030/viewer/2022040609/5ecd24e35f998d2c8838dd80/html5/thumbnails/20.jpg)
20
![Page 21: Fundamentals of Deep (Artificial) Neural Networks (DNN) · Recurrent Neural Networks Motivation Feed forward networks accept a fixed-sized vector as input and produce a fixed-sized](https://reader030.fdocuments.in/reader030/viewer/2022040609/5ecd24e35f998d2c8838dd80/html5/thumbnails/21.jpg)
21
![Page 22: Fundamentals of Deep (Artificial) Neural Networks (DNN) · Recurrent Neural Networks Motivation Feed forward networks accept a fixed-sized vector as input and produce a fixed-sized](https://reader030.fdocuments.in/reader030/viewer/2022040609/5ecd24e35f998d2c8838dd80/html5/thumbnails/22.jpg)
22
![Page 23: Fundamentals of Deep (Artificial) Neural Networks (DNN) · Recurrent Neural Networks Motivation Feed forward networks accept a fixed-sized vector as input and produce a fixed-sized](https://reader030.fdocuments.in/reader030/viewer/2022040609/5ecd24e35f998d2c8838dd80/html5/thumbnails/23.jpg)
23
![Page 24: Fundamentals of Deep (Artificial) Neural Networks (DNN) · Recurrent Neural Networks Motivation Feed forward networks accept a fixed-sized vector as input and produce a fixed-sized](https://reader030.fdocuments.in/reader030/viewer/2022040609/5ecd24e35f998d2c8838dd80/html5/thumbnails/24.jpg)
24
![Page 25: Fundamentals of Deep (Artificial) Neural Networks (DNN) · Recurrent Neural Networks Motivation Feed forward networks accept a fixed-sized vector as input and produce a fixed-sized](https://reader030.fdocuments.in/reader030/viewer/2022040609/5ecd24e35f998d2c8838dd80/html5/thumbnails/25.jpg)
25
![Page 26: Fundamentals of Deep (Artificial) Neural Networks (DNN) · Recurrent Neural Networks Motivation Feed forward networks accept a fixed-sized vector as input and produce a fixed-sized](https://reader030.fdocuments.in/reader030/viewer/2022040609/5ecd24e35f998d2c8838dd80/html5/thumbnails/26.jpg)
26
![Page 27: Fundamentals of Deep (Artificial) Neural Networks (DNN) · Recurrent Neural Networks Motivation Feed forward networks accept a fixed-sized vector as input and produce a fixed-sized](https://reader030.fdocuments.in/reader030/viewer/2022040609/5ecd24e35f998d2c8838dd80/html5/thumbnails/27.jpg)
Visualization
27
![Page 28: Fundamentals of Deep (Artificial) Neural Networks (DNN) · Recurrent Neural Networks Motivation Feed forward networks accept a fixed-sized vector as input and produce a fixed-sized](https://reader030.fdocuments.in/reader030/viewer/2022040609/5ecd24e35f998d2c8838dd80/html5/thumbnails/28.jpg)
Training Characteristics
28
Under-fitting
Over-fitting
![Page 29: Fundamentals of Deep (Artificial) Neural Networks (DNN) · Recurrent Neural Networks Motivation Feed forward networks accept a fixed-sized vector as input and produce a fixed-sized](https://reader030.fdocuments.in/reader030/viewer/2022040609/5ecd24e35f998d2c8838dd80/html5/thumbnails/29.jpg)
Supervised Learning
29
![Page 30: Fundamentals of Deep (Artificial) Neural Networks (DNN) · Recurrent Neural Networks Motivation Feed forward networks accept a fixed-sized vector as input and produce a fixed-sized](https://reader030.fdocuments.in/reader030/viewer/2022040609/5ecd24e35f998d2c8838dd80/html5/thumbnails/30.jpg)
Supervised Learning
Exploiting prior knowledge
Expert users
Crowdsourcing
Other instruments
Spiral
Elliptical
?
30
ModelPrediction
Data Labels
![Page 31: Fundamentals of Deep (Artificial) Neural Networks (DNN) · Recurrent Neural Networks Motivation Feed forward networks accept a fixed-sized vector as input and produce a fixed-sized](https://reader030.fdocuments.in/reader030/viewer/2022040609/5ecd24e35f998d2c8838dd80/html5/thumbnails/31.jpg)
State-of-the-art (before Deep Learning)Support Vector Machines Binary classification
31
![Page 32: Fundamentals of Deep (Artificial) Neural Networks (DNN) · Recurrent Neural Networks Motivation Feed forward networks accept a fixed-sized vector as input and produce a fixed-sized](https://reader030.fdocuments.in/reader030/viewer/2022040609/5ecd24e35f998d2c8838dd80/html5/thumbnails/32.jpg)
State-of-the-art (before Deep Learning)Support Vector Machines Binary classification
Kernels <-> non-linearities
32
![Page 33: Fundamentals of Deep (Artificial) Neural Networks (DNN) · Recurrent Neural Networks Motivation Feed forward networks accept a fixed-sized vector as input and produce a fixed-sized](https://reader030.fdocuments.in/reader030/viewer/2022040609/5ecd24e35f998d2c8838dd80/html5/thumbnails/33.jpg)
State-of-the-art (before Deep Learning)Support Vector Machines Binary classification
Kernels <-> non-linearities
Random Forests Multi-class classification
33
![Page 34: Fundamentals of Deep (Artificial) Neural Networks (DNN) · Recurrent Neural Networks Motivation Feed forward networks accept a fixed-sized vector as input and produce a fixed-sized](https://reader030.fdocuments.in/reader030/viewer/2022040609/5ecd24e35f998d2c8838dd80/html5/thumbnails/34.jpg)
State-of-the-art (before Deep Learning)Support Vector Machines Binary classification
Kernels <-> non-linearities
Random Forests Multi-class classification
Markov Chains/Fields Temporal data
34
![Page 35: Fundamentals of Deep (Artificial) Neural Networks (DNN) · Recurrent Neural Networks Motivation Feed forward networks accept a fixed-sized vector as input and produce a fixed-sized](https://reader030.fdocuments.in/reader030/viewer/2022040609/5ecd24e35f998d2c8838dd80/html5/thumbnails/35.jpg)
State-of-the-art (since 2015)Deep Learning (DL)
Convolutional Neural Networks (CNN) <-> Images
Recurrent Neural Networks (RNN) <-> Audio
35
![Page 36: Fundamentals of Deep (Artificial) Neural Networks (DNN) · Recurrent Neural Networks Motivation Feed forward networks accept a fixed-sized vector as input and produce a fixed-sized](https://reader030.fdocuments.in/reader030/viewer/2022040609/5ecd24e35f998d2c8838dd80/html5/thumbnails/36.jpg)
Convolutional Neural Networks
(Convolution + Subsampling) + () … + Fully Connected
36
![Page 37: Fundamentals of Deep (Artificial) Neural Networks (DNN) · Recurrent Neural Networks Motivation Feed forward networks accept a fixed-sized vector as input and produce a fixed-sized](https://reader030.fdocuments.in/reader030/viewer/2022040609/5ecd24e35f998d2c8838dd80/html5/thumbnails/37.jpg)
Convolutional Layers
channels
hei
ght
32x32x1 Image
5x5x1 filter
hei
ght
28x28xK activation map
K filters
37
![Page 38: Fundamentals of Deep (Artificial) Neural Networks (DNN) · Recurrent Neural Networks Motivation Feed forward networks accept a fixed-sized vector as input and produce a fixed-sized](https://reader030.fdocuments.in/reader030/viewer/2022040609/5ecd24e35f998d2c8838dd80/html5/thumbnails/38.jpg)
Convolutional LayersCharacteristics
Hierarchical features
Location invariance
Parameters
Number of filters (32,64…)
Filter size (3x3, 5x5)
Stride (1)
Padding (2,4)
“Machine Learning and AI for Brain Simulations” –Andrew Ng Talk, UCLA, 2012
38
![Page 39: Fundamentals of Deep (Artificial) Neural Networks (DNN) · Recurrent Neural Networks Motivation Feed forward networks accept a fixed-sized vector as input and produce a fixed-sized](https://reader030.fdocuments.in/reader030/viewer/2022040609/5ecd24e35f998d2c8838dd80/html5/thumbnails/39.jpg)
Subsampling (pooling) Layers
39
<-> downsampling
Scale invariance
Parameters
• Type
• Filter Size
• Stride
![Page 40: Fundamentals of Deep (Artificial) Neural Networks (DNN) · Recurrent Neural Networks Motivation Feed forward networks accept a fixed-sized vector as input and produce a fixed-sized](https://reader030.fdocuments.in/reader030/viewer/2022040609/5ecd24e35f998d2c8838dd80/html5/thumbnails/40.jpg)
Activation LayerIntroduction of non-linearity
◦ Brain: thresholding -> spike trains
40
![Page 41: Fundamentals of Deep (Artificial) Neural Networks (DNN) · Recurrent Neural Networks Motivation Feed forward networks accept a fixed-sized vector as input and produce a fixed-sized](https://reader030.fdocuments.in/reader030/viewer/2022040609/5ecd24e35f998d2c8838dd80/html5/thumbnails/41.jpg)
Activation LayerReLU: x=max(0,x)
Simplifies backprop
Makes learning faster
Avoids saturation issues
~ non-negativity constraint
(Note: The brain)
No saturated gradients
41
![Page 42: Fundamentals of Deep (Artificial) Neural Networks (DNN) · Recurrent Neural Networks Motivation Feed forward networks accept a fixed-sized vector as input and produce a fixed-sized](https://reader030.fdocuments.in/reader030/viewer/2022040609/5ecd24e35f998d2c8838dd80/html5/thumbnails/42.jpg)
Fully Connected LayersFull connections to all activations in previous layer
Typically at the end
Can be replaced by conv
42
Feat
ure
s
Cla
sses
![Page 43: Fundamentals of Deep (Artificial) Neural Networks (DNN) · Recurrent Neural Networks Motivation Feed forward networks accept a fixed-sized vector as input and produce a fixed-sized](https://reader030.fdocuments.in/reader030/viewer/2022040609/5ecd24e35f998d2c8838dd80/html5/thumbnails/43.jpg)
LeNet [1998]
43
![Page 44: Fundamentals of Deep (Artificial) Neural Networks (DNN) · Recurrent Neural Networks Motivation Feed forward networks accept a fixed-sized vector as input and produce a fixed-sized](https://reader030.fdocuments.in/reader030/viewer/2022040609/5ecd24e35f998d2c8838dd80/html5/thumbnails/44.jpg)
AlexNet [2012]
Alex Krizhevsky, Ilya Sutskever and Geoff Hinton, ImageNet ILSVRC challenge in 2012http://vision03.csail.mit.edu/cnn_art/data/single_layer.png
44
![Page 45: Fundamentals of Deep (Artificial) Neural Networks (DNN) · Recurrent Neural Networks Motivation Feed forward networks accept a fixed-sized vector as input and produce a fixed-sized](https://reader030.fdocuments.in/reader030/viewer/2022040609/5ecd24e35f998d2c8838dd80/html5/thumbnails/45.jpg)
K. Simonyan, A. Zisserman Very Deep Convolutional Networks for Large-Scale Image Recognition, arXiv technical report, 2014
VGGnet [2014]
45
![Page 46: Fundamentals of Deep (Artificial) Neural Networks (DNN) · Recurrent Neural Networks Motivation Feed forward networks accept a fixed-sized vector as input and produce a fixed-sized](https://reader030.fdocuments.in/reader030/viewer/2022040609/5ecd24e35f998d2c8838dd80/html5/thumbnails/46.jpg)
VGGnet
D: VGG16E: VGG19All filters are 3x3
More layers smaller filters
46
![Page 47: Fundamentals of Deep (Artificial) Neural Networks (DNN) · Recurrent Neural Networks Motivation Feed forward networks accept a fixed-sized vector as input and produce a fixed-sized](https://reader030.fdocuments.in/reader030/viewer/2022040609/5ecd24e35f998d2c8838dd80/html5/thumbnails/47.jpg)
Inception (GoogLeNet, 2014)
Inception moduleInception module with dimensionality reduction
47
![Page 48: Fundamentals of Deep (Artificial) Neural Networks (DNN) · Recurrent Neural Networks Motivation Feed forward networks accept a fixed-sized vector as input and produce a fixed-sized](https://reader030.fdocuments.in/reader030/viewer/2022040609/5ecd24e35f998d2c8838dd80/html5/thumbnails/48.jpg)
Residuals
48
![Page 49: Fundamentals of Deep (Artificial) Neural Networks (DNN) · Recurrent Neural Networks Motivation Feed forward networks accept a fixed-sized vector as input and produce a fixed-sized](https://reader030.fdocuments.in/reader030/viewer/2022040609/5ecd24e35f998d2c8838dd80/html5/thumbnails/49.jpg)
ResNet, 2015
49
He, Kaiming, et al. "Deep residual learning for image recognition." IEEE CVPR. 2016.
![Page 50: Fundamentals of Deep (Artificial) Neural Networks (DNN) · Recurrent Neural Networks Motivation Feed forward networks accept a fixed-sized vector as input and produce a fixed-sized](https://reader030.fdocuments.in/reader030/viewer/2022040609/5ecd24e35f998d2c8838dd80/html5/thumbnails/50.jpg)
Training protocolsFully Supervised• Random initialization of weights
• Train in supervised mode (example + label)
Unsupervised pre-training + standard classifier• Train each layer unsupervised
• Train a supervised classifier (SVM) on top
Unsupervised pre-training + supervised fine-tuning• Train each layer unsupervised
• Add a supervised layer
50
![Page 51: Fundamentals of Deep (Artificial) Neural Networks (DNN) · Recurrent Neural Networks Motivation Feed forward networks accept a fixed-sized vector as input and produce a fixed-sized](https://reader030.fdocuments.in/reader030/viewer/2022040609/5ecd24e35f998d2c8838dd80/html5/thumbnails/51.jpg)
Dropout
51
Srivastava, Nitish, et al. "Dropout: a simple way to prevent neural networks from overfitting." Journal of machine learning research15.1 (2014): 1929-1958.
![Page 52: Fundamentals of Deep (Artificial) Neural Networks (DNN) · Recurrent Neural Networks Motivation Feed forward networks accept a fixed-sized vector as input and produce a fixed-sized](https://reader030.fdocuments.in/reader030/viewer/2022040609/5ecd24e35f998d2c8838dd80/html5/thumbnails/52.jpg)
Batch Normalization
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift [Ioffe and Szegedy 2015] 52
![Page 53: Fundamentals of Deep (Artificial) Neural Networks (DNN) · Recurrent Neural Networks Motivation Feed forward networks accept a fixed-sized vector as input and produce a fixed-sized](https://reader030.fdocuments.in/reader030/viewer/2022040609/5ecd24e35f998d2c8838dd80/html5/thumbnails/53.jpg)
Transfer LearningLayer 1 Layer 2 Layer L
1x
2x
……
……
……
……
……
elephant
……Nx
Pixels
……
……
……
![Page 54: Fundamentals of Deep (Artificial) Neural Networks (DNN) · Recurrent Neural Networks Motivation Feed forward networks accept a fixed-sized vector as input and produce a fixed-sized](https://reader030.fdocuments.in/reader030/viewer/2022040609/5ecd24e35f998d2c8838dd80/html5/thumbnails/54.jpg)
Transfer Learning
54
Layer 1 Layer 2 Layer L
1x
2x
……
……
……
……
……
elephant
……Nx
Pixels
……
……
……
Layer 1 Layer 2 Layer L
1x
2x…
…
……
……
……
Healthy
Malignancy
Nx
Pixels
……
……
……
![Page 55: Fundamentals of Deep (Artificial) Neural Networks (DNN) · Recurrent Neural Networks Motivation Feed forward networks accept a fixed-sized vector as input and produce a fixed-sized](https://reader030.fdocuments.in/reader030/viewer/2022040609/5ecd24e35f998d2c8838dd80/html5/thumbnails/55.jpg)
Layer Transfer - Image
Jason Yosinski, Jeff Clune, Yoshua Bengio, Hod Lipson, “How transferable are features in deep neural networks?”, NIPS, 2014
Only train the rest layers
fine-tune the whole network
Source: 500 classes from ImageNet
Target: another 500 classes from ImageNet
![Page 56: Fundamentals of Deep (Artificial) Neural Networks (DNN) · Recurrent Neural Networks Motivation Feed forward networks accept a fixed-sized vector as input and produce a fixed-sized](https://reader030.fdocuments.in/reader030/viewer/2022040609/5ecd24e35f998d2c8838dd80/html5/thumbnails/56.jpg)
ImageNET
Validation classification
Validation classification
Validation classification
• ~14 million labeled images, 20k classes
• Images gathered from Internet
• Human labels via Amazon MTurk
• ImageNet Large-Scale Visual Recognition Challenge (ILSVRC): 1.2 million training images, 1000 classes
www.image-net.org/challenges/LSVRC/
56
![Page 57: Fundamentals of Deep (Artificial) Neural Networks (DNN) · Recurrent Neural Networks Motivation Feed forward networks accept a fixed-sized vector as input and produce a fixed-sized](https://reader030.fdocuments.in/reader030/viewer/2022040609/5ecd24e35f998d2c8838dd80/html5/thumbnails/57.jpg)
Summary: ILSVRC 2012-2015
Team Year Place Error (top-5) External data
(AlexNet, 7 layers) 2012 - 16.4% no
SuperVision 2012 1st 15.3% ImageNet 22k
Clarifai – NYU (7 layers) 2013 - 11.7% no
Clarifai 2013 1st 11.2% ImageNet 22k
VGG – Oxford (16 layers) 2014 2nd 7.32% no
GoogLeNet (19 layers) 2014 1st 6.67% no
ResNet (152 layers) 2015 1st 3.57%
Human expert* 5.1%
http://karpathy.github.io/2014/09/02/what-i-learned-from-competing-against-a-convnet-on-imagenet/
57
![Page 58: Fundamentals of Deep (Artificial) Neural Networks (DNN) · Recurrent Neural Networks Motivation Feed forward networks accept a fixed-sized vector as input and produce a fixed-sized](https://reader030.fdocuments.in/reader030/viewer/2022040609/5ecd24e35f998d2c8838dd80/html5/thumbnails/58.jpg)
Esteva, Andre, et al. "Dermatologist-level classification of skin cancer with deep neural networks." Nature 542.7639 (2017): 115-118.
Skin cancer detection
58
![Page 59: Fundamentals of Deep (Artificial) Neural Networks (DNN) · Recurrent Neural Networks Motivation Feed forward networks accept a fixed-sized vector as input and produce a fixed-sized](https://reader030.fdocuments.in/reader030/viewer/2022040609/5ecd24e35f998d2c8838dd80/html5/thumbnails/59.jpg)
CNN & FMRI
59
![Page 60: Fundamentals of Deep (Artificial) Neural Networks (DNN) · Recurrent Neural Networks Motivation Feed forward networks accept a fixed-sized vector as input and produce a fixed-sized](https://reader030.fdocuments.in/reader030/viewer/2022040609/5ecd24e35f998d2c8838dd80/html5/thumbnails/60.jpg)
Different types of mapping
Image classification
Image captioning
Sentiment analysis
Machine translation
Synced sequence(video classification)
60
![Page 61: Fundamentals of Deep (Artificial) Neural Networks (DNN) · Recurrent Neural Networks Motivation Feed forward networks accept a fixed-sized vector as input and produce a fixed-sized](https://reader030.fdocuments.in/reader030/viewer/2022040609/5ecd24e35f998d2c8838dd80/html5/thumbnails/61.jpg)
Recurrent Neural NetworksMotivation
Feed forward networks accept a fixed-sized vector as input and produce a fixed-sized vector as output
fixed amount of computational steps
recurrent nets allow us to operate over sequences of vectors
Use cases
Video
Audio
Text
61
![Page 62: Fundamentals of Deep (Artificial) Neural Networks (DNN) · Recurrent Neural Networks Motivation Feed forward networks accept a fixed-sized vector as input and produce a fixed-sized](https://reader030.fdocuments.in/reader030/viewer/2022040609/5ecd24e35f998d2c8838dd80/html5/thumbnails/62.jpg)
RNN Architecture
Output
DelayHidden Units
Inputs
𝑥(𝑡)
𝑠(𝑡)
𝑠(𝑡 − 1)
o(𝑡)
𝑤𝑒𝑖𝑔ℎ𝑡𝑠 𝑈
𝑤𝑒𝑖𝑔ℎ𝑡𝑠 𝑊
𝑤𝑒𝑖𝑔ℎ𝑡𝑠 𝑉
![Page 63: Fundamentals of Deep (Artificial) Neural Networks (DNN) · Recurrent Neural Networks Motivation Feed forward networks accept a fixed-sized vector as input and produce a fixed-sized](https://reader030.fdocuments.in/reader030/viewer/2022040609/5ecd24e35f998d2c8838dd80/html5/thumbnails/63.jpg)
Unfolding RNNsEach node represents a layer of network units at a single time step.
The same weights are reused at every time step.
63
![Page 64: Fundamentals of Deep (Artificial) Neural Networks (DNN) · Recurrent Neural Networks Motivation Feed forward networks accept a fixed-sized vector as input and produce a fixed-sized](https://reader030.fdocuments.in/reader030/viewer/2022040609/5ecd24e35f998d2c8838dd80/html5/thumbnails/64.jpg)
Unsupervised Learning
64
![Page 65: Fundamentals of Deep (Artificial) Neural Networks (DNN) · Recurrent Neural Networks Motivation Feed forward networks accept a fixed-sized vector as input and produce a fixed-sized](https://reader030.fdocuments.in/reader030/viewer/2022040609/5ecd24e35f998d2c8838dd80/html5/thumbnails/65.jpg)
AgendaAutoencoders
Sparse coding
Generative Adversarial Networks
65
![Page 66: Fundamentals of Deep (Artificial) Neural Networks (DNN) · Recurrent Neural Networks Motivation Feed forward networks accept a fixed-sized vector as input and produce a fixed-sized](https://reader030.fdocuments.in/reader030/viewer/2022040609/5ecd24e35f998d2c8838dd80/html5/thumbnails/66.jpg)
AutoencodersUnsupervised feature learning
Network is trained to output the input (learn identify function).
Encoder
Decoder
66
x4
x5
x6
+1
Layer 1
Layer 2
x1
x2
x3
x4
x5
x6
x1
x2
x3
+1
Layer 3
![Page 67: Fundamentals of Deep (Artificial) Neural Networks (DNN) · Recurrent Neural Networks Motivation Feed forward networks accept a fixed-sized vector as input and produce a fixed-sized](https://reader030.fdocuments.in/reader030/viewer/2022040609/5ecd24e35f998d2c8838dd80/html5/thumbnails/67.jpg)
Sparse neuron activation
Contractive auto-encoders
Denoising auto-encoders
Convolutional AE
Regularized Autoencoders
67
![Page 68: Fundamentals of Deep (Artificial) Neural Networks (DNN) · Recurrent Neural Networks Motivation Feed forward networks accept a fixed-sized vector as input and produce a fixed-sized](https://reader030.fdocuments.in/reader030/viewer/2022040609/5ecd24e35f998d2c8838dd80/html5/thumbnails/68.jpg)
Stacked Autoencoders
Extended AE with multiple layers of hidden units
Challenges of Backpropagation
Efficient training◦ Normalization of input
Unsupervised pre-training◦ Greedy layer-wise training
◦ Fine-tune w.r.t criterion
68
Bengio, Learning deep architectures for AI, Foundations and Trends in Machine Learning ,2009
![Page 69: Fundamentals of Deep (Artificial) Neural Networks (DNN) · Recurrent Neural Networks Motivation Feed forward networks accept a fixed-sized vector as input and produce a fixed-sized](https://reader030.fdocuments.in/reader030/viewer/2022040609/5ecd24e35f998d2c8838dd80/html5/thumbnails/69.jpg)
x4
x5
x6
+1
Layer 1
Layer 2
x1
x2
x3
x4
x5
x6
x1
x2
x3
+1
Layer 3
a1
a2
a3
69
![Page 70: Fundamentals of Deep (Artificial) Neural Networks (DNN) · Recurrent Neural Networks Motivation Feed forward networks accept a fixed-sized vector as input and produce a fixed-sized](https://reader030.fdocuments.in/reader030/viewer/2022040609/5ecd24e35f998d2c8838dd80/html5/thumbnails/70.jpg)
x4
x5
x6
+1
Layer 1
Layer 2
x1
x2
x3
+1
a1
a2
a3
New representation for input.
70
![Page 71: Fundamentals of Deep (Artificial) Neural Networks (DNN) · Recurrent Neural Networks Motivation Feed forward networks accept a fixed-sized vector as input and produce a fixed-sized](https://reader030.fdocuments.in/reader030/viewer/2022040609/5ecd24e35f998d2c8838dd80/html5/thumbnails/71.jpg)
x4
x5
x6
+1
Layer 1
Layer 2
x1
x2
x3
+1
a1
a2
a3
71
![Page 72: Fundamentals of Deep (Artificial) Neural Networks (DNN) · Recurrent Neural Networks Motivation Feed forward networks accept a fixed-sized vector as input and produce a fixed-sized](https://reader030.fdocuments.in/reader030/viewer/2022040609/5ecd24e35f998d2c8838dd80/html5/thumbnails/72.jpg)
x4
x5
x6
+1
x1
x2
x3
+1
a1
a2
a3
+1
b1
b2
b3
Train parameters so that ,
subject to bi’s being sparse.
72
![Page 73: Fundamentals of Deep (Artificial) Neural Networks (DNN) · Recurrent Neural Networks Motivation Feed forward networks accept a fixed-sized vector as input and produce a fixed-sized](https://reader030.fdocuments.in/reader030/viewer/2022040609/5ecd24e35f998d2c8838dd80/html5/thumbnails/73.jpg)
x4
x5
x6
+1
x1
x2
x3
+1
a1
a2
a3
+1
b1
b2
b3
Train parameters so that ,
subject to bi’s being sparse.
73
![Page 74: Fundamentals of Deep (Artificial) Neural Networks (DNN) · Recurrent Neural Networks Motivation Feed forward networks accept a fixed-sized vector as input and produce a fixed-sized](https://reader030.fdocuments.in/reader030/viewer/2022040609/5ecd24e35f998d2c8838dd80/html5/thumbnails/74.jpg)
x4
x5
x6
+1
x1
x2
x3
+1
a1
a2
a3
+1
b1
b2
b3
Train parameters so that ,
subject to bi’s being sparse. 74
![Page 75: Fundamentals of Deep (Artificial) Neural Networks (DNN) · Recurrent Neural Networks Motivation Feed forward networks accept a fixed-sized vector as input and produce a fixed-sized](https://reader030.fdocuments.in/reader030/viewer/2022040609/5ecd24e35f998d2c8838dd80/html5/thumbnails/75.jpg)
x4
x5
x6
+1
x1
x2
x3
+1
a1
a2
a3
+1
b1
b2
b3
New representation for input.
75
![Page 76: Fundamentals of Deep (Artificial) Neural Networks (DNN) · Recurrent Neural Networks Motivation Feed forward networks accept a fixed-sized vector as input and produce a fixed-sized](https://reader030.fdocuments.in/reader030/viewer/2022040609/5ecd24e35f998d2c8838dd80/html5/thumbnails/76.jpg)
x4
x5
x6
+1
x1
x2
x3
+1
a1
a2
a3
+1
b1
b2
b3
76
![Page 77: Fundamentals of Deep (Artificial) Neural Networks (DNN) · Recurrent Neural Networks Motivation Feed forward networks accept a fixed-sized vector as input and produce a fixed-sized](https://reader030.fdocuments.in/reader030/viewer/2022040609/5ecd24e35f998d2c8838dd80/html5/thumbnails/77.jpg)
x4
x5
x6
+1
x1
x2
x3
+1
a1
a2
a3
+1
b1
b2
b3
+1
c1
c2
c3
77
![Page 78: Fundamentals of Deep (Artificial) Neural Networks (DNN) · Recurrent Neural Networks Motivation Feed forward networks accept a fixed-sized vector as input and produce a fixed-sized](https://reader030.fdocuments.in/reader030/viewer/2022040609/5ecd24e35f998d2c8838dd80/html5/thumbnails/78.jpg)
x4
x5
x6
+1
x1
x2
x3
+1
a1
a2
a3
+1
b1
b2
b3
+1
c1
c2
c3
New representation
for input.
Use [c1, c3, c3] as representation to feed
to learning algorithm.
78
![Page 79: Fundamentals of Deep (Artificial) Neural Networks (DNN) · Recurrent Neural Networks Motivation Feed forward networks accept a fixed-sized vector as input and produce a fixed-sized](https://reader030.fdocuments.in/reader030/viewer/2022040609/5ecd24e35f998d2c8838dd80/html5/thumbnails/79.jpg)
TensorFlowDeep learning library, open-sourced by Google (11/2015)
TensorFlow provides primitives for ◦ defining functions on tensors
◦ automatically computing their derivatives
What is a tensor
What is a computational graph
79
Material from lecture by Bharath Ramsundar, March 2018, Stanford
![Page 80: Fundamentals of Deep (Artificial) Neural Networks (DNN) · Recurrent Neural Networks Motivation Feed forward networks accept a fixed-sized vector as input and produce a fixed-sized](https://reader030.fdocuments.in/reader030/viewer/2022040609/5ecd24e35f998d2c8838dd80/html5/thumbnails/80.jpg)
Introduction to KerasOfficial high-level API of TensorFlow
◦ Python
◦ 250K developers
Same front-end <-> Different back-ends◦ TensorFlow (Google)
◦ CNTK (Microsoft)
◦ MXNet (Apache)
◦ Theano (RIP)
Hardware◦ GPU (Nvidia)
◦ CPU (Intel/AMD)
◦ TPU (Google)
Companies: Netflix, Uber, Google, Nvidia…
80
Material from lecture by Francois Chollet, 2018, Stanford
![Page 81: Fundamentals of Deep (Artificial) Neural Networks (DNN) · Recurrent Neural Networks Motivation Feed forward networks accept a fixed-sized vector as input and produce a fixed-sized](https://reader030.fdocuments.in/reader030/viewer/2022040609/5ecd24e35f998d2c8838dd80/html5/thumbnails/81.jpg)
Keras modelsInstallation
◦ Anaconda -> Tensorflow -> Keras
Build-in◦ Conv1D, Conv2D, Conv3D…
◦ MaxPooling1D, MaxPooling2D, MaxPooling3D…
◦ Dense, Activation, RNN…
The Sequential Model◦ Very simple
◦ Single-input, Single-output, sequential layer stacks
The functional API◦ Mix & Match
◦ Multi-input, multi-output, arbitrary static graph topologies
81
![Page 82: Fundamentals of Deep (Artificial) Neural Networks (DNN) · Recurrent Neural Networks Motivation Feed forward networks accept a fixed-sized vector as input and produce a fixed-sized](https://reader030.fdocuments.in/reader030/viewer/2022040609/5ecd24e35f998d2c8838dd80/html5/thumbnails/82.jpg)
Sequential>>from keras.models import Sequential
>>model = Sequential()
>> from keras.layers import Dense
>> model.add(Dense(units=64, activation='relu', input_dim=100))
>> model.add(Dense(units=10, activation='softmax'))
>> model.compile(loss='categorical_crossentropy', optimizer='sgd', metrics=['accuracy'])
>> model.fit(x_train, y_train, epochs=5, batch_size=32)
>> loss_and_metrics = model.evaluate(x_test, y_test, batch_size=128)
>> classes = model.predict(x_test)
82
![Page 83: Fundamentals of Deep (Artificial) Neural Networks (DNN) · Recurrent Neural Networks Motivation Feed forward networks accept a fixed-sized vector as input and produce a fixed-sized](https://reader030.fdocuments.in/reader030/viewer/2022040609/5ecd24e35f998d2c8838dd80/html5/thumbnails/83.jpg)
Functional >> from keras.layers import Input, Dense
>> from keras.models import Model
>> inputs = Input(shape=(784,))
>> x = Dense(64, activation='relu')(inputs)
>> x = Dense(64, activation='relu')(x)
>> predictions = Dense(10, activation='softmax')(x)
>> model = Model(inputs=inputs, outputs=predictions)
>> model.compile(optimizer='rmsprop', loss='categorical_crossentropy', metrics=['accuracy'])
>> model.fit(data, labels)
83
![Page 84: Fundamentals of Deep (Artificial) Neural Networks (DNN) · Recurrent Neural Networks Motivation Feed forward networks accept a fixed-sized vector as input and produce a fixed-sized](https://reader030.fdocuments.in/reader030/viewer/2022040609/5ecd24e35f998d2c8838dd80/html5/thumbnails/84.jpg)
ReferencesStephens, Zachary D., et al. "Big data: astronomical or genomical?." PLoS biology 13.7 (2015): e1002195.
LeCun, Yann, Yoshua Bengio, and Geoffrey Hinton. "Deep learning.“ Nature 521.7553 (2015): 436-444.
Goodfellow, Ian, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.
Kietzmann, Tim Christian, Patrick McClure, and Nikolaus Kriegeskorte. "Deep Neural Networks In Computational Neuroscience. " bioRxiv (2017): 133504.
84