Deep Learning with RFrancesca Lazzeri - @frlazzeriData Scientist II - Microsoft, AI Research

Agenda

Better understanding of R DL tools

Demo

Deep Learning

with R

What is Deep

Learning

What is Deep

LearningFundamental concepts in Deep Learning

Forward Propagation Algorithm

Activation Functions

Gradient Descent Backpropagation

What is Deep

LearningFundamental concepts in Deep Learning

Forward Propagation Algorithm

Gradient Descent Backpropagation

Activation Functions

What is Deep

Learning

Age

Bank Balance

Retirement Status

…

Example as seen by linear regression

Number of Transactions

What is Deep

LearningInteractions

o Neural networks account for interactions really well

o Deep learning uses especially powerful neural networks for:* Text* Images* Videos* Audio* Source code

What is Deep

Learning

Age

Bank Balance

Retirement Status

…

Deep learning models capture interactions

Number of Transactions

What is Deep

Learning

Age

Bank Balance

Retirement Status

# Accounts

Interactions in neural networks

Number of Transactions

Input Layer

Hidden Layer

Output Layer

What is Deep

Learning

2

3

Forward Propagation Algorithm

9

5

1

InputHidden

Output

# Children

# Accounts

# Transactions-1

1

1

1

-1

2

What is Deep

LearningFundamental concepts in Deep Learning

Activation Functions

Gradient Descent Backpropagation

Forward Propagation Algorithm

What is Deep

LearningActivation Functions

2

3

9

tanh (2+3)

tanh (-2+3)

InputHidden

Output

# Accounts

# Transactions-1

1

1

1

-1

2

# Children

What is Deep

LearningReLU Activation Function

3

5

Input

4

2

4

-5

364

0

52

Hidden

Output

7

-3

26

0

Hidden

1

-1

2

2

What is Deep

LearningRepresentation Learning

o Deep networks internally build representations of patterns in the data

o Partially replace the need for feature engineering

o Subsequent layers build increasingly sophisticated representations of raw data

o Modeler doesn’t need to specify the interactions

o When you train the model, the neural network gets weights that find the relevant patterns to make better predictions

What is Deep

LearningFundamental concepts in Deep Learning

Gradient Descent

Forward Propagation Algorithm

Backpropagation

Activation Functions

What is Deep

LearningThe Need for Optimization

o Predictions with multiple points* Making accurate predictions gets harder with more points* At any set of weights, there are many values of the error* Correspond to the many points we make predictions for

o Loss function* Aggregate errors in predictions from many data points into single number* Measure of model’s predictive performance

What is Deep

LearningThe Need for Optimization

o Squared error loss function

o Total Squared Error: 150o Mean Squared Error: 50o Lower loss function value means a better modelo Goal: find the weights that give the lowest value for the loss functiono Gradient descent!

Prediction Actual Error Squared Error

10 20 -10 100

8 3 5 25

6 1 5 25

What is Deep

LearningGradient Descent

Loss(w)

w

What is Deep

LearningGradient Descent

o Slope calculation example

Actual Target Value = 10

o To calculate the slope for a weight, need to multiply:* Slope of the loss function w.r.t value at the node we feed into* The value of the node that feeds into our weight* Slope of activation function w.r.t value we feed into

32

6

What is Deep

LearningFundamental concepts in Deep Learning

Backpropagation

Forward Propagation Algorithm

Gradient Descent

Activation Functions

What is Deep

LearningBackpropagation

3

5

Input

364

0

52

Hidden

Output26

0

Hidden

What is Deep

LearningBackpropagation

o Allows gradient descent to update all weights in neural network (by getting gradientsfor all weights)

o Go back one layer at a time

o Important to understand the process, but you will generally use a library thatimplements this

Deep Learning

with R

• Feed-forward neural network

• Convolutional neural network (CNN)

MXNetR

• Restricted Boltzmann machine

• Deep belief network

darch

• Feed-forward neural network

• Restricted Boltzmann machine

• Deep belief network

• Stacked autoencoders

deepnet

• Feed-forward neural network

• Deep autoencoders

H2O

• Simplify some functions from H2O

• Deepnet packages

deepr

Deep Learning

with R

Model/Dataset

MNIST Iris Forest Cover Type

Accuracy (%) Runtime (sec) Accuracy (%) Runtime (sec) Accuracy (%) Runtime (sec)

MXNetR (CPU) 98.33 147.78 83.04 1.46 66.8 30.24

MXNetR (GPU) 98.27 336.94 84.77 3.09 67.75 80.89

darch 100 92.09 1368.31 69.12 1.71 – –

darch 500/300 95.88 4706.23 54.78 2.1 – –

deepnet DBN 97.85 6775.4 30.43 0.89 14.06 67.97

deepnet DNN 97.05 2183.92 78.26 0.42 26.01 25.67

H2O 98.08 543.14 89.56 0.53 67.36 5.78

Random Forest 96.77 125.28 91.3 2.89 86.25 9.41

Deep Learning

with RR interface to Keras

Demo Deep Learning with R on Azure with Keras and CNTK

DSVM CNTK R & Keras

Demo Keras Workflow Steps to Build your Model

o Specify architecture

o Compile the model

o Fit the model

o Predict

Demo Preparing the Data

Demo Defining the Model

Demo Defining the Model

Demo Defining the Model

Demo Training and Evaluation

References

http://blog.revolutionanalytics.com/2017/08/keras-and-cntk.html

http://www.rblog.uni-freiburg.de

https://keras.rstudio.com/

https://campus.datacamp.com/courses/deep-learning

http://gluon.mxnet.io

Thank You!Francesca Lazzeri - @frlazzeriData Scientist II - Microsoft, AI Research