LTspice IV Getting Started GuideLTspice IV Getting Started Guide
Getting started with theanoApr1 -...
Transcript of Getting started with theanoApr1 -...
![Page 1: Getting started with theanoApr1 - NVIDIAon-demand.gputechconf.com/gtc/2015/webinar/deep-learning...Seminar #5 - Getting Started with the Torch Framework 38 Hands-on Lab 1. Create an](https://reader033.fdocuments.in/reader033/viewer/2022050303/5f6bb941c46dbd1f92695a0f/html5/thumbnails/1.jpg)
Jon Barker, Solutions Architect
Getting started with Theano
![Page 2: Getting started with theanoApr1 - NVIDIAon-demand.gputechconf.com/gtc/2015/webinar/deep-learning...Seminar #5 - Getting Started with the Torch Framework 38 Hands-on Lab 1. Create an](https://reader033.fdocuments.in/reader033/viewer/2022050303/5f6bb941c46dbd1f92695a0f/html5/thumbnails/2.jpg)
2
Agenda
What is Theano?
Simple first application
Theano for deep learning
Step-by-step example
Related projects
Example applications
![Page 3: Getting started with theanoApr1 - NVIDIAon-demand.gputechconf.com/gtc/2015/webinar/deep-learning...Seminar #5 - Getting Started with the Torch Framework 38 Hands-on Lab 1. Create an](https://reader033.fdocuments.in/reader033/viewer/2022050303/5f6bb941c46dbd1f92695a0f/html5/thumbnails/3.jpg)
3
What is Theano?
A Python library for symbolic maths - far broader than just Deep Learning
Tightly integrated with the Python ecosystem
Fast C/CUDA back-end and transparent GPU acceleration
A mathematical symbolic expression compiler
![Page 4: Getting started with theanoApr1 - NVIDIAon-demand.gputechconf.com/gtc/2015/webinar/deep-learning...Seminar #5 - Getting Started with the Torch Framework 38 Hands-on Lab 1. Create an](https://reader033.fdocuments.in/reader033/viewer/2022050303/5f6bb941c46dbd1f92695a0f/html5/thumbnails/4.jpg)
4
What is Theano?
Variables and expressions are symbolic - more like maths than code…
…but, symbolic expressions use a familiar NumPy-like syntax
Symbolic expression compiler
![Page 5: Getting started with theanoApr1 - NVIDIAon-demand.gputechconf.com/gtc/2015/webinar/deep-learning...Seminar #5 - Getting Started with the Torch Framework 38 Hands-on Lab 1. Create an](https://reader033.fdocuments.in/reader033/viewer/2022050303/5f6bb941c46dbd1f92695a0f/html5/thumbnails/5.jpg)
5
What is Theano?
Developed and used since January 2008, created at Université de Montréal
Large contributor community
Tools for inspecting and debugging code
Great tutorials and examples - http://deeplearning.net/tutorial/
Additional features
![Page 6: Getting started with theanoApr1 - NVIDIAon-demand.gputechconf.com/gtc/2015/webinar/deep-learning...Seminar #5 - Getting Started with the Torch Framework 38 Hands-on Lab 1. Create an](https://reader033.fdocuments.in/reader033/viewer/2022050303/5f6bb941c46dbd1f92695a0f/html5/thumbnails/6.jpg)
6
What is Theano?
Recipe for a Theano application:
Define symbolic expressions
Compile a function that can compute numeric values using those expressions
Execute that function on data
Theano defines a language, a compiler and a library
![Page 7: Getting started with theanoApr1 - NVIDIAon-demand.gputechconf.com/gtc/2015/webinar/deep-learning...Seminar #5 - Getting Started with the Torch Framework 38 Hands-on Lab 1. Create an](https://reader033.fdocuments.in/reader033/viewer/2022050303/5f6bb941c46dbd1f92695a0f/html5/thumbnails/7.jpg)
7
Example Theano applicationy = a⇥ b
a, b 2 R
![Page 8: Getting started with theanoApr1 - NVIDIAon-demand.gputechconf.com/gtc/2015/webinar/deep-learning...Seminar #5 - Getting Started with the Torch Framework 38 Hands-on Lab 1. Create an](https://reader033.fdocuments.in/reader033/viewer/2022050303/5f6bb941c46dbd1f92695a0f/html5/thumbnails/8.jpg)
8
Example Theano application
Initialize symbolic variables
y = a⇥ b
a, b 2 R
![Page 9: Getting started with theanoApr1 - NVIDIAon-demand.gputechconf.com/gtc/2015/webinar/deep-learning...Seminar #5 - Getting Started with the Torch Framework 38 Hands-on Lab 1. Create an](https://reader033.fdocuments.in/reader033/viewer/2022050303/5f6bb941c46dbd1f92695a0f/html5/thumbnails/9.jpg)
9
Example Theano application
Initialize symbolic variables
Define symbolic expression
y = a⇥ b
a, b 2 R
![Page 10: Getting started with theanoApr1 - NVIDIAon-demand.gputechconf.com/gtc/2015/webinar/deep-learning...Seminar #5 - Getting Started with the Torch Framework 38 Hands-on Lab 1. Create an](https://reader033.fdocuments.in/reader033/viewer/2022050303/5f6bb941c46dbd1f92695a0f/html5/thumbnails/10.jpg)
10
Example Theano application
Initialize symbolic variables
Define symbolic expression
Compile a function
y = a⇥ b
a, b 2 R
![Page 11: Getting started with theanoApr1 - NVIDIAon-demand.gputechconf.com/gtc/2015/webinar/deep-learning...Seminar #5 - Getting Started with the Torch Framework 38 Hands-on Lab 1. Create an](https://reader033.fdocuments.in/reader033/viewer/2022050303/5f6bb941c46dbd1f92695a0f/html5/thumbnails/11.jpg)
11
Example Theano application
Initialize symbolic variables
Define symbolic expression
Compile a function
Use on numeric data
y = a⇥ b
a, b 2 R
![Page 12: Getting started with theanoApr1 - NVIDIAon-demand.gputechconf.com/gtc/2015/webinar/deep-learning...Seminar #5 - Getting Started with the Torch Framework 38 Hands-on Lab 1. Create an](https://reader033.fdocuments.in/reader033/viewer/2022050303/5f6bb941c46dbd1f92695a0f/html5/thumbnails/12.jpg)
12
Theano for deep learning
Symbolic computation for tensors
Sub-modules of tensor operations relevant to DL
Highly expressive
Symbolic differentiation
Transparent GPU acceleration
Easily integrates with Python ecosystem
![Page 13: Getting started with theanoApr1 - NVIDIAon-demand.gputechconf.com/gtc/2015/webinar/deep-learning...Seminar #5 - Getting Started with the Torch Framework 38 Hands-on Lab 1. Create an](https://reader033.fdocuments.in/reader033/viewer/2022050303/5f6bb941c46dbd1f92695a0f/html5/thumbnails/13.jpg)
13
Theano for deep learningSymbolic computation for tensors
x1 x2 x3
y1 y2
w11 w12
w21 w22
w31 w33
y2 = �(w12x1 + w22x2 + w32x3)y1 = �(w11x1 + w21x2 + w31x3)=)
y = �(Wx)
![Page 14: Getting started with theanoApr1 - NVIDIAon-demand.gputechconf.com/gtc/2015/webinar/deep-learning...Seminar #5 - Getting Started with the Torch Framework 38 Hands-on Lab 1. Create an](https://reader033.fdocuments.in/reader033/viewer/2022050303/5f6bb941c46dbd1f92695a0f/html5/thumbnails/14.jpg)
14
Theano for deep learningHighly expressive: easily add new activation or loss functions
![Page 15: Getting started with theanoApr1 - NVIDIAon-demand.gputechconf.com/gtc/2015/webinar/deep-learning...Seminar #5 - Getting Started with the Torch Framework 38 Hands-on Lab 1. Create an](https://reader033.fdocuments.in/reader033/viewer/2022050303/5f6bb941c46dbd1f92695a0f/html5/thumbnails/15.jpg)
15
Theano for deep learningSymbolic differentiation
The update is a function of partial derivative of error w.r.t
parameters
![Page 16: Getting started with theanoApr1 - NVIDIAon-demand.gputechconf.com/gtc/2015/webinar/deep-learning...Seminar #5 - Getting Started with the Torch Framework 38 Hands-on Lab 1. Create an](https://reader033.fdocuments.in/reader033/viewer/2022050303/5f6bb941c46dbd1f92695a0f/html5/thumbnails/16.jpg)
16
Example: Linear model
Fit a linear model to this data, i.e. find such that
Example credit: “Introduction to Deep Learning with Python”, Alec Radford, https://www.youtube.com/watch?v=S75EdAcXHKk
w 2 R y = wx
![Page 17: Getting started with theanoApr1 - NVIDIAon-demand.gputechconf.com/gtc/2015/webinar/deep-learning...Seminar #5 - Getting Started with the Torch Framework 38 Hands-on Lab 1. Create an](https://reader033.fdocuments.in/reader033/viewer/2022050303/5f6bb941c46dbd1f92695a0f/html5/thumbnails/17.jpg)
17
Example: Linear model
Initialize symbolic variables
Example credit: “Introduction to Deep Learning with Python”, Alec Radford, https://www.youtube.com/watch?v=S75EdAcXHKk
![Page 18: Getting started with theanoApr1 - NVIDIAon-demand.gputechconf.com/gtc/2015/webinar/deep-learning...Seminar #5 - Getting Started with the Torch Framework 38 Hands-on Lab 1. Create an](https://reader033.fdocuments.in/reader033/viewer/2022050303/5f6bb941c46dbd1f92695a0f/html5/thumbnails/18.jpg)
18
Example: Linear model
Initialize symbolic variables
Define symbolic model
Example credit: “Introduction to Deep Learning with Python”, Alec Radford, https://www.youtube.com/watch?v=S75EdAcXHKk
![Page 19: Getting started with theanoApr1 - NVIDIAon-demand.gputechconf.com/gtc/2015/webinar/deep-learning...Seminar #5 - Getting Started with the Torch Framework 38 Hands-on Lab 1. Create an](https://reader033.fdocuments.in/reader033/viewer/2022050303/5f6bb941c46dbd1f92695a0f/html5/thumbnails/19.jpg)
19
Example: Linear model
Initialize symbolic variables
Define symbolic model
Initialize model parameter
Example credit: “Introduction to Deep Learning with Python”, Alec Radford, https://www.youtube.com/watch?v=S75EdAcXHKk
![Page 20: Getting started with theanoApr1 - NVIDIAon-demand.gputechconf.com/gtc/2015/webinar/deep-learning...Seminar #5 - Getting Started with the Torch Framework 38 Hands-on Lab 1. Create an](https://reader033.fdocuments.in/reader033/viewer/2022050303/5f6bb941c46dbd1f92695a0f/html5/thumbnails/20.jpg)
20
Example: Linear model
Initialize symbolic variables
Define symbolic model
Initialize model parameter
Define symbolic loss
Example credit: “Introduction to Deep Learning with Python”, Alec Radford, https://www.youtube.com/watch?v=S75EdAcXHKk
![Page 21: Getting started with theanoApr1 - NVIDIAon-demand.gputechconf.com/gtc/2015/webinar/deep-learning...Seminar #5 - Getting Started with the Torch Framework 38 Hands-on Lab 1. Create an](https://reader033.fdocuments.in/reader033/viewer/2022050303/5f6bb941c46dbd1f92695a0f/html5/thumbnails/21.jpg)
21
Example: Linear model
Initialize symbolic variables
Define symbolic model
Initialize model parameter
Define symbolic lossDetermine partial derivative of loss w.r.t parameter
Example credit: “Introduction to Deep Learning with Python”, Alec Radford, https://www.youtube.com/watch?v=S75EdAcXHKk
![Page 22: Getting started with theanoApr1 - NVIDIAon-demand.gputechconf.com/gtc/2015/webinar/deep-learning...Seminar #5 - Getting Started with the Torch Framework 38 Hands-on Lab 1. Create an](https://reader033.fdocuments.in/reader033/viewer/2022050303/5f6bb941c46dbd1f92695a0f/html5/thumbnails/22.jpg)
22
Example: Linear model
Initialize symbolic variables
Define symbolic model
Initialize model parameter
Define symbolic lossDetermine partial derivative of loss w.r.t parameterDefine how to update parameter based on gradient
Example credit: “Introduction to Deep Learning with Python”, Alec Radford, https://www.youtube.com/watch?v=S75EdAcXHKk
![Page 23: Getting started with theanoApr1 - NVIDIAon-demand.gputechconf.com/gtc/2015/webinar/deep-learning...Seminar #5 - Getting Started with the Torch Framework 38 Hands-on Lab 1. Create an](https://reader033.fdocuments.in/reader033/viewer/2022050303/5f6bb941c46dbd1f92695a0f/html5/thumbnails/23.jpg)
23
Example: Linear model
Initialize symbolic variables
Define symbolic model
Initialize model parameter
Define symbolic lossDetermine partial derivative of loss w.r.t parameterDefine how to update parameter based on gradient
compile theano function
Example credit: “Introduction to Deep Learning with Python”, Alec Radford, https://www.youtube.com/watch?v=S75EdAcXHKk
![Page 24: Getting started with theanoApr1 - NVIDIAon-demand.gputechconf.com/gtc/2015/webinar/deep-learning...Seminar #5 - Getting Started with the Torch Framework 38 Hands-on Lab 1. Create an](https://reader033.fdocuments.in/reader033/viewer/2022050303/5f6bb941c46dbd1f92695a0f/html5/thumbnails/24.jpg)
24
Example: Linear model
Initialize symbolic variables
Define symbolic model
Initialize model parameter
Define symbolic lossDetermine partial derivative of loss w.r.t parameterDefine how to update parameter based on gradient
compile theano functionIterate through data 100 times, updating parameter after each iteration
Example credit: “Introduction to Deep Learning with Python”, Alec Radford, https://www.youtube.com/watch?v=S75EdAcXHKk
![Page 25: Getting started with theanoApr1 - NVIDIAon-demand.gputechconf.com/gtc/2015/webinar/deep-learning...Seminar #5 - Getting Started with the Torch Framework 38 Hands-on Lab 1. Create an](https://reader033.fdocuments.in/reader033/viewer/2022050303/5f6bb941c46dbd1f92695a0f/html5/thumbnails/25.jpg)
25
Example: Linear model
Before training After 100 iterations
![Page 26: Getting started with theanoApr1 - NVIDIAon-demand.gputechconf.com/gtc/2015/webinar/deep-learning...Seminar #5 - Getting Started with the Torch Framework 38 Hands-on Lab 1. Create an](https://reader033.fdocuments.in/reader033/viewer/2022050303/5f6bb941c46dbd1f92695a0f/html5/thumbnails/26.jpg)
26
Designing more complex architectures
Layer types are typically defined as Python classes
![Page 27: Getting started with theanoApr1 - NVIDIAon-demand.gputechconf.com/gtc/2015/webinar/deep-learning...Seminar #5 - Getting Started with the Torch Framework 38 Hands-on Lab 1. Create an](https://reader033.fdocuments.in/reader033/viewer/2022050303/5f6bb941c46dbd1f92695a0f/html5/thumbnails/27.jpg)
27
Designing more complex architectures
Networks can be defined symbolically as sequences of class instances
![Page 28: Getting started with theanoApr1 - NVIDIAon-demand.gputechconf.com/gtc/2015/webinar/deep-learning...Seminar #5 - Getting Started with the Torch Framework 38 Hands-on Lab 1. Create an](https://reader033.fdocuments.in/reader033/viewer/2022050303/5f6bb941c46dbd1f92695a0f/html5/thumbnails/28.jpg)
28
Related projects
Blocks
Keras
Lasagne
Morb
Pylearn2
PyMC 3
sklearn-theano
theano-rnn
more…
Built on top of Theano (mostly machine learning)
Typically simplify syntax and interface for artificial neural network training at the expense of
expressiveness
![Page 29: Getting started with theanoApr1 - NVIDIAon-demand.gputechconf.com/gtc/2015/webinar/deep-learning...Seminar #5 - Getting Started with the Torch Framework 38 Hands-on Lab 1. Create an](https://reader033.fdocuments.in/reader033/viewer/2022050303/5f6bb941c46dbd1f92695a0f/html5/thumbnails/29.jpg)
29
Related projectsCNN example in Keras
Input Conv MaxPool Dense DropOut Dense
![Page 30: Getting started with theanoApr1 - NVIDIAon-demand.gputechconf.com/gtc/2015/webinar/deep-learning...Seminar #5 - Getting Started with the Torch Framework 38 Hands-on Lab 1. Create an](https://reader033.fdocuments.in/reader033/viewer/2022050303/5f6bb941c46dbd1f92695a0f/html5/thumbnails/30.jpg)
30
Example applicationsMusic recommendation at Spotify
http://benanne.github.io/2014/08/05/spotify-cnns.html
Audio spectrogram input
Latent factors output
![Page 31: Getting started with theanoApr1 - NVIDIAon-demand.gputechconf.com/gtc/2015/webinar/deep-learning...Seminar #5 - Getting Started with the Torch Framework 38 Hands-on Lab 1. Create an](https://reader033.fdocuments.in/reader033/viewer/2022050303/5f6bb941c46dbd1f92695a0f/html5/thumbnails/31.jpg)
31
Example applicationsClassifying galaxies and plankton and winning Kaggle contests
http://benanne.github.io/2014/04/05/galaxy-zoo.html
http://benanne.github.io/2015/03/17/plankton.html
![Page 32: Getting started with theanoApr1 - NVIDIAon-demand.gputechconf.com/gtc/2015/webinar/deep-learning...Seminar #5 - Getting Started with the Torch Framework 38 Hands-on Lab 1. Create an](https://reader033.fdocuments.in/reader033/viewer/2022050303/5f6bb941c46dbd1f92695a0f/html5/thumbnails/32.jpg)
32
Example applications
Many more examples at: http://deeplearning.net/tutorial/
Unsupervised training and auto-encoders
Recurrent Neural Networks
![Page 33: Getting started with theanoApr1 - NVIDIAon-demand.gputechconf.com/gtc/2015/webinar/deep-learning...Seminar #5 - Getting Started with the Torch Framework 38 Hands-on Lab 1. Create an](https://reader033.fdocuments.in/reader033/viewer/2022050303/5f6bb941c46dbd1f92695a0f/html5/thumbnails/33.jpg)
33
Installation
Linux, OS X or Windows
Requirements:
Python >= 2.6
g++, python-dev
NumPy, SciPy
BLAS
![Page 34: Getting started with theanoApr1 - NVIDIAon-demand.gputechconf.com/gtc/2015/webinar/deep-learning...Seminar #5 - Getting Started with the Torch Framework 38 Hands-on Lab 1. Create an](https://reader033.fdocuments.in/reader033/viewer/2022050303/5f6bb941c46dbd1f92695a0f/html5/thumbnails/34.jpg)
34
Installation
Basic instructions:
pip install Theano
easy_install Theano
Advanced instructions (for bleeding edge installs):
git clone git://github.com/Theano/Theano.git
cd Theano
python setup.py develop –-user
![Page 35: Getting started with theanoApr1 - NVIDIAon-demand.gputechconf.com/gtc/2015/webinar/deep-learning...Seminar #5 - Getting Started with the Torch Framework 38 Hands-on Lab 1. Create an](https://reader033.fdocuments.in/reader033/viewer/2022050303/5f6bb941c46dbd1f92695a0f/html5/thumbnails/35.jpg)
35
Configuration
Three ways to configure Theano:
~/.theanorc: settings you always want
THEANO_FLAGS: setting for one job
theano.config: mid-code settings changes
![Page 36: Getting started with theanoApr1 - NVIDIAon-demand.gputechconf.com/gtc/2015/webinar/deep-learning...Seminar #5 - Getting Started with the Torch Framework 38 Hands-on Lab 1. Create an](https://reader033.fdocuments.in/reader033/viewer/2022050303/5f6bb941c46dbd1f92695a0f/html5/thumbnails/36.jpg)
36
GPU acceleration
Three ways to invoke:
Add device=gpu to .theanorc
Add device=gpu to THEANO_FLAGS
Set theano.config.device=‘gpu’ in code
cuDNN acceleration (including v4) is automatic if installed on system
![Page 37: Getting started with theanoApr1 - NVIDIAon-demand.gputechconf.com/gtc/2015/webinar/deep-learning...Seminar #5 - Getting Started with the Torch Framework 38 Hands-on Lab 1. Create an](https://reader033.fdocuments.in/reader033/viewer/2022050303/5f6bb941c46dbd1f92695a0f/html5/thumbnails/37.jpg)
37
Deep Learning Lab Series Scheduledeveloper.nvidia.com/deep-learning-courses
▪ Review the other seminars in series
Seminar #2 – Introduction to DIGITs
Seminar #3 - Getting Started with the CaffeFramework
Seminar #5 - Getting Started with the Torch Framework
![Page 38: Getting started with theanoApr1 - NVIDIAon-demand.gputechconf.com/gtc/2015/webinar/deep-learning...Seminar #5 - Getting Started with the Torch Framework 38 Hands-on Lab 1. Create an](https://reader033.fdocuments.in/reader033/viewer/2022050303/5f6bb941c46dbd1f92695a0f/html5/thumbnails/38.jpg)
38
Hands-on Lab
1. Create an account at nvidia.qwiklab.com
2. Go to “Getting started with Theano” lab at bit.ly/dlnvlab4
3. Start the lab and enjoy!
Only requires a supported browser, no NVIDIA GPU necessary!
Lab is free until end of Deep Learning Lab series
![Page 39: Getting started with theanoApr1 - NVIDIAon-demand.gputechconf.com/gtc/2015/webinar/deep-learning...Seminar #5 - Getting Started with the Torch Framework 38 Hands-on Lab 1. Create an](https://reader033.fdocuments.in/reader033/viewer/2022050303/5f6bb941c46dbd1f92695a0f/html5/thumbnails/39.jpg)