Introduction to Deep Learning (Dmytro Fishman Technology Stream)
-
Upload
lviv-it-arena -
Category
Technology
-
view
1.050 -
download
4
Transcript of Introduction to Deep Learning (Dmytro Fishman Technology Stream)
![Page 1: Introduction to Deep Learning (Dmytro Fishman Technology Stream)](https://reader031.fdocuments.in/reader031/viewer/2022021422/5882198a1a28ab3f4c8b61e1/html5/thumbnails/1.jpg)
Introduction to Deep LearningDmytro Fishman, University of Tartu
![Page 2: Introduction to Deep Learning (Dmytro Fishman Technology Stream)](https://reader031.fdocuments.in/reader031/viewer/2022021422/5882198a1a28ab3f4c8b61e1/html5/thumbnails/2.jpg)
What is Machine Learning?Was born with a goal to create Artificial Intelligence !There were two main branches: • Symbolic AI, based on if/then like rules (GOFAI) !• Inspired by Neuroscience principle - learn through
examples (later developed into ML)
![Page 3: Introduction to Deep Learning (Dmytro Fishman Technology Stream)](https://reader031.fdocuments.in/reader031/viewer/2022021422/5882198a1a28ab3f4c8b61e1/html5/thumbnails/3.jpg)
Machine Learning founding disciplines
Computer Science
Statistics
Information Theory Databases
Probability Theory
Optimisation
Machine Learning
![Page 4: Introduction to Deep Learning (Dmytro Fishman Technology Stream)](https://reader031.fdocuments.in/reader031/viewer/2022021422/5882198a1a28ab3f4c8b61e1/html5/thumbnails/4.jpg)
What is Machine Learning?
Machine learning is a scientific field that develops computer programs that make data-driven decisions without being explicitly programmed and also automatically improve over time.
![Page 5: Introduction to Deep Learning (Dmytro Fishman Technology Stream)](https://reader031.fdocuments.in/reader031/viewer/2022021422/5882198a1a28ab3f4c8b61e1/html5/thumbnails/5.jpg)
What is Machine Learning?
Machine learning is • a scientific field • develops computer programs • make data-driven decisions • without being explicitly programmed to • also automatically improve over time.
![Page 6: Introduction to Deep Learning (Dmytro Fishman Technology Stream)](https://reader031.fdocuments.in/reader031/viewer/2022021422/5882198a1a28ab3f4c8b61e1/html5/thumbnails/6.jpg)
Examples of Machine LearningSpam Filtering
![Page 7: Introduction to Deep Learning (Dmytro Fishman Technology Stream)](https://reader031.fdocuments.in/reader031/viewer/2022021422/5882198a1a28ab3f4c8b61e1/html5/thumbnails/7.jpg)
Examples of Machine LearningFace Recognition
![Page 8: Introduction to Deep Learning (Dmytro Fishman Technology Stream)](https://reader031.fdocuments.in/reader031/viewer/2022021422/5882198a1a28ab3f4c8b61e1/html5/thumbnails/8.jpg)
Examples of Machine LearningRecommendation Systems
![Page 9: Introduction to Deep Learning (Dmytro Fishman Technology Stream)](https://reader031.fdocuments.in/reader031/viewer/2022021422/5882198a1a28ab3f4c8b61e1/html5/thumbnails/9.jpg)
Comes from nature…
http://www.borgenmagazine.com/wp-content/uploads/2014/06/natural_disasters.jpg
![Page 10: Introduction to Deep Learning (Dmytro Fishman Technology Stream)](https://reader031.fdocuments.in/reader031/viewer/2022021422/5882198a1a28ab3f4c8b61e1/html5/thumbnails/10.jpg)
…and industryhttp://forums.vwvortex.com/showthread.php?3208922-2008-Dodge-Challenger-Factory-production-pics./page2
![Page 11: Introduction to Deep Learning (Dmytro Fishman Technology Stream)](https://reader031.fdocuments.in/reader031/viewer/2022021422/5882198a1a28ab3f4c8b61e1/html5/thumbnails/11.jpg)
Often is very messy
http://insights.wired.com/profiles/blogs/what-s-the-big-deal-with-unstructured-data
![Page 12: Introduction to Deep Learning (Dmytro Fishman Technology Stream)](https://reader031.fdocuments.in/reader031/viewer/2022021422/5882198a1a28ab3f4c8b61e1/html5/thumbnails/12.jpg)
Ideally a long !list of examples of the same nature
![Page 13: Introduction to Deep Learning (Dmytro Fishman Technology Stream)](https://reader031.fdocuments.in/reader031/viewer/2022021422/5882198a1a28ab3f4c8b61e1/html5/thumbnails/13.jpg)
Types of data
255,255,255,255 255,255,255,255 255, 56, 65,255 255, 98, 89,110
To be or not to be.
x = (0.5, 0.9, 0.7, -0.3, …)
x = (255, 255, 255, 255, …)
x = (0, …, 0, 2, 1, 0, 0, 1, 2, …)
to or not
bethe
cat
low John
Adopted from P.Vincent http://videolectures.net/deeplearning2015_vincent_machine_learning/
![Page 14: Introduction to Deep Learning (Dmytro Fishman Technology Stream)](https://reader031.fdocuments.in/reader031/viewer/2022021422/5882198a1a28ab3f4c8b61e1/html5/thumbnails/14.jpg)
Machine Learning tasksSupervised Learning • Classification • Regression!!Unsupervised Learning!• Clustering • Dimensionality reduction • Density Estimation !Reinforcement Learning - taking actions in an environment to maximise reward
![Page 15: Introduction to Deep Learning (Dmytro Fishman Technology Stream)](https://reader031.fdocuments.in/reader031/viewer/2022021422/5882198a1a28ab3f4c8b61e1/html5/thumbnails/15.jpg)
Machine Learning tasksSupervised Learning • Classification • Regression!!Unsupervised Learning!• Clustering • Dimensionality reduction • Density Estimation !Reinforcement Learning - taking actions in an environment to maximise reward
![Page 16: Introduction to Deep Learning (Dmytro Fishman Technology Stream)](https://reader031.fdocuments.in/reader031/viewer/2022021422/5882198a1a28ab3f4c8b61e1/html5/thumbnails/16.jpg)
Supervised LearningCat!
Cat
???
Dog
x1 = (87, 205, 255, 155, …); y1 = 1
x2 = (95, 195, 245, 155, …); y2 = 1
x3 = (43, 159, 255, 5, …); y3 = -1
x4 = (180, 66, 245, …); !y4 = f(x4) = 1
Features Labels
Adopted from P.Vincent http://videolectures.net/deeplearning2015_vincent_machine_learning/
![Page 17: Introduction to Deep Learning (Dmytro Fishman Technology Stream)](https://reader031.fdocuments.in/reader031/viewer/2022021422/5882198a1a28ab3f4c8b61e1/html5/thumbnails/17.jpg)
Evolution of ML methodsRule-based Classic Machine
LearningRepresentation
Learning Deep Learning
Adopted from Y.Bengio http://videolectures.net/deeplearning2015_bengio_theoretical_motivations/
Input Input Input Input
Fixed set of rules
Output
Hand designed features
Learning
Automated feature extraction
Output
Learning
Output
Low level features
High level features
Learning
Output
![Page 18: Introduction to Deep Learning (Dmytro Fishman Technology Stream)](https://reader031.fdocuments.in/reader031/viewer/2022021422/5882198a1a28ab3f4c8b61e1/html5/thumbnails/18.jpg)
Mimicking the brainEngineering perspective • Compact!• Energy efficient (12 watts—a fifth of
the power of a lightbulb) • 10^11 neurons • for each neurone there are 10-50
supporting cells (glial cells) • 10^14 synapses (connections)
Computing power • Fairly slow in arithmetic calculations • Very efficient in computer vision, speech, language etc.
Adopted from L.Bottou http://videolectures.net/deeplearning2015_bottou_neural_networks/
![Page 19: Introduction to Deep Learning (Dmytro Fishman Technology Stream)](https://reader031.fdocuments.in/reader031/viewer/2022021422/5882198a1a28ab3f4c8b61e1/html5/thumbnails/19.jpg)
Biological Neuron
![Page 20: Introduction to Deep Learning (Dmytro Fishman Technology Stream)](https://reader031.fdocuments.in/reader031/viewer/2022021422/5882198a1a28ab3f4c8b61e1/html5/thumbnails/20.jpg)
Caution! Here comes the slide with
some math!
![Page 21: Introduction to Deep Learning (Dmytro Fishman Technology Stream)](https://reader031.fdocuments.in/reader031/viewer/2022021422/5882198a1a28ab3f4c8b61e1/html5/thumbnails/21.jpg)
Artificial Neuronex0
w0
x1w1
x2w2
X
i
xiwi + b
f
axon from neuronedendrite
synapse
output axonactivation!function
f(X
i
xiwi + b)
Slide from CS231n: Convolutional Neural Networks for Visual Recognition http://cs231n.github.io/ by Andrej Karpathy
![Page 22: Introduction to Deep Learning (Dmytro Fishman Technology Stream)](https://reader031.fdocuments.in/reader031/viewer/2022021422/5882198a1a28ab3f4c8b61e1/html5/thumbnails/22.jpg)
Feed Forwardx0
w0
x1w1
x2w2
X
i
xiwi + b
f
axon from neuronedendrite
synapse
output axonactivation!function
f(X
i
xiwi + b)
Slide from CS231n: Convolutional Neural Networks for Visual Recognition http://cs231n.github.io/ by Andrej Karpathy
class Neuron: # ... def forward(inputs): """ assume inputs and weights are 1-D numpy arrays and bias is a number """ cell_body_sum = np.sum(inputs * self.weights) + self.bias firing_rate = 1.0 / (1.0 + math.exp(-cell_body_sum)) # sigmoid activation function return firing_rate
![Page 23: Introduction to Deep Learning (Dmytro Fishman Technology Stream)](https://reader031.fdocuments.in/reader031/viewer/2022021422/5882198a1a28ab3f4c8b61e1/html5/thumbnails/23.jpg)
Training Neural Netowrks
43
255
32
Output
Hidden
…
Input
Image
![Page 24: Introduction to Deep Learning (Dmytro Fishman Technology Stream)](https://reader031.fdocuments.in/reader031/viewer/2022021422/5882198a1a28ab3f4c8b61e1/html5/thumbnails/24.jpg)
32
143
255
Output
Hidden
…
Input
Image
Training Neural Netowrks
![Page 25: Introduction to Deep Learning (Dmytro Fishman Technology Stream)](https://reader031.fdocuments.in/reader031/viewer/2022021422/5882198a1a28ab3f4c8b61e1/html5/thumbnails/25.jpg)
32
143
255
Output
Hidden
…
Input
Image
Training Neural Netowrks
![Page 26: Introduction to Deep Learning (Dmytro Fishman Technology Stream)](https://reader031.fdocuments.in/reader031/viewer/2022021422/5882198a1a28ab3f4c8b61e1/html5/thumbnails/26.jpg)
0.6
32
143
Input
255
0.4
OutputHidden
Image
…Cat
Dog
Error
Training Neural Netowrks
![Page 27: Introduction to Deep Learning (Dmytro Fishman Technology Stream)](https://reader031.fdocuments.in/reader031/viewer/2022021422/5882198a1a28ab3f4c8b61e1/html5/thumbnails/27.jpg)
Deep Neural Networks
![Page 28: Introduction to Deep Learning (Dmytro Fishman Technology Stream)](https://reader031.fdocuments.in/reader031/viewer/2022021422/5882198a1a28ab3f4c8b61e1/html5/thumbnails/28.jpg)
GoogLeNet
![Page 29: Introduction to Deep Learning (Dmytro Fishman Technology Stream)](https://reader031.fdocuments.in/reader031/viewer/2022021422/5882198a1a28ab3f4c8b61e1/html5/thumbnails/29.jpg)
Types of Deep Neural Networks
• Convolutional Neural Networks (CNN) • Convolutional Deep Belief Networks (CDBN)
!• Recurrent Neural Networks (RNN)
• Long Short-Term Memory Networks (LSTM) !
https://en.wikipedia.org/wiki/Deep_learning
![Page 30: Introduction to Deep Learning (Dmytro Fishman Technology Stream)](https://reader031.fdocuments.in/reader031/viewer/2022021422/5882198a1a28ab3f4c8b61e1/html5/thumbnails/30.jpg)
Convolutional Neural Networks
![Page 31: Introduction to Deep Learning (Dmytro Fishman Technology Stream)](https://reader031.fdocuments.in/reader031/viewer/2022021422/5882198a1a28ab3f4c8b61e1/html5/thumbnails/31.jpg)
ImageNet Large Scale Visual Recognition Challenge (ILSVRC)
![Page 32: Introduction to Deep Learning (Dmytro Fishman Technology Stream)](https://reader031.fdocuments.in/reader031/viewer/2022021422/5882198a1a28ab3f4c8b61e1/html5/thumbnails/32.jpg)
Breed?
http://cs.stanford.edu/people/karpathy/ilsvrc/
![Page 33: Introduction to Deep Learning (Dmytro Fishman Technology Stream)](https://reader031.fdocuments.in/reader031/viewer/2022021422/5882198a1a28ab3f4c8b61e1/html5/thumbnails/33.jpg)
Misclassified examples
Overall:trained expert got 5.1% of errorsGoogLeNet 6.7% of errorsBefore Deep Neural Networks the best result was 20.7%
![Page 34: Introduction to Deep Learning (Dmytro Fishman Technology Stream)](https://reader031.fdocuments.in/reader031/viewer/2022021422/5882198a1a28ab3f4c8b61e1/html5/thumbnails/34.jpg)
Galaxy Zoo Challenge
61578 JPG images, made by Hubble
![Page 35: Introduction to Deep Learning (Dmytro Fishman Technology Stream)](https://reader031.fdocuments.in/reader031/viewer/2022021422/5882198a1a28ab3f4c8b61e1/html5/thumbnails/35.jpg)
Galaxy Zoo Challenge
http://benanne.github.io/2014/04/05/galaxy-zoo.html
![Page 36: Introduction to Deep Learning (Dmytro Fishman Technology Stream)](https://reader031.fdocuments.in/reader031/viewer/2022021422/5882198a1a28ab3f4c8b61e1/html5/thumbnails/36.jpg)
Diabetic Retinopathy Detection Challenge
0 stage
4 stage
and 80 000 more images
![Page 37: Introduction to Deep Learning (Dmytro Fishman Technology Stream)](https://reader031.fdocuments.in/reader031/viewer/2022021422/5882198a1a28ab3f4c8b61e1/html5/thumbnails/37.jpg)
Diabetic Retinopathy Detection Challenge
![Page 38: Introduction to Deep Learning (Dmytro Fishman Technology Stream)](https://reader031.fdocuments.in/reader031/viewer/2022021422/5882198a1a28ab3f4c8b61e1/html5/thumbnails/38.jpg)
…
…
Diabetic Retinopathy Detection Challenge
![Page 39: Introduction to Deep Learning (Dmytro Fishman Technology Stream)](https://reader031.fdocuments.in/reader031/viewer/2022021422/5882198a1a28ab3f4c8b61e1/html5/thumbnails/39.jpg)
DeepMind
![Page 40: Introduction to Deep Learning (Dmytro Fishman Technology Stream)](https://reader031.fdocuments.in/reader031/viewer/2022021422/5882198a1a28ab3f4c8b61e1/html5/thumbnails/40.jpg)
DeepMind
![Page 41: Introduction to Deep Learning (Dmytro Fishman Technology Stream)](https://reader031.fdocuments.in/reader031/viewer/2022021422/5882198a1a28ab3f4c8b61e1/html5/thumbnails/41.jpg)
0.6
32
143
Input
255
0.4
OutputHidden
Image
…Right
Left
Lost one life?
Training DeepMind
![Page 42: Introduction to Deep Learning (Dmytro Fishman Technology Stream)](https://reader031.fdocuments.in/reader031/viewer/2022021422/5882198a1a28ab3f4c8b61e1/html5/thumbnails/42.jpg)
Recurrent Neural Networks
![Page 43: Introduction to Deep Learning (Dmytro Fishman Technology Stream)](https://reader031.fdocuments.in/reader031/viewer/2022021422/5882198a1a28ab3f4c8b61e1/html5/thumbnails/43.jpg)
Shakespeare?Recurrent Neural Network
PANDARUS: Alas, I think he shall be come approached and the day When little srain would be attain'd into being never fed, And who is but a chain and subjects of his death, I should not sleep. !Second Senator: They are away this miseries, produced upon my soul, Breaking and strongly should be buried, when I perish The earth and thoughts of many states. !DUKE VINCENTIO: Well, your wit is in the care of side and that. !Second Lord: They would be ruled after this chamber, and my fair nues begun out of the fact, to be conveyed, Whose noble souls I'll have the heart of the wars. !Clown: Come, sir, I will make did behold your worship. !VIOLA: I'll drink it.
VIOLA. On your attendance, my lord; here. !DUKE. Stand you awhile aloof.—Cesario, Thou know'st no less but all; I have unclasp'd To thee the book even of my secret soul: Therefore, good youth, address thy gait unto her; Be not denied access, stand at her doors, And tell them there thy fixed foot shall grow Till thou have audience. !VIOLA. Sure, my noble lord, If she be so abandon'd to her sorrow As it is spoke, she never will admit me. !DUKE. Be clamorous and leap all civil bounds, Rather than make unprofited return. !VIOLA. Say I do speak with her, my lord. What then?
4.4МB Shakespeare
texts
Generated text is from The Unreasonable Effectiveness of Recurrent Neural Networks http://karpathy.github.io/2015/05/21/rnn-effectiveness/ by Andrej Karpathy
![Page 44: Introduction to Deep Learning (Dmytro Fishman Technology Stream)](https://reader031.fdocuments.in/reader031/viewer/2022021422/5882198a1a28ab3f4c8b61e1/html5/thumbnails/44.jpg)
Recurrent Neural NetworkPANDARUS: Alas, I think he shall be come approached and the day When little srain would be attain'd into being never fed, And who is but a chain and subjects of his death, I should not sleep. !Second Senator: They are away this miseries, produced upon my soul, Breaking and strongly should be buried, when I perish The earth and thoughts of many states. !DUKE VINCENTIO: Well, your wit is in the care of side and that. !Second Lord: They would be ruled after this chamber, and my fair nues begun out of the fact, to be conveyed, Whose noble souls I'll have the heart of the wars. !Clown: Come, sir, I will make did behold your worship. !VIOLA: I'll drink it.
VIOLA. On your attendance, my lord; here. !DUKE. Stand you awhile aloof.—Cesario, Thou know'st no less but all; I have unclasp'd To thee the book even of my secret soul: Therefore, good youth, address thy gait unto her; Be not denied access, stand at her doors, And tell them there thy fixed foot shall grow Till thou have audience. !VIOLA. Sure, my noble lord, If she be so abandon'd to her sorrow As it is spoke, she never will admit me. !DUKE. Be clamorous and leap all civil bounds, Rather than make unprofited return. !VIOLA. Say I do speak with her, my lord. What then?
Shakespeare?
Generated text is from The Unreasonable Effectiveness of Recurrent Neural Networks http://karpathy.github.io/2015/05/21/rnn-effectiveness/ by Andrej Karpathy
![Page 45: Introduction to Deep Learning (Dmytro Fishman Technology Stream)](https://reader031.fdocuments.in/reader031/viewer/2022021422/5882198a1a28ab3f4c8b61e1/html5/thumbnails/45.jpg)
Linux 474MB of the source code
/*! * Increment the size file of the new incorrect UI_FILTER group information! * of the size generatively.! */!static int indicate_policy(void)!{! int error;! if (fd == MARN_EPT) {! /*! * The kernel blank will coeld it to userspace.! */! if (ss->segment < mem_total)! unblock_graph_and_set_blocked();! else! ret = 1;! goto bail;! }! segaddr = in_SB(in.addr);! selector = seg / 16;! setup_works = true;! for (i = 0; i < blocks; i++) {! seq = buf[i++];! bpf = bd->bd.next + i * search;! if (fd) {! current = blocked;! }! }! rw->name = "Getjbbregs";! bprm_self_clearl(&iv->version);! regs->new = blocks[(BPF_STATS << info->historidac)] | PFMR_CLOBATHINC_SECONDS << 12;! return segtable;!}
Generated text is from The Unreasonable Effectiveness of Recurrent Neural Networks http://karpathy.github.io/2015/05/21/rnn-effectiveness/ by Andrej Karpathy
![Page 46: Introduction to Deep Learning (Dmytro Fishman Technology Stream)](https://reader031.fdocuments.in/reader031/viewer/2022021422/5882198a1a28ab3f4c8b61e1/html5/thumbnails/46.jpg)
Tools• Python (many implementations) • Torch (http://torch.ch/) • Theano (http://deeplearning.net/software/theano/) • Caffe (http://caffe.berkeleyvision.org/)
!!
Deep Learning Libraries by programming language: http://www.teglor.com/b/deep-learning-libraries-language-cm569
![Page 47: Introduction to Deep Learning (Dmytro Fishman Technology Stream)](https://reader031.fdocuments.in/reader031/viewer/2022021422/5882198a1a28ab3f4c8b61e1/html5/thumbnails/47.jpg)
Online courses
CS224d: Deep Learning for Natural Language Processing by Richard Socher http://cs224d.stanford.edu/
Machine Learning by Andrew Ng https://www.coursera.org/learn/machine-learning
Neural Networks for Machine Learning by Geoffrey Hinton https://www.coursera.org/course/neuralnets
CS231n: Convolutional Neural Networks for Visual Recognition by Andrej Karpathy http://cs231n.github.io/
![Page 48: Introduction to Deep Learning (Dmytro Fishman Technology Stream)](https://reader031.fdocuments.in/reader031/viewer/2022021422/5882198a1a28ab3f4c8b61e1/html5/thumbnails/48.jpg)
Offline placesComputer Science Bachelor’s program at CS@UCU http://cs.ucu.edu.ua/en/
Applied Computer Science Stream at Summer School AACIMP (Kyiv) http://summerschool.ssa.org.ua/
The Deep Learning Summer School at Montreal, Canada https://sites.google.com/site/deeplearningsummerschool/
Computer Science Master’s program at Tartu, Estonia https://sites.google.com/site/deeplearningsummerschool/
International Conference on Machine Learning at New York, USA http://icml.cc/2016/
![Page 49: Introduction to Deep Learning (Dmytro Fishman Technology Stream)](https://reader031.fdocuments.in/reader031/viewer/2022021422/5882198a1a28ab3f4c8b61e1/html5/thumbnails/49.jpg)
Popular Blogs10 Machine Learning Terms Explained in Simple English by Aylien http://blog.aylien.com/post/121281850733/10-machine-learning-terms-explained-in-simple
The Unreasonable Effectiveness of Recurrent Neural Networks by Andrej Karpathy http://karpathy.github.io/2015/05/21/rnn-effectiveness/
A Neural Network in 11 lines of Python by iamtrask http://iamtrask.github.io/2015/07/12/basic-python-network/
Understand LSTM Networks by Christopher Olah https://colah.github.io/posts/2015-08-Understanding-LSTMs/
Sign up for Data Elixir mailing list: http://dataelixir.com/
![Page 50: Introduction to Deep Learning (Dmytro Fishman Technology Stream)](https://reader031.fdocuments.in/reader031/viewer/2022021422/5882198a1a28ab3f4c8b61e1/html5/thumbnails/50.jpg)
KagglePlatform for data science competitions
Sponsored
Research
Public competitions
![Page 51: Introduction to Deep Learning (Dmytro Fishman Technology Stream)](https://reader031.fdocuments.in/reader031/viewer/2022021422/5882198a1a28ab3f4c8b61e1/html5/thumbnails/51.jpg)
Computers are useless. They can only give answers.
Pablo Picasso