A Brief Introduction to Deep Learningdcor/Graphics/pdf.slides/YY-Deep Learning.pdfInternational...
Transcript of A Brief Introduction to Deep Learningdcor/Graphics/pdf.slides/YY-Deep Learning.pdfInternational...
![Page 1: A Brief Introduction to Deep Learningdcor/Graphics/pdf.slides/YY-Deep Learning.pdfInternational Conference on Artificial Intelligence and Statistics. 2011. • Hinton, Geoffrey E.,](https://reader030.fdocuments.in/reader030/viewer/2022040522/5e7eeec32438c90b4b58513b/html5/thumbnails/1.jpg)
A Brief Introduction to Deep Learning
--Yangyan Li
![Page 2: A Brief Introduction to Deep Learningdcor/Graphics/pdf.slides/YY-Deep Learning.pdfInternational Conference on Artificial Intelligence and Statistics. 2011. • Hinton, Geoffrey E.,](https://reader030.fdocuments.in/reader030/viewer/2022040522/5e7eeec32438c90b4b58513b/html5/thumbnails/2.jpg)
How would you crack it?
![Page 3: A Brief Introduction to Deep Learningdcor/Graphics/pdf.slides/YY-Deep Learning.pdfInternational Conference on Artificial Intelligence and Statistics. 2011. • Hinton, Geoffrey E.,](https://reader030.fdocuments.in/reader030/viewer/2022040522/5e7eeec32438c90b4b58513b/html5/thumbnails/3.jpg)
How to avoid being cracked?
![Page 4: A Brief Introduction to Deep Learningdcor/Graphics/pdf.slides/YY-Deep Learning.pdfInternational Conference on Artificial Intelligence and Statistics. 2011. • Hinton, Geoffrey E.,](https://reader030.fdocuments.in/reader030/viewer/2022040522/5e7eeec32438c90b4b58513b/html5/thumbnails/4.jpg)
Seam Carving!
![Page 5: A Brief Introduction to Deep Learningdcor/Graphics/pdf.slides/YY-Deep Learning.pdfInternational Conference on Artificial Intelligence and Statistics. 2011. • Hinton, Geoffrey E.,](https://reader030.fdocuments.in/reader030/viewer/2022040522/5e7eeec32438c90b4b58513b/html5/thumbnails/5.jpg)
![Page 6: A Brief Introduction to Deep Learningdcor/Graphics/pdf.slides/YY-Deep Learning.pdfInternational Conference on Artificial Intelligence and Statistics. 2011. • Hinton, Geoffrey E.,](https://reader030.fdocuments.in/reader030/viewer/2022040522/5e7eeec32438c90b4b58513b/html5/thumbnails/6.jpg)
Labradoodle or fried chicken
![Page 7: A Brief Introduction to Deep Learningdcor/Graphics/pdf.slides/YY-Deep Learning.pdfInternational Conference on Artificial Intelligence and Statistics. 2011. • Hinton, Geoffrey E.,](https://reader030.fdocuments.in/reader030/viewer/2022040522/5e7eeec32438c90b4b58513b/html5/thumbnails/7.jpg)
Puppy or bagel
![Page 8: A Brief Introduction to Deep Learningdcor/Graphics/pdf.slides/YY-Deep Learning.pdfInternational Conference on Artificial Intelligence and Statistics. 2011. • Hinton, Geoffrey E.,](https://reader030.fdocuments.in/reader030/viewer/2022040522/5e7eeec32438c90b4b58513b/html5/thumbnails/8.jpg)
Sheepdog or mop
![Page 9: A Brief Introduction to Deep Learningdcor/Graphics/pdf.slides/YY-Deep Learning.pdfInternational Conference on Artificial Intelligence and Statistics. 2011. • Hinton, Geoffrey E.,](https://reader030.fdocuments.in/reader030/viewer/2022040522/5e7eeec32438c90b4b58513b/html5/thumbnails/9.jpg)
Chihuahua or muffin
![Page 10: A Brief Introduction to Deep Learningdcor/Graphics/pdf.slides/YY-Deep Learning.pdfInternational Conference on Artificial Intelligence and Statistics. 2011. • Hinton, Geoffrey E.,](https://reader030.fdocuments.in/reader030/viewer/2022040522/5e7eeec32438c90b4b58513b/html5/thumbnails/10.jpg)
Barn owl or apple
![Page 11: A Brief Introduction to Deep Learningdcor/Graphics/pdf.slides/YY-Deep Learning.pdfInternational Conference on Artificial Intelligence and Statistics. 2011. • Hinton, Geoffrey E.,](https://reader030.fdocuments.in/reader030/viewer/2022040522/5e7eeec32438c90b4b58513b/html5/thumbnails/11.jpg)
Parrot or guacamole
![Page 12: A Brief Introduction to Deep Learningdcor/Graphics/pdf.slides/YY-Deep Learning.pdfInternational Conference on Artificial Intelligence and Statistics. 2011. • Hinton, Geoffrey E.,](https://reader030.fdocuments.in/reader030/viewer/2022040522/5e7eeec32438c90b4b58513b/html5/thumbnails/12.jpg)
Raw chicken or Donald Trump
![Page 13: A Brief Introduction to Deep Learningdcor/Graphics/pdf.slides/YY-Deep Learning.pdfInternational Conference on Artificial Intelligence and Statistics. 2011. • Hinton, Geoffrey E.,](https://reader030.fdocuments.in/reader030/viewer/2022040522/5e7eeec32438c90b4b58513b/html5/thumbnails/13.jpg)
But, we human actually lose!
• A demo that shows we, human, lose, on the classification task, we are proud of, we have been
trained for millions of years!
• If we want to make it hard for bots, it has to be hard for human as well.
![Page 14: A Brief Introduction to Deep Learningdcor/Graphics/pdf.slides/YY-Deep Learning.pdfInternational Conference on Artificial Intelligence and Statistics. 2011. • Hinton, Geoffrey E.,](https://reader030.fdocuments.in/reader030/viewer/2022040522/5e7eeec32438c90b4b58513b/html5/thumbnails/14.jpg)
How would you crack it?
![Page 15: A Brief Introduction to Deep Learningdcor/Graphics/pdf.slides/YY-Deep Learning.pdfInternational Conference on Artificial Intelligence and Statistics. 2011. • Hinton, Geoffrey E.,](https://reader030.fdocuments.in/reader030/viewer/2022040522/5e7eeec32438c90b4b58513b/html5/thumbnails/15.jpg)
We human lose on Go!
![Page 16: A Brief Introduction to Deep Learningdcor/Graphics/pdf.slides/YY-Deep Learning.pdfInternational Conference on Artificial Intelligence and Statistics. 2011. • Hinton, Geoffrey E.,](https://reader030.fdocuments.in/reader030/viewer/2022040522/5e7eeec32438c90b4b58513b/html5/thumbnails/16.jpg)
![Page 17: A Brief Introduction to Deep Learningdcor/Graphics/pdf.slides/YY-Deep Learning.pdfInternational Conference on Artificial Intelligence and Statistics. 2011. • Hinton, Geoffrey E.,](https://reader030.fdocuments.in/reader030/viewer/2022040522/5e7eeec32438c90b4b58513b/html5/thumbnails/17.jpg)
We (will) lose on many specific tasks!
• Speech recognition
• Translation
• Self-driving
• …
• BUT, they are not AI yet…
• Don’t worry until it dates with your girl/boy friend…
![Page 18: A Brief Introduction to Deep Learningdcor/Graphics/pdf.slides/YY-Deep Learning.pdfInternational Conference on Artificial Intelligence and Statistics. 2011. • Hinton, Geoffrey E.,](https://reader030.fdocuments.in/reader030/viewer/2022040522/5e7eeec32438c90b4b58513b/html5/thumbnails/18.jpg)
Deep learning is so cool for so many problems…
![Page 19: A Brief Introduction to Deep Learningdcor/Graphics/pdf.slides/YY-Deep Learning.pdfInternational Conference on Artificial Intelligence and Statistics. 2011. • Hinton, Geoffrey E.,](https://reader030.fdocuments.in/reader030/viewer/2022040522/5e7eeec32438c90b4b58513b/html5/thumbnails/19.jpg)
![Page 20: A Brief Introduction to Deep Learningdcor/Graphics/pdf.slides/YY-Deep Learning.pdfInternational Conference on Artificial Intelligence and Statistics. 2011. • Hinton, Geoffrey E.,](https://reader030.fdocuments.in/reader030/viewer/2022040522/5e7eeec32438c90b4b58513b/html5/thumbnails/20.jpg)
![Page 21: A Brief Introduction to Deep Learningdcor/Graphics/pdf.slides/YY-Deep Learning.pdfInternational Conference on Artificial Intelligence and Statistics. 2011. • Hinton, Geoffrey E.,](https://reader030.fdocuments.in/reader030/viewer/2022040522/5e7eeec32438c90b4b58513b/html5/thumbnails/21.jpg)
A Brief Introduction to Deep Learning
• Artificial Neural Network
• Back-propagation
• Fully Connected Layer
• Convolutional Layer
• Overfitting
![Page 22: A Brief Introduction to Deep Learningdcor/Graphics/pdf.slides/YY-Deep Learning.pdfInternational Conference on Artificial Intelligence and Statistics. 2011. • Hinton, Geoffrey E.,](https://reader030.fdocuments.in/reader030/viewer/2022040522/5e7eeec32438c90b4b58513b/html5/thumbnails/22.jpg)
Artificial Neural Network
1. Activation function 2. Weights 3. Cost function 4. Learning algorithm
Live Demo
![Page 23: A Brief Introduction to Deep Learningdcor/Graphics/pdf.slides/YY-Deep Learning.pdfInternational Conference on Artificial Intelligence and Statistics. 2011. • Hinton, Geoffrey E.,](https://reader030.fdocuments.in/reader030/viewer/2022040522/5e7eeec32438c90b4b58513b/html5/thumbnails/23.jpg)
Neurons are functions
![Page 24: A Brief Introduction to Deep Learningdcor/Graphics/pdf.slides/YY-Deep Learning.pdfInternational Conference on Artificial Intelligence and Statistics. 2011. • Hinton, Geoffrey E.,](https://reader030.fdocuments.in/reader030/viewer/2022040522/5e7eeec32438c90b4b58513b/html5/thumbnails/24.jpg)
Neurons are functions
![Page 25: A Brief Introduction to Deep Learningdcor/Graphics/pdf.slides/YY-Deep Learning.pdfInternational Conference on Artificial Intelligence and Statistics. 2011. • Hinton, Geoffrey E.,](https://reader030.fdocuments.in/reader030/viewer/2022040522/5e7eeec32438c90b4b58513b/html5/thumbnails/25.jpg)
Back-propagation
![Page 26: A Brief Introduction to Deep Learningdcor/Graphics/pdf.slides/YY-Deep Learning.pdfInternational Conference on Artificial Intelligence and Statistics. 2011. • Hinton, Geoffrey E.,](https://reader030.fdocuments.in/reader030/viewer/2022040522/5e7eeec32438c90b4b58513b/html5/thumbnails/26.jpg)
Now, serious stuff, a bit…
![Page 27: A Brief Introduction to Deep Learningdcor/Graphics/pdf.slides/YY-Deep Learning.pdfInternational Conference on Artificial Intelligence and Statistics. 2011. • Hinton, Geoffrey E.,](https://reader030.fdocuments.in/reader030/viewer/2022040522/5e7eeec32438c90b4b58513b/html5/thumbnails/27.jpg)
Fully Connected Layers
![Page 28: A Brief Introduction to Deep Learningdcor/Graphics/pdf.slides/YY-Deep Learning.pdfInternational Conference on Artificial Intelligence and Statistics. 2011. • Hinton, Geoffrey E.,](https://reader030.fdocuments.in/reader030/viewer/2022040522/5e7eeec32438c90b4b58513b/html5/thumbnails/28.jpg)
“When in doubt, use brute force.” --Ken Thompson
![Page 29: A Brief Introduction to Deep Learningdcor/Graphics/pdf.slides/YY-Deep Learning.pdfInternational Conference on Artificial Intelligence and Statistics. 2011. • Hinton, Geoffrey E.,](https://reader030.fdocuments.in/reader030/viewer/2022040522/5e7eeec32438c90b4b58513b/html5/thumbnails/29.jpg)
“If brute force is possible...” --Yangyan Li
![Page 30: A Brief Introduction to Deep Learningdcor/Graphics/pdf.slides/YY-Deep Learning.pdfInternational Conference on Artificial Intelligence and Statistics. 2011. • Hinton, Geoffrey E.,](https://reader030.fdocuments.in/reader030/viewer/2022040522/5e7eeec32438c90b4b58513b/html5/thumbnails/30.jpg)
Convolutional Layers
![Page 31: A Brief Introduction to Deep Learningdcor/Graphics/pdf.slides/YY-Deep Learning.pdfInternational Conference on Artificial Intelligence and Statistics. 2011. • Hinton, Geoffrey E.,](https://reader030.fdocuments.in/reader030/viewer/2022040522/5e7eeec32438c90b4b58513b/html5/thumbnails/31.jpg)
Convolutional Layers
![Page 32: A Brief Introduction to Deep Learningdcor/Graphics/pdf.slides/YY-Deep Learning.pdfInternational Conference on Artificial Intelligence and Statistics. 2011. • Hinton, Geoffrey E.,](https://reader030.fdocuments.in/reader030/viewer/2022040522/5e7eeec32438c90b4b58513b/html5/thumbnails/32.jpg)
Convolution Filters
![Page 33: A Brief Introduction to Deep Learningdcor/Graphics/pdf.slides/YY-Deep Learning.pdfInternational Conference on Artificial Intelligence and Statistics. 2011. • Hinton, Geoffrey E.,](https://reader030.fdocuments.in/reader030/viewer/2022040522/5e7eeec32438c90b4b58513b/html5/thumbnails/33.jpg)
![Page 34: A Brief Introduction to Deep Learningdcor/Graphics/pdf.slides/YY-Deep Learning.pdfInternational Conference on Artificial Intelligence and Statistics. 2011. • Hinton, Geoffrey E.,](https://reader030.fdocuments.in/reader030/viewer/2022040522/5e7eeec32438c90b4b58513b/html5/thumbnails/34.jpg)
![Page 35: A Brief Introduction to Deep Learningdcor/Graphics/pdf.slides/YY-Deep Learning.pdfInternational Conference on Artificial Intelligence and Statistics. 2011. • Hinton, Geoffrey E.,](https://reader030.fdocuments.in/reader030/viewer/2022040522/5e7eeec32438c90b4b58513b/html5/thumbnails/35.jpg)
Feature Engineering vs. Learning
• Feature engineering is the process of using domain knowledge of the data to create features that make machine learning algorithms work.
• “When working on a machine learning problem, feature engineering is manually designing what the input x's should be.”
-- Shayne Miel
• “Coming up with features is difficult, time-consuming, requires expert knowledge.”
--Andrew Ng
![Page 36: A Brief Introduction to Deep Learningdcor/Graphics/pdf.slides/YY-Deep Learning.pdfInternational Conference on Artificial Intelligence and Statistics. 2011. • Hinton, Geoffrey E.,](https://reader030.fdocuments.in/reader030/viewer/2022040522/5e7eeec32438c90b4b58513b/html5/thumbnails/36.jpg)
![Page 37: A Brief Introduction to Deep Learningdcor/Graphics/pdf.slides/YY-Deep Learning.pdfInternational Conference on Artificial Intelligence and Statistics. 2011. • Hinton, Geoffrey E.,](https://reader030.fdocuments.in/reader030/viewer/2022040522/5e7eeec32438c90b4b58513b/html5/thumbnails/37.jpg)
![Page 38: A Brief Introduction to Deep Learningdcor/Graphics/pdf.slides/YY-Deep Learning.pdfInternational Conference on Artificial Intelligence and Statistics. 2011. • Hinton, Geoffrey E.,](https://reader030.fdocuments.in/reader030/viewer/2022040522/5e7eeec32438c90b4b58513b/html5/thumbnails/38.jpg)
How to detect it in training process?
![Page 39: A Brief Introduction to Deep Learningdcor/Graphics/pdf.slides/YY-Deep Learning.pdfInternational Conference on Artificial Intelligence and Statistics. 2011. • Hinton, Geoffrey E.,](https://reader030.fdocuments.in/reader030/viewer/2022040522/5e7eeec32438c90b4b58513b/html5/thumbnails/39.jpg)
Dropout
![Page 40: A Brief Introduction to Deep Learningdcor/Graphics/pdf.slides/YY-Deep Learning.pdfInternational Conference on Artificial Intelligence and Statistics. 2011. • Hinton, Geoffrey E.,](https://reader030.fdocuments.in/reader030/viewer/2022040522/5e7eeec32438c90b4b58513b/html5/thumbnails/40.jpg)
Sigmod ReLU
![Page 41: A Brief Introduction to Deep Learningdcor/Graphics/pdf.slides/YY-Deep Learning.pdfInternational Conference on Artificial Intelligence and Statistics. 2011. • Hinton, Geoffrey E.,](https://reader030.fdocuments.in/reader030/viewer/2022040522/5e7eeec32438c90b4b58513b/html5/thumbnails/41.jpg)
Sigmod ReLU
![Page 42: A Brief Introduction to Deep Learningdcor/Graphics/pdf.slides/YY-Deep Learning.pdfInternational Conference on Artificial Intelligence and Statistics. 2011. • Hinton, Geoffrey E.,](https://reader030.fdocuments.in/reader030/viewer/2022040522/5e7eeec32438c90b4b58513b/html5/thumbnails/42.jpg)
Compute, connect, evaluate, correct, train madly…
Non-linearity, distributed representation, parallel computation, adaptive, self-organizing…
![Page 43: A Brief Introduction to Deep Learningdcor/Graphics/pdf.slides/YY-Deep Learning.pdfInternational Conference on Artificial Intelligence and Statistics. 2011. • Hinton, Geoffrey E.,](https://reader030.fdocuments.in/reader030/viewer/2022040522/5e7eeec32438c90b4b58513b/html5/thumbnails/43.jpg)
A brief history • McCulloch, Warren S., and Walter Pitts. "A logical calculus of the ideas immanent in nervous
activity." The bulletin of mathematical biophysics 5.4 (1943): 115-133.
• Rosenblatt, Frank. "The perceptron: a probabilistic model for information storage and organization in the brain." Psychological review 65.6 (1958): 386.
• Rumelhart, David E., Geoffrey E. Hinton, and Ronald J. Williams. "Learning representations by back-propagating errors." Cognitive modeling 5.3 (1988): 1.
• LeCun, Yann, et al. "Backpropagation applied to handwritten zip code recognition." Neural computation 1.4 (1989): 541-551.
• 1993: Nvidia started…
• Hinton, Geoffrey E., Simon Osindero, and Yee-Whye Teh. "A fast learning algorithm for deep belief nets." Neural computation 18.7 (2006): 1527-1554.
• Raina, Rajat, Anand Madhavan, and Andrew Y. Ng. "Large-scale deep unsupervised learning using graphics processors." Proceedings of the 26th annual international conference on machine learning. ACM, 2009.
• Deng, Jia, et al. "Imagenet: A large-scale hierarchical image database."Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on. IEEE, 2009.
• 2010: “GPUS ARE ONLY UP TO 14 TIMES FASTER THAN CPUS” SAYS INTEL –Nvidia
• Glorot, Xavier, Antoine Bordes, and Yoshua Bengio. "Deep sparse rectifier neural networks." International Conference on Artificial Intelligence and Statistics. 2011.
• Hinton, Geoffrey E., et al. "Improving neural networks by preventing co-adaptation of feature detectors." arXiv preprint arXiv:1207.0580 (2012).
• Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E. Hinton. "Imagenet classification with deep convolutional neural networks." Advances in neural information processing systems. 2012.
![Page 44: A Brief Introduction to Deep Learningdcor/Graphics/pdf.slides/YY-Deep Learning.pdfInternational Conference on Artificial Intelligence and Statistics. 2011. • Hinton, Geoffrey E.,](https://reader030.fdocuments.in/reader030/viewer/2022040522/5e7eeec32438c90b4b58513b/html5/thumbnails/44.jpg)
“Now this is not the end. It is not even the beginning of the end. But it is, perhaps, the end of the beginning.”
--Winston Churchill
![Page 45: A Brief Introduction to Deep Learningdcor/Graphics/pdf.slides/YY-Deep Learning.pdfInternational Conference on Artificial Intelligence and Statistics. 2011. • Hinton, Geoffrey E.,](https://reader030.fdocuments.in/reader030/viewer/2022040522/5e7eeec32438c90b4b58513b/html5/thumbnails/45.jpg)
Is Deep Learning Taking Over the World?
• What applications are likely/unlikely to benefit from DL? Why?
![Page 46: A Brief Introduction to Deep Learningdcor/Graphics/pdf.slides/YY-Deep Learning.pdfInternational Conference on Artificial Intelligence and Statistics. 2011. • Hinton, Geoffrey E.,](https://reader030.fdocuments.in/reader030/viewer/2022040522/5e7eeec32438c90b4b58513b/html5/thumbnails/46.jpg)
Deep learning, yay or nay?
A piece of cake, elementary math…
It eats, a lot!