A machine learning perspective on neural networks and learning tools
description
Transcript of A machine learning perspective on neural networks and learning tools
![Page 1: A machine learning perspective on neural networks and learning tools](https://reader033.fdocuments.in/reader033/viewer/2022061615/56816628550346895dd98612/html5/thumbnails/1.jpg)
A machine learning perspective on neural networks and learning tools
Tom Schaul
![Page 2: A machine learning perspective on neural networks and learning tools](https://reader033.fdocuments.in/reader033/viewer/2022061615/56816628550346895dd98612/html5/thumbnails/2.jpg)
2
Overview
PyBrain: training artificial neural networks for classification, (sequence) prediction and control
1. Neural networks– Modular structure– Available architectures
2. Training– Supervised learning– Optimization– Reinforcement learning (RL)
4th FACETS CodeJam Workshop - Tom Schaul - PyBrain
![Page 3: A machine learning perspective on neural networks and learning tools](https://reader033.fdocuments.in/reader033/viewer/2022061615/56816628550346895dd98612/html5/thumbnails/3.jpg)
3
Disclaimer
• Only version 0.3, you may encounter– inconsistencies– bugs– undocumented “features”
• But growing– 10+ contributors– 100+ followers (github, mailing list)– 1000+ downloads
4th FACETS CodeJam Workshop - Tom Schaul - PyBrain
![Page 4: A machine learning perspective on neural networks and learning tools](https://reader033.fdocuments.in/reader033/viewer/2022061615/56816628550346895dd98612/html5/thumbnails/4.jpg)
4
(Our) Neural Networks
• No spikes• Continuous activations• Discrete time steps
4th FACETS CodeJam Workshop - Tom Schaul - PyBrain
![Page 5: A machine learning perspective on neural networks and learning tools](https://reader033.fdocuments.in/reader033/viewer/2022061615/56816628550346895dd98612/html5/thumbnails/5.jpg)
5
Network Structure: Modules
4th FACETS CodeJam Workshop - Tom Schaul - PyBrain
Module
input
output
Parametersparameters
input error
output error
Derivativesderivatives
![Page 6: A machine learning perspective on neural networks and learning tools](https://reader033.fdocuments.in/reader033/viewer/2022061615/56816628550346895dd98612/html5/thumbnails/6.jpg)
6
Network Structure: Connections
4th FACETS CodeJam Workshop - Tom Schaul - PyBrain
Module
input
output
input error
output error
Module
input
output
input error
output error
Module
input
output
input error
output error
FullConnection
![Page 7: A machine learning perspective on neural networks and learning tools](https://reader033.fdocuments.in/reader033/viewer/2022061615/56816628550346895dd98612/html5/thumbnails/7.jpg)
7
Network Structure:Graphs, Recurrency, Nesting
4th FACETS CodeJam Workshop - Tom Schaul - PyBrain
Module
Module Module
Module
Module Module
![Page 8: A machine learning perspective on neural networks and learning tools](https://reader033.fdocuments.in/reader033/viewer/2022061615/56816628550346895dd98612/html5/thumbnails/8.jpg)
8
Network Components: Modules
• Module types– layers of neurons• additive or multiplicative• sigmoidal squashing functions• stochastic outputs
– gate units– memory cells (e.g. LSTM cells)– …
4th FACETS CodeJam Workshop - Tom Schaul - PyBrain
![Page 9: A machine learning perspective on neural networks and learning tools](https://reader033.fdocuments.in/reader033/viewer/2022061615/56816628550346895dd98612/html5/thumbnails/9.jpg)
9
Network Components: Connections
• Connection – Fully connected or sparse– Time-recurrent– Weight-sharing– may contain parameters– …
4th FACETS CodeJam Workshop - Tom Schaul - PyBrain
![Page 10: A machine learning perspective on neural networks and learning tools](https://reader033.fdocuments.in/reader033/viewer/2022061615/56816628550346895dd98612/html5/thumbnails/10.jpg)
10
Network Architectures
• Feed-forward networks, including– Deep Belief Nets– Restricted Boltzmann Machines (RBM)
• Recurrent networks, including– Reservoirs (Echo State networks)– Bidirectional networks– Long Short-Term Memory (LSTM) architectures– Multi-Dimensional Recurrent Networks (MDRNN)
• Custom-designed topologies4th FACETS CodeJam Workshop - Tom Schaul - PyBrain
![Page 11: A machine learning perspective on neural networks and learning tools](https://reader033.fdocuments.in/reader033/viewer/2022061615/56816628550346895dd98612/html5/thumbnails/11.jpg)
11
Overview
1. Neural networks– Modular structure– Available architectures
2. Training– Supervised learning– Optimization– Reinforcement learning (RL)
4th FACETS CodeJam Workshop - Tom Schaul - PyBrain
![Page 12: A machine learning perspective on neural networks and learning tools](https://reader033.fdocuments.in/reader033/viewer/2022061615/56816628550346895dd98612/html5/thumbnails/12.jpg)
12
Training: Supervised Learning
4th FACETS CodeJam Workshop - Tom Schaul - PyBrain
Module
input
output
Parametersparameters
input error
output error
Derivativesderivatives
compare to target
gradientupdate on
parameters
Backpropagation
![Page 13: A machine learning perspective on neural networks and learning tools](https://reader033.fdocuments.in/reader033/viewer/2022061615/56816628550346895dd98612/html5/thumbnails/13.jpg)
13
Training: Black-box Optimization
• fitness function based on e.g. MSE, accuracy, rewards
• multiple fitness values: multi-objective optimization
4th FACETS CodeJam Workshop - Tom Schaul - PyBrain
Black box
update parameters
fitness
Parametersparameters
BlackBoxOptimizer
![Page 14: A machine learning perspective on neural networks and learning tools](https://reader033.fdocuments.in/reader033/viewer/2022061615/56816628550346895dd98612/html5/thumbnails/14.jpg)
14
Optimization Algorithms
• (Stochastic) Hill-climbing• Particle Swarm Optimization (PSO)• (Natural) Evolution Strategies (ES)• Covariance Matrix Adaptation (CMA)• Genetic Algorithms (GA)• Co-evolution• Multi-Objective Optimization (NSGA-II)• …
4th FACETS CodeJam Workshop - Tom Schaul - PyBrain
![Page 15: A machine learning perspective on neural networks and learning tools](https://reader033.fdocuments.in/reader033/viewer/2022061615/56816628550346895dd98612/html5/thumbnails/15.jpg)
15
Training: Reinforcement Learning
4th FACETS CodeJam Workshop - Tom Schaul - PyBrain
Agent
action observationreward
Environment
stateaction
EnvironmentTask
Expe
rimen
t
![Page 16: A machine learning perspective on neural networks and learning tools](https://reader033.fdocuments.in/reader033/viewer/2022061615/56816628550346895dd98612/html5/thumbnails/16.jpg)
16
RL: Agents, Learners, Exploration
4th FACETS CodeJam Workshop - Tom Schaul - PyBrain
action observationreward
LearningAgent
Module
Learner
DataSetExplorer
![Page 17: A machine learning perspective on neural networks and learning tools](https://reader033.fdocuments.in/reader033/viewer/2022061615/56816628550346895dd98612/html5/thumbnails/17.jpg)
17
RL: Learning Algorithms and Exploration
• Value-based RL– Q-Learning, SARSA– Fitted-Q Iteration
• Policy Gradient RL– REINFORCE– Natural Actor-Critic
• Exploration methods– Epsilon-Greedy– Boltzmann– State-Dependent Exploration
4th FACETS CodeJam Workshop - Tom Schaul - PyBrain
![Page 18: A machine learning perspective on neural networks and learning tools](https://reader033.fdocuments.in/reader033/viewer/2022061615/56816628550346895dd98612/html5/thumbnails/18.jpg)
18
RL: Environments and Tasks
• 2D Mazes (MDP / POMDP)• Pole balancing
• 3D environments (ODE, FlexCube)
• Board games (e.g. Atari-Go, Pente)
4th FACETS CodeJam Workshop - Tom Schaul - PyBrain
![Page 19: A machine learning perspective on neural networks and learning tools](https://reader033.fdocuments.in/reader033/viewer/2022061615/56816628550346895dd98612/html5/thumbnails/19.jpg)
19
Also in PyBrain
• Unsupervised learning and preprocessing• Support Vector Machines (through LIBSVM)• Tools– Plotting / Visualization– netCDF support– XML read/write support
• arac: fast C version
4th FACETS CodeJam Workshop - Tom Schaul - PyBrain
![Page 20: A machine learning perspective on neural networks and learning tools](https://reader033.fdocuments.in/reader033/viewer/2022061615/56816628550346895dd98612/html5/thumbnails/20.jpg)
20
References
• Source download, documentationwww.pybrain.org
• Mailing list (200+ members)groups.google.com/group/pybrain
• Feature requestsgithub.com/pybrain/pybrain/issues
• CitationT. Schaul, J. Bayer, D. Wierstra, Y. Sun, M. Felder, F. Sehnke, T. Rückstieß and J. Schmidhuber. PyBrain. Journal of Machine Learning Research, 2010.
4th FACETS CodeJam Workshop - Tom Schaul - PyBrain
![Page 21: A machine learning perspective on neural networks and learning tools](https://reader033.fdocuments.in/reader033/viewer/2022061615/56816628550346895dd98612/html5/thumbnails/21.jpg)
21
Acknowledgements
Justin BayerMartin FelderThomas RückstiessFrank SehnkeDaan Wierstra
and many more who contributed code, suggestions, bug fixes …
… and you for your attention!
4th FACETS CodeJam Workshop - Tom Schaul - PyBrain