NEURAL NETWORKS,CELLULAR NEURAL NETWORKS AND ADAPTIVE FUZZY FILTERS
A Gentle Introduction to Neural Networks - Agile on the Beach...A Gentle Introduction to Neural...
Transcript of A Gentle Introduction to Neural Networks - Agile on the Beach...A Gentle Introduction to Neural...
![Page 1: A Gentle Introduction to Neural Networks - Agile on the Beach...A Gentle Introduction to Neural Networks (with Python) Tariq Rashid @rzeta0 July 2018 Background Ideas DIY Handwriting](https://reader035.fdocuments.in/reader035/viewer/2022081407/604ac5d1b8491d452f03115b/html5/thumbnails/1.jpg)
A Gentle Introductionto Neural Networks(with Python)
Tariq Rashid @rzeta0
July 2018
![Page 2: A Gentle Introduction to Neural Networks - Agile on the Beach...A Gentle Introduction to Neural Networks (with Python) Tariq Rashid @rzeta0 July 2018 Background Ideas DIY Handwriting](https://reader035.fdocuments.in/reader035/viewer/2022081407/604ac5d1b8491d452f03115b/html5/thumbnails/2.jpg)
BackgroundIdeasDIYHandwriting
… and a live demo!
![Page 3: A Gentle Introduction to Neural Networks - Agile on the Beach...A Gentle Introduction to Neural Networks (with Python) Tariq Rashid @rzeta0 July 2018 Background Ideas DIY Handwriting](https://reader035.fdocuments.in/reader035/viewer/2022081407/604ac5d1b8491d452f03115b/html5/thumbnails/3.jpg)
Background
![Page 4: A Gentle Introduction to Neural Networks - Agile on the Beach...A Gentle Introduction to Neural Networks (with Python) Tariq Rashid @rzeta0 July 2018 Background Ideas DIY Handwriting](https://reader035.fdocuments.in/reader035/viewer/2022081407/604ac5d1b8491d452f03115b/html5/thumbnails/4.jpg)
Start With Two Questions
2403343781289312+ 2843033712837981+ 2362142787897881+ 3256541312323213+ 9864479802118978+ 8976677987987897+ 8981257890087988
= ?
locate people in this photo add these numbers
![Page 5: A Gentle Introduction to Neural Networks - Agile on the Beach...A Gentle Introduction to Neural Networks (with Python) Tariq Rashid @rzeta0 July 2018 Background Ideas DIY Handwriting](https://reader035.fdocuments.in/reader035/viewer/2022081407/604ac5d1b8491d452f03115b/html5/thumbnails/5.jpg)
AI is Huge!
![Page 6: A Gentle Introduction to Neural Networks - Agile on the Beach...A Gentle Introduction to Neural Networks (with Python) Tariq Rashid @rzeta0 July 2018 Background Ideas DIY Handwriting](https://reader035.fdocuments.in/reader035/viewer/2022081407/604ac5d1b8491d452f03115b/html5/thumbnails/6.jpg)
Google’s and Go
![Page 7: A Gentle Introduction to Neural Networks - Agile on the Beach...A Gentle Introduction to Neural Networks (with Python) Tariq Rashid @rzeta0 July 2018 Background Ideas DIY Handwriting](https://reader035.fdocuments.in/reader035/viewer/2022081407/604ac5d1b8491d452f03115b/html5/thumbnails/7.jpg)
Ideas
![Page 8: A Gentle Introduction to Neural Networks - Agile on the Beach...A Gentle Introduction to Neural Networks (with Python) Tariq Rashid @rzeta0 July 2018 Background Ideas DIY Handwriting](https://reader035.fdocuments.in/reader035/viewer/2022081407/604ac5d1b8491d452f03115b/html5/thumbnails/8.jpg)
Simple Predicting Machine
![Page 9: A Gentle Introduction to Neural Networks - Agile on the Beach...A Gentle Introduction to Neural Networks (with Python) Tariq Rashid @rzeta0 July 2018 Background Ideas DIY Handwriting](https://reader035.fdocuments.in/reader035/viewer/2022081407/604ac5d1b8491d452f03115b/html5/thumbnails/9.jpg)
Simple Predicting Machine
![Page 10: A Gentle Introduction to Neural Networks - Agile on the Beach...A Gentle Introduction to Neural Networks (with Python) Tariq Rashid @rzeta0 July 2018 Background Ideas DIY Handwriting](https://reader035.fdocuments.in/reader035/viewer/2022081407/604ac5d1b8491d452f03115b/html5/thumbnails/10.jpg)
Kilometres to Miles
try a model - this one is linear
random starting parameter
![Page 11: A Gentle Introduction to Neural Networks - Agile on the Beach...A Gentle Introduction to Neural Networks (with Python) Tariq Rashid @rzeta0 July 2018 Background Ideas DIY Handwriting](https://reader035.fdocuments.in/reader035/viewer/2022081407/604ac5d1b8491d452f03115b/html5/thumbnails/11.jpg)
Kilometres to Miles
not great
![Page 12: A Gentle Introduction to Neural Networks - Agile on the Beach...A Gentle Introduction to Neural Networks (with Python) Tariq Rashid @rzeta0 July 2018 Background Ideas DIY Handwriting](https://reader035.fdocuments.in/reader035/viewer/2022081407/604ac5d1b8491d452f03115b/html5/thumbnails/12.jpg)
Kilometres to Miles
better
![Page 13: A Gentle Introduction to Neural Networks - Agile on the Beach...A Gentle Introduction to Neural Networks (with Python) Tariq Rashid @rzeta0 July 2018 Background Ideas DIY Handwriting](https://reader035.fdocuments.in/reader035/viewer/2022081407/604ac5d1b8491d452f03115b/html5/thumbnails/13.jpg)
Kilometres to Miles
worse !
![Page 14: A Gentle Introduction to Neural Networks - Agile on the Beach...A Gentle Introduction to Neural Networks (with Python) Tariq Rashid @rzeta0 July 2018 Background Ideas DIY Handwriting](https://reader035.fdocuments.in/reader035/viewer/2022081407/604ac5d1b8491d452f03115b/html5/thumbnails/14.jpg)
Kilometres to Miles
best yet !
![Page 15: A Gentle Introduction to Neural Networks - Agile on the Beach...A Gentle Introduction to Neural Networks (with Python) Tariq Rashid @rzeta0 July 2018 Background Ideas DIY Handwriting](https://reader035.fdocuments.in/reader035/viewer/2022081407/604ac5d1b8491d452f03115b/html5/thumbnails/15.jpg)
Key Points
1. Don’t know how something works exactly? Try a
model with adjustable parameters.
2. Use the error to refine the parameters.
![Page 16: A Gentle Introduction to Neural Networks - Agile on the Beach...A Gentle Introduction to Neural Networks (with Python) Tariq Rashid @rzeta0 July 2018 Background Ideas DIY Handwriting](https://reader035.fdocuments.in/reader035/viewer/2022081407/604ac5d1b8491d452f03115b/html5/thumbnails/16.jpg)
Garden Bugs
![Page 17: A Gentle Introduction to Neural Networks - Agile on the Beach...A Gentle Introduction to Neural Networks (with Python) Tariq Rashid @rzeta0 July 2018 Background Ideas DIY Handwriting](https://reader035.fdocuments.in/reader035/viewer/2022081407/604ac5d1b8491d452f03115b/html5/thumbnails/17.jpg)
Classifying Bugs
![Page 18: A Gentle Introduction to Neural Networks - Agile on the Beach...A Gentle Introduction to Neural Networks (with Python) Tariq Rashid @rzeta0 July 2018 Background Ideas DIY Handwriting](https://reader035.fdocuments.in/reader035/viewer/2022081407/604ac5d1b8491d452f03115b/html5/thumbnails/18.jpg)
Classifying Bugs
![Page 19: A Gentle Introduction to Neural Networks - Agile on the Beach...A Gentle Introduction to Neural Networks (with Python) Tariq Rashid @rzeta0 July 2018 Background Ideas DIY Handwriting](https://reader035.fdocuments.in/reader035/viewer/2022081407/604ac5d1b8491d452f03115b/html5/thumbnails/19.jpg)
Classifying Bugs
![Page 20: A Gentle Introduction to Neural Networks - Agile on the Beach...A Gentle Introduction to Neural Networks (with Python) Tariq Rashid @rzeta0 July 2018 Background Ideas DIY Handwriting](https://reader035.fdocuments.in/reader035/viewer/2022081407/604ac5d1b8491d452f03115b/html5/thumbnails/20.jpg)
Classifying Bugs
![Page 21: A Gentle Introduction to Neural Networks - Agile on the Beach...A Gentle Introduction to Neural Networks (with Python) Tariq Rashid @rzeta0 July 2018 Background Ideas DIY Handwriting](https://reader035.fdocuments.in/reader035/viewer/2022081407/604ac5d1b8491d452f03115b/html5/thumbnails/21.jpg)
Key Points
1. Classifying things is kinda like predicting things.
![Page 22: A Gentle Introduction to Neural Networks - Agile on the Beach...A Gentle Introduction to Neural Networks (with Python) Tariq Rashid @rzeta0 July 2018 Background Ideas DIY Handwriting](https://reader035.fdocuments.in/reader035/viewer/2022081407/604ac5d1b8491d452f03115b/html5/thumbnails/22.jpg)
Learning from Data
Example Width Length Bug
1 3.0 1.0 ladybird
2 1.0 3.0 caterpillar
![Page 23: A Gentle Introduction to Neural Networks - Agile on the Beach...A Gentle Introduction to Neural Networks (with Python) Tariq Rashid @rzeta0 July 2018 Background Ideas DIY Handwriting](https://reader035.fdocuments.in/reader035/viewer/2022081407/604ac5d1b8491d452f03115b/html5/thumbnails/23.jpg)
Learning from Data
![Page 24: A Gentle Introduction to Neural Networks - Agile on the Beach...A Gentle Introduction to Neural Networks (with Python) Tariq Rashid @rzeta0 July 2018 Background Ideas DIY Handwriting](https://reader035.fdocuments.in/reader035/viewer/2022081407/604ac5d1b8491d452f03115b/html5/thumbnails/24.jpg)
Learning from Data
not a good separator
![Page 25: A Gentle Introduction to Neural Networks - Agile on the Beach...A Gentle Introduction to Neural Networks (with Python) Tariq Rashid @rzeta0 July 2018 Background Ideas DIY Handwriting](https://reader035.fdocuments.in/reader035/viewer/2022081407/604ac5d1b8491d452f03115b/html5/thumbnails/25.jpg)
Learning from Data
shift the line upjust above the training data point
![Page 26: A Gentle Introduction to Neural Networks - Agile on the Beach...A Gentle Introduction to Neural Networks (with Python) Tariq Rashid @rzeta0 July 2018 Background Ideas DIY Handwriting](https://reader035.fdocuments.in/reader035/viewer/2022081407/604ac5d1b8491d452f03115b/html5/thumbnails/26.jpg)
Learning from Data
![Page 27: A Gentle Introduction to Neural Networks - Agile on the Beach...A Gentle Introduction to Neural Networks (with Python) Tariq Rashid @rzeta0 July 2018 Background Ideas DIY Handwriting](https://reader035.fdocuments.in/reader035/viewer/2022081407/604ac5d1b8491d452f03115b/html5/thumbnails/27.jpg)
How Do We Update The Parameter?
error = target - actual
E = (A + ΔA)x - Ax
ΔA = E / x
![Page 28: A Gentle Introduction to Neural Networks - Agile on the Beach...A Gentle Introduction to Neural Networks (with Python) Tariq Rashid @rzeta0 July 2018 Background Ideas DIY Handwriting](https://reader035.fdocuments.in/reader035/viewer/2022081407/604ac5d1b8491d452f03115b/html5/thumbnails/28.jpg)
Hang On!
Oh no!each update ignoresprevious examples
![Page 29: A Gentle Introduction to Neural Networks - Agile on the Beach...A Gentle Introduction to Neural Networks (with Python) Tariq Rashid @rzeta0 July 2018 Background Ideas DIY Handwriting](https://reader035.fdocuments.in/reader035/viewer/2022081407/604ac5d1b8491d452f03115b/html5/thumbnails/29.jpg)
Calm Down the Learning
ΔA = L · (E / x)
learning rate
![Page 30: A Gentle Introduction to Neural Networks - Agile on the Beach...A Gentle Introduction to Neural Networks (with Python) Tariq Rashid @rzeta0 July 2018 Background Ideas DIY Handwriting](https://reader035.fdocuments.in/reader035/viewer/2022081407/604ac5d1b8491d452f03115b/html5/thumbnails/30.jpg)
Calm Down the Learning
learning rate = 0.5
![Page 31: A Gentle Introduction to Neural Networks - Agile on the Beach...A Gentle Introduction to Neural Networks (with Python) Tariq Rashid @rzeta0 July 2018 Background Ideas DIY Handwriting](https://reader035.fdocuments.in/reader035/viewer/2022081407/604ac5d1b8491d452f03115b/html5/thumbnails/31.jpg)
Key Points
1. Moderating your learning is good - ensures you
learn from all your data, and reduces impact of
outliers or noisy training data.
![Page 32: A Gentle Introduction to Neural Networks - Agile on the Beach...A Gentle Introduction to Neural Networks (with Python) Tariq Rashid @rzeta0 July 2018 Background Ideas DIY Handwriting](https://reader035.fdocuments.in/reader035/viewer/2022081407/604ac5d1b8491d452f03115b/html5/thumbnails/32.jpg)
Boolean Logic
Input A Input B AND OR
0 0 0 0
0 1 0 1
1 0 0 1
1 1 1 1
IF I have eaten my vegetables AND I am still hungry THEN I can have ice cream.
IF it’s the weekend OR I am on annual leave THEN I’ll go to the park.
![Page 33: A Gentle Introduction to Neural Networks - Agile on the Beach...A Gentle Introduction to Neural Networks (with Python) Tariq Rashid @rzeta0 July 2018 Background Ideas DIY Handwriting](https://reader035.fdocuments.in/reader035/viewer/2022081407/604ac5d1b8491d452f03115b/html5/thumbnails/33.jpg)
Boolean Logic
![Page 34: A Gentle Introduction to Neural Networks - Agile on the Beach...A Gentle Introduction to Neural Networks (with Python) Tariq Rashid @rzeta0 July 2018 Background Ideas DIY Handwriting](https://reader035.fdocuments.in/reader035/viewer/2022081407/604ac5d1b8491d452f03115b/html5/thumbnails/34.jpg)
Boolean Logic
![Page 35: A Gentle Introduction to Neural Networks - Agile on the Beach...A Gentle Introduction to Neural Networks (with Python) Tariq Rashid @rzeta0 July 2018 Background Ideas DIY Handwriting](https://reader035.fdocuments.in/reader035/viewer/2022081407/604ac5d1b8491d452f03115b/html5/thumbnails/35.jpg)
XOR Puzzle!
Input A Input B XOR
0 0 0
0 1 1
1 0 1
1 1 0
![Page 36: A Gentle Introduction to Neural Networks - Agile on the Beach...A Gentle Introduction to Neural Networks (with Python) Tariq Rashid @rzeta0 July 2018 Background Ideas DIY Handwriting](https://reader035.fdocuments.in/reader035/viewer/2022081407/604ac5d1b8491d452f03115b/html5/thumbnails/36.jpg)
XOR Solution!
… Use more than one node!
![Page 37: A Gentle Introduction to Neural Networks - Agile on the Beach...A Gentle Introduction to Neural Networks (with Python) Tariq Rashid @rzeta0 July 2018 Background Ideas DIY Handwriting](https://reader035.fdocuments.in/reader035/viewer/2022081407/604ac5d1b8491d452f03115b/html5/thumbnails/37.jpg)
Key Points
1. Some problems can’t be solved with just a single
simple linear classifier.
2. You can use multiple nodes working together to
solve many of these problems.
![Page 38: A Gentle Introduction to Neural Networks - Agile on the Beach...A Gentle Introduction to Neural Networks (with Python) Tariq Rashid @rzeta0 July 2018 Background Ideas DIY Handwriting](https://reader035.fdocuments.in/reader035/viewer/2022081407/604ac5d1b8491d452f03115b/html5/thumbnails/38.jpg)
Brains in Nature
![Page 39: A Gentle Introduction to Neural Networks - Agile on the Beach...A Gentle Introduction to Neural Networks (with Python) Tariq Rashid @rzeta0 July 2018 Background Ideas DIY Handwriting](https://reader035.fdocuments.in/reader035/viewer/2022081407/604ac5d1b8491d452f03115b/html5/thumbnails/39.jpg)
https://en.wikipedia.org/wiki/List_of_animals_by_number_of_neuronshttps://faculty.washington.edu/chudler/facts.html
Brains in Nature
brain 0.4 grams 11,000 neurons
302 neurons
37 billion neurons
(humans 20 billion)
nature’s brains can eat, fly, navigate, fight, communicate, play, learn …
.. and they’re resilient
![Page 40: A Gentle Introduction to Neural Networks - Agile on the Beach...A Gentle Introduction to Neural Networks (with Python) Tariq Rashid @rzeta0 July 2018 Background Ideas DIY Handwriting](https://reader035.fdocuments.in/reader035/viewer/2022081407/604ac5d1b8491d452f03115b/html5/thumbnails/40.jpg)
Brains in Nature
![Page 41: A Gentle Introduction to Neural Networks - Agile on the Beach...A Gentle Introduction to Neural Networks (with Python) Tariq Rashid @rzeta0 July 2018 Background Ideas DIY Handwriting](https://reader035.fdocuments.in/reader035/viewer/2022081407/604ac5d1b8491d452f03115b/html5/thumbnails/41.jpg)
Brains in Nature
logistic function
y = 1 / (1 + e-x)
![Page 42: A Gentle Introduction to Neural Networks - Agile on the Beach...A Gentle Introduction to Neural Networks (with Python) Tariq Rashid @rzeta0 July 2018 Background Ideas DIY Handwriting](https://reader035.fdocuments.in/reader035/viewer/2022081407/604ac5d1b8491d452f03115b/html5/thumbnails/42.jpg)
Brains in Nature
![Page 43: A Gentle Introduction to Neural Networks - Agile on the Beach...A Gentle Introduction to Neural Networks (with Python) Tariq Rashid @rzeta0 July 2018 Background Ideas DIY Handwriting](https://reader035.fdocuments.in/reader035/viewer/2022081407/604ac5d1b8491d452f03115b/html5/thumbnails/43.jpg)
Artificial Neuron
![Page 44: A Gentle Introduction to Neural Networks - Agile on the Beach...A Gentle Introduction to Neural Networks (with Python) Tariq Rashid @rzeta0 July 2018 Background Ideas DIY Handwriting](https://reader035.fdocuments.in/reader035/viewer/2022081407/604ac5d1b8491d452f03115b/html5/thumbnails/44.jpg)
Artificial Neural Network .. finally!
![Page 45: A Gentle Introduction to Neural Networks - Agile on the Beach...A Gentle Introduction to Neural Networks (with Python) Tariq Rashid @rzeta0 July 2018 Background Ideas DIY Handwriting](https://reader035.fdocuments.in/reader035/viewer/2022081407/604ac5d1b8491d452f03115b/html5/thumbnails/45.jpg)
Pause.
...
![Page 46: A Gentle Introduction to Neural Networks - Agile on the Beach...A Gentle Introduction to Neural Networks (with Python) Tariq Rashid @rzeta0 July 2018 Background Ideas DIY Handwriting](https://reader035.fdocuments.in/reader035/viewer/2022081407/604ac5d1b8491d452f03115b/html5/thumbnails/46.jpg)
Where Does The Learning Happen?
sigmoid function slope?link weight?
![Page 47: A Gentle Introduction to Neural Networks - Agile on the Beach...A Gentle Introduction to Neural Networks (with Python) Tariq Rashid @rzeta0 July 2018 Background Ideas DIY Handwriting](https://reader035.fdocuments.in/reader035/viewer/2022081407/604ac5d1b8491d452f03115b/html5/thumbnails/47.jpg)
Key Points
1. Natural brains can do sophisticated things, and
are incredibly resilient to damage and imperfect
signals .. unlike traditional computing.
2. Trying to copy biological brains partly inspired
artificial neural networks.
3. Link weights are the adjustable parameter - it’s
where the learning happens.
![Page 48: A Gentle Introduction to Neural Networks - Agile on the Beach...A Gentle Introduction to Neural Networks (with Python) Tariq Rashid @rzeta0 July 2018 Background Ideas DIY Handwriting](https://reader035.fdocuments.in/reader035/viewer/2022081407/604ac5d1b8491d452f03115b/html5/thumbnails/48.jpg)
Feeding Signals Forward
![Page 49: A Gentle Introduction to Neural Networks - Agile on the Beach...A Gentle Introduction to Neural Networks (with Python) Tariq Rashid @rzeta0 July 2018 Background Ideas DIY Handwriting](https://reader035.fdocuments.in/reader035/viewer/2022081407/604ac5d1b8491d452f03115b/html5/thumbnails/49.jpg)
Feeding Signals Forward
![Page 50: A Gentle Introduction to Neural Networks - Agile on the Beach...A Gentle Introduction to Neural Networks (with Python) Tariq Rashid @rzeta0 July 2018 Background Ideas DIY Handwriting](https://reader035.fdocuments.in/reader035/viewer/2022081407/604ac5d1b8491d452f03115b/html5/thumbnails/50.jpg)
Feeding Signals Forward
![Page 51: A Gentle Introduction to Neural Networks - Agile on the Beach...A Gentle Introduction to Neural Networks (with Python) Tariq Rashid @rzeta0 July 2018 Background Ideas DIY Handwriting](https://reader035.fdocuments.in/reader035/viewer/2022081407/604ac5d1b8491d452f03115b/html5/thumbnails/51.jpg)
Matrix Multiplication
![Page 52: A Gentle Introduction to Neural Networks - Agile on the Beach...A Gentle Introduction to Neural Networks (with Python) Tariq Rashid @rzeta0 July 2018 Background Ideas DIY Handwriting](https://reader035.fdocuments.in/reader035/viewer/2022081407/604ac5d1b8491d452f03115b/html5/thumbnails/52.jpg)
Matrix Multiplication
W·I = Xdot product
weights incoming signals
![Page 53: A Gentle Introduction to Neural Networks - Agile on the Beach...A Gentle Introduction to Neural Networks (with Python) Tariq Rashid @rzeta0 July 2018 Background Ideas DIY Handwriting](https://reader035.fdocuments.in/reader035/viewer/2022081407/604ac5d1b8491d452f03115b/html5/thumbnails/53.jpg)
Key Points
1. The many feedforward calculations can be
expressed concisely as matrix multiplication, no
matter what shape the network.
2. Some programming languages can do matrix
multiplication really efficiently and quickly.
![Page 54: A Gentle Introduction to Neural Networks - Agile on the Beach...A Gentle Introduction to Neural Networks (with Python) Tariq Rashid @rzeta0 July 2018 Background Ideas DIY Handwriting](https://reader035.fdocuments.in/reader035/viewer/2022081407/604ac5d1b8491d452f03115b/html5/thumbnails/54.jpg)
Network Error
![Page 55: A Gentle Introduction to Neural Networks - Agile on the Beach...A Gentle Introduction to Neural Networks (with Python) Tariq Rashid @rzeta0 July 2018 Background Ideas DIY Handwriting](https://reader035.fdocuments.in/reader035/viewer/2022081407/604ac5d1b8491d452f03115b/html5/thumbnails/55.jpg)
Network Error
![Page 56: A Gentle Introduction to Neural Networks - Agile on the Beach...A Gentle Introduction to Neural Networks (with Python) Tariq Rashid @rzeta0 July 2018 Background Ideas DIY Handwriting](https://reader035.fdocuments.in/reader035/viewer/2022081407/604ac5d1b8491d452f03115b/html5/thumbnails/56.jpg)
Internal Error
![Page 57: A Gentle Introduction to Neural Networks - Agile on the Beach...A Gentle Introduction to Neural Networks (with Python) Tariq Rashid @rzeta0 July 2018 Background Ideas DIY Handwriting](https://reader035.fdocuments.in/reader035/viewer/2022081407/604ac5d1b8491d452f03115b/html5/thumbnails/57.jpg)
Internal Error
![Page 58: A Gentle Introduction to Neural Networks - Agile on the Beach...A Gentle Introduction to Neural Networks (with Python) Tariq Rashid @rzeta0 July 2018 Background Ideas DIY Handwriting](https://reader035.fdocuments.in/reader035/viewer/2022081407/604ac5d1b8491d452f03115b/html5/thumbnails/58.jpg)
Matrices Again!
![Page 59: A Gentle Introduction to Neural Networks - Agile on the Beach...A Gentle Introduction to Neural Networks (with Python) Tariq Rashid @rzeta0 July 2018 Background Ideas DIY Handwriting](https://reader035.fdocuments.in/reader035/viewer/2022081407/604ac5d1b8491d452f03115b/html5/thumbnails/59.jpg)
Key Points
1. Remember we use the error to guide how we
refine a model’s parameter - link weights.
2. The error at the output nodes is easy - the
difference between the desired and actual
outputs.
3. The error at internal nodes isn’t obvious. A
heuristic approach is to split it in proportion to
the link weights.
4. … and back propagating the error can be
expressed as a matrix multiplication too!
![Page 60: A Gentle Introduction to Neural Networks - Agile on the Beach...A Gentle Introduction to Neural Networks (with Python) Tariq Rashid @rzeta0 July 2018 Background Ideas DIY Handwriting](https://reader035.fdocuments.in/reader035/viewer/2022081407/604ac5d1b8491d452f03115b/html5/thumbnails/60.jpg)
Yes, But How Do We Actually Update The Weights?
Aaarrrggghhh !!
![Page 61: A Gentle Introduction to Neural Networks - Agile on the Beach...A Gentle Introduction to Neural Networks (with Python) Tariq Rashid @rzeta0 July 2018 Background Ideas DIY Handwriting](https://reader035.fdocuments.in/reader035/viewer/2022081407/604ac5d1b8491d452f03115b/html5/thumbnails/61.jpg)
Perfect is the Enemy of Good
landscape is a complicated difficult mathematical function .. … with all kinds of lumps, bumps, kinks …
![Page 62: A Gentle Introduction to Neural Networks - Agile on the Beach...A Gentle Introduction to Neural Networks (with Python) Tariq Rashid @rzeta0 July 2018 Background Ideas DIY Handwriting](https://reader035.fdocuments.in/reader035/viewer/2022081407/604ac5d1b8491d452f03115b/html5/thumbnails/62.jpg)
Gradient Descent
smaller gradient .. you’re closer to the bottom … take smaller steps?
![Page 63: A Gentle Introduction to Neural Networks - Agile on the Beach...A Gentle Introduction to Neural Networks (with Python) Tariq Rashid @rzeta0 July 2018 Background Ideas DIY Handwriting](https://reader035.fdocuments.in/reader035/viewer/2022081407/604ac5d1b8491d452f03115b/html5/thumbnails/63.jpg)
Key Points
1. Gradient descent is a practical way of finding the
minimum of difficult functions.
2. You can avoid the chance of overshooting by
taking smaller steps if the gradient gets shallower.
3. The error of a neural network is a difficult function
of the link weights … so maybe gradient descent
will help ...
![Page 64: A Gentle Introduction to Neural Networks - Agile on the Beach...A Gentle Introduction to Neural Networks (with Python) Tariq Rashid @rzeta0 July 2018 Background Ideas DIY Handwriting](https://reader035.fdocuments.in/reader035/viewer/2022081407/604ac5d1b8491d452f03115b/html5/thumbnails/64.jpg)
Climbing Down the Network Error Landscape
We need to find this gradient
![Page 65: A Gentle Introduction to Neural Networks - Agile on the Beach...A Gentle Introduction to Neural Networks (with Python) Tariq Rashid @rzeta0 July 2018 Background Ideas DIY Handwriting](https://reader035.fdocuments.in/reader035/viewer/2022081407/604ac5d1b8491d452f03115b/html5/thumbnails/65.jpg)
Error Gradient
A gentle intro to calculus
http://makeyourownneuralnetwork.blogspot.co.uk/2016/01/a-gentle-introduction-to-calculus.html
E = (desired - actual)2
dE/dwij = - ej . oj . (1 - oj) . oi
school level calculus (chain rule)
previous node
![Page 66: A Gentle Introduction to Neural Networks - Agile on the Beach...A Gentle Introduction to Neural Networks (with Python) Tariq Rashid @rzeta0 July 2018 Background Ideas DIY Handwriting](https://reader035.fdocuments.in/reader035/viewer/2022081407/604ac5d1b8491d452f03115b/html5/thumbnails/66.jpg)
Updating the Weights
remember that learning rate
move wjk in the opposite direction to the slope
![Page 67: A Gentle Introduction to Neural Networks - Agile on the Beach...A Gentle Introduction to Neural Networks (with Python) Tariq Rashid @rzeta0 July 2018 Background Ideas DIY Handwriting](https://reader035.fdocuments.in/reader035/viewer/2022081407/604ac5d1b8491d452f03115b/html5/thumbnails/67.jpg)
DIY
![Page 68: A Gentle Introduction to Neural Networks - Agile on the Beach...A Gentle Introduction to Neural Networks (with Python) Tariq Rashid @rzeta0 July 2018 Background Ideas DIY Handwriting](https://reader035.fdocuments.in/reader035/viewer/2022081407/604ac5d1b8491d452f03115b/html5/thumbnails/68.jpg)
Python Class and Functions
Neural NetworkClass
Initialise Train Query
set size, initial weights do the learning query for answers
![Page 69: A Gentle Introduction to Neural Networks - Agile on the Beach...A Gentle Introduction to Neural Networks (with Python) Tariq Rashid @rzeta0 July 2018 Background Ideas DIY Handwriting](https://reader035.fdocuments.in/reader035/viewer/2022081407/604ac5d1b8491d452f03115b/html5/thumbnails/69.jpg)
Python has Cool Tools
numpyscipy
matplotlibnotebook
matrix maths
![Page 70: A Gentle Introduction to Neural Networks - Agile on the Beach...A Gentle Introduction to Neural Networks (with Python) Tariq Rashid @rzeta0 July 2018 Background Ideas DIY Handwriting](https://reader035.fdocuments.in/reader035/viewer/2022081407/604ac5d1b8491d452f03115b/html5/thumbnails/70.jpg)
Function - Initialise
# initialise the neural network
def __init__(self, inputnodes, hiddennodes, outputnodes, learningrate):
# set number of nodes in each input, hidden, output layer
self.inodes = inputnodes
self.hnodes = hiddennodes
self.onodes = outputnodes
# link weight matrices, wih and who
# weights inside the arrays are w_i_j, where link is from node i to node j in the next layer
# w11 w21
# w12 w22 etc
self.wih = numpy.random.normal(0.0, pow(self.hnodes, -0.5), (self.hnodes, self.inodes))
self.who = numpy.random.normal(0.0, pow(self.onodes, -0.5), (self.onodes, self.hnodes))
# learning rate
self.lr = learningrate
# activation function is the sigmoid function
self.activation_function = lambda x: scipy.special.expit(x)
pass
random initial weightsnumpy.random.normal()
![Page 71: A Gentle Introduction to Neural Networks - Agile on the Beach...A Gentle Introduction to Neural Networks (with Python) Tariq Rashid @rzeta0 July 2018 Background Ideas DIY Handwriting](https://reader035.fdocuments.in/reader035/viewer/2022081407/604ac5d1b8491d452f03115b/html5/thumbnails/71.jpg)
Function - Query
combined weighted signals into hidden layer
# query the neural network
def query(self, inputs_list):
# convert inputs list to 2d array
inputs = numpy.array(inputs_list, ndmin=2).T
# calculate signals into hidden layer
hidden_inputs = numpy.dot(self.wih, inputs)
# calculate the signals emerging from hidden layer
hidden_outputs = self.activation_function(hidden_inputs)
# calculate signals into final output layer
final_inputs = numpy.dot(self.who, hidden_outputs)
# calculate the signals emerging from final output layer
final_outputs = self.activation_function(final_inputs)
return final_outputs
then sigmoid applied
similar for output layernumpy.dot()
![Page 72: A Gentle Introduction to Neural Networks - Agile on the Beach...A Gentle Introduction to Neural Networks (with Python) Tariq Rashid @rzeta0 July 2018 Background Ideas DIY Handwriting](https://reader035.fdocuments.in/reader035/viewer/2022081407/604ac5d1b8491d452f03115b/html5/thumbnails/72.jpg)
Function - Train
# train the neural network
def train(self, inputs_list, targets_list):
# convert inputs list to 2d array
inputs = numpy.array(inputs_list, ndmin=2).T
targets = numpy.array(targets_list, ndmin=2).T
# calculate signals into hidden layer
hidden_inputs = numpy.dot(self.wih, inputs)
# calculate the signals emerging from hidden layer
hidden_outputs = self.activation_function(hidden_inputs)
# calculate signals into final output layer
final_inputs = numpy.dot(self.who, hidden_outputs)
# calculate the signals emerging from final output layer
final_outputs = self.activation_function(final_inputs)
# output layer error is the (target - actual)
output_errors = targets - final_outputs
# hidden layer error is the output_errors, split by weights, recombined at hidden nodes
hidden_errors = numpy.dot(self.who.T, output_errors)
# update the weights for the links between the hidden and output layers
self.who += self.lr * numpy.dot((output_errors * final_outputs * (1.0 - final_outputs)),
numpy.transpose(hidden_outputs))
# update the weights for the links between the input and hidden layers
self.wih += self.lr * numpy.dot((hidden_errors * hidden_outputs * (1.0 - hidden_outputs)),
numpy.transpose(inputs))
pass
output layer errors
hidden layer errors
update weights
same feed forward as before
![Page 73: A Gentle Introduction to Neural Networks - Agile on the Beach...A Gentle Introduction to Neural Networks (with Python) Tariq Rashid @rzeta0 July 2018 Background Ideas DIY Handwriting](https://reader035.fdocuments.in/reader035/viewer/2022081407/604ac5d1b8491d452f03115b/html5/thumbnails/73.jpg)
Handwriting
![Page 74: A Gentle Introduction to Neural Networks - Agile on the Beach...A Gentle Introduction to Neural Networks (with Python) Tariq Rashid @rzeta0 July 2018 Background Ideas DIY Handwriting](https://reader035.fdocuments.in/reader035/viewer/2022081407/604ac5d1b8491d452f03115b/html5/thumbnails/74.jpg)
Handwritten Numbers Challenge
![Page 75: A Gentle Introduction to Neural Networks - Agile on the Beach...A Gentle Introduction to Neural Networks (with Python) Tariq Rashid @rzeta0 July 2018 Background Ideas DIY Handwriting](https://reader035.fdocuments.in/reader035/viewer/2022081407/604ac5d1b8491d452f03115b/html5/thumbnails/75.jpg)
MNIST Datasets
MNIST dataset:60,000 training data examples10,000 test data examples
![Page 76: A Gentle Introduction to Neural Networks - Agile on the Beach...A Gentle Introduction to Neural Networks (with Python) Tariq Rashid @rzeta0 July 2018 Background Ideas DIY Handwriting](https://reader035.fdocuments.in/reader035/viewer/2022081407/604ac5d1b8491d452f03115b/html5/thumbnails/76.jpg)
MNIST Datasets
label784 pixelsvalues
28 by 28 pixel image
![Page 77: A Gentle Introduction to Neural Networks - Agile on the Beach...A Gentle Introduction to Neural Networks (with Python) Tariq Rashid @rzeta0 July 2018 Background Ideas DIY Handwriting](https://reader035.fdocuments.in/reader035/viewer/2022081407/604ac5d1b8491d452f03115b/html5/thumbnails/77.jpg)
Output Layer Values
![Page 78: A Gentle Introduction to Neural Networks - Agile on the Beach...A Gentle Introduction to Neural Networks (with Python) Tariq Rashid @rzeta0 July 2018 Background Ideas DIY Handwriting](https://reader035.fdocuments.in/reader035/viewer/2022081407/604ac5d1b8491d452f03115b/html5/thumbnails/78.jpg)
Experiments
96% is very good!
we’ve only used simple ideas and code
random processesdo go wonky!
![Page 79: A Gentle Introduction to Neural Networks - Agile on the Beach...A Gentle Introduction to Neural Networks (with Python) Tariq Rashid @rzeta0 July 2018 Background Ideas DIY Handwriting](https://reader035.fdocuments.in/reader035/viewer/2022081407/604ac5d1b8491d452f03115b/html5/thumbnails/79.jpg)
More Experiments
98% is amazing!
![Page 80: A Gentle Introduction to Neural Networks - Agile on the Beach...A Gentle Introduction to Neural Networks (with Python) Tariq Rashid @rzeta0 July 2018 Background Ideas DIY Handwriting](https://reader035.fdocuments.in/reader035/viewer/2022081407/604ac5d1b8491d452f03115b/html5/thumbnails/80.jpg)
Thoughts
![Page 81: A Gentle Introduction to Neural Networks - Agile on the Beach...A Gentle Introduction to Neural Networks (with Python) Tariq Rashid @rzeta0 July 2018 Background Ideas DIY Handwriting](https://reader035.fdocuments.in/reader035/viewer/2022081407/604ac5d1b8491d452f03115b/html5/thumbnails/81.jpg)
Peek Inside The Mind Of a Neural Network?
![Page 82: A Gentle Introduction to Neural Networks - Agile on the Beach...A Gentle Introduction to Neural Networks (with Python) Tariq Rashid @rzeta0 July 2018 Background Ideas DIY Handwriting](https://reader035.fdocuments.in/reader035/viewer/2022081407/604ac5d1b8491d452f03115b/html5/thumbnails/82.jpg)
Peek Inside The Mind Of a Neural Network?
this isn’t done very often
![Page 83: A Gentle Introduction to Neural Networks - Agile on the Beach...A Gentle Introduction to Neural Networks (with Python) Tariq Rashid @rzeta0 July 2018 Background Ideas DIY Handwriting](https://reader035.fdocuments.in/reader035/viewer/2022081407/604ac5d1b8491d452f03115b/html5/thumbnails/83.jpg)
Thanks!
live demo!
![Page 84: A Gentle Introduction to Neural Networks - Agile on the Beach...A Gentle Introduction to Neural Networks (with Python) Tariq Rashid @rzeta0 July 2018 Background Ideas DIY Handwriting](https://reader035.fdocuments.in/reader035/viewer/2022081407/604ac5d1b8491d452f03115b/html5/thumbnails/84.jpg)
Finding Out More
makeyourownneuralnetwork.blogspot.co.uk
github.com/makeyourownneuralnetwork
www.amazon.co.uk/dp/B01EER4Z4G
twitter.com/myoneuralnet
slides goo.gl/JKsb62
![Page 85: A Gentle Introduction to Neural Networks - Agile on the Beach...A Gentle Introduction to Neural Networks (with Python) Tariq Rashid @rzeta0 July 2018 Background Ideas DIY Handwriting](https://reader035.fdocuments.in/reader035/viewer/2022081407/604ac5d1b8491d452f03115b/html5/thumbnails/85.jpg)
Raspberry Pi Zero
It all works on a Raspberry Pi Zero … and it only costs £4 / $5 !!
![Page 86: A Gentle Introduction to Neural Networks - Agile on the Beach...A Gentle Introduction to Neural Networks (with Python) Tariq Rashid @rzeta0 July 2018 Background Ideas DIY Handwriting](https://reader035.fdocuments.in/reader035/viewer/2022081407/604ac5d1b8491d452f03115b/html5/thumbnails/86.jpg)
Cornwall