THE PERCEPTRON · program in matlab a preceptron with a perceptron learning rule and solve the OR,...
Transcript of THE PERCEPTRON · program in matlab a preceptron with a perceptron learning rule and solve the OR,...
![Page 1: THE PERCEPTRON · program in matlab a preceptron with a perceptron learning rule and solve the OR, AND and XOR problems. (Due Feb 8’th) Show Demo . Summary – what can perceptrons](https://reader034.fdocuments.in/reader034/viewer/2022052313/5bb0955609d3f289598bf295/html5/thumbnails/1.jpg)
THE PERCEPTRON: (Classification)
Threshold unit:
where is the output for input pattern , are the synaptic weights and is the desired output
w1 w2 w3 w4 w5
![Page 2: THE PERCEPTRON · program in matlab a preceptron with a perceptron learning rule and solve the OR, AND and XOR problems. (Due Feb 8’th) Show Demo . Summary – what can perceptrons](https://reader034.fdocuments.in/reader034/viewer/2022052313/5bb0955609d3f289598bf295/html5/thumbnails/2.jpg)
x1 x2 y 1 1 1 1 0 0 0 1 0 0 0 0
AND
0 1
1
-1.5
1 1 Linearly seprable
![Page 3: THE PERCEPTRON · program in matlab a preceptron with a perceptron learning rule and solve the OR, AND and XOR problems. (Due Feb 8’th) Show Demo . Summary – what can perceptrons](https://reader034.fdocuments.in/reader034/viewer/2022052313/5bb0955609d3f289598bf295/html5/thumbnails/3.jpg)
x1 x2 y 1 1 1 1 0 1 0 1 1 0 0 0
OR
-0.5
1 1
0 1
1
Linearly separable
![Page 4: THE PERCEPTRON · program in matlab a preceptron with a perceptron learning rule and solve the OR, AND and XOR problems. (Due Feb 8’th) Show Demo . Summary – what can perceptrons](https://reader034.fdocuments.in/reader034/viewer/2022052313/5bb0955609d3f289598bf295/html5/thumbnails/4.jpg)
x1 x2 y 1 1 0 1 0 1 0 1 1 0 0 0
XOR
0 1
1
Linearly separable?
![Page 5: THE PERCEPTRON · program in matlab a preceptron with a perceptron learning rule and solve the OR, AND and XOR problems. (Due Feb 8’th) Show Demo . Summary – what can perceptrons](https://reader034.fdocuments.in/reader034/viewer/2022052313/5bb0955609d3f289598bf295/html5/thumbnails/5.jpg)
Linearly separable Linearly inseparable
0 1
1
0 1
1
Linearly inseparable is not necessarily difficult
![Page 6: THE PERCEPTRON · program in matlab a preceptron with a perceptron learning rule and solve the OR, AND and XOR problems. (Due Feb 8’th) Show Demo . Summary – what can perceptrons](https://reader034.fdocuments.in/reader034/viewer/2022052313/5bb0955609d3f289598bf295/html5/thumbnails/6.jpg)
Perceptron learning rule:
w1 w2 w3 w4 w5
Convergence proof:
Hertz, Krough, Palmer (HKP)
Assignment 3a: program in matlab a preceptron with a perceptron learning rule and solve the OR, AND and XOR problems. (Due Feb 8’th)
Show Demo
![Page 7: THE PERCEPTRON · program in matlab a preceptron with a perceptron learning rule and solve the OR, AND and XOR problems. (Due Feb 8’th) Show Demo . Summary – what can perceptrons](https://reader034.fdocuments.in/reader034/viewer/2022052313/5bb0955609d3f289598bf295/html5/thumbnails/7.jpg)
Summary – what can perceptrons do and how?
![Page 8: THE PERCEPTRON · program in matlab a preceptron with a perceptron learning rule and solve the OR, AND and XOR problems. (Due Feb 8’th) Show Demo . Summary – what can perceptrons](https://reader034.fdocuments.in/reader034/viewer/2022052313/5bb0955609d3f289598bf295/html5/thumbnails/8.jpg)
Linear single layer network: ( approximation, curve fitting)
Linear unit:
where is the output for input pattern , are the synaptic weights and is the desired output
w1 w2 w3 w4 w5
Minimize mean square error:
or *
![Page 9: THE PERCEPTRON · program in matlab a preceptron with a perceptron learning rule and solve the OR, AND and XOR problems. (Due Feb 8’th) Show Demo . Summary – what can perceptrons](https://reader034.fdocuments.in/reader034/viewer/2022052313/5bb0955609d3f289598bf295/html5/thumbnails/9.jpg)
Linear single layer network: ( approximation, curve fitting)
Linear unit:
where is the output for input pattern , are the synaptic weights and is the desired output
w1 w2 w3 w4 w5
Minimize mean square error:
![Page 10: THE PERCEPTRON · program in matlab a preceptron with a perceptron learning rule and solve the OR, AND and XOR problems. (Due Feb 8’th) Show Demo . Summary – what can perceptrons](https://reader034.fdocuments.in/reader034/viewer/2022052313/5bb0955609d3f289598bf295/html5/thumbnails/10.jpg)
The best solution is obtained when E is minimal.
For linear neurons there is an exact solution for this called the pseudo-inverse (see HKP).
Looking for a solution by gradient descent:
E
w
-gradient
Chain rule
![Page 11: THE PERCEPTRON · program in matlab a preceptron with a perceptron learning rule and solve the OR, AND and XOR problems. (Due Feb 8’th) Show Demo . Summary – what can perceptrons](https://reader034.fdocuments.in/reader034/viewer/2022052313/5bb0955609d3f289598bf295/html5/thumbnails/11.jpg)
and
Since:
Error:
Therefore:
Which types of problems can a linear network solve?
![Page 12: THE PERCEPTRON · program in matlab a preceptron with a perceptron learning rule and solve the OR, AND and XOR problems. (Due Feb 8’th) Show Demo . Summary – what can perceptrons](https://reader034.fdocuments.in/reader034/viewer/2022052313/5bb0955609d3f289598bf295/html5/thumbnails/12.jpg)
Sigmoidal neurons:
Which types of problems can a sigmoidal networks solve?
Assignment 3b – Implement a one layer linear and sigmoidal network, fit a 1D a linear, a sigmoid and a quadratic function, for both networks.
for example:
![Page 13: THE PERCEPTRON · program in matlab a preceptron with a perceptron learning rule and solve the OR, AND and XOR problems. (Due Feb 8’th) Show Demo . Summary – what can perceptrons](https://reader034.fdocuments.in/reader034/viewer/2022052313/5bb0955609d3f289598bf295/html5/thumbnails/13.jpg)
![Page 14: THE PERCEPTRON · program in matlab a preceptron with a perceptron learning rule and solve the OR, AND and XOR problems. (Due Feb 8’th) Show Demo . Summary – what can perceptrons](https://reader034.fdocuments.in/reader034/viewer/2022052313/5bb0955609d3f289598bf295/html5/thumbnails/14.jpg)
Multi layer networks:
• Can solve non linearly separable classification problems.
• Can approximate any arbitrary function, given ‘enough’ units in the hidden layer.
Hidden layer
Output layer
Input layer
![Page 15: THE PERCEPTRON · program in matlab a preceptron with a perceptron learning rule and solve the OR, AND and XOR problems. (Due Feb 8’th) Show Demo . Summary – what can perceptrons](https://reader034.fdocuments.in/reader034/viewer/2022052313/5bb0955609d3f289598bf295/html5/thumbnails/15.jpg)
Multi layer networks:
Hidden layer
Output layer
Input layer
€
σ(x) =1
1+ exp(−βx)
Example sigmoid
![Page 16: THE PERCEPTRON · program in matlab a preceptron with a perceptron learning rule and solve the OR, AND and XOR problems. (Due Feb 8’th) Show Demo . Summary – what can perceptrons](https://reader034.fdocuments.in/reader034/viewer/2022052313/5bb0955609d3f289598bf295/html5/thumbnails/16.jpg)
€
oµ =σ 2 wk2σ( wk, j
1 x jµ )
j∑
k∑
Note: is not a vector but a matrix
![Page 17: THE PERCEPTRON · program in matlab a preceptron with a perceptron learning rule and solve the OR, AND and XOR problems. (Due Feb 8’th) Show Demo . Summary – what can perceptrons](https://reader034.fdocuments.in/reader034/viewer/2022052313/5bb0955609d3f289598bf295/html5/thumbnails/17.jpg)
Solving linearly inseparable problems
x1 x2 y 1 1 0 1 0 1 0 1 1 0 0 0
XOR
Hint: XOR = or and not and
![Page 18: THE PERCEPTRON · program in matlab a preceptron with a perceptron learning rule and solve the OR, AND and XOR problems. (Due Feb 8’th) Show Demo . Summary – what can perceptrons](https://reader034.fdocuments.in/reader034/viewer/2022052313/5bb0955609d3f289598bf295/html5/thumbnails/18.jpg)
How do we learn a multi-layer network The credit assignment problem !
.5 0
-.5
x1 x2 y 1 1 0 1 0 1 0 1 1 0 0 0
XOR
0.5 -0.5 1 -1
1 0.5
![Page 19: THE PERCEPTRON · program in matlab a preceptron with a perceptron learning rule and solve the OR, AND and XOR problems. (Due Feb 8’th) Show Demo . Summary – what can perceptrons](https://reader034.fdocuments.in/reader034/viewer/2022052313/5bb0955609d3f289598bf295/html5/thumbnails/19.jpg)
Gradient descent/ Back Propagation, the solution to the credit assignment problem:
€
E =12
yµ − oµ( )2
µ=1
P
∑ Where:
From hidden layer to output weights:
{ Note –for simplicity of derivation left out second sigma – in general need to keep it.
![Page 20: THE PERCEPTRON · program in matlab a preceptron with a perceptron learning rule and solve the OR, AND and XOR problems. (Due Feb 8’th) Show Demo . Summary – what can perceptrons](https://reader034.fdocuments.in/reader034/viewer/2022052313/5bb0955609d3f289598bf295/html5/thumbnails/20.jpg)
€
E =12
yµ − oµ( )2
µ=1
P
∑ Where:
For input to hidden layer:
and
and
{
![Page 21: THE PERCEPTRON · program in matlab a preceptron with a perceptron learning rule and solve the OR, AND and XOR problems. (Due Feb 8’th) Show Demo . Summary – what can perceptrons](https://reader034.fdocuments.in/reader034/viewer/2022052313/5bb0955609d3f289598bf295/html5/thumbnails/21.jpg)
For input to hidden layer:
and
Assignment 3c: Program a 2 layer network in matlab, solve the XOR problem. Fit the curve: x(x-1) between 0 and 1, how many hidden units did you need?
![Page 22: THE PERCEPTRON · program in matlab a preceptron with a perceptron learning rule and solve the OR, AND and XOR problems. (Due Feb 8’th) Show Demo . Summary – what can perceptrons](https://reader034.fdocuments.in/reader034/viewer/2022052313/5bb0955609d3f289598bf295/html5/thumbnails/22.jpg)
Formal neural networks can accomplish many tasks, for example:
• Perform complex classification
• Learn arbitrary functions
• Account for associative memory Some applications: Robotics, Character recognition, Speech recognition, Medical diagnostics.
This is not Neuroscience, but is motivated loosely by neuroscience and carries important information for neuroscience as well.
For example: Memory, learning and some aspects of development are assumed to be based on synaptic plasticity.
![Page 23: THE PERCEPTRON · program in matlab a preceptron with a perceptron learning rule and solve the OR, AND and XOR problems. (Due Feb 8’th) Show Demo . Summary – what can perceptrons](https://reader034.fdocuments.in/reader034/viewer/2022052313/5bb0955609d3f289598bf295/html5/thumbnails/23.jpg)
What did we learn today?
Is Backprop biologically realistic?