Artificial Neural Networks Lect1: Introduction & neural computation
COMP53311 Other Classification Models: Neural Network Prepared by Raymond Wong Some of the notes...
-
Upload
donald-morris -
Category
Documents
-
view
219 -
download
0
description
Transcript of COMP53311 Other Classification Models: Neural Network Prepared by Raymond Wong Some of the notes...
COMP5331 1
COMP5331
Other Classification Models:Neural Network
Prepared by Raymond WongSome of the notes about Neural Network are borrowed from LW Chan’s notes
Presented by Raymond Wongraywong@cse
COMP5331 2
What we learnt for Classification Decision Tree Bayesian Classifier Nearest Neighbor Classifier
COMP5331 3
Other Classification Models Neural Network
COMP5331 4
Neural Network Neural Network
A computing system made up of simple and highly interconnected processing elements
Other terminologies: Connectionist Models Parallel distributed processing models (PDP) Artificial Neural Networks Computational Neural Networks Neurocomputers
COMP5331 5
Neural Network This approach is inspired by the way
that brains process information, which is entirely differ from the way that conventional computers do
Information processing occurs at many identical and simple processing elements called neurons (or also called units, cells or nodes)
Interneuron connection strengths known as synaptic weights are used to store the knowledge
COMP5331 6
Advantages of Neural Network Parallel Processing – each
neuron operates individually Fault tolerance – if a small
number of neurons break down, the whole system is still able to operate with slight degradation in performance.
COMP5331 7
Neuron Network Neuron Network for OR
Neuron Network
input
outputx1
x2y
x1 x2 y0 0 00 1 11 0 11 1 1
OR Function
COMP5331 8
Neuron Network Neuron Network for OR
input
outputx1
x2y
x1 x2 y0 0 00 1 11 0 11 1 1
OR Function
Neuron
COMP5331 9
Neuron Network Neuron Network for OR
input
outputx1
x2y
x1 x2 y0 0 00 1 11 0 11 1 1
OR Function
Front Backnet
COMP5331 10
Neuron Network Neuron Network for OR
input
outputx1
x2y
x1 x2 y0 0 00 1 11 0 11 1 1
OR Function
Front Backnetw1
w2
Weight net = w1x1 + w2x2 + b
COMP5331 11
Neuron Network Neuron Network for OR
input
outputx1
x2y
x1 x2 y0 0 00 1 11 0 11 1 1
OR Function
Front Backnetw1
w2
Activation function-Linear function: y = net or y = a . net-Non-linear function
COMP5331 12
Activation Function Non-linear functions
Threshold function, Step Function, Hard Limiter
Piecewise-Linear Function Sigmoid Function Radial Basis Function
COMP5331 13
Threshold function, Step Function, Hard Limiter
y = if net 0
if net <0
1
0
0 net
y
1
COMP5331 14
Piecewise-Linear Function
y =
if net a
if –a < net < a
1
0
0 net
y
1
if net -a
½(1/a x net + 1)
-a a
COMP5331 15
Sigmoid Function
y =
0 net
y
1
11 + e-net
COMP5331 16
Radial Basis Function
y =
0 net
y
1
-net2e
COMP5331 17
Neuron Network Neuron Network for OR
input
outputx1
x2y
x1 x2 y0 0 00 1 11 0 11 1 1
OR Function
Front Backnetw1
w2
net = w1x1 + w2x2 + b
Threshold function
y = if net 0
if net <0
1
0
COMP5331 18
Learning Let be the learning rate (a real
number) Learning is done by
wi wi + (d – y)xi where
d is the desired output y is the output of our neural network
b b + (d – y)
COMP5331 19
Neuron Network Neuron Network for OR
input
outputx1
x2y
x1 x2 d0 0 00 1 11 0 11 1 1
OR Function
Front Backnetw1
w2
net = w1x1 + w2x2 + b
Threshold function
y = if net 0
if net <0
1
0
net = w1x1 + w2x2 + b y = if net 0
if net <0
1
0
COMP5331 20
Neuron Networkx1 x2 d0 0 00 1 11 0 11 1 1
net = w1x1 + w2x2 + b y = if net 0
if net <0
1
0
net = w1x1 + w2x2 + b
b w1 w21 1 1
=1
y = 1
w1 = w1 + (d – y)x1
= 1+0.8*(0-1)*0 = 1
w2 = w2 + (d – y)x2
= 1+0.8*(0-1)*0 = 1
b = b + (d – y)
= 1+0.8*(0-1) = 0.2
0.2 1 1
Incorrect!
= 0.8
COMP5331 21
Neuron Networkx1 x2 d0 0 00 1 11 0 11 1 1
net = w1x1 + w2x2 + b y = if net 0
if net <0
1
0
net = w1x1 + w2x2 + b
b w1 w20.2 1 1
=1.2
y = 1
w1 = w1 + (d – y)x1
= 1+0.8*(1-1)*0 = 1
w2 = w2 + (d – y)x2
= 1+0.8*(1-1)*1 = 1
b = b + (d – y)
= 0.2+0.8*(1-1) = 0.2
0.2 1 1
Correct!
= 0.8
COMP5331 22
Neuron Networkx1 x2 d0 0 00 1 11 0 11 1 1
net = w1x1 + w2x2 + b y = if net 0
if net <0
1
0
net = w1x1 + w2x2 + b
b w1 w20.2 1 1
=1.2
y = 1
w1 = w1 + (d – y)x1
= 1+0.8*(1-1)*1 = 1
w2 = w2 + (d – y)x2
= 1+0.8*(1-1)*0 = 1
b = b + (d – y)
= 0.2+0.8*(1-1) = 0.2
0.2 1 1
Correct!
= 0.8
COMP5331 23
Neuron Networkx1 x2 d0 0 00 1 11 0 11 1 1
net = w1x1 + w2x2 + b y = if net 0
if net <0
1
0
net = w1x1 + w2x2 + b
b w1 w20.2 1 1
=2.2
y = 1
w1 = w1 + (d – y)x1
= 1+0.8*(1-1)*1 = 1
w2 = w2 + (d – y)x2
= 1+0.8*(1-1)*1 = 1
b = b + (d – y)
= 0.2+0.8*(1-1) = 0.2
0.2 1 1
Correct!
= 0.8
COMP5331 24
Neuron Networkx1 x2 d0 0 00 1 11 0 11 1 1
net = w1x1 + w2x2 + b y = if net 0
if net <0
1
0
net = w1x1 + w2x2 + b
b w1 w20.2 1 1
=0.2
y = 1
w1 = w1 + (d – y)x1
= 1+0.8*(0-1)*0 = 1
w2 = w2 + (d – y)x2
= 1+0.8*(0-1)*0 = 1
b = b + (d – y)
= 0.2+0.8*(0-1) = -0.6
-0.6 1 1
Incorrect!
We repeat the above process until the neural networks output the correct values of y (i.e., y = d for each possible input)
= 0.8
COMP5331 25
Neuron Network Neuron Network for OR
input
outputx1
x2y
x1 x2 y0 0 00 1 11 0 11 1 1
OR Function
Front Backnetw1
w2
net = w1x1 + w2x2 + b
Threshold function
y = if net 0
if net <0
1
0
COMP5331 26
Limitation It can only solve linearly separable
problems
COMP5331 27
Multi-layer Perceptron (MLP)
Neuron Network
input
outputx1
x2y
input
outputx1
x2yNeuron
COMP5331 28
Multi-layer Perceptron (MLP)
Neuron Network
input
outputx1
x2y
inputoutputx1
x2
y
COMP5331 29
Multi-layer Perceptron (MLP)
inputoutput
x1y1
x2
x3
x4
x5
y2
y3
y4
Input layer Hidden layer Output layer
COMP5331 30
Advantages of MLP Can solve
Linear separable problems Non-linear separable problems
A universal approximator MLP has proven to be a universal
approximator, i.e., it can model all types function y = f(x)