Dr. kiani artificial neural network lecture 1
-
Upload
parinaz-faraji -
Category
Education
-
view
556 -
download
5
description
Transcript of Dr. kiani artificial neural network lecture 1
![Page 1: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/1.jpg)
Semiconductors, BP&A Planning, 2003-01-29 1
x0
xn
w0
wn
oi
n
ii xw
0
o/w 0 and 0 if 10
i
n
ii xwo
Threshold units
![Page 2: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/2.jpg)
Semiconductors, BP&A Planning, 2003-01-29 2
History spiking neural networks
Vapnik (1990) ---support vector machine
Broomhead & Lowe (1988) ----Radial basis functions (RBF)
Linsker (1988) ----- Informax principle
Rumelhart, Hinton -------- Back-propagation & Williams (1986)
Kohonen(1982) ------ Self-organizing mapsHopfield(1982) ------ Hopfield Networks
Minsky & Papert(1969) ------ Perceptrons
Rosenblatt(1960) ------ Perceptron
Minsky(1954) ------ Neural Networks (PhD Thesis)
Hebb(1949) --------The organization of behaviour
McCulloch & Pitts (1943) -----neural networks and artificial intelligence were born
![Page 3: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/3.jpg)
Semiconductors, BP&A Planning, 2003-01-29 3
History of Neural Networks
• 1943: McCullough and Pitts - Modeling the Neuron for Parallel Distributed Processing
• 1958: Rosenblatt - Perceptron• 1969: Minsky and Papert publish limits on
the ability of a perceptron to generalize• 1970’s and 1980’s: ANN renaissance• 1986: Rumelhart, Hinton + Williams
present backpropagation• 1989: Tsividis: Neural Network on a chip
![Page 4: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/4.jpg)
Semiconductors, BP&A Planning, 2003-01-29 4
William McCulloch
![Page 5: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/5.jpg)
Semiconductors, BP&A Planning, 2003-01-29 5
Neural Networks
• McCulloch & Pitts (1943) are generally recognised as the designers of the first neural network
• Many of their ideas still used today (e.g. many simple units combine to give increased computational power and the idea of a threshold)
![Page 6: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/6.jpg)
Semiconductors, BP&A Planning, 2003-01-29 6
Neural Networks
• Hebb (1949) developed the first learning rule (on the premise that if two neurons were active at the same time the strength between them should be increased)
![Page 7: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/7.jpg)
Semiconductors, BP&A Planning, 2003-01-29 7
![Page 8: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/8.jpg)
Semiconductors, BP&A Planning, 2003-01-29 8
Neural Networks
• During the 50’s and 60’s many researchers worked on the perceptron amidst great excitement.
• 1969 saw the death of neural network research for about 15 years – Minsky & Papert
• Only in the mid 80’s (Parker and LeCun) was interest revived (in fact Werbos discovered algorithm in 1974)
![Page 9: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/9.jpg)
Semiconductors, BP&A Planning, 2003-01-29 9
How Does the Brain Work ? (1)
NEURON• The cell that perform information
processing in the brain• Fundamental functional unit of
all nervous system tissue
![Page 10: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/10.jpg)
Semiconductors, BP&A Planning, 2003-01-29 10
How Does the Brain Work ? (2)
Each consists of : SOMA, DENDRITES, AXON, and SYNAPSE
![Page 11: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/11.jpg)
Semiconductors, BP&A Planning, 2003-01-29 11
Biological neurons
axon
dendrites
dendrites
synapse
cell
![Page 12: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/12.jpg)
Semiconductors, BP&A Planning, 2003-01-29 12
Neural Networks
• We are born with about 100 billion neurons
• A neuron may connect to as many as 100,000 other neurons
![Page 13: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/13.jpg)
Semiconductors, BP&A Planning, 2003-01-29 13
Biological inspiration
Dendrites
Soma (cell body)
Axon
![Page 14: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/14.jpg)
Semiconductors, BP&A Planning, 2003-01-29 14
Biological inspiration
synapses
axon dendrites
The information transmission happens at the synapses.
![Page 15: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/15.jpg)
Semiconductors, BP&A Planning, 2003-01-29 15
Biological inspiration
The spikes travelling along the axon of the pre-synaptic neuron trigger the release of neurotransmitter substances at the synapse.
The neurotransmitters cause excitation or inhibition in the dendrite of the post-synaptic neuron.
The integration of the excitatory and inhibitory signals may produce spikes in the post-synaptic neuron.
The contribution of the signals depends on the strength of the synaptic connection.
![Page 16: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/16.jpg)
Semiconductors, BP&A Planning, 2003-01-29 16
Biological Neurons
• human information processing system consists of brain neuron: basic building block
– cell that communicates information to and from various parts of body
• Simplest model of a neuron: considered as a threshold unit –a processing element (PE)
• Collects inputs & produces output if the sum of the input exceeds an internal threshold value
![Page 17: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/17.jpg)
Semiconductors, BP&A Planning, 2003-01-29 17
Artificial Neural Nets (ANNs)
• Many neuron-like PEs units– Input & output units receive and broadcast signals to the
environment, respectively
– Internal units called hidden units since they are not in contact with external environment
– units connected by weighted links (synapses)
• A parallel computation system because– Signals travel independently on weighted channels & units
can update their state in parallel– However, most NNs can be simulated in serial computers
• A directed graph, with labeled edges by weights is typically used to describe the connections among units
![Page 18: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/18.jpg)
Semiconductors, BP&A Planning, 2003-01-29 18
Each processing unit has a simple program that: a) computes a weighted sum of the input data it receives from those units which feed into it b) outputs of a single value, which in general is a non-linear function of the weighted sum of the its inputs ---this output then becomes an input to those units into which the original units feeds
activationlevel
A NODE
inig
ai
inputfunctionactivation function
output
input linksoutputlinks
aj Wj,i
ai = g(ini)
![Page 19: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/19.jpg)
Semiconductors, BP&A Planning, 2003-01-29 19
g = Activation functions for units
Step function(Linear Threshold Unit)
Sign function Sigmoid function
step(x) = 1, if x >= threshold 0, if x < threshold
sign(x) = +1, if x >= 0 -1, if x < 0
sigmoid(x) = 1/(1+e-x)
![Page 20: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/20.jpg)
Semiconductors, BP&A Planning, 2003-01-29 20
Real vs artificial neurons
axon
dendrites
dendrites
synapse
cell
x0
xn
w0
wn
oi
n
ii xw
0
o/w 0 and 0 if 10
i
n
ii xwo
Threshold units
![Page 21: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/21.jpg)
Semiconductors, BP&A Planning, 2003-01-29 21
Artificial neurons
Neurons work by processing information. They receive and provide information in form of spikes.
The McCullogh-Pitts model
Inputs
Outputw2
w1
w3
wn
wn-1
. . .
x1
x2
x3
…
xn-1
xn
y)(;
1
zHyxwzn
iii
![Page 22: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/22.jpg)
Semiconductors, BP&A Planning, 2003-01-29 22
Mathematical representation
The neuron calculates a weighted sum of inputs and compares it to a threshold. If the sum is higher than the threshold, the output is set to 1, otherwise to -1.
Non-linearity
![Page 23: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/23.jpg)
Semiconductors, BP&A Planning, 2003-01-29 23
• x1
• x2
• xn
• …
• w1• w2
• …
• wn
i
n
iiwx
1threshold threshold • f
i
n
iin wxxxxf
121 if,1),...,,(
otherwise,0
Artificial neurons
![Page 24: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/24.jpg)
Semiconductors, BP&A Planning, 2003-01-29 24
![Page 25: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/25.jpg)
Semiconductors, BP&A Planning, 2003-01-29 25
Basic Concepts
Definition of a node:
• A node is an element which performs the function
y = fH(∑(wixi) + Wb)fH(x)
Input 0 Input 1 Input n...
W0 W1 Wn
+
Output
+
...
Wb
NodeNode
ConnectionConnection
![Page 26: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/26.jpg)
Semiconductors, BP&A Planning, 2003-01-29 26
Anatomy of an Artificial Neuron
bias
inputs
h(w0 ,wi , xi )
y f h
y
x1
w1
xi
wi
xn
wn
1
w0 f : activation function
output
h : combine wi & xi
![Page 27: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/27.jpg)
Semiconductors, BP&A Planning, 2003-01-29 27
Simple Perceptron
• Binary logic application• fH(x) = u(x) [linear
threshold]• Wi = random(-1,1)
• Y = u(W0X0 + W1X1 + Wb)
• Now how do we train it?
fH(x)
Input 0 Input 1
W0 W1
+
Output
Wb
![Page 28: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/28.jpg)
Semiconductors, BP&A Planning, 2003-01-29 28
• From experience: examples / training data
• Strength of connection between the neurons is stored as a weight-value for the specific connection.
• Learning the solution to a problem = changing the connection weights
Artificial Neuron
An artificial neuron
A physical neuron
![Page 29: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/29.jpg)
Semiconductors, BP&A Planning, 2003-01-29 29
Mathematical Representation
bw1
w2
wn
x1
x2
xn
+
b
x0
f(n)..
.
.
ny
Inputs Weights Summation Activation Output
Inputs
Outputw2
w1
wn. .
…
y1
net b
y f (net)
n
i ii
w x
+x2
xn
b
x1
![Page 30: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/30.jpg)
Semiconductors, BP&A Planning, 2003-01-29 30
![Page 31: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/31.jpg)
Semiconductors, BP&A Planning, 2003-01-29 31
![Page 32: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/32.jpg)
Semiconductors, BP&A Planning, 2003-01-29 32
A simple perceptron
• It’s a single-unit network• Change the weight by
an amount proportional to the difference between the desired output and the actual output.
Δ Wi = η * (D-Y).Ii
Perceptron Learning Rule
Learning rate Desired output
Input
Actual output
![Page 33: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/33.jpg)
Semiconductors, BP&A Planning, 2003-01-29 33
Linear Neurons
•Obviously, the fact that threshold units can only output the values 0 and 1 restricts their applicability to certain problems.
•We can overcome this limitation by eliminating the threshold and simply turning fi into the identity function so that we get:
)(net )( tto ii
•With this kind of neuron, we can build networks with m input neurons and n output neurons that compute a function
f: Rm Rn.
![Page 34: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/34.jpg)
Semiconductors, BP&A Planning, 2003-01-29 34
Linear Neurons
•Linear neurons are quite popular and useful for applications such as interpolation.
•However, they have a serious limitation: Each neuron computes a linear function, and therefore the overall network function f: Rm Rn is also linear.
•This means that if an input vector x results in an output vector y, then for any factor the input x will result in the output y.
•Obviously, many interesting functions cannot be realized by networks of linear neurons.
![Page 35: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/35.jpg)
Semiconductors, BP&A Planning, 2003-01-29 35
Mathematical Representation
nenfa
1
1)(
00
01)(
n
nnfa nnfa )(
2
( ) na f n e
![Page 36: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/36.jpg)
Semiconductors, BP&A Planning, 2003-01-29 36
Gaussian Neurons
•Another type of neurons overcomes this problem by using a Gaussian activation function:
•1
•0
•1
ffii(net(netii(t))(t))
netnetii(t)(t)•-1
2
1)(net
))(net(
t
ii
i
etf
![Page 37: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/37.jpg)
Semiconductors, BP&A Planning, 2003-01-29 37
Gaussian Neurons
•Gaussian neurons are able to realize non-linear functions.
•Therefore, networks of Gaussian units are in principle unrestricted with regard to the functions that they can realize.
•The drawback of Gaussian neurons is that we have to make sure that their net input does not exceed 1.
•This adds some difficulty to the learning in Gaussian networks.
![Page 38: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/38.jpg)
Semiconductors, BP&A Planning, 2003-01-29 38
Sigmoidal Neurons
•Sigmoidal neurons accept any vectors of real numbers as input, and they output a real number between 0 and 1.
•Sigmoidal neurons are the most common type of artificial neuron, especially in learning networks.
•A network of sigmoidal units with m input neurons and n output neurons realizes a network function f: Rm (0,1)n
![Page 39: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/39.jpg)
Semiconductors, BP&A Planning, 2003-01-29 39
Sigmoidal Neurons
•The parameter controls the slope of the sigmoid function, while the parameter controls the horizontal offset of the function in a way similar to the threshold neurons.
•1
•0
•1
ffii(net(netii(t))(t))
netnetii(t)(t)•-1
/))(net(1
1))(net( tii ie
tf
= 1
= 0.1
![Page 40: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/40.jpg)
Semiconductors, BP&A Planning, 2003-01-29 40
Example: A simple single unit adaptive network
• The network has 2 inputs, and one output. All are binary. The output is – 1 if W0I0 + W1I1 + Wb > 0 – 0 if W0I0 + W1I1 + Wb ≤ 0
• We want it to learn simple OR: output a 1 if either I0 or I1 is 1.
![Page 41: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/41.jpg)
Semiconductors, BP&A Planning, 2003-01-29 41
Artificial neurons
The McCullogh-Pitts model:
• spikes are interpreted as spike rates;
• synaptic strength are translated as synaptic weights;
• excitation means positive product between the incoming spike rate and the corresponding synaptic weight;
• inhibition means negative product between the incoming spike rate and the corresponding synaptic weight;
![Page 42: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/42.jpg)
Semiconductors, BP&A Planning, 2003-01-29 42
Artificial neurons
Nonlinear generalization of the McCullogh-Pitts neuron:
),( wxfy y is the neuron’s output, x is the vector of inputs, and w is the vector of synaptic weights.
Examples:
2
2
2
||||
1
1
a
wx
axw
ey
ey T
sigmoidal neuron
Gaussian neuron
![Page 43: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/43.jpg)
Semiconductors, BP&A Planning, 2003-01-29 43
NNs: Dimensions of a Neural Network
– Knowledge about the learning task is given in the form of examples called training examples.
– A NN is specified by:– an architecture: a set of neurons and links connecting
neurons. Each link has a weight, – a neuron model: the information processing unit of the
NN,– a learning algorithm: used for training the NN by
modifying the weights in order to solve the particular learning task correctly on the training examples.
The aim is to obtain a NN that generalizes well, that is, that behaves correctly on new instances of the learning task.
![Page 44: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/44.jpg)
Semiconductors, BP&A Planning, 2003-01-29 44
Neural Network Architectures
Many kinds of structures, main distinction made between two classes:
a) feed- forward (a directed acyclic graph (DAG): links are unidirectional, no cycles
b) recurrent: links form arbitrary topologies e.g., Hopfield Networks and Boltzmann machines
Recurrent networks: can be unstable, or oscillate, or exhibit chaoticbehavior e.g., given some input values, can take a long time to compute stable output and learning is made more difficult….However, can implement more complex agent designs and can
modelsystems with state
We will focus more on feed- forward networks
![Page 45: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/45.jpg)
Semiconductors, BP&A Planning, 2003-01-29 45
![Page 46: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/46.jpg)
Semiconductors, BP&A Planning, 2003-01-29 46
![Page 47: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/47.jpg)
Semiconductors, BP&A Planning, 2003-01-29 47
![Page 48: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/48.jpg)
Semiconductors, BP&A Planning, 2003-01-29 48
![Page 49: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/49.jpg)
Semiconductors, BP&A Planning, 2003-01-29 49
![Page 50: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/50.jpg)
Semiconductors, BP&A Planning, 2003-01-29 50
![Page 51: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/51.jpg)
Semiconductors, BP&A Planning, 2003-01-29 51
![Page 52: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/52.jpg)
Semiconductors, BP&A Planning, 2003-01-29 52
![Page 53: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/53.jpg)
Semiconductors, BP&A Planning, 2003-01-29 53
![Page 54: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/54.jpg)
Semiconductors, BP&A Planning, 2003-01-29 54
![Page 55: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/55.jpg)
Semiconductors, BP&A Planning, 2003-01-29 55
Single Layer Feed-forward
Input layerof
source nodes
Output layerof
neurons
![Page 56: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/56.jpg)
Semiconductors, BP&A Planning, 2003-01-29 56
Multi layer feed-forward
Inputlayer
Outputlayer
Hidden Layer
3-4-2 Network
![Page 57: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/57.jpg)
Semiconductors, BP&A Planning, 2003-01-29 57
Feed-forward networks:
Advantage: lack of cycles = > computation proceeds uniformly from input units to output units.
-activation from the previous time step plays no part in computation, as it is not fed back to an earlier unit
- simply computes a function of the input values that depends on the weight settings –it has no internal state other than the weights themselves.
- fixed structure and fixed activation function g: thus the functions representable by a feed-forward network are restricted to have acertain parameterized structure
![Page 58: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/58.jpg)
Semiconductors, BP&A Planning, 2003-01-29 58
Learning in biological systems
Learning = learning by adaptation
The young animal learns that the green fruits are sour, while the yellowish/reddish ones are sweet. The learning happens by adapting the fruit picking behavior.
At the neural level the learning happens by changing of the synaptic strengths, eliminating some synapses, and building new ones.
![Page 59: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/59.jpg)
Semiconductors, BP&A Planning, 2003-01-29 59
Learning as optimisation
The objective of adapting the responses on the basis of the information received from the environment is to achieve a better state. E.g., the animal likes to eat many energy rich, juicy fruits that make its stomach full, and makes it feel happy.
In other words, the objective of learning in biological organisms is to optimise the amount of available resources, happiness, or in general to achieve a closer to optimal state.
![Page 60: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/60.jpg)
Semiconductors, BP&A Planning, 2003-01-29 60
Synapse concept
• The synapse resistance to the incoming signal can be changed during a "learning" process [1949]
Hebb’s Rule: If an input of a neuron is repeatedly and
persistently causing the neuron to fire, a metabolic change happens in the synapse of that particular
input to reduce its resistance
![Page 61: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/61.jpg)
Semiconductors, BP&A Planning, 2003-01-29 61
Neural Network Learning
• Objective of neural network learning: given a set of examples, find parameter settings that minimize the error.
• Programmer specifies- numbers of units in each layer - connectivity between units,
• Unknowns- connection weights
![Page 62: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/62.jpg)
Semiconductors, BP&A Planning, 2003-01-29 62
Supervised Learning in ANNs
•In supervised learning, we train an ANN with a set of vector pairs, so-called exemplars.
•Each pair (x, y) consists of an input vector x and a corresponding output vector y.
•Whenever the network receives input x, we would like it to provide output y.
•The exemplars thus describe the function that we want to “teach” our network.
•Besides learning the exemplars, we would like our network to generalize, that is, give plausible output for inputs that the network had not been trained with.
![Page 63: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/63.jpg)
Semiconductors, BP&A Planning, 2003-01-29 63
Supervised Learning in ANNs
•There is a tradeoff between a network’s ability to precisely learn the given exemplars and its ability to generalize (i.e., inter- and extrapolate).
•This problem is similar to fitting a function to a given set of data points.
•Let us assume that you want to find a fitting function f:RR for a set of three data points.
•You try to do this with polynomials of degree one (a straight line), two, and nine.
![Page 64: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/64.jpg)
Semiconductors, BP&A Planning, 2003-01-29 64
Supervised Learning in ANNs
•Obviously, the polynomial of degree 2 provides the most plausible fit.
•f(x)
•x
•deg. 1
•deg. 2
•deg. 9
![Page 65: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/65.jpg)
Semiconductors, BP&A Planning, 2003-01-29 65
Overfitting
Overfitted ModelReal Distribution
![Page 66: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/66.jpg)
Semiconductors, BP&A Planning, 2003-01-29 66
![Page 67: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/67.jpg)
Semiconductors, BP&A Planning, 2003-01-29 67
![Page 68: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/68.jpg)
Semiconductors, BP&A Planning, 2003-01-29 68
![Page 69: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/69.jpg)
Semiconductors, BP&A Planning, 2003-01-29 69
![Page 70: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/70.jpg)
Semiconductors, BP&A Planning, 2003-01-29 70
Supervised Learning in ANNs
•The same principle applies to ANNs:
• If an ANN has too few neurons, it may not have enough degrees of freedom to precisely approximate the desired function.
• If an ANN has too many neurons, it will learn the exemplars perfectly, but its additional degrees of freedom may cause it to show implausible behavior for untrained inputs; it then presents poor ability of generalization.
•Unfortunately, there are no known equations that could tell you the optimal size of your network for a given application; you always have to experiment.
![Page 71: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/71.jpg)
Semiconductors, BP&A Planning, 2003-01-29 71
Learning in Neural Nets
Learning Tasks
Supervised UnsupervisedData:Labeled examples (input , desired output)
Tasks:classificationpattern recognition regressionNN models:perceptron adalinefeed-forward NN radial basis functionsupport vector machines
Data:Unlabeled examples (different realizations of the input)
Tasks:clusteringcontent addressable memory
NN models:self-organizing maps (SOM)Hopfield networks
![Page 72: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/72.jpg)
Semiconductors, BP&A Planning, 2003-01-29 72
Learning Algorithms
Depend on the network architecture:
• Error correcting learning (perceptron)• Delta rule (AdaLine, Backprop)• Competitive Learning (Self Organizing
Maps)
![Page 73: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/73.jpg)
Semiconductors, BP&A Planning, 2003-01-29 73
Perceptrons
• Perceptrons are single-layer feedforward networks• Each output unit is independent of the others• Can assume a single output unit• Activation of the output unit is calculated by:
• O = Step( )
where xj is the activation of input unit j, and we assume an additional weight and input to represent the threshold
n
j jxjw0
![Page 74: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/74.jpg)
Semiconductors, BP&A Planning, 2003-01-29 74
Perceptron
x1
x2
xn
w1
w2
wn
.
.
X0 = 1
w0
n
j jxjw0 > 01 if
-1 otherwise
O =
n
j jxjw0
![Page 75: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/75.jpg)
Semiconductors, BP&A Planning, 2003-01-29 75
![Page 76: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/76.jpg)
Semiconductors, BP&A Planning, 2003-01-29 76
Perceptron
Rosenblatt (1958) defined a perceptron to be a machine that learns, using examples, to assign input vectors (samples) to different classes, using linear functions of the inputs
Minsky and Papert (1969) instead describe perceptron as a stochastic gradient-descent algorithm that attempts to linearly separate a set of n-dimensional training data.
![Page 77: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/77.jpg)
Semiconductors, BP&A Planning, 2003-01-29 77
Linear Separable
++
+
-
-
-x1
x2
(a)
+
- +
-
x1
x2
some functions not representable - e.g., (b) not linearly separable
(b)
![Page 78: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/78.jpg)
Semiconductors, BP&A Planning, 2003-01-29 78
So what can be represented using perceptrons?
and or
Representation theorem: 1 layer feedforward networks canonly represent linearly separable functions. That is,the decision surface separating positive from negativeexamples has to be a plane.
![Page 79: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/79.jpg)
Semiconductors, BP&A Planning, 2003-01-29 79
Learning Boolean AND
![Page 80: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/80.jpg)
Semiconductors, BP&A Planning, 2003-01-29 80
XOR
• No w0, w1, w2 satisfy: (Minsky and Papert, 1969)
0
0
0
0
021
01
02
0
www
ww
ww
w
![Page 81: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/81.jpg)
Semiconductors, BP&A Planning, 2003-01-29 81
Expressive limits of perceptrons
• Can the XOR function be represented by a perceptron
(a network without a hidden layer)?
XOR cannot be represented.
![Page 82: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/82.jpg)
Semiconductors, BP&A Planning, 2003-01-29 82
How can perceptrons be designed?
• The Perceptron Learning Theorem (Rosenblatt, 1960): Given enough training examples, there is an algorithm that will learn any linearly separable function.
Theorem 1 (Minsky and Papert, 1969) The perceptron rule converges to weights that correctly classify all training examples provided the given data set represents a function that is linearly separable
![Page 83: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/83.jpg)
Semiconductors, BP&A Planning, 2003-01-29 83
The perceptron learning algorithm
• Inputs: training set {(x1,x2,…,xn,t)}• Method
– Randomly initialize weights w(i), -0.5<=i<=0.5– Repeat for several epochs until convergence:
• for each example– Calculate network output o.– Adjust weights:
iii
ii
www
xotw
)( Perceptron training
rule
learning rate error
![Page 84: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/84.jpg)
Semiconductors, BP&A Planning, 2003-01-29 84
Why does the method work?
• The perceptron learning rule performs gradient descent in weight space.– Error surface: The surface that describes the error on
each example as a function of all the weights in the network. A set of weights defines a point on this surface.
– We look at the partial derivative of the surface with respect to each weight (i.e., the gradient -- how much the error would change if we made a small change in each weight). Then the weights are being altered in an amount proportional to the slope in each direction (corresponding to a weight). Thus the network as a whole is moving in the direction of steepest descent on the error surface.
• The error surface in weight space has a single global minimum and no local minima. Gradient descent is guaranteed to find the global minimum, provided the learning rate is not so big that that you overshoot it.
![Page 85: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/85.jpg)
Semiconductors, BP&A Planning, 2003-01-29 85
Multi-layer, feed-forward networks
Perceptrons are rather weak as computing models since they can only learn linearly-separable functions.
Thus, we now focus on multi-layer, feed forward networks of non- linear sigmoid units: i.e.,
g(x) =
xe11
![Page 86: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/86.jpg)
Semiconductors, BP&A Planning, 2003-01-29 86
Multi-layer feed-forward networks
Multi-layer, feed forward networks extend perceptrons i.e., 1-layernetworks into n-layer by:• Partition units into layers 0 to L such that;
•lowermost layer number, layer 0 indicates the input units
•topmost layer numbered L contains the output units.
•layers numbered 1 to L are the hidden layers
•Connectivity means bottom-up connections only, with no cycles, hence the name"feed-forward" nets
•Input layers transmit input values to hidden layer nodes hence do notperform any computation.
Note: layer number indicates the distance of a node from the inputnodes
![Page 87: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/87.jpg)
Semiconductors, BP&A Planning, 2003-01-29 87
Multilayer feed forward network
x0 x1 x2 x3 x4
v1v2 v3
o1o2
Layer of input units
Layer of hidden units
Layer of output units
![Page 88: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/88.jpg)
Semiconductors, BP&A Planning, 2003-01-29 88
Multi-layer feed-forward networks
• Multi-layer feed-forward networks can be trained by back-propagation provided the activation function g is a differentiable function.– Threshold units don’t qualify, but the sigmoid function
does.
• Back-propagation learning is a gradient descent search through the parameter space to minimize the sum-of-squares error.
– Most common algorithm for learning algorithms in multilayer networks
![Page 89: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/89.jpg)
Semiconductors, BP&A Planning, 2003-01-29 89
Sigmoid units
x0
xn
w0
wn
oi
n
ii xw
0
Sigmoid unit for g
aea
1
1)(
))(1)(()(
aaa
a
This is g’ (the basis for gradient descent)
![Page 90: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/90.jpg)
Semiconductors, BP&A Planning, 2003-01-29 90
Weight updating in backprop • Learning in backprop is similar to learning with perceptrons, i.e.,
– Example inputs are fed to the network.
• If the network computes an output vector that matches the target, nothing is done.
• If there is a difference between output and target (i.e., an error), then the weights are adjusted to reduce this error.
• The key is to assess the blame for the error and divide it among the contributing weights.
• The error term (T - o) is known for the units in the output layer. To adjust the weights between the hidden and the output layer, the gradient descent rule can be applied as done for perceptrons.
• To adjust weights between the input and hidden layer some way of estimating the errors made by the hidden units in needed.
![Page 91: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/91.jpg)
Semiconductors, BP&A Planning, 2003-01-29 91
Estimating Error
• Main idea: each hidden node contributes for some fraction of the error in each of the output nodes. – This fraction equals the strength of the connection
(weight) between the hidden node and the output node.
ioutputsi
ijw
j nodehidden at error
where is the error at output node i.i
A goal of neural network learning is, given a set of examples, to find parameter settings that minimize the error
![Page 92: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/92.jpg)
Semiconductors, BP&A Planning, 2003-01-29 92
Back-propagation Learning
• Inputs: – Network topology: includes all units & their
connections – Some termination criteria – Learning Rate (constant of proportionality
of gradient descent search) – Initial parameter values– A set of classified training data
• Output: Updated parameter values
![Page 93: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/93.jpg)
Semiconductors, BP&A Planning, 2003-01-29 93
Back-propagation algorithm for updating weights in a multilayer network
1.Initialize the weights in the network (often randomly) 2.repeat for each example e in the training set do i.O = neural-net-output(network, e) ; forward pass ii.T = teacher output for e iii.Calculate error (T - O) at the output units iv.Compute wj = wj + * Err * Ij for all weights from
hidden layer to output layer;backward pass v.Compute wj = wj + * Err * Ij for all weights from input layer
to hidden layer; backward pass continued vi.Update the weights in the network end 3.until all examples classified correctly or stopping criterion met 4.return(network)
![Page 94: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/94.jpg)
Semiconductors, BP&A Planning, 2003-01-29 94
Back-propagation Using Gradient Descent
• Advantages– Relatively simple implementation– Standard method and generally works
well
• Disadvantages– Slow and inefficient– Can get stuck in local minima resulting
in sub-optimal solutions
![Page 95: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/95.jpg)
Semiconductors, BP&A Planning, 2003-01-29 95
Number of training pairs needed?
Difficult question. Depends on the problem, the training examples, andnetwork architecture. However, a good rule of thumb is:
Where W = No. of weights; P = No. of training pairs, e = error rate
For example, for e = 0.1, a net with 80 weights will require 800training patterns to be assured of getting 90% of the test patternscorrect (assuming it got 95% of the training examples correct).
epw
![Page 96: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/96.jpg)
Semiconductors, BP&A Planning, 2003-01-29 96
How long should a net be trained?
• The objective is to establish a balance between correct responses for the training patterns and correct responses for new patterns. (a balance between memorization and generalization).
• If you train the net for too long, then you run the risk of overfitting.
• In general, the network is trained until it reaches an acceptable error rate (e.g., 95%)
![Page 97: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/97.jpg)
Semiconductors, BP&A Planning, 2003-01-29 97
Implementing Backprop – Design Decisions
1. Choice of r2. Network architecture
a) How many Hidden layers? how many hidden units per a layer?b) How should the units be connected? (e.g., Fully, Partial, using
domain knowledge3. Stopping criterion – when should training stop?
![Page 98: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/98.jpg)
Semiconductors, BP&A Planning, 2003-01-29 98
Determining optimal network structure
Weak point of fixed structure networks: poor choice can lead to poor performance
Too small network: model incapable of representing the desired Function
Too big a network: will be able to memorize all examples but forming a large lookup table, but will not generalize well to inputs that have not been seen before.
Thus finding a good network structure is another example of asearch problems.Some approaches to search for a solution for this problem includeGenetic algorithmsBut using GAs is very cpu-intensive.
![Page 99: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/99.jpg)
Semiconductors, BP&A Planning, 2003-01-29 99
Learning rate
• Ideally, each weight should have its own learning rate
• As a substitute, each neuron or each layer could have its own rate
• Learning rates should be proportional to the sqrt of the number of inputs to the neuron
![Page 100: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/100.jpg)
Semiconductors, BP&A Planning, 2003-01-29 100
Setting the parameter values
• How are the weights initialized?• Do weights change after the presentation of
each pattern or only after all patterns of the training set have been presented?
• How is the value of the learning rate chosen?• When should training stop?• How many hidden layers and how many nodes
in each hidden layer should be chosen to build a feedforward network for a given problem?
• How many patterns should there be in a training set?
• How does one know that the network has learnt something useful?
![Page 101: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/101.jpg)
Semiconductors, BP&A Planning, 2003-01-29 101
When should neural nets be used for learning a problem
• If instances are given as attribute-value pairs.– Pre-processing required: Continuous
input values to be scaled in [0-1] range, and discrete values need to converted to Boolean features.
• Noise in training examples.
• If long training time is acceptable.
![Page 102: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/102.jpg)
Semiconductors, BP&A Planning, 2003-01-29 102
Neural Networks: Advantages
•Distributed representations •Simple computations
•Robust with respect to noisy data
•Robust with respect to node failure
•Empirically shown to work well for many problem domains
•Parallel processing
![Page 103: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/103.jpg)
Semiconductors, BP&A Planning, 2003-01-29 103
Neural Networks: Disadvantages•Training is slow
•Interpretability is hard
•Network topology layouts ad hoc
•Can be hard to debug
•May converge to a local, not global, minimum of error
•May be hard to describe a problem in terms of features with numerical values
![Page 104: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/104.jpg)
Semiconductors, BP&A Planning, 2003-01-29 104
Back-propagation Algorithm
yj(n)
y0=+1
y1(n)
yi(n) wji(n)
wj1(n)
wj0(n)=bj (n)
∑ netj(n) f(.)
0
( ) ( ) ( )
( ) ( ( ))
( ) ( ) ( )
m
j ji ii
j j j
j j j
net n w n y n
y n f net n
e n d n y n
Total error )(2
1)( 2 nen
Cj
All output neurons
N
nav n
N 1
)(1 Average squared error where N=No. of items in the training set
![Page 105: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/105.jpg)
Semiconductors, BP&A Planning, 2003-01-29 105
( ) ( ) ( )( ) ( )
( ) ( ) ( ) ( ) ( )j j j
ji j j j ji
e n y n net nn n
w n e n y n net n w n
( )( ) ( ( )) ( )
( ) j j j iji
ne n v n y n
w n
)()(
)(ne
ne
nj
j
)(2
1)( 2 nen
Cj
as
1)(
)(
ny
ne
j
j
as
)()()( nyndne jjj
( )' ( ( ))
( )j
j jj
y nf net n
net n
as
( ) ( ( ))j j jy n f net n
( )( )
( )j
iji
net ny n
w n
as
0
( ) ( ) ( )m
j ji ii
net n w n y n
Back-propagation Algorithm
Gradient decent
)()()( nynnw ijji
Error term
where ( ) ( ) '( ( ))j j jn e n f net n
)()()( nynnw ijji 1
( ( ))1 exp( ( ))j j
j
net nnet n
if
as
'( ( )) ( )[1 ( )]j j jnet n y n y n
( ) ( ( ))j j jy n net n
![Page 106: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/106.jpg)
Semiconductors, BP&A Planning, 2003-01-29 106
Back-propagation Algorithm
Neuron k is an output node
( ) ( ) '( ( )) [ ( ) ( )] ( )[1 ( )]k k k k k k kn e n net n d n y n y n y n
( ) '( ( )) ( ) ( )
( )[1 ( )] ( ) ( )
j j k kjk
j j k kjk
n net n n w n
y n y n n w n
Neuron j is a hidden node
Output layer (k)Hidden layer(j)
1
2
jw1
jw2
j
)()()( nynnw ijji
Weight adjustment
Learningrate
Localgradient
Input signal
( 1) ( ) ( )ji ji jiw n w n w n
Output of neuron k
![Page 107: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/107.jpg)
Semiconductors, BP&A Planning, 2003-01-29 107
![Page 108: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/108.jpg)
Semiconductors, BP&A Planning, 2003-01-29 108
-3.0
00
-2.0
00
-1.0
00
0.00
0
1.00
0
2.00
0
3.00
0
4.00
0
5.00
0
6.00
0
-3.0
00
-2.0
00
-1.0
00
0.00
0
1.00
0
2.00
0
3.00
0
4.00
0
5.00
0
6.00
00.0
2.0
4.0
6.0
8.0
10.0
12.0
14.0
Error
W1W2
![Page 109: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/109.jpg)
Semiconductors, BP&A Planning, 2003-01-29 109
Brain vs. Digital Computers (1)
Computers require hundreds of cycles to simulate a firing of a neuron
The brain can fire all the neurons in a single step. ParallelismSerial computers require billions of cycles to
perform some tasks but the brain takes less than a seconde.g. Face Recognition
![Page 110: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/110.jpg)
Semiconductors, BP&A Planning, 2003-01-29 110
What are Neural Networks
• An interconnected assembly of simple processing elements, units, neurons or nodes, whose functionality is loosely based on the animal neuron
• The processing ability of the network is stored in the interunit connection strengths, or weights, obtained by a process of adaptation to, or learning from, a set of training patterns.
![Page 111: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/111.jpg)
Semiconductors, BP&A Planning, 2003-01-29 111
Definition of Neural Network
A Neural Network is a system composed of manysimple processing elements operating in parallelwhich can acquire, store, and utilize experientialKnowledge.
![Page 112: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/112.jpg)
Semiconductors, BP&A Planning, 2003-01-29 112
Architecture
• Connectivity:– fully connected– partially connected
• Feedback– feedforward network: no feedback
• simpler, more stable, proven most useful
– recurrent network: feedback from output to input units
• complex dynamics, may be unstable
• Number of layers i.e. presence of hidden layers
![Page 113: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/113.jpg)
Semiconductors, BP&A Planning, 2003-01-29 113
Input
Layer
Hidden
Layer
Output
Layer
Inputs
NodeConnection
Outputs
Feedforward, Fully-Connected with One Hidden Layer
![Page 114: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/114.jpg)
Semiconductors, BP&A Planning, 2003-01-29 114
Hidden Units
• Layer of nodes between input and output nodes
• Allow a network to learn non-linear functions
• Allow the net to represent combinations of the input features
![Page 115: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/115.jpg)
Semiconductors, BP&A Planning, 2003-01-29 115
Learning Algorithms
• How the network learns the relationship between the inputs and outputs
• Type of algorithm used depends on type of network- architecture, type of learning,etc.
• Back Propagation: most popular– modifications exist: quick prop, Delta-bar-Delta
• Others: Conjugate gradient descent, Levenberg-Marquardt, K-Means, Kohonen, standard pseudo-inverse (SVD) linear optimization
![Page 116: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/116.jpg)
Semiconductors, BP&A Planning, 2003-01-29 116
Types of Networks
• Multilayer Perceptron• Radial Basis Function• Kohonen• Linear• Hopfield• Adaline/Madaline• Probabilistic Neural Network (PNN)• General Regression Neural Network
(GRNN) • and at least thirty others
![Page 117: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/117.jpg)
Semiconductors, BP&A Planning, 2003-01-29 117
• A Neural Network is a system composed of many simple processing elements operating in parallel which can acquire, store, and utilize experiential knowledge
• Basic Artificial Model– Consists of simple processing elements called neurons, units or
nodes– Each neuron is connected to other nodes with an associated
weight(strength) which typically multiplies the signal transmitted. Each neuron has a single threshold value
• Characterization– Architecture: the pattern of nodes and connections between
them– Learning algorithm, or training method: method for
determining weights of the connections– Activation function: function that produces an output based on
the input values received by node
![Page 118: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/118.jpg)
Semiconductors, BP&A Planning, 2003-01-29 118
Perceptrons
• First studied in the late 1950s• Also known as Layered Feed-Forward
Networks• The only efficient learning element at that
time was for single-layered networks• Today, used as a synonym for a single-
layer, feed-forward network
![Page 119: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/119.jpg)
Semiconductors, BP&A Planning, 2003-01-29 119
Perceptrons
• First studied in the late 1950s• Also known as Layered Feed-Forward
Networks• The only efficient learning element at that
time was for single-layered networks• Today, used as a synonym for a single-
layer, feed-forward network
![Page 120: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/120.jpg)
Semiconductors, BP&A Planning, 2003-01-29 120
Single Layer Perceptron
OUT = F(NET)OUT
X1
X2 • • • Xn
w1
w2
wn
OUT = F(NET)
X1
X2 • • • Xn
w1
w2
wn
Squashing function need not be sigmoidal
![Page 121: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/121.jpg)
Semiconductors, BP&A Planning, 2003-01-29 121
Perceptron Architecture
![Page 122: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/122.jpg)
Semiconductors, BP&A Planning, 2003-01-29 122
Single-Neuron Perceptron
![Page 123: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/123.jpg)
Semiconductors, BP&A Planning, 2003-01-29 123
Decision Boundary
![Page 124: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/124.jpg)
Semiconductors, BP&A Planning, 2003-01-29 124
Example OR
![Page 125: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/125.jpg)
Semiconductors, BP&A Planning, 2003-01-29 125
OR solution
![Page 126: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/126.jpg)
Semiconductors, BP&A Planning, 2003-01-29 126
Multiple-Neuron Perceptron
![Page 127: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/127.jpg)
Semiconductors, BP&A Planning, 2003-01-29 127
Learning Rule Test Problem
![Page 128: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/128.jpg)
Semiconductors, BP&A Planning, 2003-01-29 128
Starting Point
![Page 129: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/129.jpg)
Semiconductors, BP&A Planning, 2003-01-29 129
Tentative Learning Rule
![Page 130: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/130.jpg)
Semiconductors, BP&A Planning, 2003-01-29 130
Second Input Vector
![Page 131: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/131.jpg)
Semiconductors, BP&A Planning, 2003-01-29 131
Third Input Vector
![Page 132: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/132.jpg)
Semiconductors, BP&A Planning, 2003-01-29 132
Unified Learning Rule
![Page 133: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/133.jpg)
Semiconductors, BP&A Planning, 2003-01-29 133
Multiple-Neuron Perceptron
![Page 134: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/134.jpg)
Semiconductors, BP&A Planning, 2003-01-29 134
Apple / Banana Example
![Page 135: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/135.jpg)
Semiconductors, BP&A Planning, 2003-01-29 135
Apple / Banana Example, Second iteration
![Page 136: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/136.jpg)
Semiconductors, BP&A Planning, 2003-01-29 136
Apple / Banana Example, Check
![Page 137: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/137.jpg)
Semiconductors, BP&A Planning, 2003-01-29 137
Historical Note
There was great interest in Perceptrons in the '50s and '60s - centred on the work of Rosenblatt.
This was crushed by the publication of "Perceptrons" by Minsky and Papert.
• EOR problem
regions must be linearly separable.
(0,1)
(0,0)
(1,1)
(1,0)
• Training Problems
esp. with higher order nets.
![Page 138: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/138.jpg)
Semiconductors, BP&A Planning, 2003-01-29 138
Multilayer Perceptron(MLP)
• Type of Back Propagation Network• Arguably the most popular network architecture
today• Can model functions of almost arbitrary
complexity, with number of layers and number of units/layer determining the function complexity
• Number of hidden layers and units: good starting point is to use one hidden layer with the number of units equal to half the sum of the number of input and output units
![Page 139: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/139.jpg)
Semiconductors, BP&A Planning, 2003-01-29 139
Perceptrons
![Page 140: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/140.jpg)
Semiconductors, BP&A Planning, 2003-01-29 140
Network Structures
Feed-forward:Links can only go in one direction.
Recurrent : Links can go anywhere and form arbitrary
topologies.
![Page 141: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/141.jpg)
Semiconductors, BP&A Planning, 2003-01-29 141
Feed-forward Networks
• Arranged in layers• Each unit is linked only in the unit in next layer• No units are linked between the same layer,
back to the previous layer or skipping a layer• Computations can proceed uniformly from input
to output units• No internal state exists
![Page 142: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/142.jpg)
Semiconductors, BP&A Planning, 2003-01-29 142
I2
W24= -1
H4
W46 = 1
H6
W67 = 1
I1
W13 = -1
H3
W35 = 1
H5
O7
W57 = 1
W25 = 1
W16 = 1
Feed-Forward Example
![Page 143: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/143.jpg)
Semiconductors, BP&A Planning, 2003-01-29 143
Introduction to Backpropagation
• In 1969 a method for learning in multi-layer network, Backpropagation, was invented by Bryson and Ho
• The Backpropagation algorithm is a sensible approach for dividing the contribution of each weight
• Works basically the same as perceptrons
![Page 144: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/144.jpg)
Semiconductors, BP&A Planning, 2003-01-29 144
![Page 145: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/145.jpg)
Semiconductors, BP&A Planning, 2003-01-29 145
![Page 146: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/146.jpg)
Semiconductors, BP&A Planning, 2003-01-29 146
![Page 147: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/147.jpg)
Semiconductors, BP&A Planning, 2003-01-29 147
![Page 148: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/148.jpg)
Semiconductors, BP&A Planning, 2003-01-29 148
Backpropagation Learning
• There are two differences for the updating rule :– The activation of the hidden unit is used
instead of the input value– The rule contains a term for the gradient of
the activation function
![Page 149: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/149.jpg)
Semiconductors, BP&A Planning, 2003-01-29 149
Backpropagation Learning
• There are two differences for the updating rule :– The activation of the hidden unit is used
instead of the input value– The rule contains a term for the gradient of
the activation function
![Page 150: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/150.jpg)
Semiconductors, BP&A Planning, 2003-01-29 150
Backpropagation Algorithm Summary
The ideas of the algorithm can be summarized as follows :
• Computes the error term for the output units using the observed error
• From output layer, repeat propagating the error term back to the previous layer and updating the weights between the two layers until the earliest hidden layer is reached
![Page 151: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/151.jpg)
Semiconductors, BP&A Planning, 2003-01-29 151
Back Propagation
• Best-known example of a training algorithm• Uses training data to adjust weights and thresholds
of neurons so as to minimize the networks errors of prediction
• Slower than modern second-order algorithms such as gradient descent & Levenberg-Marquardt for many problems
• Still has some advantages over these in some instances
• Easiest algorithm to understand• Heuristic modifications exist: quick propagation and
Delta-bar-Delta
![Page 152: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/152.jpg)
Semiconductors, BP&A Planning, 2003-01-29 152
Training a BackPropagation Net
• Feedforward training of input patterns– each input node receives a signal, which is
broadcast to all of the hidden units– each hidden unit computes its activation which is
broadcast to all output nodes
• Back propagation of errors– each output node compares its activation with the
desired output– based on these differences, the error is propagated
back to all previous nodes
• Adjustment of weights– weights of all links computed simultaneously based
on the errors that were propagated back
![Page 153: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/153.jpg)
Semiconductors, BP&A Planning, 2003-01-29 153
Training Back Prop Net: Feedforward Stage
1. Initialize weights with small, random values2. While stopping condition is not true
– for each training pair (input/output):• each input unit broadcasts its value to all hidden units• each hidden unit sums its input signals & applies
activation function to compute its output signal• each hidden unit sends its signal to the output units• each output unit sums its input signals & applies its
activation function to compute its output signal
![Page 154: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/154.jpg)
Semiconductors, BP&A Planning, 2003-01-29 154
Training Back Prop Net: Backpropagation
3. Each output computes its error term, its own weight correction term and its bias(threshold) correction term & sends it to layer below
4. Each hidden unit sums its delta inputs from above & multiplies by the derivative of its activation function; it also computes its own weight correction term and its bias correction term
![Page 155: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/155.jpg)
Semiconductors, BP&A Planning, 2003-01-29 155
Training a Back Prop Net: Adjusting the Weights
5. Each output unit updates its weights and bias6. Each hidden unit updates its weights and bias
– Each training cycle is called an epoch. The weights are updated in each cycle
– It is not analytically possible to determine where the global minimum is. Eventually the algorithm stops in a low point, which may just be a local minimum.
![Page 156: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/156.jpg)
Semiconductors, BP&A Planning, 2003-01-29 156
Gradient Descent Methods
• Error Function: How far off are we?– Example Error function:
depends on weight values• Gradient Descent: Minimize error by
moving weights along the decreasing slope of error
• The Idea: iterate through the training set and adjust the weights to minimize the gradient of the error
ΞΧ
ii
i
fdε 2
![Page 157: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/157.jpg)
Semiconductors, BP&A Planning, 2003-01-29 157
Gradient Descent: The Math
We have = (d - f)2
Gradient of :
Using the chain rule:
Since , we have
Also:
Which finally gives:
11
,...,,...,ni wwwW
W
s
sW
W
s
sW
s
ffd
s
)(2
s
ffd
W)(2
![Page 158: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/158.jpg)
Semiconductors, BP&A Planning, 2003-01-29 158
Gradient Descent: Back to reality
• So we have • The problem: f / s is not
differentiable• Three solutions:
– Ignore It: The Error-Correction Procedure– Fudge It: Widrow-Hoff– Approximate it: The Generalized Delta
Procedure
s
ffd
W)(2
![Page 159: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/159.jpg)
Semiconductors, BP&A Planning, 2003-01-29 159
Training a Back Prop Net: Adjusting the Weights
5. Each output unit updates its weights and bias6. Each hidden unit updates its weights and
bias– Each training cycle is called an epoch. The weights
are updated in each cycle– It is not analytically possible to determine where the
global minimum is. Eventually the algorithm stops in a low point, which may just be a local minimum.
![Page 160: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/160.jpg)
Semiconductors, BP&A Planning, 2003-01-29 160
How long should you train?
• Goal: balance between correct responses for training patterns & correct responses for new patterns (memorization v. generalization)
• In general, network is trained until it reaches an acceptable error rate (e.g. 95%)
• If train too long, you run the risk of overfitting
![Page 161: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/161.jpg)
Semiconductors, BP&A Planning, 2003-01-29 161
Learning in the BPN
•Before the learning process starts, all weights (synapses) in the network are initialized with pseudorandom numbers.
•We also have to provide a set of training patterns (exemplars). They can be described as a set of ordered vector pairs {(x1, y1), (x2, y2), …, (xP, yP)}.
•Then we can start the backpropagation learning algorithm.
•This algorithm iteratively minimizes the network’s error by finding the gradient of the error surface in weight-space and adjusting the weights in the opposite direction (gradient-descent technique).
![Page 162: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/162.jpg)
Semiconductors, BP&A Planning, 2003-01-29 162
Learning in the BPN•Gradient-descent example: Finding the absolute minimum of a one-dimensional error function f(x):
f(x)
xx0
slope: f’(x0)
x1 = x0 - f’(x0)
•Repeat this iteratively until for some xi, f’(xi) is sufficiently close to 0.
![Page 163: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/163.jpg)
Semiconductors, BP&A Planning, 2003-01-29 163
Learning in the BPN•Gradients of two-dimensional functions:
•The two-dimensional function in the left diagram is represented by contour lines in the right diagram, where arrows indicate the gradient of the function at different locations. Obviously, the gradient is always pointing in the direction of the steepest increase of the function. In order to find the function’s minimum, we should always move against the gradient.
![Page 164: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/164.jpg)
Semiconductors, BP&A Planning, 2003-01-29 164
Gradient Descent
(w1,w2)
(w1+w1,w2 +w2)
![Page 165: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/165.jpg)
Semiconductors, BP&A Planning, 2003-01-29 165
Learning in the BPN
• In the BPN, learning is performed as follows:
1. Randomly select a vector pair (xp, yp) from the training set and call it (x, y).
2. Use x as input to the BPN and successively compute the outputs of all neurons in the network (bottom-up) until you get the network output o.
3. Compute the error opk, for the pattern p across all K
output layer units by using the formula:
)(')( okkk
opk netfoy
![Page 166: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/166.jpg)
Semiconductors, BP&A Planning, 2003-01-29 166
Learning in the BPN
4. Compute the error hpj, for all J hidden layer units by
using the formula:
kj
K
k
opk
hk
hpj wnetf
1
)('
5. Update the connection-weight values to the hidden layer by using the following equation:
ihpjjiji xtwtw )()1(
![Page 167: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/167.jpg)
Semiconductors, BP&A Planning, 2003-01-29 167
Learning in the BPN
•Repeat steps 1 to 6 for all vector pairs in the training set; this is called a training epoch.
•Run as many epochs as required to reduce the network error E to fall below a threshold :
6. Update the connection-weight values to the output layer by using the following equation:
)()()1( hj
opkkjkj netftwtw
2
1 1
)(
P
p
K
k
opkE
![Page 168: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/168.jpg)
Semiconductors, BP&A Planning, 2003-01-29 168
Learning in the BPN
•The only thing that we need to know before we can start our network is the derivative of our sigmoid function, for example, f’(netk) for the output neurons:
kef k net1
1)net(
)1(net
)net()net(' kk
k
kk oo
ff
![Page 169: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/169.jpg)
Semiconductors, BP&A Planning, 2003-01-29 169
Learning in the BPN
•Now our BPN is ready to go!
•If we choose the type and number of neurons in our network appropriately, after training the network should show the following behavior:
• If we input any of the training vectors, the network should yield the expected output vector (with some margin of error).
• If we input a vector that the network has never “seen” before, it should be able to generalize and yield a plausible output vector based on its knowledge about similar input vectors.
![Page 170: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/170.jpg)
Semiconductors, BP&A Planning, 2003-01-29 170
Weights and Gradient Descent
• Calculate the partial derivatives of the error E with respect to each of the weights:
• Minimize E by gradient descent: change each weight by an amount proportional to the partial derivative
wE
E
w
If slope is negative increase wIf slope is positive decrease w
Local minima for E are places wherederivative equals zero
wEw
Gradient descent
![Page 171: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/171.jpg)
Semiconductors, BP&A Planning, 2003-01-29 171
Gradient descent
Local minimum
Global minimum
Error
![Page 172: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/172.jpg)
Semiconductors, BP&A Planning, 2003-01-29 172
Faster Convergence: Momentum rule• Add a fraction (=momentum) of the last change to the
current change
)1()( twwEtw
![Page 173: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/173.jpg)
Semiconductors, BP&A Planning, 2003-01-29 173
Back-propagation: a simple “chain rule” procedure for updating all weights
output
input
hidden
Weight updates for hidden to output weights wo are easy to calculate. For sigmoid activation function
hi
oj
oji yw wo
wh Weight updates for input to hidden layer require “back-propagated” quantities
oj
oj
oj
oj
oj yyyd 1
oy1oy2
hy1hy2
1i 2i
j
oj
oji
hi
hi
hi wyy 1
khj
hjk iw
(a.k.a delta rule)
![Page 174: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/174.jpg)
Semiconductors, BP&A Planning, 2003-01-29 174
Procedure for two-layer network (with sigmoid activation functions)
1. Initialize all weights to small random values
2. Choose an input pattern c and set the input nodes to that pattern
3. Propagate signal forward by calculating hidden node activation
4. Propagate signal forward to output nodes
5. Compute deltas for the output node layer
6. Compute deltas for the hidden node layer
7. Update weights immediately after this pattern (no waiting to accumulate weight changes over all patterns)
k
hikk
hi wiy exp11
i
ojii
oj wyy exp11
oj
oj
oj
oj
oj yyyd 1
j
oj
oji
hi
hi
hi wyy 1
hi
oj
oji yw
khj
hjk iw
![Page 175: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/175.jpg)
Artificial Neural Networks
Perceptrons
o(x1,x2...,xn) = 1 if w0+ w1 x1+.. + wn xn > 0
-1 otherwise
o(x) = sgn(w.x) (x0=1)
Hypothesis Space: H = {w | w n+1}
![Page 176: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/176.jpg)
Semiconductors, BP&A Planning, 2003-01-29 176
![Page 177: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/177.jpg)
Semiconductors, BP&A Planning, 2003-01-29 177
Artificial Neural Networks
• Representational Power
– Perceptrons can represent all the primitive Boolean functions AND, OR, NAND (AND) and NOR (OR)
– They cannot represent all Boolean functions (for example, XOR)
– Every Boolean function can be represented by some network of perceptrons two levels deep
![Page 178: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/178.jpg)
Semiconductors, BP&A Planning, 2003-01-29 178
Artificial Neural Networks
![Page 179: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/179.jpg)
Semiconductors, BP&A Planning, 2003-01-29 179
• The Perceptron Training Rule
wi wi + wi wi = (t - o) xi
t: target output for the current training example
o: output generated by the perceptron
: learning rate
Artificial Neural Networks
![Page 180: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/180.jpg)
Semiconductors, BP&A Planning, 2003-01-29 180
Multilayer Networks and the BP Algorithm
ANNs with two or more layers are able to represent complex nonlinear decision surfaces
– Differentiable Threshold (Sigmoid) Units
o = (w.x) (y) = 1/(1+e-y)
/y = (y) [1-(y)]
![Page 181: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/181.jpg)
Semiconductors, BP&A Planning, 2003-01-29 181
![Page 182: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/182.jpg)
Semiconductors, BP&A Planning, 2003-01-29 182
![Page 183: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/183.jpg)
Semiconductors, BP&A Planning, 2003-01-29 183
![Page 184: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/184.jpg)
Semiconductors, BP&A Planning, 2003-01-29 184
![Page 185: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/185.jpg)
Semiconductors, BP&A Planning, 2003-01-29 185
![Page 186: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/186.jpg)
Semiconductors, BP&A Planning, 2003-01-29 186
First find the outputs OI , OII . In order to do this, propagate the inputs forward. First find the outputs for the neurons of hidden layer
)1.()(
)1.()(
)1.()(
2321310
3033
2221210
2022
2121110
1011
XWXWWOXWOO
XWXWWOXWOO
XWXWWOXWOO
iii
iii
iii
Example: Learning addition
![Page 187: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/187.jpg)
Semiconductors, BP&A Planning, 2003-01-29 187
)1.()(
)1.()(
3322110
0
3322110
0
OWOWOWWOOWOO
OWOWOWWOOWOO
IIIIIIi
IIiIIiII
IIIi
IiIiI
Then find the outputs of the neurons of output layer
Example: Learning addition
![Page 188: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/188.jpg)
Semiconductors, BP&A Planning, 2003-01-29 188
Now propagate back the errors. In order to do that first find the errors for the output layer, also update the weights between hidden layer and output layer
))(1(
))(1(
IIIIIIIIII
IIIII
OtOO
OtOO
33
22
11
0
OW
OW
OW
W
II
II
II
II
33
22
11
0
OW
OW
OW
W
IIII
IIII
IIII
IIII
Example: Learning addition
![Page 189: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/189.jpg)
Semiconductors, BP&A Planning, 2003-01-29 189
And backpropagate the errors to hidden layer.
Example: Learning addition
![Page 190: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/190.jpg)
Semiconductors, BP&A Planning, 2003-01-29 190
Example: Learning addition
))(1()1(
))(1()1(
))(1()1(
33331
3333
22221
2222
11111
1111
IIIIIIk
kk
IIIIIIk
kk
IIIIIIk
kk
WWOOWOO
WWOOWOO
WWOOWOO
2112
1111
10110
XW
XW
XW
2222
1221
20220
XW
XW
XW
2332
1331
30330
XW
XW
XW
And backpropagate the errors to hidden layer.
![Page 191: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/191.jpg)
Semiconductors, BP&A Planning, 2003-01-29 191333
222
111
000
333
222
111
000
323232
313131
303030
222222
212121
202020
121212
111111
101010
IIIIII
IIIIII
IIIIII
IIIIII
III
III
III
III
WWW
WWW
WWW
WWW
WWW
WWW
WWW
WWW
WWW
WWW
WWW
WWW
WWW
WWW
WWW
WWW
WWW
Example: Learning addition
Finally update weights!!!!
![Page 192: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/192.jpg)
Semiconductors, BP&A Planning, 2003-01-29 192
![Page 193: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/193.jpg)
Semiconductors, BP&A Planning, 2003-01-29 193
![Page 194: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/194.jpg)
Semiconductors, BP&A Planning, 2003-01-29 194
![Page 195: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/195.jpg)
Semiconductors, BP&A Planning, 2003-01-29 195
![Page 196: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/196.jpg)
Semiconductors, BP&A Planning, 2003-01-29 196
![Page 197: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/197.jpg)
Semiconductors, BP&A Planning, 2003-01-29 197
![Page 198: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/198.jpg)
Semiconductors, BP&A Planning, 2003-01-29 198
![Page 199: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/199.jpg)
Semiconductors, BP&A Planning, 2003-01-29 199
![Page 200: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/200.jpg)
Semiconductors, BP&A Planning, 2003-01-29 200
![Page 201: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/201.jpg)
Semiconductors, BP&A Planning, 2003-01-29 201
![Page 202: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/202.jpg)
Semiconductors, BP&A Planning, 2003-01-29 202
![Page 203: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/203.jpg)
Semiconductors, BP&A Planning, 2003-01-29 203
![Page 204: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/204.jpg)
Semiconductors, BP&A Planning, 2003-01-29 204
![Page 205: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/205.jpg)
Semiconductors, BP&A Planning, 2003-01-29 205
![Page 206: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/206.jpg)
Semiconductors, BP&A Planning, 2003-01-29 206
![Page 207: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/207.jpg)
Semiconductors, BP&A Planning, 2003-01-29 207
Gradient-Descent Training Rule
•As described in Machine Learning (Mitchell, 1997).•Also called delta rule, although not quite the same as the adaline delta rule. •Compute Ei, the error at output node i over the set of training instances, D.
ij
iij
w
Ew
Intuitive: Do what you can to reduce Ei, so: If increases in wij will increase Ei (i.e., dEi/dwij > 0), then reduce wij, but ” ” decrease Ei ” ” < 0, then increase wij
• Compute dEi/dwij (i.e. wij’s contribution to error at node i) for every input weight to node i.
• Gradient Descent Method: Updating all wij by the delta rule amounts to moving along the path of steepest descent on the error surface.
• Difficult part: computing dEi/dwij .
• Base weight updates on dEi/dwij
2)(
2
1
Dd
ididi otE D = training set
= distance * direction (to move in error space)
![Page 208: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/208.jpg)
Semiconductors, BP&A Planning, 2003-01-29 208
Computing dEi/dwij
2)(
2
1
Ddidid
ijij
i otww
E
)()(22
1idid
ijDdidid ot
wot
)()( idijDd
idid ow
ot
))(()( idTijDd
idid sumfw
ot
jjdijid xwsumwhere
ij
id
id
TidT
ij wsum
sumf
sumfw
)( jdid
T xsum
f
In general:
![Page 209: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/209.jpg)
Semiconductors, BP&A Planning, 2003-01-29 209
Computing )( idTij
sumfw
idsumidT esumf
1
1)(Sigmoidal ft :
))(1)(()1(1
12 idtidtsum
sum
sumid
sumfsumfe
eesum id
id
id
ididT osumf )(But since: jdidididTij
xoosumfw
)1()(
ididT sumsumf )(Identity ft :
1
id
idT
id
sumsum
fsum
jdjdij
id
id
TidT
ij
xxw
sumsum
fsumf
w
)1()(
If fT is not continuous, andhence not differentiableeverywhere, then we cannotuse the Delta Rule.
![Page 210: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/210.jpg)
Semiconductors, BP&A Planning, 2003-01-29 210
Weight Updates for Simple Units
)()( jdDd
idid xot
jdDd
ididij
iij xot
wE
w
)(
))(()( idTijDd
ididij
i sumfw
otwE
fT = identity function
wijxjd
oid
tid
Eid
![Page 211: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/211.jpg)
Semiconductors, BP&A Planning, 2003-01-29 211
Weight Updates for Sigmoidal Units
))(1()( jdididDd
idid xooot
jdididDd
ididij
iij xooot
wE
w )1()(
))(()( idTijDd
ididij
i sumfw
otwE
fT = sigmoidal function
wijxjd
oid
tid
Eid
![Page 212: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/212.jpg)
Semiconductors, BP&A Planning, 2003-01-29 212
![Page 213: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/213.jpg)
Semiconductors, BP&A Planning, 2003-01-29 213
Error gradient for the sigmoid function
![Page 214: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/214.jpg)
Semiconductors, BP&A Planning, 2003-01-29 214
Error gradient for the sigmoid function
![Page 215: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/215.jpg)
Semiconductors, BP&A Planning, 2003-01-29 215
Bias of a Neuron
• The bias b has the effect of applying an affine transformation to the weighted sum u
v = u + b
x1-x2=0
x1-x2= 1
x1
x2 x1-x2= -1
![Page 216: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/216.jpg)
Semiconductors, BP&A Planning, 2003-01-29 216
Bias as extra input• The bias is an external parameter of the neuron. It can be modeled by adding an extra input.
Inputsignal
Synapticweights
Summingfunction
ActivationfunctionLocal
Field
vOutput
y
x1
x2
xm
w2
wm
w1
)(
w0x0 = +1
bw
xwv j
m
j
j
0
0
![Page 217: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/217.jpg)
Semiconductors, BP&A Planning, 2003-01-29 217
Apples/Bananas Sorter
![Page 218: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/218.jpg)
Semiconductors, BP&A Planning, 2003-01-29 218
![Page 219: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/219.jpg)
Semiconductors, BP&A Planning, 2003-01-29 219
Prototype Vectors
![Page 220: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/220.jpg)
Semiconductors, BP&A Planning, 2003-01-29 220
Perceptron
![Page 221: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/221.jpg)
Semiconductors, BP&A Planning, 2003-01-29 221
Two-Input Case
![Page 222: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/222.jpg)
Semiconductors, BP&A Planning, 2003-01-29 222
Apple / Banana Example
![Page 223: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/223.jpg)
Semiconductors, BP&A Planning, 2003-01-29 223
Testing the Network
![Page 224: Dr. kiani artificial neural network lecture 1](https://reader035.fdocuments.in/reader035/viewer/2022062220/554e933eb4c90573338b4f51/html5/thumbnails/224.jpg)
Semiconductors, BP&A Planning, 2003-01-29 224
Questions? Suggestions ?