Back Propagation Network by bhabanath sahu
-
Upload
sbhabanath -
Category
Documents
-
view
225 -
download
0
Transcript of Back Propagation Network by bhabanath sahu
-
7/30/2019 Back Propagation Network by bhabanath sahu
1/21
Back Propagation Network
-
7/30/2019 Back Propagation Network by bhabanath sahu
2/21
Back Propagation Network
What is BPN?
A single-layer NN has many restriction.
limited classes of task.
Minsky ans Papert (1969) showed that a two
layer FF Network can over come many
restriction, but they did not present the
solution to the problem as how to adjust the
weight from input to hidden layer?
-
7/30/2019 Back Propagation Network by bhabanath sahu
3/21
Back Propagation Network (Continue)
In 1986 an answer was presented by
Rumelhart, Hinton and williams.
The central idea behind this solution is that,
the error for the hidden layer units are
determined by back-propagating the errors of
the output layer units.
This method is often called the Back-
propagation learning rule.
-
7/30/2019 Back Propagation Network by bhabanath sahu
4/21
Back Propagation Network (Continue)
Back-propagation is a systematic method of
training artificial neural networks.
A BackProp network consist of at least three
layers of units:
An input layer,
At least one intermediate hidden layer, and
An output layer.
-
7/30/2019 Back Propagation Network by bhabanath sahu
5/21
Back Propagation Network (Continue)
Units are connected in a feed-forward fashion
(i.e. i/p layer neuron is fully connected to
hidden layer neuron and hidden layer neuron
is fully connected to output layer neuron.)
When a BackProp n/w is cycles, an input
pattern is propagated forward to the o/p units
through the i/p to hidden and hidden to o/pweights.
The o/p of BackProp n/w is interpreted as a
classification decision or result of application.
-
7/30/2019 Back Propagation Network by bhabanath sahu
6/21
Back Propagation Network (Continue)
With BPN, learning occurs during trainingphase. The steps are:
1. Each i/p pattern in a training set is applied to
the i/p units and then propagated forward.2. At the o/p layer error is calculated. (i.e.
compression between n/w o/p and target).
3. The error signal for each such target o/ppattern is then back-propagated from the o/pto the i/p in order to adjust the weights ineach layer of the n/w
-
7/30/2019 Back Propagation Network by bhabanath sahu
7/21
Back Propagation Network (Continue)
4. After BackProp n/w has learned the correct
classification for a set of i/ps, it can be tested
for untrained pattern.
Learning: implement AND function in NN
AND function
AND
X1 X2 Y
0 0 0
0 1 0
1 0 0
1 1 1
X1
X2
Y
I1
I2
Output OW1
W2
-
7/30/2019 Back Propagation Network by bhabanath sahu
8/21
Back Propagation Network (Continue)
There are 4 inequalities in the AND function
and they must be satisfied.
W1
0 + W2
0 < , W1
0 + W2
1 < ,
W11 + W20 < , W11 + W21 >
One possible solution: if both weight are set
to 1 and the threshold is set to 1.5, then
(1)(0) + (1)(0) < 1.5 assign 0, (1)(0) + (1)(1) < 1.5 assign 0,
(1)(1) + (1)(0) < 1.5 assign 0, (1)(1) + (1)(1) > 1.5 assign 1,
-
7/30/2019 Back Propagation Network by bhabanath sahu
9/21
Back Propagation Network (Continue)
Example 1
AND Problem
AND
X1 X2 Y
0 0 0
0 1 0
1 0 0
1 1 1
X1
X2
Y
I1
I2
Output O
W1
W2
The o/p of the n/w is determined by
calculating a weighted sum of its two i/p and
comparing this value with a threshold .
if the net input (net) is greater than , than
the o/p is 1, else it is 0.
the computation is performed by the o/p
unit is.
net= W1 I1 + W2 I2
if net > then O = 1, otherwise O = 0.
-
7/30/2019 Back Propagation Network by bhabanath sahu
10/21
Simple Learning Machines
Rosenblatt proposed learning n/w called
Perceptron.
A perceptron consist of a set of input units
and a single output unit.
As in the AND N/w, the output of the
perceptron is calculated by comparing the net
i/p net = (Wi Ii) where i = 1 to n, and a
threshold .
If net > the O/p = 1, else O/p = 0
-
7/30/2019 Back Propagation Network by bhabanath sahu
11/21
Simple Learning Machines (Continue)
The learning rule that Rosenblatt developed, is
based on determining the difference between
the actual o/p of the n/w with the target o/p
(0 or 1), called ErrorMeasure.
Error Measure (Learning rule): Difference
between actual o/p of the n/w with target o/p
(0 or 1).
-
7/30/2019 Back Propagation Network by bhabanath sahu
12/21
Simple Learning Machines (Continue)
Error Measure (Learning rule) : Difference
between actual o/p of the n/w with target o/p
(0 or 1).
If the i/p vector is correctly classified (i.e., zero
error), then the weight are left unchanged, and
the next i/p vector is presented.
Else there are two cases to consider
-
7/30/2019 Back Propagation Network by bhabanath sahu
13/21
Simple Learning Machines (Continue)
Case 1: if the o/p unit is 1 and need to be 0 then.
the threshold is incremented by 1.
if the i/p Ii is 0, then the corresponding weight Wi is
left unchanged.
If the i/p Ii is 1, then the corresponding weight Wi is
decreased by 1.
Case 2: If O/p unit is 0 but need to be 1 then theopposite changes are made.
-
7/30/2019 Back Propagation Network by bhabanath sahu
14/21
Simple Learning Machines (Continue)
Perceptron Learning Rule : Equations
The perceptron learning rules are govern by two
equations,
1. Change in threshold
2. Change in weights.
The change in threshold is given by = -(tp op) = -dp
The change in the weight are given by
Wi = (tp op) Ipi = dp Ipi
-
7/30/2019 Back Propagation Network by bhabanath sahu
15/21
Back Propagation Network
Learning by Example: Consider the Multilayer
feed-forward back-propagation network
1 1 1
2 2 2
l m n
II1
II2
IIl
OI1
OI2
OIl
V11
V21
Vij
IH1
IH2
IHm
OH1
OH2
OHm
W11W21
Wij
IO1
IO2
IOn
OO1
OO2
OOn
Input Layer i nodes Hidden Layer m nodes O/p Layer n nodes
-
7/30/2019 Back Propagation Network by bhabanath sahu
16/21
Back Propagation Network
Consider a problem in which an nset of l
inputs and the corresponding nset of n
output data is shown in table
No Input Output
l1
L2..
ll
O1
O2
O
l
1 0.3 0.4 0.8 0.1 0.56 0.82
2
.
.
Nset
-
7/30/2019 Back Propagation Network by bhabanath sahu
17/21
Input Layer Computation
Consider linear activation function.
if the o/p of input layer is the input of the i/p
layer and the transfer function is 1, then
{ O }I = { I }I
lx 1 lx 1(denotes matrix row, column size)
The i/p to the hidden neuron is the weightedsum of the o/ps of the i/p neurons. Thus
IHP = V1pOI1 + V2p OI2+ .+ V1p OIl so that
{ I }H = [V]T
{ O }I
-
7/30/2019 Back Propagation Network by bhabanath sahu
18/21
Hidden Layer Computation
Consider sigmoid function. The output of the pth
hidden neuron is given by
OHP
= 1/(1 + e-(IHP-HP))
The input to the o/p neuron is the weighted sum
of the o/p of the hidden neuron. Ioq the input to
the qth output neuron is given by
Ioq = W1q Ohp + W2q OH2+.+ Wmq Ohm
So that,
{ I }o = [W]T
{ O }H
-
7/30/2019 Back Propagation Network by bhabanath sahu
19/21
Output Layer Computation
Consider sigmoid function. The output of the qth
output neuron is given by
Ooq
= 1/(1 + e-(Ioq-oq)
Where : Ooq is the o/p of the qth o/p neuron,
Ioq is the i/p of the qth o/p neuron,
oq is the threshold of the qth
o/pneuron,
-
7/30/2019 Back Propagation Network by bhabanath sahu
20/21
Calculation of Error
Consider any rth output neuron,
The error norm in the output for the rth output
neuron is calculated by:
E1r = (1/2) e2
r = (1/2) (T-O)2
-
7/30/2019 Back Propagation Network by bhabanath sahu
21/21
Basic algorithm loop structure
Initialize the weight
Repeat
foreach training pattern
Train on that pattern
EndUntil the error is acceptably low.