Post on 26-Dec-2015
1
Introduction to Artificial Neural Networks
Andrew L. Nelson
Visiting Research FacultyUniversity of South Florida
2/9/2004 Neural Networks 2
Overview• Outline to the left
• Current topic in red• Introduction• History and Origins• Biologically Inspired • Applications • Perceptron• Activation Functions• Hidden Layer Networks• Training with BP• Examples
• References• Introduction• History• Biologically
Inspired• Applications• The Perceptron• Activation
Functions• Hidden Layer
Networks
• Training with BP
• Examples
2/9/2004 Neural Networks 3
References• W. S. McCulloch, W. Pitts, "A logical calculus
of the ideas immanent in nervous activity," Bulletin of Mathematical Biophysics, vol. 5 pp. 115-133, 1943.
• J. L. McClelland, D. E. Rumelhart, Parallel Distributed Processing: Explorations in the Microstructure of Cognition, The MIT Press, 1986.
• C. M. Bishop, Neural Networks for Pattern Recognition, Oxford University Press, 1995.
• References• Introduction• History• Biologically
Inspired• Applications• The Perceptron• Activation
Functions• Hidden Layer
Networks
• Training with BP
• Examples
2/9/2004 Neural Networks 4
Introduction• Artificial Neural Networks (ANN)
• Connectionist computation• Parallel distributed processing• Computational models
• Biologically Inspired computational models
• Machine Learning
• Artificial intelligence
• References• Introduction• History• Biologically
Inspired• Applications• The Perceptron• Activation
Functions• Hidden Layer
Networks
• Training with BP
• Examples
2/9/2004 Neural Networks 5
History• McCulloch and Pitts introduced the
Perceptron in 1943.• Simplified model of a biological neuron
• Fell out of favor in the late 1960's • (Minsky and Papert)
• Perceptron limitations
• Resurgence in the mid 1980's• Nonlinear Neuron Functions
• Back-propagation training
• References• Introduction• History• Biologically
Inspired• Applications• The Perceptron• Activation
Functions• Hidden Layer
Networks
• Training with BP
• Examples
2/9/2004 Neural Networks 6
Summary of Applications
• Function approximation
• Pattern recognition
• Signal processing
• Modeling
• Control
• Machine learning
• References• Introduction• History• Biologically
Inspired• Applications• The Perceptron• Activation
Functions• Hidden Layer
Networks
• Training with BP
• Examples
2/9/2004 Neural Networks 7
Biologically Inspired• Electro-chemical signals
• Threshold output firing
Axon
Terminal Branches of Axon
Dendrites
• References• Introduction• History• Biologically
Inspired• Applications• The Perceptron• Activation
Functions• Hidden Layer
Networks
• Training with BP
• Examples
2/9/2004 Neural Networks 8
The Perceptron• Binary classifier functions
• Threshold activation function
Axon
Terminal Branches of Axon
Dendrites
S
x1
x2
w1
w2
wnxn
x3 w3
• References• Introduction• History• Biologically
Inspired• Applications• The Perceptron• Activation
Functions• Hidden Layer
Networks
• Training with BP
• Examples
2/9/2004 Neural Networks 9
The Perceptron: Threshold Activation Function
• Binary classifier functions
• Threshold activation function
Step Threshold
• References• Introduction• History• Biologically
Inspired• Applications• The Perceptron• Activation
Functions• Hidden Layer
Networks
• Training with BP
• Examples
2/9/2004 Neural Networks 10
Linear Activation functions• Output is scaled sum of inputs
n
N
nn xwuy
1
Linear
• References• Introduction• History• Biologically
Inspired• Applications• The Perceptron• Activation
Functions• Hidden Layer
Networks
• Training with BP
• Examples
2/9/2004 Neural Networks 11
Nonlinear Activation Functions
• Sigmoid Neuron unit function
uhide
uy
1
1)(
Sigmoid
• References• Introduction• History• Biologically
Inspired• Applications• The Perceptron• Activation
Functions• Hidden Layer
Networks
• Training with BP
• Examples
2/9/2004 Neural Networks 12
Layered Networks
. x1 y1
Hidden Neurons Output NeuronsInputs Outputs
x2 y2
xn ym
yh1(u ) yo1(u )
yo2(u )yh2(u )
yhn(u ) yom(u )
• References• Introduction• History• Biologically
Inspired• Applications• The Perceptron• Activation
Functions• Hidden Layer
Networks
• Training with BP
• Examples
2/9/2004 Neural Networks 13
SISO Single Hidden Layer Network
• Can represent and single input single output functions: y = f(x)
x y
Hidden Neurons
Output NeuronInput
Output
yhid,1 (u )
yout(u )
yhid,2 (u )
yhid,N (u )
whid,1
whid,2
wout,1
wout,2
wout,Nwhid,N
• References• Introduction• History• Biologically
Inspired• Applications• The Perceptron• Activation
Functions• Hidden Layer
Networks
• Training with BP
• Examples
2/9/2004 Neural Networks 14
Training Data SetAdjust weights (w) to learn a given target
function: y = f(x)
Given a set of training data X→Y
x y
Hidden Neurons
Output NeuronInput
Output
yhid,1 (u )
yout(u )
yhid,2 (u )
yhid,N (u )
whid,1
whid,2
wout,1
wout,2
wout,Nwhid,N
• References• Introduction• History• Biologically
Inspired• Applications• The Perceptron• Activation
Functions• Hidden Layer
Networks
• Training with BP
• Examples
2/9/2004 Neural Networks 15
Training Weights: Error Back-Propagation (BP)
• Weight update formula:
wkwkw )()( 1
w
ieiw
)(
*)(
• References• Introduction• History• Biologically
Inspired• Applications• The Perceptron• Activation
Functions• Hidden Layer
Networks
• Training with BP
• Examples
2/9/2004 Neural Networks 16
Error Back-Propagation (BP)Training error term: e
yout(Suout,n)
yhid(uhid,n)
Nn
N1
yhid(uhid,1)
yhid(uhid,2)
N2e(yout, ytrain)
wout,1
wout,2
whid,n wout,n
uhid,1
x
yhid
uout,1
yout
ytrain
whid,2
whid,1
x
Hidden Neurons
Output Neuron
e
2
2
1)( trainout yye
• References• Introduction• History• Biologically
Inspired• Applications• The Perceptron• Activation
Functions• Hidden Layer
Networks
• Training with BP
• Examples
2/9/2004 Neural Networks 17
BP Formulation
))),(((
))),(((
)),((
)),((),(
,,
,,
,,
,
trainhidhidoutout
trainhidhidoutout
trainhidoutout
trainoutouttrainout
yxwywye
yuywye
yywye
yuyeyye
11
11
11
1
• References• Introduction• History• Biologically
Inspired• Applications• The Perceptron• Activation
Functions• Hidden Layer
Networks
• Training with BP
• Examples
yout(Suout,n)
yhid(uhid,n)
Nn
N1
yhid(uhid,1)
yhid(uhid,2)
N2e(yout, ytrain)
wout,1
wout,2
whid,n wout,n
uhid,1
x
yhid
uout,1
yout
ytrain
whid,2
whid,1
x
Hidden Neurons
Output Neuron
e
2/9/2004 Neural Networks 18
BP Formulation
1
1
1
1
1 ,
,
,
,
, hid
hid
hid
hid
hid
out
out
out
outhid w
u
u
y
y
u
u
y
y
e
w
e
outout
out
hid
out
hid
hid
hid
hid
hid y
e
u
y
y
u
u
y
w
u
w
e
1,
1,
1,1,
1,
• References• Introduction• History• Biologically
Inspired• Applications• The Perceptron• Activation
Functions• Hidden Layer
Networks
• Training with BP
• Examples
yout(Suout,n)
yhid(uhid,n)
Nn
N1
yhid(uhid,1)
yhid(uhid,2)
N2e(yout, ytrain)
wout,1
wout,2
whid,n wout,n
uhid,1
x
yhid
uout,1
yout
ytrain
whid,2
whid,1
x
Hidden Neurons
Output Neuron
e
2/9/2004 Neural Networks 19
BP Formulation
x
xwww
uhid
hidhid
hid
111
1,
,,
,
• References• Introduction• History• Biologically
Inspired• Applications• The Perceptron• Activation
Functions• Hidden Layer
Networks
• Training with BP
• Examples
yout(Suout,n)
yhid(uhid,n)
Nn
N1
yhid(uhid,1)
yhid(uhid,2)
N2e(yout, ytrain)
wout,1
wout,2
whid,n wout,n
uhid,1
x
yhid
uout,1
yout
ytrain
whid,2
whid,1
x
Hidden Neurons
Output Neuron
e
2/9/2004 Neural Networks 20
BP Formulation
)()(
)()(
)()(
)()(
)(
)()(
)(
,, 11
2
22
2
2
1
1
11
1
1
1
1
1
1
1
1
1
1
1
11
1
1
1
hidhidhidhid
uu
uu
uu
u
u
u
uu
uhid
uyuy
ee
ee
ee
e
e
e
ee
edu
d
du
udy
• References• Introduction• History• Biologically
Inspired• Applications• The Perceptron• Activation
Functions• Hidden Layer
Networks
• Training with BP
• Examples
yout(Suout,n)
yhid(uhid,n)
Nn
N1
yhid(uhid,1)
yhid(uhid,2)
N2e(yout, ytrain)
wout,1
wout,2
whid,n wout,n
uhid,1
x
yhid
uout,1
yout
ytrain
whid,2
whid,1
x
Hidden Neurons
Output Neuron
e
2/9/2004 Neural Networks 21
BP Formulation
1
111
,
,,,
out
hidouthidhid
out
w
ywyy
u
• References• Introduction• History• Biologically
Inspired• Applications• The Perceptron• Activation
Functions• Hidden Layer
Networks
• Training with BP
• Examples
yout(Suout,n)
yhid(uhid,n)
Nn
N1
yhid(uhid,1)
yhid(uhid,2)
N2e(yout, ytrain)
wout,1
wout,2
whid,n wout,n
uhid,1
x
yhid
uout,1
yout
ytrain
whid,2
whid,1
x
Hidden Neurons
Output Neuron
e
2/9/2004 Neural Networks 22
BP Formulation
)(
)(
trainout
trainoutoutout
yy
yyyy
e
2
2
1
• References• Introduction• History• Biologically
Inspired• Applications• The Perceptron• Activation
Functions• Hidden Layer
Networks
• Training with BP
• Examples
yout(Suout,n)
yhid(uhid,n)
Nn
N1
yhid(uhid,1)
yhid(uhid,2)
N2e(yout, ytrain)
wout,1
wout,2
whid,n wout,n
uhid,1
x
yhid
uout,1
yout
ytrain
whid,2
whid,1
x
Hidden Neurons
Output Neuron
e
2/9/2004 Neural Networks 23
BP Formulation
))()(()()()( ,,,
,
,
,,
,
trainoutouthidhidhidhid
outout
out
hid
out
hid
hid
hid
hid
hid
yywuyuyx
y
e
u
y
y
u
u
y
w
u
w
e
11 111
1
1
11
1
• References• Introduction• History• Biologically
Inspired• Applications• The Perceptron• Activation
Functions• Hidden Layer
Networks
• Training with BP
• Examples
yout(Suout,n)
yhid(uhid,n)
Nn
N1
yhid(uhid,1)
yhid(uhid,2)
N2e(yout, ytrain)
wout,1
wout,2
whid,n wout,n
uhid,1
x
yhid
uout,1
yout
ytrain
whid,2
whid,1
x
Hidden Neurons
Output Neuron
e
2/9/2004 Neural Networks 24
BP Formulation
))()((
)(... ,,,,
,,
,
,
trainouthid
trainoutout
Noutoutoutout
hidoutout
outout
out
out
out
out
yyy
yyy
uuuu
yww
y
e
u
y
w
u
w
e
1
2
1 221
111
1
1
• References• Introduction• History• Biologically
Inspired• Applications• The Perceptron• Activation
Functions• Hidden Layer
Networks
• Training with BP
• Examples
yout(Suout,n)
yhid(uhid,n)
Nn
N1
yhid(uhid,1)
yhid(uhid,2)
N2e(yout, ytrain)
wout,1
wout,2
whid,n wout,n
uhid,1
x
yhid
uout,1
yout
ytrain
whid,2
whid,1
x
Hidden Neurons
Output Neuron
e
2/9/2004 Neural Networks 25
Example: The XOR problem:
• Single hidden layer: 3 Sigmoid neurons
• 2 inputs, 1 output
Desired I/O table (XOR):
x1 x2 y
Example 1 0 0 0
Example 2 0 1 1
Example 3 1 0 1
Example 4 1 1 0
• References• Introduction• History• Biologically
Inspired• Applications• The Perceptron• Activation
Functions• Hidden Layer
Networks
• Training with BP
• Examples
2/9/2004 Neural Networks 26
Example: The XOR problem:
• Training error over epoch• References• Introduction• History• Biologically
Inspired• Applications• The Perceptron• Activation
Functions• Hidden Layer
Networks
• Training with BP
• Examples
2/9/2004 Neural Networks 27
Example: The XOR problem:
initial_weights =0.0654 0.2017 0.0769 0.1782 0.0243 0.0806 0.0174 0.1270 0.0599 0.1184 0.1335 0.0737 0.1511
final_weights =4.6970 -4.6585 2.0932 5.5168 -5.7073 -3.2338 -0.1886 1.6164 -0.1929 -6.8066 6.8477 -1.6886 4.1531
• References• Introduction• History• Biologically
Inspired• Applications• The Perceptron• Activation
Functions• Hidden Layer
Networks
• Training with BP
• Examples
2/9/2004 Neural Networks 28
Example: The XOR problem:
Mapping produced by the trained neural net:
x1 x2 y
Example 1 0 0 0.0824
Example 2 0 1 0.9095
Example 3 1 0 0.9470
Example 4 1 1 0.0464
• References• Introduction• History• Biologically
Inspired• Applications• The Perceptron• Activation
Functions• Hidden Layer
Networks
• Training with BP
• Examples
2/9/2004 Neural Networks 29
Example: Overtraining • Single hidden layer: 10 Sigmoid
neurons• 1 input, 1 output
• References• Introduction• History• Biologically
Inspired• Applications• The Perceptron• Activation
Functions• Hidden Layer
Networks
• Training with BP
• Examples