1 Introduction to Artificial Neural Networks Andrew L. Nelson Visiting Research Faculty University...

1

Introduction to Artificial Neural Networks

Andrew L. Nelson

Visiting Research FacultyUniversity of South Florida

2/9/2004 Neural Networks 2

Overview• Outline to the left

• Current topic in red• Introduction• History and Origins• Biologically Inspired • Applications • Perceptron• Activation Functions• Hidden Layer Networks• Training with BP• Examples

• References• Introduction• History• Biologically

Inspired• Applications• The Perceptron• Activation

Functions• Hidden Layer

Networks

• Training with BP

• Examples


References• W. S. McCulloch, W. Pitts, "A logical calculus

of the ideas immanent in nervous activity," Bulletin of Mathematical Biophysics, vol. 5 pp. 115-133, 1943.

• J. L. McClelland, D. E. Rumelhart, Parallel Distributed Processing: Explorations in the Microstructure of Cognition, The MIT Press, 1986.

• C. M. Bishop, Neural Networks for Pattern Recognition, Oxford University Press, 1995.




Networks


• Examples


Introduction• Artificial Neural Networks (ANN)

• Connectionist computation• Parallel distributed processing• Computational models

• Biologically Inspired computational models

• Machine Learning

• Artificial intelligence




Networks


• Examples


History• McCulloch and Pitts introduced the

Perceptron in 1943.• Simplified model of a biological neuron

• Fell out of favor in the late 1960's • (Minsky and Papert)

• Perceptron limitations

• Resurgence in the mid 1980's• Nonlinear Neuron Functions

• Back-propagation training




Networks


• Examples


Summary of Applications

• Function approximation

• Pattern recognition

• Signal processing

• Modeling

• Control

• Machine learning




Networks


• Examples


Biologically Inspired• Electro-chemical signals

• Threshold output firing

Axon

Terminal Branches of Axon

Dendrites




Networks


• Examples


The Perceptron• Binary classifier functions

• Threshold activation function

Axon

Terminal Branches of Axon

Dendrites

S

x1

x2

w1

w2

wnxn

x3 w3




Networks


• Examples


The Perceptron: Threshold Activation Function

• Binary classifier functions

• Threshold activation function

Step Threshold




Networks


• Examples


Linear Activation functions• Output is scaled sum of inputs

n

N

nn xwuy

1

Linear




Networks


• Examples


Nonlinear Activation Functions

• Sigmoid Neuron unit function

uhide

uy

1

1)(

Sigmoid




Networks


• Examples


Layered Networks

. x1 y1

Hidden Neurons Output NeuronsInputs Outputs

x2 y2

xn ym

yh1(u ) yo1(u )

yo2(u )yh2(u )

yhn(u ) yom(u )




Networks


• Examples


SISO Single Hidden Layer Network

• Can represent and single input single output functions: y = f(x)

x y

Hidden Neurons

Output NeuronInput

Output

yhid,1 (u )

yout(u )

yhid,2 (u )

yhid,N (u )

whid,1

whid,2

wout,1

wout,2

wout,Nwhid,N




Networks


• Examples


Training Data SetAdjust weights (w) to learn a given target

function: y = f(x)

Given a set of training data X→Y

x y

Hidden Neurons

Output NeuronInput

Output

yhid,1 (u )

yout(u )

yhid,2 (u )

yhid,N (u )

whid,1

whid,2

wout,1

wout,2

wout,Nwhid,N




Networks


• Examples


Training Weights: Error Back-Propagation (BP)

• Weight update formula:

wkwkw )()( 1

w

ieiw

)(

*)(




Networks


• Examples


Error Back-Propagation (BP)Training error term: e

yout(Suout,n)

yhid(uhid,n)

Nn

N1

yhid(uhid,1)

yhid(uhid,2)

N2e(yout, ytrain)

wout,1

wout,2

whid,n wout,n

uhid,1

x

yhid

uout,1

yout

ytrain

whid,2

whid,1

x

Hidden Neurons

Output Neuron

e

2

2

1)( trainout yye




Networks


• Examples


BP Formulation

))),(((

))),(((

)),((

)),((),(

,,

,,

,,

,

trainhidhidoutout

trainhidhidoutout

trainhidoutout

trainoutouttrainout

yxwywye

yuywye

yywye

yuyeyye

11

11

11

1




Networks


• Examples

yout(Suout,n)

yhid(uhid,n)

Nn

N1

yhid(uhid,1)

yhid(uhid,2)

N2e(yout, ytrain)

wout,1

wout,2

whid,n wout,n

uhid,1

x

yhid

uout,1

yout

ytrain

whid,2

whid,1

x

Hidden Neurons

Output Neuron

e


BP Formulation

1

1

1

1

1 ,

,

,

,

, hid

hid

hid

hid

hid

out

out

out

outhid w

u

u

y

y

u

u

y

y

e

w

e

outout

out

hid

out

hid

hid

hid

hid

hid y

e

u

y

y

u

u

y

w

u

w

e

1,

1,

1,1,

1,




Networks


• Examples

yout(Suout,n)

yhid(uhid,n)

Nn

N1

yhid(uhid,1)

yhid(uhid,2)

N2e(yout, ytrain)

wout,1

wout,2

whid,n wout,n

uhid,1

x

yhid

uout,1

yout

ytrain

whid,2

whid,1

x

Hidden Neurons

Output Neuron

e


BP Formulation

x

xwww

uhid

hidhid

hid

111

1,

,,

,




Networks


• Examples

yout(Suout,n)

yhid(uhid,n)

Nn

N1

yhid(uhid,1)

yhid(uhid,2)

N2e(yout, ytrain)

wout,1

wout,2

whid,n wout,n

uhid,1

x

yhid

uout,1

yout

ytrain

whid,2

whid,1

x

Hidden Neurons

Output Neuron

e


BP Formulation

)()(

)()(

)()(

)()(

)(

)()(

)(

,, 11

2

22

2

2

1

1

11

1

1

1

1

1

1

1

1

1

1

1

11

1

1

1

hidhidhidhid

uu

uu

uu

u

u

u

uu

uhid

uyuy

ee

ee

ee

e

e

e

ee

edu

d

du

udy




Networks


• Examples

yout(Suout,n)

yhid(uhid,n)

Nn

N1

yhid(uhid,1)

yhid(uhid,2)

N2e(yout, ytrain)

wout,1

wout,2

whid,n wout,n

uhid,1

x

yhid

uout,1

yout

ytrain

whid,2

whid,1

x

Hidden Neurons

Output Neuron

e


BP Formulation

1

111

,

,,,

out

hidouthidhid

out

w

ywyy

u




Networks


• Examples

yout(Suout,n)

yhid(uhid,n)

Nn

N1

yhid(uhid,1)

yhid(uhid,2)

N2e(yout, ytrain)

wout,1

wout,2

whid,n wout,n

uhid,1

x

yhid

uout,1

yout

ytrain

whid,2

whid,1

x

Hidden Neurons

Output Neuron

e


BP Formulation

)(

)(

trainout

trainoutoutout

yy

yyyy

e

2

2

1




Networks


• Examples

yout(Suout,n)

yhid(uhid,n)

Nn

N1

yhid(uhid,1)

yhid(uhid,2)

N2e(yout, ytrain)

wout,1

wout,2

whid,n wout,n

uhid,1

x

yhid

uout,1

yout

ytrain

whid,2

whid,1

x

Hidden Neurons

Output Neuron

e


BP Formulation

))()(()()()( ,,,

,

,

,,

,

trainoutouthidhidhidhid

outout

out

hid

out

hid

hid

hid

hid

hid

yywuyuyx

y

e

u

y

y

u

u

y

w

u

w

e

11 111

1

1

11

1




Networks


• Examples

yout(Suout,n)

yhid(uhid,n)

Nn

N1

yhid(uhid,1)

yhid(uhid,2)

N2e(yout, ytrain)

wout,1

wout,2

whid,n wout,n

uhid,1

x

yhid

uout,1

yout

ytrain

whid,2

whid,1

x

Hidden Neurons

Output Neuron

e


BP Formulation

))()((

)(... ,,,,

,,

,

,

trainouthid

trainoutout

Noutoutoutout

hidoutout

outout

out

out

out

out

yyy

yyy

uuuu

yww

y

e

u

y

w

u

w

e

1

2

1 221

111

1

1




Networks


• Examples

yout(Suout,n)

yhid(uhid,n)

Nn

N1

yhid(uhid,1)

yhid(uhid,2)

N2e(yout, ytrain)

wout,1

wout,2

whid,n wout,n

uhid,1

x

yhid

uout,1

yout

ytrain

whid,2

whid,1

x

Hidden Neurons

Output Neuron

e


Example: The XOR problem:

• Single hidden layer: 3 Sigmoid neurons

• 2 inputs, 1 output

Desired I/O table (XOR):

x1 x2 y

Example 1 0 0 0

Example 2 0 1 1

Example 3 1 0 1

Example 4 1 1 0




Networks


• Examples



• Training error over epoch• References• Introduction• History• Biologically



Networks


• Examples



initial_weights =0.0654 0.2017 0.0769 0.1782 0.0243 0.0806 0.0174 0.1270 0.0599 0.1184 0.1335 0.0737 0.1511

final_weights =4.6970 -4.6585 2.0932 5.5168 -5.7073 -3.2338 -0.1886 1.6164 -0.1929 -6.8066 6.8477 -1.6886 4.1531




Networks


• Examples



Mapping produced by the trained neural net:

x1 x2 y

Example 1 0 0 0.0824

Example 2 0 1 0.9095

Example 3 1 0 0.9470

Example 4 1 1 0.0464




Networks


• Examples


Example: Overtraining • Single hidden layer: 10 Sigmoid

neurons• 1 input, 1 output




Networks


• Examples

1 Introduction to Artificial Neural Networks Andrew L. Nelson Visiting Research Faculty University...

Documents

Transcript of 1 Introduction to Artificial Neural Networks Andrew L. Nelson Visiting Research Faculty University...