Introduction to Artificial Neural Networks - Faculty Web...
Transcript of Introduction to Artificial Neural Networks - Faculty Web...
• The brain has been extensively studied byscientists.
• Vast complexity prevents all but rudimentaryunderstanding.
• Even the behaviour of an individual neuronis extremely complex.
Artificial Neural Networks:Biological Inspiration
Features of the Brain• >Ten billion (1010) neurons• Neuron switching time >10-3 secs• Face Recognition ~0.1 secs• On average, each neuron has several thousand
connections (to other neurons)• Hundreds of operations per second• Very high degree of parallel computation• Distributed representations (not all info in one spot)• Die off frequently (never replaced):
• redundancy & rewired
• Compensated for problems by massive parallelism
Computers vs Brains
• One CPU
• Fixed connections
• Fast
• Repeatable/reliable
• Absolute arithmeticprecision
• Good computation
• Poor recognition
• Highly parallel processing
• Changing connections
• Good Pattern Recognition
• Good association & context
• Good for complex issues
• Good noise tolerance
• Good with incomplete info
• Unreliable, slow
• Poor computational ability
A Contrast in Architecture
Why Artificial Neural Networks?
There are two basic reasons why we are interested in buildingartificial neural networks (ANNs):
• Technical viewpoint: Some problems such as character recognition or the prediction of future states of a system require massively parallel and adaptive processing.
• Biological viewpoint: ANNs can be used to replicate and simulate components of the human (or animal) brain, thereby giving us insight into natural information processing.
Why do we need another paradigm otherthan symbolic AI or Fuzzy logic for
building “intelligent” machines?
• Symbolic AI and Fuzzy logic are well-suited for representing explicit knowledge that can be appropriately formalized, or stated.
• However, learning in complex systems (e.g., biological) is mostly implicit – it is an adaptation process based on uncertain information by uncertain reasoning via experiences.
• Good in mechatronics for• Developing “unwritten” rules of operation and control.• Making a model or estimator of a system by observation.
How do NNs and ANNs work?
• The “building blocks” of real neural networks are the neurons(nerve cells).
axon
cell body
synapse
nucleus
dendrites
axon
cell body
synapse
nucleus
dendrites
How does a neuron work
• A neuron only fires (via the axon) if its inputsignal exceeds a certain amount (threshold).• All or nothing (on or off)
• Synapses (junctions/connections betweenneurons) vary in strength– Good connections allowing a large signal– Slight connections allow only a weak signal.– Synapses can be either excitatory (+) or inhibitory (-).
How do they all piece together?
• Basically, each neuron• Transmits information as a series of electric impulses, so-
called spikes.• Receives inputs from many other neurons. One neuron can
be connected.• Changes its internal state (activation) based on all the current
inputs• Sends one output signal to many other neurons (>10,000 in
cases), possibly including itself (recurrent network).• Phase, frequency are also important in real neuron.
• In ANN we refer to neurons as units or nodes.
How do NNs (and ANNs) work?• In biological systems, neurons of similar functionality
are usually organized in separate areas (or layers).
• Often, there is a hierarchy of interconnected layerswith the lowest layer receiving sensory input andneurons in higher layers computing more complexfunctions.
• For example, neurons in a monkey’s visual cortex havebeen identified that are activated only when there is aface (monkey, human, or drawing) in the primate’svisual field.
Artificial Neural Network History &Terminology
• Originally hailed as a breakthrough in AI• 1943-McCulloch & Pitts model -first neuron model
• 1957 Rosenblatt - Perceptron networks
• 1969-Minsky, Papert - Defined perceptron limitations: ANN setback
• 1980’s - Re-emergence of ANN - multi-layer networks
• Referred to as connectionist networks, paralleldistributed processing, adaptive networks
• Now, better understood
• Hundreds of variants
• Less a model of the actual brain than a useful tool
• Numerous applications• handwriting, face, speech recognition,CMU van that drives itself
Components of Modeled Neuron
• The internal components of the neural unit consist of;
• Weights. There is a weight associated with each input. Theseweights are two-valued as well ( {-1,1} ). ( w1, w2, … , wn ).
• Threshold / Bias:θ. A threshold weight, w0, associated with a biasvalue of x0. θ= w0 x0
• Summing Unit. A summation unit which produces the weightedsum of the binary inputs ( s = w0x0 + w1x1 + … + wnxn ).
• Activation Function. An activation function which determineswhether the neural unit ‘fires’ or not. This function takes theweighted sum, s, as its input and outputs a single value ( ex. {0,1},or{-1,1} ).
• The following diagram displays the above components
Basic Artificial Neuron with Threshold
threshold threshold θθ
x1
w1
x2 w2
wn
xn
… i
n
iiwx∑
=1
θ≥= ∑=
i
n
iinn wxxwxwxwf
12211 if,1),...,,(
otherwise,0=
activation output
inputs
f
summing
weights
The Threshold Activation Function•One possible of neuron activation model is a threshold function
•The graph of this function looks like this:
1
0
θ
f
i
n
iiwx∑
=1
Properties of Artificial NeuralNets (ANNs)
• Many simple neuron-like threshold switching units
• Many weighted interconnections among units
• Highly parallel, distributed processing
• Learning is accomplished by tuning the connectionweights (wi) through training
• Training is usually separate from actual usage
Appropriate Problem Domainsfor Neural Network Learning
• Input is high-dimensional discrete or real-valued(e.g. raw sensor input, pixels of a CCD)
• Output is discrete or real valued
• Output is a vector of values
• Form of target function is unknown
• Humans do not need to interpret the results (blackbox model)
Perceptron Learning: Delta Rulewi = wi + ∆wi∆wi = η (t - o) xit is the target valueo is the perceptron outputη Is a small constant (e.g. 0.1) called learning rate
• If the output is correct (t=o) the weights wi are not changed• If the output is incorrect (t≠o) the weights wi are changed such that the output of the perceptron for the new weights is closer to t.• The algorithm converges to the correct classification
• if the training data is linearly separable• and η is sufficiently small
Supervised Learning
• Training and test data sets
• Training set:• 2 data components required: inputs & target
Perceptron Training
• Linear threshold is used.
• W - weight value
• t - threshold value
1 if Σ wi xi >tOutput= 0 otherwise{ i=0
Training Perceptrons
t = 0.0
B
A
-1W = ?
W = ?
W = ?
For ANDA B Output0 0 00 1 01 0 01 1 1
••What are the starting weightWhat are the starting weightvalues?values?••Initialize with random weight valuesInitialize with random weight values
Training Perceptrons
t = 0.0
B
A
-1W = 0.3
W = -0.4
W = 0.5
I1 I2 I3 Summation Output-1 0 0 (-1*0.3) + (0*0.5) + (0*-0.4) = -0.3 0-1 0 1 (-1*0.3) + (0*0.5) + (1*-0.4) = -0.7 0-1 1 0 (-1*0.3) + (1*0.5) + (0*-0.4) = 0.2 1-1 1 1 (-1*0.3) + (1*0.5) + (1*-0.4) = -0.2 0
For ANDA B Output0 0 00 1 01 0 01 1 1
I1
I2
I3
Learning algorithmEpoch : Presentation of the entire training set to the neural
network.In the case of the AND function an epoch consistsof four sets of inputs being presented to thenetwork (i.e. [0,0], [0,1], [1,0], [1,1])
Error: The error value is the amount by which the valueoutput by the network differs from the targetvalue. For example, if we required the network tooutput 0 and it output a 1, then Error = -1
Learning algorithmTarget Value, T : When we are training a network we
not only present it with the input but also with avalue that we require the network to produce. Forexample, if we present the network with [1,1] forthe AND function the training value will be 1
Output , O : The output value from the neuron
Xi : Inputs being presented to the neuron
Wi : Weight from input neuron (Ij) to the output neuron
η: The learning rate. This dictates how quickly thenetwork converges. It is set by a matter ofexperimentation. It is typically 0.1
∆wi = η (T - o) xi
Decision boundaries
• In simple cases, divide feature space bydrawing a hyperplane across it.
• Known as a decision boundary.• Discriminant function: returns different values
on opposite sides. (straight line)
• Problems which can be thus classified arelinearly separable.
Decision Surface of aPerceptron
+
++
+ -
-
-
-x1
x2
+
+-
-
x1
x2
• Perceptron is able to represent some useful functions• AND(x1,x2) choose weights w0=-1.5, w1=1, w2=1• But functions that are not linearly separable (e.g. XOR) are not representable•Do additional layers help? No.
Linearly separable Non-Linearly separable
Multilayer Perceptron (MLP)FeedForward Networks
Output Values
Input Signals (External Stimuli)
Output Layer
AdjustableWeights
Input Layer
Different Non-LinearlySeparable Problems
StructureTypes of
Decision RegionsExclusive-OR
ProblemClasses with
Meshed regionsMost General
Region Shapes
Single-Layer
Two-Layer
Three-Layer
Half PlaneBounded ByHyperplane
Convex OpenOr
Closed Regions
Arbitrary(Complexity
Limited by No.of Nodes)
A
AB
B
A
AB
B
A
AB
B
BA
BA
BA
Types of Layers• The input layer.
– Introduces input values into the network.– No activation function or other processing.
• The hidden layer(s).– Perform classification of features– Two hidden layers are generally sufficient to solve
any problem– More solution details imply more layers may be better
• The output layer.– Functionally just like the hidden layers– Outputs are passed on to the world outside the neural
network (using weights).
Standard activationfunctions
• The hard-limiting threshold function– Corresponds to the biological paradigm
• either fires or not
• Sigmoid functions ('S'-shaped curves)
– The logistic function– The hyperbolic tangent (symmetrical)– Both functions have a simple differential– Only the shape is important– Sigmoid is differentiable
φ(x) = 1
1 + e -ax
Activation functions• Transforms neuron’s input into output.• Features of activation functions:
• A squashing effect is required• Prevents accelerating growth of activation
levels through the network.
• Simple and easy to calculate
Standard activationfunctions
• The hard-limiting threshold function– Corresponds to the biological paradigm
• either fires or not
• Sigmoid functions ('S'-shaped curves)
– The logistic function– The hyperbolic tangent (symmetrical)– Both functions have a simple differential– Only the shape is important– Sigmoid is differentiable
φ(x) = 1
1 + e -ax
Training/Learning Algorithms
• Adjust neural network weights to map inputs tooutputs.
• Use a training set of sample patterns/data asbefore, where the desired output (given theinputs presented) is known.
• The purpose is to learn to generalize– Recognize features which are common to
good and bad examples
Back-Propagation
• A training procedure which allows multi-layerfeed-forward Neural Networks to be trained;
• Can theoretically perform “any” input-outputmapping;
• Can learn to solve linearly inseparable problems.
Activation functions andtraining
• For feed-forward networks:• A continuous function can be differentiated
allowing gradient-descent.• Back-propagation is an example of a
gradient-descent technique.• Reason for prevalence of sigmoid
Gradient Descent LearningRule
• Consider the linear sum unit without threshold andcontinuous output o (not just –1,1 or 0, 1)
o=w0 + w1 x1 + … + wn xn
• Train the wi’s such that they minimize the squarederror
E[w1,…,wn] = ½ Σd∈ D (td-od)2
where D is the set of training examples
Gradient Descent
(w1,w2)
(w1+∆w1,w2 +∆w2)∆w=-η ∇ E[w]
∆wi=-η ∂E/∂wi
=∂/∂wi 1/2Σd(td-od)2
= ∂/∂wi 1/2Σd(td-Σi wi xi)2
= Σd(td- od)(-xi)
Gradient:∇ E[w]=[∂E/∂w0,… ∂E/∂wn]
Gradient Descent AlgorithmGradient-Descent(training_examples, η)
Each training example is a pair of the form <(x1,…xn),t> where (x1,…,xn) is thevector of input values, and t is the target output value, η is the learning rate (e.g.0.1)
• Initialize each wi to some small random value
• Until the termination condition is met, Do
• Initialize each ∆wi to zero
• For each <(x1,…xn),t> in training_examples Do
• Input the instance (x1,…,xn) to the linear unit and compute the output o
• For each linear unit weight wi Do
– ∆wi= ∆wi + η (t-o) xi
• For each linear unit weight wi Do
• wi=wi+∆wi
Incremental StochasticGradient Descent
• Batch mode : gradient descent
w=w - η ∇ ED[w] over the entire data D
ED[w]=1/2Σd(td-od)2
• Incremental mode: gradient descent
w=w - η ∇ Ed[w] over individual training examples d Ed[w]=1/2 (td-od)2
Incremental Gradient Descent can approximate Batch GradientDescent arbitrarily closely if η is small enough
Comparison Perceptron andGradient Descent Rule
Perceptron or Delta learning rule guaranteed to succeed if• Training examples are linearly separable• Sufficiently small learning rate η
Linear unit training rules using gradient descent• Guaranteed to converge to hypothesis with minimum squared
error• Given sufficiently small learning rate η• Even when training data contains noise• Even when training data not linearly separable
Sigmoid Unit
Σ
x1
x2
xn
...
w1
w2
wn
w0
x0=1
sum=Σi=0n wi xi
o
o=σ(sum)=1/(1+e-net)
σ(x) is the sigmoid function: 1/(1+e-x)
dσ(x)/dx= σ(x) (1- σ(x))Derive gradient decent rules to train:• one sigmoid function ∂E/∂wi = -Σd(td-od) od (1-od) xi• Multilayer networks of sigmoid units backpropagation:
Backpropagation Algorithm
• Initialize each wi to some small random value
• Until the termination condition is met, Do
• For each training example <(x1,…xn),t> Do
• Input the instance (x1,…,xn) to the network and compute thenetwork outputs ok
• For each output unit k
δk=ok(1-ok)(tk-ok)
• For each hidden unit h
δh=oh(1-oh) Σk wh,k δk
• For each network weight w,j Do
• wi,j=wi,j+∆wi,j where
∆wi,j= η δj xi,j
Backpropagation• Gradient descent over entire network weight vector• Easily generalized to arbitrary directed graphs• Will find a local, not necessarily global error minimum
-in practice often works well (can be invoked multiple timeswith different initial weights)
• Often include weight momentum term∆wi,j(n)= η δj xi,j + α ∆wi,j (n-1)
• Minimizes error training examples• Will it generalize well to unseen instances (over-fitting)?
• Training can be slow typical 1000-10000 iterations (use Levenberg-Marquardt instead of gradient descent)• Using network after training is fast
Hidden weights.89 .04 .08.01 .11 .88.01 .97 .27.99 .97 .71.03 .05 .02.22 .99 .99.80 .01 .98.60 .94 .01
Ed[w]=1/2 (td-od)2
Sum of Squared Errors for theOutput Units
Convergence of Backprop
Gradient descent to some local minimum• Perhaps not global minimum• Add momentum• Stochastic gradient descent• Train multiple nets with different initial weights
Nature of convergence• Initialize weights near zero• Therefore, initial networks near-linear• Increasingly non-linear functions possible as training
progresses
Applications• The properties of neural networks define where
they are useful.– Can learn complex mappings from inputs to outputs,
based solely on samples.– Difficult to analyse: firm predictions about neural
network behaviour difficult;• Questionable for safety-critical applications.
– Require limited understanding from trainer, who canbe guided by heuristics.
– Defined operational area: ANNs cannot extrapolate.Training sets bounds model accuracy.
Neural network for OCR
• feedforwardnetwork
• trained using Back-propagation
��������������
������������������������
��������
������������������������
����������������
��������������
���������������
����������������
����������������
������������������������
������������������������
��������
����������������
����������������
��������������
������������������������
��������
������������������������
���������������
���������������
������������������������
������������������������
����������������
��������������
���������������
����������������
����������������
��������������
����������������
������������������������
��������������������������������
������������������������
��������������
����������������
����������������
����������������
����������������
��������������
������������������������
������������������������
����������������
��������������
������������������������
������������������������
����������������
����������������
��������������
���������������
����������������
����������������
A
B
E
D
C
Output Layer
Input Layer
Hidden Layer
OCR for 8x10 characters
• NN are able to generalize
• learning involves generating a partitioning of the input space
• for single layer network input space must be linearly separable
• what is the dimension of this input space?
• how many points in the input space?
• this network is binary(uses binary values)
• networks may also be continuous
����������������
����������������
����������������
���������������������
�������
������������������������
��������������������������������
��������������������������������
������������������������
����������������������
������������������������
����������������
����������������
���������������
�������
����������������
��������������
������������������������
��������
������������������������
��������
������������������������
��������������������������������
������������������������
������������������������
������������������������
��������
������������������������
���������������
���������������
��������
����������������
����������������
����������������
��������������
����������������
�����������������������
�������
�����������������������
����������������������������
���������������������
����������������������������
���������������������
������������������������
����������������
����������������
���������������
���������������
����������������
��������
8
10
8 8
1010
Engine management
• The behaviour of a car engine is influencedby a large number of parameters– temperature at various points– fuel/air mixture– lubricant viscosity.
• Major companies have used neural networksto dynamically tune an engine depending oncurrent settings.
ALVINNDrives 70 mph on a public highway
30x32 pixelsas inputs
30 outputsfor steering4 hiddenunits
30x32 weightsinto one out offour hiddenunit
Signature recognition
• Each person's signature is different.• There are structural similarities which are
difficult to quantify.• One company has manufactured a machine
which recognizes signatures to within a highlevel of accuracy.– Considers speed in addition to gross shape.– Makes forgery even more difficult.
Sonar target recognition
• Distinguish mines from rocks on sea-bed• The neural network is provided with a large
number of parameters which are extractedfrom the sonar signal.
• The training set consists of sets of signalsfrom rocks and mines.
Stock market prediction
• “Technical trading” refers to trading basedsolely on known statistical parameters; e.g.previous price
• Neural networks have been used to attemptto predict changes in prices.
• Difficult to assess success since companiesusing these techniques are reluctant todisclose information.
Mortgage assessment
• Assess risk of lending to an individual.• Difficult to decide on marginal cases.• Neural networks have been trained to make
decisions, based upon the opinions of expertunderwriters.
• Neural network produced a 12% reduction indelinquencies compared with human experts.
Parameter setting
• Number of layers• Number of neurons
• too many neurons, require more training time
• Learning rate• from experience, value should be small ~0.1
• Momentum term• ..
Over-fitting
• With sufficient nodes can classify anytraining set exactly
• May have poor generalization ability.• Cross-validation with some patterns
– Typically 30% of training patterns– Validation set error is checked each epoch– Stop training if validation error goes up
Training time
• How many epochs of training?– Stop if the error fails to improve (has reached a
minimum)– Stop if the rate of improvement drops below a
certain level– Stop if the error reaches an acceptable level– Stop when a certain number of epochs have
passed