Module 1Introduction :Artificial Neuron-Activation function-Single layer and Multi layer networks-Training Artificial Neural Networks- Perceptron-Representation-Linear Separability-Learning-Training Algorithms

What is Artificial Neural Networks? Artificial neural systems can be considered as simplified mathematical models of brain like systems and they function as parallel distributed computing networks.Artificial neural systems, or neural networks, are physical cellular systems which can acquire, store, and utilize experiential knowledge

Definition of ANN (by Hetct-Nelsen)An artificial Neural Network is a parallel ,distributed information processing structure consisting of Processing Units interconnected via unidirectional signal channels called connections.Each processing unit has a single output connection that branches into as many collateral connections as desired.-each carries the same signal-the out put of processing units

Contd...

Processing Units:-

Can Possess a Local Memory

Can carry out localized information processingProcessing unit output can be of any desired mathematical type desired.The information processes in the unit is completely local, ie, input arriving at the units,values stored in memory.

Traditional Algorithamic Approach Vs Artificial Neural NetworkTraditional algorithmic approach is useful for problems where it is possible to finda precise sequence of mathematical operationsa precise sequence of rules.

Main weakness of Traditional approachSequential ComputationUsually instructions have to be executed in sequentially even if the two instruction are not related

Local RepresentationAny corruption in the sequence can ruin the entire process.As complexity of program increases the reliability decreases.OOPS tries to eliminate this problem

Main weakness of Traditional approach (Contd)Learning DifficultiesIf we define learning as the construction or modification of some computational representation or model, it is difficult to simulate learning by traditional methods.

Self decision ProblemsDigital computer can solve problems that are difficult for human,but it is very difficult to use them to automate tasks that human can solve with little effort.

ANN: Is it similar to Neuron?Artificial Neural Networks are biologically inspiredANN composed of elements similar to biological neuronThe anatomy may similar to that of anatomy of brain

ANN also has surprising number of brain Characteristicssuch as..1.Learn from previous examples2.Generalize from previous examples to new ones.3.Abstract essentials data from input with irrelevant dataDespite of this similarities nobody can say ANN will soon replace the functions of human brainThe actual intelligence exhibited by the most sophisticated ANN works is below the level of a tape worm.

REPRESENTATION OF A NEURON

Artificial NeuronSet of input applied is output from another neuronInput is multiplied by a corresponding weight ~ synaptic strength.All the weighted inputs are summed to get the activation level of the neuron.The Artificial Neuron was designed to mimic the first order Characteristics of biological neuron.

X1X2

X3X4w1w2w3w4NET= XWNET=X1*w1+ X2*w2+X3*w3+X4*w4Artificial Neuron

ACTIVATION FUNCTION(Transfer Functions)A function used in between the actual output and the NET.Activation function processes the NET .

NET= X*WOUT=F(NET)X,W are vectorsActivation function can be Simple Linear functionThe Threshold FunctionThe Sigmoid Function.

Simple Linear Function

OUT = K(NET)K is a constant

Threshold Function

OUT=1 if NET>TOUT=0 otherwise.

Hyperbolic Tangent Functions

OUT=tanh(x)

Simple Linear Function

OUT = K(NET)K is a constant

Threshold Function

OUT=1 if NET>TOUT=0 otherwise.

Sigmoid Function provides a nonlinear gain for Artificial Neuron.Sigmoid Function

Matlab Activation (Transfer) Functions

Single Input Neuron.Scalar input given (p) is multiplied by the scalar weight.A bias b is passed to the neuron.The summer output is given to the activation function The output a=f (wp+b)

InputGeneral Neurona= f( w*p+b)Single Input Neuron.

X1X2

X3X4W1,1W1,2W1,3W1,4NET= XWFOUTArtificial NeuronARTIFICIAL NEURON WITH ACTIVATION FUNCTIONbMulti input Neuron.

Neuron with R inputs (Abbreviated notation)

The first index represent the particular neuron destination for that weight.The second index represents the source of the signal fed to the neuron. The indices in W1,2 say that weight represents the connection to the first neuron from the second source. Weight Indices

Single Layer Neural NetworksOne Neuron , even with many inputs, may not be sufficient. A Layer of neurons connected in parallel is called single layer neuron.Each elements of the input vector is connected to all of the neuron of that layer.Layer includes: weight matrix , the summers (soma) ,bias b, activation function.

Single Layer Neural Networks with 3 NeuronW1,1W3,3W1,2p4

The Vector Notation Input vectorWeight MatrixW =

Multiple Layer Neural Networks

Layer of s neurons- Abbreviated notation.

Abbreviated representation-Two Layer Network

TRAINING OF NEURAL NETWORKSA network is trained so that a set of inputs produces the desired set of outputs.Training is accomplished by sequentially applying input vectors ,while adjusting networks weights according to a predetermined procedure.During training the network weights gradually converge to a value such that each input vector produces ad desired output vector.

Types Of Training Supervised Training.Unsupervised Training.

Supervised Training.Supervised training requires the pairing of input vector with a target vector representing the desired output. (Training Pair)The Network is usually trained with a number of such training pairs.

An input vector is applied output vector calculated difference (error) fed back-Network weights are changed accordingly-to minimize the error.The Training pairs are applied sequentially, errors are calculated and the weights adjusted for each vector, until the error for entire training set is in the acceptably low level. Supervised Training.(Contd)

Unsupervised trainingSupervised training methods are biologically implausible.Unsupervised training Methods are far more plausible It requires no target vectors for output.No comparison to predetermined ideal response.Training set consists solely of input vectors.

The Training Algorithms modifies the weights to produce output vectors that are consistent.Consistent similar input vectors will produce same output.Unsupervised method utilizes the statistical property of input vectors. Applying a vector from a given class to the input will produce a specific output vector, but there is no way to determine prior to training which output pattern will be produced by a given input vector class.

TYPES OF NETWORKSFEED FORWARD NETWORKSCOMPETATIVE NETWORKSRECURRENT NETWORKS

PerceptronPerceptron is a feed forward network. In this the summing unit multiplies the input vector by a weight and sums the weighted output.If this sum is greater than a predetermined threshold value, the output is one; otherwise Zero (in case of Hardlim and -1 in case of Hardlims)

X1X2

X3X4w1w2w3w4NET= XWFOUTArtificial NeuronThreshold

Perceptron RepresentationRepresentation & LearningRepresentation refers to the ability of the network to simulate a specified function.Learning requires the existence of a systematic procedure for adjusting the weights to produce that function.

Example: RepresentationCan we represent a Odd/Even Number discriminating machine by a perceptron?

A Basic Pattern Recognition Problem using Perceptron.

P=P1=Prototype of orangeP2=Prototype of apple

Two Input case: Single Neuron Preceptronp2a = hardlims (wp+b)

Single input neuron can classify the input vectors into two categories.Example 1:Let for the above two input perceptron w11=1 and w12=1Then a=hardlims([ 1 1 ]p+b)if b=1, n=[1 1 ]p+1=0 represent a boundary line.

P2Perceptron Decision Boundary

Example 2:Let for the above two input perceptron w11=-1 and w12=1Then a=hardlims([ -1 1 ]p+b)if b= -1, n=[-1 1 ]p-1=0 represent a boundary line.

1-11n>0n

The Key Property of Single-neuron perceptron is that it can separate the input vectors into two category.This category is determined by the equationWp + b =0.Single layer perceptron can be used only to recognize patterns which are LINEARLY SEPARABLE

Pattern recognition Example (Contd.)There are only two category, Hence we can use single-neuron perceptron.The input vector is of order 3x1.Perceptron equationa=Hardlims([w11 w12 w13] +b )

Here ,to implement this pattern recognition problem we have to select a linear boundary which separates the proto type vectors ( Here it is Apple and Orange ).Orange =Apple =

P3P1P2Apple (1 1 -1)

Hence the linear boundary between the output are a plane P1P3 . That is, P2 .=0.Wp+b=0 is here P2 .=0.([w11 w12 w13] +b )=0.

[0 1 0 ] + 0 =0.Hence weight matrix = [0 1 0 ].

Bias ,b =0.

Here the weight matrix is orthogonal to the Boundary Layer.

Is X-OR Problem is representational?Example 2Take two input XOR gate

X ValueY ValueDesired OutputPoint000A0011B0101B1110A1

A0B0B1A1xw1+ yw2

Exapmle:3Check whether AND, OR functions are linearly seperable ?Linear SeparabilityFor some class of function the input vectors can be separated geometrically .For two input case ,the separator is a straight line. For three inputs it can be done with a flat plane., cutting the resultant three dimensional space. For four or more inputs visualization is difficult . we can generalize to a space of n dimensions divided by a HYPERPLANE, which divides the space into four or more regions.

Overcoming Linear separability LimitationLinear separability limitation of single layer networks can be overcome by adding more layers.Multilayer networks can perform more general tasks.

Perceptron training AlgorithmTraining methods used can be summarized as followsApply an input pattern and calculate the output..If the output is correct, go to step 1.If the output is incorrect, and is zero, add each input to its corresponding weight;orIf the output is incorrect, and is one, subtract each input to its corresponding weight.3.Go to step 1.

THE DELTA RULEDelta rule is an important generalization of perceptron training algorithm .Perceptron training algorithm is generalized by introducing a term = ( T-A )

T = Target Output.A = Actual Output

If = 0 step 2a >0 step2b < 0 step2c

In any of these case ,the perceptron training algorithm is satisfied if is multiplied the value of each input xi and this product is added to the corresponding weight.a Learning Rate coeifficent is multiplied with xi product to allow the average size of weight changes.

= ( T-A ) i = xi wi(n+1) =wi(n)+ i Wherei = the correction associated with i th input xiwi(n+1) = the value of weight i after adjustmentwi(n) = the value of weight i before adjustment

Problems with Perceptron Training AlgorithmIt is difficult to determine whether the input sets are lineaerly seperable or not.In real world situation the inputs are often time varying and may be sepearable at one time and not at another.The number of steps required is not properly defined.There is no proof that the perceptron algorithms are faster than simply changing the values.

Module2Back propagation: Training Algorithm - Application - Network Configurations - Network Paralysis - Local Minima - Temporal instability.

BACK PROPAGATION Back propagation is a systematic method for training multilayer artificial neural networks