EE04 804(B) Soft Computing Ver. 1.2

Class 2. Neural Networks - I

Feb – 23 , 2012

1

Sasidharan Sreedharan www.sasidharan.webs.com

3/1/2012

Syllabus

Artificial Intelligence Systems- Neural

Networks, fuzzy logic, genetic

algorithms, Artificial neural networks:

Biological neural networks, model of an

artificial neuron, Activation functions,

architectures, characteristics- learning

methods, brief history of ANN research-

Early ANN architectures (basics only)-

McCulloh & Pitts model, Perceptron,

ADALINE, MADALINE

Biological Neural Nets

• Pigeons as art experts (Watanabe et al.

1995)

– Experiment:

• Pigeon in Skinner box

• Present paintings of two different artists (e.g.

Chagall / Van Gogh)

• Reward for pecking when presented a

particular artist (e.g. Van Gogh)

• Pigeons were able to discriminate between Van

Gogh and Chagall with 95% accuracy (when

presented with pictures they had been trained on)

• Discrimination still 85% successful for previously

unseen paintings of the artists

• Pigeons do not simply memorise the pictures

• They can extract and recognise patterns (the

‘style’)

• They generalise from the already seen to make

predictions

• This is what neural networks (biological and artificial)

are good at (unlike conventional computer)

•Human brain contains a massively

interconnected net of 1010-1011 (10 billion)

neurons (cortical cells)

•Biological Neuron

•- The simple “arithmetic computing”

element

Brain Computer: What is it?

8

•The schematic model of a biological

neuron

•Synapses

•Dendrites

•Soma

•Axon •Dendrite

from other

•Axon from other neuron

1. Soma or body cell - is a large, round central body in which almost all the logical functions of the neuron are realized.

2. The axon (output), is a nerve fibre

attached to the soma which can serve as a final output channel of the neuron. An axon is usually highly branched.

3. The dendrites (inputs)- represent a highly branching tree of fibers. These long irregularly shaped nerve fibers

(processes) are attached to the soma.

4. Synapses are specialized contacts on a neuron which are the termination points for the axons from other neurons.

Biological Neurons

9

•?

•Brain-Like Computer

•Brain-like computer –

•is a mathematical model of humane-brain principles of computations. This computer consists of those elements which can be called the biological neuron prototypes, which are interconnected by direct links called connections and which cooperate to perform parallel distributed processing (PDP) in order to solve a desired computational task.

•Neurons and Neural Net

•The new paradigm of computing mathematics consists of the combination of such artificial neurons into some artificial neuron net.

•Artificial Neural Network – Mathematical Paradigms of Brain-Like Computer

Brain-like Computer

10

•NN as an model of brain-like Computer

An artificial neural network (ANN) is a massively parallel distributed processor that has a natural property for storing experimental knowledge and making it available for use. It means that:

•

Knowledge is acquired by the network through a learning (training) process;

The strength of the interconnections between neurons is implemented by means of the synaptic weights used to store the knowledge.

•The learning process is a procedure of the adapting the weights with a learning algorithm in order to capture the knowledge. On more mathematically, the aim of the learning process is to map a given relation between inputs and output (outputs) of the network.

•Brain

The human brain is still not well understood and indeed its behavior is very complex!

There are about 10 billion neurons in the human cortex and 60 trillion synapses of connections

The brain is a highly complex, nonlinear and parallel computer (information-processing system)

ANN as a Brain-Like Computer

11

•Artificial Intellect with

Neural Networks

•Intelligent

Control

•Technical

Diagnistics

•Intelligent

• Data Analysis

•and Signal

Processing

•Advance

Robotics

•Machine

Vision

•Image &

Pattern

Recognition

•Intelligent

Security

Systems

•Intelligent

Medicine

Devices

•Intelligent

Expert

Systems

Applications of Artificial Neural

Networks

12

ANNs – The basics

• ANNs incorporate the two fundamental components

of biological neural nets:

1. Neurones (nodes)

2. Synapses (weights)

• Neurone vs. Node

• Structure of a node:

• Squashing function limits node output:

• Synapse vs. weight

Feed-forward nets

• Information flow is unidirectional

• Data is presented to Input layer

• Passed on to Hidden Layer

• Passed on to Output layer

• Information is distributed

• Information processing is parallel

Internal representation (interpretation) of data

• Feeding data through the net:

(1 0.25) + (0.5 (-1.5)) = 0.25 + (-0.75) = - 0.5

0.37751

1

5.0

eSquashing:

Artificial neuron

19

• Human brain is a complex structure of highly

interconnected network of simple processing element called

neurons.

• The behaviour of a neuron can be captured as a simple

model.

• Every component of the model bears a direct analogy to the

actual constituents of a biological neuron and is termed as the

artificial neuron and is the basis for ANN.

• Biological neuron receives all inputs through the dendrites

sums them and produces an output if the sum is greater than

a threshold value.

• The input signals are passed on to the cell body through the

synapse which may accelerate and retard an arriving signal.

• The acceleration or retardation of the input signals are

modelled by the weights.

Artificial Neuron – Simple Model

20

1 1 2 2 n n

n

i i

i=1

I=w x +w x +...+w x

= w x

y= (I):W here is

activation function or transfer function or squash function

which releases the output.

I z

•Linear activation

•Threshold activation •Hyperbolic tangent activation

•Logistic activation

2

2

1

1

u

u

eu tanh u

e

1

1I

Ie

1, 0,

sign ( )1, 0.

if II I

if I

•input

•z

•1

•-1

•1

•0

•0

Σ

1

-1

Artificial Neuron:

Classical Activation Functions

21

•input

1. Threshold Activation function

22

i

1

y = w x

w here the step function

know n as the H eaviside function

( ) = 1, I 0

0, I 0

n

i

i

is

I

Output signal is either 1 or 0 resulting in the neuron being on or off

2. Signum function

23

( ) = 1, I

1, I

I

3. Sigmoid function

24

1 ( )

1T

Ie

The function is a continuous function that varies gradually between the

asymptotic values 0 and 1 or -1 and +1

4. Hyperbolic tangent function

25

( ) tanh( )I I

McCulloch – Pitts Model (1943)

Neural Network Architectures

27

• ANN is defined as a data processing system consisting of a

large number of simple highly interconnected processing

elements (artificial neurons).

• ANN structure can be represented by a directed graph

(digraph – directed graph or oriented graph)

• The vertices of the graph represents neurons (input/output)

and the edges, the synaptic links. The edges are labelled by

the weights attached to the synaptic links

1.Single Layer Feed Forward

Network

2.Multi Layer Feed forward

network

• Presence of hidden intermediate layer which are useful for

computations.

3.Recurrent ANN Architecture

• Presence of at least one feedback loop.

• There could be self – feedback links

Regards

Dr. Sasidharan Sreedharan

www.sasisreedhar.webs.com