An Introduction Artificial Neural Network

8/9/2019 An Introduction Artificial Neural Network

http://slidepdf.com/reader/full/an-introduction-artificial-neural-network 1/22

An Introduction to Artificial neural network | Kaushik Bose

An Introduction toArtificial neural network

TERM PAPER DOCUMENT

B.TECH 6TH SEMESTER

PAPER – CSEB 605(P)

Kaushik Bose 13/06/2014 ROLL NO. – 91/CSE/111029




1

TABLE OF CONTENTS

Introduction ......................................................... ................................................................. .................................. 3

Basics ................................................................ ................................................................ .................................. 3

Networks ............................................................................................................................................................ 3

Why Neural Networks? ...................................................................................................................................... 4

Technical Viewpoint .......................................................... ................................................................. ............ 4

Biological Viewpoint ...................................................................................................................................... 4

Biological neural networks ..................................................................................................................................... 5

Characteristics that ann share with biological neural system ............................................................................ 6

Artificial Neural Networks ...................................................................................................................................... 7

What is neural network ......................................................... ................................................................. ............ 7

Formal definition of artificial neuron network ................................................................. .................................. 7 Characterization of ANN..................................................................................................................................... 7

A General framework for ann models ......................................................... ....................................................... 8

Neurons the basic computational entities ............................................................ ............................................. 8

The Perceptron and Linear Separability ........................................................................................................... 10

Perceptron for Classification ........................................................ .............................................................. .. 11

limitations of perceptron ............................................................................................................................. 11

Artificial Neural Network Architecture or topology ............................................................................................. 12

Architecture based on number of layers ............................................................... ........................................... 12

single layer neural network ......................................................................................................................... 12

Multilayer Neural Network .......................................................................................................................... 12

Architecture based on the connection pattern ................................................................ ................................ 14

totally connected neuron network ................................................................... ........................................... 14

partially connected neuron network ........................................................................................................... 14

ARCHITECTURE BASED ON information flow ................................................................................................... 15

Feed-forward neural network ................................................................ ...................................................... 15

feed-back or recurrent neural network ...................................... ............................................................... .. 15

ANN learning process ........................................................................................................................................... 16

supervised learning ........................................................... ................................................................. .......... 16

reinforcement learning ........................................................................................................... ..................... 16

unsupervised learning ....................................................... ................................................................. .......... 16

Back propagation ......................................................................................................................................... 17

learning laws .................................................................................................................................................... 17

Hebb's Rule: ................................................................................................................................................. 17

Hopfield Law: ............................................................................................................................................... 17




2

The Delta Rule: ....................................................... ................................................................. ..................... 17

Kohonen’s Learning Law: ............................................................................................................................. 17

Benefits of neural networks ................................................................................................................................. 18

Nonlinearity ..................................................... ................................................................. ................................ 18

input-output mapping ........................................................... ................................................................. .......... 18

adaptivity.......................................................................................................................................................... 19

fault tolerance ............................................................ ................................................................. ..................... 19

Neurobiological Analogy .................................................................................................................................. 19

Applications of ann ............................................................................................................... ................................ 19

Signal Processing ........................................................ ................................................................. ..................... 19

Pattern Recognition ......................................................................................................................................... 19

Medicine .......................................................... ................................................................. ................................ 20

Speech Production ........................................................................................................................................... 20

Speech Recognition ............................................................... ................................................................. .......... 20

clustering/categorization ................................................................................................................................. 20

prediction/forecasting...................................................................................................................................... 20

optimization ..................................................................................................................................................... 20

Future scope OF artificial NEURAL NETWORKS .................................................................................................... 21

References ............................................................................................................................................................ 21




3

INTRODUCTION

BASICS

The great majority of digital computers in use today are based around the

principle of using one very powerful processor through which all computations

are channelled. This is the so called von Neumann architecture, after John von

Neumann, one of the pioneers of modern computing. The power of such a

processor can be measured in terms of its speed (number of instructions that it

can execute in a unit of time) and complexity (the number of different

instructions that it can execute).

Nowadays there is a new field of computational science that integrates

the different methods of problem solving that cannot be so easily described

without an algorithmic traditional focus. These methods, in one way or another,

have their origin in the emulation, more or less intelligent, of the behaviour of

the biological systems.

It is a new way of computing denominated Artificial Intelligence, which

through different methods is capable of managing the impressions and

uncertainties that appear when trying to solve problems related to the real

world, offering strong solution and easy implementation. One of thosetechnique is known as Artificial Neuron Networks (ANN), inspired by the

functioning of human brain.

NETWORKS

One efficient way of solving complex problems is following the lemma

“divide and conquer”. A complex system may be decomposed into simpler

elements, in order to be able to understand it. Also simple elements may be

gathered to produce a complex system (Bar Yam, 1997). Networks are one

approach for achieving this. There are a large number of different types of

networks, but they all are characterized by the following components: a set of

nodes, and connections between nodes. The nodes can be seen as

computational units. They receive inputs, and process them to obtain an output.

This processing might be very simple (such as summing the inputs), or quite

complex (a node might contain another network...) the connections determine




4

the information flow between nodes. They can be unidirectional, when the

information flows only in one sense, and bidirectional, when the information

flows in either sense.

Networks are used to model a wide range of phenomena in physics,computer science, biochemistry, ethology, mathematics, sociology, economics,

telecommunications, and many other areas. This is because many systems can

be seen as a network: proteins, computers, communities, etc.

WHY NEURAL NETWORKS?

There are problem categories that cannot be formulated as an algorithm.

Problems that depend on many subtle factors, for example the purchase price of a real

estate which our brain can (approximately) calculate. Without an algorithm a

computer cannot do the same. Therefore the question to be asked is: How do we learn

to explore such problems?

So we need to learn; a capability computers obviously do not have. Humans

have a brain that can learn. Computers have some processing units and memory. They

allow the computer to perform the most complex numerical calculations in a very

short time, but they are not adaptive.

The largest part of the brain is working continuously, while the largest part ofthe computer is only passive data storage. Thus, the brain is parallel and therefore

performing close to its theoretical maximum, from which the computer is orders of

magnitude away. Additionally, a computer is static - the brain as a biological neural

network can reorganize itself during its "lifespan" and therefore is able to learn, to

compensate errors and so forth. There are two basic reasons why we are interested in

building artificial neural networks (ANNs):

TECHNICAL VIEWPOINT

Some problems such as character recognition or the prediction of future states

of a system require massively parallel and adaptive processing.

BIOLOGICAL VIEWPOINT

Artificial Neural Networks can be used to replicate and simulate components of

the human (or animal) brain, thereby giving us insight into natural information

processing.




5

BIOLOGICAL NEURAL NETWORKS

A biological neuron has three types of components that are of particular

interest in understanding an artificial neuron:

Dendrites

Soma

Axon

The many dendrites receive signals from other neurons. The signals are electric

impulses that are transmitted across a synaptic gap by means of a chemical process.

The action of the chemical transmitter modifies the incoming signal (typically, by

scaling the frequency of the signals that are received) in a manner similar to the action

of the weights in an artificial neural network.

The soma, or cell body, sums the incoming signals. When sufficient input is

received, the cell fires; that is, it transmits a signal over its axon to other cells. It is

often supposed that a cell either fires or doesn't at any instant of time, so that

transmitted signals can be treated as binary. However, the frequency of firing varies

and can be viewed as a signal of either greater or lesser magnitude. This corresponds

to looking at discrete time steps and summing all activity (signals received or signals

sent) at a particular point in time.

The transmission of the signal from a particular neuron is accomplished by an

action potential resulting from differential concentrations of ions on either side of the

neuron's axon sheath (the brain's "white matter"). The ions most directly involved are

potassium, sodium, and chloride.

Figure 1: Biological Neuron




6

Figure 2: Biological Neuron to Neuron Connection

CHARACTERISTICS THAT ANN SHARE WITH BIOLOGICAL NEURAL SYSTEM

The processing element receives many signals.

Signals may be modified by a weight at the receiving synapse.

The processing element sums the weighted inputs.

Under appropriate circumstances (sufficient input), the neuron

transmits a single output.

The output from a particular neuron may go to many other neurons (the

axon branches).

Systems are fault tolerant

Information processing is local (although other means of transmission,

such as the action of hormones, may suggest means of overall process

control).

Memory is distributed:

Long-term memory resides in the neurons' synapses or weights.

Short-term memory corresponds to the signals sent by the

neurons.

A synapse's strength may be modified by experience.

Neurotransmitters for synapses may be excitatory or inhibitory.




7

ARTIFICIAL NEURAL NETWORKS

WHAT IS NEURAL NETWORK

Work on artificial neural networks commonly referred to as “neural networks”,

has been motivated right from its inception by the recognition that human braincomputes in an entirely different way from the conventional digital computer. The

brain is a highly complex, nonlinear and parallel computer (information processing

system). It has the capability to organize its structural constituents, known as neurons,

so as to perform certain computations (e.g., pattern recognition, perception, and

motor control) many times faster than the fastest digital computer in existence today.

A neural network is a machine that is designed to model the way in which the

brain performs a particular task or function of interest; the network is usually

implemented by using electronic components or is simulated in software on a digital

computer. Our interest is confined largely to an important class of neural networks

that perform useful computations through a process of learning. To achieve good

performance, neural networks employ a massive interconnection of simple computing

cells referred to as "neurons" or "processing units."

FORMAL DEFINITION OF ARTIFICIAL NEURON NETWORK

An artificial neural network is a massively parallel distributed processor made

up of simple processing units, which has a natural propensity for storing experiential

knowledge and making it available for use.

It resembles the brain in two respects:

Knowledge is acquired by the network from its environment through a learning

process.

Interneuron connection strengths, known as synaptic weights, are used to store

the acquired knowledge.

CHARACTERIZATION OF ANN

A neural network is characterized by:

Its pattern of connections between the neurons (called its architecture)

Its method of determining the weights on the connections (called its training,

or learning, algorithm)

Its activation function.




8

A GENERAL FRAMEWORK FOR ANN MODELS

A neural net consists of a large number of simple processing elements called

neurons, units, cells, or nodes. Each neuron is connected to other neurons by means

of directed communication links, each with an associated weight. The weights

represent information being used by the net to solve a problem. Neural nets can be

applied to a wide variety of problems, such as storing and recalling data or patterns,

classifying patterns, performing general mappings from input patterns to output

patterns, grouping similar patterns, or finding solutions to constrained optimization

problems.

There are many different ANN models but each model can be precisely specified

by the following eight major aspects:

A set of processing units

A state of activation for each unit

An output function for each unit

A pattern of connectivity among units or topology of the network

A propagation rule, or combining function, to propagate the activities of

the units through the network

An activation rule to update the activities of each unit by using the

current activation value and the inputs received from other units

An external environment that provides information to the network

and/or interacts with it.

A learning rule to modify the pattern of connectivity by using

information provided by the external environment.

NEURONS THE BASIC CO MPUTATIONAL ENTITIES

The basic unit of neural networks, the artificial neurons, simulates the four basic

functions of natural neurons (receives inputs from other sources, combines them in

some way, performs a generally nonlinear operation on the result, and then output

the final result). Artificial neurons are much simpler than the biological neuron. Here

we identify three basic elements of the artificial neural model:

A set of synapses or connecting links, each of which is characterized by a weight

or strength of its own. Specifically, a signal x j at the input of synapse j connected

to neuron k is multiplied by the synaptic weight w j. Unlike a synapse in the brain,

the synaptic weight of an artificial neuron may lie in a range that includes

negative as well as positive values.




9

An adder for summing the input signals, weighted by the respective synapses

of the neuron; the operations described here constitute a linear combiner.

An activation function for limiting the amplitude of the output of a neuron. The

activation function is also referred to as a squashing function in that it squashes

(limits) the permissible amplitude range of the output signal to some finite

value.

Figure 3: Nonlinear model of an Artificial Neuron

The neuronal model of Figure 1 also includes an externally applied bias, denoted by b.

The bias b, has the effect of increasing or lowering the net input of the activation

function; depending on whether it is positive or negative, respectively.

In mathematical terms, we may describe a neuron by writing the following pair of

equations:

m

1 j jxwu

j

And

)(uy b




10

Where x1, x2, …, xm are the input signals; w1, w2, … , wm are the synaptic weights

of neuron; u is the linear combiner output due to the input signals; b is the bias; ( . )

is the activation function; and y is the output signal of the neuron. The use of bias b,

has the effect of applying an affine transformation to the output u of the linearcombiner in the model of Figure 1.

v = u + b

v is called induced field of the neuron.

THE PERCEPTRON AND LINEAR SEPARABILITY

The Perceptron was the first supervised model of artificial neural network. In

1958 Frank Rosenblatt proposed the Perceptron model that can also be used as a

pattern classifier. The single-layer perceptron model consists of one layer of binary

input units and one layer of binary output units. There are no hidden layers and

therefore there is only one layer of modifiable weights.

A perceptron uses a step function that returns +1 if weighted sum of its input

>=0 and -1 otherwise.

Figure 4: The Single Layer Perceptron

0if 1

0if 1)(

v

v

v




12

ARTIFICIAL NEURAL NETWORK ARCHITECTURE OR TOPOLOGY

With the expression architecture, structure or topology of an artificial neuron

network we talk about the way in which computational neurons are organized in the

network. Particularly how the nodes are connected and how the information istransmitted through the network. The architecture can be classified in terms of three

aspects.

Number of levels or layers

Connection pattern

Information flow

ARCHITECTURE BASED ON NUMBER OF LAYERS

SINGLE LAYER NEURAL NETWORK

This is the simplest form of layered neural network. Here an input layer of the

source nodes (input nodes) that projects onto an output layer of neurons or vice versa.

Figure 6: Single layer Neural Network

MULTILAYER NEURAL NETWORK

A multilayer neural network is a network with one or more layers (or levels) of

nodes (the so-called hidden units) between the input layers and the output layers.

Multilayer neural networks can solve more complicated problems than can single-layer neural networks, but training may be more difficult.




13

Figure 7: Multilayer (Three-layer) Neural Network

Figure 8: Multilayer Neural Network




14

ARCHITECTURE BASED ON THE CONNECTION PATTERN

TOTALLY CONNECTED NEURON NETWORK

A neural network is said to be totally connected neural network when all the

output from a level get to all and each of the nodes in the following node. In this case

there will be more connections than nodes.

Figure 9: Totally connected Neural Network

PARTIALLY CONNECTED NEURON NETWORK

A neural network is said to be partially connected if a neuron of the first layer

does not have to be connected to all neurons on the second layer and so on.

Figure 10: Partially connected Neural Network




15

ARCHITECTURE BASED ON INFORMATION FLOW

FEED-FORWARD NEURAL NETWORK

In a feed forward Artificial Neuron Network a unit only sends its output to units

from which it does not receive an input directly or indirectly (via other units). In other

words, there are no feedback loops. A feed forward ANN arranged in layers, where the

units are connected only to the units situated in the next consecutive layer, is called a

strictly feed forward ANN.

Figure 11: A strictly Feed-Forward Neural Network

FEED-BACK OR RECURRENT NEURAL NETWORK

A neural network is said to be feed-back or recurrent if there is at least one

feedback loop.

Figure 12: A Feed-back Neural Network




16

ANN LEARNING PROCESS

Learning is a process by which the free parameters of a neural network are

adapted through a process of stimulation by the environment in which the network isembedded. The type of learning is determined by the manner in which the parameter

changes take place.

This definition of the learning process implies the following sequence of events:

The neural network is stimulated by an environment.

The neural network undergoes changes in its free parameters as a result

of this stimulation.

The neural network responds in a new way to the environment becauseof the changes that have occurred in its internal structure.

By learning we mean the procedure for modifying the weights and biases of a

network. The purpose of learning rule is to train the network to perform some task.

The learning process fall into four broad categories.

SUPERVISED LEARNING

In supervised learning the external environment also provides a desired output

for each one of the training input vectors and it is said that the external environment

acts as a "teacher".

REINFORCEMENT LEARNING

A special case of supervised learning is reinforcement learning where the

external environment only provides the information that the network output is "good"

or "bad", instead of giving the correct output. In the case of reinforcement learning it

is said that the external environment acts as a "critic".

UNSUPERVISED LEARNING

In unsupervised learning the external environment does not provide the

desired network output nor classifies it as good or bad. By using the correlation of the

input vector the learning rule changes the network weights in order to group the input

vector into "clusters" such that similar input vectors will produce similar network

outputs since they will belong to the same cluster. Ideally, the learning rule finds the

number of clusters and their respective centres, if they exist, for the training data. This

learning method is also called self-organization.




17

BACK PROPAGATION

This method is proven highly successful in training of multi-layered neural nets.

The network is not just given reinforcement for how it is doing on a task. Information

about errors is also filtered back through the system and is used to adjust the

connections between the layers, thus improving performance. A form of supervised

learning.

LEARNING LAWS

HEBB'S RULE:

The first, and undoubtedly the best known, learning rule was introduced by

Donald Hebb. The description appeared in his book T h e Organization of Behaviour in

1949. His basic rule is: If a neuron receives an input from another neuron, and if both

are highly active (mathematically have the same sign), the weight between the

neurons should be strengthened.

HOPFIELD LAW:

This law is similar to Hebb’s Rule with the exception that it specifies the

magnitude of the strengthening or weakening. It states, "If the desired output and the

input are both active or both inactive, increment the connection weight by the learning

rate, otherwise decrement the weight by the learning rate." (Most learning functions

have some provision for a learning rate, or a learning constant. Usually this term is

positive and between zero and one.)

THE DELTA RULE:

This rule is a further variation of Hebb's Rule. It is one of the most commonly

used. This rule is based on the simple idea of continuously modifying the strengths of

the input connections to reduce the difference (the delta) between the desired outputvalue and the actual output of a processing element. This rule changes the synaptic

weights in the way that minimizes the mean squared error of the network. This rule is

also referred to as the Widrow-Hoff Learning Rule and the Least Mean Square (LMS)

Learning Rule.

KOHONEN’S LEARNING LAW:

This procedure, developed by Teuvo Kohonen, was inspired by learning in

biological systems. In this procedure, the neurons compete for the opportunity tolearn, or to update their weights. The processing neuron with the largest output is




18

declared the winner and has the capability of inhibiting its competitors as well as

exciting its neighbours. Only the winner is permitted output, and only the winner plus

its neighbours are allowed to update their connection weights.

The Kohonen rule does not require desired output. Therefor it is implementedin the unsupervised methods of learning.

BENEFITS OF NEURAL NETWORKS

It is apparent that a neural network derives its computing power through, first,

its massively parallel distributed structure and, second, its ability to learn and

therefore generalize. Generalization refers to the neural network producing

reasonable outputs for inputs not encountered during training (learning). These two

information-processing capabilities make it possible for neural networks to solve

complex (large-scale) problems that are currently intractable.

NONLINEARITY

An artificial neuron can be linear or nonlinear. A neural network, made up of an

interconnection of nonlinear neurons, is itself nonlinear.

INPUT-OUTPUT MAPPING

A popular paradigm of learning called learning with a teacher or supervised

learning involves modification of the synaptic weights of a neural network by applyinga set of labelled training samples or task examples. Each example consists of a unique

input signal and a corresponding desired response. The network is presented with ·an

example picked at random from the set, and the synaptic weights (free parameters) of

the network are modified to minimize the difference between the desired response

and the actual response of the network produced by the input signal in accordance

with an appropriate statistical criterion. The training of the network is repeated for

many examples in the set until the network reaches a steady state where there are no

further significant changes in the synaptic weights. Thus the network learns from the

examples by constructing an input-output mapping for the problem at hand.




19

ADAPTIVITY

Neural networks have a built-in capability to adapt their synaptic weights to

changes in the surrounding environment. In particular, a neural network trained to

operate in a specific environment can be easily retrained to deal with minor changes

in the operating environmental conditions. Moreover when it is operating in a non-

stationary environment (i.e., one where statistics change with time), a neural network

can be designed to change its synaptic weights in real time.

FAULT TOLERANCE

A neural network, implemented in hardware form, has the potential to be

inherently fault tolerant, or capable of robust computation, in the sense that its

performance degrades gracefully under adverse operating conditions. For example, if

a neuron or its connecting links are damaged, recall of a stored pattern is impaired in

quality. However, due to the distributed nature of information stored in the network,

the damage has to be extensive before the overall response of the network is

degraded seriously.

NEUROBIOLOGICAL ANALOGY

The design of a neural network is motivated by analogy with the brain, which is

a living proof that fault tolerant parallel processing is not only physically possible but

also fast and powerful. Neurobiologists look to (artificial) neural networks as a

research tool for the interpretation of neurobiological phenomena. On the other hand,

engineers look to neurobiology for new ideas to solve problems more complex than

those based on conventional hard-wired design techniques.

APPLICATIONS OF ANN

SIGNAL PROCESSING

There are many applications of neural networks in the general area of signal

processing. One of the first commercial applications was (and still is) to suppress noise

on a telephone line. The neural net used for this purpose is a form of ADALINE.

PATTERN RECOGNITION

Many interesting problems fall into the general area of pattern recognition. One

specific area in which many neural network applications have been developed is the

automatic recognition of handwritten characters (digits or letters).




20

MEDICINE

One of many examples of the application of neural networks to medicine was

developed by Anderson et al. in the mid-1980s [Anderson, 1986; Anderson, Golden,

and Murphy, 1986]. It has been called the "Instant Physician" [Hecht Nielsen, 1990].

The idea behind this application is to train an auto associative memory neural network

(the "Brain-State-in-a-Box," described in Section 3.4.2) to store a large number of

medical records, each of which includes information on symptoms, diagnosis, and

treatment for a particular case. After training, the net can be presented with input

consisting of a set of symptoms; it will then find the fun stored pattern that represents

the "best" diagnosis and treatment.

SPEECH PRODUCTION

Learning to read English text aloud is a difficult task, because the correct

phonetic pronunciation of a letter depends on the context in which the letter appears.

SPEECH RECOGNITION

Progress is being made in the difficult area of speaker-independent recognition

of speech. Several types of neural networks have been used for speech recognition,

including multilayer nets.

CLUSTERING/CATEGORIZATION

In clustering, there are no training data with known class labels. A clustering

technique explores similarity between the patterns and places similar pattern in a

cluster.

PREDICTION/FORECASTING

Artificial neural network is used for Stock-market prediction and weather

forecasting.

OPTIMIZATION

A wide variety of problems in mathematics, statistics, engineering, science,

etc. can be posed as optimization problems. The goal of an optimization algorithm is

to find a solution satisfying a set of constraints such that an objective function is

minimized or maximized. ANN is used to solve this kind of problems.




21

FUTURE SCOPE OF ARTIFICIAL NEURAL NETWORKS

A great deal of research is going on in neural networks worldwide:

The basic research to networks which can respond to temporally varying

patterns.

The research on the techniques for implementing neural networks directly in

silicon. Already one chip commercially available exists, but it does not include

adaptation. Edinburgh University have implemented a neural network chip, and are

working on the learning problem.

There is particular interest in sensory and sensing applications: nets which learnto interpret real-world sensors and learn about their environment.

REFERENCES

“Neural Networks-A Comprehensive Foundation”, Simon Haykin

“Fundamentals Of Neural Networks”, Laurene Fausett “Artificial Neural Network: A Tutorial”, Anil K. Jain

“A Brief Introduction to Neural Networks”, David Kriesel

“Artificial Neural Networks”, Girish Kumar Jha

“Artificial Neural Networks for Beginners”, Carlos Gershenson

“Artificial Neural Networks Technology”, Dave Anderson and George

McNeill

http://en.wikipedia.org/wiki/Artificial_neural_network

An Introduction Artificial Neural Network

Documents

Transcript of An Introduction Artificial Neural Network