NEURAL NETWORKS

• Biological analogy• Introduction to Artificial Neural Networks• Typical architectures

João Sousa, José Borges XI.2

Biological neuron

• Soma: body of the neuron.• Dendrites: receptors (inputs) of the neuron.• Axon: output of neuron; connected to dendrites of other

neurons via synapses.• Synapses: transfer of information between neurons

(electrical-chemical-electrical).

João Sousa, José Borges XI.3

Neural networks

• Biological neural networks• Neuron switching time: 0.001 second• Number of neurons: 1010

• Connections per neuron (synapses): 104,5

• Recognition time: 0.1 s

para

llel c

ompu

tati

on

• Artificial neural networks• Weighted connections amongst units• Highly parallel, distributed process• Emphasis on tuning weights automatically

João Sousa, José Borges XI.4

Artificial Neural Networks

• Artificial Neuron

Threshold function

Piece-wise Linear Sigmoidal function

João Sousa, José Borges XI.5

Use of Artificial Neural Networks

• Input is high-dimensional • Output is multidimensional• Mathematical form of system is unknown• Interpretability of identified model is unimportant

Biological neural network

Artificial neural network

Soma Neuron

Dendrite Input

Axon Output

Synapse Weight

• Applications• Pattern recognition

• Classification

• Prediction

• Modeling

João Sousa, José Borges XI.6

Architectures of typical ANN

Out

puts

igna

ls

• Feedforward ANN

João Sousa, José Borges XI.7

Architectures of typical ANN

• Recurrent ANN

ADAPTIVE NETWORKS

• Adaptive ANN• Network Classification• Backpropagation

João Sousa, José Borges XI.9

Adaptive (neural) networks

• Massively connected computational units inspired by the working of the human brain

• Provide a mathematical model for biological neural networks (brains)

• Characteristics:• learning from examples• adaptive and fault tolerant• robust for fulfilling complex tasks

João Sousa, José Borges XI.10

Network classification

• Learning methods: supervised, unsupervised

• Architectures: feedforward, recurrent

• Output types: binary, continuous

• Node types: uniform, hybrid

• Implementations: software, hardware

• Connection weights: adjustable, hard-wired

• Inspirations: biological, psychological

João Sousa, José Borges XI.11

Adaptive network

• Nodes can be static or parametric• Network can consist of heterogeneous nodes• Links do not have parameters associated• Node functions are differentiable except at a finite number

of points

adaptive nodes

x1

x2

4

5

36

7 9

8

Input layer Layer 1 Layer 2 Output layer

x8

x9

fixed nodes

João Sousa, José Borges XI.12

Calculating with a network

),( axfy

x f y

x

a

f y

),(),,( ayhvaxgu

x g u

y h va

a

y h v

x g u

João Sousa, José Borges XI.13

Backpropagation learning rule

• Simple gradient descent applied to layered networks• An overall error measure is minimized for P data

points and L layers

2

,11 1

LNPP

p k L kpp k

E E d x

change inparameter

change inoutputs of nodes

containing

change innetwork'soutputs

change inerror measure

• Derivative information propagated by the use of chain rule,

João Sousa, José Borges XI.14

Ordered vs. partial derivatives

y

x

f

g z )(

),(

xfy

yxgz

( , )z g x y

x x

x

xf

y

yxg

x

yxg

x

xfxg

x

z

xfyxfy

)(),(),(

))(,(

)()(

partial derivative

ordered derivative

João Sousa, José Borges XI.15

BP for feedforward networks

• Define an error signal at each node

iL

p

iL

piL x

E

x

E

,,,

output node

il

mlN

mml

il

mlN

m ml

p

iL

piL x

f

x

f

x

E

x

E ll

,

,1

1,1

,

,1

1 ,1,,

11

hidden layer node

il

ilil

il

pp ff

x

EE ,,

,

,

P

p

pEE

1

João Sousa, José Borges XI.16

Error propagation network

x1

x2

4

5

36

7 9

8 x8

x9

1

24

5

36

7 9

8 8

9

w83

w97w52 w75

w31

6

99

6

88

6

9

96

8

866 x

f

x

f

x

f

x

E

x

f

x

E

x

E ppp