NN 5 2.09.98
Neural Networks 1
Associative Memories We consider now NN models for unsupervised learning problems called auto‐association problems
HOP
problems, called auto association problems. Association is the task of mapping patterns to patterns.In an associative memory the stimulus of an incomplete or corrupted pattern leads to the response of a stored pattern that corresponds in some manner to the input pattern.A neural network model most commonly used for (auto‐)
i ti bl i th H fi ld t kassociation problems is the Hopfield network.
NN 5 2
NN 5 2.09.98
Neural Networks 2
Example HOP
• States Bit maps
Intermediate state of Hopfield net
Corrupted image Output
NN 5 3
States Bit maps.• Attractors Prototype patterns.• Input An arbitrary pattern
(e.g. picture with noise).• Output The best prototype for that pattern.
HOPFIELD NETWORKSThe Hopfield network implements a so‐called content addressable memory
HOP
addressable memory. A collection of patterns called fundamental memories is stored in the NN by means of weights.Each neuron represents a component of the input.The weight of the link between two neurons measures the correlation between the two corresponding components correlation between the two corresponding components over the fundamental memories. If the weight is high then the corresponding components are often equal in the fundamental memories.
NN 5 4
NN 5 2.09.98
Neural Networks 3
ARCHITECTURE: recurrent
HOP
z-1
z-1
z-1
Multiple-loop feedback systemwith no self-feedback
NN 5 5
z 1
unit-delay operator
Hopfield discrete NN Input vectors values are in {‐1,1} (or {0,1}).The number of neurons is equal to the input dimension.
HOP
q pEvery neuron has a link from every other neuron (recurrent architecture) except itself (no self‐feedback).The neuron state at time n is its output value.The network state at time n is the vector of neurons states. The activation function used to update a neuron state is the i f ti b t if th i t f th ti ti f ti i sign function but if the input of the activation function is 0 then the new output (state) of the neuron is equal to the old one.Weights are symmetric:
NN 5 6
jiij ww =
NN 5 2.09.98
Neural Networks 4
Notation
HOP
N: input dimension.M: number of fundamental memories.
i‐th component of the fundamental memory.State of neuron i at time n.
if μ μ)(nxi
NN 5 7
Weights computation 1. Storage. Let f1, f2, … , fM denote a known set of N-dimensional fundamental memories. The synaptic weights of
HOP
the network are:
⎪⎩
⎪⎨
⎧
=
=≠= ∑
=
ij
Nji jffw
M
jiMji
0
.......1, 1
,,1
μμμ
NN 5 8
where wji is the weight from neuron i to neuron j. The elements of the vectors fμ are in {-1,+1}. Once computed, the synaptic weights are fixed.
NN 5 2.09.98
Neural Networks 5
NN Execution
HOP
2. Initialisation. Let x probe denote an input vector (probe) presented to the network. The algorithm is initialised by setting:
Njxx jbj 1)0( ==where xj(0) is the state of neuron j at time n = 0, and x probe,j is the j-th element of the probe vector x probe.
3. Iteration Until Convergence. Update the elements of network state vector x(n) asynchronously (i.e. randomly and one at the time) according to the rule
NjnxwsignnxN
ijij , ... 2, ,1 )()1( =⎥⎦
⎤⎢⎣
⎡=+ ∑
, ... , Njxx jprobej 1 )0( ,
NN 5 9
4. Outputting. Let denote the fixed point (or stable state, that is such
that x(n+1)=x(n)) computed at the end of step 3.
The resulting output y of the network is:
Repeat the iteration until the state vector x remains unchanged.i
jj1
⎥⎦
⎢⎣∑=
fixedxy =
fixedx
Example 1 (1, 1, 1)(-1, 1, 1)
(-1, 1, -1)(1, 1, -1)
1+
weight
HOP
(-1, -1, -1)
(-1, -1, 1)(1, -1, 1)
(1, -1, -1)
neuron2 3
--
-
-
+
+
-1 1 -1stable states
NN 5 10
attraction basin 1 attraction basin 2-1 -1 -1
1 -1 -1
1 1 -1
-1 -1 1 1 1 1
-1 1 1
1 -1 1
NN 5 2.09.98
Neural Networks 6
Example 2 Separation of patterns using the two fundamental memories (‐1 ‐1 ‐1) ( 1 1 1):
HOP
1 1 1) ( 1 1 1): Find weights to obtain the following behavior:
-1 -1 -1
-1 -1 1
-1 1 1
-1 1 -1
1 -1 1 1 1 -1
1 -1 -11
NN 5 11
-1 1 1 1 -1 1 1 1 -1
1 1 1 2 3
wij = ?
CONVERGENCEEvery stable state x is at an “energy minimum”. (A state x is stable if x(n+1)=x(n))
HOP
What is “energy”?Energy is a function (Lyapunov function):
E: States →
such that every firing (change of its output value) of a neuron decreases the value of E:
NN 5 12
∑−=
−=
ji
T xWxxE
,jiji2
121
xxw
)(x → x’ implies E(x’) < E(x)
NN 5 2.09.98
Neural Networks 7
CONVERGENCEA stored pattern establishes a correlation between pairs of neurons: neurons tend to have the same state or opposite state according to their values in the pattern.
HOP
according to their values in the pattern.If wji is large, this expresses an expectation that neurons i and j are positively correlated similarly. If it is small (very negative) this indicates a negative correlation.
• will thus be large for a state x, which is a stored pattern (since wji will be positive if xi xj > 0 and negative if xi xj )
∑ji
jiij xxw,
NN 5 13
< 0).
• The negative of the sum will thus be small.
Energy DecreasingClaim:
Firing a neuron decreases the energy E.
f
HOP
Proof:Let l be the neuron that fires:
Either:
1. xl goes from -1 to 1 which implies
(above threshold)01
>∑=
N
iii xwl
2 f 1 t 1 hi h i li
NN 5 14
2. xl goes from 1 to -1 which implies
(below threshold)01
<∑=
N
iii xwl
Thus in both cases: 0)'(1
>− ∑=
N
iii xwxx lll
value after
NN 5 2.09.98
Neural Networks 8
ProofNow we try to “factor” xl from the energy function E:
∑∑−== =
xxw1 1
jiji21 N
i
N
jE
HOP
∑ ∑−∑−=
∑ ∑−=
≠= =
≠=
= =
= =
xwxxw
xwx
111
1 1jjii2
1
1jj2
1
1 1jjii2
1
1 1
N NNN
N
ii
N
j
N
jj
N
i
N
j
i j
xll
ll
pulled out the i = l case
can be added since wll = 0
NN 5 15
+∑−=
∑ ∑−∑−∑−=
≠=
≠=
≠=
≠=
≠=
xw
xwxxxw
1jj
1 1jjii2
1
1i2
1
1jj2
1
N
jj
ii
jj
ii
i
jj
x
xwx
l
ll
l l
l
l
l
l
ll
Term independent of xl
ProofHow does E change with xl changing?
=E change before Eof value
HOP
∑−−=
∑+∑−=−
=
≠=
≠=
≠=
N
jj
N
jj
N
jj
xx
xxEE
E
l
lll
l
ll
l
ll
1jj
1jj
1jj
xw)'(
xwxw''
changeafter Eof value'
NN 5 16
≠j l
we showed in previous slide that this quantity is > 0(without ‘-’ sign)
So E’-E < 0 always.
NN 5 2.09.98
Neural Networks 9
Convergence result
HOP
We have shown that the energy E decreases with each neuron firing.The overall number of states is finite ⊆ {+1, ‐1}N.
ThenThe energy cannot decrease forever.Firing cannot continue forever.
NN 5 17
Computer experiment
We illustrate the behavior of the discrete Hopfield network as a content
HOP
We illustrate the behavior of the discrete Hopfield network as a content addressable memory.n = 120 neurons (⇒n2 ‐ n = 14280 weights).The network is trained to retrieve 8 black and white patterns. Each pattern contains 120 pixels. The inputs of the net assume value +1 for black pixels and ‐1 for white pixels.Retrieval 1: the fundamental memories are presented to the network to test its ability to recover them correctly from the information stored in the its ability to recover them correctly from the information stored in the weights. In each case, the desired pattern was produced by the network after 1 iteration.
NN 5 18
NN 5 2.09.98
Neural Networks 10
Computer experiment Computer experiment
NN 5 19
These patterns are used as fundamentalThese patterns are used as fundamentalmemories to create the weight matrix.memories to create the weight matrix.
Computer experiment Retrieval 2: to test error‐correcting capability of the network, a pattern of interest is distorted by p yrandomly and independently reversing each pixel of the pattern with a probability of 0.25. The distorted pattern is used as a probe for the network.The average number of iterations needed to recall, averaged over the eight patterns, is about 31. The net behaves as expected.
NN 5 20
NN 5 2.09.98
Neural Networks 11
Computer experiment
NN 5 21
Computer experiment
NN 5 22
NN 5 2.09.98
Neural Networks 12
Spurious attractor states
bl f l l h k hProblem of incorrect retrieval. For example the network with input corrupted pattern “2” converges to “6”.
Spurious attractor states: the next slide shows 108 spurious attractors found in 43097 tests of randomly selected digits corrupted with the probability of flipping a bit at 0.25.
NN 5 23
Spurious attractor states There are some local minima of E not correlated with any of the fundamentalwith any of the fundamental memories, because of weights definition.
⎪
⎪⎨
⎧=≠
= ∑=
Nji jffw
M
jiMji
.......1, 1
,,1
μμμ
NN 5 24
Combination of digit 1, digit 4 and digit 9
⎪⎩ = ij 0
NN 5 2.09.98
Neural Networks 13
Storage CapacityStorage capacity: the quantity of information that can be stored in a network in such a way that it can be retrieved correctly.Definition of storage capacity:
networkin theneuronsofnumber patterns lfundamenta ofnumber
=C
NN 5 25
network in the weightsofnumber patterns lfundamenta ofnumber
=C
or equivalently
Storage Capacity: boundsTheorem: The maximum storage capacity of a discrete Hopfield NN is defined as C=M/N and is bounded above by the quantity
Defining Max as the storage capacity almost without error, i.e. the largest number of fundamental memories that can be
NC
ln21
=
the largest number of fundamental memories that can be stored in the network with correctly recalling of most of them we have:
NN 5 26
)ln( NN/Max 2=
NN 5 2.09.98
Neural Networks 14
Hopfield NN for Optimization
Optimization problems (like Traveling Salesman) can be encoded into Hopfield Networksp
Objective function corresponds to the energy of the network
Good solutions are stable states of the networkGood solutions are stable states of the network
NN 5 27
Travelling Salesman ProblemTravelling Salesman Problem (TSP):Given N cities with distances dij .What is the shortest tour?
NN 5 28
NN 5 2.09.98
Neural Networks 15
EncodingConstruct a Hopfield network with N 2 nodes.Semantics: iff town i is visited at step a1=nSemantics: iff town i is visited at step aConstraints:
ina
ia ∀=∑ ,1ani
ia ∀=∑ ,1
1=ian
NN 5 29
The town distancies are encoded by weights, i.e.
ijijab dw =
Hopfield Network for TSPplacecity 1 2 3 4A 0 1 0 0A 0 1 0 0B 1 0 0 0C 0 0 0 1D 0 0 1 0
NN 5 30
Configuration not allowed
Configuration allowed
NN 5 2.09.98
Neural Networks 16
Energy and Weights∑−=
bajijbiaijab nnwE
,,,21
Nodes within each row ( i=j ) connected with weights
Nodes within each column ( a=b ) connected with weights
γ−=ijabw
j ,,,
γ−=wweights
Each node is connected to nodes in columns left and right with weight
NN 5 31
γ=ijabw
ijijab dw −=
Top Related