Lecture 13: Associative Memory

Lecture 13: Associative Memory

References:

D Amit, N Brunel, Cerebral Cortex 7, 237-252 (1997)

N Brunel, Network 11, 261-280 (2000)

N Brunel, Cerebral Cortex 13, 1151-1161 (2003)

J Hertz, in Models of Neural Networks IV (L van Hemmen, J Cowan and E Domany, eds) Springer Verlag, 2002; sect 1.4

What is associative memory?


• “Patterns”: firing activity of specific sets of neurons (Hebb: “assemblies”)



• “Store” patterns in synaptic strengths




• Recall: Given input (initial activity pattern) not equal to any stored pattern, network dynamics should take it to “nearest” (most similar) stored pattern




• Recall: Given input (initial activity pattern) not equal to any stored pattern, network dynamics should take it to “nearest” (most similar) stored pattern

(categorization, error correction, …)

Implementation in balanced excitatory-inhibitory network

Model (Amit & Brunel): p non-overlapping excitatory subpopulations


Model (Amit & Brunel): p non-overlapping excitatory subpopulationseach of size n = fN (fp < 1)



stronger connections within subpopulations (“assemblies”)



stronger connections within subpopulations (“assemblies”)weakened connections between subpopulations



stronger connections within subpopulations (“assemblies”)weakened connections between subpopulations

Looking for selective states: higher rates in a single assembly

Model

Like Amit-Brunel model (Lecture 9) except for exc-exc synapses:

Model


From within the same assembly:

Model


From within the same assembly: 1111 JgJ

Model


From within the same assembly: 1111 JgJ (strengthened, “Hebb” rule)

Model



From outside the assembly:

1111 JgJ

1111 JgJ

(strengthened, “Hebb” rule)

(weakened, “anti-Hebb”)

Model




Otherwise: no change

1111 JgJ

1111 JgJ



Model





1111 JgJ

1111 JgJ



To conserve average strength: 1)1( gffg

Model





1111 JgJ

1111 JgJ



To conserve average strength:

=>

1)1( gffg

ffg

g

11

Mean field theoryRates: active assembly

inactive assembliesrest of excitatory neuronsinhibitory neuronsext input neurons

actr

r1r

2r0r

Mean field theory

Input current to neurons in the active assembly:

212111010 ])1()1([ rJrgpfrgpfrfgJrJI actact

Rates: active assemblyinactive assembliesrest of excitatory neuronsinhibitory neuronsext input neurons

actr

r1r

2r0r

Mean field theory




actr

r1r

2r0r

to rest of assemblies:

212111010 ])1(])2([[ rJrgpfrgpgfrfgJrJI act

Mean field theory




actr

r1r

2r0r



to other excitatory neurons:

2121110101 ])1()1([ rJrpfrpffrJrJI act

Mean field theory




actr

r1r

2r0r



to other excitatory neurons:


to inhibitory neurons:


Mean field theory (2)Noise variances (white noise approximation):


2

22

121

222

1

211

0

02

102 ])1()1([K

rJrgpfrgpfrfg

KJ

KrJ

actact


2

22

121

222

1

211

0

02

102 ])1()1([K

rJrgpfrgpfrfg

KJ

KrJ

actact

2

22

121

2222

1

211

0

02

102 ])1(])2([[K

rJrgpfrgpgfrfg

KJ

KrJ

act


2

22

121

222

1

211

0

02

102 ])1()1([K

rJrgpfrgpfrfg

KJ

KrJ

actact

2

22

121

2222

1

211

0

02

102 ])1(])2([[K

rJrgpfrgpgfrfg

KJ

KrJ

act

2

22

121

1

211

0

02

1021 ])1()1([

KrJ

rpfrpffrKJ

KrJ

act


2

22

121

222

1

211

0

02

102 ])1()1([K

rJrgpfrgpfrfg

KJ

KrJ

actact

2

22

121

2222

1

211

0

02

102 ])1(])2([[K

rJrgpfrgpgfrfg

KJ

KrJ

act

2

22

121

1

211

0

02

1021 ])1()1([

KrJ

rpfrpffrKJ

KrJ

act

2

2222

11

221

0

02202

2 ])1()1([K

rJrpfrpffr

KJ

KrJ

act


2

22

121

222

1

211

0

02

102 ])1()1([K

rJrgpfrgpfrfg

KJ

KrJ

actact

2

22

121

2222

1

211

0

02

102 ])1(])2([[K

rJrgpfrgpgfrfg

KJ

KrJ

act

2

22

121

1

211

0

02

1021 ])1()1([

KrJ

rpfrpffrKJ

KrJ

act

2

2222

11

221

0

02202

2 ])1()1([K

rJrpfrpffr

KJ

KrJ

act

Rate of an I&F neuron driven by white noise:


2

22

121

222

1

211

0

02

102 ])1()1([K

rJrgpfrgpfrfg

KJ

KrJ

actact

2

22

121

2222

1

211

0

02

102 ])1(])2([[K

rJrgpfrgpgfrfg

KJ

KrJ

act

2

22

121

1

211

0

02

1021 ])1()1([

KrJ

rpfrpffrKJ

KrJ

act

2

2222

11

221

0

02202

2 ])1()1([K

rJrpfrpffr

KJ

KrJ

act

),()erf1)(exp(

12/)(

/)(

aaI

IVr

a IFxxdx

raa

aareset

Rate of an I&F neuron driven by white noise:

Spontaneous activity:All assemblies inactive:

Spontaneous activity:


All assemblies inactive:


212111010 })1(])1([{ rJrgpfrgpgfJrJI



becomes



2121110101 ])1([ rJrpfpfrJrJI



becomes

Similarly,



2121110101 ])1([ rJrpfpfrJrJI 2221210202 ])1([ rJrpfpfrJrJI



becomes

Similarly,




2

22

121

222

1

211

0

02

102 })1(])1([{K

rJrgpfrgpgf

KJ

KrJ




becomes

Similarly,

and




2

22

121

222

1

211

0

02

102 })1(])1([{K

rJrgpfrgpgf

KJ

KrJ


2

22

121

1

211

0

02

1021 ])1([

KrJ

rpfpfrKJ

K

rJ 2

2222

11

221

0

02202

2 ])1([K

rJrpfpfr

KJ

KrJ



becomes

Similarly,

and




2

22

121

222

1

211

0

02

102 })1(])1([{K

rJrgpfrgpgf

KJ

KrJ


2

22

121

1

211

0

02

1021 ])1([

KrJ

rpfpfrKJ

K

rJ 2

2222

11

221

0

02202

2 ])1([K

rJrpfpfr

KJ

KrJ



becomes

Similarly,

and

Solve for 2121 ,,;,, rrr

Simplified model (Brunel 2000)


• pf << 1


• pf << 1

• g+ ~1/f >> 1


• pf << 1

• g+ ~1/f >> 1

• variances + =act, 1 as in spontaneous-activity state


• pf << 1

• g+ ~1/f >> 1


Define L = fJ11g+


• pf << 1

• g+ ~1/f >> 1


Define L = fJ11g+

Then

(1) spontaneous activity state has r+= r1,


• pf << 1

• g+ ~1/f >> 1


Define L = fJ11g+

Then


(2) In recall state with ract > r+, r1 and r2 are same as in spontaneous activity state


• pf << 1

• g+ ~1/f >> 1


Define L = fJ11g+

Then


(2) In recall state with ract > r+, r1 and r2 are same as in spontaneous activity state

(3) ract is determined by]),([ ,11,1 spontactspontact rrLIFr

Graphical solution

)( mactact I (This L = (our L) x m)

(r -> )

Graphical solution

)( mactact I (This L = (our L) x m)

(r -> )

1-assembly memory/recall state stable for big enough L (or g+ )~ describes “working memory” in prefrontal cortex

Capacity problem

In this model, memory assemblies were non-overlapping.This is unrealistic.

Capacity problem


Alternative model: neurons in each assembly independently chosenA single neuron can be in many assemblies

Capacity problem



How many patterns can be stored using N neurons before interference between patterns destroys the recall ability?

Capacity problem



How many patterns can be stored using N neurons before interference between patterns destroys the recall ability?

Here: solve this for a simplified model (binary neurons, can beeither excitatory on inhibitory, “Hebbian” synapse formula)

Model N Binary neurons: )(0),(1 firing"not"firing""iS


Assemblies/patterns : p sets of n = fN neurons with Si = 1i



ff ii 1)0(P;)1(P



ff ii 1)0(P;)1(P

)1(,1

OnfNOfNnN

Of



ff ii 1)0(P;)1(P

(Synchronous) dynamics:

jjiji tSJtS )()1(

)1(,1

OnfNOfNnN

Of



ff ii 1)0(P;)1(P


jjiji tSJtS )()1(

Synapses:Npf

nJ j

p

iij

1

1

)1(,1

OnfNOfNnN

Of



ff ii 1)0(P;)1(P


jjiji tSJtS )()1(

Synapses:Npf

nJ j

p

iij

1

1 (global inhibitory term makesaverage Jij = 0)

)1(,1

OnfNOfNnN

Of

Order parameters

Order parameters

(normalized) overlap with pattern 1:i

ii S

nm 11

Order parameters

(normalized) overlap with pattern 1:i

ii S

nm 11

Total average activity: i

iSN

fQ1

Net input to neuron i:

j

jiji SJh


j

jiji SJh

j

jjj

ji SNpf

Sn

,

1


j

jiji SJh

j

jjj

ji SNpf

Sn

,

1

j

jjj

jijj

ji SNpf

Sn

Sn 1,

11 11


j

jiji SJh

j

jjj

ji SNpf

Sn

,

1

j

jjj

jijj

ji SNpf

Sn

Sn 1,

11 11

01 21 fQpffQfNp

Nfhhm iii

Fluctuations QpfS

nS

Npf

Sn

h jj

jij

jjj

jii2

1,1,

11

Fluctuations QpfS

nS

Npf

Sn

h jj

jij

jjj

jii2

1,1,

11

kjjk

kjikjjk

kijii SSn

SSn

h

222 11

Fluctuations QpfS

nS

Npf

Sn

h jj

jij

jjj

jii2

1,1,

11

kjjk

kjikjjk

kijii SSn

SSn

h

222 11

kjkj

kjijj

ji SSn

Sn

,

22

11

Fluctuations QpfS

nS

Npf

Sn

h jj

jij

jjj

jii2

1,1,

11

kjjk

kjikjjk

kijii SSn

SSn

h

222 11

kjkj

kjijj

ji SSn

Sn

,

22

11

2322

22 )(

)(1

)(1

fQfpNNf

fQfNpNf

Fluctuations QpfS

nS

Npf

Sn

h jj

jij

jjj

jii2

1,1,

11

kjjk

kjikjjk

kijii SSn

SSn

h

222 11

kjkj

kjijj

ji SSn

Sn

,

22

11

2322

22 )(

)(1

)(1

fQfpNNf

fQfNpNf

)1( 2NQffQN

p

Fluctuations QpfS

nS

Npf

Sn

h jj

jij

jjj

jii2

1,1,

11

kjjk

kjikjjk

kijii SSn

SSn

h

222 11

with = p/N

kjkj

kjijj

ji SSn

Sn

,

22

11

2322

22 )(

)(1

)(1

fQfpNNf

fQfNpNf

)1( 2NQffQN

p )1( nfQfQ

Fluctuations QpfS

nS

Npf

Sn

h jj

jij

jjj

jii2

1,1,

11

kjjk

kjikjjk

kijii SSn

SSn

h

222 11

with = p/N (recall nf = O(1))

kjkj

kjijj

ji SSn

Sn

,

22

11

2322

22 )(

)(1

)(1

fQfpNNf

fQfNpNf

)1( 2NQffQN

p )1( nfQfQ

Mean field equations

For neurons in pattern 1, h = m + Gaussian noise



=>

2

2

21

2exp

2 h

h

h

dhm

m



=>

2

2

21

2exp

2 h

h

h

dhm

m

2

Hh

m

with )exp(2

)(H 221 x

dxx

x



=>

2

2

21

2exp

2 h

h

h

dhm

m

2

Hh

m

with )exp(2

)(H 221 x

dxx

x

For other neurons, h = Gaussian noise



=>

2

2

21

2exp

2 h

h

h

dhm

m

2

Hh

m

with )exp(2

)(H 221 x

dxx

x


=>

2H)1(

hffmfQ



=>

2

2

21

2exp

2 h

h

h

dhm

m

2

Hh

m

with )exp(2

)(H 221 x

dxx

x


=>

2H)1(

hffmfQ

Solve for m and Q

Graphical interpretation

Capacity estimateWeight in tail (h > m) of big gaussian centered at 0 must be < weight in small one centered at m


Can take

1,1 mQ


Can take

=>

1,1 mQ

1)1(

exp)1(2

2

21

1

nffh

nff

dh


Can take

=>

1,1 mQ

1)1(

exp)1(2

2

21

1

nffh

nff

dh

i.e., 1)1(

1H

nff


Can take

=>

1,1 mQ

1)1(

exp)1(2

2

21

1

nffh

nff

dh

i.e., 1)1(

1H

nff

Use asymptotic form of H: )exp(2

1)(H 2

21 x

xx


Can take

=>

1,1 mQ

1)1(

exp)1(2

2

21

1

nffh

nff

dh

i.e., 1)1(

1H

nff

Use asymptotic form of H: )exp(2

1)(H 2

21 x

xx

=> capacity estimate)/1log()1(

12fnff

Lecture 13: Associative Memory

Documents

Transcript of Lecture 13: Associative Memory