Error-driven learning and XCAL - Brown...
Transcript of Error-driven learning and XCAL - Brown...
![Page 1: Error-driven learning and XCAL - Brown Universityski.clps.brown.edu/cogsim/cogsim.7combo.pdfCombining Error-driven + Hebbian in single XCAL rule Get benefits of both: ∆ w ij ≈](https://reader036.fdocuments.in/reader036/viewer/2022062912/611a0dab8543f519cb05e72f/html5/thumbnails/1.jpg)
Erro
r-driv
enlearn
ingan
dXCAL
•Self-o
rgan
izingHeb
bian
learningarises
from
biologically
-motiv
ated
XCALrule,ad
justin
gweig
htsas
afunctio
nofsynap
ticactiv
ityxi y
j
over
thesh
ort-term
,relativ
eto
long-term
valu
es(floatin
gthresh
old).
•TheXCALrule
caneasily
bead
apted
tosu
pporterro
r-driv
enlearn
ing,
byusin
gthe’m
edium-term
’synap
ticactiv
ityas
thecompariso
n.
•Med
ium-term
activity
:av
erageofboth
recentoutco
mean
dearlier
expectatio
n-justav
erages
synap
ticactiv
ityover
alonger
horizo
n.
Short-term
activity
reflects
outco
mepreferen
tially.
•So,th
edifferen
cebetw
eenthem
will
largely
reflect
differen
cesbetw
een
expectatio
nan
doutco
me-thedelta.
![Page 2: Error-driven learning and XCAL - Brown Universityski.clps.brown.edu/cogsim/cogsim.7combo.pdfCombining Error-driven + Hebbian in single XCAL rule Get benefits of both: ∆ w ij ≈](https://reader036.fdocuments.in/reader036/viewer/2022062912/611a0dab8543f519cb05e72f/html5/thumbnails/2.jpg)
med
ium
termactiv
ityas
expectatio
nin
XCAL
∆w
≈xs y
s−
xmym
Thisrelatio
nsh
ipistru
eformost
oftheXCALfunctio
n(th
elin
earpart).
Then
goes
back
tozero
forlow
xs y
s–nolearn
ingfornoactiv
ity(C
a++).
![Page 3: Error-driven learning and XCAL - Brown Universityski.clps.brown.edu/cogsim/cogsim.7combo.pdfCombining Error-driven + Hebbian in single XCAL rule Get benefits of both: ∆ w ij ≈](https://reader036.fdocuments.in/reader036/viewer/2022062912/611a0dab8543f519cb05e72f/html5/thumbnails/3.jpg)
Thisiswhyitiscalled
Contrastiv
eAttracto
rLearn
ing
![Page 4: Error-driven learning and XCAL - Brown Universityski.clps.brown.edu/cogsim/cogsim.7combo.pdfCombining Error-driven + Hebbian in single XCAL rule Get benefits of both: ∆ w ij ≈](https://reader036.fdocuments.in/reader036/viewer/2022062912/611a0dab8543f519cb05e72f/html5/thumbnails/4.jpg)
Other
mech
anism
sforErro
r-driv
enLearn
ing
•Neu
romodulato
rysig
nals:
Dopam
ine,A
cetylch
olin
e,etc.
•“P
hasic”
signals
elicitedbybrain
system
scomputin
g’ex
pected
reward
’
anddev
iationsfro
mthisexpectatio
n
![Page 5: Error-driven learning and XCAL - Brown Universityski.clps.brown.edu/cogsim/cogsim.7combo.pdfCombining Error-driven + Hebbian in single XCAL rule Get benefits of both: ∆ w ij ≈](https://reader036.fdocuments.in/reader036/viewer/2022062912/611a0dab8543f519cb05e72f/html5/thumbnails/5.jpg)
Other
mech
anism
sforErro
r-driv
enLearn
ing
•Neu
romodulato
rysig
nals:
Dopam
ine,A
cetylch
olin
e,etc.
•“P
hasic”
signals
elicitedbybrain
system
scomputin
g’ex
pected
reward
’
anddev
iationsfro
mthisexpectatio
n
•Resu
ltingsig
nals,w
hen
combined
with
target
inform
ation( w
hat
should
hav
ebeen
expected
)in
subseq
uen
tstate,can
enhan
ceco
ntrast
betw
eentw
osu
cceedingattracto
rstates
•Lots
ofev
iden
cethat
LTP,L
TD
under
neu
romodulato
rycontro
l
![Page 6: Error-driven learning and XCAL - Brown Universityski.clps.brown.edu/cogsim/cogsim.7combo.pdfCombining Error-driven + Hebbian in single XCAL rule Get benefits of both: ∆ w ij ≈](https://reader036.fdocuments.in/reader036/viewer/2022062912/611a0dab8543f519cb05e72f/html5/thumbnails/6.jpg)
Other
mech
anism
sforErro
r-driv
enLearn
ing
•Neu
romodulato
rysig
nals:
Dopam
ine,A
cetylch
olin
e,etc.
•“P
hasic”
signals
elicitedbybrain
system
scomputin
g’ex
pected
reward
’
anddev
iationsfro
mthisexpectatio
n
•Resu
ltingsig
nals,w
hen
combined
with
target
inform
ation( w
hat
should
hav
ebeen
expected
)in
subseq
uen
tstate,can
enhan
ceco
ntrast
betw
eentw
osu
cceedingattracto
rstates
•Lots
ofev
iden
cethat
LTP,L
TD
under
neu
romodulato
rycontro
l
•Heb
bian
learningalw
aysoccu
rslocally,in
every
synap
se(m
odel
learning,statistics)
•Brain
regionsinnerv
atedbyDA,A
Chhav
een
han
cedweig
htch
anges
durin
gerro
rs,lead
ingto
contrastiv
eattracto
rlearn
ing(ap
proxim
ated
bydelta
rule)
![Page 7: Error-driven learning and XCAL - Brown Universityski.clps.brown.edu/cogsim/cogsim.7combo.pdfCombining Error-driven + Hebbian in single XCAL rule Get benefits of both: ∆ w ij ≈](https://reader036.fdocuments.in/reader036/viewer/2022062912/611a0dab8543f519cb05e72f/html5/thumbnails/7.jpg)
Combined
Model
&Task
Learn
ing
1.Prosan
dCons:Use
Both.
2.Inhibitio
nisalso
anIm
portan
tBias.
![Page 8: Error-driven learning and XCAL - Brown Universityski.clps.brown.edu/cogsim/cogsim.7combo.pdfCombining Error-driven + Hebbian in single XCAL rule Get benefits of both: ∆ w ij ≈](https://reader036.fdocuments.in/reader036/viewer/2022062912/611a0dab8543f519cb05e72f/html5/thumbnails/8.jpg)
Functio
nal:
Prosan
dCons
... ... ... ... ...E
rror
error−driven
is based onrem
ote errors
Hebbian
is local
Pro
Con
Heb
bian
autonomous,
myopic,
(local)
reliable
greed
yErro
r-driv
entask
-driv
en,
co-dep
enden
t,(rem
ote)
cooperativ
elazy
Erro
r-driv
en=Left-w
ing,H
ebbian
=Right-w
ing(?!)
![Page 9: Error-driven learning and XCAL - Brown Universityski.clps.brown.edu/cogsim/cogsim.7combo.pdfCombining Error-driven + Hebbian in single XCAL rule Get benefits of both: ∆ w ij ≈](https://reader036.fdocuments.in/reader036/viewer/2022062912/611a0dab8543f519cb05e72f/html5/thumbnails/9.jpg)
CombiningErro
r-driv
en+Heb
bian
insin
gle
XCALrule
Get
ben
efits
ofboth:
∆wij
≈∆
hebb+
∆err
![Page 10: Error-driven learning and XCAL - Brown Universityski.clps.brown.edu/cogsim/cogsim.7combo.pdfCombining Error-driven + Hebbian in single XCAL rule Get benefits of both: ∆ w ij ≈](https://reader036.fdocuments.in/reader036/viewer/2022062912/611a0dab8543f519cb05e72f/html5/thumbnails/10.jpg)
CombiningErro
r-driv
en+Heb
bian
insin
gle
XCALrule
Get
ben
efits
ofboth:
∆wij
≈∆
hebb+
∆err
Θp=
λyl+
(1−
λ)xmym
∆wij
=λl fxcal (xs y
s ,yl )
+λmfxcal (xs y
s ,xmym)
•λ
(“thr_l_mix”in
simulato
r)isaparam
eteraffectin
gdeg
reeto
which
XCAL
thresh
old
isdeterm
ined
byylorym.
•can
differ
betw
eenbrain
system
s,orev
enbemodulated
dynam
ically
(e.g.byneu
romodulato
rs)
![Page 11: Error-driven learning and XCAL - Brown Universityski.clps.brown.edu/cogsim/cogsim.7combo.pdfCombining Error-driven + Hebbian in single XCAL rule Get benefits of both: ∆ w ij ≈](https://reader036.fdocuments.in/reader036/viewer/2022062912/611a0dab8543f519cb05e72f/html5/thumbnails/11.jpg)
Heb
bian
bias
help
sso
that
weig
htsare
constrain
edto
smaller
setof
solutio
ns(otherw
isetoointerd
epen
den
tin
err-driv
en)
![Page 12: Error-driven learning and XCAL - Brown Universityski.clps.brown.edu/cogsim/cogsim.7combo.pdfCombining Error-driven + Hebbian in single XCAL rule Get benefits of both: ∆ w ij ≈](https://reader036.fdocuments.in/reader036/viewer/2022062912/611a0dab8543f519cb05e72f/html5/thumbnails/12.jpg)
Inhibito
ryCompetitio
nas
aBias
Inhibitio
n:
•Cau
sessp
arse,distrib
uted
represen
tations
(man
yaltern
atives,
only
afew
relevan
tat
anytim
e).
•Competitio
nan
dsp
ecialization:su
rvival
offittest.
•Self-o
rgan
izinglearn
ing.
(Often
more
importan
tthan
Heb
bian
bias)
![Page 13: Error-driven learning and XCAL - Brown Universityski.clps.brown.edu/cogsim/cogsim.7combo.pdfCombining Error-driven + Hebbian in single XCAL rule Get benefits of both: ∆ w ij ≈](https://reader036.fdocuments.in/reader036/viewer/2022062912/611a0dab8543f519cb05e72f/html5/thumbnails/13.jpg)
TheWhole
Ench
ilada
![Page 14: Error-driven learning and XCAL - Brown Universityski.clps.brown.edu/cogsim/cogsim.7combo.pdfCombining Error-driven + Hebbian in single XCAL rule Get benefits of both: ∆ w ij ≈](https://reader036.fdocuments.in/reader036/viewer/2022062912/611a0dab8543f519cb05e72f/html5/thumbnails/14.jpg)
Gen
eralization
How
well
dowedeal
with
thingswe’v
enev
erseen
befo
re?
![Page 15: Error-driven learning and XCAL - Brown Universityski.clps.brown.edu/cogsim/cogsim.7combo.pdfCombining Error-driven + Hebbian in single XCAL rule Get benefits of both: ∆ w ij ≈](https://reader036.fdocuments.in/reader036/viewer/2022062912/611a0dab8543f519cb05e72f/html5/thumbnails/15.jpg)
Gen
eralization
How
well
dowedeal
with
thingswe’v
enev
erseen
befo
re?
nust
![Page 16: Error-driven learning and XCAL - Brown Universityski.clps.brown.edu/cogsim/cogsim.7combo.pdfCombining Error-driven + Hebbian in single XCAL rule Get benefits of both: ∆ w ij ≈](https://reader036.fdocuments.in/reader036/viewer/2022062912/611a0dab8543f519cb05e72f/html5/thumbnails/16.jpg)
Gen
eralization
How
well
dowedeal
with
thingswe’v
enev
erseen
befo
re?
nust
![Page 17: Error-driven learning and XCAL - Brown Universityski.clps.brown.edu/cogsim/cogsim.7combo.pdfCombining Error-driven + Hebbian in single XCAL rule Get benefits of both: ∆ w ij ≈](https://reader036.fdocuments.in/reader036/viewer/2022062912/611a0dab8543f519cb05e72f/html5/thumbnails/17.jpg)
Gen
eralization
How
well
dowedeal
with
thingswe’v
enev
erseen
befo
re?
nust
eachtim
eyouwalk
into
class,eachsocial
interactio
n,each
senten
ceyou
hear,etc.
![Page 18: Error-driven learning and XCAL - Brown Universityski.clps.brown.edu/cogsim/cogsim.7combo.pdfCombining Error-driven + Hebbian in single XCAL rule Get benefits of both: ∆ w ij ≈](https://reader036.fdocuments.in/reader036/viewer/2022062912/611a0dab8543f519cb05e72f/html5/thumbnails/18.jpg)
Gen
eralization
How
well
dowedeal
with
thingswe’v
enev
erseen
befo
re?
nust
eachtim
eyouwalk
into
class,eachsocial
interactio
n,each
senten
ceyou
hear,etc.
We’re
constan
tlyfaced
with
new
situatio
ns,an
dgen
eralizereaso
nab
lywell
tothem
.
![Page 19: Error-driven learning and XCAL - Brown Universityski.clps.brown.edu/cogsim/cogsim.7combo.pdfCombining Error-driven + Hebbian in single XCAL rule Get benefits of both: ∆ w ij ≈](https://reader036.fdocuments.in/reader036/viewer/2022062912/611a0dab8543f519cb05e72f/html5/thumbnails/19.jpg)
Gen
eralization
How
well
dowedeal
with
thingswe’v
enev
erseen
befo
re?
nust
eachtim
eyouwalk
into
class,eachsocial
interactio
n,each
senten
ceyou
hear,etc.
We’re
constan
tlyfaced
with
new
situatio
ns,an
dgen
eralizereaso
nab
lywell
tothem
.
How
dowedoit?
![Page 20: Error-driven learning and XCAL - Brown Universityski.clps.brown.edu/cogsim/cogsim.7combo.pdfCombining Error-driven + Hebbian in single XCAL rule Get benefits of both: ∆ w ij ≈](https://reader036.fdocuments.in/reader036/viewer/2022062912/611a0dab8543f519cb05e72f/html5/thumbnails/20.jpg)
![Page 21: Error-driven learning and XCAL - Brown Universityski.clps.brown.edu/cogsim/cogsim.7combo.pdfCombining Error-driven + Hebbian in single XCAL rule Get benefits of both: ∆ w ij ≈](https://reader036.fdocuments.in/reader036/viewer/2022062912/611a0dab8543f519cb05e72f/html5/thumbnails/21.jpg)
![Page 22: Error-driven learning and XCAL - Brown Universityski.clps.brown.edu/cogsim/cogsim.7combo.pdfCombining Error-driven + Hebbian in single XCAL rule Get benefits of both: ∆ w ij ≈](https://reader036.fdocuments.in/reader036/viewer/2022062912/611a0dab8543f519cb05e72f/html5/thumbnails/22.jpg)
![Page 23: Error-driven learning and XCAL - Brown Universityski.clps.brown.edu/cogsim/cogsim.7combo.pdfCombining Error-driven + Hebbian in single XCAL rule Get benefits of both: ∆ w ij ≈](https://reader036.fdocuments.in/reader036/viewer/2022062912/611a0dab8543f519cb05e72f/html5/thumbnails/23.jpg)
![Page 24: Error-driven learning and XCAL - Brown Universityski.clps.brown.edu/cogsim/cogsim.7combo.pdfCombining Error-driven + Hebbian in single XCAL rule Get benefits of both: ∆ w ij ≈](https://reader036.fdocuments.in/reader036/viewer/2022062912/611a0dab8543f519cb05e72f/html5/thumbnails/24.jpg)
![Page 25: Error-driven learning and XCAL - Brown Universityski.clps.brown.edu/cogsim/cogsim.7combo.pdfCombining Error-driven + Hebbian in single XCAL rule Get benefits of both: ∆ w ij ≈](https://reader036.fdocuments.in/reader036/viewer/2022062912/611a0dab8543f519cb05e72f/html5/thumbnails/25.jpg)
![Page 26: Error-driven learning and XCAL - Brown Universityski.clps.brown.edu/cogsim/cogsim.7combo.pdfCombining Error-driven + Hebbian in single XCAL rule Get benefits of both: ∆ w ij ≈](https://reader036.fdocuments.in/reader036/viewer/2022062912/611a0dab8543f519cb05e72f/html5/thumbnails/26.jpg)
![Page 27: Error-driven learning and XCAL - Brown Universityski.clps.brown.edu/cogsim/cogsim.7combo.pdfCombining Error-driven + Hebbian in single XCAL rule Get benefits of both: ∆ w ij ≈](https://reader036.fdocuments.in/reader036/viewer/2022062912/611a0dab8543f519cb05e72f/html5/thumbnails/27.jpg)
Heb
b:
•Sometim
esfails
tolearn
thetrain
ingset
•Rep
resents
mean
ingful“th
ings”
intheworld
(correlatio
ns)
•Showsgoodgen
eralization
Erro
r(G
eneR
ec/X
CA
Lw
ithxm
ym):
•Alw
ayslearn
sthetrain
ingset
•Rep
resentatio
nsare
“mush
y”
•Can
show
poorgen
eralization
Erro
r+
Heb
b:
•Learn
sthetrain
ingset
(more
quick
lythan
erroralo
ne)
•Rep
resents
mean
ingfulfeatu
res
•Showsgoodgen
eralization!
![Page 28: Error-driven learning and XCAL - Brown Universityski.clps.brown.edu/cogsim/cogsim.7combo.pdfCombining Error-driven + Hebbian in single XCAL rule Get benefits of both: ∆ w ij ≈](https://reader036.fdocuments.in/reader036/viewer/2022062912/611a0dab8543f519cb05e72f/html5/thumbnails/28.jpg)
Deep
Netw
orks
Need
man
yhidden
layers
toach
ieveman
ystag
esoftran
sform
ations
(dram
aticallyre-rep
resentin
gtheproblem
).(cf.
recentsu
rgein
interest
in“d
eeplearn
ing”
inmach
inelearn
ingforsp
eechreco
gnitio
n,G
oogle,etc)
Butthen
theerro
rsig
nals
arevery
remote
&weak
.
Need
toad
dconstrain
tsan
dself-o
rgan
izinglearn
ing:
![Page 29: Error-driven learning and XCAL - Brown Universityski.clps.brown.edu/cogsim/cogsim.7combo.pdfCombining Error-driven + Hebbian in single XCAL rule Get benefits of both: ∆ w ij ≈](https://reader036.fdocuments.in/reader036/viewer/2022062912/611a0dab8543f519cb05e72f/html5/thumbnails/29.jpg)
Deep
Netw
orks
Need
man
yhidden
layers
toach
ieveman
ystag
esoftran
sform
ations
(dram
aticallyre-rep
resentin
gtheproblem
).(cf.
recentsu
rgein
interest
in“d
eeplearn
ing”
inmach
inelearn
ingforsp
eechreco
gnitio
n,G
oogle,etc)
Butthen
theerro
rsig
nals
arevery
remote
&weak
.
Need
toad
dconstrain
tsan
dself-o
rgan
izinglearn
ing:
•Heb
bgives
eachlay
erlocal
guidan
ceonrep
resentatio
ns
•Inhib
competitio
nrestricts
flexibility
(only
certainstates
arevalid
)
•Combined
heb
b+err
→few
erdeg
reesoffreed
om
toad
apt
![Page 30: Error-driven learning and XCAL - Brown Universityski.clps.brown.edu/cogsim/cogsim.7combo.pdfCombining Error-driven + Hebbian in single XCAL rule Get benefits of both: ∆ w ij ≈](https://reader036.fdocuments.in/reader036/viewer/2022062912/611a0dab8543f519cb05e72f/html5/thumbnails/30.jpg)
Exam
ple:
Fam
ilyTrees
(Hinton,1986)
Christo=
Penny
Andy=
Christi
Marge=
Art
Vicky=
James
Jenn=C
huck
Colin
Charlot
Rob=
Maria
Pierro=
Francy
Gina=
Em
ilioLucia=
Marco
Angela=
Tom
aso
Alf
Sophia
Agent
Agent_C
ode
Relation
Relation_C
odePatient
Patient_C
ode
Hidden
![Page 31: Error-driven learning and XCAL - Brown Universityski.clps.brown.edu/cogsim/cogsim.7combo.pdfCombining Error-driven + Hebbian in single XCAL rule Get benefits of both: ∆ w ij ≈](https://reader036.fdocuments.in/reader036/viewer/2022062912/611a0dab8543f519cb05e72f/html5/thumbnails/31.jpg)
Exam
ple:
Fam
ilyTrees
(Hinton,1986)
Christo=
Penny
Andy=
Christi
Marge=
Art
Vicky=
James
Jenn=C
huck
Colin
Charlot
Rob=
Maria
Pierro=
Francy
Gina=
Em
ilioLucia=
Marco
Angela=
Tom
aso
Alf
Sophia
Agent
Agent_C
ode
Relation
Relation_C
odePatient
Patient_C
ode
Hidden
24peo
ple,
12relatio
nsh
ips(brother,
mother,g
randdau
ghter,etc)
![Page 32: Error-driven learning and XCAL - Brown Universityski.clps.brown.edu/cogsim/cogsim.7combo.pdfCombining Error-driven + Hebbian in single XCAL rule Get benefits of both: ∆ w ij ≈](https://reader036.fdocuments.in/reader036/viewer/2022062912/611a0dab8543f519cb05e72f/html5/thumbnails/32.jpg)
Exam
ple:
Fam
ilyTrees
(Hinton,1986)
Christo=
Penny
Andy=
Christi
Marge=
Art
Vicky=
James
Jenn=C
huck
Colin
Charlot
Rob=
Maria
Pierro=
Francy
Gina=
Em
ilioLucia=
Marco
Angela=
Tom
aso
Alf
Sophia
Agent
Agent_C
ode
Relation
Relation_C
odePatient
Patient_C
ode
Hidden
24peo
ple,
12relatio
nsh
ips(brother,
mother,g
randdau
ghter,etc)
WhoisAlf’s
gran
dmother?
WhoisLucia’s
dau
ghter?
![Page 33: Error-driven learning and XCAL - Brown Universityski.clps.brown.edu/cogsim/cogsim.7combo.pdfCombining Error-driven + Hebbian in single XCAL rule Get benefits of both: ∆ w ij ≈](https://reader036.fdocuments.in/reader036/viewer/2022062912/611a0dab8543f519cb05e72f/html5/thumbnails/33.jpg)
[family
trees.proj]