Adaptive Resonance Theory
description
Transcript of Adaptive Resonance Theory
38 1
Adaptive Resonance Theory
Unsupervised Learning
22
Learning Objectives
1.1. IntroductionIntroduction
2.2. ART architectureART architecture
3.3. ART implementationART implementation
4.4. ART2ART2
5.5. ConclusionConclusion
33
Introduction
1. Developed by Carpenter and Grossberg
2. ART1-accept binary data
3. ART2-accept continuous value data
4. ART3-improved ART
5. Fuzzy ART- accept fuzzy data
44
ART Architecture (1/20)1.1. Brief description:Brief description:
a.a. Accept an input vector and Accept an input vector and classify it into one of categories.classify it into one of categories.
b.b. The category it belongs to The category it belongs to depend on which one it most depend on which one it most resembles.resembles.
c.c. If the input vector does not If the input vector does not match any of the stored match any of the stored pattern, a new category is created.pattern, a new category is created.
55
ART Architecture (2/20)d.d. If the input vector matched (within If the input vector matched (within
vigilance level) any of stored vigilance level) any of stored pattern, the pattern is adjusted pattern, the pattern is adjusted
(trained) to make it more (trained) to make it more like the like the input vector.input vector.
2.2. Simplified ART ArchitectureSimplified ART Architecturea.a. two layers: two layers: comparison(F1), comparison(F1),
recognition(F2)recognition(F2)b.b. 3 control function: 3 control function: Gain 1, Gain 2Gain 1, Gain 2, ,
and and ResetReset. .
66
ART Architecture (3/20)
77
ART Architecture (4/20)
3.3. Functions of each models:Functions of each models:
a.a. Comparison layer:Comparison layer:
-accept binary input -accept binary input XX
-initially pass -initially pass XX to to CC . So . So CC = = XX
-binary vector -binary vector RR is produced from is produced from recognition layer to modify recognition layer to modify
CC
88
ART Architecture (5/20)
99
ART Architecture (6/20)-each neuron in the comparison layer -each neuron in the comparison layer
has 3 has 3 inputs:inputs:XX: input vector: input vector
PPjj: : weighted sum of recognition layer weighted sum of recognition layer outputoutput
Gain 1Gain 1: : same signal to all neuronssame signal to all neurons-use “two-third” rule-use “two-third” rule
=> at least => at least two of a neuron’s three two of a neuron’s three inputs must be one, otherwise, inputs must be one, otherwise,
the the output is zero.output is zero.-initially, -initially, Gain 1Gain 1 is set to one and is set to one and RR are are
set set to 0.to 0.
1010
ART Architecture (7/20)
b.b. Recognition layer:Recognition layer:
-compute -compute dot productdot product of of BB and and CC
--tthe neuron with largest output winshe neuron with largest output wins
-the winning neuron is set to one -the winning neuron is set to one others are set to zero.others are set to zero.
c.c. Gain 2Gain 2
-OR -OR of components ofof components of XX
1111
ART Architecture (8/20)
1212
ART Architecture (9/20)
d.d. Gain 1Gain 1
OR of OR of XX
components G2 OR of components G2 OR of RR G1G1
--------------------------------------------------------------------------------------------------------
00 0 0 00 0 0
11 1 1 00 1 1
11 1 1 1 1 0 0
00 0 0 1 1 0 0
1313
ART Architecture (10/20)
e.e. ResetReset
-measures the similarity between -measures the similarity between XX and and CC
1414
ART Architecture (11/20)
4.4. OperationsOperationsa. recognition phasea. recognition phase
i) when no input vector i) when no input vector XX , then , then G2 = 0G2 = 0
ii) G2 = 0 disables all neurons in ii) G2 = 0 disables all neurons in recognition layerrecognition layer=> => RR=0=0
iii) This makes sure all neurons in iii) This makes sure all neurons in this layer start out at same state.this layer start out at same state.
1515
ART Architecture (13/20) iv) Then, vector iv) Then, vector XX is applied, this is applied, this XX
must have at least one must have at least one component is “1”. (component is “1”. (∵∵OR of OR of
XX is is “1” => G2=1; “1” => G2=1; ∵∵OR of OR of RR is is “0” “0” => G1=1). => G1=1).
v) So G2, G1 =1, and v) So G2, G1 =1, and CC = = XX..
vi) dot product vi) dot product BBjj.C.C. find neuron . find neuron jj that that has the largest result, fire has the largest result, fire “one”, “one”, others fire “zero”.others fire “zero”.
vii) This vii) This jjthth neuron has output neuron has output rrjj of of RR equal to one, and all others equal to one, and all others
equal equal to zero.to zero.
1616
ART Architecture (14/20)
1717
ART Architecture (15/20)
b. Comparison phaseb. Comparison phase
i) i) rrjj going through weight going through weight ttjiji (binary) (binary)
to each neuron in comparison to each neuron in comparison layer providing an input signal layer providing an input signal PPii..
ii) Now ii) Now RR is not zero anymore, G1 = is not zero anymore, G1 = 0 and by 2/3 rule, the 0 and by 2/3 rule, the
components of components of CC will be one will be one only when only when XX and and P P are ones. are ones.
1818
ART Architecture (16/20)
iii) If there is a substantial iii) If there is a substantial mismatch between mismatch between XX
and and PP, , CC contains many contains many zeros, zeros, while while XX contains ones. This contains ones. This will will trigger Reset function to trigger Reset function to inhabit the output of firing inhabit the output of firing
neuron in the recognition layer to neuron in the recognition layer to zero and disable it for the zero and disable it for the
duration of the current duration of the current classification. classification.
1919
ART Architecture (17/20)
2020
ART Architecture (18/20)
c. Search phasec. Search phase i) if no Reset signal generated, the i) if no Reset signal generated, the
match is in tolerate level, and match is in tolerate level, and the classification is finished. the classification is finished. Otherwise, search other nodes.Otherwise, search other nodes.
ii) to search other node, let ii) to search other node, let rrjj = 0 = 0 => => RR = 0 => G1 =1 => = 0 => G1 =1 => CC = = XX
again.again.
iii) different neuron wins => different iii) different neuron wins => different pattern pattern PP is fed back to is fed back to
comparison layer.comparison layer.
2121
ART Architecture (19/20)
1)1) AA stored pattern is found then the stored pattern is found then the network enters training cycle.network enters training cycle.
2)2) All stored pattern are tried and no All stored pattern are tried and no match , assign new neuron set match , assign new neuron set BBjj
and and TTjj..
2222
ART Architecture (20/20)
d.d. Performance issues:Performance issues:
i)i) sequential searchsequential search
ii)ii) stabilizationstabilization
2323
ART Implementation(1/3)1.1. InitializationInitialization
TTjj, , BBjj, Vigilance level , Vigilance level BBjj::
0< 0< bbijij < < L/(L-1+m)L/(L-1+m) for all for all ii, , jj
mm: # of components in the input vector: # of components in the input vectorLL: as constant > 1 (: as constant > 1 (LL=2, typically)=2, typically)
all all bbij ij are the same valueare the same value
TTjj: : ttijij=1 for all =1 for all ii, , jj
ρρ: 0<: 0<ρρ<1 coarse distinction at first, fine <1 coarse distinction at first, fine distinction at last.distinction at last.
2424
ART Implementation(2/3)2.2. RecognitionRecognition
NETNETj j = (= (BBjjCC))
OUTOUTjj=1=1 NETNETjj>>TT
00 otherwiseotherwise
3.3. ComparisonComparison
Xinof
CinofS
1 #
1 #
Xinof
CinofS
1 #
1 #
4.4. SearchSearchSearching process is going until a pattern is Searching process is going until a pattern is matched or no pattern is matched.matched or no pattern is matched.
2525
ART Implementation(3/3)5.5. TrainingTraining
If the input If the input XX is matched, is matched,
2
neuron firing :
ofcomponent the:
)1/()(
L
j
Cic
cLcLb
thi
kkiij
2
neuron firing :
ofcomponent the:
)1/()(
L
j
Cic
cLcLb
thi
kkiij
ict iji allfor ict iji allfor
for newly stored for newly stored TTjj : :
2626
ART Algorithm(1/4)1.1. InitializationInitialization
L > 1, 0<ρ<1L > 1, 0<ρ<1
0< 0< bbijij < < L/(L-1+m)L/(L-1+m) for all for all ii, , jj
ttijij=1=1 for all for all ii, , jj2.2. While stopping condition is false, do 3-While stopping condition is false, do 3-
12:12:3.3. For each training pattern(For each training pattern(XX),),
do 4-11:do 4-11:4.4. Set Set R R = 0,= 0, C C = = XX
5.5. For each node in For each node in recognition layer that is recognition layer that is not inhibited (rnot inhibited (r-1):-1):
2727
ART Algorithm(2/4)6.6. If If rrjj -1, r -1, rjj = = ((BBjjCC))
7.7. While Reset is trueWhile Reset is true8.8. Find winning Find winning
neuron neuron j, j, If If rrjj - -1 1 then this then this pattern pattern cannot cannot be be clusteredclustered..
9.9. Let Let PP ==TTJJ,,
compute compute CC
2828
ART Algorithm(3/4)
10.10.# 1
, 1# 1
# 1 , proceed to 11.
# 1
J
of in Cif S r
of in X
of in Cif S
of in X
# 1 , 1
# 1
# 1 , proceed to 11.
# 1
J
of in Cif S r
of in X
of in Cif S
of in X
11.11. Update Update BBJJ, , TTJJ==CC
12.12. Test for stopping Test for stopping conditionsconditions
a.a. no weight changeno weight change
b.b. maximum number of maximum number of epochs epochs
reached.reached.
2929
ART Algorithm(4/4)
**** Stopping conditions:Stopping conditions:
a.a. no weight changeno weight change
b.b. maximum number of epochs maximum number of epochs reached.reached.
3030
ART2 (1/9)1.1. Several sublayers replace F1 layerSeveral sublayers replace F1 layer2.2. Update functions:Update functions:
)()( , ,
, ,
iiijiiiiii
ii
ii
ii
qbfxfvdtupausw
Pe
pq
We
wx
Ve
vu
)()( , ,
, ,
iiijiiiiii
ii
ii
ii
qbfxfvdtupausw
Pe
pq
We
wx
Ve
vu
3.3. Parameters:Parameters:mm:: dimension of input vectordimension of input vectornn:: number of cluster units number of cluster units
3131
ART2 (2/9)
3232
ART2 (3/9)3.3. Parameters(continue):Parameters(continue):
a, ba, b:: fixed weight in F1, cannot be zero.fixed weight in F1, cannot be zero.cc:: fixed weight used in testing for fixed weight used in testing for
Reset, 0<c<1.Reset, 0<c<1.dd:: output activation of F2,output activation of F2,e:e: a small parameter to prevent a small parameter to prevent
division by zero when norm of a division by zero when norm of a vector is zero.vector is zero.
:: noise suppression parameternoise suppression parameter:: learning ratelearning rate:: vigilance levelvigilance level
11
d
cd1
1
d
cd
3333
ART2 (4/9)3.3. Algorithm:Algorithm:
aa.. Initialization: Initialization:
a, b, c, d, e, a, b, c, d, e, , , , , b.b. perform # of epochs of trainingperform # of epochs of training
c.c. For each input pattern, For each input pattern, SS,,
do d. to m. do d. to m.
d.d. update F1update F1
)( ,0 ,0
, , ,0
iiii
iii
ii
xfvpq
swSe
sxu
)( ,0 ,0
, , ,0
iiii
iii
ii
xfvpq
swSe
sxu
3434
ART2 (5/9)d.d. update F1 againupdate F1 again
)()( ,
, ,
, ,
iiiii
iiii
i
ii
ii
qbfxfvup
auswPe
pq
We
wx
Ve
vu
)()( ,
, ,
, ,
iiiii
iiii
i
ii
ii
qbfxfvup
auswPe
pq
We
wx
Ve
vu
e.e. compute signal to F2compute signal to F2
j ij ii
y b pj ij ii
y b p
3535
ART2 (6/9)f.f. While Reset is true, do g. to h.While Reset is true, do g. to h.g.g. Find winning neuron Find winning neuron JJ in in F2 F2 h.h. Check for Reset:Check for Reset:
PcUe
cpur
dtupVe
vu
iii
Jiiii
i
, ,
PcUe
cpur
dtupVe
vu
iii
Jiiii
i
, ,
3636
ART2 (7/9)i) i) If ||If ||RR|| < || < - e - e, ,
then then yyJ J = -1 (inhibit = -1 (inhibit
JJ) ) Reset is true; Reset is true; repeat repeat f. f.
ii) If ||ii) If ||RR|| >= || >= - e - e, ,
thenthen
).()(,
, ,
iiii
i
iiiii
qbfxfvPe
pq
We
wxausw
).()(,
, ,
iiii
i
iiiii
qbfxfvPe
pq
We
wxausw
3737
ART2 (8/9)i.i. Do j. to Do j. to ll. for # of learning . for # of learning
iterations.iterations.
j.j. update weight for update weight for neuron neuron JJ
iJiiJ
JiiJi
bdddub
tdddut
)1(1
)1(1
iJiiJ
JiiJi
bdddub
tdddut
)1(1
)1(1
k.k. Update F1Update F1
)()( ,
,,
, ,
iiijiii
iiii
i
ii
ii
qbfxfvdtup
auswPe
pq
We
wx
Ve
vu
)()( ,
,,
, ,
iiijiii
iiii
i
ii
ii
qbfxfvdtup
auswPe
pq
We
wx
Ve
vu
3838
ART2 (9/9)
l.l. Test stopping condition Test stopping condition for weight updates.for weight updates.
m.m. Test stopping condition for # Test stopping condition for # of epochs.of epochs.