An Introduction to Complex-Valued Recurrent Correlation ...valle/PDFfiles/Talk_IJCNN2014.pdf ·...
Transcript of An Introduction to Complex-Valued Recurrent Correlation ...valle/PDFfiles/Talk_IJCNN2014.pdf ·...
An Introduction to Complex-Valued RecurrentCorrelation Neural Networks
Marcos Eduardo Valle
Department of Applied MathematicsInstitute of Mathematics, Statistics, and Scientific Computing
University of Campinas - Brazil
July 11, 2014
Marcos Eduardo Valle (Brazil) CV-RCNNs July 11, 2014 1 / 23
Introduction
Complex-Valued Hopfield NetworksLow storage capacity!
Hopfield Network
Low storage capacity!
Complex-valuedRecurrent CorrelationNeural NetworkHigh storage capacity
Recurrent Correlation NetworksHigh storage capacity!
Marcos Eduardo Valle (Brazil) CV-RCNNs July 11, 2014 2 / 23
Introduction
Complex-Valued Hopfield NetworksLow storage capacity!
Hopfield NetworkLow storage capacity!
Complex-valuedRecurrent CorrelationNeural NetworkHigh storage capacity
Recurrent Correlation NetworksHigh storage capacity!
Marcos Eduardo Valle (Brazil) CV-RCNNs July 11, 2014 2 / 23
Introduction
Complex-Valued Hopfield NetworksLow storage capacity!
Hopfield NetworkLow storage capacity!
(Chiueh&Goodman,1991)
��
Complex-valuedRecurrent CorrelationNeural NetworkHigh storage capacity
Recurrent Correlation NetworksHigh storage capacity!
Marcos Eduardo Valle (Brazil) CV-RCNNs July 11, 2014 2 / 23
Introduction
Complex-Valued Hopfield NetworksLow storage capacity!
Hopfield NetworkLow storage capacity!
(Chiueh&Goodman,1991)
��
(Jankowski et al.,1996)
OO
Complex-valuedRecurrent CorrelationNeural NetworkHigh storage capacity
Recurrent Correlation NetworksHigh storage capacity!
Marcos Eduardo Valle (Brazil) CV-RCNNs July 11, 2014 2 / 23
Introduction
Complex-Valued Hopfield NetworksLow storage capacity!
))
Hopfield NetworkLow storage capacity!
(Chiueh&Goodman,1991)
��
(Jankowski et al.,1996)
OO
Complex-valuedRecurrent CorrelationNeural NetworkHigh storage capacity
Recurrent Correlation NetworksHigh storage capacity!
55
Marcos Eduardo Valle (Brazil) CV-RCNNs July 11, 2014 2 / 23
Outline
1 Introduction
2 Some Complex-Valued Dynamic Networks
3 Review the Bipolar Recurrent Correlation Neural Networks
4 Complex-Valued Recurrent Correlation Neural Networks
5 Computational Experiments
6 Concluding Remarks
Marcos Eduardo Valle (Brazil) CV-RCNNs July 11, 2014 3 / 23
Some Complex-Valued Dynamic Networks
Many complex-valued dynamic neural networks (CV-DNNs) aredefined as follows:
Complex-Valued Dynamic Neural Networks (CV-DNNs)
Given a complex-valued input z(0) = [z1(0), . . . , zn(0)]T , compute
zj(t + 1) = ϕ
(n∑
k=1
mjkzk (t)
), ∀j = 1, . . . ,n,
whereϕ is a complex-valued activation function.M = (mjk ) ∈ Cn×n is the synaptic weight matrix,
Marcos Eduardo Valle (Brazil) CV-RCNNs July 11, 2014 4 / 23
Recording Recipe
Given a set of vectors u1, . . . ,up, M is usually defined byThe outer product rule (or hebbian learning), i.e.,
M = UU∗.
The generalized-inverse learning (or projection rule), i.e.,
M = UU†,
whereU = [u1, . . . ,up],U∗ is the adoint (or hermitian conjungate) of U,U† is the pseudo-inverse (or generalized inverse) of U.
Marcos Eduardo Valle (Brazil) CV-RCNNs July 11, 2014 5 / 23
Some Complex-Valued Activation Functions
Definition (Complex-Signum Activation Function)
csgnK (z) =
1, 0 ≤ Arg(z) < θK ,
eiθK , θK ≤ Arg(z) < 2θK ,...
...ei(K−1)θK , (K − 1)θK ≤ Arg(z) < 2π,
for some integer K and θK = 2πK is called angular size.
Proposition (Alternative Representation of csgn)
csgnK (z) = rK ◦ φK ◦ qK (z),
where φK denotes the floor function,
qK (z) =Arg(z)θK
, and rK (x) = eixθK .
Marcos Eduardo Valle (Brazil) CV-RCNNs July 11, 2014 6 / 23
Definition (Complex-Sigmoid Activation Function)
csgmK (z) = rK ◦mK ◦ qK (z),
where
mK (x) = mod
([(K∑κ=1
11− e−(x−κ)/ε
)− 1
2
],K
).
RemarkThe csgmK has an aditional parameter ε:
The csgmK → csgnK as ε→ 0.Tanaka and Aihara suggested the value ε = 0.2.
Marcos Eduardo Valle (Brazil) CV-RCNNs July 11, 2014 7 / 23
For ε = 0.2, mK (x) ≈ x :
0
1
2
3
4
5
6
7
8
0 1 2 3 4 5 6 7 8
mK
x
Marcos Eduardo Valle (Brazil) CV-RCNNs July 11, 2014 8 / 23
Definition (Continuous-valued Activation Function)
σ(z) = rK ◦ ι ◦ qK (z),
where ι denotes the identity function. Alternatively,
σ(z) =z|z|.
RemarkWe consier σ because it is simpler than csgn and csgm.
Marcos Eduardo Valle (Brazil) CV-RCNNs July 11, 2014 9 / 23
From Hopfield to RCNNs
Consider U = {u1, . . . ,up} ⊆ {−1,+1}n.Given x(0) = [x1(0), x2(0), . . . , xn(0)]T ∈ Bn, define
xi(t + 1) = sgn
n∑j=1
mijxj(t)
, ∀i = 1, . . . ,n,
where
mij =
p∑ξ=1
uξi uξj .
Marcos Eduardo Valle (Brazil) CV-RCNNs July 11, 2014 10 / 23
From Hopfield to RCNNs
Consider U = {u1, . . . ,up} ⊆ {−1,+1}n.Given x(0) = [x1(0), x2(0), . . . , xn(0)]T ∈ Bn, define
xi(t + 1) = sgn
n∑j=1
p∑ξ=1
uξi uξj
xj(t)
, ∀i = 1, . . . ,n.
Marcos Eduardo Valle (Brazil) CV-RCNNs July 11, 2014 10 / 23
From Hopfield to RCNNs
Consider U = {u1, . . . ,up} ⊆ {−1,+1}n.Given x(0) = [x1(0), x2(0), . . . , xn(0)]T ∈ Bn, define
xi(t + 1) = sgn
p∑ξ=1
n∑j=1
xj(t)uξj
uξi
, ∀i = 1, . . . ,n.
Marcos Eduardo Valle (Brazil) CV-RCNNs July 11, 2014 10 / 23
From Hopfield to RCNNs
Consider U = {u1, . . . ,up} ⊆ {−1,+1}n.Given x(0) = [x1(0), x2(0), . . . , xn(0)]T ∈ Bn, define
xi(t + 1) = sgn
p∑ξ=1
⟨uξ,x(t)
⟩uξi
, ∀i = 1, . . . ,n.
Marcos Eduardo Valle (Brazil) CV-RCNNs July 11, 2014 10 / 23
From Hopfield to RCNNs
Consider U = {u1, . . . ,up} ⊆ {−1,+1}n.Given x(0) = [x1(0), x2(0), . . . , xn(0)]T ∈ Bn, define
xi(t + 1) = sgn
p∑ξ=1
f(⟨
uξ,x(t)⟩)
uξi
, ∀i = 1, . . . ,n,
where f : [−n,n]→ R is a continuous nondecreasing function.
Marcos Eduardo Valle (Brazil) CV-RCNNs July 11, 2014 10 / 23
Recurrent Correlation Neural Networks
Definition (Recurrent Correlation Neural Networks (RCNNs))
Consider U = {u1, . . . ,up} ⊆ {−1,+1}n.Given x(0) = [x1(0), x2(0), . . . , xn(0)]T ∈ Bn, define
xi(t + 1) = sgn
p∑ξ=1
wξ(t)uξi
, ∀i = 1, . . . ,n,
wherewξ(t) = f
(⟨uξ,x(t)
⟩), ∀ξ = 1, . . . ,p,
for some (fixed) continuous nondecreasing function f : [−n,n]→ R.
PropositionThe sequence produced by a RCNN always converges.
Marcos Eduardo Valle (Brazil) CV-RCNNs July 11, 2014 11 / 23
Complex-Valued Recurrent Correlation NNs
Definition (Complex-Valued RCNNs)
Consider U = {u1, . . . ,up} ⊆ Sn, S = {z ∈ C : |z| = 1}.Given z(0) = [z1(0), z2(0), . . . , zn(0)]T ∈ Sn, define
zi(t + 1) = σ
p∑ξ=1
wξ(t)uξi
, ∀i = 1, . . . ,n,
wherewξ(t) = f
(<{⟨
uξ, z(t)⟩})
, ∀ξ = 1, . . . ,p,
for some (fixed) continuous nondecreasing function f : [−n,n]→ R.
Theorem (Convergence of the CV-RCNNs)The sequence produced by a CV-RCNN always converges.
Marcos Eduardo Valle (Brazil) CV-RCNNs July 11, 2014 12 / 23
Examples:
The correlation CV-RCNN is obtained by considering
fc(x) = x/n.
The exponential CV-RCNN is obtained considering
fe(x) = eαx/n, α > 0.
The high-order CV-RCNN is obtained by considering
fh(x) = (1 + x/n)q, q > 1 is an integer.
The potential-function CV-RCNN is obtained by considering
fp(x) = 1/(1− x/n)L, L ≥ 1.
Marcos Eduardo Valle (Brazil) CV-RCNNs July 11, 2014 13 / 23
Experiments: Exponential CV-RCNN
0
2
4
6
8
10
12
14
16
0 2 4 6 8 10 12 14 16
Out
put E
rror
Input Error
alpha=1alpha=3alpha=5
alpha=10alpha=20alpha=30
Marcos Eduardo Valle (Brazil) CV-RCNNs July 11, 2014 14 / 23
Experiments: High-order CV-RCNN
0
2
4
6
8
10
12
14
16
0 2 4 6 8 10 12 14 16
Out
put E
rror
Input Error
q=2q=3q=5
q=10q=20q=30
Marcos Eduardo Valle (Brazil) CV-RCNNs July 11, 2014 15 / 23
Experiments: Potential-function CV-RCNN
0
2
4
6
8
10
12
14
16
0 2 4 6 8 10 12 14 16
Out
put E
rror
Input Error
L=1L=3L=5
L=10L=20L=30
Marcos Eduardo Valle (Brazil) CV-RCNNs July 11, 2014 16 / 23
Comparison between the four CV-RCNNs
Parameters: q = 10,L = 10, and α = 10.
0
2
4
6
8
10
12
14
16
0 2 4 6 8 10 12 14 16
Out
put E
rror
Input Error
CorrelationHigh-Order
PotentialExponential
Marcos Eduardo Valle (Brazil) CV-RCNNs July 11, 2014 17 / 23
General remarks:
The high-order CV-RCNN and the exponential CV-RCNN failed toperfectly recall a fundamental vector for small values of theparameters q and α, i.e., for q, α ≤ 5.The error correction capability of the high-order and exponentialCV-RCNNs have similar dependence on the parameters.The vector recalled by the potential-function CV-RCNN is alwaysvery similar to the desired fundamental vector u1.The correlation CV-RCNN failed to retrieve the fundamental vectoru1 in many steps.The high-order, potential-function, and exponential CV-RCNNs,besides apparently giving perfect recall of undistorted vectors,exhibited similar error capability for large values of theirparameters, i.e., for q,L, α ≥ 10.
In the following, we focus on the exponential CV-RCNN.
Marcos Eduardo Valle (Brazil) CV-RCNNs July 11, 2014 18 / 23
On the Effect of the Activation Function
Let us compare the effect of the three activation functions σ, csgn, andcsgm in the exponential CV-RCNN with α = 10.
For p = 12, the three variations of the exponential CV-RCNNalways recalled one of the fundamental vectors.We increased considerably the number of fundamental vectorsfrom 12 to 4096.
Marcos Eduardo Valle (Brazil) CV-RCNNs July 11, 2014 19 / 23
Probability of Successful Recall
0
0.2
0.4
0.6
0.8
1
1.2
0 0.2 0.4 0.6 0.8 1 1.2
Rec
all P
roba
bilit
y
Epsilon
csgmcsgn
sigma
Marcos Eduardo Valle (Brazil) CV-RCNNs July 11, 2014 20 / 23
Recall Phase of the Exponential CV-RCNN
Number of times the exponential CV-RCNN recalls the vector that iscloser to the input vector using the Euclidean distance.
0
0.2
0.4
0.6
0.8
1
12 4096
alpha=10alpha=20alpha=30
Marcos Eduardo Valle (Brazil) CV-RCNNs July 11, 2014 21 / 23
Storage Capacity of the Exponential CV-RCNN
101
102
103
2 4 6 8 10 12 14 16 18 20
Avera
ge o
f th
e E
stim
ate
d S
tora
ge C
ap
aci
ty
n
alpha=3 alpha=4
alpha=5alpha=6alpha=7
Straight lines:
Acn.
(least-squares).α A c3 3.43 1.054 4.41 1.125 5.39 1.226 4.18 1.397 3.50 1.58
The storage capacity of the exponential CV-RCNN visually scalesexponentially with the length n of the vectors.
Marcos Eduardo Valle (Brazil) CV-RCNNs July 11, 2014 22 / 23
Concluding Remarks
We generalized the bipolar RCNNs using complex.A CV-RCNNs is characterized by a continuous non-decreasingfunction f .The sequence produced by a CV-RCNN always converges to astationary state.Preliminary computational experiments revealed that the storagecapacity of the exponential CV-RCNN scales exponentially withthe length of the stored vectors.
A detailed account on CV-RCNNs have been submitted for publicationon IEEE TNNLS.
Thank you!
Marcos Eduardo Valle (Brazil) CV-RCNNs July 11, 2014 23 / 23