Study on Some Key Issues of Synergetic Neural Network
-
Upload
jie-bao -
Category
Technology
-
view
637 -
download
2
Transcript of Study on Some Key Issues of Synergetic Neural Network
![Page 1: Study on Some Key Issues of Synergetic Neural Network](https://reader034.fdocuments.in/reader034/viewer/2022042715/559dba091a28ab3c3b8b47c2/html5/thumbnails/1.jpg)
1
Study on Some Key Issues of Synergetic Neural
Network
![Page 2: Study on Some Key Issues of Synergetic Neural Network](https://reader034.fdocuments.in/reader034/viewer/2022042715/559dba091a28ab3c3b8b47c2/html5/thumbnails/2.jpg)
2
Synergetic Associative Memory, or Synergetic Neural Network is one kind of recently
developed associative memory. Compared with traditional associative memory, it has
remarkable advantages as closing to biological pattern, short training time, low space
complexity, low time complexity in recall process, no fault states, and good associative
effect
![Page 3: Study on Some Key Issues of Synergetic Neural Network](https://reader034.fdocuments.in/reader034/viewer/2022042715/559dba091a28ab3c3b8b47c2/html5/thumbnails/3.jpg)
3
Main Work :
1、 Structure Dividing of Network
2 、 Generating Order Parameters
3 、 Order Measurement of Network
4 、 Quick Haken Network
![Page 4: Study on Some Key Issues of Synergetic Neural Network](https://reader034.fdocuments.in/reader034/viewer/2022042715/559dba091a28ab3c3b8b47c2/html5/thumbnails/4.jpg)
4
Synergetics and Synergetic Information Processing
Synergetic Information Processing is a new concept proposed by Hermann Haken (Germany) in 1980s by applying Synergetics in information science
Synergetics is a research inter-disciplinary field. It studies the cooperation among “synergized” individuals and the formation of new space structure , time structure or functional structure by cooperation
![Page 5: Study on Some Key Issues of Synergetic Neural Network](https://reader034.fdocuments.in/reader034/viewer/2022042715/559dba091a28ab3c3b8b47c2/html5/thumbnails/5.jpg)
5
The concept of synergetic computer is come from Synergetics. It utilizes the similarity between the pattern formation and pattern recognition and its most pronounced feature is that it can strictly control the behavior of the network on the mathematic meaning, so we can know its feature precisely and get no “fault status” The construction of synergetic computer is a top-down method: we recognize the expected feature first and then create the algorithm to avoid the dis-uniqueness and uncontrollability of dynamic behavior.
![Page 6: Study on Some Key Issues of Synergetic Neural Network](https://reader034.fdocuments.in/reader034/viewer/2022042715/559dba091a28ab3c3b8b47c2/html5/thumbnails/6.jpg)
6
Memory status
Memory status(fault)
Attractive Domain
![Page 7: Study on Some Key Issues of Synergetic Neural Network](https://reader034.fdocuments.in/reader034/viewer/2022042715/559dba091a28ab3c3b8b47c2/html5/thumbnails/7.jpg)
7
The progress of pattern recognition is also the progress of pattern formation. When the feature sets are given, order parameters will compete. The order parameter with the strongest initial support will win eventually and force the system to resume the absent features. This progress is corresponding to the transformation from experiment sample to basic sample.
![Page 8: Study on Some Key Issues of Synergetic Neural Network](https://reader034.fdocuments.in/reader034/viewer/2022042715/559dba091a28ab3c3b8b47c2/html5/thumbnails/8.jpg)
8
∑ ∑≠ =
−−=kk
k
M
kkkkkkk CB
' 1'
2'
2' )( ξξξξξλξ
2
1'
2'
'
22'
1
2 )(4
1
4
1
2
1 ∑∑∑=≠=
++−=M
kk
kkkk
M
kkk CBV ξξξξλ
![Page 9: Study on Some Key Issues of Synergetic Neural Network](https://reader034.fdocuments.in/reader034/viewer/2022042715/559dba091a28ab3c3b8b47c2/html5/thumbnails/9.jpg)
9
1 Division of synergetic neural network
In this chapter, we proposed that synergetic neural network is conformed to the general construction of competition neural network and can be divided into matching subnet and competition subnet. This division is helpful to better understand and improve the generation and dynamic evolution progress of synergetic neural network, and is also advantageous for applying it in some special problems 。
![Page 10: Study on Some Key Issues of Synergetic Neural Network](https://reader034.fdocuments.in/reader034/viewer/2022042715/559dba091a28ab3c3b8b47c2/html5/thumbnails/10.jpg)
10
Explanation of some key concepts in Haken Model
•Adjacent Vector and Original Vector •Order parameter •Attention parameter •Energy Function •Evolution Progress
![Page 11: Study on Some Key Issues of Synergetic Neural Network](https://reader034.fdocuments.in/reader034/viewer/2022042715/559dba091a28ab3c3b8b47c2/html5/thumbnails/11.jpg)
11
Initial value of input pattern
Order Parameter
Original Parameter
![Page 12: Study on Some Key Issues of Synergetic Neural Network](https://reader034.fdocuments.in/reader034/viewer/2022042715/559dba091a28ab3c3b8b47c2/html5/thumbnails/12.jpg)
12
Synergetic Neural Network is a kind of Competitive Neural Network
Matching Layer: Order parameter stands for the similarity between patterns.
Competition layer:
lateral suppression
winner takes all
selective attention
![Page 13: Study on Some Key Issues of Synergetic Neural Network](https://reader034.fdocuments.in/reader034/viewer/2022042715/559dba091a28ab3c3b8b47c2/html5/thumbnails/13.jpg)
13
2 Generation of Order Parameter
Key issue of constructing matching layer is to find a transformation to measure the similarity between input patterns
and original vectors and obtain the initial value of order parameters. Classic Pseudo-Inverse Method can obtain precise order parameter in the sense of inner production, but is greatly
limited on computational ability and the adaptability. We proposed that order parameter can be obtained by other
methods under the criterion of Minimal Remnant Vector and gave experiments with Mean Square Error Method, Correlation
Coefficient Method, Distance Method and Absolute Value Distance Method to overcome those problems
![Page 14: Study on Some Key Issues of Synergetic Neural Network](https://reader034.fdocuments.in/reader034/viewer/2022042715/559dba091a28ab3c3b8b47c2/html5/thumbnails/14.jpg)
14
Original Pattern
Adjacent Pattern
∑−=k
kkvqw ξ
Remnant Vector
Pattern to be recognized
![Page 15: Study on Some Key Issues of Synergetic Neural Network](https://reader034.fdocuments.in/reader034/viewer/2022042715/559dba091a28ab3c3b8b47c2/html5/thumbnails/15.jpg)
15
0 5 10 15 20 25 30 35-0.1
0
0.1
0.2
0.3
0.4
0.5
0.6
Method : Order Parameter, Remnant Vector, Evolutional Processing
kkkkTkk vvvv ′′
+′
+ =⋅= δ),(
′≠′=
=kk
kk
0
1
Definition of Pseudo-Inverse
20 40 60 80 100 120
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
![Page 16: Study on Some Key Issues of Synergetic Neural Network](https://reader034.fdocuments.in/reader034/viewer/2022042715/559dba091a28ab3c3b8b47c2/html5/thumbnails/16.jpg)
16
The shortcomings of Pseudo-Inverse Method:
•1) High limitation on the dimension N and the number M of input patterns;
•2) Those M original vectors must be linear irrelative;
•3) All samples must be recomputed to get new pseudo-inverse when new pattern is appended. It is very disadvantageous for dynamic recognition.
•4) Cannot represent the distance between input pattern and original pattern.
![Page 17: Study on Some Key Issues of Synergetic Neural Network](https://reader034.fdocuments.in/reader034/viewer/2022042715/559dba091a28ab3c3b8b47c2/html5/thumbnails/17.jpg)
17
Construct order parameter based on its basic meaning
•Distance
•Mean Square Error
•Correlative coefficient
•Absolute Value Difference
2
)0(1)0(
qvkk
−−=ξ
2/)0( esk =ξ
∑∑∑
=
ii
iki
iiki
kqv
qv
22)0(ξ
ηξ /||1 ∑ −−=i
ikik qv
![Page 18: Study on Some Key Issues of Synergetic Neural Network](https://reader034.fdocuments.in/reader034/viewer/2022042715/559dba091a28ab3c3b8b47c2/html5/thumbnails/18.jpg)
18
Strength and weakness of new methods
•Don’t need Pseudo-Inversion
• Also stand for the similarity between input pattern and original pattern
• They all can’t meet the requirement of zero remnant vector, so we need more work on minimizing remnant vector
q
vΣ
w
w*
vΣ*
![Page 19: Study on Some Key Issues of Synergetic Neural Network](https://reader034.fdocuments.in/reader034/viewer/2022042715/559dba091a28ab3c3b8b47c2/html5/thumbnails/19.jpg)
19
Experiment Result
0 5 1 0 1 5 2 0 2 5 3 0 350 .0 25
0 . 03
0 .0 35
0 . 04
0 .0 45
0 . 05
Distance Method
0 5 10 15 20 25 30 35-0.1
0
0.1
0.2
0.3
0.4
0.5
0.6
Pseudo-Inverse
0 5 10 15 20 25 30 350.5
0.6
0.7
0.8
0.9
1
1.1
1.2
1.3
Mean Square
Error Method
0 5 10 15 20 25 30 350.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
Correlation Coefficient
Method
0 5 10 15 20 25 30 350
0.05
0.1
0.15
0.2
0.25
0.3
0.35
Absolute Value Distance Method
Correct recognized pattern
![Page 20: Study on Some Key Issues of Synergetic Neural Network](https://reader034.fdocuments.in/reader034/viewer/2022042715/559dba091a28ab3c3b8b47c2/html5/thumbnails/20.jpg)
20
flops Comparing with PI Pseudo-Inverse(PI) 155,528,036 1 Distance Method 174,148 1/893 Correlation Coefficient Method 69,632 1/2233 Absolute Value Distance Method 69,700 1/2231 Mean Square Error Method 313,514 1/496
Running Cost
![Page 21: Study on Some Key Issues of Synergetic Neural Network](https://reader034.fdocuments.in/reader034/viewer/2022042715/559dba091a28ab3c3b8b47c2/html5/thumbnails/21.jpg)
21
3 ordering measurement of the network
We proposed a ordering measurement of the network: Entropy, based on the relative distribution of order parameters. And it is proved that it never decreases during the running of network. It shows that the work processing of synergetic neural network is a self-organizing processing.
![Page 22: Study on Some Key Issues of Synergetic Neural Network](https://reader034.fdocuments.in/reader034/viewer/2022042715/559dba091a28ab3c3b8b47c2/html5/thumbnails/22.jpg)
22
Definition:
∑=
ii
kk ξ
ξξ~
∑−=k
kkH ξξ ~ln
~
![Page 23: Study on Some Key Issues of Synergetic Neural Network](https://reader034.fdocuments.in/reader034/viewer/2022042715/559dba091a28ab3c3b8b47c2/html5/thumbnails/23.jpg)
23
Experiment:
0 20 40 60 80 100 120 140 160 180 2000
0.5
1
1.5
2
2.5
0 50 100 150 200 250 300 350 4000
0.5
1
1.5
2
2.5
3
3.5
4
Pseudo-Inverse Correlation Coefficient Method
It can be proved that the entropy is non-decreasing during the evolution of synergetic neural network:
0≤∂
∂t
H
![Page 24: Study on Some Key Issues of Synergetic Neural Network](https://reader034.fdocuments.in/reader034/viewer/2022042715/559dba091a28ab3c3b8b47c2/html5/thumbnails/24.jpg)
24
4 Quick Haken Network
In the study of competitive sub-network, we mainly analyzed the improvement on the time performance of classic Haken network and proposed Quick Haken Network. With detailed analysis for potential function, the evolutionary locus of order parameter with certain initial value can be full determined by the initial value of order parameter and the attention parameter. This method can greatly reduce the required time for iteration by classic Haken network.
![Page 25: Study on Some Key Issues of Synergetic Neural Network](https://reader034.fdocuments.in/reader034/viewer/2022042715/559dba091a28ab3c3b8b47c2/html5/thumbnails/25.jpg)
25
Defect of classic Haken Model: Classic Haken model requires resolving high-dimension nonlinear equation groups when the number of patterns is very large. This iterating procedure
requires large time and memory resource.
Resolution:1) Hierarchical Competition in Matching layer
2) Improvement in Competition Layer
2.1) From the view of time cost, determine the competitive result from the initial condition of the equation.
2.2) From the view space cost, convert differential function groups into sparse matrix and then solve.
We only discuss method 2.1 here.
![Page 26: Study on Some Key Issues of Synergetic Neural Network](https://reader034.fdocuments.in/reader034/viewer/2022042715/559dba091a28ab3c3b8b47c2/html5/thumbnails/26.jpg)
26
Quick Haken Network
• Balanced attention parameter: the greatest order parameter will win eventually
•Unbalanced attention parameter :
Then the order parameter corresponding to pattern j will die away.
)()()0()0(
))0(/)0(ln()1(
τξτξξξ
ξξτλλτλτλ
ijij
jiij
ij ee >⇔>⇔
>−
![Page 27: Study on Some Key Issues of Synergetic Neural Network](https://reader034.fdocuments.in/reader034/viewer/2022042715/559dba091a28ab3c3b8b47c2/html5/thumbnails/27.jpg)
27
Potential Function Analysis : Balanced Attention Parameter
Potential function: 4 attractors can be found clearly in this figure and is axial symmetrical each other. The hill at original point is represented for instable equilibrium point . We can also find 4
saddle points on the diagonal direction.
A
B
C
D
E
F
G
H
I
Note: there are 4 attractors (-0.25): B,C,D,E; 1 unstable stationary point (A),4 saddle points (F, G, H, I). And the shape of two attractor is same. It shows the meaning of “balanced” attention parameter.
![Page 28: Study on Some Key Issues of Synergetic Neural Network](https://reader034.fdocuments.in/reader034/viewer/2022042715/559dba091a28ab3c3b8b47c2/html5/thumbnails/28.jpg)
28
Potential Function Analysis: unbalanced attention parameter 0.6-0.4
Potential function. Note that both the width and depth of attractive domain of pattern 2 are greatly reduced comparing with that of balanced parameters
Note: Attractive domain of attractor 1 (pattern 1) is enlarged while that of pattern 2 is reduced. It shows that attention parameters have important influence on the attractive domain of patterns.
![Page 29: Study on Some Key Issues of Synergetic Neural Network](https://reader034.fdocuments.in/reader034/viewer/2022042715/559dba091a28ab3c3b8b47c2/html5/thumbnails/29.jpg)
29
Depth of Attractors
1.0 – 1.0 0.6 – 0.4 0.8 – 0.2
1ξ
0 50 1 00 15 0 200 2 50
-0 .09
-0 .08
-0 .07
-0 .06
-0 .05
-0 .04
-0 .03
-0 .02
-0 .01
0
0 0 50 10 0 15 0 2 00 2 50
-0 .16
-0 .14
-0 .12
-0 .1
-0 .08
-0 .06
-0 .04
-0 .02
0
0
2ξ
0 50 1 00 15 0 200 2 50-0 .04
-0 .03
-0 .02
-0 .01
0
0 .01
0 .02
0 .03
0 .04
0 .05
0 0 5 0 1 00 1 5 0 2 0 0 25 0-0 .0 2
0
0 .02
0 .04
0 .06
0 .08
0 .1
0 .12
0 .14
0 .16
0
![Page 30: Study on Some Key Issues of Synergetic Neural Network](https://reader034.fdocuments.in/reader034/viewer/2022042715/559dba091a28ab3c3b8b47c2/html5/thumbnails/30.jpg)
30
Attractive Domain
Balance Attention Parameter
0.6-0.4 0.8-0.2
![Page 31: Study on Some Key Issues of Synergetic Neural Network](https://reader034.fdocuments.in/reader034/viewer/2022042715/559dba091a28ab3c3b8b47c2/html5/thumbnails/31.jpg)
31
Experiment : Balanced Attention Parameters
10 20 30 40 50 60 70 80 90 100-0.5
0
0.5
1
1.5
2
2.5
3
Time Cost flops Classic Algorithm 2.0420 40900
Quick Algorithm <0.00005 34
![Page 32: Study on Some Key Issues of Synergetic Neural Network](https://reader034.fdocuments.in/reader034/viewer/2022042715/559dba091a28ab3c3b8b47c2/html5/thumbnails/32.jpg)
32
Experiment : Unbalanced Attention Parameters
50 100 150 200 250 300-0.5
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
1 2 3 4 5 6 7 8 9 10 11
-0.1
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
Time Cost
flops
Classic Algorithm 7.2890 122700
Quick Algorithm 0.0667 143
![Page 33: Study on Some Key Issues of Synergetic Neural Network](https://reader034.fdocuments.in/reader034/viewer/2022042715/559dba091a28ab3c3b8b47c2/html5/thumbnails/33.jpg)
33
Outlook for High dimensional competition
Basic principles for constructing V
penaltytV += cos∑ ••ξcp )(ξ••∑ fr
Restriction should can be transformed into the summation of multiplication form of the square of order parameters. There will be same # of penalty term with the dimension of problem.
The evolution function of order parameters should include three parts:• One order term with attention parameter as self-exciting item• Cubic order term with negative parameter as self-suppressive item• Lateral suppressive item between an order parameter and other order parameters. It stands for constrain in the evolution process.