Fourth International Symposium on Neural Networks (ISNN) June 3-7, 2007, Nanjing, China

A Hierarchical Self-organizing Associative Memory forMachine Learning

Janusz A. Starzyk, Ohio UniversityHaibo He, Stevens Institute of Technology

Yue Li, O2 Micro Inc

2/23

Outline

Introduction;

Associative learning algorithm;

Memory network architecture and operation;

Simulation analysis;

Conclusion and future research;

3/23

Introduction: A biological point of view

Source: “The computational brain” by

P. S. Churchland and T. J. Sejnowski

Memory is a critical component for understanding and developing natural intelligent machines/systems

The question is: How???

4/23

Introduction: self-organizing learning array(SOLAR)

Characteristics:

* Self-organization

* Sparse and local interconnections

* Dynamically reconfigurable

* Online data-driven learning

Other Neurons

Nearest neighbour neuron

Remote neurons System clock

ID: information deficiency

II: information index

5/23

Introduction: from SOLAR to AM

Characteristics: Self-organization; Sparse and local interconnections; Feedback propagation; Information inference; Hierarchical organization; Robust and self-adaptive; Capable of both hetero-associative (HA) and auto-associative (AA)

Feed forward only Feed forward

Feed backward

6/23

Outline

Introduction;

Associative learning algorithm;

Memory network architecture and operation;

Simulation analysis;

Conclusion and future research;

7/23

Basic learning element

Self-determination of the function value:

An example:

8/23

Signal strength (SS)

Signal strength (SS) =| Signal value – logic threshold|

(SS range: [0, 1])

Provides a coherent way to determine when to trigger an association; Helps to resolve multiple feedback signals;

9/23

Three types of associations

IOA: Input only association;

OOA: Output only association;

INOUA: Input-output association;

10/23

Probability based associative learning algorithm

Case 1: Given the values of both inputs, decide the output value;

0021

2110

21

21

01

21

2111

21

21

)0,0(

)1,0,0(

)0,1(

)1,0,1(

)1,0(

)1,1,0(

)1,1(

)1,1,1()(

VIIp

FIIpV

IIp

FIIp

VIIp

FIIpV

IIp

FIIpOV

•==

===+•==

===+

•==

===+•==

====

)1)(1();1(

;)1(;

0010

0111

nmVnmV

nmVmnV

−−=−=−==

11/23

Probability based associative learning algorithm

Case 2: Given the values of one input and an un-defined output, decide the value of the other input;

( ) ( )( )1

1

211

1

212 1

)0(

)1,0(

)1(

)1,1()( IV

Ip

IIpIV

Ip

IIpIV −•

===+•

====

01001

11101

)0(

)1(

ppIp

ppIp

+==+==

For instance:

12/23

Probability based associative learning algorithm

Case 3: Given the values of the output, decide the values of both inputs;

( ) ( )( )

( ) ( )( )OVFp

IFpOV

Fp

IFpIV

OVFp

IFpOV

Fp

IFpIV

−•=

==+•=

===

−•=

==+•=

===

1)0(

)1,0(

)1(

)1,1()(

1)0(

)1,0(

)1(

)1,1()(

222

111

13/23

Probability based associative learning algorithm

Case 4: Given the values of one input and the output, decide the other input value;

For instance:

00

1

2110

1

21

011

2111

1

212

ˆ)0,0(

)1,0,0(ˆ)0,1(

)1,0,1(

ˆ)1,0(

)1,1,0(ˆ)1,1(

)1,1,1()(

VFIp

IFIpV

FIp

IFIp

VFIp

IFIpV

FIp

IFIpIV

•==

===+•==

===+

•==

===+•==

====

14/23

Outline

Introduction;

Associative learning algorithm;

Memory network architecture and operation;

Simulation analysis;

Conclusion and future research;

15/23

Network operations

Feedback operation Feed forward operation

Depth

Input data Input data

Depth

?

.

?

16/23

2I

1I

Memory operation

1

2

3

4 5

1O

1I

2I

2I

2O

3O

4O

fI2

fI1

fI2

5O

fI2

fO

fOfI2

KT = 1+= KT 2+= KT

Undefined signal

Defined signal

Recovered signal Input data

Signal resolved

based on SS

17/23

Outline

Introduction;

Associative learning algorithm;

Memory network architecture and operation;

Simulation analysis;

Conclusion and future research;

18/23

M 3M

M 3M

Class 1

Class 2

M 3M

Class 3

Hetero-associative memory: Iris database classification

N-bits sliding-bar coding mechanism:

Features:

Class identity labels:

In our simulation:

N=80, L=20, M=30

3 classes, 4 numeric attributes, 150 instances

19/23

Neuron association pathway

Classification accuracy: 96%

20/23

Auto-associative memory: Panda image recovery

30% missing pixels

Original image64x64 binary image

Error: 0.4394%

Block half Error: 2.42%

64 x 64 binary panda image:

( ) 4096,...21 == nxxxp ni 1=ix 0=ix for a black pixel; for a white pixel;

21/23

Outline

Introduction;

Associative learning algorithm;

Memory network architecture and operation;

Simulation analysis;

Conclusion and future research;

22/23

Conclusion and future research

Hierarchical associative memory architecture;

Probabilistic information processing, transmission, association and prediction;

Self-organization;

Self-adaptive;

Robustness;

23/23

It’s all about design natural intelligent machines !

Future research

Multiple-inputs (>2) association mechanism;

Dynamically self-reconfigurable;

Hardware implementation;

Facilitate goal-driven learning;

Spatio-temporal memory organization;

How far are we???

“Brain On Silicon” will not just be a

dream or scientific fiction in the future!

3DANN

Picture source: http://www.cs.utexas.edu/users/ai-lab/fai/; and Irvine Sensors Corporation (Costa Mesa, CA)