SOLAR @ ICCINC'2003

1

Janusz Starzyk, Yongtao Guo and Zhineng ZhuJanusz Starzyk, Yongtao Guo and Zhineng ZhuOhio University, Athens, OH 45701, U.S.A.Ohio University, Athens, OH 45701, U.S.A.

66thth International Conference on International Conference on Computational Intelligence and Neural ComputingComputational Intelligence and Neural Computing

Cary, NC, September 30Cary, NC, September 30thth, 2003, 2003

SOLAR @ ICCINC'2003

2

OUTLINEOUTLINE Neural Networks Neural Networks Traditional Hardware ImplementationTraditional Hardware Implementation Principle of Self-Organizing Learning Principle of Self-Organizing Learning Advantages & Simulation AlgorithmAdvantages & Simulation Algorithm Hardware ArchitectureHardware Architecture Hardware/software CodesignHardware/software Codesign Routing and InterfaceRouting and Interface PCB SOLARPCB SOLAR Future WorkFuture Work ConclusionConclusion

SOLAR @ ICCINC'2003

3

Traditional ANN HardwareTraditional ANN Hardware– Limited routing

resource.– Quadratic relationship

between the routing and the number of neuron makes classical ANNs wire dominated.

input

output

information flow

hidden

Interconnect is Interconnect is 70% of chip area70% of chip area

SOLAR @ ICCINC'2003

4

Biological Neural NetworksBiological Neural Networks

Cell body

From IFC’s webpage Dowling, 1998, p. 17

SOLAR @ ICCINC'2003

5

• What is SOLAR?What is SOLAR?New Biologically Inspired Learning Network New Biologically Inspired Learning Network

OrganizationOrganizationBasic Fabric:Basic Fabric:

A fixed lattice of distributed, parallel processing units A fixed lattice of distributed, parallel processing units (neurons)(neurons)

Self-organization:Self-organization: NNeurons chose inputs adaptively from routing channels.eurons chose inputs adaptively from routing channels. Neurons are adaptively self re-configured.Neurons are adaptively self re-configured. Neurons send output signals to the routing channels.Neurons send output signals to the routing channels. Number of neurons results automatically from problem Number of neurons results automatically from problem

complexity.complexity.

Self Organizing Learning Array Self Organizing Learning Array SOLARSOLAR

SOLAR @ ICCINC'2003

6

Self Organizing Learning Array Self Organizing Learning Array SOLAR-OrganizationSOLAR-Organization

Neurons organized in a cell array

Sparse randomized connections

Local self-organization Data driven Entropy based learning Regular structure Suitable for large scale

circuit implementation

SOLAR @ ICCINC'2003

7

Neuron’s Simulation StructureNeuron’s Simulation Structure

Neuron InputsNeuron Inputs–System clockSystem clock–Data inputData input–Control input TCIControl input TCI–Information deficiency IDInformation deficiency ID

Other Neurons

This neuronThis neuron

System clockSystem clockSystem clockSystem clock

NearestNearest neighborneighbor neuronneuron

RemoteRemote neuronsneurons

TCITCITCITCIIDIDIDID

Neuron OutputsNeuron Outputs-Data output-Data output-Control output-Control output-Information -Information deficiencydeficiency

SOLAR @ ICCINC'2003

8

Self-Organizing ProcessSelf-Organizing Process

SOLAR @ ICCINC'2003

9

Self-organizing PrincipleSelf-organizing Principle

ccc

ssssc

s csc

s

PP

PPPP

E

EI

)log(

)log()log(11

max

Information indexInformation index

Neuron self-organizes Neuron self-organizes by maximizing the by maximizing the information indexinformation index

SOLAR @ ICCINC'2003

10

Self-organizing PrincipleSelf-organizing Principle

ccc

sssc

scscs

s

pp

pppp

E

E

)log(

)log()log(

max

sio

Output information deficiency.

Information deficiency (helps to organize SOLAR learning)Information deficiency (helps to organize SOLAR learning)

The learning array grows by adding more neurons until The learning array grows by adding more neurons until input information deficiency of a subsequent neuron input information deficiency of a subsequent neuron falls below thresholdfalls below threshold

Self-organizing Self-organizing Process Matlab SimulationProcess Matlab Simulation

Initial interconnectionInitial interconnection Learning processLearning process

SOLAR @ ICCINC'2003

12

Software SimulationSoftware Simulation

TrainingData

SOLAR & otherAlgorithms

Credit card approval data

(ftp:cs.uci.edu)

SOLAR & otherClassifiers

(Simulation)

Method Miss Detection Probability

Method Miss Detection Probability

CAL5 .131 Naivebay .151

DIPOL92 .141 CASTLE .148

Logdisc .141 ALLOC80 .201

SMART .158 CART .145

C4.5 .155 NewID .181

IndCART .152 CN2 .204

Bprop .154 LVQ .197

RBF .145 Quadisc .207

Baytree .171 Default .440

ITule .137 k-NN .181

AC2 .181 SOLAR .135

SOLAR @ ICCINC'2003

13

Structure of a single neuronStructure of a single neuron

RPU: reconfigurable processing unit

CU: control unit

DPE: dynamic probability estimator

EBE: entropy based evaluator

DSRU: dynamic self-reconfiguration memory.

NI/NO: Data input/output

CI/CO: Control input/output

SOLAR @ ICCINC'2003

14

Routing StructureRouting Structure – CSU:configurable

switching unit– BRU: bidirectional

routing unit

SOLAR @ ICCINC'2003

15

Configurable Switching Unit Configurable Switching Unit (CSU)(CSU)

CSU is used to realize flexible connections among neuronsCSU is used to realize flexible connections among neurons

– Butterfly structure

– CSU can take any number of inputs

Even number of inputs

Odd number of inputs

SOLAR @ ICCINC'2003

16

Configurable Switching Unit(cont’d)Configurable Switching Unit(cont’d) Random connections of neurons with branching ratio of 50%

for 3*6 and 6*15 neurons array

Routing resources used 62.7% Routing resources used 85.3%

SOLAR @ ICCINC'2003

17

Branching Ratio of 10% Branching Ratio of 90%

Random connections of 4*7 neurons array with branching ratio of 10% and 90%

Configurable Switching Unit(cont’d)Configurable Switching Unit(cont’d)

SOLAR @ ICCINC'2003

18

HW/SW CodesignHW/SW Codesign

Partition of System Co-simulation Neuron’s architecture System initialization,

organization and management Interface JTAG

Programming

Software run in PC

PCI Bus

Hardware Board

Virtex XCV800FPGA dynamic configuration

SOLAR @ ICCINC'2003

19

SoftwareModel

InBehavioural

VHDL

Hardware Model

InStructural

VHDL

SW/HW Co-simulationSW/HW Co-simulation

• A software process– Written in

behavioral VHDL

• A hardware process– Written in RTL

VHDL which is synthesizable

• HW/SW communication– FSM and FIFOs

SOLAR @ ICCINC'2003

20

Hardware ArchitectureHardware Architecture

SOLAR @ ICCINC'2003

21

Software ArchitectureSoftware Architecture

System Design

Data I/O

API

PCI FUNC

Kernel Driver

Ctrl I/O

API Sys

Func

Har

dwar

e A

cces

s F

unct

ion

Data I/OmatDIME_DMARead.dllmatDIME_DMAWrite.dllmatviDIME_ReadRegister.dllmatviDIME_WriteRegister.dll…

Ctrl I/O

matCloseDIMEBoard.dll

matConfigDIMEBoard.dll

matOpenDIMEBoard.dll

…

PCI BUS

SOLAR @ ICCINC'2003

22

PCB DesignPCB DesignSingle SOLAR PCB contains 2x2 VIRTEX XCV1000 chips

SOLAR @ ICCINC'2003

23

SOLAR PCB Design BoardsSOLAR PCB Design Boards

Interface Board SOLAR Board

SOLAR @ ICCINC'2003

24

Neurons PrototypingNeurons Prototyping

Problem:

Neurons need to be carefully placed - otherwise some resources are lost.

Neurons memory needs to be optimized for best resource utilization.

SOLAR @ ICCINC'2003

25

Future WorkFuture Work- System SOLAR- System SOLAR

SOLAR @ ICCINC'2003

26

SOLAR is different from SOLAR is different from traditional neural networks …traditional neural networks … Expandable modular architectureExpandable modular architecture Dynamically reconfigurable hardware Dynamically reconfigurable hardware

structurestructure Interconnection number grows linearly with Interconnection number grows linearly with

the number of neuronsthe number of neurons Data-driven self-organizing learning Data-driven self-organizing learning

hardwarehardware Learning and organization is based on local Learning and organization is based on local

informationinformation

SOLAR @ ICCINC'2003

27

Why to focus on networks of Why to focus on networks of neurons?neurons?

Increases computational speedIncreases computational speed Improves fault toleranceImproves fault tolerance Constraints us to use distributed solutionsConstraints us to use distributed solutions

Brain does itBrain does it www.ent.ohiou.edu/~starzykwww.ent.ohiou.edu/~starzyk

SOLAR @ ICCINC'2003

28

Can we set milestones in developing Can we set milestones in developing

intelligent networks of neurons?intelligent networks of neurons?

HowHow to represent a distributed cognition? to represent a distributed cognition?

HowHow to model machine will to learn and act? to model machine will to learn and act?

HowHow to introduce association between patterns? to introduce association between patterns?

HowHow a machine shell implement temporal learning? a machine shell implement temporal learning?

HowHow machine shell block repetitive information machine shell block repetitive information from being processed over and over again? from being processed over and over again?

HowHow machine shell evaluate its state with respect to machine shell evaluate its state with respect to set objectives and plan its actions?set objectives and plan its actions?

HowHow to implement elements of reinforcement to implement elements of reinforcement learning in distributed networks?learning in distributed networks?

SOLAR @ ICCINC'2003

29

Questions