1

Robotics

2

Rule-based Integration of Multiple Neural Networks Evolved Based on Cellular Automata

3

Motivation

• Construction of mobile robot controller– Evolving neural networks using genetic algorithm (Floreano,

1996)– Genetic programming (Nordin, 1997)– Fuzzy (Cho, 1996)

• Integration of multi-modules– Solving complex task

→ Simpler components or subtasks– Learn each module by separate systems– Combine them to solve goal task

4

Behavior-based Robot : Khepera

8 distance sensors ( 0 ~ 1023 )8 light sensors ( 50 ~ 500 )2 motors ( -10 ~ 10 )

Battery level sensor ( 0~2500 )Floor-brightness sensor

Battery RechargeArea

LightSource

5

Evolved NNs based on CA

Generate initial population

Develop NNs based on CA from chromosome

Apply it to a problem

Evaluate fitness of each NN

Select good NNs in population

Maniplate them with genetic operater

Generate new population

Satisfied NN found ? Yes

NoStop

6

CA-Based Neural Networks

Chromosome Input

Output

Growth phase Signal phase

Transmit signals from input cells to output cells

Make neural network structurewith chromosome

Applying to Control a Mobile Robot

Growth phase Signaling phase Khepera Simulator

Fitness value

Evolution

Chromosome

Neural Networks

Sensor value

Velocity of wheels

8

Cell States

• Blank – Empty space– No participate in any cell interaction during signaling

• Neuron– Collect neural signals from surrounding dendrite cells– Send them to surrounding axon cells

• Axon– Distribute signals to neighborhoods originating from neuron

• Dendrite – Collect signals from neighborhoods – Pass them to neuron cells

Growth Phase

(a)

(d) (f)

(c)(b)

(e)

y1

x1

y2

y3

y4

x2 x3 x4

Signaling phase

Collecting signals

Distributing signals

ExcitatoryAxonInhibitory

Axon

Excitatory signalInhibitory signal

Neuron

Dendrite

11

Basis Behaviors

• Battery Recharge– If a robot arrives at battery recharge area, battery is recharged.

• Go Ahead– If there is nothing around the robot, it goes ahead.

• Follow Light– The robot goes to stronger light.

• Avoid Obstacles– If the obstacles exist around the robot, it avoids obstacles

without bumping against them.

12

“Battery Recharge”

• Programmed Module• Enables it to live for a long time• “Battery Recharge” area

– Black area in environment– Light source exists

13

“Go Ahead”

• Programmed module– The velocity of left : 5– The velocity of right : 5

• Make it to move continuously without stopping

14

“Follow Light” (1)

• The robot goes to stronger light => Go to “Battery Recharge” area• Evolved module

– Evolving CAM-Brain to follow light• Fitness Function

S : Average speed of the two wheelsV : Difference between the velocity of two wheelsc : 1 (if the robot does not bump against obstacles)

1/2 (if the robot bumps)D : Distance from the robot to the goal position

15

“Follow Light” (2)

Best Fitness Average Fitness

16

“Follow Light” (3)

0 degree

180 degrees 270 degrees

90 degrees

17

“Avoid Obstacles”

• Evolved module– Evolving CAM-Brain to avoid obstacles without bumping– Using CAM-Brain module evolved incrementally in previous work

18

Incremental Evolution

• Single system learns a succession of tasks

• Task {t1, t2, t3, …. , tn}

– tn : goal task

– ti is easier than ti+1 for all i: 0 < i <= n

– Each task is derived by transforming a goal-task

19

Environments

(a) (b) (c)

(f)(e)(d)

20

Fitness

Incremental Evolution Direct Evolution

21

Architecture of NNs

1

2

3

4

5

7

6

8

1

2

3

4

5

7

6

8

1

2

3

4

5

8

7

9

1

10

11

12

13

15

14

11

17

18

16

22

Applying to Other Environments

23

Rules of Combining (1)

IF (“Battery Recharge” area)Battery Recharge module

ELSEIF (Battery sensor < α) AND (Minimum value of light sensors ≤ γ)

IF (Maximum value of distance sensors ≤ β1)Follow Light module

ELSEAvoid Obstacles module

ELSEIF (Maximum value of distance sensors ≤ β2)

Go Ahead moduleELSE

Avoid Obstacles module

24

Rules of Combining (2)

• α : If battery sensor value is less than α, battery is needed to recharging.

• β1, β2 : If the maximum values of distance sensors are larger than β1, β2, the robot recognize as obstacles.

• γ : If the minimum values of light sensor is less than γ, the robot recognizes as light.

25

Results of Combining

Moving 5000 step Moving 14000 step Moving 10000 step

26

Coordination of Multiple Behavior Modules Evolved on CAM-Brain

27

Motivation

• Construction of mobile robot controller– Evolving neural networks using genetic algorithm (Floreano, 1996)– Genetic programming (Nordin, 1997)– Fuzzy system evolved by genetic algorithm(Cho, 1996)

• Integration of multi-modules by action selection mechanism– Solving complex task

→ Simpler components or subtasks– Learn each module by separate system– Combine them to solve goal task by action selection mechanism

28

Action Selection Mechanism

• Proposed by Maes (1989)• Idea : Integrate high-level behavior into a system using lower-level

behaviors• Distributed, non-hierarchical network• Two waves of input to the network

– Sensors of external environment– Motivations or goals

• Different types of links encoding various relationships• Components of ASM : Nodes, internal links and external links

29

Example

30

Nodes

PRECONDITIONS

ADD LIST

DELETE LIST

ACTIVATION

EXECUTABLECODE

INPUT OUTPUT

PREDECESSOR LINKS

SUCCESSOR LINKS

CONFLICTOR LINKS

GOALS

ENVIRONMENT

PREDECESSOR LINKS

SUCCESSOR LINKS

CONFLICTOR LINKS

31

Internal Links

• A→B is active– Predecessor link

If (p = false) ∧ (p ∈ precondition of A) ∧ (p ∈ add list of B)– Successor link

If (p = false) ∧ (p ∈ add list of A ) ∧ (A is executable) ∧ (p ∈ precondition of B)

– Conflictor linkIf (p = true) ∧ (p ∈ precondition of A ) ∧ (p ∈ delete list of B)

32

External Links

• p→A is active– From sensors of the environment

If (p = true) ∧ (p ∈ precondition of A) – From goals

If (p ≥ 0 ) ∧ (p ∈ add list of A) – From protected goals

If (p ≥ 0 ) ∧ (p ∈ delete list of A)

33

Action Selection Procedure

• Activation of a node is updated by excitation coming in from theenvironment and the motivations

• By the type of internal links, activation of a node is exchanged between two nodes

• Normalize the node activation AVG(SUM( (activation of node1) + (activation of node2) ….)) = π

• If (all preconditions of a node are true) and (activation of a node ≥ θ )

then the node is executable • If (executable node exist )

then execute the node else θ is reduced by 10% and repeat cycle

34

Experiment Environments

Battery RechargeArea

LightSource

Battery RechargeArea

LightSource

Simple Chaotic

35

Basic Behaviors

• Recharging batteryIf a robot is in battery recharge area, then battery is recharged

• Following light Robot goes to stronger light

• Avoiding obstacles If obstacles exist around the robot, then robot avoids them without bumping

• Going straight Robot goes ahead

36

Sensors and Goals

Sensors of environment (Binary value) Goals (Real or binary value)

“In battery recharge area”“Near battery recharge area”

“In shade area”“Nothing around the robot”

“Obstacles are near”

“Battery is not zero”“Battery is not below the half”

37

Precodition and Add List of Nodes

Precondition Add list

“In battery recharge area” “Battery is not zero”“Battery is not below the half”

Recharging battery

Followinglight

GoingStraight “Nothing around the robot”

“Obstacles are near””In battery recharge area”

“Near battery recharge area”

“Obstacles are near” “Nothing around the robot”

“In shade area”“Near battery recharge area” “In battery recharge area”

Avoiding Obstacles

38

Internal Links between Nodes

Predecessor Links Successor Links

(Recharging battery → Following light)(Recharging battery → Going straight)

(Following light → Going straight)(Going straight → Avoiding obstacles)

(Avoiding obstacles → Goinging straight)

(Following light → Recharging battery)(Going straight → Following light) (Going straight → Recharging battery)(Going straight → Avoiding obstacles)(Avoiding obstacles → Going straight)

39

Model of ASM

Battery is not zero

Battery is not below the half

In battery recharge area

In shade area

Obstacles are near

Near battery recharge area

Nothing around the robot

Rechargingbattery

Followinglight

AvoidingObstacles

Goingstraight

40

Simulation Results

Simple Chaotic

41

Action Sequence (1)

0

1

2

3

4

5

1 1001 2001 3001 4001

1 = Recharging Battery2 = Following Light3 = Avoiding Obstacle4 = Going Straight

42

Action Sequence (2)

0

1

2

3

4

5

1 501 1001 1501

A B C D

43

A Framework of Evolvable Systems and Measures for

Intelligent Agents

44

Need for Agents

• Conventional information processing– Inference, planning, and commands directed by users

• Increase of the amount of information

Need for an agent that works for a user

45

Intelligent Agent

• Intelligent agent: achieve user’s goals autonomously instead of users

• Properties of intelligent agents– Autonomy– Reactivity– Proactivity– Reasoning and learning– social ability

• Cooperation• Communication

Cooperate Learn

Autonomous

Collaborativelearning agents

Smart agents

Collaborative agents Interface agents

A category of intelligent agents

46

Related Works:Application of Soft Computing Techniques

• Soft computing techniques– Neural networks, fuzzy, probabilistic inference, evolutionary

computation– Flexible inference/random searching capability

• Type 1: application of each soft computing technique– To Simple problems– Difficult to tune internal parameters– Require expert’s knowledge/time and effort

• Type 2: combination with evolutionary algorithms– Tuning of internal parameters by evolutionary algorithms

47

AdaptiveEvolution

HighEvolvability

LowEvolvability

Non-AdaptiveEvolution

GoodSolution

BadSolution

DesirableEvolutionary Causes

and Effects

High probability

Low probability

Emergence

AdaptiveBehavior

Problems of Type 2

• Can the same results be obtained? Adaptive evolution( )• What properties are genetically preferred? Adaptive behaviors( )• How the solutions are formed? Evolutionary pathways to the solutions( )• Behavioral properties? Illustration of emergence( )

1

2

1

2

4

3

3

4

48

Research Objectives

• A framework for intelligent agents easy to – represent expert’s knowledge– analyze evolution

• Application to a real-world agent to show the usefulness

49

A Soft Computing Framework

AgentConstruction

Rule-BasedSystem

EvolutionaryAlgorithms

HardwareAgent

SoftwareAgent

EvolutionAnalysis

EvolutionaryActivity

Statistics

SchemaAnaysis

ObservationalEmergence

Analysis ofEvolution

AdaptiveEvolution

AdaptiveBehavior Emergence

Evolutionary

Pathways

Research goals How to achieve the goals What we can get or show

BehaviorAnalysis

50

Rule-based Systems

• Working memory: represents current facts• Inference engine: infer appropriate actions based on current facts• Knowledge base: a set of rules for inference

IF (condition) THEN (action)….IF (condition) THEN (action)

• Merits– Easy to represent expert’s knowledge– Easily understood by humans

• Demerits– Need knowledge base

Working Memory(Facts)

InferenceEngine

Knowledge Base(Rules)

RuseBasedSystems

51

Genetic Algorithm (GA)

CandidateSolutions

Test

Terminate ?

End

CrossoverMutation

ReproduceY

N

Genetic Algorithm Parameters

Encoding of chromosomes

Types of algorithms: Simple GA, Overlapped GA

Genetic operations: Mutation/Crossover

Others

52

Evolutionary Activity Statistics

• Attach a counter to each component (e.g., alleles, individuals, …)• Accumulate counter values over generations• Need to define increment function for the counter

MeanActivity

NewActivity

Adaptability ofeach component

Overalladaptabilityof a system

Adaptiveinnovations

EvolutionaryActivity

53

Schema

• Definition– A similarity template representing a subset of strings with

similarity at certain string positions ( Holland 1968)• Composed of a character set and a don’t care character• Examples

– Character set = {0,1}, don’t care=#– #0000 {10000, 00000}– #111# {01110, 01111, 11110, 11111}

54

Observational Emergence

• “creation of new properties” – Morgan, C.L., Emergent Evolution, Williams and Norgate, 1923

• Observational emergence– Proposed by Bass, N.A.

• S : structure - system, organization, organism, machine, …• P : a property of S observed by observational mechanism

Obs

55

Application to a Robot Agent

• Robot Agent– Khepera mobile robot: 8 proximity sensors, 2 motors

56

AgentConstruction

Rule-BasedSystem

Fuzzy Logic Controller

Evolutionary Algorithms

Genetic Algorithm

EvolutionAnalysis

EvolutionaryActivity

Schema Anaysis

ObservationalEmergence

Analysis ofEvolution

AdaptiveEvolution

AdaptiveBehavior Emergence

EvolutionaryPathways

to Solutions

Research goals How to achieve the goals What we can get or show

BehaviorAnalysis

57

Fuzzy Rule-based System

Working Memory

Inference Engine

Knowledge BaseFuzzy

Rule-basedSystem

Min-Max Correlation

Eight proximitysensors

Centroid Defuzzification

20 rules maximum

OthersNumber of fuzzy sets

Membership functions

Output:Two motors

58

Genetic Algorithm

변수의퍼지집합코딩

Parameters

Encoding of chromosomes

Types of algorithms: Simple GA, Overlapped GA

Genetic operations:Mutation/crossover

Others

Overlapped GA (50%)

Two point crossover (50%)Mutation (1%)

50 individuals

59

Adaptive Evolution

0 100 200 300 400 500 600 700 800 9000

20

40

60

80

100

120

Generation

Mea

n A

ctiv

ity

Fuzzy ModelNeutral Shadow

0 100 200 300 400 500 600 700 800 9000

0.02

0.04

0.06

0.08

0.1

0.12

0.14

0.16

0.18

0.2

Generation

New

Act

ivity

Fuzzy ModelNeutral Shadow

60

Adaptive Behaviors

+ +

61

Evolutionary Pathways

62

14th individual (780)

0111001112140000311

0003120304010102401

0304000410130204031

0002040003010411441

0400010114000110211

......

...

0111001112140410311

...

9th individual (600)

0003120304020102401

0301100410130104031

...

0111101112140411311

...

6th individual (578)

...

0003120304020102401

...

25th individual (578)

0301100410130104031

...

0111001112140000311

...

5th individual (775)

0003120304010102401

0304000410130204031

...

0002040003010411441

...

27th individual (777)

0400010114000110211

...

0002030003010411441

...

24th individual (605)

0400040014040010211

...

0400040014040010211

...

6th individual (599)

...

0302020112010300001

...

6th individual (6)

...

0114041013001300130

...

33th individual (8)

...

0403120200100210101

...

21th individual (1)

...

0002030003010411441

...

12th individual (603)

...

1111111100141100131

...

1001010001030411441

27th individual (0)

M

C

M M M

CC

M

M

M

C

M

...

0301100410130104031

...

12th individual (574)

1

4

4

1

1

1

1

4

4

4

2

2

...

0003120304020102401

...

47th individual (576)

2

2

2

2

3

3

3

3

3

3

5

5

5

5

5

M Mutation

C Crossover

Best (816th generation)

0111001102140000311

0003120304010103401

0304000410130204031

0102041003010411441

0400110114000110211

......

1

2

3

4

5

1 5~ Tags for information

63

Observational Emergence:Turning Around Behavior

First order structures Obs1

Int

Obs2

• The interactions of the three first-order structures make Obs2(S2) different from the Obs1( ), of the first-order

• This implies that

• Therefore, we can conclude that Obs2(S2) is observationally emergentbehavior

17

15

12 111

,, SSS11iS }7,5,2{∈i