AMAM Conference 2005

Adaptive Motion in Animals and Machines

Outline of the talk

Short AMAM conference overview Introduction to Embodied Artificial

Intelligence (keynotes, R. Pfeifer) More detailed look at:

Sensory Motor Coordination Value-Systems

AMAM: Conference Overview

Motivation of studying Biology Source of inspiration for robotics

Model features of rather simple animals (insects…)

Robots and animals have to solve the same physical problems

Robots are useful tools for computational neuroscience Testing Neural Models within a complete

sensing-acting loop

Biorobotics

Bio-inspired technologies New sensors: Whiskers and Antennas Muscle-Like (flexible) actuators Flexible robotic arms and hands Biped and humanoid robots Numerical Models of animal and human

locomotion Central Pattern Generator based and other

control methods Some robots for illustratoin:

AMAM: robots

Scorpion [Kirchner05] 8 legged robot

BigDog [Buehler, Boston Dynamics]

AMAM: Robots

Fish Robot Iida

Stumpy „Special“ robot to

investigate cheap design locomotion (Iida)

AMAM Conference: Robots

ZAR 4 [boblan05] Bionic robot arm driven

by artificial muscels

And many more: Insects :

Coackroaches[ritzmann05]

Worm [menciassi05] Amoebic Robots

[ishiguro05] Bisam Rat [albiez05]

Embodied Artificial Intelligence [Pfeifer99, Iida03]

Not interested in the control aspects of robots alone, but rather in designing entire systems Morphology, Materials + Control

Synthetic Methology: Understanding intelligent behavior by building Concentrate on complete autonomous robots

Self-Sufficient: Sustain itself over a extended period of time Situatedness: acquires all information about the environment from its

own sensory system „Lives“ in a specified ecological niche: no need for universal robots Embodiment: real physical agents Adaptivity

„Why do plants have no brain? They do not move.“ [Brooks] Often aspects of only simple animals are modeled by robots

(locomotion of insects…) It took evolution 3 billion years to evolve insects/legged locomotion, but

only 500 million more years to develop humans => locomotion must be a hard problem

Embodied AI: Principles

Emergence: Emergent Behaviours: „emerge“ by the

interaction of the robot with the environment Not preprogrammed Agent is the result of its history Exploit the dynamics of the system More adaptive : developmental mechanisms

Diversity Compliance: Exploiting ecologicol niche / behavioral diversity Exploration/Exploitation trade off

Embodied AI: Principles

Parallel, loosely coupled processes Intelligence emerge from a lager number of parallel processes Processes are connected to the agent‘s sensory-motor aparatus

Coupling through embodiment or coordination No functional decompositon/hierarchical control like in traditional

robotic Supsumption architecture [brooks86]

Sensory-Motor Coordination Structuring sensory input Generation of good sensory-motor patterns:

Correlated Stationarity Can simplify learning

Dimensionality Reduction of sensory-motor space [lungeralla05, boekhorst03]

Embodied AI: Principles

„Morphological“ Computation Parts of the control can be „computed“ by the morphology

Facets in flies, motion paralax Springs and flexible material Exploit system dynamics for control

E.g. Exploit gravity and flexible actuators Can simplify control considerably Increase learning speed by morphology „Extreme“ Example: Passive dynamic walker

Cheap Design: Exploit physics and constraints of ecological niche Use the most simple architecture for a given task

Embodied AI: Principles

Redundancy: Overlap of functionality in the subsystems

Sensory system, Motor system Required for diversity and adaptivity

Ecological Balance: Complexity of the sensory, motor and neural system has to

match for a given task Balance between morphology, materials and control [Ishiguro03]

Value Principle Motivation of the robot to do something (should be more general

than RL) Essential for every complete autonomous agent No generally accepted solution exists 2 approaches will be discussed in more detail

Traditional Robotics / AI

In difference to traditional robotics Limited numbers of degrees of freedom (e.g. wheels) Stiff structure and joints (servo motors)

Easy to control All Computation has to be done by the control system

Limited natural dynamics Centralized rule-based control

Functional decomposition „Sense-think-act“ cycle

Problems: Frame problem Symbol grounding problem

Sensory-Motor coordination (SMC) [Pfeifer99, Lungarella05]

Used for categorization Traditional approach: Sensory-input to

category mapping Prototype or example matching

Difficulties: Often this mapping is not learnable Noise and Inaccuracies in Sensors Ambigious sensory input (Type 2 problems)

Categorization: Example [Nolfi97]

Learn 2 categories (Wall, Cylinder) with IR sensors Data for:

180 orientations, 20 distances

Learn with neural network Just linear output units 4 resp 8 hidden

neurons Very bad results: 35 %

correct categorization

Back dots: correct categoritization

SMC: Categorization

Approach the problem through interacting with the environment

Object related actions to structure the input Simplifies the problem of categorization No real internal category representation

Just different behaviors for different categories Empirical studies about Dimensionality

Reduction [lungarella05] Example in infants: Look at object from

different directions in the same distance

SMC: Example

Learning optimal categorization strategy through a genetic algorithm Nolfi‘s experiment:

Fitness: Time the robot is near the cylinder Evolved Behavior:

Robot never stops in front of target: Move back/forth and left/right hand side

SMC: Example

Learning to distinguish circles and diamonds [Beer96]

Catching circles, avoiding diamonds Agent can only move horizontally Again evolved controller

SMC: Example

Results: Not merely centering and

statically pattern matching Dynamic strategy, with active

scanning Both policies evolve sensory-

motor coordination strategies Examples show quite good the

idea of sensory-motor coordination

Other examples: Darwin II [Reeke89] Garbage Collector [Pfeifer97,

Schleier96]

Catching Circle

Avoiding Diamond

SMC: Conclusion

Nice new ideas for categorization tasks and robotics in generell Simple examples that illustrate the use of SMC for

categorization Examples are „well-suited“ for SMC

No complex categorization problem (e.g for visual object recognition) found in the literature Only numerical results which proofs dimensionality reduction

How to use them?

Critic: Humans are also able to do categorization very well without sensory-motor interaction The emphasis of SMC is a bit overstressed by the authors

Value Systems & Developmental Learning [oudeyer04/05, steels03]

Intrinsic Motivation of the Agent: learn more about the environment Ideal case: open-end learning Many different behaviors may emerge

Very adaptive 2 approaches to this problem discussed in more detail

Intelligent Adaptive Curiosity (IAC) [oudeyer04] Autotelic Principle [steels03] Still in the beginning, only for toy examples

Other approaches comming from RL Intrinsically motivated RL [singh04] Self Motivated Development [schmidhuber05]

IAC: Motivation

Push agent towards situations in which it maximizes learning progress Balance between the „unknown“ and the

„predictable“ Goal: Improve prediction machine

A(t) … action SM(t)… sensory-motor context S(t+1)… prediction

)1())(),(( tStSMtAP

IAC: framework

Prediction error

=> Decrease E(t) First naive approach

Learning Progress

Em(t)… mean Error at time t Do not reward high error values, reward high LP Meta Learning Machine (predicts error)

Choose action which maximizes Learning Progress Problem ?

||)1()1(||)( tSatStE

))()(()( DELAYtEmtEmtLP

)1())(),(( tEptSMtAMP

IAC:

Problem of naive approach: Transition from complex, not predictable

situations to simple situations is considered as learning progress

Solution: Instead of comparing the LP succesive in

time, compare the LP succesive in state space

IAC: algorithm

Prediction machine P Consists of a set of local experts. Each expert consists of training examples

Simple NN algorithm is used for prediction Build kd-tree incrementally : experts in the

leaves Each expert stores prediction errors and the

mean Calculate local learning progress

LPi(t) = -(Empi(t) – Empi(t – DELAY) Used for action selection

Very simple algorithms used More sophisticated algorithms have a good chance to

improve performance

IAC: experiments

Toy example: 2 wheeled robot, can produce sound Toy: position depends on sound

frequency intervall f1 : moves randomly f2 : stops moving f3 : toy jumps to robot

Predictor: predict relative position of the toy

IAC: experiments

Results:

Basically 3 experts First explores intervall f3, then intervall f2 f1 is not explored : not predictable

IAC: experiments

Playground experiment AIBO robot on a baby play mat Various toys: can be bitten, bashed or

simply detected

IAC: Playground Experiment

Motor Control: Turning head (2 DoF, pan + tilt) Bashing (2 DoF, strength + angle) Crouch + Bite (1 DoF, crouches given distance in direction it is

looking at) Perception:

3 High level sensors (just binary values) Visual object detection Biting Sensor Infra-red distance sensor

Bashing + Biting only produce visible results if applied in front of an appropriate object

Agent knows nothing about sensorimotor affordances

IAC: Results

Different stages evolves Stage 1: random exploration +

body babbling Stage 2: Most of the time looking

around (no biting + bashing) Stage 3: biting and bashing

Sometimes produces something, robot still not oriented to objects

Stage 4: Starts to look at objects Learns precise location of the

object Stage 5: Trying bite biteable

object, trying to bash bashable object

The Autotelic principle [steels03]

Autotelic activities: no real reward Climbing, painting…

Motivational driving signal comes from the individual itself

Balance between high challenge and required skill too high: withdrawal too low: boredom

Operational description given in [steels03], no real experiments found

Autotelic Principle: Operational Descripion

Agent: Organised in number of sub-agencies

(components) Establish input/output mapping based on knowledge Each component must be parameterized to self adjust

challenge levels Precision of movement, weights of objects… Parameter vector pi for each component

Goal: not to reach a stable state, keep exploring parameter landscape

Each component has also an associated skill vector

Autotelic Principle: Operational Descripion

Self Regulation: Operation phase: Clamp challenge

parameters, learn skills through learning Shake-Up phase:

Increase challenge: skill level already too high

Decrease challenge: performance could not be reached

Conclusion: Value Systems

Both approaches try to create open-ended learner Interesting ideas Only very simple algorithms used, or not even

implemented Open for improvement

Can help to structure learning progress in complex environments Complete autonomous agents will need some sort of

developmental value system No complex real-world experiments found

Scalable?

Conclusion: Embodied Intelligence

Provides new ways of thinking about robotic / intelligence in general

Provides a better understanding of intelligent behavior by modelling the behavior.

Good principles to design an agent Claims to solve many problems of traditionial AI

Good and promising ideas Somehow the algorithmic solutions for more complex

systems are missing Actually: same problems as for traditional AI

Works for small problems Hard to scale up

The End

Thank you!

Literature

[pfeifer99] R. Pfeifer and C. Schleier, Understanding Intelligence, MIT Press [iida03] F. Iida and R. Pfeifer, Embodied Artificial Intelligence [kirchner05] D. Spenneberg, F. Kirchner, Embodied Categorization of spatial

environments on the Basis of Proprioceptive Data, AMAM 2005 [ritzmann05] R. Ritzmann, R. Quinn, Convergent Evolution and locomotion through

complex terrain by insects, vertebrates and robots, AMAM 2005 [menciassi05] A. Menciassi, S. Spina, Bioinspired robotic worms for locomotion in

unstructered environments, AMAM2005 [ishiguro05] A. Ishiguro, M. Shimizu, Slimebot: A Modular robot that exhibits amoebic

locomotion, AMAM2005 [albiez05] J. Albiez, T. Hinkel, Reactive Foot-control for quadruped walking, AMAM2005 [boblan05] I. Boblan, R. Bannasch, A Humanlike Robot Arm and Hand with fluidic

muscles: The human muscle and the control of technical realization, AMAM 2005 [lungeralla05] M. Lungarella, O. Sporns, Information Self-Structuring: Key Principle for

Learning and Development [broekhorst03] R. Broekhorst, M. Lungarella, Dimensionality Reduction through sensory

motor-coordination

Literature

[ishiguro03] A. Ishiguro, T. Kawakatsu, How should control and body systems be coupled? A robotic case study, Embodied artificial intellingence 2003

[nolfi97] S. Nolfi, Evolving non-trivial behavior on autonomous robots: Adaptation is more powerful than decompositionand integration

[beer96] R. Beer, Toward the Evolution of Dynamical Neural Networks for Minimally Cognitive Behavior

[reeke89] G. Reeke, O. Sporns, Synthetic neural modeling: A multilevel approach to analysis of brain complexity

[pfeifer97] R. Pfeifer, C. Schleier, Sensory-motor coordination: The metaphor and beyond: Practice and future of autonmous robots

[schleier96] C. Schleier, D. Lambrinos, Categorization in a real world agent using haptic exploration and active perception

[oudeyer04] P. Oudeyer, F. Kaplan, Intelligent Adaptive Curiosity: a source of Self-Development

[oudeyer05] P. Oudeyer, F. Kaplan, The Playground Experiment: Task independent development of a curious robot.

[steels03] L. Steels, The Autotelic Principle [singh04] S. Singh, A. Barto, Intrinsically Motivated Learning of Hierarical Collections of

Skills [schmidhuber05] J. Schmidhuber, Self-Motivated Development Through Rewards for

Predictor Errors/Improvements

Measure influence of SMC [lungeralla05, broekhorst03]

New experiments with SMC Measure the effect of SMC with information processing

quantities Experiments of Broekhorst:

Robot: Wheeled CCD camera (compressed to 10 x 10 pixels) IR sensors (12) Measure angular velocity

5 different Experiments: Control setup: Move forward Moving object Wiggling : Move forward in oscillatory movement Tracking 1: Move forward + track object Tracking 2: Move forward + track moving object

Preprogrammed control

Measure Influence of SMC [broekhorst03]

Quantify dimension of the sensory information Measure Correlation on most significant

principal components from the different modalities (R*)

3 different information quantities Shannon entropy

Dominance of the highest eigenvector Number of PC‘s that explain 95% of variance

N

iii ppH

1

)(log)(

i …Eigenvalue of R*

Results:

Difference: Variance in the experiments

SMC experiments have higher variance

SMC experiments and non SMC experiments can be distinguished

No further straithforward results

Measure Influence of SMC [lungarella05]

Experimental Setup: Active Vision: (compressed 55 x 75 pixels) looking at screen 2 behaviors:

Foveation: „follow red area“ Random: Same motion structure, not coordinated

2 scenarios Artificial Scene: Random Data with moving red block Natural Images

Measure Influence of SMC [lungarella05]

Quantify sensory information Entropy Joint-Entropy Mutual Information Integration : Multivariate Mutual Information

Complexity :

Quantify Dimensionality Reduction PCA Isomap ([tenenbaum01], also recognizes non-linear

dimensions)

i

i XHxHXI )()()(

i

ii xXxHXHXC )|()()(

Results for foveation behavior

Entropy in central regions decreased

Mutual information increased

Results for foveation behavior

Integration and Complexity where much larger in the center

Results for foveation behavior

Reduced dimensionality (isomap)

Mutual information between center and motor actions also increased