Outline IM-CLeVeR: Intrinsically Motivated Cumulative Learning Versatile Robots

Gianluca Baldassarre 1/20Sestri Levante, 18 January 2010

IM-CLeVeR: Intrinsically MotivatedCumulative Learning

Versatile Robots

Gianluca Baldassarre, Marco Mirolli,Francesco Mannella, Vincenzo Fiore, Stefano Zappacosta,

Daniele Caligiore, Fabian Chersi, Vieri Santucci, Simona Bosco


OutlineIM-CLeVeR: Intrinsically Motivated

Cumulative Learning Versatile Robots

The “numbers” of the project The partners The project vision The 3 pillars of the project hypothesis + 4 S/T objectives WP3: Experiments WP4: Abstraction WP5: Intrinsic motivations WP6: Hierarchical architectures WP7: Integration and demonstrators Conclusions


The “Numbers” of the ProjectIM-CLeVeR: Intrinsically Motivated

Cumulative Learning Versatile Robots

Integrated Project Coordinator: ISTC-CNR Call: Cognitive Systems, Interactions and Robotics EU funds: 5.9 ml euros 7 Partners Start: May 2009 End: April 2013


Partners1. CNR-ISTC-LOCEN Coordinator

(Gianluca Baldassarre, Marco Mirolli)

1. CNR-ISTC-UCP (Elisabetta Visalberghi)

1. UMASS (Andrew Barto)

2. USFD (Peter Redgrave & Kevin Gurney)

3. UCBM-LBRB (Eugenio Guglielmelli)

3. UCBM-LDN (Flavio Keller)

4. FIAS (Jochen Triesch)

5. AU (Mark Lee)

6. UU (Ulrich Nehmzow)

7. IDSIA (Juergen Schmidhuber)

Institutes Groups


Vision: the problem How can we create “truly intelligent” robots?

Versatile: have many goals; re-use actions Robust: function in different conditions, with noise Autonomous: learning is paramount

Weng, McClelland, Pentland, Sporns, Stockman, Sur, Thelen, (Science, 2001):

…knowledge-based systems (e.g. production systems)… …learning systems focussed on single tasks (e.g. RL)… …evolutionary systems… Important results, but limited autonomy and scalability. . . . . . on the contrary . . .

. . . organisms do scale, are flexible, and are robust!


Vision: the idea Why are organisms so special? Let’s give a closer look at children…


Vision: the ideaIngredients: Powerful abstractions: “elefant on table leg”, “it slides down” Explore and record interesting states:

Based on intrinsic motivations (novelty, learning rates, …) Such states motivate to reach them (= goals) Furnish learning signals which guide learning

Acquired skills are: Re-used to explore and discover new goals Composed to produce new skills

Science: which brain and behavioural mechanisms are behind these processes?

Technology: Can we reverse engineer them?


Vision: 2 promises Science: we can understand the mechanisms in organisms Technology: we can develop a new methodology for designing robots…

… in particular robots that (we will get 3 iCubs!)…

Learn actions cumulatively:

…on the basis of abstraction

(sensory andmotor)…

…on the basis of intrinsic

motivations…

…on the basis of already learned

actions.


Vision: how we will do it:3 pillars + 4 S/T objectives

WP4: Abstraction and attention

WP5: Intrinsic motivations

WP6: Hierarchical architectures to support

cumulative learning

1. Empirical investigations:

- Monkeys - Children - Adults - Parkinson patients

4. Two robotic demonstrators:- CLEVER-B- CLEVER-K

2. Computational bio-constrained models:mechanisms underlying brainand behaviour

Suitable representations

Focussing learning

Science

From Science to Technology

Technology

3. Machine-learning models:powerful algorithms and architectures

From Technologyto Science


WP3: Experiments and mechatronic board




cumulative learning




2. Computational bio-constrained models:mechanisms underlying brainand behaviour


Focussing learning

Science


Technology


WP3


WP3: Empirical Experiments: “Board experiment” UCBM-LBRB (Eugenio Guglielmelli); UCBM-LDN (Flavio Keller): children

CNR-ISTC-UCP (Elisabetta Visalberghi): monkeys;

Inertial/magnetic unit + battery + wireless

Tactile sensors

Sabbatini, Stammati, Tavares, Visalberghi, 2007,Amer. J. PrimatologyCampolo, Taffoni, Schiavone,

Formica, Guglielmelli, Keller, 2009, Int. J. Sicial Robotics


WP4: Abstraction




cumulative learning




2. Computational bio-contrained models:mechanisms underlying brainand behaviour


Focussing learning

Science


Technology


WP4


WP4 Abstraction: motor, perception, attention, vergence,

Abstraction is a key ingredient for action hierarchies Abstraction is a key ingredient for intrinsic motivations

Schembri, Mirolli, Baldassare, 2007,ICDL, ECAL, EPIROB

Neto Nehmzow, 2007, Rob. & Aut. Syst.


WP5: Novelty detection




cumulative learning




2. Computational bio-contrained models:mechanisms underlying brainand behaviour


Focussing learning

Science


Technology


WP5


WP5 Intrinsic (extrinsic) motivations Extrinsic motivations

(e.g. food, sex, money): Psychology (Berlyne,

White, Deci & Rayan):motivate actions to achieve specific goals

Drive actions whose effects directly increase fitness

Come back again with the homeostatic needs they are associated with

Intrinsic motivations (skill/knowledge acquis.):

Psychology: motivate actions for their own sake

Drive actions whose effects are an increase in:(a) knowledge or prediction ability;(b) competence to do

Terminate to drive actions when knowledge or competence is acquired


WP5 Intrinsic motivations CNR-LOCEN (Gianluca Baldassarre, Marco Mirolli) Young robot: low level of hierarchy develps skills based on

evolved ‘reinforcers’ (knowledge-based intrinsic motivations) Young robot: high level of hierarchy selects skills which produce

the highest suprise (competence-based intrinsic motivations) Adult robot: high level of hierarchy performs skill composition to

achieve salient goals (external rewards fitness measure)Adult robot tasks

Child robot taskYoung robot: resultsBefore learning

After learning

Adult robot: results

Schembri, Mirolli, Baldassare, 2007, ICDL, ECAL, EPIROB


WP6: Hierarchical architectures




cumulative learning




2. Computational bio-mimetic models:mechanisms underlying brainand behaviour


Focussing learning

Science


Technology


WP6


WP6 Bio-inspired / bio-constrained hierarchical reinforcement learning

CNR-LOCEN (Gianluca Baldassarre & Marco Mirolli) Piaget theory: actions support learning of other actions Camera, dynamic arm, reaching tasks Continuous state/action reinforcement learning Hierarchical RL: segmentation, Piaget

Caligiore Borghi Parisi Mirolli Baldassarre, ongoing From Fuster, 2001, Neuron


Conclusions: A timely project! Timely research goals:

sensorimot. abstraction, intrinsic motiv., hierarchical architect. Within important trends:

Developmental robotics Computational system neuroscience Emotions/motivations

In synergy with various events:EpiRob, ICDL, IEEE Journal Automonous Mental Development

In line with EU calls:“Cognitive Systems, Interactions and Robotics”

First EU Integrated Project wholly focussed on these topics

www.im-clever.eu


What we need from iCub

Robustness! Usability (assistance):

Software: Yarp, simulator for rapid prototyping Hardware: when it will break

One standardised simulator (e.g., based on Bullet) Compliance: for safety, for more bio-realism That it actually becomes a standard in EU research

Outline IM-CLeVeR: Intrinsically Motivated Cumulative Learning Versatile Robots

Documents

Transcript of Outline IM-CLeVeR: Intrinsically Motivated Cumulative Learning Versatile Robots