MindRACES, First Review Meeting, Lund, 11/01/2006

Fovea-Based Robot Control for Anticipation Studies in Various

Scenarios

Alexander Förster, Daan Wierstra, Jürgen Schmidhuber

IDSIA - Lugano - Switzerland

MindRACES, First Review Meeting, Lund, 11/01/2006 2

Overview

• Real world scenarios

• Fovea

• Software Framework

• Robertino Robot

• Simulated 3D scenarios

• Integration and Future Work

MindRACES, First Review Meeting, Lund, 11/01/2006 3

Instances of scenario 1: find an object

Still Image

Robot Lab

Office Room

Movie (LUCS)

2D E

nviro

nmen

ts3D

Environm

ents

MindRACES, First Review Meeting, Lund, 11/01/2006 4

Fovea Centralis Simulation

Three regions with increasing resolutions

MindRACES, First Review Meeting, Lund, 11/01/2006 5

Fovea Centralis Simulation

Three regions with increasing resolutions

Subsample each region to 13x11 pixels

MindRACES, First Review Meeting, Lund, 11/01/2006 6

Fovea Centralis Simulation

Three regions with increasing resolutions

Subsample each region to 13x11 pixels

Combine the subsampled image

MindRACES, First Review Meeting, Lund, 11/01/2006 7

?? ? ? ? ? ? ? ? ?? ?

Simulated Vision: 1-D Fovea Simulation

Task: Find the sad smiley

Bins Objects

1-D world

Transformation

Fovea

Memory

State 0

MindRACES, First Review Meeting, Lund, 11/01/2006 8

?? ? ? ? ? ? ? ? ?? ?

Simulated Vision: 1-D Fovea Simulation

Task: Find the sad smiley State 1

Bins Objects

1-D world

Transformation

Fovea

Memory

MindRACES, First Review Meeting, Lund, 11/01/2006 9

?? ? ? ? ? ? ? ? ?? ?

Simulated Vision: 1-D Fovea Simulation

Task: Find the sad smiley State 2

Bins Objects

1-D world

Transformation

Fovea

Memory

MindRACES, First Review Meeting, Lund, 11/01/2006 10

?? ? ? ? ? ? ? ? ?? ?

Simulated Vision: 1-D Fovea Simulation

Task: Find the sad smiley State 3

Bins Objects

1-D world

Transformation

Fovea

Memory

MindRACES, First Review Meeting, Lund, 11/01/2006 11

? ? ? ? ? ?

Simulated Vision: 1-D Fovea Simulation

Task: Find the sad smiley State 4

Bins Objects

1-D world

Transformation

Fovea

Memory

MindRACES, First Review Meeting, Lund, 11/01/2006 12

? ? ? ? ? ?

Simulated Vision: 1-D Fovea Simulation

Task: Find the sad smiley State 5

Bins Objects

1-D world

Transformation

Fovea

Memory

MindRACES, First Review Meeting, Lund, 11/01/2006 13

? ? ? ? ?

Simulated Vision: 1-D Fovea Simulation

Task: Find the sad smiley State 6

Bins Objects

1-D world

Transformation

Fovea

Memory

MindRACES, First Review Meeting, Lund, 11/01/2006 14

? ? ? ? ?

Simulated Vision: 1-D Fovea Simulation

Task: Find the sad smiley State 7

Bins Objects

1-D world

Fovea

Memory

Transformation

MindRACES, First Review Meeting, Lund, 11/01/2006 15

Simulated Vision: 1-D Fovea Simulation

Task: Find the sad smiley State 8

Bins Objects

1-D world

Fovea

Memory

Transformation

MindRACES, First Review Meeting, Lund, 11/01/2006 16

Simulated Vision: 1-D Fovea Simulation

Task: Find the sad smiley State 9

Bins Objects

1-D world

Fovea

Memory

Transformation

MindRACES, First Review Meeting, Lund, 11/01/2006 17

Simulated Vision: abstract 2-D Simulation

• Extended 1-D Simulation• Only the centered sensor can differentiate between

objects

MindRACES, First Review Meeting, Lund, 11/01/2006 18

Simulated Vision: 2-D Simulation

Original image with 2 objects

MindRACES, First Review Meeting, Lund, 11/01/2006 19

Simulated Vision: 2-D Simulation

Original image with 2 objects

Simulated fovea images

Center of the fovea

Step 1

MindRACES, First Review Meeting, Lund, 11/01/2006 20

Simulated Vision: 2-D Simulation

Original image with 2 objects

Simulated fovea images

Center of the fovea

Detected!!!

Step 1 Step n

MindRACES, First Review Meeting, Lund, 11/01/2006 21

Software Framework

Server

Robertino

•Debian/GNU Linux operating system•CAN bus interface •FireWire (IEEE 1394) interface•Video4Linux library•Fovea simulation

Client

Robosim

•Linux or Windows•Ogre framework•Fovea simulation•Simple physical and collision detection system

•Same interface as the real robot

Robomon

•Linux or Windows•Experiment management•Interface for learning algorithms•Full remote control of the robot/simulation

TCP/IP

Previously recorded data

MindRACES, First Review Meeting, Lund, 11/01/2006 24

Robertino – Client Software

Manual control of the robot

Learning control interface

Fovea image

MindRACES, First Review Meeting, Lund, 11/01/2006 25

Robertino - Overview

• Diameter: 40 cm• Height: 43 cm• Weight: 6.5 kg. • Holonomic three wheeled drive• PC-103 (industry standard)

with a 500MHz Intel Mobile-Pentium II processor on-board

• WLAN (IEEE 802.11a)• 2 cameras, used to simulate

the fovea• Actuators: the three wheels

and the simulated foveawww.openrobertino.org

MindRACES, First Review Meeting, Lund, 11/01/2006 26

Omnidirectional Camera

Web cam

Omnidirectional mirror

Image of an office environment, as seen by the robot

MindRACES, First Review Meeting, Lund, 11/01/2006 27

Image Transformation

• Used only for human convenience for navigation• Simple transformation algorithm

MindRACES, First Review Meeting, Lund, 11/01/2006 28

High Quality Camera

Imaging Source DFK21 AF04Image of an office environment, as seen by the robot

MindRACES, First Review Meeting, Lund, 11/01/2006 29

Robot Lab Environment

• Robot lab 550 cm x 280 cm• Robot can navigate through the world and move the fovea• Robot can be observed from a top view camera and local camera

MindRACES, First Review Meeting, Lund, 11/01/2006 30

Robot Lab Environment

Camera image Fovea data Composition

No video for his presentation!

MindRACES, First Review Meeting, Lund, 11/01/2006 31

3D Simulation of the Robot

• Environment design in 3D Studio MAX or G-MAX. • Simulation in Ogre3D• Same objects as in the real world• Shadows (optional)

MindRACES, First Review Meeting, Lund, 11/01/2006 32

Simulated vs. Real Vision (Fovea)

Simulation Real Robot

• Fovea transformation brings both world more together• Artificial noise simulates camera noise (optional)

MindRACES, First Review Meeting, Lund, 11/01/2006 33

Integration and Future Work

• Develop and evaluate anticipatory learning algorithms in the simulated environments (UW, ÖFAI); compare them also to non-anticipatory ones

• Share scenarios for attentive vision (LUCS, NBU)• Real robot with simulation as anticipation for surprise studies (CNR) • Simulation-based learning of anticipation (UW)• Share fovea simulation (IST)• Evaluate transfer of learned behavior to the real robot• Systematically increase the complexity of simulated and real

environments

• Possibly use a movable camera and zoom lens mounted on the robot