From Vision to Actions - Towards Adaptive & Autonomous Humanoid Robots [PhD Defense]
-
Upload
juxi-leitner -
Category
Science
-
view
353 -
download
3
description
Transcript of From Vision to Actions - Towards Adaptive & Autonomous Humanoid Robots [PhD Defense]
From Vision to Actions towards adaptive & autonomous robots
Jürgen ‘Juxi’ Leitner i'stituto dalle molle di studi sull’intelligenza artificiale
idsia / supsi / usi
humanoidiCub
overview
a novel way of object segmentation (CGPIP)
learning and teaching spatial perception
integration action-perception sidereactive reaching/graspingimproving perception with (inter-)actions
!
perceptionvisual
thanks to G. Metta and IIT for this picture
!
objectsdetecting
!
objectsdetecting
Harding, Leitner, Schmidhuber, 2013Leitner et al., ICDL 2012, IJARS 2012, BICA 2012, CEC 2013
+ min dilate avg INP INP INP diff
using building blocksOpenCV
+ min dilate avg INP INP INP thresh+ min dilate avg INP INP INP blur+ min dilate avg INP INP INP normalize+ min dilate avg INP INP INP input
icVision
! icImage* BlueCupFilter::runFilter() { icImage* node43 = InputImages[4]; icImage* node49 = node43->LocalAvg(15); ! icImage* out = node49->threshold(81.532f); return out; }
framework
embedding domain knowledge
+ min dilate avg INP INP INP OpenCV functions
full images
cartesian genetic programming
+ min dilate avg INP INP INP
Harding, Leitner, Schmidhuber, 2013Leitner et al., ICDL 2012, IJARS 2012, BICA 2012, CEC 2013
learningapproach
[Matthews, 1975]
TP… true positive, TN .. true negative!FP… false positive, FN .. false negative
learningapproach
detection
! icImage GreenTeaBoxDetector::runFilter() { icImage node0 = InputImages[6]; icImage node1 = InputImages[1]; icImage node2 = node0.absdiff(node1); icImage node5 = node2.SmoothBilateral(11); icImage node12 = InputImages[0]; icImage node16 = node12.Sqrt(); icImage node33 = node16.erode(6); icImage node34 = node33.log(); icImage node36 = node34.min(node5); icImage node49 = node36.Normalize(); ! //cleanup ... icImage out = node49.threshold(230.7218f); return out; }
detect
detect
detection
approachcgp
[Leitner et al, iSAIRAS 2012]
approachsupervised learning
BUT
clusteringfeature
saliencymap
Autonomous Approach
[Leitner et al, ICDL/EpiRob 2012]
presegmentation
localisation approachescurrent
[P2,1] * F * [P1,1]' = 0
!
learningspatial perception
trainingset
9DOF!
iCubpositio
n in the frame!
2/6 per eye
Carte
sian!
Coor
dinate
s
setuplearning
trainingset
9DOF!
iCubpositio
n in the frame!
2/6 per eye
Carte
sian!
Coor
dinate
s
.
.
.~1000
spatial perception neural network
.!.!.
9DO
F!iC
ubpo
sitio
n in
the
imag
e!2/
6 pe
r eye
Cart
esian!
Coor
dina
tes
!fu
lly c
onne
cted!
!fu
lly c
onne
cted!
.!.!.
results ANN
-700
-600
-500
-400
-300
-200
-100
0
100
200
300
400
0 200 400 600 800 1000
Y (
mm
)
Sample Index
PredictedExpected
100
150
200
250
300
350
400
450
500
550
600
0 200 400 600 800 1000
X (
mm
)
Sample Index
-1300
-1200
-1100
-1000
-900
-800
-700
-600
-500
-400
-300
-200
0 200 400 600 800 1000
Z (
mm
)
Sample Index
-700
-600
-500
-400
-300
-200
-100
0
100
200
300
400
0 200 400 600 800 1000
Y (
mm
)
Sample Index
PredictedExpected
100
150
200
250
300
350
400
450
500
550
600
0 200 400 600 800 1000
X (
mm
)
Sample Index
-1300
-1200
-1100
-1000
-900
-800
-700
-600
-500
-400
-300
-200
0 200 400 600 800 1000
Z (
mm
)
Sample Index
[Leitner et al, IJCNN 2013]
results
[Leitner et al, IJARS 2012]
x"="17.81"−"0.0191"v1"+"0.1527"v4"+"0.1378"v7"+"0.0111"v10""""""−"0.0296"v11"−"0.1207"v12"!
y"="1.1242"+"0.1296"v10"+"0.1156"v8"+"0.0170"v0
GPv0 … left camera x coordinate,v1 … left camera y coordinate,
v2 … right camera x coordinate,v3 … right camera y coordinate,
v4-6 … neckv7-9 … eyes v10-12 … torso
simplified caseon the table
MoBeE[Frank, Leitner et al., ICINCO, 2012]
MoBeEframework [Frank, Leitner et al., ICINCO, 2012]
[Frank, Leitner et al., ICINCO, 2012]
generationmotion
Shak
ey 2
013
Win
ner
MoBeEv2[Frank, Leitner et al. 2012, 2013]
hand/armop-space forcing
CSWorld
CSHand
CSR/CSL
[Leitner et al, in prep]
-
repertoireaction
[Leitner et al, in prep]
operationtele
hand/armop-space forcing
CSWorld
CSHand
CSR/CSL
[Leitner et al, in prep]
- CSHand’
teleoperation
coordinationhand-eye
model
http://robotics.idsia.ch/
manipulation for improved perception
manipulation actions
extracting information
improveddetection
detection
! icImage* BlueCupFilter::runFilter() { icImage* node43 = InputImages[4]; icImage* node49 = node43->LocalAvg(15); ! icImage* out = node49->threshold(81.532f); return out; }
detection
! icImage* BlueCupFilter::runFilter() { icImage* node0 = InputImages[4].Exp(); icImage* node5 = InputImages[0]; icImage* node16 = node0->Gabor(-8,14,1,13); icImage* node17 = InputImages[4]->LocalAvg(6); icImage* node18 = node16->Laplace(5); icImage* node19 = node5->Sobel(13,9); icImage* node24 = node17->Erode(5); icImage* node28 = node19->Min(node18); icImage* node29 = node28->Min(node24); icImage* node41 = node29->LocalAvg(7); icImage* node49 = node41->LocalMax(7); ! icImage* out = node49->threshold(68.032f); return out; }
resulting
conclusions
a novel way of object segmentation
learning and teaching spatial perception
integration action-perception sidereactive reaching/graspingimproving perception with (inter-)actions
S. Harding!M. Frank!
A. Förster!M. Stollenga!
L. Pape!
http://dilbert.com/strips/comic/2013-10-24/
thanks to
for listeningthank you
[email protected] !http://Juxi.net/projects
1/2referencesReactive Reaching and Grasping on a Humanoid: Towards Closing the Action-Perception Loop on the iCub. Jürgen Leitner, Mikhail Frank, Alexander Förster, Jürgen Schmidhuber. ICINCO 2014. !Improving Robot Vision Models for Object Detection Through Interaction. Jürgen Leitner, Alexander Förster, Jürgen Schmidhuber. WCCI - IJCNN 2014. !Teleoperation of a 7 DOF Humanoid Robot Arm Using Human Arm Accelerations and EMG Signals.. J Leitner, M Luciw, A Förster, J Schmidhuber. iSAIRAS 2014. !Curiosity driven reinforcement learning for motion planning on humanoids. Mikhail Frank, Jürgen Leitner, Alexander Förster, Jürgen Schmidhuber. Frontiers in Neurorobotics, 7:25, 2014. !Task-Relevant Roadmaps: A Framework for Humanoid Motion Planning. M Stollenga, L Pape, M Frank, J Leitner, A Förster, J Schmidhuber. IROS 2013. !ALife in Humanoids: Developing a Framework to Employ Artificial Life Techniques for High-Level Perception and Cognition Tasks on Humanoid Robots. Jürgen Leitner, Simon Harding, Mikhail Frank, Alexander Förster, Jürgen Schmidhuber. Alife WS @ ECAL 2013. !Artificial Neural Networks For Spatial Perception: Towards Visual Object Localisation in Humanoid Robots. Jürgen Leitner, Simon Harding, Mikhail Frank, Alexander Förster, Jürgen Schmidhuber. IJCNN 2013. !Learning Visual Object Detection and Localisation Using icVision. Jürgen Leitner, Simon Harding, Pramod Chandrashekhariah, Mikhail Frank, Alexander Förster, Jochen Triesch, Jürgen Schmidhuber. Biologically Inspired Cognitive Architectures, Vol. 5, 2013. !Humanoid Learns to Detect Its Own Hands. Jürgen Leitner, Simon Harding, Mikhail Frank, Alexander Förster, Jürgen Schmidhuber. CEC 2013. !Cartesian Genetic Programming for Image Processing (CGP-IP). Simon Harding, Jürgen Leitner, Jürgen Schmidhuber. In Genetic Programming Theory and Practice X, pp 31-44. ISBN: 978-1-4614-6845-5. Springer, 2013.
http://Juxi.net/papers
referencesCartesian Genetic Programming for Image Processing and Robot Vision. J. Leitner, S. Harding, A. Förster, J. Schmidhuber. IEEE RAS Summer School on "Robot Vision and Applications". Santiago, Chile. December 2012. !Learning Spatial Object Localization from Vision on a Humanoid Robot. Jürgen Leitner, Simon Harding, Mikhail Frank, Alexander Förster, Jürgen Schmidhuber. International Journal of Advanced Robotic Systems, Vol. 9, 2012. !Autonomous Learning Of Robust Visual Object Detection And Identification On A Humanoid. J. Leitner, P. Chandrashekhariah, S. Harding, M. Frank, G. Spina, A. Förster, J. Triesch, J. Schmidhuber. ICDL/EpiRob 2012. Paper of Excellence Award !An Integrated, Modular Framework for Computer Vision and Cognitive Robotics Research (icVision). Jürgen Leitner, Simon Harding, Mikhail Frank, Alexander Förster, Jürgen Schmidhuber. BICA 2012. !Transferring Spatial Perception Between Robots Operating In A Shared Workspace. J Leitner, S Harding, M Frank, A Förster, J Schmidhuber. IROS 2012. !Towards Spatial Perception: Learning to Locate Objects From Vision. J Leitner, S Harding, M Frank, A Förster, J Schmidhuber. RobotDoC-PhD 2012. !Mars Terrain Image Classification using Cartesian Genetic Programming. J Leitner, S Harding, A Förster, J Schmidhuber. i-SAIRAS 2012. !The Modular Behavioral Environment for Humanoids & other Robots (MoBeE). Mikhail Frank, Jürgen Leitner, Marijn Stollenga, Gregor Kaufmann, Simon Harding, Alexander Förster, Jürgen Schmidhuber. ICINCO 2012. !MT-CGP: Mixed Type Cartesian Genetic Programming. S. Harding, V. Graziano, J. Leitner, Jürgen Schmidhuber. GECCO 2012. !icVision: A Modular Vision System for Cognitive Robotics Research. J. Leitner, S. Harding, M. Frank, A. Förster, J. Schmidhuber. CogSys 2012. !Artificial Curiosity for Autonomous Space Exploration. Vincent Graziano, Tobias Glasmachers, Tom Schaul, Leo Pape, Giuseppe Cuccu, Jürgen Leitner and Jürgen Schmidhuber. Acta Futura, 4, pp. 41-52, 2011.
http://Juxi.net/papers2/2