A Supervised Learning Architecture for Pose Recognition in a Social Robot
-
Upload
victor-pacheco -
Category
Technology
-
view
673 -
download
0
description
Transcript of A Supervised Learning Architecture for Pose Recognition in a Social Robot
A Supervised Learning Architecture for Pose Recognition in a Social
Robot
Víctor González Pacheco
domingo 17 de julio de 2011
Outline
pose_trainer
pose_labeler_bridge
Dataset
WekaModel
Learning Framework/Architecture
pose_classier
/skeleton
/classied_pose
OpenNI + pi_tracker
Kinect
/pose_labeled
asr_skill
etts_skill
pose_labeler_skill
RECOGNITION_RESULTSLABELED_POSE
pose_tellerSAY_TEXT
ROS AD
torsoOrient_w
<=0.956792 > 0.956792
torsoOrient_y
<=0.11679 > 0.11679
left_kneeOrient_w
<=0.609245 > 0.609245
STAND_TURNED_RIGHT
STAND_TURNED_FORWARD
STAND_TURNED_LEFTSTAND_TURNED_RIGHT
1. Introduction 2. System Description 3. Architecture
4. Evaluation 5. Conclusions
domingo 17 de julio de 2011
1. Introduction
domingo 17 de julio de 2011
How we interact with robots.
domingo 17 de julio de 2011
Children learn by “seeing and listening”
domingo 17 de julio de 2011
Gesture Recognition.
Fiction?
domingo 17 de julio de 2011
Reality is Better!
domingo 17 de julio de 2011
Outline
pose_trainer
pose_labeler_bridge
Dataset
WekaModel
Learning Framework/Architecture
pose_classier
/skeleton
/classied_pose
OpenNI + pi_tracker
Kinect
/pose_labeled
asr_skill
etts_skill
pose_labeler_skill
RECOGNITION_RESULTSLABELED_POSE
pose_tellerSAY_TEXT
ROS AD
torsoOrient_w
<=0.956792 > 0.956792
torsoOrient_y
<=0.11679 > 0.11679
left_kneeOrient_w
<=0.609245 > 0.609245
STAND_TURNED_RIGHT
STAND_TURNED_FORWARD
STAND_TURNED_LEFTSTAND_TURNED_RIGHT
1. Introduction 2. System Description 3. Architecture
4. Evaluation 5. Conclusions
domingo 17 de julio de 2011
2. Description of the System
domingo 17 de julio de 2011
5 ModulesOverview
High-level view
!"#
"#
$%&'()*)+', #-.'/0&'1 21%3'1)
4'))56'(75))%86
4',91*(,5856','8+ :;<
='>(>19?)'1@A,5%&(-&%'8+
=91/(719-'))91
B&%'8+(&%>151%')
C%)05&%D5+%98E2FG2(H'1-'7+%98
49>%&'(,58%70&5+%98
I53%65+%98J5)K('L'-0+%3'
#%,0&5+%98
Robot Operating SystemRadu Bogdan Rusu
Maggie KinectROS
Voice System Weka
domingo 17 de julio de 2011
Maggie
domingo 17 de julio de 2011
Maggie
Social Robot
HRI Research
RoboticsLab, UC3M
domingo 17 de julio de 2011
Maggie
Bumpers
Laser
Kinect
Touch Sensors
Camera
Touch Sensor
2 DoF Neck
Touch Sensors
2 DoF Base
RFID Reader
RFID Reader
1 DoF Arms
domingo 17 de julio de 2011
AD Architecture
Actuators
Short Term Memory
Long Term Memory
Sensors
Delib. Skill n
Delib. Skill 1
Delib. Skill 2
Autom. Skill n
Autom. Skill 1
Autom. Skill 2
events
domingo 17 de julio de 2011
5 ModulesOverview
High-level view
!"#
"#
$%&'()*)+', #-.'/0&'1 21%3'1)
4'))56'(75))%86
4',91*(,5856','8+ :;<
='>(>19?)'1@A,5%&(-&%'8+
=91/(719-'))91
B&%'8+(&%>151%')
C%)05&%D5+%98E2FG2(H'1-'7+%98
49>%&'(,58%70&5+%98
I53%65+%98J5)K('L'-0+%3'
#%,0&5+%98
Robot Operating SystemRadu Bogdan Rusu
Maggie KinectROS
Voice System Weka
domingo 17 de julio de 2011
ROS
Retrieved from: http://www.ros.org/wiki/Events/CoTeSys-ROS-School
Overview
High-level view
!"#
"#
$%&'()*)+', #-.'/0&'1 21%3'1)
4'))56'(75))%86
4',91*(,5856','8+ :;<
='>(>19?)'1@A,5%&(-&%'8+
=91/(719-'))91
B&%'8+(&%>151%')
C%)05&%D5+%98E2FG2(H'1-'7+%98
49>%&'(,58%70&5+%98
I53%65+%98J5)K('L'-0+%3'
#%,0&5+%98
Robot Operating SystemRadu Bogdan Rusu
domingo 17 de julio de 2011
ROS
Robot Operating System
Retrieved from: http://www.ros.org/wiki/Events/CoTeSys-ROS-School
Overview
High-level view
!"#
"#
$%&'()*)+', #-.'/0&'1 21%3'1)
4'))56'(75))%86
4',91*(,5856','8+ :;<
='>(>19?)'1@A,5%&(-&%'8+
=91/(719-'))91
B&%'8+(&%>151%')
C%)05&%D5+%98E2FG2(H'1-'7+%98
49>%&'(,58%70&5+%98
I53%65+%98J5)K('L'-0+%3'
#%,0&5+%98
Robot Operating SystemRadu Bogdan Rusu
domingo 17 de julio de 2011
ROS
Robot Operating System
C++, Python, Java, Octave, Lisp.
Ubuntu
(Mac and Windows in experimental phase)
Retrieved from: http://www.ros.org/wiki/Events/CoTeSys-ROS-School
Overview
High-level view
!"#
"#
$%&'()*)+', #-.'/0&'1 21%3'1)
4'))56'(75))%86
4',91*(,5856','8+ :;<
='>(>19?)'1@A,5%&(-&%'8+
=91/(719-'))91
B&%'8+(&%>151%')
C%)05&%D5+%98E2FG2(H'1-'7+%98
49>%&'(,58%70&5+%98
I53%65+%98J5)K('L'-0+%3'
#%,0&5+%98
Robot Operating SystemRadu Bogdan Rusu
domingo 17 de julio de 2011
Robot Operating System
Node Node
/topicPublication Subscription
domingo 17 de julio de 2011
5 ModulesOverview
High-level view
!"#
"#
$%&'()*)+', #-.'/0&'1 21%3'1)
4'))56'(75))%86
4',91*(,5856','8+ :;<
='>(>19?)'1@A,5%&(-&%'8+
=91/(719-'))91
B&%'8+(&%>151%')
C%)05&%D5+%98E2FG2(H'1-'7+%98
49>%&'(,58%70&5+%98
I53%65+%98J5)K('L'-0+%3'
#%,0&5+%98
Robot Operating SystemRadu Bogdan Rusu
Maggie KinectROS
Voice System Weka
domingo 17 de julio de 2011
Kinect RGB-D Sensor
domingo 17 de julio de 2011
Light Coding Technology
domingo 17 de julio de 2011
domingo 17 de julio de 2011
domingo 17 de julio de 2011
5 ModulesOverview
High-level view
!"#
"#
$%&'()*)+', #-.'/0&'1 21%3'1)
4'))56'(75))%86
4',91*(,5856','8+ :;<
='>(>19?)'1@A,5%&(-&%'8+
=91/(719-'))91
B&%'8+(&%>151%')
C%)05&%D5+%98E2FG2(H'1-'7+%98
49>%&'(,58%70&5+%98
I53%65+%98J5)K('L'-0+%3'
#%,0&5+%98
Robot Operating SystemRadu Bogdan Rusu
Maggie KinectROS
Voice System Weka
domingo 17 de julio de 2011
Automatic Speech Recognition (ASR)
HelloMicrophone
ASR “Hello”
domingo 17 de julio de 2011
Text To Speech (TTS)
Hello
TTS “Hello”
Speaker
Speaker
domingo 17 de julio de 2011
5 ModulesOverview
High-level view
!"#
"#
$%&'()*)+', #-.'/0&'1 21%3'1)
4'))56'(75))%86
4',91*(,5856','8+ :;<
='>(>19?)'1@A,5%&(-&%'8+
=91/(719-'))91
B&%'8+(&%>151%')
C%)05&%D5+%98E2FG2(H'1-'7+%98
49>%&'(,58%70&5+%98
I53%65+%98J5)K('L'-0+%3'
#%,0&5+%98
Robot Operating SystemRadu Bogdan Rusu
Maggie KinectROS
Voice System Weka
domingo 17 de julio de 2011
Weka
Machine Learning Framework
Classification, Clustering, etc.
GPL
Java API
Decision Tree: J48 (C 4.5)
(Waikato Environment for Knowledge Analysis)
domingo 17 de julio de 2011
Outline
pose_trainer
pose_labeler_bridge
Dataset
WekaModel
Learning Framework/Architecture
pose_classier
/skeleton
/classied_pose
OpenNI + pi_tracker
Kinect
/pose_labeled
asr_skill
etts_skill
pose_labeler_skill
RECOGNITION_RESULTSLABELED_POSE
pose_tellerSAY_TEXT
ROS AD
torsoOrient_w
<=0.956792 > 0.956792
torsoOrient_y
<=0.11679 > 0.11679
left_kneeOrient_w
<=0.609245 > 0.609245
STAND_TURNED_RIGHT
STAND_TURNED_FORWARD
STAND_TURNED_LEFTSTAND_TURNED_RIGHT
1. Introduction 2. System Description 3. Architecture
4. Evaluation 5. Conclusions
domingo 17 de julio de 2011
3. Architecture
pose_trainer
pose_labeler_bridge
Dataset
WekaModel
Learning Framework/Architecture
pose_classier
/skeleton
/classied_pose
OpenNI + pi_tracker
Kinect
/pose_labeled
asr_skill
etts_skill
pose_labeler_skill
RECOGNITION_RESULTSLABELED_POSE
pose_tellerSAY_TEXT
ROS AD
domingo 17 de julio de 2011
System Overview
MODEL
Dataset
Kinect
Microphone ASRMachine Learning
Framework
I'm seated
Kinect
TTS
Pose Classifier
Speaker
What's my pose?
MODEL
You're seated
domingo 17 de julio de 2011
System Overview
MODEL
Dataset
Kinect
Microphone ASRMachine Learning
Framework
I'm seated
Kinect
TTS
Pose Classifier
Speaker
What's my pose?
MODEL
You're seated
domingo 17 de julio de 2011
System Overview
MODEL
Dataset
Kinect
Microphone ASRMachine Learning
Framework
I'm seated
Kinect
TTS
Pose Classifier
Speaker
What's my pose?
MODEL
You're seated
domingo 17 de julio de 2011
System Overview
MODEL
Dataset
Kinect
Microphone ASRMachine Learning
Framework
I'm seated
Kinect
TTS
Pose Classifier
Speaker
What's my pose?
MODEL
You're seated
domingo 17 de julio de 2011
domingo 17 de julio de 2011
pose_trainer
pose_labeler_bridge
Dataset
WekaModel
Learning Framework/Architecture
pose_classier
/skeleton
/classied_pose
OpenNI + pi_tracker
Kinect
/pose_labeled
asr_skill
etts_skill
pose_labeler_skill
RECOGNITION_RESULTSLABELED_POSE
pose_tellerSAY_TEXT
ROS AD
domingo 17 de julio de 2011
pose_trainer
pose_labeler_bridge
Dataset
WekaModel
Learning Framework/Architecture
pose_classier
/skeleton
/classied_pose
OpenNI + pi_tracker
Kinect
/pose_labeled
asr_skill
etts_skill
pose_labeler_skill
RECOGNITION_RESULTSLABELED_POSE
pose_tellerSAY_TEXT
ROS AD
domingo 17 de julio de 2011
pose_trainer
pose_labeler_bridge
Dataset
WekaModel
Learning Framework/Architecture
pose_classier
/skeleton
/classied_pose
OpenNI + pi_tracker
Kinect
/pose_labeled
asr_skill
etts_skill
pose_labeler_skill
RECOGNITION_RESULTSLABELED_POSE
pose_tellerSAY_TEXT
ROS AD
domingo 17 de julio de 2011
pose_trainer
pose_labeler_bridge
Dataset
WekaModel
Learning Framework/Architecture
pose_classier
/skeleton
/classied_pose
OpenNI + pi_tracker
Kinect
/pose_labeled
asr_skill
etts_skill
pose_labeler_skill
RECOGNITION_RESULTSLABELED_POSE
pose_tellerSAY_TEXT
ROS AD
domingo 17 de julio de 2011
pose_trainer
pose_labeler_bridge
Dataset
WekaModel
Learning Framework/Architecture
pose_classier
/skeleton
/classied_pose
OpenNI + pi_tracker
Kinect
/pose_labeled
asr_skill
etts_skill
pose_labeler_skill
RECOGNITION_RESULTSLABELED_POSE
pose_tellerSAY_TEXT
ROS AD
domingo 17 de julio de 2011
pose_trainer
pose_labeler_bridge
Dataset
WekaModel
Learning Framework/Architecture
pose_classier
/skeleton
/classied_pose
OpenNI + pi_tracker
Kinect
/pose_labeled
asr_skill
etts_skill
pose_labeler_skill
RECOGNITION_RESULTSLABELED_POSE
pose_tellerSAY_TEXT
ROS AD
domingo 17 de julio de 2011
pose_trainer
pose_labeler_bridge
Dataset
WekaModel
Learning Framework/Architecture
pose_classier
/skeleton
/classied_pose
OpenNI + pi_tracker
Kinect
/pose_labeled
asr_skill
etts_skill
pose_labeler_skill
RECOGNITION_RESULTSLABELED_POSE
pose_tellerSAY_TEXT
ROS AD
domingo 17 de julio de 2011
pose_trainer
pose_labeler_bridge
Dataset
WekaModel
Learning Framework/Architecture
pose_classier
/skeleton
/classied_pose
OpenNI + pi_tracker
Kinect
/pose_labeled
asr_skill
etts_skill
pose_labeler_skill
RECOGNITION_RESULTSLABELED_POSE
pose_tellerSAY_TEXT
ROS AD
domingo 17 de julio de 2011
pose_trainer
pose_labeler_bridge
Dataset
WekaModel
Learning Framework/Architecture
pose_classier
/skeleton
/classied_pose
OpenNI + pi_tracker
Kinect
/pose_labeled
asr_skill
etts_skill
pose_labeler_skill
RECOGNITION_RESULTSLABELED_POSE
pose_tellerSAY_TEXT
ROS AD
domingo 17 de julio de 2011
pose_trainer
pose_labeler_bridge
Dataset
WekaModel
Learning Framework/Architecture
pose_classier
/skeleton
/classied_pose
OpenNI + pi_tracker
Kinect
/pose_labeled
asr_skill
etts_skill
pose_labeler_skill
RECOGNITION_RESULTSLABELED_POSE
pose_tellerSAY_TEXT
ROS AD
domingo 17 de julio de 2011
pose_trainer
pose_labeler_bridge
Dataset
WekaModel
Learning Framework/Architecture
pose_classier
/skeleton
/classied_pose
OpenNI + pi_tracker
Kinect
/pose_labeled
asr_skill
etts_skill
pose_labeler_skill
RECOGNITION_RESULTSLABELED_POSE
pose_tellerSAY_TEXT
ROS AD
domingo 17 de julio de 2011
Outline
pose_trainer
pose_labeler_bridge
Dataset
WekaModel
Learning Framework/Architecture
pose_classier
/skeleton
/classied_pose
OpenNI + pi_tracker
Kinect
/pose_labeled
asr_skill
etts_skill
pose_labeler_skill
RECOGNITION_RESULTSLABELED_POSE
pose_tellerSAY_TEXT
ROS AD
torsoOrient_w
<=0.956792 > 0.956792
torsoOrient_y
<=0.11679 > 0.11679
left_kneeOrient_w
<=0.609245 > 0.609245
STAND_TURNED_RIGHT
STAND_TURNED_FORWARD
STAND_TURNED_LEFTSTAND_TURNED_RIGHT
1. Introduction 2. System Description 3. Architecture
4. Evaluation 5. Conclusions
domingo 17 de julio de 2011
4. Evaluation
torsoOrient_w
<=0.956792 > 0.956792
torsoOrient_y
<=0.11679 > 0.11679
left_kneeOrient_w
<=0.609245 > 0.609245
STAND_TURNED_RIGHT
STAND_TURNED_FORWARD
STAND_TURNED_LEFTSTAND_TURNED_RIGHT
domingo 17 de julio de 2011
Demo Video
domingo 17 de julio de 2011
Experiment Setup
180 cm240 cm
180
cm RobotU
ser
User Area
Kinect Horizontal Field of View
domingo 17 de julio de 2011
Datasets and models
No Expert Knowledge
Expert Knowledge
2 TrainingDatasets
Moving during training
2 TrainingDatasets
Not moving during training
12 Users Tested the four models
torsoOrient_w
<=0.956792 > 0.956792
torsoOrient_y
<=0.11679 > 0.11679
left_kneeOrient_w
<=0.609245 > 0.609245
STAND_TURNED_RIGHT
STAND_TURNED_FORWARD
STAND_TURNED_LEFTSTAND_TURNED_RIGHT
right_kneeOrient_z
<=0.196633 > 0.196633
right_kneePos_x
<=0.100138 > 0.100138
STAND_TURNED_RIGHT
STAND_TURNED_FORWARDSTAND_TURNED_LEFT
right_kneeOrient_z
<=0.196633 > 0.196633
torsoOrient_w
<=0.699489 > 0.699489
STAND_TURNED_RIGHT
STAND_TURNED_FORWARDSTAND_TURNED_LEFT
torsoOrient_w
<=0.956792 > 0.956792
torsoOrient_y
<=0.11679 > 0.11679
left_kneeOrient_w
<=0.609245 > 0.609245
STAND_TURNED_RIGHT
STAND_TURNED_FORWARD
STAND_TURNED_LEFTSTAND_TURNED_RIGHT
domingo 17 de julio de 2011
Models of “moving dataset” were the same
No Expert Knowledge
Expert Knowledge
2 TrainingDatasets
Moving during training
2 TrainingDatasets
Not moving during training
12 Users Tested the four models
right_kneeOrient_z
<=0.196633 > 0.196633
right_kneePos_x
<=0.100138 > 0.100138
STAND_TURNED_RIGHT
STAND_TURNED_FORWARDSTAND_TURNED_LEFT
right_kneeOrient_z
<=0.196633 > 0.196633
torsoOrient_w
<=0.699489 > 0.699489
STAND_TURNED_RIGHT
STAND_TURNED_FORWARDSTAND_TURNED_LEFT
torsoOrient_w
<=0.956792 > 0.956792
torsoOrient_y
<=0.11679 > 0.11679
left_kneeOrient_w
<=0.609245 > 0.609245
STAND_TURNED_RIGHT
STAND_TURNED_FORWARD
STAND_TURNED_LEFTSTAND_TURNED_RIGHT
domingo 17 de julio de 2011
torsoOrient_w
<=0.956792 > 0.956792
torsoOrient_y
<=0.11679 > 0.11679
left_kneeOrient_w
<=0.609245 > 0.609245
STAND_TURNED_RIGHT
STAND_TURNED_FORWARD
STAND_TURNED_LEFTSTAND_TURNED_RIGHT
Moving.w/ & wo/ Expert
Knowledge
domingo 17 de julio de 2011
Models of “moving dataset” were the same
No Expert Knowledge
Expert Knowledge
2 TrainingDatasets
Moving during training
2 TrainingDatasets
Not moving during training
12 Users Tested the four models
right_kneeOrient_z
<=0.196633 > 0.196633
right_kneePos_x
<=0.100138 > 0.100138
STAND_TURNED_RIGHT
STAND_TURNED_FORWARDSTAND_TURNED_LEFT
right_kneeOrient_z
<=0.196633 > 0.196633
torsoOrient_w
<=0.699489 > 0.699489
STAND_TURNED_RIGHT
STAND_TURNED_FORWARDSTAND_TURNED_LEFT
torsoOrient_w
<=0.956792 > 0.956792
torsoOrient_y
<=0.11679 > 0.11679
left_kneeOrient_w
<=0.609245 > 0.609245
STAND_TURNED_RIGHT
STAND_TURNED_FORWARD
STAND_TURNED_LEFTSTAND_TURNED_RIGHT
domingo 17 de julio de 2011
right_kneeOrient_z
<=0.196633 > 0.196633
right_kneePos_x
<=0.100138 > 0.100138
STAND_TURNED_RIGHT
STAND_TURNED_FORWARDSTAND_TURNED_LEFT
Not Moving wo/ Expert Knowledge
(Position + Orientation)
domingo 17 de julio de 2011
Models of “moving dataset” were the same
No Expert Knowledge
Expert Knowledge
2 TrainingDatasets
Moving during training
2 TrainingDatasets
Not moving during training
12 Users Tested the four models
right_kneeOrient_z
<=0.196633 > 0.196633
right_kneePos_x
<=0.100138 > 0.100138
STAND_TURNED_RIGHT
STAND_TURNED_FORWARDSTAND_TURNED_LEFT
right_kneeOrient_z
<=0.196633 > 0.196633
torsoOrient_w
<=0.699489 > 0.699489
STAND_TURNED_RIGHT
STAND_TURNED_FORWARDSTAND_TURNED_LEFT
torsoOrient_w
<=0.956792 > 0.956792
torsoOrient_y
<=0.11679 > 0.11679
left_kneeOrient_w
<=0.609245 > 0.609245
STAND_TURNED_RIGHT
STAND_TURNED_FORWARD
STAND_TURNED_LEFTSTAND_TURNED_RIGHT
domingo 17 de julio de 2011
Not Moving w/ Expert Knowledge
(Orientation)right_kneeOrient_z
<=0.196633 > 0.196633
torsoOrient_w
<=0.699489 > 0.699489
STAND_TURNED_RIGHT
STAND_TURNED_FORWARDSTAND_TURNED_LEFT
domingo 17 de julio de 2011
Models of “moving dataset” were the same
No Expert Knowledge
Expert Knowledge
2 TrainingDatasets
Moving during training
2 TrainingDatasets
Not moving during training
12 Users Tested the four models
right_kneeOrient_z
<=0.196633 > 0.196633
right_kneePos_x
<=0.100138 > 0.100138
STAND_TURNED_RIGHT
STAND_TURNED_FORWARDSTAND_TURNED_LEFT
right_kneeOrient_z
<=0.196633 > 0.196633
torsoOrient_w
<=0.699489 > 0.699489
STAND_TURNED_RIGHT
STAND_TURNED_FORWARDSTAND_TURNED_LEFT
torsoOrient_w
<=0.956792 > 0.956792
torsoOrient_y
<=0.11679 > 0.11679
left_kneeOrient_w
<=0.609245 > 0.609245
STAND_TURNED_RIGHT
STAND_TURNED_FORWARD
STAND_TURNED_LEFTSTAND_TURNED_RIGHT
domingo 17 de julio de 2011
Results
ModelModel Correctly Classified (%)
False Positives (%)
Moving TrainingMoving Training
Not MovingTraining
wo/ Expert Knowledge
(Pos + Orient)Not Moving
Trainingw/ Expert Knowledge
(Orient)
92,6 3,9
56,3 21,3
70,0 14,1
domingo 17 de julio de 2011
Outline
pose_trainer
pose_labeler_bridge
Dataset
WekaModel
Learning Framework/Architecture
pose_classier
/skeleton
/classied_pose
OpenNI + pi_tracker
Kinect
/pose_labeled
asr_skill
etts_skill
pose_labeler_skill
RECOGNITION_RESULTSLABELED_POSE
pose_tellerSAY_TEXT
ROS AD
torsoOrient_w
<=0.956792 > 0.956792
torsoOrient_y
<=0.11679 > 0.11679
left_kneeOrient_w
<=0.609245 > 0.609245
STAND_TURNED_RIGHT
STAND_TURNED_FORWARD
STAND_TURNED_LEFTSTAND_TURNED_RIGHT
1. Introduction 2. System Description 3. Architecture
4. Evaluation 5. Conclusions
domingo 17 de julio de 2011
5. Conclusions
domingo 17 de julio de 2011
Conclusions
• Architecture works
• 92% Correctly classified instances
• Now the robot is able to understand contextual information
domingo 17 de julio de 2011
Future Work
• Gesture Recognition
• Activity Recognition
• Active Learning
domingo 17 de julio de 2011
Questions?Thanks for your attention.
domingo 17 de julio de 2011