A Supervised Learning Architecture for Pose Recognition in a Social Robot

65
A Supervised Learning Architecture for Pose Recognition in a Social Robot Víctor González Pacheco domingo 17 de julio de 2011

description

A main activity of Social Robots is to interact with people. To do that, the robot must be able to understand what the user is saying or doing. This document presents a supervised learning architecture to enable a social robot to recognise human poses. The architecture is trained using data obtained from a depth camera that allows the creation of a kinematic model of the user. The user labels each set of poses by telling it directly to the robot, which identifies these labels with an Automatic Speech Recognition System (ASR). The architecture is evaluated with two different datasets where the quality of the training examples varies. In both datasets, a user trains the classifier to recognise three different poses. The learned classifiers are evaluated against twelve different users demonstrating high accuracy and robustness when representative examples are provided in the training phase. Using this architecture in a social robot might improve the quality of the human-robot interaction since the robot is able to detect non-verbal cues from the user, making the robot more aware of the interaction context.

Transcript of A Supervised Learning Architecture for Pose Recognition in a Social Robot

Page 1: A Supervised Learning Architecture for Pose Recognition in a Social Robot

A Supervised Learning Architecture for Pose Recognition in a Social

Robot

Víctor González Pacheco

domingo 17 de julio de 2011

Page 2: A Supervised Learning Architecture for Pose Recognition in a Social Robot

Outline

pose_trainer

pose_labeler_bridge

Dataset

WekaModel

Learning Framework/Architecture

pose_classier

/skeleton

/classied_pose

OpenNI + pi_tracker

Kinect

/pose_labeled

asr_skill

etts_skill

pose_labeler_skill

RECOGNITION_RESULTSLABELED_POSE

pose_tellerSAY_TEXT

ROS AD

torsoOrient_w

<=0.956792 > 0.956792

torsoOrient_y

<=0.11679 > 0.11679

left_kneeOrient_w

<=0.609245 > 0.609245

STAND_TURNED_RIGHT

STAND_TURNED_FORWARD

STAND_TURNED_LEFTSTAND_TURNED_RIGHT

1. Introduction 2. System Description 3. Architecture

4. Evaluation 5. Conclusions

domingo 17 de julio de 2011

Page 3: A Supervised Learning Architecture for Pose Recognition in a Social Robot

1. Introduction

domingo 17 de julio de 2011

Page 4: A Supervised Learning Architecture for Pose Recognition in a Social Robot

How we interact with robots.

domingo 17 de julio de 2011

Page 5: A Supervised Learning Architecture for Pose Recognition in a Social Robot

Children learn by “seeing and listening”

domingo 17 de julio de 2011

Page 6: A Supervised Learning Architecture for Pose Recognition in a Social Robot

Gesture Recognition.

Fiction?

domingo 17 de julio de 2011

Page 7: A Supervised Learning Architecture for Pose Recognition in a Social Robot

Reality is Better!

domingo 17 de julio de 2011

Page 8: A Supervised Learning Architecture for Pose Recognition in a Social Robot

Outline

pose_trainer

pose_labeler_bridge

Dataset

WekaModel

Learning Framework/Architecture

pose_classier

/skeleton

/classied_pose

OpenNI + pi_tracker

Kinect

/pose_labeled

asr_skill

etts_skill

pose_labeler_skill

RECOGNITION_RESULTSLABELED_POSE

pose_tellerSAY_TEXT

ROS AD

torsoOrient_w

<=0.956792 > 0.956792

torsoOrient_y

<=0.11679 > 0.11679

left_kneeOrient_w

<=0.609245 > 0.609245

STAND_TURNED_RIGHT

STAND_TURNED_FORWARD

STAND_TURNED_LEFTSTAND_TURNED_RIGHT

1. Introduction 2. System Description 3. Architecture

4. Evaluation 5. Conclusions

domingo 17 de julio de 2011

Page 9: A Supervised Learning Architecture for Pose Recognition in a Social Robot

2. Description of the System

domingo 17 de julio de 2011

Page 10: A Supervised Learning Architecture for Pose Recognition in a Social Robot

5 ModulesOverview

High-level view

!"#

"#

$%&'()*)+', #-.'/0&'1 21%3'1)

4'))56'(75))%86

4',91*(,5856','8+ :;<

='>(>19?)'1@A,5%&(-&%'8+

=91/(719-'))91

B&%'8+(&%>151%')

C%)05&%D5+%98E2FG2(H'1-'7+%98

49>%&'(,58%70&5+%98

I53%65+%98J5)K('L'-0+%3'

#%,0&5+%98

Robot Operating SystemRadu Bogdan Rusu

Maggie KinectROS

Voice System Weka

domingo 17 de julio de 2011

Page 11: A Supervised Learning Architecture for Pose Recognition in a Social Robot

Maggie

domingo 17 de julio de 2011

Page 12: A Supervised Learning Architecture for Pose Recognition in a Social Robot

Maggie

Social Robot

HRI Research

RoboticsLab, UC3M

domingo 17 de julio de 2011

Page 13: A Supervised Learning Architecture for Pose Recognition in a Social Robot

Maggie

Bumpers

Laser

Kinect

Touch Sensors

Camera

Touch Sensor

2 DoF Neck

Touch Sensors

2 DoF Base

RFID Reader

RFID Reader

1 DoF Arms

domingo 17 de julio de 2011

Page 14: A Supervised Learning Architecture for Pose Recognition in a Social Robot

AD Architecture

Actuators

Short Term Memory

Long Term Memory

Sensors

Delib. Skill n

Delib. Skill 1

Delib. Skill 2

Autom. Skill n

Autom. Skill 1

Autom. Skill 2

events

domingo 17 de julio de 2011

Page 15: A Supervised Learning Architecture for Pose Recognition in a Social Robot

5 ModulesOverview

High-level view

!"#

"#

$%&'()*)+', #-.'/0&'1 21%3'1)

4'))56'(75))%86

4',91*(,5856','8+ :;<

='>(>19?)'1@A,5%&(-&%'8+

=91/(719-'))91

B&%'8+(&%>151%')

C%)05&%D5+%98E2FG2(H'1-'7+%98

49>%&'(,58%70&5+%98

I53%65+%98J5)K('L'-0+%3'

#%,0&5+%98

Robot Operating SystemRadu Bogdan Rusu

Maggie KinectROS

Voice System Weka

domingo 17 de julio de 2011

Page 16: A Supervised Learning Architecture for Pose Recognition in a Social Robot

ROS

Retrieved from: http://www.ros.org/wiki/Events/CoTeSys-ROS-School

Overview

High-level view

!"#

"#

$%&'()*)+', #-.'/0&'1 21%3'1)

4'))56'(75))%86

4',91*(,5856','8+ :;<

='>(>19?)'1@A,5%&(-&%'8+

=91/(719-'))91

B&%'8+(&%>151%')

C%)05&%D5+%98E2FG2(H'1-'7+%98

49>%&'(,58%70&5+%98

I53%65+%98J5)K('L'-0+%3'

#%,0&5+%98

Robot Operating SystemRadu Bogdan Rusu

domingo 17 de julio de 2011

Page 17: A Supervised Learning Architecture for Pose Recognition in a Social Robot

ROS

Robot Operating System

Retrieved from: http://www.ros.org/wiki/Events/CoTeSys-ROS-School

Overview

High-level view

!"#

"#

$%&'()*)+', #-.'/0&'1 21%3'1)

4'))56'(75))%86

4',91*(,5856','8+ :;<

='>(>19?)'1@A,5%&(-&%'8+

=91/(719-'))91

B&%'8+(&%>151%')

C%)05&%D5+%98E2FG2(H'1-'7+%98

49>%&'(,58%70&5+%98

I53%65+%98J5)K('L'-0+%3'

#%,0&5+%98

Robot Operating SystemRadu Bogdan Rusu

domingo 17 de julio de 2011

Page 18: A Supervised Learning Architecture for Pose Recognition in a Social Robot

ROS

Robot Operating System

C++, Python, Java, Octave, Lisp.

Ubuntu

(Mac and Windows in experimental phase)

Retrieved from: http://www.ros.org/wiki/Events/CoTeSys-ROS-School

Overview

High-level view

!"#

"#

$%&'()*)+', #-.'/0&'1 21%3'1)

4'))56'(75))%86

4',91*(,5856','8+ :;<

='>(>19?)'1@A,5%&(-&%'8+

=91/(719-'))91

B&%'8+(&%>151%')

C%)05&%D5+%98E2FG2(H'1-'7+%98

49>%&'(,58%70&5+%98

I53%65+%98J5)K('L'-0+%3'

#%,0&5+%98

Robot Operating SystemRadu Bogdan Rusu

domingo 17 de julio de 2011

Page 19: A Supervised Learning Architecture for Pose Recognition in a Social Robot

Robot Operating System

Node Node

/topicPublication Subscription

domingo 17 de julio de 2011

Page 20: A Supervised Learning Architecture for Pose Recognition in a Social Robot

5 ModulesOverview

High-level view

!"#

"#

$%&'()*)+', #-.'/0&'1 21%3'1)

4'))56'(75))%86

4',91*(,5856','8+ :;<

='>(>19?)'1@A,5%&(-&%'8+

=91/(719-'))91

B&%'8+(&%>151%')

C%)05&%D5+%98E2FG2(H'1-'7+%98

49>%&'(,58%70&5+%98

I53%65+%98J5)K('L'-0+%3'

#%,0&5+%98

Robot Operating SystemRadu Bogdan Rusu

Maggie KinectROS

Voice System Weka

domingo 17 de julio de 2011

Page 21: A Supervised Learning Architecture for Pose Recognition in a Social Robot

Kinect RGB-D Sensor

domingo 17 de julio de 2011

Page 22: A Supervised Learning Architecture for Pose Recognition in a Social Robot

Light Coding Technology

domingo 17 de julio de 2011

Page 23: A Supervised Learning Architecture for Pose Recognition in a Social Robot

domingo 17 de julio de 2011

Page 24: A Supervised Learning Architecture for Pose Recognition in a Social Robot

domingo 17 de julio de 2011

Page 25: A Supervised Learning Architecture for Pose Recognition in a Social Robot

5 ModulesOverview

High-level view

!"#

"#

$%&'()*)+', #-.'/0&'1 21%3'1)

4'))56'(75))%86

4',91*(,5856','8+ :;<

='>(>19?)'1@A,5%&(-&%'8+

=91/(719-'))91

B&%'8+(&%>151%')

C%)05&%D5+%98E2FG2(H'1-'7+%98

49>%&'(,58%70&5+%98

I53%65+%98J5)K('L'-0+%3'

#%,0&5+%98

Robot Operating SystemRadu Bogdan Rusu

Maggie KinectROS

Voice System Weka

domingo 17 de julio de 2011

Page 26: A Supervised Learning Architecture for Pose Recognition in a Social Robot

Automatic Speech Recognition (ASR)

HelloMicrophone

ASR “Hello”

domingo 17 de julio de 2011

Page 27: A Supervised Learning Architecture for Pose Recognition in a Social Robot

Text To Speech (TTS)

Hello

TTS “Hello”

Speaker

Speaker

domingo 17 de julio de 2011

Page 28: A Supervised Learning Architecture for Pose Recognition in a Social Robot

5 ModulesOverview

High-level view

!"#

"#

$%&'()*)+', #-.'/0&'1 21%3'1)

4'))56'(75))%86

4',91*(,5856','8+ :;<

='>(>19?)'1@A,5%&(-&%'8+

=91/(719-'))91

B&%'8+(&%>151%')

C%)05&%D5+%98E2FG2(H'1-'7+%98

49>%&'(,58%70&5+%98

I53%65+%98J5)K('L'-0+%3'

#%,0&5+%98

Robot Operating SystemRadu Bogdan Rusu

Maggie KinectROS

Voice System Weka

domingo 17 de julio de 2011

Page 29: A Supervised Learning Architecture for Pose Recognition in a Social Robot

Weka

Machine Learning Framework

Classification, Clustering, etc.

GPL

Java API

Decision Tree: J48 (C 4.5)

(Waikato Environment for Knowledge Analysis)

domingo 17 de julio de 2011

Page 30: A Supervised Learning Architecture for Pose Recognition in a Social Robot

Outline

pose_trainer

pose_labeler_bridge

Dataset

WekaModel

Learning Framework/Architecture

pose_classier

/skeleton

/classied_pose

OpenNI + pi_tracker

Kinect

/pose_labeled

asr_skill

etts_skill

pose_labeler_skill

RECOGNITION_RESULTSLABELED_POSE

pose_tellerSAY_TEXT

ROS AD

torsoOrient_w

<=0.956792 > 0.956792

torsoOrient_y

<=0.11679 > 0.11679

left_kneeOrient_w

<=0.609245 > 0.609245

STAND_TURNED_RIGHT

STAND_TURNED_FORWARD

STAND_TURNED_LEFTSTAND_TURNED_RIGHT

1. Introduction 2. System Description 3. Architecture

4. Evaluation 5. Conclusions

domingo 17 de julio de 2011

Page 31: A Supervised Learning Architecture for Pose Recognition in a Social Robot

3. Architecture

pose_trainer

pose_labeler_bridge

Dataset

WekaModel

Learning Framework/Architecture

pose_classier

/skeleton

/classied_pose

OpenNI + pi_tracker

Kinect

/pose_labeled

asr_skill

etts_skill

pose_labeler_skill

RECOGNITION_RESULTSLABELED_POSE

pose_tellerSAY_TEXT

ROS AD

domingo 17 de julio de 2011

Page 32: A Supervised Learning Architecture for Pose Recognition in a Social Robot

System Overview

MODEL

Dataset

Kinect

Microphone ASRMachine Learning

Framework

I'm seated

Kinect

TTS

Pose Classifier

Speaker

What's my pose?

MODEL

You're seated

domingo 17 de julio de 2011

Page 33: A Supervised Learning Architecture for Pose Recognition in a Social Robot

System Overview

MODEL

Dataset

Kinect

Microphone ASRMachine Learning

Framework

I'm seated

Kinect

TTS

Pose Classifier

Speaker

What's my pose?

MODEL

You're seated

domingo 17 de julio de 2011

Page 34: A Supervised Learning Architecture for Pose Recognition in a Social Robot

System Overview

MODEL

Dataset

Kinect

Microphone ASRMachine Learning

Framework

I'm seated

Kinect

TTS

Pose Classifier

Speaker

What's my pose?

MODEL

You're seated

domingo 17 de julio de 2011

Page 35: A Supervised Learning Architecture for Pose Recognition in a Social Robot

System Overview

MODEL

Dataset

Kinect

Microphone ASRMachine Learning

Framework

I'm seated

Kinect

TTS

Pose Classifier

Speaker

What's my pose?

MODEL

You're seated

domingo 17 de julio de 2011

Page 36: A Supervised Learning Architecture for Pose Recognition in a Social Robot

domingo 17 de julio de 2011

Page 37: A Supervised Learning Architecture for Pose Recognition in a Social Robot

pose_trainer

pose_labeler_bridge

Dataset

WekaModel

Learning Framework/Architecture

pose_classier

/skeleton

/classied_pose

OpenNI + pi_tracker

Kinect

/pose_labeled

asr_skill

etts_skill

pose_labeler_skill

RECOGNITION_RESULTSLABELED_POSE

pose_tellerSAY_TEXT

ROS AD

domingo 17 de julio de 2011

Page 38: A Supervised Learning Architecture for Pose Recognition in a Social Robot

pose_trainer

pose_labeler_bridge

Dataset

WekaModel

Learning Framework/Architecture

pose_classier

/skeleton

/classied_pose

OpenNI + pi_tracker

Kinect

/pose_labeled

asr_skill

etts_skill

pose_labeler_skill

RECOGNITION_RESULTSLABELED_POSE

pose_tellerSAY_TEXT

ROS AD

domingo 17 de julio de 2011

Page 39: A Supervised Learning Architecture for Pose Recognition in a Social Robot

pose_trainer

pose_labeler_bridge

Dataset

WekaModel

Learning Framework/Architecture

pose_classier

/skeleton

/classied_pose

OpenNI + pi_tracker

Kinect

/pose_labeled

asr_skill

etts_skill

pose_labeler_skill

RECOGNITION_RESULTSLABELED_POSE

pose_tellerSAY_TEXT

ROS AD

domingo 17 de julio de 2011

Page 40: A Supervised Learning Architecture for Pose Recognition in a Social Robot

pose_trainer

pose_labeler_bridge

Dataset

WekaModel

Learning Framework/Architecture

pose_classier

/skeleton

/classied_pose

OpenNI + pi_tracker

Kinect

/pose_labeled

asr_skill

etts_skill

pose_labeler_skill

RECOGNITION_RESULTSLABELED_POSE

pose_tellerSAY_TEXT

ROS AD

domingo 17 de julio de 2011

Page 41: A Supervised Learning Architecture for Pose Recognition in a Social Robot

pose_trainer

pose_labeler_bridge

Dataset

WekaModel

Learning Framework/Architecture

pose_classier

/skeleton

/classied_pose

OpenNI + pi_tracker

Kinect

/pose_labeled

asr_skill

etts_skill

pose_labeler_skill

RECOGNITION_RESULTSLABELED_POSE

pose_tellerSAY_TEXT

ROS AD

domingo 17 de julio de 2011

Page 42: A Supervised Learning Architecture for Pose Recognition in a Social Robot

pose_trainer

pose_labeler_bridge

Dataset

WekaModel

Learning Framework/Architecture

pose_classier

/skeleton

/classied_pose

OpenNI + pi_tracker

Kinect

/pose_labeled

asr_skill

etts_skill

pose_labeler_skill

RECOGNITION_RESULTSLABELED_POSE

pose_tellerSAY_TEXT

ROS AD

domingo 17 de julio de 2011

Page 43: A Supervised Learning Architecture for Pose Recognition in a Social Robot

pose_trainer

pose_labeler_bridge

Dataset

WekaModel

Learning Framework/Architecture

pose_classier

/skeleton

/classied_pose

OpenNI + pi_tracker

Kinect

/pose_labeled

asr_skill

etts_skill

pose_labeler_skill

RECOGNITION_RESULTSLABELED_POSE

pose_tellerSAY_TEXT

ROS AD

domingo 17 de julio de 2011

Page 44: A Supervised Learning Architecture for Pose Recognition in a Social Robot

pose_trainer

pose_labeler_bridge

Dataset

WekaModel

Learning Framework/Architecture

pose_classier

/skeleton

/classied_pose

OpenNI + pi_tracker

Kinect

/pose_labeled

asr_skill

etts_skill

pose_labeler_skill

RECOGNITION_RESULTSLABELED_POSE

pose_tellerSAY_TEXT

ROS AD

domingo 17 de julio de 2011

Page 45: A Supervised Learning Architecture for Pose Recognition in a Social Robot

pose_trainer

pose_labeler_bridge

Dataset

WekaModel

Learning Framework/Architecture

pose_classier

/skeleton

/classied_pose

OpenNI + pi_tracker

Kinect

/pose_labeled

asr_skill

etts_skill

pose_labeler_skill

RECOGNITION_RESULTSLABELED_POSE

pose_tellerSAY_TEXT

ROS AD

domingo 17 de julio de 2011

Page 46: A Supervised Learning Architecture for Pose Recognition in a Social Robot

pose_trainer

pose_labeler_bridge

Dataset

WekaModel

Learning Framework/Architecture

pose_classier

/skeleton

/classied_pose

OpenNI + pi_tracker

Kinect

/pose_labeled

asr_skill

etts_skill

pose_labeler_skill

RECOGNITION_RESULTSLABELED_POSE

pose_tellerSAY_TEXT

ROS AD

domingo 17 de julio de 2011

Page 47: A Supervised Learning Architecture for Pose Recognition in a Social Robot

pose_trainer

pose_labeler_bridge

Dataset

WekaModel

Learning Framework/Architecture

pose_classier

/skeleton

/classied_pose

OpenNI + pi_tracker

Kinect

/pose_labeled

asr_skill

etts_skill

pose_labeler_skill

RECOGNITION_RESULTSLABELED_POSE

pose_tellerSAY_TEXT

ROS AD

domingo 17 de julio de 2011

Page 48: A Supervised Learning Architecture for Pose Recognition in a Social Robot

Outline

pose_trainer

pose_labeler_bridge

Dataset

WekaModel

Learning Framework/Architecture

pose_classier

/skeleton

/classied_pose

OpenNI + pi_tracker

Kinect

/pose_labeled

asr_skill

etts_skill

pose_labeler_skill

RECOGNITION_RESULTSLABELED_POSE

pose_tellerSAY_TEXT

ROS AD

torsoOrient_w

<=0.956792 > 0.956792

torsoOrient_y

<=0.11679 > 0.11679

left_kneeOrient_w

<=0.609245 > 0.609245

STAND_TURNED_RIGHT

STAND_TURNED_FORWARD

STAND_TURNED_LEFTSTAND_TURNED_RIGHT

1. Introduction 2. System Description 3. Architecture

4. Evaluation 5. Conclusions

domingo 17 de julio de 2011

Page 49: A Supervised Learning Architecture for Pose Recognition in a Social Robot

4. Evaluation

torsoOrient_w

<=0.956792 > 0.956792

torsoOrient_y

<=0.11679 > 0.11679

left_kneeOrient_w

<=0.609245 > 0.609245

STAND_TURNED_RIGHT

STAND_TURNED_FORWARD

STAND_TURNED_LEFTSTAND_TURNED_RIGHT

domingo 17 de julio de 2011

Page 50: A Supervised Learning Architecture for Pose Recognition in a Social Robot

Demo Video

domingo 17 de julio de 2011

Page 51: A Supervised Learning Architecture for Pose Recognition in a Social Robot

Experiment Setup

180 cm240 cm

180

cm RobotU

ser

User Area

Kinect Horizontal Field of View

domingo 17 de julio de 2011

Page 52: A Supervised Learning Architecture for Pose Recognition in a Social Robot

Datasets and models

No Expert Knowledge

Expert Knowledge

2 TrainingDatasets

Moving during training

2 TrainingDatasets

Not moving during training

12 Users Tested the four models

torsoOrient_w

<=0.956792 > 0.956792

torsoOrient_y

<=0.11679 > 0.11679

left_kneeOrient_w

<=0.609245 > 0.609245

STAND_TURNED_RIGHT

STAND_TURNED_FORWARD

STAND_TURNED_LEFTSTAND_TURNED_RIGHT

right_kneeOrient_z

<=0.196633 > 0.196633

right_kneePos_x

<=0.100138 > 0.100138

STAND_TURNED_RIGHT

STAND_TURNED_FORWARDSTAND_TURNED_LEFT

right_kneeOrient_z

<=0.196633 > 0.196633

torsoOrient_w

<=0.699489 > 0.699489

STAND_TURNED_RIGHT

STAND_TURNED_FORWARDSTAND_TURNED_LEFT

torsoOrient_w

<=0.956792 > 0.956792

torsoOrient_y

<=0.11679 > 0.11679

left_kneeOrient_w

<=0.609245 > 0.609245

STAND_TURNED_RIGHT

STAND_TURNED_FORWARD

STAND_TURNED_LEFTSTAND_TURNED_RIGHT

domingo 17 de julio de 2011

Page 53: A Supervised Learning Architecture for Pose Recognition in a Social Robot

Models of “moving dataset” were the same

No Expert Knowledge

Expert Knowledge

2 TrainingDatasets

Moving during training

2 TrainingDatasets

Not moving during training

12 Users Tested the four models

right_kneeOrient_z

<=0.196633 > 0.196633

right_kneePos_x

<=0.100138 > 0.100138

STAND_TURNED_RIGHT

STAND_TURNED_FORWARDSTAND_TURNED_LEFT

right_kneeOrient_z

<=0.196633 > 0.196633

torsoOrient_w

<=0.699489 > 0.699489

STAND_TURNED_RIGHT

STAND_TURNED_FORWARDSTAND_TURNED_LEFT

torsoOrient_w

<=0.956792 > 0.956792

torsoOrient_y

<=0.11679 > 0.11679

left_kneeOrient_w

<=0.609245 > 0.609245

STAND_TURNED_RIGHT

STAND_TURNED_FORWARD

STAND_TURNED_LEFTSTAND_TURNED_RIGHT

domingo 17 de julio de 2011

Page 54: A Supervised Learning Architecture for Pose Recognition in a Social Robot

torsoOrient_w

<=0.956792 > 0.956792

torsoOrient_y

<=0.11679 > 0.11679

left_kneeOrient_w

<=0.609245 > 0.609245

STAND_TURNED_RIGHT

STAND_TURNED_FORWARD

STAND_TURNED_LEFTSTAND_TURNED_RIGHT

Moving.w/ & wo/ Expert

Knowledge

domingo 17 de julio de 2011

Page 55: A Supervised Learning Architecture for Pose Recognition in a Social Robot

Models of “moving dataset” were the same

No Expert Knowledge

Expert Knowledge

2 TrainingDatasets

Moving during training

2 TrainingDatasets

Not moving during training

12 Users Tested the four models

right_kneeOrient_z

<=0.196633 > 0.196633

right_kneePos_x

<=0.100138 > 0.100138

STAND_TURNED_RIGHT

STAND_TURNED_FORWARDSTAND_TURNED_LEFT

right_kneeOrient_z

<=0.196633 > 0.196633

torsoOrient_w

<=0.699489 > 0.699489

STAND_TURNED_RIGHT

STAND_TURNED_FORWARDSTAND_TURNED_LEFT

torsoOrient_w

<=0.956792 > 0.956792

torsoOrient_y

<=0.11679 > 0.11679

left_kneeOrient_w

<=0.609245 > 0.609245

STAND_TURNED_RIGHT

STAND_TURNED_FORWARD

STAND_TURNED_LEFTSTAND_TURNED_RIGHT

domingo 17 de julio de 2011

Page 56: A Supervised Learning Architecture for Pose Recognition in a Social Robot

right_kneeOrient_z

<=0.196633 > 0.196633

right_kneePos_x

<=0.100138 > 0.100138

STAND_TURNED_RIGHT

STAND_TURNED_FORWARDSTAND_TURNED_LEFT

Not Moving wo/ Expert Knowledge

(Position + Orientation)

domingo 17 de julio de 2011

Page 57: A Supervised Learning Architecture for Pose Recognition in a Social Robot

Models of “moving dataset” were the same

No Expert Knowledge

Expert Knowledge

2 TrainingDatasets

Moving during training

2 TrainingDatasets

Not moving during training

12 Users Tested the four models

right_kneeOrient_z

<=0.196633 > 0.196633

right_kneePos_x

<=0.100138 > 0.100138

STAND_TURNED_RIGHT

STAND_TURNED_FORWARDSTAND_TURNED_LEFT

right_kneeOrient_z

<=0.196633 > 0.196633

torsoOrient_w

<=0.699489 > 0.699489

STAND_TURNED_RIGHT

STAND_TURNED_FORWARDSTAND_TURNED_LEFT

torsoOrient_w

<=0.956792 > 0.956792

torsoOrient_y

<=0.11679 > 0.11679

left_kneeOrient_w

<=0.609245 > 0.609245

STAND_TURNED_RIGHT

STAND_TURNED_FORWARD

STAND_TURNED_LEFTSTAND_TURNED_RIGHT

domingo 17 de julio de 2011

Page 58: A Supervised Learning Architecture for Pose Recognition in a Social Robot

Not Moving w/ Expert Knowledge

(Orientation)right_kneeOrient_z

<=0.196633 > 0.196633

torsoOrient_w

<=0.699489 > 0.699489

STAND_TURNED_RIGHT

STAND_TURNED_FORWARDSTAND_TURNED_LEFT

domingo 17 de julio de 2011

Page 59: A Supervised Learning Architecture for Pose Recognition in a Social Robot

Models of “moving dataset” were the same

No Expert Knowledge

Expert Knowledge

2 TrainingDatasets

Moving during training

2 TrainingDatasets

Not moving during training

12 Users Tested the four models

right_kneeOrient_z

<=0.196633 > 0.196633

right_kneePos_x

<=0.100138 > 0.100138

STAND_TURNED_RIGHT

STAND_TURNED_FORWARDSTAND_TURNED_LEFT

right_kneeOrient_z

<=0.196633 > 0.196633

torsoOrient_w

<=0.699489 > 0.699489

STAND_TURNED_RIGHT

STAND_TURNED_FORWARDSTAND_TURNED_LEFT

torsoOrient_w

<=0.956792 > 0.956792

torsoOrient_y

<=0.11679 > 0.11679

left_kneeOrient_w

<=0.609245 > 0.609245

STAND_TURNED_RIGHT

STAND_TURNED_FORWARD

STAND_TURNED_LEFTSTAND_TURNED_RIGHT

domingo 17 de julio de 2011

Page 60: A Supervised Learning Architecture for Pose Recognition in a Social Robot

Results

ModelModel Correctly Classified (%)

False Positives (%)

Moving TrainingMoving Training

Not MovingTraining

wo/ Expert Knowledge

(Pos + Orient)Not Moving

Trainingw/ Expert Knowledge

(Orient)

92,6 3,9

56,3 21,3

70,0 14,1

domingo 17 de julio de 2011

Page 61: A Supervised Learning Architecture for Pose Recognition in a Social Robot

Outline

pose_trainer

pose_labeler_bridge

Dataset

WekaModel

Learning Framework/Architecture

pose_classier

/skeleton

/classied_pose

OpenNI + pi_tracker

Kinect

/pose_labeled

asr_skill

etts_skill

pose_labeler_skill

RECOGNITION_RESULTSLABELED_POSE

pose_tellerSAY_TEXT

ROS AD

torsoOrient_w

<=0.956792 > 0.956792

torsoOrient_y

<=0.11679 > 0.11679

left_kneeOrient_w

<=0.609245 > 0.609245

STAND_TURNED_RIGHT

STAND_TURNED_FORWARD

STAND_TURNED_LEFTSTAND_TURNED_RIGHT

1. Introduction 2. System Description 3. Architecture

4. Evaluation 5. Conclusions

domingo 17 de julio de 2011

Page 62: A Supervised Learning Architecture for Pose Recognition in a Social Robot

5. Conclusions

domingo 17 de julio de 2011

Page 63: A Supervised Learning Architecture for Pose Recognition in a Social Robot

Conclusions

• Architecture works

• 92% Correctly classified instances

• Now the robot is able to understand contextual information

domingo 17 de julio de 2011

Page 64: A Supervised Learning Architecture for Pose Recognition in a Social Robot

Future Work

• Gesture Recognition

• Activity Recognition

• Active Learning

domingo 17 de julio de 2011

Page 65: A Supervised Learning Architecture for Pose Recognition in a Social Robot

Questions?Thanks for your attention.

domingo 17 de julio de 2011