Recognising the Social Attitude in Natural Interaction with Pedagogical Agents

Università degli studi di Bari “Aldo Moro”Dipartimento di Informatica

Recognising the Social Attitude in NaturalInteraction with Pedagogical Agents

B. De Carolis, S. Ferilli, N. Novielli, F. Leuzzi, F. Rotella{decarolis, ferilli, novielli}@di.uniba.it, {fabio.leuzzi, fulvio.rotella}@uniba.it

DIDAMATICA, Informatica per la DidatticaTaranto, Italy, May 14-16, 2012

Overview

● Introduction

● Objective

● The proposed model

● The proposed approach

● Signs of social attitude

● Evaluation

● Conclusions

● Future works

Recognising the Social Attitude in Natural Interaction with Pedagogical AgentsB. De Carolis, S. Ferilli, N. Novielli, F. Leuzzi, F. Rotella 2

Introduction


Pedagogical Conversational Agent (PCA)

● fulfil pedagogical goals

● interact with the user through a natural dialog by

appropriately mixing verbal and non verbal expressions:

● recognize verbal and non-verbal inputs

● generate verbal and non-verbal outputs

● handle typical functions of human conversations,

with particular emphasis on social aspects

Objective


Aim: building a multimodal framework for the recognition

of the social response of users to a PCA.

In particular: building a framework that integrates the

analysis of the linguistic component of the user's

communicative act with the analysis of the acoustic

features of the spoken sentence and of the gestures.

ObjectiveThe intuition


The combination of these different input modalities may

improve the recognition of multimodal behaviours that

may denote the openness attitude of the users towards

the embodied agent.

Steps:

● Recognize signs of social attitude

● Build a model to infer the user attitude toward the PCA

● Adapt the dialog strategies accordingly

The proposed model


Dynamic Belief Network (DBN):

● handling uncertainty and incompleteness of data

● representing situations which gradually evolve from a

dialog step to the next one

The proposed approach


● The model is initialized

● At every dialog step:

● Knowledge about the evidence is produced

● The produced knowledge is entered and propagated in the

network

● The model revises the probabilities of the social attitude node

● The new probabilities of the signs of social attitude are used

for planning the next agent move

● The probability of the social attitude node supports revising

high-level planning of the agent behaviour

Signs of social attitudein the language


● Sense of intimacy (use of common jargon)

● Friendly self-introduction

● Familiar style

● Attempt to establish a common ground

● Talk about self

● Personal questions about the agent

● Irony and humour

● Benevolent/polemic attitude towards the system failures

● Favourable/negative comments

● Interest to protract interaction

● Friendly farewell


Praat functions to extract features related to:

● variation of the fundamental frequency

● variation of energy

● variation of harmonicity

● Spectrum Central Moment, Standard Deviation, Gravity

centre, Skeweness and Kurtosis

● speech rate

Classification (using NNge algorithm) of user's spoken

sentence into 3 classes: positive, negative and neutral.

Signs of social attitudein the prosody

Signs of social attitudein the gestures


Gesture recognition performed using

Microsoft Kinect + KinectDTW

need to consider only a subset of gestures compatible

with the nodes in

the skeleton that

the Kinect SDK

can detect.

Signs of social attitudein the gestures


Signal Possible meaning(s)

Crossed arms Defensiveness, closure

Gripping own upper arms

Insecurity, closure

Adjusting cuff, watchstrap, tic, using

an arm across the body

Nervouseness, negative attitude

touching or scratching shoulder using arm

across body

Nervouseness, negative attitude

EvaluationCollecting a corpus


We collected moltimodal dialog moves, consisting in

linguistic, acoustic and gesture data.

Participants: 2 groups of 5 italian students aged between

16 and 25 (equally distributed by gender).

Goal: getting information about a correct diet in order to

stay in shape.

PCA role: nutrition expert.

Collected: about 300 moves.

EvaluationCollecting a corpus


EvaluationResults


Move U6 Move U7

Move U8

Conclusions


Existing model for recognising social attitude enriched

with the analysis of signals regarding non-verbal

communication: prosody and gesture.

We propose an extension of the multimodal analysis to

gesture modeling, according to the meanings that

psycholinguistic researchers attach to gestures in

conversations.

Preliminary experiments show promising results.

Future works


● Improving gesture recognition since the new Kinect

should allow for a better hand recognition

● extending the social attitude analysis with facial

expressions

Carrying out more evaluation studies in order to test the

robustness of our framework:

● for social attitude recognition in different scenarios

● with respect to different interaction modalities with both

ECAs and Robots

Recognising the Social Attitude in Natural Interaction with Pedagogical Agents

Technology

Transcript of Recognising the Social Attitude in Natural Interaction with Pedagogical Agents