1

Multimodal emotion recognition and expressivity

2

Reference

S. Kollias, K. Karpouzis, “Multimodal emotion recognition and expressivity,” Multimedia and Expo, 2005. ICME 2005. IEEE International Conference on , 6-8 July 2005, p.p. 779- 783

3

Introduction

People express their emotions through multiple modalities Humans’ speech Facial expressions Body pose

Emotional feature and signs

4

Recognition of the user’s emotional state

Emotion analysis and recognition Audio Visual Physiological signal

Emotional psychological background

Human computer interaction (HCI)

5

Emotional speech analysis

Speech is a major channel for communicating emotion

Speech signal conveys Textual, lexical, emotional and gestural i

nformation The set of features in the speech signa

l Classification algorithm

6

Emotion recognition system

7

Paralinguistic speech analysis

Prosody is composed of Intonation Duration Intensity Speech quality

Voice quality is influenced by physiological factors

8

Feature extraction

Extracting information from Pitch contour, range, variance, mean,

jitter, intensity, shimmer Voice quality Duration : pauses, speaking rate Background information on the

speaker

9

Emotional facial analysis

Facial action coding system (FACS) Facial definition parameter (FDP) Facial animation parameter (FAP) MPEG-4 standard

10

Facial animation

11

Emotional Gesture Analysis Hand tracking systems Tracking the centroid of skin masks Estimates of user’s movements

12

Gesture recognition

13

Targeting Emotion Recognition Facial animation parameter from the

user’s face Future merging of different emotional

representations

14

Targeting Expressivity

Facial Expressivity Time-varying facial movements

Quantity and quality of movement Interaction Transition

Gesture Expressivity Speed, acceleration, direction variation

15

Physiologocal signal analysis Visceral differences between emotion

al states Heart rate Skin conductance level Finger temperature Muscle activity

16

Measurementwith physiological information

Biosensor The value of skin conductivity Electromyography (EMG) sensors for

muscle-activity

17

Multimodal emotion recognition

Define the processes and functions Visual, auditory and physiological modali

ties Identify different emotions in the reco

gnition processes Synchronization and temporal seque

nce in different modalities

18

Conclusions

Multimodal emotion recognition and expressivity analysis

Human computer interaction (HCI) Pattern recognition in combination wi

th different techniques