1

Affect and Personality inInteraction with Ubiquitous

SystemsProfessor Ruth Aylett

Vision Interactive Systems & GraphicalEnvironments

MACS, Heriot-Watt Universitywww.macs.hw.ac.uk/~ruth

Summary of programme

Introduction and overview Affective outputs

– speech, language, gesture, facial expressions, music, colour

Affective/Personality models and action-selectionapproaches

Affective inputs Applications

– Embodied Conversational Characters, Intelligent Virtual,Agents, human-robot interaction

Evaluation approaches

Today’s topics

Embodied Conversational Characters -ECAs

Laban Movement analysis Gesture FACS and facial expressions

Embodied Conversational Agents

Affective EmbodiedConversational Agents

2

ECAs Definition

Software entities capable of autonomouslycommunicating with users through verbal and nonverbalmeans

Autonomous they can plan what to say, know whento start a conversation, when to answer and when totake the conversation turn

ECA’s cognitive and expressive capabilities simulatehuman capabilities

Models are based on theories from human studies,particularly in the domains of linguistic, phonetics,cognitive science, emotion, psychology, and sociology

GretaU Paris8 Pelachaud, Mancini, Bevacqua et al

Affective multimodal ECA– Produce dynamic expressive

visual behaviors– Achieve coordination of signals

in multiple modalities– Encode behavior variability

MAX in the MuseumUniversity Bielefeld, Wachsmuth, Kopp et al

Located in a museum for use by all Engages visitors in Natural Language

small-talk conversations– Gives information about the museum and

exhibition– Knows about external environment– Tells jokes, plays animal guessing game– Internet lookup of information, e.g.

weather report

Emotions– Continuously updated, always shown– Influence behavior, e.g. leaving the scene

when very annoyed by rude visitorbehavior

Expressive movement andfacial expression

How to make this ‘human’?– But are we doing life or drama?

Look for formalisms that we can re-apply– Expressive movement: Laban Movement

Analysis– Facial expression: Facial Action Coding

System

3

Laban Movement Analysis

Originates in ballet choreography– As a notation for dance

5 major components:– Body: part of the body in used– Space: description of the directions and paths of motion– Shape: changing forms of the body– Effort: dynamics of the body moves– Relationship: modes of interaction with oneself, others and

the environment (facing, contact…)

Textual and symbolic language of movementdescription

EMOTE (Badler et al)

EMOTE model for Effort and Shape Based on Movement Observation

Science Laban Movement Analysis Computational model of Effort and

Shape components

Effort

Four motion factors: space, weight, time,flow– Space– Weight– Time– Flow

Each factor ranges from:– Reinforcing the factor– Fighting against the factor

Effort

Space: indirect direct– Waving away bugs pointing to a particular

spot Weight: light strong

– Feather movement punching

Time: sustained sudden– Stretching grabbing a falling object

Flow: free bound– Waving widely moving in slow motion

4

Shape

Three distinct qualities of change in theform of movement:– Shape flow: mover’s attitude toward the

changing relationship among body parts– Directional movement: mover’s intent to

bridge the action to a point in theenvironment

– Shaping: mover’s carving or moldingattitude with the environment

Shape

Three dimensions:– Horizontal: spreading enclosing

• Opening arms to embrace claspingsomeone in a hug

– Vertical: rising sinking• Reaching for something in a high shelf

stamping the floor with indignation– Sagittal: advancing retreating

• Reaching out to shake hand Avoiding apunch

Capturing the movement

5

GRETA example Greta with different emotional parameters

Facial Expression

Facial Action Coding System

Duchenne de Bologne

French anatomist of19thC– In 1860s used

electric currents onfacially paralysedsubject

– Showed whichmuscles go withwhich expressions

6

Expressions - facial muscles

Main human muscle groupsfor articulation and facialexpressions

Muscle numbered in white: 1.Frontalis, 2. Corrugator, 3.Orbicularis oculi, 4. Levatorpalpebrae, 5. Nose andupper lip, 6. Upper lip, 7.Zygomaticus major, 8. Lipstretch, 9. Triangularis, 10.Lower lip, 11. Mentalis, 12.Orbicularis oris, 13. Jaw

Reflector-point positionsnumbered in yellow

Facial expression

Drawing on psychology:– FACS (Facial Action Coding System)– Developed by Ekman & Friesen (1978)– Distinguishes among 44 Action Units– Each action unit (AU) describes a separate,

visible, and muscle-based facial change

Ekman’sFACs

For primitiveemotions:– Surprise, fear,

disgust, anger,happiness, sadness

The Duchenne smile

Associated withexpression ofhappiness

AU12 (Lip Corner Puller)

Plus

AU6 (Cheek Raiser)

7

Mapping onto mpeg-4

Defines FAPs– Facial action

parameters– Set of feature points– Model calibration– Morphology

parameters– Expression

parameters

CG facial expressionshttp://mrl.nyu.edu/perlin/experiments/facedemo

Neutral Frightened Kiss Disappointed Sleeping Annoyed Surprised Happy Arrogant Angry Talking

– (CG face – Ken Perlin, NY)

Facial expressions simplified

Use of texture with morphing– Can work well with cartoon-like faces

Affective ListenerAndré, U Ausgburg

Creation of rapportbetween a virtual agentand the human user byproviding the agentwith emotionalsensitivity

Responding to theuser’s emotional voicewhen telling a story tothe agent

E André

8

Agents for training University of Southern California

QUESTIONS?