Michael Frydrych, 11.6.2004 Laboratory of Computational Engineering

Making the head smileMaking the head smile

Smile? -> part of non-verbal communication

Functions of non-verbal communication To enhance linguistic information To give new information

Michael Frydrych, 11.6.2004 Laboratory of Computational Engineering

Non-verbal communicationNon-verbal communication

Figure by Jari Kätsyri

Michael Frydrych, 11.6.2004 Laboratory of Computational Engineering

Voice prosodyVoice prosody

Figure by Jari Kätsyri

Michael Frydrych, 11.6.2004 Laboratory of Computational Engineering

HeadHead

Figure by Jari Kätsyri

Michael Frydrych, 11.6.2004 Laboratory of Computational Engineering

EyesEyes

Figure by Jari Kätsyri

Michael Frydrych, 11.6.2004 Laboratory of Computational Engineering

Facial expressionsFacial expressions

Figure by Jari Kätsyri

Michael Frydrych, 11.6.2004 Laboratory of Computational Engineering

GesturesGestures

Figure by Jari Kätsyri

Michael Frydrych, 11.6.2004 Laboratory of Computational Engineering

Conversation regulationConversation regulation

Figure by Jari Kätsyri

Michael Frydrych, 11.6.2004 Laboratory of Computational Engineering

Conversational signalsConversational signals

Figure by Jari Kätsyri

Michael Frydrych, 11.6.2004 Laboratory of Computational Engineering

MaintenanceMaintenance

Figure by Jari Kätsyri

Michael Frydrych, 11.6.2004 Laboratory of Computational Engineering

EmotionsEmotions

Figure by Jari Kätsyri

Michael Frydrych, 11.6.2004 Laboratory of Computational Engineering

Expressing emotionsExpressing emotions

Figure by Jari Kätsyri

Michael Frydrych, 11.6.2004 Laboratory of Computational Engineering

Conversational signalsConversational signals

Figure by Jari Kätsyri

Michael Frydrych, 11.6.2004 Laboratory of Computational Engineering

VoiceVoice

Figure by Jari Kätsyri

Michael Frydrych, 11.6.2004 Laboratory of Computational Engineering

HeadHead

Figure by Jari Kätsyri

Michael Frydrych, 11.6.2004 Laboratory of Computational Engineering

EyesEyes

Figure by Jari Kätsyri

Michael Frydrych, 11.6.2004 Laboratory of Computational Engineering

FaceFace

Figure by Jari Kätsyri

Michael Frydrych, 11.6.2004 Laboratory of Computational Engineering

ExamplesExamples

Michael Frydrych, 11.6.2004 Laboratory of Computational Engineering

Dimitris Metaxas, Normal Dimitris Metaxas, Normal BadlerBadler Modeling of hand gestures Modeling emotions, personality…

Michael Frydrych, 11.6.2004 Laboratory of Computational Engineering

Facial expressionsFacial expressions

Emotional expressions in face Are expressions showing emotions? How they can be described? What kind of emotions are there?

Michael Frydrych, 11.6.2004 Laboratory of Computational Engineering

Emotion in face: To Emotion in face: To communicate or to express?communicate or to express? Communicating vs. expressing emotions

Intentional communication (Chovil) vs. Involuntary behaviour (Ekman)

Technology viewpoint Emotion = Emotional expression = (Non-

verbal) communication (In most situations emotions

inappropriate)

Michael Frydrych, 11.6.2004 Laboratory of Computational Engineering

Emotions and facial Emotions and facial expressions (Ekman)expressions (Ekman) Emotion Facial expression ?

Emotional expressions can be inhibited No evidence for non-expressed emotions

Facial expression Emotion ? Emotional expressions can be faked There are facial muscles with no

voluntary control Facial expression Emotion (?)

Michael Frydrych, 11.6.2004 Laboratory of Computational Engineering

Description of expressionDescription of expression

Several descriptions of expressions exist: Facial Action Coding System – FACS

By Eckman and Friesen ”Language of facial expressions”

MPEG-4 Synthetic-Natural hybrid coding (SNHC)

“Technical” description for low-bandwidth transmission

Michael Frydrych, 11.6.2004 Laboratory of Computational Engineering

FACSFACS

Noticeable changes, not exactly facial muscles Tool for identifying facial actions Not for modeling facial actions Does not describe dynamics

Action units (AUs) AU1: Inner brow raiser … AU23: Lip tightener … AU66: Crosseye

Michael Frydrych, 11.6.2004 Laboratory of Computational Engineering

AU examples: AU7 (lid AU examples: AU7 (lid tightener)tightener)

Michael Frydrych, 11.6.2004 Laboratory of Computational Engineering

AU examples: AU9 (nose AU examples: AU9 (nose wrinkler)wrinkler)

Michael Frydrych, 11.6.2004 Laboratory of Computational Engineering

AU examples: AU11AU examples: AU11(nasolabial furrow deepener)(nasolabial furrow deepener)

Michael Frydrych, 11.6.2004 Laboratory of Computational Engineering

Pure and blended emotionsPure and blended emotions

7 pure emotions Pleasure (pl), surprise (su), fear (f), hate

(h), sorrow (so), disgust (d), (interest) Blended emotions

Combinations of pure emotions Easy: pl+su, h+d, so+f Difficult: su+f, d+pl, h+so, f+pl (Some) additivity

Michael Frydrych, 11.6.2004 Laboratory of Computational Engineering

Possible causes of blended expressions in real world Mixed emotions (sadness+disgust) Consequent emotions

(surprise+happiness) Masking (anger+happiness) …

Michael Frydrych, 11.6.2004 Laboratory of Computational Engineering

Attempts to add emotionsAttempts to add emotions

Michael Frydrych, 11.6.2004 Laboratory of Computational Engineering

neu

tral

sad

su

rpri

se

Michael Frydrych, 11.6.2004 Laboratory of Computational Engineering

6 basic expressions6 basic expressions

courtesy Jari Kätsyri

Michael Frydrych, 11.6.2004 Laboratory of Computational Engineering

courtesy Jari Kätsyri

Blended expressionsBlended expressions

Michael Frydrych, 11.6.2004 Laboratory of Computational Engineering

Textured × non-texturedTextured × non-textured

Identification of expressions from pictures and videos of textured and non-textured version of AP

0.0%

20.0%

40.0%

60.0%

80.0%

100.0%

anger disgust fear happin. sadn. surprise

picture

picture+tex

video

video+tex

Michael Frydrych, 11.6.2004 Laboratory of Computational Engineering

Differences among databasesDifferences among databases

Number of correct responses for pictures and videos of different databases

Static

Dynamic5

6

7

8

9

10

CK HUT EF TH

Michael Frydrych, 11.6.2004 Laboratory of Computational Engineering

Neutral expressionNeutral expression

Average evaluations for AP textured and non-textured neutral expressions

Michael Frydrych, 11.6.2004 Laboratory of Computational Engineering

Better implementationBetter implementation

Collecting data

Michael Frydrych, 11.6.2004 Laboratory of Computational Engineering

Collecting data for Collecting data for expressionsexpressions

happy anger

Michael Frydrych, 11.6.2004 Laboratory of Computational Engineering

HUT facial expression HUT facial expression databasedatabase First Finnish

database of moving facial expressions

Michael Frydrych, 11.6.2004 Laboratory of Computational Engineering

Study:Study:Interaction of blur x type x Interaction of blur x type x emotionemotion New database, 6 actors 3 levels of blur Pictures, short video sequences 6 basic expressions

3 subject groups (17 each) each group – 1 level of blur each actor 6 expressions 3 actors pictures, 3 actors videos

Michael Frydrych, 11.6.2004 Laboratory of Computational Engineering

Interaction of blur x type x Interaction of blur x type x emotionemotion

Michael Frydrych, 11.6.2004 Laboratory of Computational Engineering

The endThe end

Thank you!

and … a little exercise again