The supplementary material includes description of sce-narios that we developed to record emotionally chargedperson-person interactions for the MPIIEmo dataset aswell as a summary of existing datasets for analysis of emo-tional expressions.

The details on scenarios and sub-scenarios are shownin Fig. 1 - 7. Description of each scenario includes a shortbackground of the situation that led to a scene and a de-scription of each sub-scenario that correspond to possibledevelopments. These descriptions were given to subjectsprior to recording and are used to guide their conversa-tion. Fig. 1 - 7 also include example snapshots to illustrateeach of the recordings. For the purpose of illustration wechoose images from recordings of different pairs of sub-jects for every scenario.

1

Emotion Recognition from Embedded Bodily Expressions and Speech duringDyadic Interactions

Supplementary Material

1. B is happy that A reminded himand throws away the garbage.

2. B is angry about himself,because A has needed to remind

him. A tries to calm him down, itsok to forget once.

3. B is angry about the As irony. Athinks that his remarks were ok and

becomes angry himself.

4. B is bewildered because heknows that today is not his turn. Aforgot about it. A is surprised and

apologizes himself.

Scenario 1: “MuellWe”. Scenario description: A is in the kitchen, B comes in. A asks B with irony if he is perhaps thenext one to throw away the garbage. (A: white shirt, B: black shirt)

1. A is coming along and is happyabout the party. B is happy, that A

will join.

2. A was invited, but has a lot ofwork to do and can’t come. She issad about it. B sympathizes with

her.

3. A is surprised. She didn’t knowabout the party but she would like

to come along. B is happy about it.

4. The party is organized by thecompany where A and B are bothtogether. A was not invited. She isangry. She has the impression thatshe is a second-class employee inthe company. B is surprised and

bewildered that A was not invited.

Scenario 2: “PartyPe”. Scenario description: A is in the kitchen, B comes in. B asks A if she is also coming to theparty. (A: grey pullover, B: green pullover)

2

1. A is happy and says thank you.B is happy that she could do A a

favor.

2. A is surprised and happy that Bbrought a new flavour. B is proud

that she discovered it.

3. A ist angry because B didn’tbring what he asked for. B is sadthat she made a mistake. Actually,she was looking forward to do a

favour for A.

4. A reacts indifferently. B isirritated and angry, she was

expecting to be thanked for makingan effort.

Scenario 3: “Mitbrin”. Scenario description: B comes back from shopping and gives A a pack of candy that A askedher to buy. A reacts in various ways. (A: male person, B: female person)

1. A is happy together with B.

2. A applied for the same job, butdidn’t get it. A is angry at B,

because B took away the job fromA and on top of that talks about it

happily.

3. A is sad because it means that Bwill be moving away soon. Theywere very good friends. B also

becomes sad when he thinks aboutit.

4. A is surprised because B toldhim before that the job was very

difficult to get. B is proud ofhermself.

Scenario 4: “JobOffe”. Scenario description: B has just learned that she passed a job interview for a position that shewanted for a long time. She enters the kitchen and talks about it happily. (A: white shirt, B: black shirt)

1. B’s girlfriend broke up with him.B is very sad and talks about it. Ahas sympathy for B and tries to

solace him.

2. B’s girlfriend broke up with him.B is very sad and talks about it. A

is angry at B’s ex-girlfriend.

3. B broke up with his girlfriend.He is happy about it, because shegot on his nerves since a long time.A is bewildered that B is not sad

because of the break up.

4. B broke up with his girlfriend.He is happy about it, because shegot on his nerves since a long time.

A is sad because he was gettingalong with B’s ex-girlfriend very

well.

Scenario 5: “SplitUp”. Scenario description: B and his girlfriend broke up. B comes into the kitchen. A is there too.(A: female person, B: male person)

3

1. A is angry that B arrives so late.B is sorry.

2. A is angry and points out withirony how late B is. B is

textbfangry because of the irony.

3. B apologizes immediately, A isfine with it.

4. A is surprised that B finallyarrived after all. B is sorry.

Scenario 6: “ZuSpaet”. Scenario description: A und B were supposed to meet at home to go climbing together. B ishalf an hour late. A is waiting in the kitchen, B comes in a hurry. (A: male person, B: female person)

1. A is surprised and very happythat he can go on a trip with B for

free.

2. A has exams during the time ofthe trip and can’t go. Both are

disappointed and sad.

3. A hates trips on a ship. B hasalready registered A for the trip.

This makes A angry. A thinks thatB should have known that he

would rather not go on the sea trip.

4. B is surprised and bewilderedthat B wants to go on this trip,

because it will be his exam time.However, A is still happy that B

takes him on the trip.

Scenario 7: “ReiseGe”. Scenario description: B just learned that he won two tickets for a journey on a cruise ship. Heis extremely happy about it and since he has an extra ticket he would like to take A with him. B goes to the kitchen totell A about it. (A: person on the left, B: person on the right)

4

Nam

em

odal

ities

#vi

ews

#pe

ople

emot

ion

anno

ta-

tions

natu

raln

ess

ofin

tera

c-tio

n

size

natu

reof

data

lang

uage

USC

Cre

-at

ive

IT[7

]

audi

o,vi

deo,

moc

ap(f

ullb

ody)

3Dpo

si-

tions

2va

lenc

e,ac

tivat

ion,

dom

inan

ce

acte

d19

acto

rs,

ca.9

hac

tors

im-

prov

isin

gon

thea

tri-

cale

xer-

cise

san

dex

cerp

tsof

play

s

Eng

lish

IEM

OC

AP

[1]

audi

o,vi

deo,

moc

ap(f

ace

and

wri

sts)

3Dpo

si-

tions

(mo-

cap)

2va

lenc

e,ac

tivat

ion,

dom

inan

ce+

disc

rete

acte

d10

acto

rs,

ca.1

2h

scri

pted

and

impr

ovis

atio

n-ba

sed,

acto

rssi

tting

Eng

lish

Ver

aam

Mitt

ag[5

]

audi

o,vi

deo

12-

5no

nefo

rfu

ll-bo

dyvi

deo

part

ofda

tase

t

natu

ral

12h

unsc

ript

ed,

auth

entic

disc

ussi

ons

betw

een

gues

tsof

ata

lksh

ow

Ger

man

SEM

AIN

E[6

]

audi

o,vi

deo

22

(1av

atar

)va

lenc

e,ac

tivat

ion,

pow

er,a

n-tic

ipat

ion,

over

all

emot

ion

inte

nsity

+op

tiona

ldi

men

sion

s

natu

ral

100

reco

rd-

ings

,ap-

prox

5m

inea

ch

user

inte

r-ac

ting

with

avat

arin

fron

tof

com

pute

r

Eng

lish

Bel

fast

natu

ralis

ticda

tase

t[3]

audi

o,vi

deo

12

vale

nce,

ac-

tivat

ion

and

cate

gori

alan

nota

tions

natu

ral

125

spea

k-er

s,29

8cl

ips

(10-

60s

each

)

Inte

ract

ive

unsc

ript

eddi

scou

rse,

from

TV

show

san

dla

bin

tera

c-tio

ns

Eng

lish

Met

anet

[2]

audi

o,vi

deo

54

cate

gori

es:

neut

ral,

emph

asiz

ed,

nega

tive

(fru

stra

ted)

,en

thus

iast

ic

Spon

tane

ous,

guid

e-lin

edco

nver

sa-

tion

6pe

ople

,5h

inte

ract

ions

ina

mee

ting-

room

envi

-ro

nmen

t

Eng

lish,

Span

ish

and

Cat

alan

MPI

IEm

o(o

urs)

audi

o,vi

deo

(ful

lbod

y)8

2va

lenc

e,ac

tivat

ion,

pow

er,a

n-tic

ipat

ion

and

cate

go-

rial

anno

ta-

tions

acte

d14

acto

rs,

196

clip

s,12

5m

in

Impr

ovis

atio

nof

coar

sely

scri

pted

daily

-lif

esi

tuat

ions

ina

kitc

hen

Ger

man

Tabl

e8:

Com

pari

son

ofdi

ffer

enta

vaila

ble

data

sets

for

stud

ying

affe

ctiv

ebo

dily

expr

essi

ons

inin

tera

ctio

ns.

The

data

sets

pres

ente

dhe

rear

ea

subs

etof

the

ones

liste

din

[4].

5

References[1] C. Busso, M. Bulut, C.-C. Lee, A. Kazemzadeh, E. Mower,

S. Kim, J. N. Chang, S. Lee, and S. S. Narayanan. Iemo-cap: Interactive emotional dyadic motion capture database.Language resources and evaluation, 42(4):335–359, 2008.5

[2] T. Butko, C. Nadeu, and A. Moreno. A multilingual corpusfor rich audio-visual scene description in a meeting-roomenvironment. In ICMI workshop on multimodal corpora formachine learning: Taking Stock and Roadmapping the Fu-ture, 2011. 5

[3] E. Douglas-Cowie, R. Cowie, and M. Schroder. A newemotion database: considerations, sources and scope. InISCA Tutorial and Research Workshop (ITRW) on Speechand Emotion, 2000. 5

[4] A. Eerekoviæ. An insight into multimodal databases forsocial signal processing: acquisition, efforts, and directions.Artificial Intelligence Review, pages 1–30. 5

[5] M. Grimm, K. Kroschel, and S. Narayanan. The vera ammittag german audio-visual emotional speech database. InMultimedia and Expo, 2008 IEEE International Conferenceon, pages 865–868. IEEE, 2008. 5

[6] G. McKeown, M. F. Valstar, R. Cowie, and M. Pantic. Thesemaine corpus of emotionally coloured character interac-tions. In Multimedia and Expo (ICME), 2010 IEEE Interna-tional Conference on, pages 1079–1084. IEEE, 2010. 5

[7] A. Metallinou, C.-C. Lee, C. Busso, S. Carnicke, andS. Narayanan. The usc creativeit database: a multimodaldatabase of theatrical improvisation. Multimodal Corpora:Advances in Capturing, Coding and Analyzing Multimodal-ity 18 May 2010, page 55, 2010. 5

6