An Emotion Recognition Journey

Lucy Kuncheva

A long time agoin a galaxy far, far

away...

around the summer of 2008,we came in contact with the School of Psychology at Bangor University.

I’d better move to Cardiff...

I’d better move to Brunel...

Areas of activation in the brain in response to emotion stimuli.

Amygdala

Areas of activation in the brain in response to emotion stimuli.

The limbic system

fMRI data were acquired from 16 right-handed healthy US college male students (aged 20–25).

The Dead Salmon Lesson

How far are we from MIND READNING?

this far...

March 2009

EPSRC proposal

New approaches for fMRI data analysis

April 2010

FP7 proposal

Lost an entire month of my life to this....

1. Kuncheva L.I., J. J. Rodriguez, C. O. Plumpton, D. E. J. Linden and S. J. Johnston, Random Subspace Ensembles for fMRI Classification, IEEE Transactions on Medical Imaging, 29 (2), 2010, 531-542.

2. Kuncheva L.I., J. J. Rodriguez, Classifier Ensembles for fMRI Data Analysis: An Experiment, Magnetic Resonance Imaging, 28 (4), 2010, 583-593.

3. Kuncheva L.I. and C. O. Plumpton, Choosing parameters for Random Subspace ensembles for fMRI classification, Proc. Multiple Classifier Systems (MCS'10), Cairo, Egypt, LNCS 5997, 2010, 54-63.

4. Plumpton C. O., L. I. Kuncheva, N. N. Oosterhof and S. J. Johnston, Naive random subspace ensemble with linear classifiers for real-time classification of fMRI data, Pattern Recognition, 45 (6), 2012, 2101-2108.

5. Plumpton C. O., L. I. Kuncheva, D. E. J. Linden and S. J. Johnston, On-line fMRI Data Classification Using Linear and Ensemble Classifiers, Proc. ICPR 2010, Istanbul, Turkey, 2010, 4312-4315.

9

41

70

7

5

Cat Plumpton

PhDreal-time fMRI data analysis

Tom Gardner 10/113rd year projectEnvironments for emotion recognition

Adam Williams 09/103rd year project

Emotion recognition from fMRI data

Jamie Blacker 09/103rd year project

Fractals and fMRI

Joe Freeman 10/113rd year projectfMRI Visualiser

Joey Owen 10/113rd year projectfMRI Voxel selection

Colin Steele 08/093rd year projectfMRI Data analysis

10 20 30 40 50

10

20

30

40

50

10 20 30 40

10

20

30

40

10 20 30 40 50

10

20

30

40

50

50 100 150 200

195

200

205

210

215

2010

End of the fMRI era...

Enter Tom Christy!

Summer 2010

Brain-computer interface through EEG

Game accessories

EEG headsets

Peripheral devices

Accessible

Inexpensive

And just like that...The idea was born...

A game controlled by emotion

Summer 2010

Summer 2010

Sa’ad

Martin

Not as easy as it looked...

Autumn 2010

AFFECTIVE COMPUTING

TAC celebrates its 5th Anniversary

The Galvactivator: A glove that senses and communicates skin conductivity

“Affective Computing is an area of computing that relates to, arises from, or influences emotions.” Rosalind Picard, 1995

HAHV

HALV

POSITIVE

Arousal

LALV

LAHV happy

excited

angry

fearful

content

calm

sad

depressed

PASSIVE

ACTIVE

Valence

NEGATIVE

EXPRESSION OF EMOTION - MODALITIES

facial expression

posture

behaviouralphysiologic

al

peripheral nervous system

central nervous system

EEG

fMRI

Galvanic skin response

blood pressure

skin to

respiration

EMG

speech

gesture

interaction with

the compute

r

pressure on mouse

drag-click speed

eye tracking

fNIRS

pulse rate

pulse variation

dialogue with tutor

facial expression

posture

EEG

fMRI

Galvanic skin response

blood pressure

skin to

respiration

EMGspeech

gesture

pressure on mouse

drag-click-zoom-type

speed

eye tracking

fNIRS

pulse ratepulse variation

dialogue with tutor

Detecting Stress During Real-World Driving Tasks Using Physiological SensorsJennifer A. Healey and Rosalind W. PicardIEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS, VOL. 6, NO. 2, JUNE 2005

The subject wore five physiological sensors:

• an electrocardiogram (EKG) on the chest

• an electromyogram (EMG) on the left shoulder

• a chest cavity expansion respiration sensor (Resp.) around the diaphragm,

• skin conductivity sensor on the left hand

• skin conductivity sensor on the left foot

AMBER Experiment #1

Tom

NeuroSky EEG

Pulse signal

Galvanic skin response

(+) Positive emotionWhale and ocean sounds...

(-) Negative emotion• Michael Jackson• Cheeky girls

  3 4 5 6 10 12 15 20

1nn 62.8 64.9 63.4 62.0 61.1 60.9 56.5 59.9

DT 64.2 58.6 67.4 65.9 58.5 62.8 70.0 58.9

RT 60.0 63.0 61.9 62.6 57.1 66.7 66.8 57.1

NB 64.7 63.8 64.5 64.5 65.0 67.8 65.4 61.1

LOG 62.0 60.4 62.6 63.3 59.3 59.2 57.6 57.5

MLP 62.5 59.4 63.3 63.4 63.4 64.2 57.1 58.5SVM-

L 62.1 61.4 63.5 62.4 62.3 59.1 58.7 56.8SVM-

R 50.8 51.2 50.6 50.5 50.2 51.2 51.7 51.3

BAG 65.6 65.6 68.3 67.1 67.4 68.8 66.5 64.4

RAF 64.5 64.7 66.1 65.3 65.9 69.6 67.3 61.6

ADA 63.4 62.2 70.0 67.6 61.1 66.3 73.8 63.3

LB 65.3 62.9 68.8 68.1 62.0 64.0 68.3 60.7

RS 65.0 64.8 66.3 68.2 64.6 67.4 69.0 61.8

ROF 66.9 65.4 66.9 67.2 67.4 69.3 65.5 62.3

Individual classifiers

Ensembles

Serious wired-up man Tom Christy

Very important supervisorLucy Kuncheva

A.M.B.E.R.Advanced Multimodal Biometric Emotion Recognition

NeuroSky

EMOTIV

Nia

My lovely plant

Tom’s R2D2

(show off!)

EDAPulse reader

Cast:

A.M.B.E.R.Advanced Multimodal Biometric Emotion Recognition

Media stars overnight!

HARDWARE

DATA&

CollaboratorsTom Lucy

Tom

HARDWARE201320122011

2013“emotional mouse”

Version 1

EDA sensors(Galvanic skin response)

pulse reader

ReincarnationVersion 2

In the meantime:

Guillaume Thierry

This is what Google returned 2nd on “Guillaume Thierry”!

Me: Would you like to collaborate on emotion recognition from EEG data? Guillaume: Yes, of course, but listen what a fantastic idea occurred to me just now!!!”

Christoph Klein

Stephan Boehm

Let’s give it a go

March 2011

EPSRC proposal

No One is a Prophet in their Own Land

Hello Salzburg!Seminar

May 2012

On-going collaboratio

n with the University of

Salzburg

Hello Juan Rodriguez!

Hello Ramon Mollineda!

happy

elated

excited

alertPOSITIVE AROUSAL

tense

nervous

stressed

upset

UNPLEASANT

sad

depressed

bored

drowsy

NEGATIVE AROUSAL

calm

relaxed

tranquil

contended

PLEASANT

Video: Arch Enemy (My Apocalypse)

Volunteers Participants All

0 1 2 3 4 5 6 7 8 9 100

1

2

3

4

5

6

7

8

9

10

0 1 2 3 4 5 6 7 8 9 100

1

2

3

4

5

6

7

8

9

10

What emotion is being provoked?!?

0 1 2 3 4 5 6 7 8 9 100

1

2

3

4

5

6

7

8

9

10

Valence

Aro

usal

T: Intended emotion

E: EXPERIENCED emotion

P: Reported emotionProvoked?

Acted?

Spontaneous?

Self-reported?

So, WHAT are we RECOGNISING?

1. Emotions are very difficult to define and explicate.2. Experiments for provoking emotion vary considerably, and so do

the results reported in the literature ( from near chance to 95% accuracy).

3. Most emotion measuring modalities are intrusive and annoying.4. Emotions are individual for each person.5. The measured signals are difficult to analyse. There is a

bottleneck of idiosyncratic feature extraction and parameter tuning.

6. There is no unified protocol. 7. Benchmark data collections are not available. 8. There is no consensus about the type of experiment to validate

a hypothesis (provoked, controlled, acted, spontaneous emotion).

Are we doomed?

Maybe not...

We can detect CHANGE in the physiological responses and the EEG, which may be associated with some emotion. If we need to ACT upon detecting an emotion rather than NAMING it, we still may have a chance.

The emotional mouse

Other ingenious input devices

User-friendly EEG headsets

Simple, transparent and generic technologies for feature extraction.

State-of-the-

art data analys

is

LucyTom

Unified protocols, benchmark data

The emotional mouse

Other ingenious input devices

User-friendly EEG headsets

Simple, transparent and generic technologies for feature extraction.

State-of-the-

art data analys

is

LucyTom

Unified protocols, benchmark data

THE END