4837410-Automatic-Facial-Emotion-Recognition.ppt

7/25/2019 4837410-Automatic-Facial-Emotion-Recognition.ppt

1/52

Automatic Facial Emotion Recognition

Aitor Azcarate

Felix HagelohKoen van de Sande

Roberto Valenti

Supervisor: Nicu Sebe


2/52

OverviewINTR!"#TIN

R$%AT$! &RK

$'TIN R$#(NITIN

#%ASSIFI#ATIN

VIS"A%I)ATIN

FA#$ !$T$#TR

!$'

$VA%"ATIN

F"T"R$ &RKS

#N#%"SIN

*"$STINS


3/52

Emotions

Emotions are reflected in voice, handand body gestures, and mainly through

facial expressions


4/52

Emotions (2)

&h+ is it i,portant to recognize e,otions-

Human beings express emotions in day to

day interactions

Understanding emotions and knowing how

to react to peoples expressions greatly

enriches the interaction


5/52

Human-Computer interaction

Knowing the user

emotion, the system canadapt to the user

ensing !and responding

appropriately"# to theusers emotional state will

be perceived as more

natural, persuasive, andtrusting

$e only focus on emotion

recognition%


6/52

Related work

#ross.cultural research b+ $/,an sho0s

that so,e e,otional expressions areuniversal:

Happiness

Sadness

Anger

Fear

!isgust 1,a+be2

Surprise 1,a+be2

ther e,otional expressions are

culturall+ variable3


7/52

Related work (2)

$/,an developedthe Facial Action

#oding S+ste,

1FA#S2:

!escription o4 4acial

,uscles and

5a06tongue derived4ro, anal+sis o4

4acial anato,+


8/52

Facial Expression Recognition

&antic ' (othkrant) in &*+ -...performed a survey of the field

(ecogni)e a generic procedure

amongst all systems/ Extract features !provided by a tracking

system, for example#

0eed the features into a classifier 1lassify to one of the pre2selected emotion

categories !3 universal emotions, or

34neutral, or 54neutral, etc#


9/52

Field overview Extracting !eatures

ystems have a model of the face and

update the model using video frames/ $avelets

6ual2view point2based model

7ptical flow urface patches in 8e)ier volumes

+any, many more

0rom these models, features areextracted9


10/52

Facial !eatures

$e use features similar to Ekmans/6isplacement vectors of facial features

(oughly corresponds to facial movement

!more exact description soon#


11/52

Our Facial "odelNice to usecertain

4eatures7 but ho0 do0e get the,-

0ace tracking, basedon a system

developed by :ao andHuang ;1?,subse@uently used by1ohen, ebe et al;1&(.-?

0irst, landmark facialfeatures !e9g9, eyecorners# are selectedinteractively


12/52

Our Facial "odel (2)

* generic face model is then warped to

fit the selected facial features :he face model consists of A3 surface

patches embedded in 8e)ier volumes


13/52

Face tracking-6 image motionsare measured usingtemplate matchingbetween frames atdifferent resolutions

B6 motion can beestimated from the -6motions of manypoints of the mesh

:he recoveredmotions arerepresented in termsof magnitudes of facialfeatures


14/52

Related work Classi!iers

&eople have used the whole range ofclassifiers available on their set offeatures !rule2based, 8ayesiannetworks, Ceural networks, H++, C8,k2Cearest Ceighbour, etc#9

ee &antic ' (othkrant) for an overviewof their performance9

8oils down to/ there is little training dataavailable, so if you need to estimatemany parameters for your classifier, youcan get in trouble9


15/52

OverviewINTR!"#TIN

R$%AT$! &RK

$'TIN R$#(NITIN

#%ASSIFI#ATIN

VIS"A%I)ATIN

FA#$ !$T$#TR

!$'

$VA%"ATIN

F"T"R$ &RKS

#N#%"SIN

*"$STINS


16/52

Classi!ication # $eneral %tructure

ava Server

1lassifier


17/52

Classi!ication - &asics

&e 0ould li/e to assign a class label c toan observed 4eature vector 8 0ith n

di,ensions 14eatures23

The opti,al classi4ication rule under the

,axi,u, li/elihood 1'%2 is given as:


18/52

Classi!ication - &asics

ur 4eature vector has 9 4eatures

#lassi4ier identi4ies ; basice,otions:

Happiness

Sadness Anger

Fear

!isgust

Surprise No e,otion 1neutral2


19/52

'e Classi!iers

Na


20/52

'e Classi!iers - a*ve &a+es

&ell /no0n classi4ication ,ethod $as+ to i,ple,ent

Kno0n to give surprisingl+ good

results

Si,plicit+ ste,s 4ro, the

independence assu,ption


21/52


In a na


22/52


#onditional probabilities are,odeled 0ith a (aussian distribution

For each 4eature 0e need to

esti,ate:

'ean:

Variance:

==

N

iiN

x1

1

==

N

iiN

x1

212 )(

C *


23/52


>roble,s 0ith Na


24/52

'e Classi!iers - 'A

Tree.Aug,ented.Naive =a+es Subclass o4 =a+esian net0or/

classi4iers

=a+esian net0or/s are an eas+ andintuitive 0a+ to ,odel 5oint

distributions

1Na


25/52

'e Classi!iers - 'A

The structure o4 the =a+sian Net0or/is crucial 4or classi4ication

Ideall+ it should be learned 4ro, the

data set using '% =ut searching through all possible

dependencies is N>.#o,plete

&e should restrict ourselves to asubclass o4 possible structures

' Cl i!i 'A


26/52

'e Classi!iers - 'A

TAN ,odels are such a subclass Advantage: There exist an e44icient

algorith, ?#ho0.%iu@ to co,pute the

opti,al TAN ,odel

' Cl i!i 'A


27/52

'e Classi!iers - 'A

Structure: The class node has no parents

$ach 4eature has as parent the class

node $ach 4eature has as parent at ,ost one

other 4eature

' Cl i!i 'A


28/52

'e Classi!iers - 'A

,i li ti


29/52

,isualiation

#lassi4ication results are visualizedin t0o di44erent 0a+s

=ar !iagra,

#ircle !iagra,

=oth i,ple,ented in 5ava

,i li ti & .i


30/52

,isualiation # &ar .iagram

,i li ti Ci l .i


31/52

,isualiation # Circle .iagram

O er ie


32/52

OverviewINTR!"#TIN

R$%AT$! &RK

$'TIN R$#(NITIN

#%ASSIFI#ATIN

VIS"A%I)ATIN

FA#$ !$T$#TR

!$'

$VA%"ATIN

F"T"R$ &RKS

#N#%"SIN

*"$STINS

/andmarks and !itted model


33/52

/andmarks and !itted model

0ro1lems


34/52

0ro1lems

'as/ 4itting

Scale independent Initialization in placeB

Fitted 'odel

Reinitialize the ,esh in the correctposition 0hen it gets lost

Solution-

FA#$ !$T$#TR

ew mplementation


35/52

ew mplementation

+ovie 68

7penD

converter

1apture+odule

0ace

6etector

0ace0itting

end data to

classifier

ostF

(epositioning

yes

no

1lassify and

visuali)e results

olid mask

Face .etector


36/52

Face .etector

%oo/ing 4or a 4ast and reliable one

"sing the one proposed b+ Viola andCones

Three ,ain contributions:

Integral I,ages Adaboost

#lassi4iers in a cascade structure

"ses Haar.%i/e 4eatures to recognizeob5ects

Face .etector 3Haar /ike4 !eatures


37/52

Face .etector # 3Haar-/ike4 !eatures

Face .etector ntegral mages


38/52

Face .etector # ntegral mages

A D 9

= D .9

# D E.9

! D .A.=.#

6 G 54A2!-4B#

Face .etector Ada1oost


39/52

Face .etector - Ada1oost

Results o4 the 4irst t0o Adaboost Iterations

This ,eans:

Those 4eatures appear in all the data

'ost i,portant 4eature: e+es

Face .etector Cascade


40/52

Face .etector - Cascade

All Sub.0indo0s

T T T

Re5ect Sub.0indo0

F F F F

9 E

.emo


41/52

.emo

Overview


42/52

OverviewINTR!"#TIN

R$%AT$! &RK

$'TIN R$#(NITIN

#%ASSIFI#ATIN

VIS"A%I)ATIN

FA#$ !$T$#TR

!$'

$VA%"ATIN

F"T"R$ &RKS

#N#%"SIN

*"$STINS

Evaluation


43/52

Evaluation

>erson independent

Used two classifiers/ Cave 8ayes and:*C9

*ll data divided into three sets9 :hen two

parts are used for training and the other

part for testing9 o you get B different test

and training sets9

:he training set for person independent

tests contains samples from several peopledisplaying all seven emotions9 0or testing a

disIoint set with samples from other people

is used9

Evaluation


44/52

Evaluation>erson independent

Results Na


45/52

Evaluation>erson independent

Results TAN:

Evaluation


46/52

Evaluation

>erson dependent

*lso used two classifiers/ Cave 8ayes and:*C

*ll the data from one person is taken and

divided into three parts9 *gain two parts are

used for training and one for testing9

:raining is done for J people and is then

averaged9

Evaluation


47/52

Evaluation>erson dependent

Results Na


48/52

Evaluation>erson dependent

Results TAN:

Evaluation


49/52

Evaluation

#onclusions:

Na


50/52

Future 5ork

Handle partial occlusions better3

'a/e it ,ore robust 1lightingconditions etc32

'ore person independent 14it ,as/

auto,aticall+23 "se other classi4iers 1d+na,ics23

Appl+ e,otion recognition in

applications3 For exa,ple ga,es3

Conclusions


51/52

Conclusions

ur i,ple,entation is 4aster 1due to

server connection2 #an get input 4ro, di44erent ca,eraLs

#hanged code to be ,ore e44icient

&e have visualizations "se 4ace detection

'as/ loading and recover+

6uestions


52/52

6uestions

?

4837410-Automatic-Facial-Emotion-Recognition.ppt

Documents

Transcript of 4837410-Automatic-Facial-Emotion-Recognition.ppt