HUMAINE summer HUMAINE summer school on databases:school on databases:opening orientationopening orientation

Roddy Cowie, Ellen Douglas-Roddy Cowie, Ellen Douglas-Cowie, Edelle McMahon & Cate Cowie, Edelle McMahon & Cate

CoxCox

PracticalitiesPracticalities Welcome!Welcome!

Fire alarmsFire alarms

Break and afterBreak and after

GroupsGroups

LogonsLogons

Toilets Toilets

PhotoPhoto

INTRODUCTION: INTRODUCTION: DATABASES in HUMAINEDATABASES in HUMAINE

The aim of this session is to put the Summer The aim of this session is to put the Summer school in context.school in context.

To do that it reviewsTo do that it reviews basic points about HUMAINE basic points about HUMAINE

• (context we are working in)(context we are working in) the kind of database work relevant to HUMAINEthe kind of database work relevant to HUMAINE the databases that currently existthe databases that currently exist

Sets out natural targets for research on Sets out natural targets for research on databases in the HUMAINE contextdatabases in the HUMAINE context based on report delivered in June (D5c)based on report delivered in June (D5c)

INTRODUCTION: INTRODUCTION: DATABASES in HUMAINEDATABASES in HUMAINE

That background sets the scene for That background sets the scene for summer schoolsummer school

its function is getting hands-on feel for its function is getting hands-on feel for techniques we might use to meet the techniques we might use to meet the targetstargets

and establishing relationships among and establishing relationships among people who will do it.people who will do it.

Basic points about HUMAINE Basic points about HUMAINE (context we are working in)(context we are working in)

HUMAINE is about developing a HUMAINE is about developing a particular approach to research on particular approach to research on emotionemotion

It is spelled out in the Technical Annex It is spelled out in the Technical Annex (TA for short)(TA for short)

We are committed to follow through We are committed to follow through what we said in the TA,what we said in the TA,

and we need to be clear what it was.and we need to be clear what it was.

Basic points about HUMAINE Basic points about HUMAINE (context we are working in)(context we are working in)

Key points: HUMAINE is oriented toKey points: HUMAINE is oriented to Applications – but applications in the Applications – but applications in the

long termlong term Emotion – but emotion in a broad Emotion – but emotion in a broad

sensesense Emotion as a factor in action and Emotion as a factor in action and

interaction, not emotions as isolated interaction, not emotions as isolated episodesepisodes

These affect database objectives hugelyThese affect database objectives hugely

Basic points about HUMAINE: Basic points about HUMAINE:

Applications in the long termApplications in the long term Humaine aims to bolster sound long-term Humaine aims to bolster sound long-term development bydevelopment by clarifying the scientific, cultural and ethical clarifying the scientific, cultural and ethical

underpinnings of emotion-oriented computing; underpinnings of emotion-oriented computing; developing accessible literature and standards …; developing accessible literature and standards …; and establishing enduring relationships among and establishing enduring relationships among

research groups in the key areas ….research groups in the key areas …. To achieve real European leadership in this To achieve real European leadership in this

area, coordinated efforts must go beyond the area, coordinated efforts must go beyond the more engineering- and application-oriented more engineering- and application-oriented developmentsdevelopments undertaken in leading undertaken in leading countries outside the EU…countries outside the EU…

Basic points about HUMAINE: Basic points about HUMAINE:

emotion in a broad senseemotion in a broad sense We [will] consider emotion in an inclusive We [will] consider emotion in an inclusive sense rather than in the narrow sense of sense rather than in the narrow sense of episodes where a strong rush of feeling episodes where a strong rush of feeling briefly dominates a person’s awareness.briefly dominates a person’s awareness.

… … emotion in the broad sense pervades emotion in the broad sense pervades human communication and cognition. human communication and cognition. Human beings have positive or negative Human beings have positive or negative feelings about most things, people, events feelings about most things, people, events and symbols. These feelings strongly and symbols. These feelings strongly influence the way they attend, behave, plan, influence the way they attend, behave, plan, learn and select.learn and select.

Basic points about HUMAINE: Basic points about HUMAINE: emotion in action & interaction emotion in action & interaction Human beings have positive or negative feelings Human beings have positive or negative feelings

about most things, people, events and symbols. about most things, people, events and symbols. These feelings strongly influence the way they These feelings strongly influence the way they

attend, behave, plan, learn and select. attend, behave, plan, learn and select. The feelings are conveyed e.g. in faces, voices, The feelings are conveyed e.g. in faces, voices,

gestures, and postures; and people judge others by gestures, and postures; and people judge others by the way they respond to such signals. the way they respond to such signals.

They seek out situations that are emotionally rich They seek out situations that are emotionally rich (not necessarily pleasant), (not necessarily pleasant),

and are persuaded as much by emotional appeal as and are persuaded as much by emotional appeal as by factual content. by factual content.

The TA commits us to approach emotion in a The TA commits us to approach emotion in a way that embraces these aspectsway that embraces these aspects

Basic points about HUMAINE: Basic points about HUMAINE: terms to sum up terms to sum up

Bounded emotionsBounded emotions specific kinds of mental state where a strong rush of specific kinds of mental state where a strong rush of

feeling dominates a person’s awareness for a feeling dominates a person’s awareness for a relatively short timerelatively short time

Pervasive emotionPervasive emotion a quality that ‘colours’ almost any mental state to a a quality that ‘colours’ almost any mental state to a

greater or lesser extentgreater or lesser extent and therefore influences a wide range of actions & and therefore influences a wide range of actions &

interactionsinteractions

HUMAINE is committed to address both –HUMAINE is committed to address both –and therefore, so is its work on databasesand therefore, so is its work on databases

Basic points about HUMAINE: Basic points about HUMAINE: relevant kinds of dbase work relevant kinds of dbase work

Databases can play two kinds of role in researchDatabases can play two kinds of role in research Supportive Supportive

role of dbase is to let people fill in or check details of role of dbase is to let people fill in or check details of processes they understand in general terms, at least as well processes they understand in general terms, at least as well as they think they need to. as they think they need to.

Provocative Provocative role is to collect examples that help people to expand and role is to collect examples that help people to expand and

restructure their thinking about the area. restructure their thinking about the area.

To understand pervasive emotion in action & To understand pervasive emotion in action & interaction, interaction,

the database strand has to play a provocative role the database strand has to play a provocative role

Basic points about HUMAINE: Basic points about HUMAINE: summary summary

By the nature of HUMAINE, we need dbases toBy the nature of HUMAINE, we need dbases to Lay the ground for long term applicationsLay the ground for long term applications Deal with pervasive emotion as well as Deal with pervasive emotion as well as

bounded episodesbounded episodes Show how emotion figures in action and Show how emotion figures in action and

interaction interaction Plays a provocative role as well as a Plays a provocative role as well as a

supportive one.supportive one.

How does available material measure up?How does available material measure up?

Background to HUMAINE Background to HUMAINE database strand database strand

Review of existing materialReview of existing material

(based on D5c, available on the portal)(based on D5c, available on the portal)

Review of existing Review of existing material:database groupsmaterial:database groups

3 main groupings of databases (see 3 main groupings of databases (see handout)handout) multimodalmultimodal face onlyface only speech onlyspeech only

A lot of material, giving impression that we A lot of material, giving impression that we have the informationhave the information

But on closer inspection …But on closer inspection …

Review of existing materialReview of existing material

The older materialThe older material Overwhelmingly goes for bounded emotionsOverwhelmingly goes for bounded emotions Often relies on acted dataOften relies on acted data Rarely shows how modalities combineRarely shows how modalities combine Rarely shows emotion affecting action and interactionRarely shows emotion affecting action and interaction Often uses a narrow or arbitrary range of emotionsOften uses a narrow or arbitrary range of emotions

Suppose you have a practically important Suppose you have a practically important situation involving emotional colouring, signalled situation involving emotional colouring, signalled through multiple modalities, affecting action and through multiple modalities, affecting action and interaction - how likely that you will find empirical interaction - how likely that you will find empirical evidence relevant to it?evidence relevant to it?

Review of existing materialReview of existing material

The progressThe progress Emerging work on multimodal databases – Emerging work on multimodal databases –

Belfast Naturalistic Database, SMARTKOMBelfast Naturalistic Database, SMARTKOM Emerging interest in naturalistic sourcesEmerging interest in naturalistic sources

Lost luggage, call centreLost luggage, call centre Emerging interest in ‘colouring’Emerging interest in ‘colouring’

AIBO robot, CampbellAIBO robot, Campbell

- trends we need to consolidate. - trends we need to consolidate.

Agenda for HUMAINE work on Agenda for HUMAINE work on databasesdatabases

Goal: generate an exemplar, ieGoal: generate an exemplar, ie

a piece of work which exemplifies how a piece of work which exemplifies how a key problem in the area can be solved a key problem in the area can be solved in a principled way; in a principled way;

and which also demonstrates how work and which also demonstrates how work focused on that area can integrate with focused on that area can integrate with work focused on the other areaswork focused on the other areas

Agenda for HUMAINE work on Agenda for HUMAINE work on databasesdatabases

It will be a library of data to be accessed by It will be a library of data to be accessed by research on emotional states and their research on emotional states and their expressionexpression

But what will be the contents?But what will be the contents?

D5c outlines preliminary ideasD5c outlines preliminary ideas

They define the kinds of technique we want They define the kinds of technique we want to explore in the summer schoolto explore in the summer school - - tooling up for the tasks we defined in D5ctooling up for the tasks we defined in D5c

HUMAINE library of data:HUMAINE library of data:types of archivetypes of archive

1. Historical1. Historical Key databases assembled or otherwise made accessibleKey databases assembled or otherwise made accessible

2. Supportive (Reactive)2. Supportive (Reactive): : Related to needs of other workpackages.Related to needs of other workpackages. eg annotated recordings for evaluation of signal processing eg annotated recordings for evaluation of signal processing

routinesroutines gesture-rich samples support development of coding systems gesture-rich samples support development of coding systems (unless pre-existing material meets needs).(unless pre-existing material meets needs).

3. Provocative (Proactive)3. Provocative (Proactive): : material that will challenge fundamental research in the area material that will challenge fundamental research in the area

4. Metaknowledge4. Metaknowledge: : procedures, systems, summary analyses, etc.procedures, systems, summary analyses, etc.

HUMAINE library of data:HUMAINE library of data:emotional scopeemotional scope

There should be a prestated framework There should be a prestated framework defining the emotional and emotion-related defining the emotional and emotion-related states that should ideally be covered states that should ideally be covered (even if there are gaps)(even if there are gaps)

Standard lists of ‘primaries’ are not an Standard lists of ‘primaries’ are not an appropriate frameworkappropriate framework

D5c has a list based on Cowie et al 1999 D5c has a list based on Cowie et al 1999 and Scherer 2004 – and Scherer 2004 – What do we think of it? Watch this space …What do we think of it? Watch this space …

HUMAINE library of data:HUMAINE library of data:levels of representativenesslevels of representativeness

Naturalistic materialNaturalistic material the point of reference for other material the point of reference for other material but we need to understand its limits as well as its but we need to understand its limits as well as its

value value Acted materialActed material we would prefer in principle to avoid we would prefer in principle to avoid but how much of the table can we sample in any but how much of the table can we sample in any

other ways? other ways? Induction techniquesInduction techniques we propose to invest effort in selecting and we propose to invest effort in selecting and

developing appropriate techniquesdeveloping appropriate techniques

HUMAINE library of data:HUMAINE library of data:contexts (your team scored)contexts (your team scored)

Solo (often in a laboratory) Solo (often in a laboratory) emotion (if it is expressed at all) is likely to be impoverishedemotion (if it is expressed at all) is likely to be impoverished

Sedentary human interactionSedentary human interaction sitting talking to each othersitting talking to each other

Mobile human interaction Mobile human interaction possibility of whole-body movements & use of objectspossibility of whole-body movements & use of objects

Social facilitation Social facilitation a group can amplify various kinds of actiona group can amplify various kinds of action

Task-dominated Task-dominated scope for expression is constrained by the task (eg driving). scope for expression is constrained by the task (eg driving).

Surface manifestations vary massively with the task. Are there Surface manifestations vary massively with the task. Are there deeper constants?deeper constants?

Machine interaction Machine interaction important for us – but should be understood in the broad rangeimportant for us – but should be understood in the broad range

HUMAINE library of data:HUMAINE library of data:modalitiesmodalities

audio and visual audio and visual recorded as normrecorded as norm (but naturalism may outweigh …)(but naturalism may outweigh …)

performance measures performance measures (error rate, reaction time, etc) (error rate, reaction time, etc) where there is a task where there is a task

psychophysiological measures psychophysiological measures (heart rate, skin conductance, temperature, breathing) (heart rate, skin conductance, temperature, breathing) occasional – interesting but restrictiveoccasional – interesting but restrictive

brain scans, kinetic records brain scans, kinetic records … … realistically beyond our scope.realistically beyond our scope.

HUMAINE library of data:HUMAINE library of data: diversity and scale diversity and scale

We have to recognise as issuesWe have to recognise as issues GenderGender CultureCulture AgeAge

Can we collect hundreds of samples per closely-Can we collect hundreds of samples per closely-defined cell?defined cell? to be negotiated for supportive databasesto be negotiated for supportive databases normally, establish the technique;normally, establish the technique; let others replicate.let others replicate.

HUMAINE library of data:HUMAINE library of data:measurement & labellingmeasurement & labelling

Assessments of the person’s emotional stateAssessments of the person’s emotional state Codings of the signs that reveal itCodings of the signs that reveal it

What is WP5’s responsibility?What is WP5’s responsibility?

Many techniques are not an optionMany techniques are not an option too labour-intensive too labour-intensive too restricted in applicationtoo restricted in application

It does make sense to explore techniques that areIt does make sense to explore techniques that are efficient efficient of general interestof general interest

HUMAINE library of data:HUMAINE library of data:structure and accessibilitystructure and accessibility

Format people can access and understandFormat people can access and understand

Systematic approach to choice of unitsSystematic approach to choice of units

Mechanisms for distributionMechanisms for distribution

… … and more. and more.

Background to the Summer Background to the Summer School: summarySchool: summary

By the nature of HUMAINE, we need dbases toBy the nature of HUMAINE, we need dbases to Deal with pervasive emotion as well as Deal with pervasive emotion as well as

bounded episodes, show how emotion bounded episodes, show how emotion figures in action and interaction, play a figures in action and interaction, play a provocative role as well as a supportive oneprovocative role as well as a supportive one

They need to cover a satisfying range of They need to cover a satisfying range of emotions, be acceptably natural, address emotions, be acceptably natural, address context effects, cover key modalities, reflect context effects, cover key modalities, reflect basic forms of diversity, include basic basic forms of diversity, include basic labelling, be competently structured.labelling, be competently structured.

Background to the Summer Background to the Summer School: summarySchool: summary

HOW ?!*?@? HOW ?!*?@?

Between us, we have a range of Between us, we have a range of tools at our disposal tools at our disposal

The summer school is about The summer school is about getting to grips with them -getting to grips with them -

bearing in mind what we want to bearing in mind what we want to achieveachieve