1

Computation Approaches to Emotional Speech

Julia Hirschberg

julia@cs.columbia.edu

2

Why Study Emotional Speech?

• Recognition• Anger/frustration in call centers

• Confidence/uncertainty in online tutoring systems

• “Hot spots” in meetings

• Generation• TTS for

– Computer games

– IVR systems

• Other applications: Speaker State– Deception, Charisma, Sleepiness, Interest…

– The Love Detector (available for Skype )…

3

Assessing Health-Related Conditions

• Assessing intoxication levels (Levit et al ‘01)• Distinguishing between active and passive coping

responses in patients with breast cancer (Zei Pollermann ’02)

• Assessing schizophrenia (Bitouk et al ‘09)• Classifying degree of autistic behavior (Columbia)• Suicide notes

4

Hard Questions in Emotion Recognition

• How do we know what emotional speech is?– Acted speech vs. natural (hand labeled) corpora

• What can we classify?• Distinguish among multiple ‘classic’ emotions

• Distinguish

– Valence: is it positive or negative?

– Activation: how strongly is it felt? (sad/despair)

• What features best predict emotions?• What techniques best to use in classification?

5

happysadangryconfidentfrustratedfriendlyinterested

anxiousboredencouraging

Acted Speech: LDC Emotional Speech Corpus

6

Is Natural Emotion Different? (thanks to Liz Shriberg)

• Neutral– July 30

– Yes

• Disappointed/tired– No

• Amused/surprised

– No

• Annoyed– Yes– Late morning

• Frustrated– Yes– No

– No, I am … – …no Manila...

7

Major Problems for Classification:Different Valence/Different Activation

8

But….Different Valence/ Same Activation

9

Good Features Can be Hard to Find

• Useful features: – Automatically extracted pitch, intensity, rate,

VQ– Hand-labeled, automatically stylized pitch

contours– Context– Lexical information: Dictionary of Affect– But….individual and cultural differences

• Algorithms for classification:– Machine learning (Decision trees, Support Vector

Machines, Rule induction algorithms, HMMs,…)

10

Results: Different Emotions, Different Success Rates

Emotion Baseline Accuracy

angry 69.32% 77.27%

confident 75.00% 75.00%

happy 57.39% 80.11%

interested 69.89% 74.43%

encouraging 52.27% 72.73%

sad 61.93% 80.11%

anxious 55.68% 71.59%

bored 66.48% 78.98%

friendly 59.09% 73.86%

frustrated 59.09% 73.86%

11

Open Questions

• New features and algorithms• New types of emotion/speaker state to identify• New ways of finding/collecting useful data• New applications of more-or-less successful

emotion classification• Interspeech Paralinguistic Challenges

12

This Class

• Goals:– Learn what we know about: readings and discussion

participation

– Learn how to analyze speech, how to design a speech experiment, how to classify speaker states

– Try to contribute something new: term project

– Practice doing research

• Syllabus:– http://www.cs.columbia.edu/~julia/courses/CS6998/

syllabus11.htm

13

Readings and Discussion

• Weekly readings– Everyone prepares/hands in 3 discussion questions on each assigned paper or website

• If you read an optional paper, submit questions on that as well if you want ‘credit’

– Everyone participates in class discussion

– Each week one person leads discussion on one paper

– Submit pdf in courseworks shared files

14

Term Project

• Everyone prepares a term project on a topic of their choice– You may work alone or in teams of 2

• Deliverables– Proposal – Interim progress report– Final report– Short presentation/demo

15

Possible Topics

• Collect audio from children of different ages winning and losing a game and see if adults can distinguish those who win (happy speech) from those who lose (sad speech).

• Create hybrid speech stimuli from tokens uttered with different emotions (mixing pitch, loudness, duration, speaking rate,...) and see which features of emotional speech are most reliably associated with emotions.

• Detect different emotions from Cantonese and Mandarin speakers and compare performance of an automatic program to performance of human judges.

• Train Machine Learning algorithms on emotional speech corpora and see if you can improve over other approaches on the same corpora

• Develop an email reader that detects emotion from text and uses the appropriate emotional TTS system to read it to the use

16

Important Details

• Read the academic integrity paragraph in the syllabus and understand it.

• Do all the readings when they are due, turn in all discussion questions by noon on the day of class, come to every class

17

Questions?