CS 59000 Statistical Machine learning Lecture 13 Yuan (Alan) Qi Purdue CS Oct. 8 2008.
Machines that Recognize Human Emotion Yuan Qi MIT Media Laboratory.
-
Upload
horatio-taylor -
Category
Documents
-
view
213 -
download
0
Transcript of Machines that Recognize Human Emotion Yuan Qi MIT Media Laboratory.
![Page 1: Machines that Recognize Human Emotion Yuan Qi MIT Media Laboratory.](https://reader036.fdocuments.in/reader036/viewer/2022062718/56649e765503460f94b7803b/html5/thumbnails/1.jpg)
Machines that Recognize Human Emotion
Yuan Qi
MIT Media Laboratory
![Page 2: Machines that Recognize Human Emotion Yuan Qi MIT Media Laboratory.](https://reader036.fdocuments.in/reader036/viewer/2022062718/56649e765503460f94b7803b/html5/thumbnails/2.jpg)
A man barges into your office when you’re busy. He doesn’t apologize, doesn’t introduce himself, and
doesn’t notice you are annoyed.
He offers you useless advice.You express more annoyance. He ignores it.
He continues to be unhelpful. The clarity of your emotional expression escalates. He ignores it.
(this goes on) Finally you have to tell him explicitly “go away”
He winks, and does a little dancebefore exiting.
![Page 3: Machines that Recognize Human Emotion Yuan Qi MIT Media Laboratory.](https://reader036.fdocuments.in/reader036/viewer/2022062718/56649e765503460f94b7803b/html5/thumbnails/3.jpg)
Recognition of three “basic” states:
![Page 4: Machines that Recognize Human Emotion Yuan Qi MIT Media Laboratory.](https://reader036.fdocuments.in/reader036/viewer/2022062718/56649e765503460f94b7803b/html5/thumbnails/4.jpg)
• Expressions, behaviors“Flared nostrils, tightened lips, a quick sharp
gesture, skin conductivity=high; probably she is angry ”
• Situation, reasoningThat was an important goal to her and Bob just
thwarted it, so she probably feels angry toward Bob
“ Emotion recognition”
![Page 5: Machines that Recognize Human Emotion Yuan Qi MIT Media Laboratory.](https://reader036.fdocuments.in/reader036/viewer/2022062718/56649e765503460f94b7803b/html5/thumbnails/5.jpg)
Emotions give rise to changes that can be sensed
Face
Distance Voice
Sensing: Posture
Gestures, movement, behavior
Skin conductivity
Pupillary dilation
Up-close Respiration, heart rate, pulse
Sensing: Temperature
Blood pressure
Internal Hormones
Sensing: Neurotransmitters
…
![Page 6: Machines that Recognize Human Emotion Yuan Qi MIT Media Laboratory.](https://reader036.fdocuments.in/reader036/viewer/2022062718/56649e765503460f94b7803b/html5/thumbnails/6.jpg)
Emotions give rise to changes that can be sensed
Distance Voice
Sensing: Posture
Gestures, movement, behavior
Skin conductivity
Pupillary dilation
Up-close Respiration, heart rate, pulse
Sensing: Temperature
Blood pressure
Internal Hormones
Sensing: Neurotransmitters
…
![Page 7: Machines that Recognize Human Emotion Yuan Qi MIT Media Laboratory.](https://reader036.fdocuments.in/reader036/viewer/2022062718/56649e765503460f94b7803b/html5/thumbnails/7.jpg)
Emotions give rise to changes that can be sensed
Sensing: Posture
Gestures, movement, behavior
Skin conductivity
Pupillary dilation
Up-close Respiration, heart rate, pulse
Sensing: Temperature
Blood pressure
Internal Hormones
Sensing: Neurotransmitters
…
![Page 8: Machines that Recognize Human Emotion Yuan Qi MIT Media Laboratory.](https://reader036.fdocuments.in/reader036/viewer/2022062718/56649e765503460f94b7803b/html5/thumbnails/8.jpg)
Emotions give rise to changes that can be sensed
Gestures, movement, behavior
Skin conductivity
Pupillary dilation
Up-close Respiration, heart rate, pulse
Sensing: Temperature
Blood pressure
Internal Hormones
Sensing: Neurotransmitters
…
![Page 9: Machines that Recognize Human Emotion Yuan Qi MIT Media Laboratory.](https://reader036.fdocuments.in/reader036/viewer/2022062718/56649e765503460f94b7803b/html5/thumbnails/9.jpg)
Emotions give rise to changes that can be sensed
Skin conductivity Pupillary dilation
Up-close Respiration, heart rate, pulse
Sensing: Temperature
Blood pressure
Internal Hormones
Sensing: Neurotransmitters
...
![Page 10: Machines that Recognize Human Emotion Yuan Qi MIT Media Laboratory.](https://reader036.fdocuments.in/reader036/viewer/2022062718/56649e765503460f94b7803b/html5/thumbnails/10.jpg)
Can a machine tell if a person is bored or interested?
Attentive? Fidgeting?
Application: Computer Learning Companion, Tutor, Mentor
![Page 11: Machines that Recognize Human Emotion Yuan Qi MIT Media Laboratory.](https://reader036.fdocuments.in/reader036/viewer/2022062718/56649e765503460f94b7803b/html5/thumbnails/11.jpg)
Can we teach a chair to recognize behaviors indicative of interest and boredom? (Mota and Picard)
Sit upright Lean Forward Slump Back Side Lean
![Page 12: Machines that Recognize Human Emotion Yuan Qi MIT Media Laboratory.](https://reader036.fdocuments.in/reader036/viewer/2022062718/56649e765503460f94b7803b/html5/thumbnails/12.jpg)
What can the sensor chair contribute toward inferring the user’s state: Bored vs. interested?
9-state Posture Recognition: 89-97% accurateHigh/Low interest, Taking a Break: 69-83% accurate(Results on kids not in training data, 2002)
![Page 13: Machines that Recognize Human Emotion Yuan Qi MIT Media Laboratory.](https://reader036.fdocuments.in/reader036/viewer/2022062718/56649e765503460f94b7803b/html5/thumbnails/13.jpg)
Detecting, tracking, and recognizing facial expressions from video (Kapoor & Picard)
![Page 14: Machines that Recognize Human Emotion Yuan Qi MIT Media Laboratory.](https://reader036.fdocuments.in/reader036/viewer/2022062718/56649e765503460f94b7803b/html5/thumbnails/14.jpg)
Computer recognition of natural head nods and shakes
Kapoor and Picard, PUI ‘01
![Page 15: Machines that Recognize Human Emotion Yuan Qi MIT Media Laboratory.](https://reader036.fdocuments.in/reader036/viewer/2022062718/56649e765503460f94b7803b/html5/thumbnails/15.jpg)
Fully automatic computer recognition of six natural facial “action units”
(Kapoor and Picard)
Accuracy:“Expert” human: 75%Our first system: 67%
![Page 16: Machines that Recognize Human Emotion Yuan Qi MIT Media Laboratory.](https://reader036.fdocuments.in/reader036/viewer/2022062718/56649e765503460f94b7803b/html5/thumbnails/16.jpg)
Can the computer sense mild frustration or distress?
(e.g., for usability testing in the field?)
![Page 17: Machines that Recognize Human Emotion Yuan Qi MIT Media Laboratory.](https://reader036.fdocuments.in/reader036/viewer/2022062718/56649e765503460f94b7803b/html5/thumbnails/17.jpg)
Things to communicate frustration
(Reynolds & Picard)
![Page 18: Machines that Recognize Human Emotion Yuan Qi MIT Media Laboratory.](https://reader036.fdocuments.in/reader036/viewer/2022062718/56649e765503460f94b7803b/html5/thumbnails/18.jpg)
Example: data from pressure mouse
Forthcoming paper w/Jack Dennerlein, Harvard School of Public Health, and Carson Reynolds/Rosalind Picard at MIT, International Ergonomics Association, linking frustration and physical risk factors
![Page 19: Machines that Recognize Human Emotion Yuan Qi MIT Media Laboratory.](https://reader036.fdocuments.in/reader036/viewer/2022062718/56649e765503460f94b7803b/html5/thumbnails/19.jpg)
Can the computer sense other emotions? Stress? Pleasure?…
![Page 20: Machines that Recognize Human Emotion Yuan Qi MIT Media Laboratory.](https://reader036.fdocuments.in/reader036/viewer/2022062718/56649e765503460f94b7803b/html5/thumbnails/20.jpg)
![Page 21: Machines that Recognize Human Emotion Yuan Qi MIT Media Laboratory.](https://reader036.fdocuments.in/reader036/viewer/2022062718/56649e765503460f94b7803b/html5/thumbnails/21.jpg)
Sensing Processing Expression
Wearable skin conductivity communicator
![Page 22: Machines that Recognize Human Emotion Yuan Qi MIT Media Laboratory.](https://reader036.fdocuments.in/reader036/viewer/2022062718/56649e765503460f94b7803b/html5/thumbnails/22.jpg)
Making the light glow:
• Significant thoughts
• Exciting events
• Exercise
• Motion artifacts
• Lying
• Pain
![Page 23: Machines that Recognize Human Emotion Yuan Qi MIT Media Laboratory.](https://reader036.fdocuments.in/reader036/viewer/2022062718/56649e765503460f94b7803b/html5/thumbnails/23.jpg)
Audience’s “Glow” conveys excitement(Approximate Skin Conductivity Level)Audience’s “Glow” conveys excitement(Approximate Skin Conductivity Level)
Communicate emotion in new waysPicard and Scheirer, HCI 2001
![Page 24: Machines that Recognize Human Emotion Yuan Qi MIT Media Laboratory.](https://reader036.fdocuments.in/reader036/viewer/2022062718/56649e765503460f94b7803b/html5/thumbnails/24.jpg)
Cybernetic wearable camera(Healey & Picard, ISWC 98)
![Page 25: Machines that Recognize Human Emotion Yuan Qi MIT Media Laboratory.](https://reader036.fdocuments.in/reader036/viewer/2022062718/56649e765503460f94b7803b/html5/thumbnails/25.jpg)
StartleCam Filter
![Page 26: Machines that Recognize Human Emotion Yuan Qi MIT Media Laboratory.](https://reader036.fdocuments.in/reader036/viewer/2022062718/56649e765503460f94b7803b/html5/thumbnails/26.jpg)
Video: StartleCam(Healey & Picard, ISWC 98)
![Page 27: Machines that Recognize Human Emotion Yuan Qi MIT Media Laboratory.](https://reader036.fdocuments.in/reader036/viewer/2022062718/56649e765503460f94b7803b/html5/thumbnails/27.jpg)
![Page 28: Machines that Recognize Human Emotion Yuan Qi MIT Media Laboratory.](https://reader036.fdocuments.in/reader036/viewer/2022062718/56649e765503460f94b7803b/html5/thumbnails/28.jpg)
Subject intentionally expressing 8 emotions:
1. Neutral 5. Platonic Love2. Anger 6. Romantic Love3. Hate 7. Joy4. Grief 8. Reverence
Each emotion collected daily, for > 4 weeks4 physiological signals:EMG on jaw, skin conductivity, BVP, respiration
Classification Accuracy:81% on 8 emotions (person dependent)Picard et al., IEEE Trans. Pattern Analysis Machine Intell.,Oct 2001.
1. Neutral 5. Platonic Love2. Anger 6. Romantic Love3. Hate 7. Joy4. Grief 8. Reverence
![Page 29: Machines that Recognize Human Emotion Yuan Qi MIT Media Laboratory.](https://reader036.fdocuments.in/reader036/viewer/2022062718/56649e765503460f94b7803b/html5/thumbnails/29.jpg)
![Page 30: Machines that Recognize Human Emotion Yuan Qi MIT Media Laboratory.](https://reader036.fdocuments.in/reader036/viewer/2022062718/56649e765503460f94b7803b/html5/thumbnails/30.jpg)
Autonomic Balance = LF/HF
![Page 31: Machines that Recognize Human Emotion Yuan Qi MIT Media Laboratory.](https://reader036.fdocuments.in/reader036/viewer/2022062718/56649e765503460f94b7803b/html5/thumbnails/31.jpg)
Bayesian Spectrum Estimation of Unevenly Sampled Nonstationary Data
(Y. Qi, T.P. Minka, and R.W. Picard 01) Problem
Estimating spectrum with data that is
• Nonstationary• Unevenly
Sampled• Noisy
Bayesian Approach
Dynamic modeling of the time series
Then the spectrum at time ti can be summarized by the posterior mean of p(si|x1:i ).
)2(
)1(
]2cos,...,2cos
,2sin,...,2sin,1[
],...,,,,...,,[
1
1
1
1010
iiii
iii
iMi
iMii
TiMiiiMiii
vscx
wss
tftf
tftfc
bbbaaas
w i : the process noise at time t i
vi : the observation noise at time ti.The filtering distribution p(si|x1:i ) can be sequentially estimated as
)|(
)|()|()|(
)|()|()|(
1:1
1:1:1
11:1111:1
ii
iiiiii
iiiiiii
xxp
xspsxpxsp
dsxspsspxsp
![Page 32: Machines that Recognize Human Emotion Yuan Qi MIT Media Laboratory.](https://reader036.fdocuments.in/reader036/viewer/2022062718/56649e765503460f94b7803b/html5/thumbnails/32.jpg)
Comparison with Classical Spectrum Estimation Algorithms
Welch
Burg
Music
Multitaper
New
The signal is the sum of 19, 20, and 21 Hz real sinusoid waves with amplitudes 0.5, 1, and 1 respectively. The variance of the additive white noise is 0.1. The signal is evenly sampled 128 times at 50 Hz.
![Page 33: Machines that Recognize Human Emotion Yuan Qi MIT Media Laboratory.](https://reader036.fdocuments.in/reader036/viewer/2022062718/56649e765503460f94b7803b/html5/thumbnails/33.jpg)
Lomb-Scargle periodogram with a
window size of 200 points
Lomb-Scargle periodogram with a window size of 200 points
Spectral analysis for an unevenly sampled signal
The signal frequency jumps from 20 Hz to 40 Hz at the sampling time -0.833 second, and then jumps from 40 Hz to 60 Hz at 0.833 second.
![Page 34: Machines that Recognize Human Emotion Yuan Qi MIT Media Laboratory.](https://reader036.fdocuments.in/reader036/viewer/2022062718/56649e765503460f94b7803b/html5/thumbnails/34.jpg)
Spectrogram by the new method Spectrogram by the new method
coupled with sparsification
The signal frequency jumps from 20 Hz to 40 Hz at the sampling time -0.833 second, and then jumps from 40 Hz to 60 Hz at 0.833 second.
Spectral analysis for an unevenly sampled signal
![Page 35: Machines that Recognize Human Emotion Yuan Qi MIT Media Laboratory.](https://reader036.fdocuments.in/reader036/viewer/2022062718/56649e765503460f94b7803b/html5/thumbnails/35.jpg)
Simultaneously examine physiology and behavior for recognizing level of stress: up to 96% accurate, across 12 drivers.(Healey and Picard, ICPR 2000)
![Page 36: Machines that Recognize Human Emotion Yuan Qi MIT Media Laboratory.](https://reader036.fdocuments.in/reader036/viewer/2022062718/56649e765503460f94b7803b/html5/thumbnails/36.jpg)
Driver Stress Demo(work w/Jen Healey, Yuan Qi,incorporating new spectral estimationtechnique for assessing heart rate variability)
![Page 37: Machines that Recognize Human Emotion Yuan Qi MIT Media Laboratory.](https://reader036.fdocuments.in/reader036/viewer/2022062718/56649e765503460f94b7803b/html5/thumbnails/37.jpg)
Stress is evident for this person when:driving through cityturning around at toll boothhearing siren
New algorithm: analysis of heart-rate variability via real-time spectrum estimation with missing and irregularly sampled data (Qi and Picard, 2001)
![Page 38: Machines that Recognize Human Emotion Yuan Qi MIT Media Laboratory.](https://reader036.fdocuments.in/reader036/viewer/2022062718/56649e765503460f94b7803b/html5/thumbnails/38.jpg)
Goal: recognize stress in speech of driver,over cell phone headset.
![Page 39: Machines that Recognize Human Emotion Yuan Qi MIT Media Laboratory.](https://reader036.fdocuments.in/reader036/viewer/2022062718/56649e765503460f94b7803b/html5/thumbnails/39.jpg)
Recognizing Affect in Speech: Stress
Data: Four drivers talking over cell phone (headset)
Problem: Associate stress with cognitive load of driving/verbal task: 2 speeds of driving (~60 kph, ~120 kph)2 speeds of questioning (every 9 sec, every 4 sec)
Models: Daubechies-4 filterbank: 21 bands, Teager Energy Operator features, Models: HMM, Auto-regressive HMM, Factorial HMM, Hidden-Markov Decision Tree, Support Vector Machine, Neural Network, Mixture of HMM’s
Results: 96% training/62% testing on 4 categories stress with Mixture HMM’s; highly speaker dependent, e.g. 89-100% training, 36-96% test
Fernandez & Picard, ISCA Workshop on Speech and Emotions, Belfast 2000
![Page 40: Machines that Recognize Human Emotion Yuan Qi MIT Media Laboratory.](https://reader036.fdocuments.in/reader036/viewer/2022062718/56649e765503460f94b7803b/html5/thumbnails/40.jpg)
Extralinguistic Markers
BreathsPauses
F0
Intonation
Syllables
Tempo
Rhythmicality, …
Understanding the Structure of Spoken Language for Affect Modeling
![Page 41: Machines that Recognize Human Emotion Yuan Qi MIT Media Laboratory.](https://reader036.fdocuments.in/reader036/viewer/2022062718/56649e765503460f94b7803b/html5/thumbnails/41.jpg)
Emotions give rise to changes that can be sensed
FaceDistance VoiceSensing: Posture Gestures, movement, behavior
Skin conductivity Pupillary dilationUp-close Respiration, heart rate, pulseSensing: Temperature Blood pressure
Internal HormonesSensing: Neurotransmitters …
![Page 42: Machines that Recognize Human Emotion Yuan Qi MIT Media Laboratory.](https://reader036.fdocuments.in/reader036/viewer/2022062718/56649e765503460f94b7803b/html5/thumbnails/42.jpg)
Conclusions & Challenges
• Steady progress w/sensors, pattern rec• Put the desires of the user first:
– more visible vs. less visible signals– non-tethered, wearable, portable, – psychological comfort– cognitive load/interruptions
• Still to combine w/additional context sensing & cognitive reasoning
![Page 43: Machines that Recognize Human Emotion Yuan Qi MIT Media Laboratory.](https://reader036.fdocuments.in/reader036/viewer/2022062718/56649e765503460f94b7803b/html5/thumbnails/43.jpg)
Papers and projects/details:
http://www.media.mit.edu/affect
http://www.media.mit.edu/~yuanqi
•Machines that “have emotion”
•Emotion and consciousness
•Concerns
•Applications
•How to sense, recognize, build
•Modeling emotion
•Affective wearables