Using Electrodermal Activity to Recognize Ease of...
28
Using Electrodermal Activity to Recognize Ease of Engagement in Children during Social Interactions Javier Hernandez Ivan Riobo Agata Rozga Gregory D. Abowd Rosalind W. Picard Expeditions in Computing
Transcript of Using Electrodermal Activity to Recognize Ease of...
Using Electrodermal Activity
to Recognize Ease of
Engagement in Children
during Social Interactions
Javier Hernandez Ivan Riobo
Agata Rozga
Gregory D. Abowd
Rosalind W. Picard
Expeditions in
Computing
3
Can we characterize qualitative
aspects of children’s social
engagement with wearable biosensors?
Contents • Background
• Data Collection
• Response Characterization
• Engagement Recognition
• Conclusions
4
Background
Engagement Measurement
6
Self-reports
Fast
Direct
Subjective
Recall problems
Disruptive
Not adequate for children
Engagement Measurement
7
Self-reports
Fast
Direct
Subjective
Recall problems
Disruptive
Not adequate for children
Continuous
Widely used for faces
Time intensive
Costly
Training
Coders
Engagement Measurement
8
Self-reports Physiology
Less disruptive
Objective
Difficult to analyze
Instrumentation
Unfamiliar
Fast
Direct
Subjective
Recall problems
Disruptive
Not adequate for children
Continuous
Widely used for faces
Time intensive
Costly
Training
Coders
Engagement Measurement
9
Physiology Wedel & Pieters, 2000
LaBarbera, Tucciarone, 1995
Silveira et al, 2013
Hernandez et al 2013
Lang, 1990
Social Engagement
10
Interactional Synchrony “Coordination of behaviors between individuals during social interactions.”
Physiological linkage
Levenson & Gottman , 1983
30 married couples
Marci et al, 2007
20 patient-therapist dyads
Data
Collection
12 Project: http://cbs.gatech.edu
Computational Behavioral Science Modeling, Analysis, and Visualization of Social and Communicative Behavior
Dataset: http://cbi.gatech.edu/mmdb
Good Indicator
• Arousal
• Cognitive Load
• Wireless
• Comfortable
• 32 Hz
• 4 sensors
Limitations
• Specificity
• Artifacts
Electrodermal Activity
Affectiva QTM
Social Interaction Greet Ball Play Book Hat Tickle
5 scripted activities with an adult
External coder
“Amount of effort required to get child’s attention” Ratings for each Activity
• 0 – little effort
• 1 – some effort
• 2 – extensive effort 14
Social Interaction
23 sessions exclude due to artifacts
51 clean sessions 24 males, 27 females
Average age: 21 months (STD: 5.23)
Average duration: 2.72 minutes (STD: 1.02)
Engagement groups
• Easier to engage (N = 29)
• Harder to engage (N = 22)
Greet Ball Play Book Hat Tickle
15
EDA Response
Characterization
Preprocessing 1. Normalize EDA values (Lykken, 1971)
2. Reduce noise (Hanning filter, 1 sec. window)
3. Extract tonic and phasic EDA (Benedek and Kaembach, 2010)
17
EDA
Phasic Tonic
Feature Extraction
18
Ton
ic/P
ha
sic
C
om
po
ne
nts
Time
Child Adult
Synchrony Features
Individual Features
Mean
# Peaks
Area under the curve
Position Maximum
Position Minimum
Standard Deviation
Slope
Average Peak Amplitude
Pearson Product-moment Correlation (standard)
Canonical Correlation (explores different representations)
Dynamic Time Warping (allows for temporal flexibility)
L2-norm between means
L2-norm between #peaks
L2 norm between averages of peak amplitudes
Ton
ic/P
ha
sic
C
om
po
ne
nts
Feature Extraction
19
Ton
ic/P
ha
sic
C
om
po
ne
nts
Time
Child Adult
Synchrony Features
Individual Features
Mean
# Peaks
Area under the curve
Position Maximum
Position Minimum
Standard Deviation
Slope
Average Peak Amplitude
Pearson Product-moment Correlation (standard)
Canonical Correlation (explores different representations)
Dynamic Time Warping (allows for temporal flexibility)
L2-norm between means
L2-norm between #peaks
L2 norm between averages of peak amplitudes
Ton
ic/P
ha
sic
C
om
po
ne
nts
Feature Extraction
20
Ton
ic/P
ha
sic
C
om
po
ne
nts
Time
Child Adult
Synchrony Features
Individual Features
Mean
# Peaks
Area under the curve
Position Maximum
Position Minimum
Standard Deviation
Slope
Average Peak Amplitude
Pearson Product-moment Correlation (standard)
Canonical Correlation (explores different representations)
Dynamic Time Warping (allows for temporal flexibility)
L2-norm between means
L2-norm between #peaks
L2 norm between averages of peak amplitudes
Ton
ic/P
ha
sic
C
om
po
ne
nts
Engagement
Recognition
Experiment Details
22
10 fold cross validation for training and testing
http://www.csie.ntu.edu.tw/~cjlin/libsvm/
Easier to Engage VS Harder to Engage
Misclassification Costs: 2^[ 0:1:18]
Support Vector Machines
Linear Kernel
Easier: 29 samples
Harder: 22 samples
Boser et al., 1992
Cortes and Vapnik, 1995
Validation
Sequential Feature Selection
0 20 40 60 80 100 120 1400.25
0.3
0.35
0.4
0.45
0.5
0.55
0.6
0.65
0.7
0.75
Avg. Time (seconds)
Avg
. N
orm
aliz
ed
ED
A
0 20 40 60 80 100 120 140 1600.25
0.3
0.35
0.4
0.45
0.5
0.55
0.6
0.65
0.7
0.75
Avg. Time (seconds)
Avg
. N
orm
aliz
ed
ED
A
Avg. Normalized EDA
(±Standard Error)
Easier to Engage Children Harder to Engage Children
Average EDA Response
23
Results
50
55
60
65
70
75
80
85
90
IF+SF IF SF IF SF
IF: Individual Features (only from the child’s EDA) SF: Synchronized Features (from the dyad’s EDA)
Best performance 69% Similar performance for both IF and SF IF better from tonic SF better from phasic Best IF: Position Max. from Tonic Best SF: #Peaks from Phasic (>13% than correlation) Tonic and Phasic decomposition improves >6% and interpretability
One feature at a time
Av
erag
e A
UC
s (R
OC
s an
d P
reci
sio
n/R
ecal
l)
24
Results
50
55
60
65
70
75
80
85
90
IF+SF IF SF IF SF
IF: Individual Features (only from the child’s EDA) SF: Synchronized Features (from the dyad’s EDA)
Feature selection improves >11% SF slightly better than IF (>2%) Tonic and phasic equally represented SF and IF are complementary (81%) Some of most selected features: Difference between peaks from phasic STD and DTW from tonic
Combinations of features
Av
erag
e A
UC
s (R
OC
s an
d P
reci
sio
n/R
ecal
l)
25
Conclusions
• 51 child-adult dyads in a structured social interaction
• Evaluation of individual and synchrony features from
tonic and phasic components
• Promising results using SVMs and feature selection (81%)
• Main benefits: automated and scalable • Biggest challenge: 31% of sessions had to be excluded
• Comparison with other modalities
• Goals: better quantify and understand behavior, and
detect developmental delays
Conclusions
27
[email protected] [email protected]
Javier Hernandez
Ivan Riobo
Agata Rozga
Gregory D. Abowd
Rosalind W. Picard
Using Electrodermal Activity to
Recognize Ease of Engagement in
Children during Social Interactions
Expeditions in Computing
Project: http://cbs.gatech.edu Dataset: http://cbi.gatech.edu/mmdb
