Riding an emo+onal roller-coaster: a mulmodal study of ... · Riding an emo+onal roller-coaster: a...

Ridinganemo+onalroller-coaster:amul+modalstudyofyoungchild’smath

problemsolvingac+vi+es

Lujie(Karen)ChenCarnegieMellonUniversity

JointworkwithXinLi,ZhuyunXia,ZhanmeiSong(ShangdongYincai)

Louis-PhilippeMorencyandArturDubrawski(CarnegieMellon)

Imagecredit:hKp://markewbie.weebly.com

1

Whycan’tmathbeasport?Enthusiasm

ExpectaQonResources…….

sports

math

2Imagecredit:www.omaha.com;www.cbc.ca

Mathisasport

JohnUrschelProfessionalfootballplayerandMathemaQcian

3Imagecredit:hKp://www.sikids.com/

SportsandmathsharesRecipeforsuccess

•  CulQvateinterestatyoungage•  Need“deliberatepracQce”[Ericsson1993]•  MakepracQcefun•  MakepracQcechallenging

4

WhatismathpracQce?

5

MOEMS

Imagecredit:www.groton.k12.ct.us

AsampleMathKangarooproblem

Whatmakesmathproblemsolvinganuniqueexperienceforayoungchild?

6

AcQveexploraQonPassivesupport

Imagecreidt:hKp://americanpregnancy.org/

EmoQonalroller-coaster

HowtoopQmizetheexperiencethroughpersonalizedsupport?

7

8

When?(Assistancedilemma[Koedinger&Aleven2007])

What?(cogniQveand/oremoQonal)

“LearningCompanion”thatcanbe“watchfulforthetrajectoryandprovideappropriatelevelofsupport”[Kapoor2001]

Researchgoals

•  ExploretheuQlityoffinegrainedmul+modalsignalsfromvideorecordingstocharacterizechild’sproblemsolvingbehaviors

9

•  DemonstrateanmulQmodalanalysisframeworkonthetemporaldynamicsofinterpersonalcommunicaQonandemoQonalprofile

•  Exploreopportunityfor“scalingup”tolargenumberofsubjectsbyvalidaQngautomaQcdetectors

DatacollecQonprotocol•  VideotapedinteracQonbetweena9-year-oldboyandhismom

•  Hetriedtosolveoneproblemineachofthesessionandhismomprovidedsupportasneeded

•  21sessionsrecordedover3weeks•  ProblemsareselectedfromMathKangaroocompeQQon

BabyRoowrotedownseveralnumbersusingonlythedigits0and1.Thesumofthesenumbersis2013.Itturnedoutthatitisimpossibletogetthesamesumbyaddingupfewernumbersofthiskind.HowmanynumbersdidBabyRoowrite?(A)2(B)3(C)4(D)5(E)204

Anexampleproblem

10

C:Idon’tgetthis

M:Doyouneedhelp?

A

B

C

D

C:Oh!Isee

Problemunderstanding

PlanningandExecu+on Solvedsupport support 11

(FACET)

Roadmap

12

<Visual>Facial

Expression

<Visual>Intent-to-Connect

(ITC)

<Vocal>Momandchild’s

talkAnnotaQons FACET manual manual

InterpersonalcommunicaQondynamics

Yes Yes

Videolevelanalysis Yes Yes Yes

PredicQvemodelingofresponse

Yes Yes Yes

AutomaQcRecogniQon

WithmanualannotaQon

WithOpenFaceFeatures

WithCOVAREPFeatures

Roadmap

13

<Visual>Facial

Expression


(ITC)




Yes Yes



Yes Yes Yes

AutomaQcRecogniQon

WithmanualannotaQon


WithCOVAREPFeatures

VideoID

Floorsharingmetrics

Blue:ChildtalkRed:Momtalk

14

TemporaldynamicsofITCandvoiceacQvity(momandchild’stalk)

video12 video17

0

500

1000

1500

2000

0 2 4 6 0 1 2 3Timestamp (in minutes)

Inte

nsity

child_talk ITC mom_talk

15

SynchronizaQon:ITCvsChildtalkvs.Momtalk

12

13

14

15

16

17

18

20

21

2627

28

29

30

31

32

33

34

35

36

37

−1.0

−0.5

0.0

0.5

1.0

−1.0 −0.5 0.0 0.5 1.0ITC vs Mom talk correlation

ITC

vs

Chi

ld ta

lk c

orre

latio

n

16

Corr(ITC,mom’talk)>Corr(ITC,child’stalk)

Corr(ITC,mom’talk)

<Corr(ITC,child’stalk)

“disengaged”

“engaged”

Roadmap

17

<Visual>Facial

Expression


(ITC)




Yes Yes


PredicaQveofmodelingofresponse

Yes Yes Yes

AutomaQcRecogniQon

WithmanualannotaQon


WithCOVAREPFeatures

xxx:computedfromfulllengthofvideoxxx_b:computedfromfirst30secsofvideoxxx_e:computedfromlast30secsofvideo

VideolevelanalysisviapairwisecorrelaQon

F S CJ EmoQonstates

InterpersonalcommunicaQonbehaviors

PosiQvecorrelaQon

NegaQvecorrelaQon

•  Edgespreservedifthepvalue<0.05from1000itera8onofrandomiza8ontest•  Widthofedge~magnitudeofcorrela8on

18



F S CJ EmoQonstates


PosiQvecorrelaQon

NegaQvecorrelaQon


19



F S CJ EmoQonstates


PosiQvecorrelaQon

NegaQvecorrelaQon


20



F S CJ EmoQonstates


PosiQvecorrelaQon

NegaQvecorrelaQon


21

Roadmap

22

<Visual>Facial

Expression


(ITC)




Yes Yes



Yes Yes Yes

AutomaQcRecogniQon

WithmanualannotaQon


WithCOVAREPFeatures

PredicQvemodelforITCresponse(output)

•  Output:ifmomrespondswithin5secondsgivenanITC

23

ITC

5secs

responseOutput=1

ITC

5secs

responseOutput=0

TotalnumberofITCs:696(5ITC/mins)Responserate:57%

+me

•  ITCco-occurrence–  numberofotherITCwithin2,5,10secondsofthecurrentITC

•  OverlapstaQsQcs–  numberofchildtalkand/ormomtalkeventsthatare

overlappingwiththecurrentITC

•  Headpose–  min/max/median/meanofdetecQonsuccess,confidence,Qlt,

turn,up-down,within5secondsofITC

•  FACETaffectdetector–  min/max/median/meanofemoQonscoreofjoy,surprise,

confusion,frustraQonandbaseline,within5secondsofITC

24

PredicQvemodelforITCresponse(inputfeatures)

25

PredicQvemodelforITCresponse(modelperformance)

•  Allmodelsoutperformrandomclassifier•  NaïveBayesandLogisQcRegressionwithslightadvantages•  Whenmodelfurtherimproved,itcanbeusedtoprovideonline

decisionsupportforateacheroraspartofafullyautomaQcagent(learningcompanion)

26

tSNEembeddingusingtheemoQoncontext(affectvector)ofITC

EvidenceofITCheterogeneity

Joy

Baseline

ConfusionFrustra+on

Surprise

Roadmap

27

<Visual>Facial

Expression


(ITC)




Yes Yes



Yes Yes Yes

AutomaQcRecogniQon

WithmanualannotaQon


WithCOVAREPFeatures

AutomaQcrecogniQonofITCandvoiceacQvityarepromising

28

ITCmodelinputs:poseesQmaQonfeaturesfromOpenFace[BaltrusaiQs2016]Voiceac+vitymodelinputs:COVAREPfeatures

Basedonrandomsampleof500posiQvesamplesand500negaQvesamples

Individual video 10-fold CV

AffectRecogniQonisnotasreliable

FACETlabelbaselineconfusionfrustraQon joy surprise Total

Human

Lab

el#1

baseline 20 2 5 3 30

confusion 8 4 12

frustra+on 1 7 5 1 14

joy 1 1 20 1 23

surprise 11 11

Not_Sure 9 12 15 7 17 60Total 30 30 30 30 30 150Accuracy 67% 27% 17% 67% 37%

FACETLabelbaseline confusion frustra+on joy surprise Total

Human

Lab

el#2 baseline 28 6 12 8 13 67

confusion 14 4 18

frustra+on 6 11 1 18

joy 2 3 3 22 6 36

surprise 1 10 11

Total 30 30 30 30 30 150

Accuracy 93% 47% 37% 73% 33%

30

PairwisecorrelaQonofFACETscores

Consistentwithfindingsfrom[Bosch&D’Mello2015]:Confusionco-occurswithfrustraQon

Takeaway#1InterpersonalcommunicaQondynamics

31

video12 video17

0

500

1000

1500

2000

0 2 4 6 0 1 2 3Timestamp (in minutes)

Inte

nsity

child_talk ITC mom_talk

12

13

14

15

16

17

18

20

21

2627

28

29

30

31

32

33

34

35

36

37

−1.0

−0.5

0.0

0.5

1.0

−1.0 −0.5 0.0 0.5 1.0ITC vs Mom talk correlation

ITC

vs

Chi

ld ta

lk c

orre

latio

n

•  WedevelopedmethodtocharacterizetheinterpersonalcommunicaQondynamicsandquanQfythesynchronizaQonamongITCandvoiceacQvity

•  ThesemetricscanbeusedtoobjecQvelymeasuretheinteracQonqualitybetweenstudentandteacheraswellaschild'saKenQondrix

Takeaway#2MulQmodalvideolevelanalysis

32

•  WeexploredtheassociaQonamonginteracQondynamicsandaffecQvestatesandQmetakentosolvetheproblem

•  ThisanalysisshedlightonpotenQalfactorsthatcouldimpactthechild’semoQonalprofilethusprovidesclueshowtoopQmizeemoQonexperience

Takeaway#3MulQmodalresponsemodel

33

ITC

5secs

responseOutput=1

ITC

5secs

responseOutput=0

•  ThepredicQvemodelsforresponseperformbeKerthanrandom,thoughtheyhaveyetroomforimprovement

•  Areliablemodelassuchcanprovideonline/realQmedecisionsupportforteacher,mayalsocontributetothevisionof“LearningCompanion”(Kapoor2001)

Takeaway#4AutomaQcdetecQon:opportunityforscalingup

34

•  ReliabledetectorsforITCandvoiceacQvityusingOpenFaceandCOVARAPfeatures

•  FACET’sdetecQonofsurpriseandjoyismorereliablethanconfusionandfrustraQon

Futurework

•  Expandthestudytootherchildrenandexplorethepossibilitytotransfermodelfromonechildtoanother

•  Incorporatemeasurementsassuchasgrit/perseverance

•  ExploreaddiQonalmodaliQes–  Verbalmessage(senQmentanalysisthroughAutomaQcSpeechRecogniQon)

–  Non-verbalvocalmessage(prosodyfeaturesofspeechsignals)

35

36

“Whyshouldwelearnmaththroughproblem-solving?toputthemagicbackintomath”---Kalman&Res8vo,DirectorsfromMOEMS(MathOlympiadforElementaryandMiddleSchool)

Riding an emo+onal roller-coaster: a mulmodal study of ... · Riding an emo+onal roller-coaster: a...

Documents

Transcript of Riding an emo+onal roller-coaster: a mulmodal study of ... · Riding an emo+onal roller-coaster: a...