Observational and Log Analysis Methods for Assessing Engagement and Affect in Educational Games
-
Upload
games-for-change -
Category
Technology
-
view
300 -
download
1
description
Transcript of Observational and Log Analysis Methods for Assessing Engagement and Affect in Educational Games
Observational and Log Analysis Methods for Assessing Engagement and Affect in Educational
Games
Ryan S.J.d. BakerAssistant Professor of Psychology, Learning Science, and Computer Science
Worcester Polytechnic Institute
2
Many ways to assess engagement and affect
• I’ll discuss two methods our lab uses
3
Quantitative Field Observations (Expert Judgments)
• Repeated 20 second observations of students’ engagement and affect as they use serious game or other learning environment in genuine learning setting– Conducted using peripheral vision/side glances– Good inter-rater reliability: k 0.6-0.8– Include engaged behaviors (collaboration with other students) and
disengaged behaviors (off-task behavior)– Include positive affect (delight, engaged concentration) and negative
affect (boredom, frustration)
• Ecologically valid assessments of how much and when – Students are disengaged– Students experience specific affect
4
Automated Detectors• Models that assess student engagement and affect in real-time or
retrospectively from behavior within software• In our approach, no sensors used
– Improves scalability – lots of data being automatically collected these days– Reduces predictive power for some affective states, relative to sensor-based
detectors• Successful at detecting disengaged behaviors such as off-task
behavior, carelessness, gaming the system• Successful at detecting engaged concentration and boredom in two
learning systems– Plus sensor-free affect detectors for AutoTutor by D’Mello et al. (2008)
• Used in interventions that improve learning outcomes(Baker et al., 2006)
5
Ongoing Project (NSF PSLC)
Quantitative Field
Observation
Affect Basic Research
Comparative Analysis
Completed for intelligent tutors; in process for serious games
Detector Development
6
Use in Research
7
How does student affectdiffer between games and ITS?
(Rodrigo & Baker, 2011)
8
Aplusix .vs. MathBlaster
Matched mathematical content between systems
Student affect assessed using quantitative field observationswith real students in real classrooms
9
Interesting differences in affect
Condition Engaged Concentration Delight
Aplusix 76% 6%
MathBlaster 63% 12%
Proportions of each affective state shown
10
How does social behavior influence affective dynamics in games?(Baker, Moore, et al., under review)
11
Students compete to be first to identify a substance chosen by their opponent
Chemistry Game (Yaron et al., 2010)
Student affect assessed using quantitative field observationswith real students in real classrooms
12
Without Social Behavior(D’Mello et al., 2007; Baker et al., 2010)
Bored
Confused
Gaming the
System
13
With Social Behavior(Baker, Moore, et al., under review)
Off-Task Behavior Bored
Confused
Gaming the
System
On-Task Conversation
14
Bottom-Line
• Field observations and detectors are powerful tools
• For assessing and understanding student engagement and affect during learning
• Including in serious games