Learning Analytics Applied to Serious GamesEuropean Conference in Game-Based Learning (ECGBL 2013), Porto, Portugal

Baltasar Fernandez-Manjon, balta@fdi.ucm.es , @BaltaFMe-UCM research group, www.e-ucm.es

2

About me and context

‣ CS Professor at Complutense U.• Director of e-UCM

‣ e-UCM research group about Learning technologies

• 15 researchers• Serious games

- Application to the medical domain• European projects

- GALA- SEGAN- CHERMUG

• www.e-ucm.es

3

Learning analytics and SG?The NMC Horizon

Report: 2013 Higher Education Edition

‣ new and emerging technologies on teaching, learning, and research

Time-to-Adoption Horizon: Two to Three Years

‣ Games and Gamification

‣ Learning Analyticshttp://www.nmc.org/publications/2013-horizon-report-higher-ed

4

Serious Games use?

‣ Serious Games have probed to be educationally effective in several domains

• Medicine, military, business, corporate training‣ But still is a low adoption of Serious Games

• Cost? ROI?‣ Serious Games considered usually as a

complementary content• Mainly used for motivational purposes• No actual impact on the final mark

‣ Difficult to include Serious Games in the learning curriculum

• Assessment of acquired learning?

5

Black box model‣ Games as

independent pieces of content

‣ No information about what is happening during the in-game play

‣ Or very simple

• Completed or not completed

• Time used

del Blanco et al (2013). Using e-Learning standards in educational video games. Computer Standards & Interfaces 36 (1) pp. 178–187

6

Learning Analytics

‣ Improving education based on data analysis

• Data driven

‣ Evidence-based education

‣ Related with …• Educational data mining

• Business intelligence

• Visual analytics

www.ed.gov/edblogs/technology/files/2012/03/edm-la-brief.pdf

7

Different uses

‣ Learning analytics• Data analysis that helps students improve

learning outcomes.‣ Academic/program analytics

• Data analysis that provides information of what is happening in a specific program and how to plug holes or otherwise adjust.

‣ Institutional analytics• Data analysis that helps make decisions about

how to improve at the institutional level.

Learning Impact Blog, Big data: Cool; Small data: Cooler http://www.imsglobal.org/blog/?p=258

8

When data is analyzed?

‣ Off-line• Analyzing data after use• Discovering patterns of use• Allows to improve the experience for future

‣ Real time• Analyzing data while the system is in use to

improve/adapt the current learning experience• Allows to also use it in actual presential classes

‣ Mixed approach

9

Extensive Data

‣ Large number of participants‣ Relatively limited number of variables‣ Usually very little demographic information‣ Relatively few observations for each user‣ Wide but shallow data set

Adapted from: Learning Analytics and Educational Data Mining WorkshopNew York University – CREATE Lab April 4–5, 2013

10

Intensive Data

‣ Relatively low number of participants‣ Large number of observations for each

variable‣ Large number of variables for each

participant, such as• User actions, In-Game Events• Survey responses; Extensive demographic

information• Video Observations• Biometric Data (HR, RESP, GSR, EEG, EKG)• Eye-tracking

‣ Narrow but deep data set

‣ Correlations among different data?Adapted from : Learning Analytics and Educational Data Mining WorkshopNew York University – CREATE Lab April 4–5, 2013

11

But there is a parallel world …

http://us.battle.net/wow/en/media/screenshots/races?keywords=&view#/goblins04

12

Game Analytics‣ Application of

analytics to game development and research

‣ Telemetry

• Data obtained over distance

• Mobile games, MMOG

‣ Game metrics

• Interpretable measures of data related to games

• Player behaviour

13

Game metrics

‣ User metrics

• Customer

• Player

‣ Performance metrics

• Technical infrastructure

‣ Process metrics

• Development of the game

‣ User metrics

• Generics metrics

• Genre specific metrics

• Game specific metrics

14

Game metrics

Springer, Game Analytics Maximizing the Value of Player Data, pag 22

15

Game requirements for LA‣ Most of games are black boxes.

• No access to what is going on during game play.

‣ We need access to game “guts”• Game state, game variables

‣ Or the game must communicate with the outside world

• Using some logging framework• Not applicable to COTS games (yet)

‣ Mozila Open Badges? http://openbadges.org/

SESSION VARIABLES

PLAYER GAME

VARIABLES

GAME VARIABLES

TRACES

GOALS

From game data to educational information

PLAYER VARIABLES

User Information

Educational System Information

SERIOUS GAME

Sessions

GAME PLAYER SESSIONS+ =

A player plays a game. Produces a game session.

One game session is a set of traces of ONE specific player in ONE specific game.

SESSIONS =

TRACE

TRACE

TRACE

TRACE

TRACE

TRACE

TRACE

TRACE

TRACE

TRACE

TRACE

TRACE

TRACE

TRACE

TRACE

TRACE

TRACE

TRACE

TRACE

TRACE

player starte

d phase

1

player scored

200in

phase 1

player clicked

in Help

button

VARIABLESGENERATO

R

SESSION

Analyzing game sessions

TRACE

TRACE

TRACE

TRACE

TRACE

TRACE

TRACE

TRACE

TRACE

TRACE

TRACE

TRACE

TRACE

TRACE

TRACE

TRACE

TRACE

TRACE

TRACE

TRACE

VAR

VAR

VAR

VAR

VAR

VAR

VAR

VAR

VAR

VAR

GOAL RESULT

GOAL RESULT

GOAL RESULT

GOAL RESULT

GOAL RESULT

GOALSGENERAT

OR

VAR DEFINITI

ONVAR

DEFINITION

VAR DEFINITI

ONVAR

DEFINITION

VAR DEFINITI

ON

VAR DEFINITI

ON

GOALDEFINITI

ON

GOALDEFINITI

ONGOAL

DEFINITION

GOALDEFINITI

ON

Variable generator: User sessions

VAR DEFINITION VAR+ =

TRACE

TRACE

varName:‘beatgame‘, operation: { type:'trace_present‘,traceType: ‘logic', trace: { event:‘end_game’ }}

event: ‘end_game',timeStamp: 1380181610150

usersession.beatgame = true;+ =

Goal Generators: User sessions

VAR

VAR

GOALDEFINITION

GOAL RESULT=

usersession: { beatgame: true, score: 4747}

id: ‘finalResult’result: ‘beatgame && score > 3000’}

usersession: { beatgame: true, score: 4747, goals: { finalResult: true }}

=

+

+

PLAYER * GAME

Analyzing game results

SESSION

SESSION

VAR VAR

VAR VAR

GOAL RESULT

GOAL RESULT

SESSION

SESSION

VAR VAR

VAR VAR

GOAL RESULT

GOAL RESULT

VARIABLESGENERATO

R

GOALSGENERATO

R

VAR DEFINITI

ONVAR

DEFINITION

VAR DEFINITI

ONVAR

DEFINITION

VAR DEFINITI

ON

VAR DEFINITI

ON

GOALDEFINITI

ON

GOALDEFINITI

ONGOAL

DEFINITION

GOALDEFINITI

ON

eAdventure game platform

Open code authoring environment for the production of point-and-click adventure games & immersive learning simulations

Easy to include Learning Analytics in eAdventure games

http://www.chermug.eu

With MGH-Harvard Universityhttp://first-aid-game.e-ucm.es

With ONT, educ@ONT

eAdventure + Learning Analytics

Game Engine

CommunicationAPI

Logic

Input

Input

Logic

Input

{ type: 'input', timeStamp: some_timestamp, device: 'some_device', action: 'some_action', target: 'target_id', data: { key1: value, ...}}

{ type: 'logic', timeStamp: some_timestamp, event: 'some_event', target: 'some_id', data: { key1: value, ...}}

Learning Analytics Database

25

GLEANER

‣ GLEANER: Game Learning Analytics for education research

• Open code framework to capture game traces

Reference model in the EU NoE GALA

GLEANER Collector

‣ Implemented: Nodejs + Mongodb‣ A web server ready to receive traces‣ Cleans and sorts the data

GLEANER Analysis

‣ Reporter has access to the database, and presents its data through reports

• graphics, heat-maps, relational tables…

‣ Evaluator has access to the database and checks the educational defined goals in the assessment model

Example: A game to learn XML

http://gleaner.e-ucm.es/lostinspace/play/index.html

About the game...

‣ A basic platform game to acquire familiarity with XML documents

• Learn the syntax• Understand nesting and attributes• Gain agility writing and reading XML documents

‣ Designed as a complementary activity in a Web Programming Course

• For undergraduate computer science students

‣ Not developed with eAdventure

Understanding the game…

Goal

Main characte

r

Power-ups (new syntax elements)

Write XML snippets here to move the

main character to the goal

Educational goals

‣ Basic understanding of XML Syntax• Markup syntax• Basic documents• Attributes• Complex documents

‣ Basic programing skills• Sequencing• Loops

LA perspective

‣ What does GLEANER trace?• Higher level events• Generic traces & game specific traces• Only those relevant for our learning objectives

‣ The aggregator will filter and transmit:• Level completion events• Each XML fragment submitted by the player

Main dashboard (GLEANER Report)

33

Progress report

34

User Session

Phases complete

d

Current score

Learning achievements

Achievements are updated and

highlighted in real time

Detailed traces

35

Filters XML snipets

The fragments are updated in real time as submitted by the student. The instructor can filter to see only

mistakes, or all the fragments submitted by a specific students.

La Dama Boba : The Game

Based on The Foolish Lady by Lope de VegaThe game is available at http://damaboba.e-ucm.es/ (in Spanish)

Experiment game vs class‣ 757 students

‣ From 8 middle and high schools in Madrid region

‣ Control group and experimental group

39

Formal evaluation pre-post‣ Similar results with

Learning Analytics than with pre-post test?

learning analytics outcomes

41

42

Other issues in LA

‣ Ethical and legal aspects

‣ Security model

‣ Ownership of information• Informing the user

‣ Anonymization of information

‣ Aspects specially relevant if you are working with kids!

43

But there are new especifications and

develpments that could sistematize the work

44

ADL eXperience API (xAPI)‣ Result of Project Tin Can

‣ Tracks experiences, informal learning, real-world experiences (not just completions)

‣ Allows data storage AND retrieval (ex. 3rd party reporting and analytics tools)

‣ Enables tracking mobile, games, ITS, and virtual worlds experiences

‣ Developed by open source communityFrom Damon Regan (ADL) presentation at SINTICE2013

45

Activity Streams

‣ http://activitystrea.ms

‣ Collaboration between Google, Facebook, Microsoft and others

‣ Allows reporting of experiences, not just completions‣ Format: <Actor> <Verb> <Object> (I did this):

Simple Statement: I (actor) watched (verb) a video on protecting employee data (object)

Complex: in the context of [information assurance certification training] with result [timestamp:2013-0618T18:30:32.360Z ].

From Damon Regan (ADL) presentation at SINTICE2013

Reporting

Systems

Assessment

Services

Semantic Analysis

Statistical Services

xAPI Learning Record Store (LRS)

46

From Damon Regan (ADL) presentation at SINTICE2013

‣ Raw data can feed several systems• An LRS• A Learning Analytics System

eAdventure + LA with xAPI

Logic

Input

Logic

Input

Input

Input

Raw data

LRS

Learning Analytics System

StatementsAnalyzer

StatementsAnalyzer

EXPERIENCEAPI

EX

PE

RIE

NC

EA

PI

IMS Global Learning Analytics Interoperability Framework

http://www.imsglobal.org/IMSLearningAnalyticsWP.pdf

50

Conclusions

‣ LA in Serious Games has a great potential from the application and research perspective

• Simplify more complex and complete experiments

‣ LA in Serious Games should benefits from Games Analytics experience and work

‣ Still complex to implement LA in SG• Increases the (already high) cost of the games

‣ Frameworks and new standards specifications could greatly simplify LA implementation and adoption

51

Thank you!

balta@fdi.ucm.es

@BaltaFM

Our current projects

Increasing patient safety using games

Modelling/teaching medical procedures atNational Transplant Organization