California Educational Research Association (CERA)

San Francisco, CA – November 19, 2009

Gregory K.W.K. Chung

Research Issues in Developing Games for Learning and Assessment

2 / ∞

Overview

• Project overview

• Why study games for learning?

• Tensions along the way

• Some design variables

• Study results

• Conclusion and next steps

2

3 / ∞

Project Overview

• Center for Advanced Technology in Schools (CATS)

• USC Game Innovation Lab

• R&D focused on games and simulations for learning and assessment

• Content focus is pre-algebra (rational numbers, solving equations, functions)

• Target population is underprepared students

• Systematic testing of features (instructional variations, game-based) before full-scale implementation

4 / ∞

Why Study Games for Learning?

• If you build it, they will play (and learn) ...

• Given: Students choose to spend hours playing games

• Idea: Let’s put academic content in games

• Magic: Students will play the game, be engaged in the game, and will learn the stuff

• fait accompli

• Recall scantron (1950s), word processors (1980s), calculators (1980s), OPAC (1980s), Web (1990s) ...

• It’s going to happen with or without R&D, so let’s figure out ways to shape the process

5 / ∞

Why Study Games for Learning?

• Help determine the relationship among:

• Different instructional design variables AND

• Different game design variables AND

• Different types of learning outcomes AND

• Different types of students AND

• Different types of game outcomes

6 / ∞

Tensions: Games for Learning Math

• game <–--> learning

• fun <–--> math

• play time <–--> efficiency

• choose to play <–--> have to play

• “pure” math <–--> “applied” math

• basic skills <---> 21st century skills

• simple tasks <–--> complex tasks

• unobtrusive measures (embedded) <---> obtrusive measures (external)

6

7 / ∞

The R&D Challenge

7

Math outcomes

• Skills

• Conceptual understanding

Game outcomes

• Game level

• Gaminess

Instruction

• Tutorial

• Feedback

Core mechanics

• Must use math

Motivational elements

• Bling

?

8 / ∞

Game Design Variables

• Feedback

• Type

• Timing

• Precision

• Impasse-driven

• In-game Assessment

• Scoring

• Performance sensing

8

• Instruction

• Game mechanics

• Conceptual

• Procedural

• Core mechanics

• Part of game

• Motivation

• Bling

9 / ∞

Outcome Variables

• Math outcomes

• Skills

• Conceptual understanding

• Game outcomes

• Student perception of “gaminess”• Flow

• Game level

9

Prototype Gamelet

11 / ∞

Game Design Requirements

• The Outcome

• Conceptual and computational fluency with rational numbers (fractions)

• The Math

• Idea of “unit” and fractional parts

• Additive operations

• Denominator no. of pieces in 1 unit

• Numerator no. of pieces

• Equivalence

• The Challenge: How to do math without killing the game

12 / ∞

Prototype Game Design

• Genre

• Puzzle—need to figure out how to navigate from start to end points

• Game and Learning Mechanics

• Jumping/bouncing from point to point

• Adding coils to go from point to point

• Only allowed to add pieces of the same fractional size (i.e., common denominator)

• Need to convert among equivalent units (2/2 = 3/3 = 4/4)

13 / ∞

14 / ∞

15 / ∞

Study

17 / ∞

Research Study

• Research Question

• To what extent do different kinds of feedback affect understanding of fractions (i.e., unit), game performance, and perception of game play?

• Design

• 2 conditions that varied feedback

• Gamey: Minimal math instruction

• Mathy: Emphasized math concepts related to unit

18 / ∞

Sample

• Sample

• N = 137

• 9th (30%); 10th (18%), 11th (31%), 12th (15%)

• Amount of weekly game play

• 0hr (21%); 1-2hr (40%); 3-6hr (19%); > 6hr (23%)

• Math achievement

• Self-reported grades: A’s and B’s (55%), C’s (31%), D’s and F’s (13%)

• Math pretest: M = 6.34, SD = 3.39, Min. = 0, Max. = 11

19 / ∞

Measures

• Math outcome

• Pretest, posttest

• Game outcome

• Last level reached, perception of game

• Game process measures

• Time, correct fraction additions, incorrect fraction additions

• Background

20 / ∞

Results

• Did we build a game?

• Did students learn from the game?

• Was there an effect of type of feedback on:

• Learning?

• Game performance?

• Game perception?

Did we build a game?

Yes

23 / ∞

Results

24 / ∞

Results

25 / ∞

Results

Did students learn from the game?

It depends

28 / ∞

Did students learn from the game?

• No overall effects of game play on math posttest scores

• Not surprising—sample was composed of high and low performers

• However, our target group—low math performers—appeared to profit from game play

• Low performers’ posttest scores (M = 3.08, SD = 2.04) were significantly higher than their pretest scores (M = 2.55, SD = 1.22). t (48) = 2.0, p = .05, d = 0.32.

Was there an effect oftype of feedback on learning?

No

Was there an effect oftype of feedback on game

performance?

Yes

33 / ∞

Was there an effect oftype of feedback on game performance?

• Students in the mathy condition (vs. the gamey condition):

• Appear to have gone further in the game (p = .08, d = 0.31)

• Committed more correct additions (p = .003, d = 0.49)

• Committed fewer incorrect additions (p = .007, d = 0.48)

Was there an effect oftype of feedback on game

perception?

Probably

36 / ∞

Was there an effect oftype of feedback on game performance?

• Students in the mathy condition (vs. the gamey condition):

• Perceived the game as more game-like (p = .08)

• Were more willing to use the game as part of school work (p = .06)

• Agreed more with the statement that the game helped them understand math (p = .003, d = 0.54)

37 / ∞

Summary

• Did we build a game? (YES)

• Did students learn from the game? (ONLY LOW PERFORMERS)

• Was there an effect of type of feedback on:

• Learning? (NO)

• Game performance? (YES)

• Game perception? (PROBABLY)

38 / ∞

Conclusion and Next Steps

• Beginning to understand conditions under which “mathification” may not hurt game play

• Speculate that math instruction helped students progress in game

• Impasse-driven instruction

• Results promising for the development of a game that includes math content while preserving game aspect

• Need stronger instructional intervention

• Building tutorial, just-in-time feedback

Backup