1

What do we Know about the Effectiveness of Instructional

Strategies in Computer Games

Harold F. O’Neil

and

Joan (Yuan-Chuang) Lang

University of Southern California/CRESST

AERA v.4New YorkMarch 2008

2

CRESST Model of Learning

Content Understanding

Learning

Communication

Collaboration/Teamwork

Problem Solving

Self-Regulation

Instructional Strategy

• Definition– Prescribed sequences and methods of instruction to achieve a

learning objective

• Source Spector, J.M., Merrill, M.D., van Merrienboer, J., & Driscoll, M.P. (2008). Handbook of Research on Educational Communications and Technology. New York: Taylor & Francis Group.

• Macro vs Micro Instructional Strategies- Do games work? (macro)

- What strategies within games work? (micro)

3

Macro Instructional Strategies Issues

• Integrate into Syllabus

• Instructor training

• accuracy re standards – Limited role for

fantasy

• Integrate into recreation/family time– Increased requirement for

effective micro instructional strategies (feedback, worked examples)

• Student “training” need (e.g., self-monitoring)

• Flexible– Fantasy role

If in class If out of class

4

Macro Instructional Strategies

• Motivation provided by instructor– Game can be less fun

• Good application– Initial acquisition

training

• Available time in hours/days/weeks

• Motivation provided by student/game– Game needs to be fun

• Good application– Refresher training;

prevent skill decay

• Available time in minutes/hours

If in class If out of class

5

6

Research Questions

• Will adding effective instructional strategies to commercial off the shelf games improve problem solving?

• Trade-off between development and selection

7

Do Games Teach?: Check Validity of Micro Instructional Strategy

• Embedded in game

– Usually inductive discovery approach

— Usually doesn’t result in learning (Kirschner, P. A., Sweller, J., & Clark, R. E. 2006. Why minimal guidance during instruction does not work: An analysis of the failure of constructivist, discovery, problem-based, experiential, and inquiry-based learning. Educational Psychologist, 41, 75-86.)

• What Works in Distance Learning (O’Neil, 2005)

– Good instructional practices that can be applied to games

8

Selection of Game for Research

• Off-the shelf games lacking learning objectives and assessment of learning

• Use wrap around instructional & assessment strategies as no access to source code

9

Common Methodology

• Participants: – Young adults selected to have no

experience of playing SafeCracker but game players

• Measures: – Knowledge mapper – Retention and transfer questions analogous

to Mayers’ – Trait self-regulation questionnaire

10

Domain Specific Problem-Solving Strategy Measures

• Retention QuestionList how you solved the puzzles inthe rooms.

• Transfer Question

List some ways to improve the funor challenge of the game.

• Modifications of previous researchers (Mayer, 2001; Mayer & Moreno, 1998; Moreno & Mayer, 2004)

11

Knowledge Mapper

12

Measurement of Self-Regulation

• Trait self-regulation questionnaire (O’Neil & Herl, 1998). – planning – self-checking – self-efficacy – effort

13

Study I, II, &III

• Study I– Without effective instructional strategies.

• Study II– With worked examples.

• Study III– With just-in time worked examples

O'Neil, H. F. and Perez, R. S. (Eds.) (2008). Computer Games and Team and Individual Learning. Elsevier

14

Study I:Discussion/Implications• There was an increase in problem-solving. But it

was small.

• Existing instructional strategies (discovery learning) in the game were not effective.

• More research on a game designed with effective research-based instructional strategies

– Worked examples (Danny Shen)

– Pictorial aids (Richard Wainess)

– Just-in-Time Worked Examples (Joan Lang)

– After-Action Review

15

Study II and Study III: Worked Examples

• Worked examples are procedures that focus attention on problem states and associated operators (i.e., solution steps), enabling students to induce generalized solutions or schemas (Sweller, 1998).

• Many researchers investigated the efficacy of using worked examples in classroom and computer-based instruction and provided evidence of the effectiveness of worked examples instruction (Cooper & Sweller, 1987; Mayer & Mautone, 2002; Ward & Sweller, 1990).

• No research used worked examples in a game-based environment.

16

Study Worked Example: Discussion/Implications

• The worked example group significantly improved more than the control group in content understanding and problem-solving strategies. However, the improvement was small.

• This study provided evidence that using worked examples could be one of the good instructional methods to facilitate adults’ problem solving with a commercial off-the-shelf computer game.

17

Just in Time Worked examples

What worked examples can do?

• Facilitate learning of problem solving in the computer game SafeCracker®

• Contribute to superior content understanding in problem solving (when presented just-in-time)

• May contribute to superior retention skill (partial evidence) but not transfer skill

18

What Are Continuing R&D Issues?

• Can we leverage game technology for training?

– Embedded instructional and assessment strategies

– Wrap-around instructional and assessment strategies

19

Walk Issues• Analytically, would you predict that commercial off-the-shelf

games should teach?

– No

• What support and guidance would help training game developers to do a better job?

- Alignment with What Works in Distance Learning

• Instructional strategies that could work

• Wrap-around or embedded instructional and assessment strategies

– Worked examples

– feedback

20

CRESST Web Site

http://www.cresst.org or any search engine: type CRESST

honeil@usc.edu

21

What Is a Game?

• A computer game consists of four key components– Settings that are real or imaginary– Roles or agendas for the participants– Rules (real life vs. imaginative)– Scoring, recording, monitoring, or other kinds

of systematic measurement

• Motivation comes from challenge, complexity, fantasy

What Works in Distance Learning*

• Develop a core set of research-based DL guidelines and lessons

– Instructional Design (Clark)*

– Multimedia (Mayer)*

– Learning Strategies (Dembo et al.)*

– Assessment (Baker et al.)*

– Management Strategies (Kazlauskas)*

– Self-Regulation and Motivation (O’Neil et al.)*

– Team Training and Assessment (Cannon Bowers)

– Team Motivation (Clark)

*O’Neil, H. F. (Ed.). (2005). What works in distance learning: Guidelines. Greenwich, CT: Information Age Publishing Inc.

22

23

Sample of a Worked Example

The Note from switchsafe will show on the screen

Notice that some codes are missing

Click the Note to close it

Search for the code related to the Liberty

Safe Find a Note from switchsafe in the toolbox

Click the Note from switchsafe

24

The Specification of What We Are Teaching Is Essential

• From goal/objective of teaching leadership, situational awareness, decision making, tactical problem solving

– The instructional strategies follow

• Nature of feedback, timing of feedback, take-home packages, instructor training, homework assignments, etc.

– The type of assessment follows

• Different assessment measures, after-action reviews

25

SafeCracker

• Puzzle-solving game– Example of problem solving

• No special background knowledge, motor skills, or extraordinary visual-spatial ability required

• Adult-oriented• Single-player game• Pacing controlled by players• Not popular

26

Results: Content UnderstandingBold = as hypothesized

• The JIT worked example (M = 7.97) group demonstrated significantly better (p = .01) content understanding than the worked example group (M = 6.63) and the control group (M = 6.51)

• The worked example group did not demonstrate significantly better content understanding than the control group

27

Results: Problem Solving Strategies

Retention skill:• The worked example group (M = 2.72, SD

= .72) demonstrated significantly greater retention skill than the control group (M = 2.03, SD = .88) by opening significantly more worked-example-illustrated safes (p = .01)

Transfer skill:• No difference among groups