Gaming the System in the CALL Classroom

Peter GobelKyoto Sangyo Universitypgobel@cc.kyoto-su.ac.jp

Introduction

J. Carrol (1963)- time on task hypothesis: the learner will succeed in learning a given task to the extent that he spends the amount of time that he needs to learn the task.

Baker et. al (2004a)- off-task behavior associated with reduced learning

Cont.

Baker et. al (2004b)- gaming the system (misuse of tutoring systems) associated with substantially lower learning. 1/3 less than those who do not engage in this behavior.

Learned helplessness and ‘performance orientation’ related to gaming.

Basic Question

Is motivation related to misuse of the system or inappropriate help-seeking?

Research QuestionsIn Japanese university students:

Is frequency of off-task behavior a predictor of test performance?Does the kind of off-task behavior make a difference?Are motivational factors predictors of performance behavior and test performance?

Kyoto Sangyo CALL Curriculum

Students lower proficiency non-English majors

System meeting 90 minutes/week

• DynEd• ALC

Course requirements grade linked to completion of level tests and time spent on software system

Participants

Three intact classes - 105 students

All non-English majors

All streamed at low level

Observation

Off-task behavior (physical and virtual)

Non-system related activityInactivityMisuse of system (gaming)

On-task behavior (physical and virtual)Appropriate system-related activitySeeking help from peers, software, and teacher

Observation (continued)

Total amount of study timeRecorded automatically by software system

Study scoreBased on total amount of study time and ‘proper’ use of the software.

Observation (continued)

Test performance Gain scores on general proficiency test

Listening sectionReading sectionAdministered in April and July

Motivation Questionnaire

35seven-point Likert scale items

administered in Japanese

30 items were written referring mainly to motivational theory

5 items referring to CALL activity

Questionnaire Items

Expectancy for successAttainment ValueIntrinsic value

Extrinsic utility value

CostAttitudes toward target group

Effort

CALL related items

Results

ObservationOff-task behaviorOn-task behaviorTotal study timeStudy score

QuestionnaireStructureMotivational predictors of performanceMotivational predictors of behavior

Classroom Observation

0

10

20

30

40

50

60

70

80

90

1stQtr

2ndQtr

3rdQtr

4thQtr

EastWestNorth

on/off-task averagesoff-T nonsoft5%off-T inactive14%

off-T gaming5%off-T software help0%on-T software75%

pn-T t/peer help1%

Off-task behavior

Off-task non software

Off-task inactive

Off-task software

Help

Study record

Off-task gaming

On-task behavior

On-task softwareMicrophones and speech analyzers seldom usedCertain activities overused and recycled

On-task helpRarely usedTranslation rarely used

Behavior Regression Results

Whether students were generally off task or on task was a significant predictor of test performance.

Gaming was not a significant predictor of test performance.

Behavior Regression Results

Listening Gain

Study Score a significant predictor of performance

Reading Gain

Study Score &Total Study Time significant predictors of performance.

Questionnaire results Four factors

Attitudes toward the target group

Perceived usefulness of studying

English

Expectancy for success

Attitude towards CALL

Motivation regression analysis

Factor 1 (Attitudes

towards the target

group) was a predictor

of gain scores in both

listening and reading,

but not for behavior.

Research Questions Revisited

Task behavior was a predictor of test performance.The kind of off-task behavior did not make a difference.Attitude toward the TG was a predictor of test performance, but not off/on-task behavior.

ConclusionOff-task behavior

Evidence of off-task behavior related mostly to inactivity rather than gaming

On-task behaviorAll aspects of the software not fully used

MotivationAttitude towards TG most important for effective CALL use

References

Baker, R. S., Corbett, A. T., Koedinger, K. R., & Wagner, A. Z. (2004). Off-task behavior in the cognitive tutor classroom: When students ‘game the system’. Proceedings of ACM CHI 2004: Computer Human Interaction, 383-390.

Baker, R. S., Corbett, A. T., & Koedinger, K. R. (2004). Detecting student misuse of intelligent tutoring systems. Proceedings of the Seventh International Conference on Intelligent Tutoring Systems, 531-540.

Carrol, J. (1963). A model of school learning. Teachers College Record, 64(8), 723-733.