PRINCIPLES OF COGNITIVE SCIENCE IN EDUCATION

SELF-GENERATION, ERRORS, & FEEDBACK

Janet Metcalfe Columbia University

IES 2006

MS 143 and the Columbia University CASL project

Thanks go to IES for sponsoring this research.

Thanks go especially to Lisa Son, Bridgid Finn and Nate

Kornell

THE GOAL OF THIS PROJECT IS TO USE PRINCIPLES OF COGNITIVE

SCIENCE TO DEVELOP A PROGRAM THAT CAN BE BROADLY USED TO

HELP STUDENTS STUDY. Why do they need it?

(1)self study is vulnerable to metacognitive illusions

(2) certain effective strategies are difficult to implement on one’s own

BACKGROUND

The Bronx project

Strict Lenient0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

ComputerSelfNone

Accuracy - Advanced English Vocabulary (English-Speaking Children)

Scoring

Accu

racy

Spanish-speaking children learning English vocabulary

Strict Lenient0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

ComputerSelfNone

Accuracy - Basic English Vocabulary (Spanish-Speaking Children)

Scoring

Accuracy

Even Columbia students showed substantial gains, though not as high

as the at-risk children.

Strict Lenient0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

ComputerSelfNone

Accuracy - Basic Spanish Vocabulary (English Speaking CU Students)

Scoring

Accuracy

GOAL: TO ISOLATE COMPONENTS IN THE ORIGINAL TASK THAT ENHANCED OR HARMED LEARNING.

Generation

Errors

Feedback

SELF GENERATION

To our surprise, we, several times, failed to find beneficial effects of self generation.

There was no difference between the control condition and either generation

condition.

control forced generation free generation0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Test 1Test 2

CONDITION

PR

OP

OR

TIO

N C

OR

RE

CT

control forced generation free generation0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Test 1Test 2

CONDITION

PR

OP

OR

TIO

N C

OR

RE

CT

With Columbia students, there was also no generation effect.

So, we tried again, with slight modifications.

Bronx SIXTH GRADE children.

NO GENERATION EFFECT

control forced generation free generation0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

Immediate testDelayed test

CONDITION

PR

OP

OR

TIO

N C

OR

REC

T

Columbia students

NO GENERATION EFFECT.

control forced generation free generation0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

Immediate testDelayed test

CONDITION

PR

OP

OR

TIO

N C

OR

RE

CT

WHY NO GENERTION EFFECT ?Our hypotheses were

1. WE GAVE THE CUE FIRST ALONE WITH A 1 SECOND PAUSE WHICH ALLOWED PEOPLE TO GENERATE EVEN IN THE READ CONDITION.

2. THE GENERATE AND READ ITEMS

WERE MIXED WITHIN LIST

If we alter the conditions so people do not generate in the ‘read’ condition, we get a generation effect.

GENERATE AND READ WERE NOW BETWEEN LIST, AND THE CUE AND

TARGET WERE PRESENTED TOGETHER.

0

.05

.1

.15

.2

.25

Ce

ll M

ea

n f

or

Len

ien

t

Immediate Delayed

Retrieval

PassivePres

ActivePres

THIS PRODUCED A LARGE GENERATE EFFECT

SIMILARLY, WITH COLUMBIA STUDENTS

LEARNING GRE WORDS

0

0.1

0.2

0.3

0.4

0.5

0.6

Presentation Retrieval

Condition

Len

ien

t Acc

ura

cy

WE FOUND A LARGE GENERATE EFFECT

PEDAGOGICAL IMPLICATIONS

SELF-GENERATION OF ANSWERS PRODUCES

LARGE LEARNING BENEFITS.

IN A CLASSROOM SITUATION, WHERE ONE CAN’T HAVE EVERY STUDENT GENERATE ALL THE TIME, ONE CAN GET GENERATION BENEFITS BY DOING WHAT WE HAD DONE IN THE FIRST SET OF EXPERIMENTS:

(1)LEAVING A PAUSE BEFORE GIVING THE ANSWER

AND(2) MIXING IN QUESTIONS TO INDIVIDUAL

STUDENTS, WHO MUST THEN GENERATE THE ANSWERS THEMSELVES, FROM TIME TO TIME.

ERRORS

We observed no effect of whether we forced people to generate an answer (which resulted in a huge number of errors) or whether we allowed them to generate only if they felt confident in their answers (which resulted in few errors).

control forced generation free generation0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Test 1Test 2

CONDITION

PR

OP

OR

TIO

N C

OR

RE

CT

NO EFFFECT OF ERRORSBRONX 6TH GRADE

control forced generation free generation0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Test 1Test 2

CONDITION

PR

OP

OR

TIO

N C

OR

RE

CT

AGAIN, NO EFFECT OF MAKING ERRORS, COLUMBIA STUDENTS

Bronx SIXTH GRADE AGAINNO EFFECT OF ERRORS

control forced generation free generation0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

Immediate testDelayed test

CONDITION

PR

OP

OR

TIO

N C

OR

REC

T

Columbia students: NO EFFECT OF ERRORS

control forced generation free generation0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

Immediate testDelayed test

CONDITION

PR

OP

OR

TIO

N C

OR

RE

CT

However, it seems like committing an error should be problematic.

So we ran two more experiments directed at errors specifically.

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

Errors No errors

Condition

Len

ien

t Acc

ura

cy

BronxChildren were either forced to generate--producing many errors or told to generate, but not to guess, producing few, in a between list design. They were given immediate feedback. Errors made no difference.

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

Errors No errors

Condition

Len

ien

t Acc

ura

cy

Columbia

The same experiment was run with more difficult materials.

Again, there was no effect.

PEDAGOGICAL IMPLICATIONS

We have no evidence to support the contention that making errors is harmful, as long as corrective feedback is given, even though we have now sought such evidence 6 times.

FEEDBACK

So far, I have shown results only when feedback was given.

But, we also had no feedback conditions.

IN EVERY CASE, FEEDBACK HELPED.

Feedback No Feedback0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

Forced (many errors) free (few errors)

Data from Bronx studentsPro

port

ion C

orr

ect

Feedback No Feedback0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

forced (many errors) free (few errors))

Data from Columbia StudentsPro

port

ion

Corr

ect

Feedback No Feedback0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

ForcedFree

Data from Columbia, Experiment 2Pro

port

ion

Corr

ect

In all three experiments, the effects of initial feedback also

held up in the second test.

In the final experiment we investigated the issue of how

important it is to give immediate, as compared to

delayed, feedback.

There were five conditions:

Immediate feedback immediate test

No feedback immediate test

Immediate feedback delayed test

Delayed feedback delayed test

No feedback delayed test

The experiment was run over four days. On the fourth day we

did the delayed test.

0

.2

.4

.6

.8

1

Acc

ura

cy

De

l te

st/D

el

fee

db

ack

De

l te

st/I

mm

fe

ed

ba

ck

De

l te

st/N

o f

ee

db

ack

Imm

te

st/I

mm

fe

ed

ba

ck

Imm

te

st/N

o f

ee

db

ack

The design of this experiment allowed us to look at the effect of

immediate versus delayed feedback, holding constant

feedback lag to test (by only using the middle sessions).

delayed immediate no feedback0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

Data from Columbia Dragon Masters

Condition

Pro

port

ion C

orr

ect

WE HAVE DONE A REPLICATION STUDY WITH GRADE 6 CHILDREN AT THE SCHOOL AT COLUMBIA UNIVERSITY AND FOUND SIGNIFICANTLY BETTER FINAL TEST PERFORMANCE WHEN FEEDBACK WAS GIVEN AT A DELAY RATHER THAN IMMEDIATELY.

PEDAGOGICAL IMPLICATIONS

We DO need to give feedback, and good feedback that is very specific and that the person fully learns and understands.

BUT, we may not have to give that feedback immediately.

Conclusions(1) Having students self generate the answers rather than passively presenting the answers to them has a large effect.

(2) It makes no difference if the person makes errors in attempting to generate their answers, as long as they are given corrective feedback.

(3) Feedback matters enormously.

(4) It may not matter whether the feedback is given immediately or at a delay--indeed, delayed feedback may be better.