THE POWER OF QUIZZINGTHE POWER OF QUIZZING

Mark A. McDaniel Washington University

NCAT Meeting, March 24, 2009

Kaplan Test Prep and Admissions Survey

Eat Alaskan salmon for breakfast

Watch back-to-back episodes of ER

Wear Superman underwear

• Laboratory research: Taking a test can serve as a learning opportunity, and enhance memory retention even to a greater extent than additional studying… the “testing effect”

• Central implication for course redesigns:

In college classes, low or no stakes quizzes (tests) can be used to enhance student learning (quizzes not just a tool to reveal student mastery)

Wildman, McDaniel & Anderson (2009)

In WEB-based Brain and Behavior university course, each week students covered one chapter (40 pages)

14 central facts from chapter presented to students in one of three ways (as an assignment):

Read the facts to review Multiple choice quiz on the facts Short answer (fill in blank) quiz on the facts

14 other central facts not presented

Summative Assessments (course exams): 84 Item Multiple Choice Exams over Three Chapters

Experiment 1

1. Summative tests manipulated match between quiz stem and test stem: Quiz Stem: Touch receptors (for nocioception and hapsis) on the skin have

their cell bodies in ____________ and connect in the spinal cord to ascending spinothalamic cells.

Exam Stem: Same Touch receptors (for nocioception and hapsis) on the skin have

their cell bodies in ____________ and connect in the spinal cord to ascending spinothalamic cells.

A. dorsal root ganglia, B…., C…, D…..

Different Touch receptors (for nocioception and hapsis) on the skin have their cell bodies in the dorsal root ganglia and connect in the spinal cord to ___________.

A……, B……,C……, D. ascending spinothalamic cells

2. Quizzes presented on web Ungraded Correct answer feedback provided

3. Repeated quizzing: students required to take quizzes four times. (Maximum credit for quiz taking was earned only by taking quizzes four times)

Unit Test Performance

SA = Short answer quiz, MC = Multiple choice quiz, QR = reading, NX = no-exposure

Experiment 1 Different Stems

n = 16

0.76 0.76 0.71 0.68

00.10.20.30.40.50.60.70.80.9

1

SA MC QR NX

SA = Short answer quiz, MC = Multiple choice quiz, QR = reading, NX = no-exposure

Experiment 1 Same Stems

n = 16

0.87 0.850.71 0.68

00.10.20.30.40.50.60.70.80.9

1

SA MC QR NX

Experiment 2Can These Robust Test-Enhanced Learning Effects Be

Replicated and Extended to condensed course format?

• Condensed summer-school course (7 weeks) Individual students could start late and further condense the course

(essentially complete course in 3-4 weeks)

• Similar to Experiment 1Summative tests manipulated match between quiz stem and test stem:

Different or Same

Students required to take quizzes four times (Maximum credit for quiz taking was earned only by taking quizzes four times)

Unit Test Performance

SA = Short answer quiz, MC = Multiple choice quiz, QR = reading, NX = no-exposure

Experiment 2 Different Stems

n = 27

0.75 0.75 0.76 0.65

0

0.2

0.4

0.6

0.8

1

SA MC QR NX

SA = Short answer quiz, MC = Multiple choice quiz, QR = reading, NX = no-exposure

Experiment 2 Same Stems

n= 27

0.90 0.920.76

0.65

00.10.20.30.40.50.60.70.80.9

1

SA MC QR NX

Summary

• Repeated quizzing produces robust learning effects on classroom summative assessments

• Multiple choice quizzes are as effective as short-answer quizzes, when summative assessment is multiple choice and quizzes repeated

• With same stems on quizzes and final test, quizzing produces better performance than presenting target content for additional study

• Can quizzing enhance richer learning of conceptual information (application of content)?

• Will application quiz items promote learning of related factual content?

Short-answer quizzes (web-based course)FactualWhat is the most complex human behavior?

AppliedIf an otherwise normal child does not start talking until after his peers, what do we expect about how long it might take this child to learn to speak fluently?

Performance on Exam Multiple Choice Questions as a Function of Quiz Type

0.84

0.650.74

0.64

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Applied Factual None

Quiz Type Question

Prop

ortio

n Co

rrec

t at E

xam

Exam Question Applied

Exam Question Factual

Performance on Exam Multiple Choice Questions as a Function of Quiz Type

0.840.72

0.650.74

0.810.64

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Applied Factual None

Quiz Type Question

Prop

ortio

n C

orre

ct a

t Exa

m

Exam Question AppliedExam Question Factual

Summary

• Quizzing (short-answer) can promote transfer—it enhances performance on related questions

Does Test-Enhanced Learning Extend to Typical Lecture Courses

and Very Able Learners?• Washington University Introductory Psych

Class (summer)• Intervening quizzing: short answer quizzed not quizzed• Exam question types: Multiple Choice, Short

Answer

McDaniel and Sun(2008)

12 Short answer items

(12 Non-quiz items)

QUIZ SENT BYE-Mail (2/week) UNIT EXAM

Mult. choice (16): 8 same wording and 8 different wording from quiz

Short answer (8): 4 same wording and 4 different wording from quiz

End of

Week

Class lecture;

reading assignment

N=21

Daily

Proportion correct on exams as function of quizzing, exam question type, and match of quiz

and exam question stems

00.10.20.30.40.50.60.70.80.9

1

Same Different Same Different

MC SA

Prop

ortio

n C

orre

ctly

R

ecal

led

Quizzed

Not Quizzed

IS TEST-ENHANCED LEARNINGPERSISTENT OVER LONG DELAYS?

Columbia Middle School project

• 6th, 7th, and 8th grade science classes

• Some facts on each lesson are quizzed 3 times; comparable facts are not quizzed

• Consider recall on tests at ends of chapters, end of the semester and end of the year

0.91

0.76

0.68

0.74

0.69

0.61

0.5

0.6

0.7

0.8

0.9

1

Chapter Exams End of the Semester End of the School Year

Pro

porti

on R

ecal

led

TestedNon-Tested

COLUMBIA MIDDLE SCHOOL PROJECT8TH GRADE SCIENCE

• Can learner initiated quizzes (retrieval) enhance learning (when course constraints preclude any quizzing)?

Learner controlled testing

• Read (study)• Recite (recall material to oneself)• Review (reread the material)

McDaniel, Howard, & Einstein

(2009, Exp. 2)

• Three study conditions:– Read, recite, review– Note-taking– Read-twice control

• Each participant read two complex, ~900 word articles– Brakes & Pumps, as in Mayer & Gallini (1990)

• Students were tested on one of the articles immediately

• After a one-week delay, students were tested on both articles

• Testing consisted of:– Free recall– Multiple choice (4 alternative)– Short-answer inference questions

**

**

**

**

*

* **

Summary• Participants who used the 3R strategy

performed significantly better on a range of classroom type tests relative to reading twice or note-taking (excluding immediate inference performance)

• 3R more efficient (takes less time) than note-taking for enhancing problem solving/inference

Summary of major findings

• Basic test-enhanced learning demonstrated in college courses and classrooms

• Found even when learning at high levels.

• Similar learning benefits were not always found from additional exposure per se

• Self-testing is effective for learning

Many reasons to expect tests to confer learning benefits in educational contexts

Retrieval itself produces retrieval practiceTest itself requires student to engage in the

material (Sikorsky et al., 2001)Signals information to target for study.

(improves metamemory—both accuracy of what you know and control of study)

Leads to more (and spaced) studying—keep up with the material better (Johnson & Kiviniemi, 2009)

Acknowledgements

FundingInstitute of Education SciencesGrant #R305H030339

James S McDonnell Collaborative Activity Grant