ONLINE EDUCA 2011

Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

New Research Shows How to Use Quizzes and Tests to Slow

the Forgetting CurveEduca Online, December 1st 2011

John Kleeman, Questionmark Chairmanwww.questionmark.com

http://www.questionmark.com/

Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are

acknowledged.

Slide 2

Would you like to reduce learner forgetting curves?


acknowledged.

Francis Bacon, 1620 If you read anything

over twenty times, you will not learn it by heart so easily as if you were to read it only ten, trying to repeat it between whiles, and when memory failed, looking at the book.

Retrieval helps you learn

Slide 3


acknowledged.

Learning

Slide 4

Receive informatio

n

Working memory

Long term memory


acknowledged.

Cue

• Question• Other stimulus

Search

• Retrieve from long term memory

Act

• Act or respond

Research shows retrieval practice helps future retrieval

Successful retrieval modifies memory making it more retrievable in future

Learning useful if can retrieve

Slide 5


acknowledged.

Experimental methodology

Re-study group Quiz / retrieval practice group

Slide 6

Study

Quiz

Measure

Study

Re-study

Measure


acknowledged.

120 students divided into groups randomly Given reading comprehension passages

used in TOEFL Some students

Study for 14 mins Some students

Study for 7 mins Recall material for 7 mins

Compared results after 5 mins and 1 week

Roediger & Karpicke experiment 2006

Slide 7


acknowledged.

After 5 mins After 1 week0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

RestudyQuiz

How much learners retained

Slide 8


acknowledged.

SAT

II

Bloo

m D

efini

tion

Bloo

m A

pplic

atio

n0%

10%

20%

30%

40%

50%

60%

No quizQuiz

Marsh & others 2007. Experiments with multiple choice questions

Slide 9


acknowledged.

No qu

iz

Quiz, n

o fe

edba

ck

Quiz, im

med

iate

feed

back

Quiz, d

elay

ed fe

edba

ck0%

20%

40%

60%

Retention after one week

Roediger & Butler. Feedback on multiple choice questions

Slide 10


acknowledged.

80 students in 4 groups, studying a science text Study Repeated Study Concept Mapping (mind maps) Retrieval Practice

Surveyed learners on expectation of retention

Tested a week later to see how much retained

Karpicke & Blunt experiment 2011

Slide 11


acknowledged.

Stud

y

Repea

ted

stud

y

Conce

pt m

appi

ng

Retrie

val p

ract

ice

0%

20%

40%

60%

80%

Actual retention

Stud

y

Repea

ted

stud

y

Conce

pt m

appi

ng

Retrie

val p

ract

ice

0%

20%

40%

60%

80%

Learner expec-tation of reten-

tion

How much learners retained

Slide 12


acknowledged.

Larson experiment 2009 Quizzes reduce forgetting curve In real-life, not in the laboratory Medical residents (doctors in training) 20 doctors in each group

Quiz group Restudy group

Day 1 Study Study

Day 1 Quiz Re-study

2 weeks later Quiz Re-study

2 weeks later Quiz Re-study

6 months later Final exam Final exam


acknowledged.

Initial Quiz Quiz 6 months

62%

44%48%

36%

62%

19%

Quiz group

Part of the results

Slide 14


acknowledged.

Retrieval practice helps retention of learning Applies to facts And also concepts

Retrieval practice includes Recall Formative quizzes/tests Self-assessment Explain or read back of material

Timing of retrieval Works best when spaced out Strongest effect when retrieval is difficult

Feedback very helpful Repetition of retrieval

Research suggests 5-7 spaced repetitions helpful

Retrieval practice key findings

Slide 15


acknowledged.

What does this mean for online learning?

Slide 16


acknowledged.

Learners do not realize they will forget Quizzes encourage/force retrieval practice

Quizzes give corrective feedback And also:

Randomize so easy to repeat Mobile devices Accessible Measure that practice has happened

Computer assessments also have many “indirect“ benefits

Computer quizzes are effective retrieval practice because …

Slide 17


acknowledged.

Direct to further learning Correct misconceptions Feed back to improve the course Compare to pre-test Measure achievement Encourage study Engage students

Indirect benefits of post-learning assessments

Slide 18

http://blog.questionmark.com/wp-content/uploads/2011/06/15428_BLOGJPG_20110609051508257.jpg


acknowledged.

Give quizzes after learning sessions Focus questions on important things to remember Make quizzes effortful (delay & difficulty) Question types:

Recall (e.g. short answer) best Multiple choice work too

Include feedback Usually the correct answer Slight delay if possible

Encourage retrieval (don’t allow peek ahead) Spacing helps significantly

If you are a practitioner …

Slide 19


acknowledged.

How easy is it for learners to take quizzes? How easy is it for your instructors to quiz

learners? Could you make it easier?

If you are a manager …

Slide 20


acknowledged.

“No time” “We teach creativity not facts” “How sure are you of the research?” “Our learners don’t like to be tested” “What about privacy?”

Potential objections

Slide 21


acknowledged.

In a Perfect World…

Slide 22


acknowledged.

In a Perfect World…we would remember everything we learn!

Slide 23


acknowledged.

In the Actual World…

Slide 24


acknowledged.

In the Actual World…we forget some of it!

Slide 25


acknowledged.

Questions through Learning Process

Slide 26

Time

Know

led

ge/S

kills

Quest

ions

Quest

ions

Quest

ions

Quest

ions

Q

uest

ions

Create Intrigu

e

Strengthen Memory

Reduce Forgetting


acknowledged.

Who Title Journal Date Description URL

Roegiger & Karpicke

Test-Enhanced Learning Psychological Science

2006 Widely cited paper showing benefits of testing vs studying

Here

Marsh, Roediger, Bjork&Bjork

The memorial consequencesof multiple-choice testing

Psychonomic Bulletin & Review

2007 Shows benefits of multiple choice questions

Here

Larsen, Butler & Roediger

Test-enhanced learning in medical education.

Medical Education

2009 Shows experiment on medical education and retrieval practice

Here

Fazio, Huelser, Johnson & Marsh

Right/wrong feedback: Consequences for learning

Memory 2010 Research showing just telling students right or wrong without correct answer unhelpful

Here

Roediger & Butler

The critical role of retrieval practice in long-term retention

Trends in Cognitive Sciences

2010 8 page review article on retrieval and retention

Here

Karpicke & Blunt

Retrieval Practice Produces More Learning than Elaborative Studying with Concept Mapping

Science Magazine

2011 Clear paper comparing retrieval with study and mind maps

Select from here (non-commercial)

Kleeman Recent cognitive psychology research shows strongly that quizzes help retain learning. What does this mean for CAA?

International CAA Conference

2011 Some more detail from me

Here

References

http://psych.wustl.edu/memory/Roddy%20article%20PDF's/Roediger&Karpicke(2006)Psych_Sci.pdf

http://psych.wustl.edu/memory/Roddy%20article%20PDF's/MarshRoedigerBjork&Bjork2007.pdf

http://psych.wustl.edu/memory/Roddy%20article%20PDF's/Larsen,Butler&Roediger2008.pdf

http://pn.aas.duke.edu/sites/marsh/Fazio_Huelser_Johnson_Marsh_2010.pdf

http://duke.edu/~ab259/pubs/Roediger&Butler(2010).pdf

http://memory.psych.purdue.edu/publications/

http://caaconference.co.uk/wp-content/uploads/Kleeman-CAA2011.pdf


acknowledged.

What is the best way to remember this session for the future?

What question types seem to give best effect?

What is the best timing for feedback?

Some questions for you:

Slide 28


acknowledged.

Contact me at john at questionmark.com or Twitter: @johnkleeman

Your questions

Slide 29

http://www.twitter.com/johnkleeman

http://www.twitter.com/johnkleeman

ONLINE EDUCA 2011

Education

Transcript of ONLINE EDUCA 2011