Center for Cognitive Technology

Evidence-Based Learning and Motivation With MOOCs

Dick Clark

Center for Cognitive Technology

University of Southern Californiaclark@usc.edu http:\\cogtech.usc.edu

Learning with MOOCs: A Practitioners Workshop

Wednesday August 13th 2014 – Cambridge MA

Topics

1. Counter-intuitive findings from research on learning from technology - what are we doing that does not work or makes learning fail?

1. What does the best evidence indicate we should do to maximize learning, motivation and transfer from MOOC courses?

i. Instructional designii. Screen designiii. Motivation strategies

1. The impact of evidence-based online course design and suggested MOOC research questions.

Sources of evidence

Why evidence-based MOOC design?

• 45% 2 year and 36% 4 year students “no analytical learning gains”1

• Those whose learning improved only gained about 4% a year.1

• Students remember less than 30% of what they were taught3

• Prestige colleges produce no more learning than all others4 • STEM students writing and quantitative skills decline2.• Physics students think less like physicists than when they enter3. • Minority and underserved student learning outcomes are worse.• Faculty have discipline expertise BUT most lack pedagogical

expertise YET are overconfident about their teaching skill.3

• MOOC designers either do not know about or are not allowed to use evidence-based online pedagogy to produce online presentations.

1Aram & Roska (2011) 2 Bok (2008) 3 Armstrong (2012) 4 Pascarella (2005) 5Bloom (1984)

Counter-Intuitive Research Evidence

• The choice of media for instruction does NOT influence learning. – Lectures, power point slides and video are media. (Clark 2012).

• Learning is influenced by pedagogy (instructional method).

• Pedagogy provides external support for internal cognitive processes required to learn.

• Media influence access to, and the cost of, learning.

• Inadequate pedagogy and screen design can cause students to remember significantly less after they finish a lesson than before they started (R. E. Clark & Estes, 2008).

• 30% of students like and choose instruction from which they learn the least when offered a choice between more or less guidance (Clark & Estes, 2008) .

Counter-Intuitive Research Evidence

• Allowing students to navigate through a course in any way they wish harms both learning and transfer (Elen & Clark, 2006) .

• For novice learners, most immersive simulations and serious games are significantly less effective and more expensive than other ways to teach because of “discovery learning” (Clark, 2007).

• Faculty who are top experts are only able to describe 30% of the analytical and decision strategies they use to perform complex tasks (Clark, 2013).

• 50% of students are wrong when asked what and how much they learned from instruction (Clark & Estes, 2008).

• So what is the problem? Why have we not been more successful?

Cognitive Architecture & Neuroscience

Our cognitive architecture makes learning difficult (perhaps

to protect us?)• We can only think about 3 +/-1 things at a time• Stress and distractions take up thinking space

• When we overload our thinking space it “crashes” and we loose attention and “daydream” (cognitive overload).

• Crashes are pleasurable but learning stops (Clark et al,

2010).

• Helping students avoid cognitive overload requires a careful focus on what and how we are presenting information.

Why Learning and Motivation are Difficult

We have evolved with two integrated knowledge systems to circumvent limits on our thinking space:

1. Declarative (what and why) – can only be memorized (30% of total). Remains conscious unless “forgotten”.

2. Procedural (when and how) – application knowledge – automates and operates outside of our consciousness (70% of total).

Our procedural (application) system is more effective if courses and lessons are sequenced as students are expected to apply the knowledge.

• Or if no sequence, simpler to learn before more difficult topics.

Lessons (modules) should be constructed around a procedure (when and how to do something).

What MOOC design framework will help us solve these problems?

“Knowledge is acquired or modified during unobservable learning events inside students’ minds. These learning events are influenced by instructional events, in which the student’s learning environment is modified, and assessment events, in which student knowledge is inferred from performance either at the time of the event or later.”

Koedinger, K.R., Corbett, A.T., & Perfetti, C.

(2012). The Knowledge-learning-instruction

framework: Toward bridging the science-

practice chasm to enhance robust student

learning. Cognitive Science 36(5), 757-798

PSLC Knowledge-Learning-Instruction Framework

External support for the internal

cognitive processes required for

motivation and learning.

J. R. Anderson’s (Lee & Anderson, 2012) Neuroscience-based ACT-R suggests that lesson level Instruction requires:

• Objective and Reasons (intentional module, goal buffer)• Overview (Declarative module)• Connection to Prior Knowledge (Declarative module)• New Knowledge (Declarative, visual/manual modules)• Demonstrated Procedure (Production system, declarative module)• Practice (Integration of new and old to achieve goal)• Feedback (Intentional module to correct and perfect)

These features mirror David Merrill’s (2006) principles from evidence-based instructional design systems

From Learning to Instruction

Evidence-based lessons or modules consist of 8 sections:

1. Objective (What will be learned in this module)2. Reasons (Benefit and Risk)3. Overview (Outline and instructional method)4. Connection to Prior Knowledge (Analogy,

metaphor)5. Conceptual Knowledge (concepts related to

“how to”)6. When and How (Demonstrate decisions and

actions)7. Practice (Solve engaging problems – part to

whole - explain)8. Feedback (what worked, what needs correction)

Guided Learning - Lesson or Module Level

Evidence for Guided Learning Effectiveness?

Multimedia (screen design) Theory - Mayer

1. Coherence: Eliminate extraneous visuals and sound 31 14 of 14

1. Signaling: Highlight essential information 17 5 of 6

1. Redundancy: Do not narrate on-screen text 23 5 of 5

1. Visual Contiguity: Text next to part described 38 5 of 5

1. Time Contiguity: Simultaneous graphics & narration 425 of 5

10 Principles (Students learn better when …) % # of

Tests

Multimedia (screen design) Theory - Mayer

6. Pacing: Learner pacing better than system pacing 31 3 of 3

7.Pre-train: Teach conceptual information first 27 5 of 5

8.Modality: Narrate graphics, not text & graphics 32 7 of 17

9. Personalization: Learners imagine prior experience 36 11 of 11

10. Voice: Human voice better than machine voice 25 3 of 3

10 Principles (Students learn better when …) % # of

Tests

“What makes me (us)Effective or Successful?”

“What makes me (us)Effective or Successful?”

MOOC Student Motivation Strategies

Motivation defined as: Starting, Persisting and Mental EffortFour MOOC motivational strategies:

1.Increase the personal value of a learning task.• If students are not persisting.

2.Increase (or Decrease) student task-specific confidence.• Under or over confidence decrease persisting and effort.

•Provide more positive emotional climate toward learning task.

• Strong negative emotions decrease motivation.•Reframe students beliefs about the controllability of problems

• When students believe that problems are uncontrollable they will quit or manufacture an excuse not to start or persist.

Value, Efficacy, Attribution and Emotion vary between cultures.

Guidance based on motivation

Survey to monitor perceived value, confidence and mood

(positivity)

Survey data combined with

starting, completion, time and

mastery data to generate

personalized guidance.

© 2013 Kaplan, Inc.

For example, if a student’s profile

shows low performance, low

confidence, and low time spent,

this triggers guidance messages

aimed at increasing confidence. © 2013 Kaplan, Inc.

Checklist for MOOC design reviews

1. Learning Objectives –a) Alignment ?b) Measurability?c) Clarity? and d) Appropriateness?

2. Sequencing – courses, lessons?3. Knowledge to perform real world

tasks?4. Conceptual definitions, examples,

practice and feedback?5. Application demonstrations,

practice and corrective feedback?6. Students asked to reflect and

integrate?7. Clear directions and navigation?

Category Score

1. Program level (3 items) 2. Course Level (4 items) 3. Lesson level ( 6 items) 4. Media and Navigation Level (7 items)

TOTAL Instructional Design SCORE (max=13)

8. Focus only on Learning Objectives

9. Students control pacing?

10. Required sequence?

11. Text and graphics integrated spatially?

12. Narration rather than lots of print?

13. Conversational style?

14. LMS supports self testing, practice and

immediate corrective feedback?

Research questions for MOOC platforms

Use data mining to suggest when to guide and recommend five alternative instructional strategies to target individual learners:

① Choosing content and problems based on prior knowledge.② Providing fully OR partly guided demonstration and

practice③ Immediate OR delayed corrective feedback during practice.④ Providing OR asking explanations for guidance and

feedback.⑤ Hints when students get stuck OR letting them “figure it

out”.http://www.learnlab.orghttp://pact.cs.cmu.edu/pubs/Koedinger,%20Corbett,%20Perfetti%202012-KLI.pdf

Koedinger, K. R. & Aleven, V. (2007) Exploring the assistance dilemma in experiments with cognitive tutors. Educational Psychology Review. 19. 239-264.

Conclusion

• Our focus should be on student learning NOT on media or technology.

• MOOC design requires three specialties: 1. Faculty contribute content expertise.2. Instructional designers – Learning, motivation & media

expertise.3. Assessment specialists – measurement expertise.

• Use guided pedagogy and avoid cognitive overload.• Diagnose and treat motivation problems online.• LMS systems must change to accommodate needed pedagogy.• Research on learning from MOOCs to improve instruction using

data. mining must be based on a thorough review of solid past research.

Comments?Questions?

QUESTIONS? COMMENTS?

ReferencesLearning Styles: Plashler, H. McDaniel, M., Rohrer, D, & Bjork, R. (2008) Learning Styles: Concepts and Evidence. Psychological Science in the Public Interest. 9(3). 105-103. Student Preferences: Clark, R. E. (1989) When Teaching Kills Learning: Research on Mathemathantics. In H. Mandl, E. De Corte, N. Bennett, & H. F. Friedrich (Eds.) Learning and instruction. European Research in an International Context. Volume II. Oxford: Pergamon. Clark, R. E. (1982) Antagonism Between Achievement and Enjoyment in ATI Studies Educational Psychologist, 17(2). 92-101.Rapid Pre-testing – Staged Innovation Design:Kalyuga, S. (2006) Rapid cognitive assessment of learner’s knowledge structures. Learning and Instruction. 16. 1-11.Wagner, N (1984) Instructional product evaluation using the staged innovation design. Journal of Instructional Development. 7(2), 24-27.

Guidance, Media, Motivation, Assessment and Personalizing all discussed in:Bloom, B. S. (1984). The 2-sigma problem: The search for methods of group instruction as effective as one-to-one tutoring. Educational Researcher, 13, 4–16.

Clark, R. E., Yates, K., Early, S. & Moulton, K. (2010). An Analysis of the Failure of Electronic Media and Discovery-based learning:

Evidence for the performance benefits of Guided Training Methods. In Silber, K. H. & Foshay, R. (Eds.). Handbook of Training and

Improving Workplace Performance, Volume I: Instructional Design and Training Delivery. New York: John Wiley and Sons. 263-

297.

Clark, R. E., Howard, K. and Early, (2006). Motivational challenges experienced in highly complex learning environments. In Elen, J. and Clark, R. E. (Eds.). Handling complexity in learning environments: Research and Theory. Oxford, G.B.: Elsevier Science Ltd, 27-43.