Studying and achieving robust learning with PSLC resources
description
Transcript of Studying and achieving robust learning with PSLC resources
![Page 1: Studying and achieving robust learning with PSLC resources](https://reader036.fdocuments.in/reader036/viewer/2022062323/568159a1550346895dc6f244/html5/thumbnails/1.jpg)
1
Studying and achieving robust learning with PSLC resourcesPrepared by: Ken Koedinger
HCII & Psychology, CMUCMU Director of PSLC
Presented by: Vincent AlevenHCII, CMUMember PSLC Executive Committee
![Page 2: Studying and achieving robust learning with PSLC resources](https://reader036.fdocuments.in/reader036/viewer/2022062323/568159a1550346895dc6f244/html5/thumbnails/2.jpg)
2
7th Annual Pittsburgh Science of Learning Center Summer School
• 11th overall– ITS was focus in
2001 to 2004
• Goals:– Learning science &
technology concepts & tools
– Hands-on project => poster on Fri
![Page 3: Studying and achieving robust learning with PSLC resources](https://reader036.fdocuments.in/reader036/viewer/2022062323/568159a1550346895dc6f244/html5/thumbnails/3.jpg)
3
Vision for PSLC
• Why? Chasm between science & practiceIndicators: Ed achievement gaps persist,Low success rate of randomized controlled trials
• Underlying problem: Many ideas, too little sound scientific foundation
• Need: Basic research studies in the field
=> PSLC Purpose: Identify the conditions that cause robust student learning– Field-based rigorous science – Leverage cognitive & computational theory,
educational technologies
“rigorous, sustained scientific research in education” (NRC, 2002)
![Page 4: Studying and achieving robust learning with PSLC resources](https://reader036.fdocuments.in/reader036/viewer/2022062323/568159a1550346895dc6f244/html5/thumbnails/4.jpg)
4
Builds off past success: Intelligent Tutors Bringing Learning Science to Schools!
• Intelligent tutoring systems– Automated 1:1 tutor– Artificial Intelligence– Cognitive Psychology
• Andes: College Physics Tutor– Replaces homework
• Algebra Cognitive Tutor – Part of complete course
Students: model problems with Students: model problems with diagrams, graphs, equationsdiagrams, graphs, equations
Tutor: feedback, help, Tutor: feedback, help, reflective dialogreflective dialog
![Page 5: Studying and achieving robust learning with PSLC resources](https://reader036.fdocuments.in/reader036/viewer/2022062323/568159a1550346895dc6f244/html5/thumbnails/5.jpg)
5
Tutors make a significant difference in improving student learning!
• Andes: College Physics Tutor– Field studies: Significant
improvements in student learning
• Algebra Cognitive Tutor – 10+ full year field
studies: improvements on problem solving, concepts, basic skills
– Regularly used in 1000s of schools by 100,000s of students!
0
10
20
30
40
50
60
Iowa SAT subset ProblemSolving
Represent-ations
Traditional Algebra Course
Cognitive Tutor Algebra
![Page 6: Studying and achieving robust learning with PSLC resources](https://reader036.fdocuments.in/reader036/viewer/2022062323/568159a1550346895dc6f244/html5/thumbnails/6.jpg)
6
President Obama on Intelligent Tutoring Systems!“we will devote more than three percent of our GDP to research and development. …. Just think what this will allow us to accomplish: solar cells as cheap as paint, and green buildings that produce all of the energy they consume; learning software as effective as a personal tutor; prosthetics so advanced that you could play the piano again; an expansion of the frontiers of human knowledge about ourselves and world the around us. We can do this.”
How close to this vision are we now?What else do we need to do?
![Page 7: Studying and achieving robust learning with PSLC resources](https://reader036.fdocuments.in/reader036/viewer/2022062323/568159a1550346895dc6f244/html5/thumbnails/7.jpg)
7
Overview
• PSLC Background– Intelligent Tutoring Systems– Cognitive Task Analysis
• PSLC Methods & Tech Resources– In vivo experimentation– LearnLab courses, CTAT, TagHelper,
DataShop
• PSLC Theoretical Framework
Next
![Page 8: Studying and achieving robust learning with PSLC resources](https://reader036.fdocuments.in/reader036/viewer/2022062323/568159a1550346895dc6f244/html5/thumbnails/8.jpg)
b
CognitiveTutorTechnology
Algebra I GeometryEquationSolver
Algebra II
Cognitive Tutors
ArtificialIntelligence
CognitivePsychology
Research base
Math InstructorsMath EducatorsNCTM Standards
Curriculum Content
Cognitive Tutor Approach
![Page 9: Studying and achieving robust learning with PSLC resources](https://reader036.fdocuments.in/reader036/viewer/2022062323/568159a1550346895dc6f244/html5/thumbnails/9.jpg)
9
• Cognitive Model: A system that can solve problems in the various ways students can
Strategy 1: IF the goal is to solve a(bx+c) = d THEN rewrite this as abx + ac = d
Strategy 2: IF the goal is to solve a(bx+c) = d THEN rewrite this as bx + c =
d/a
Misconception: IF the goal is to solve a(bx+c) = d THEN rewrite this as abx + c = d
Cognitive Tutor Technology
![Page 10: Studying and achieving robust learning with PSLC resources](https://reader036.fdocuments.in/reader036/viewer/2022062323/568159a1550346895dc6f244/html5/thumbnails/10.jpg)
10
3(2x - 5) = 9
6x - 15 = 9 2x - 5 = 3 6x - 5 = 9
Cognitive Tutor Technology
• Cognitive Model: A system that can solve problems in the various ways students can
If goal is solve a(bx+c) = dThen rewrite as abx + ac = d
If goal is solve a(bx+c) = dThen rewrite as abx + c = d
If goal is solve a(bx+c) = dThen rewrite as bx+c = d/a
• Model Tracing: Follows student through their individual approach to a problem -> context-sensitive instruction
![Page 11: Studying and achieving robust learning with PSLC resources](https://reader036.fdocuments.in/reader036/viewer/2022062323/568159a1550346895dc6f244/html5/thumbnails/11.jpg)
11
3(2x - 5) = 9
6x - 15 = 9 2x - 5 = 3 6x - 5 = 9
Cognitive Tutor Technology
• Cognitive Model: A system that can solve problems in the various ways students can
If goal is solve a(bx+c) = dThen rewrite as abx + ac = d
If goal is solve a(bx+c) = dThen rewrite as abx + c = d
• Model Tracing: Follows student through their individual approach to a problem -> context-sensitive instruction
Hint message: “Distribute a across the parentheses.”
Bug message: “You need tomultiply c by a also.”
• Knowledge Tracing: Assesses student's knowledge growth -> individualized activity selection and pacing
Known? = 85% chance Known? = 45%
![Page 12: Studying and achieving robust learning with PSLC resources](https://reader036.fdocuments.in/reader036/viewer/2022062323/568159a1550346895dc6f244/html5/thumbnails/12.jpg)
b
CognitiveTutorTechnology
Algebra I GeometryEquationSolver
Algebra II
Cognitive Tutors
ArtificialIntelligence
CognitivePsychology
Research base
Math InstructorsMath EducatorsNCTM Standards
Curriculum Content
Cognitive Tutor Approach
![Page 13: Studying and achieving robust learning with PSLC resources](https://reader036.fdocuments.in/reader036/viewer/2022062323/568159a1550346895dc6f244/html5/thumbnails/13.jpg)
14
Difficulty Factors Assessment:Discovering What is Hard for Students to Learn
Which problem type is most difficult for Algebra students?
Story Problem
As a waiter, Ted gets $6 per hour. One night he made $66 in tips and earned a total of $81.90. How many hours did Ted work?
Word Problem
Starting with some number, if I multiply it by 6 and then add 66, I get 81.90. What number did I start with?
Equation
x * 6 + 66 = 81.90
![Page 14: Studying and achieving robust learning with PSLC resources](https://reader036.fdocuments.in/reader036/viewer/2022062323/568159a1550346895dc6f244/html5/thumbnails/14.jpg)
15
Algebra Student Results:Story Problems are Easier!
70%61%
42%
0%
20%
40%
60%
80%
Story Word Equation
Problem Representation
Percent Correct
Koedinger, & Nathan, (2004). The real story behind story problems: Effects of representations on quantitative reasoning. The Journal of the Learning Sciences.
Koedinger, Alibali, & Nathan (2008). Trade-offs between grounded and abstract representations: Evidence from algebra problem solving. Cognitive Science.
![Page 15: Studying and achieving robust learning with PSLC resources](https://reader036.fdocuments.in/reader036/viewer/2022062323/568159a1550346895dc6f244/html5/thumbnails/15.jpg)
17
“The Student Is Not Like Me”
• To avoid your expert blind spot, remember the mantra:
“The Student Is Not Like Me”
• Perform Cognitive Task Analysis to find out what students are like
Use Data!
![Page 16: Studying and achieving robust learning with PSLC resources](https://reader036.fdocuments.in/reader036/viewer/2022062323/568159a1550346895dc6f244/html5/thumbnails/16.jpg)
19
Prior achievement:Intelligent Tutoring Systems bring learning science to schools
A key PSLC inspiration:Educational technology as research platform to generate new learning science
![Page 17: Studying and achieving robust learning with PSLC resources](https://reader036.fdocuments.in/reader036/viewer/2022062323/568159a1550346895dc6f244/html5/thumbnails/17.jpg)
20
Overview
• PSLC Background– Intelligent Tutoring Systems– Cognitive Task Analysis
• PSLC Methods & Tech Resources– In vivo experimentation– LearnLab courses, CTAT, TagHelper,
DataShop
• PSLC Theoretical Framework
Next
![Page 18: Studying and achieving robust learning with PSLC resources](https://reader036.fdocuments.in/reader036/viewer/2022062323/568159a1550346895dc6f244/html5/thumbnails/18.jpg)
21
PSLC Statement of Purpose
Leverage cognitive and computational theory to identify the instructional conditions that cause robust student learning.
Leverage cognitive and computational theory to identify the instructional conditions that cause robust student learning.
![Page 19: Studying and achieving robust learning with PSLC resources](https://reader036.fdocuments.in/reader036/viewer/2022062323/568159a1550346895dc6f244/html5/thumbnails/19.jpg)
22
What is Robust Learning?
• Achieved through:– Conceptual understanding & sense-making
skills– Refinement of initial understanding– Development of procedural fluency with basic
skills
• Measured by:– Transfer to novel tasks– Retention over the long term, and/or – Acceleration of future learning
![Page 20: Studying and achieving robust learning with PSLC resources](https://reader036.fdocuments.in/reader036/viewer/2022062323/568159a1550346895dc6f244/html5/thumbnails/20.jpg)
23
PSLC Statement of Purpose
Leverage cognitive and computational theory to identify the instructional conditions that cause robust student learning.
![Page 21: Studying and achieving robust learning with PSLC resources](https://reader036.fdocuments.in/reader036/viewer/2022062323/568159a1550346895dc6f244/html5/thumbnails/21.jpg)
24
In Vivo Experiments: Laboratory-quality principle testing in real classrooms
![Page 22: Studying and achieving robust learning with PSLC resources](https://reader036.fdocuments.in/reader036/viewer/2022062323/568159a1550346895dc6f244/html5/thumbnails/22.jpg)
25
In Vivo Experimentation MethodologyImportant features of different research
methodologies:• What is tested?
– Instructional solution vs. causal principle
• Where & who?– Lab vs. classroom
• How?– Treatment only vs. Treatment + control
• Generalizing conclusions:– Ecological validity: What instructional activities work in real classrooms?– Internal validity: What causal mechanisms explain & predict?
![Page 23: Studying and achieving robust learning with PSLC resources](https://reader036.fdocuments.in/reader036/viewer/2022062323/568159a1550346895dc6f244/html5/thumbnails/23.jpg)
33
In Vivo Experimentation MethodologyLab
ExperimentsDesign
ResearchRandomized Field Trials
In vivo Experiments
What?
Instructional solution
√ √
Causal principle √ × × √
Where & who?
Lab √
Classroom × √ √ √
How?
Treatment only √
Treatment + control
√ × √ √
Generalizes how?
Internal validity √ × +/– √
Ecological validity
× √ √ √
![Page 24: Studying and achieving robust learning with PSLC resources](https://reader036.fdocuments.in/reader036/viewer/2022062323/568159a1550346895dc6f244/html5/thumbnails/24.jpg)
34
LearnLabA Facility for Principle-Testing
Experiments in Classrooms
![Page 25: Studying and achieving robust learning with PSLC resources](https://reader036.fdocuments.in/reader036/viewer/2022062323/568159a1550346895dc6f244/html5/thumbnails/25.jpg)
35
LearnLab courses at K12 & College Sites
• 6+ cyber-enabled courses: Chemistry, Physics, Algebra, Geometry, Chinese, English
• Data collection– Students do home/lab work
on tutors, vlab, OLI, …– Log data, questionnaires,
tests DataShop
Researchers Schools
Learn Lab
Chemistry virtual labChemistry virtual lab
Physics intelligent tutorPhysics intelligent tutor
REAP vocabolary tutorREAP vocabolary tutor
![Page 26: Studying and achieving robust learning with PSLC resources](https://reader036.fdocuments.in/reader036/viewer/2022062323/568159a1550346895dc6f244/html5/thumbnails/26.jpg)
36
PSLC Technology Resources
• Tools for developing instruction & experiments– CTAT (cognitive tutoring systems)
• SimStudent (generalizing an example-tracing tutor)
– OLI (learning management)– TuTalk (natural language dialogue)– REAP (authentic texts)
• Tools for data analysis – DataShop– TagHelper
![Page 27: Studying and achieving robust learning with PSLC resources](https://reader036.fdocuments.in/reader036/viewer/2022062323/568159a1550346895dc6f244/html5/thumbnails/27.jpg)
37
PSLC Statement of Purpose
Leverage cognitive and computational theory to identify the instructional conditions that cause robust student learning.
![Page 28: Studying and achieving robust learning with PSLC resources](https://reader036.fdocuments.in/reader036/viewer/2022062323/568159a1550346895dc6f244/html5/thumbnails/28.jpg)
38
Overview
• PSLC Background– Intelligent Tutoring Systems– Cognitive Task Analysis
• PSLC Methods & Tech Resources– In vivo experimentation– LearnLab courses, CTAT, TagHelper,
DataShop
• PSLC Theoretical Framework Next
![Page 29: Studying and achieving robust learning with PSLC resources](https://reader036.fdocuments.in/reader036/viewer/2022062323/568159a1550346895dc6f244/html5/thumbnails/29.jpg)
39
KLI Framework: Designing Instruction for Robust Learning
The conditions that yield robust learning can be decomposed at three levels:Knowledge componentsLearning eventsInstructional events
Get framework report at learnlab.org
![Page 30: Studying and achieving robust learning with PSLC resources](https://reader036.fdocuments.in/reader036/viewer/2022062323/568159a1550346895dc6f244/html5/thumbnails/30.jpg)
40
Instructional events
Explanation, practice, text, rule, example, teacher-student discussion
Assessment eventsQuestion, feedback, step in ITS
Learning events
KC accessible from long-term memory
Immediate performance
Robust performance
KEYOvals – observableRectangle - inferredSolid line – causeDashed line – inferencesKC = Knowledge Component
State test, belief survey
KLI Event Decomposition
• Decompose temporal progress of learning
• Observe/control instructional & assessment events
• Infer learning events & changes in knowledge
![Page 31: Studying and achieving robust learning with PSLC resources](https://reader036.fdocuments.in/reader036/viewer/2022062323/568159a1550346895dc6f244/html5/thumbnails/31.jpg)
41
KLI Framework: Designing Instruction for Robust Learning
The conditions that yield robust learning can be decomposed at three levels:Knowledge componentsLearning eventsInstructional events
![Page 32: Studying and achieving robust learning with PSLC resources](https://reader036.fdocuments.in/reader036/viewer/2022062323/568159a1550346895dc6f244/html5/thumbnails/32.jpg)
42
What’s the best form of instruction? Two choices?• More assistance vs. more challenge
– Basics vs. understanding– Education wars in reading, math, science…
• Researchers like binary oppositions too.We just produce a lot more of them!– Massed vs. distributed (Pashler)– Study vs. test (Roediger)– Examples vs. problem solving (Sweller,Renkl)– Direct instruction vs. discovery learning (Klahr)– Re-explain vs. ask for explanation (Chi, Renkl)– Immediate vs. delayed (Anderson vs. Bjork)– Concrete vs. abstract (Pavio vs. Kaminski)– …
Koedinger & Aleven (2007). Exploring the assistance dilemma in experiments with Cognitive Tutors. Ed Psych Review.
![Page 33: Studying and achieving robust learning with PSLC resources](https://reader036.fdocuments.in/reader036/viewer/2022062323/568159a1550346895dc6f244/html5/thumbnails/33.jpg)
43
How many options are there really? And what works best when?
What’s best?
Focused practice
Distributed practice
Study examples
Test onproblems
50/50Mix
Concrete AbstractMix
Delayed No feedback
Immediate
Block topics in chapters
Interleave topics
Fade
Explain Ask for explanations
Mix
Gradually widen
Study Test50/50 Study Test50/50Study
Concrete Mix Abstract
ImmediateNo
feedbackDelayed
Block topics in chapters
Fade Interleave topics
Explain Ask for explanations
Mix
More helpBasics
More challenge Understanding
![Page 34: Studying and achieving robust learning with PSLC resources](https://reader036.fdocuments.in/reader036/viewer/2022062323/568159a1550346895dc6f244/html5/thumbnails/34.jpg)
44
205,891,132,094,649“Big Science” effort needed to tackle this complexity
Cumulative theory development Field-based basic research with microgenetic data collection
Derivation:15 instructional dimensions 3 options per dimension 2 stages of learning=> 315*2 options
![Page 35: Studying and achieving robust learning with PSLC resources](https://reader036.fdocuments.in/reader036/viewer/2022062323/568159a1550346895dc6f244/html5/thumbnails/35.jpg)
45
An example of some PSLC studies in this space
• Researchers like binary oppositions too.We just produce a lot more of them!– Massed vs. distributed (Pashler)– Study vs. test (Roediger)– Examples vs. problem solving (Sweller,Renkl)– Direct instruction vs. discovery learning (Klahr)– Re-explain vs. ask for explanation (Chi, Renkl)– Immediate vs. delayed (Anderson vs. Bjork)– Concrete vs. abstract (Pavio vs. Kaminski)– …
![Page 36: Studying and achieving robust learning with PSLC resources](https://reader036.fdocuments.in/reader036/viewer/2022062323/568159a1550346895dc6f244/html5/thumbnails/36.jpg)
46
Learn by doing or by studying?• Testing effect (e.g., Roediger & Karpicke, 06)
– “Tests enhance later retention more than additional study of the material”
Kirschner, Sweller, & Clark (2006). Why minimal guidance during instruction does not work:…failure of… problem-based…teaching. • Worked example effect
– “a worked example constitutes the epitome of strongly guided instruction [aka optimal instruction]”
– Paas and van Merrienboer (1994), Sweller et al. (1998), van Gerven et al. (2002), van Gog et al. (2006), Kalyuga et al. (2001a, 2001b), Sweller et al. (1998), Ayres (2006), Trafton & Reiser (1993), Renkl and Atkinson (2003), … the list goes on …
• Theoretical goal: Address debate between desirable difficulties, like “testing effect”, and direct instruction, like “worked examples”
• Limitation of past worked example studies have weaker control, untutored practice– PSLC studies compare to tutored practice
![Page 37: Studying and achieving robust learning with PSLC resources](https://reader036.fdocuments.in/reader036/viewer/2022062323/568159a1550346895dc6f244/html5/thumbnails/37.jpg)
47
Worked Example Experiments within Geometry Cognitive Tutor (Alexander Renkl, Vincent Aleven, Ron Salden, et al.)
• 8 studies in US & Germany– Random assignment, vary single principle– Over 500 students– 3 in vivo studies run in Pittsburgh area
schools
• Cognitive Science ’08 Conference IES Best Paper Award
![Page 38: Studying and achieving robust learning with PSLC resources](https://reader036.fdocuments.in/reader036/viewer/2022062323/568159a1550346895dc6f244/html5/thumbnails/38.jpg)
48
Ecological Control = Standard Cognitive Tutor Students solve problems step-by-step & explain
![Page 39: Studying and achieving robust learning with PSLC resources](https://reader036.fdocuments.in/reader036/viewer/2022062323/568159a1550346895dc6f244/html5/thumbnails/39.jpg)
49
Worked out steps with calculation shown by Tutor
Treatment condition: Half of steps are given as examples
Student still has to self explain worked out step
![Page 40: Studying and achieving robust learning with PSLC resources](https://reader036.fdocuments.in/reader036/viewer/2022062323/568159a1550346895dc6f244/html5/thumbnails/40.jpg)
50
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
Pretest Transfer:Procedural
Transfer:Declarative
Percentage correct
Example
Problem
d = .73 *
Worked examples improve efficiency & understanding
Lab results• 20% less time
on instruction• Conceptual
transfer in study 2
In Vivo• Adaptively fading examples to problems yields
better long-term retention & transfer
![Page 41: Studying and achieving robust learning with PSLC resources](https://reader036.fdocuments.in/reader036/viewer/2022062323/568159a1550346895dc6f244/html5/thumbnails/41.jpg)
51
Worked example effect generalizes across domains, settings, researchers• Geometry tutor studies• Chemistry tutor studies in vivo at High School & College
(McLaren et al.)
– Same outcomes in 20% less time• Algebra Tutor study in vivo (Anthony et al.)
– Better long term retention in less time • Theory: SimStudent model (Matsuda et al.)
– Problems provide learning process with negative examples to prune misconceptions
• Research to practice– Influencing Carnegie Learning development– New applied projects with SERP, WestEd
![Page 42: Studying and achieving robust learning with PSLC resources](https://reader036.fdocuments.in/reader036/viewer/2022062323/568159a1550346895dc6f244/html5/thumbnails/42.jpg)
52
Processes of Learning within the KLI Framework
The conditions that yield robust learning can be decomposed at three levels:Knowledge
componentsLearning eventsInstructional events
Learning eventsFluency building, refinement, sense making
![Page 43: Studying and achieving robust learning with PSLC resources](https://reader036.fdocuments.in/reader036/viewer/2022062323/568159a1550346895dc6f244/html5/thumbnails/43.jpg)
53
Learning Events in the Brain & reflected in Dialogue
• Fluency buildingMemory, speed, automaticity
• Refinement processesClassification, co-training, discrimination, analogy, non-verbal explanation-based learning
• Sense-making processesReasoning, experimentation, explanation, argument, dialogue
Some PSLC Examples• ACT-R models of spacing,
testing effects & instructional efficiency (Pavlik)
• SimStudent models of learning by example & by tutoring– Inductive logic prgrming,
probabilistic grammars (Matsuda, Cohen , Li, Koedinger)
• Transactivity+ analysis of peer & classroom learning dialogues (Rose, Asterhan, Resnick)
![Page 44: Studying and achieving robust learning with PSLC resources](https://reader036.fdocuments.in/reader036/viewer/2022062323/568159a1550346895dc6f244/html5/thumbnails/44.jpg)
54
Knowledge components carry the results of learning• Knowledge component =
an acquired unit of cognitive function or structure that can be inferred from performance on a set of related tasks.
• Used in broad sense of a knowledge base– From facts to mental
models, metacognitive skills
![Page 45: Studying and achieving robust learning with PSLC resources](https://reader036.fdocuments.in/reader036/viewer/2022062323/568159a1550346895dc6f244/html5/thumbnails/45.jpg)
55
Example KCs with different features• Chinese vocabulary KCs:
const->const, explicit, no rationale– If the Chinese pinyin is “lao3shi1”, then the English word is “teacher”– If the Chinese radical is “ 日” , the English word is “sun”
• English Article KCs: var->const, implicit, no rationale
– If the referent of the target noun was previously mentioned, then use “the”
• Geometry Area KCs: var->var, implicit & explicit, rationale
– If the goal is to find the area of a triangle,and the base <B> and the height is <H>,then compute 1/2 * <B> * <H>
– If the goal is to find the area of irregular shape made up of regular shapes <S1> and <S2>, then find area <S1> and <S2> and add
Integrated KCs for mental models, central conceptual structures, strategies & complex planning
![Page 46: Studying and achieving robust learning with PSLC resources](https://reader036.fdocuments.in/reader036/viewer/2022062323/568159a1550346895dc6f244/html5/thumbnails/46.jpg)
56
Kinds of Knowledge Components
• Other kinds of KCs– Integrative, probabilistic, metacognitive, misconceptions or “buggy”
knowledge
![Page 47: Studying and achieving robust learning with PSLC resources](https://reader036.fdocuments.in/reader036/viewer/2022062323/568159a1550346895dc6f244/html5/thumbnails/47.jpg)
57
Knowledge components are not just about domain knowledge• Examples of possible domain-general KCs• Metacognitive strategy
– Novice KC: If I’m studying an example, try to remember each step– Desired KC: If I’m studying an example, try to explain how each
step follows from the previous• Motivational belief
– Novice: I am no good at math– Desired: I can get better at math by studying and practicing
• Social communicative strategy– Novice: When an authority figure speaks, remember what they
say. – Desired: Repeat another's claim in your own words and ask
whether you got it right
Can these be assessed, learned, taught? Broad transfer?
![Page 48: Studying and achieving robust learning with PSLC resources](https://reader036.fdocuments.in/reader036/viewer/2022062323/568159a1550346895dc6f244/html5/thumbnails/48.jpg)
58
Learning curves: Measuring behavior on tasks over time• Data from flash-card tutor• Tasks present a Chinese word & request the English translation• Learning curve shows average student performance (e.g., error rate, time on
correct responses) after each opportunity to practice
6 secs
3 secs
![Page 49: Studying and achieving robust learning with PSLC resources](https://reader036.fdocuments.in/reader036/viewer/2022062323/568159a1550346895dc6f244/html5/thumbnails/49.jpg)
59
Empirical comparison of KC complexity
Example KC typesChinese vocabulary KCs const->const, explicit, no rationale
English Article KCs var->const, implicit, no rationale
Geometry Area KCs var->var, implicit/explicit, rationale
Time 6 -> 3 secs
10 -> 6 secs
14 -> 10 secs
6 secs
3 secs
10 secs
6 secs
14 secs
10 secs
![Page 50: Studying and achieving robust learning with PSLC resources](https://reader036.fdocuments.in/reader036/viewer/2022062323/568159a1550346895dc6f244/html5/thumbnails/50.jpg)
60
Which instructional principles are effective for which kinds of knowledge components?
Do complex instructional events aid simple knowledge acquisition?
Do simple instructional events aid complex knowledge acquisition?
![Page 51: Studying and achieving robust learning with PSLC resources](https://reader036.fdocuments.in/reader036/viewer/2022062323/568159a1550346895dc6f244/html5/thumbnails/51.jpg)
61
Prompted self-explanation studies across domains• Physics Course - field principles
– Better transfer than providing explanations
• Geometry Course - properties of angles– Better transfer than just practice despite
solving 50% fewer problems in same time
• English Course - article use– Pure practice appears more efficient;
self-explanation may help for long-term retention & for novices
• Cross-domain hypoth: Type of KC determines when self-explanation will be effective.
Var->Var, explicit
Var->Const, implicit
![Page 52: Studying and achieving robust learning with PSLC resources](https://reader036.fdocuments.in/reader036/viewer/2022062323/568159a1550346895dc6f244/html5/thumbnails/52.jpg)
62
Summary• Obama: “learning software as effective as a
personal tutor”• How close to this vision are we now?
– Many fielded Intelligent Tutors– Students learn as much or more
• What else do we need to do?– Expand to more areas => CTAT– More sophisticated interaction => CSCL– Use tutors to advance science & improve
educational practice => In Vivo & EDM
In other words …Take the PSLC Summer School!
![Page 53: Studying and achieving robust learning with PSLC resources](https://reader036.fdocuments.in/reader036/viewer/2022062323/568159a1550346895dc6f244/html5/thumbnails/53.jpg)
63
![Page 54: Studying and achieving robust learning with PSLC resources](https://reader036.fdocuments.in/reader036/viewer/2022062323/568159a1550346895dc6f244/html5/thumbnails/54.jpg)
64
To do • Other possibilities
– Reduce in vivo slide by deleting redundancy between text and table
– Define ITS as about the intelligence in the design – Add an opener?
• Not techy enough– TEDx talk => KDD Cup => learning curves
• Lots of issues -- gaming vs. not (Shih)• Tie to WE and SE
– Get from HCI AB talk (CSCL example) Too much listing at the end
![Page 55: Studying and achieving robust learning with PSLC resources](https://reader036.fdocuments.in/reader036/viewer/2022062323/568159a1550346895dc6f244/html5/thumbnails/55.jpg)
65
LearnLab Products
Infrastructure created and highly used• LearnLab courses have supported over
150 in vivo experiments
• Established DataShop: A vast open data repository & associated tools– 110,000 student hours of data
• 21 million transactions at ~15 second intervals – New data analysis & modeling algorithms– 67 papers, >35 are secondary data analysis not
possible without DataShop
![Page 56: Studying and achieving robust learning with PSLC resources](https://reader036.fdocuments.in/reader036/viewer/2022062323/568159a1550346895dc6f244/html5/thumbnails/56.jpg)
66
Novice Expert
Learning
Instruction
Pre-test Post-test
Typical Instructional Study• Compare effects of 2 instructional conditions in lab • Pre- & post-test similar to tasks in instruction
![Page 57: Studying and achieving robust learning with PSLC resources](https://reader036.fdocuments.in/reader036/viewer/2022062323/568159a1550346895dc6f244/html5/thumbnails/57.jpg)
67
Post-test
Novice Expert(desired)
Learning
Instruction
Pre-test Post-test: Long-term retention, transfer, accelerated future learning, or desire for future learning
• Macro: Measures of robust learning
PSLC Instructional Experiments
![Page 58: Studying and achieving robust learning with PSLC resources](https://reader036.fdocuments.in/reader036/viewer/2022062323/568159a1550346895dc6f244/html5/thumbnails/58.jpg)
68
Post-test
Novice Expert(desired)
Learning
Instruction
Pre-test Post-test: Long-term retention, transfer, accelerated future learning, or desire for future learning
• Macro: Measures of robust learning
PSLC Instructional Experiments
Assessment Events
Knowledge:Shallow percepts & concepts
Knowledge: Deep percepts & concepts, fluent
• Micro analysis: knowledge, learning, interactions
Instructional Events
![Page 59: Studying and achieving robust learning with PSLC resources](https://reader036.fdocuments.in/reader036/viewer/2022062323/568159a1550346895dc6f244/html5/thumbnails/59.jpg)
69
Post-test
Coursegoals &cultural context
• Studies run in vivo as part of existing courses
Novice Expert(desired)
Learning
Instruction
Pre-test Post-test: Long-term retention, transfer, accelerated future learning, or desire for future learning
• Macro: Measures of robust learning
PSLC Instructional Experiments
Assessment Events
Knowledge:Shallow percepts & concepts
Knowledge: Deep percepts & concepts, fluent
• Micro analysis: knowledge, learning, interactions
Instructional Events
![Page 60: Studying and achieving robust learning with PSLC resources](https://reader036.fdocuments.in/reader036/viewer/2022062323/568159a1550346895dc6f244/html5/thumbnails/60.jpg)
70
Develop a research-based, but practical framework• Theoretical framework key goals
– Support reliable generalization from empirical studies to guide design of effective ed practices
Two levels of theorizing: • Macro level
– What instructional principles explain how changes in the instructional environment cause changes in robust learning?
• Micro level– Can learning be explained in terms of what knowledge
components are acquired at individual learning events?
![Page 61: Studying and achieving robust learning with PSLC resources](https://reader036.fdocuments.in/reader036/viewer/2022062323/568159a1550346895dc6f244/html5/thumbnails/61.jpg)
71
Example study at macro level: Hausmann & VanLehn 2007
• Research question– Should instruction provide explanations and/or elicit
“self-explanations” from students?
• Study design– All students see 3 examples & 3 problems
• Examples: Watch video of expert solving problem• Problems: Solve in the Andes intelligent tutor
– Treatment variables: • Videos include justifications for steps or do not• Students are prompted to “self-explain” or paraphrase
![Page 62: Studying and achieving robust learning with PSLC resources](https://reader036.fdocuments.in/reader036/viewer/2022062323/568159a1550346895dc6f244/html5/thumbnails/62.jpg)
72
Para-phrase
Self-explain
Explan XNo
explan
![Page 63: Studying and achieving robust learning with PSLC resources](https://reader036.fdocuments.in/reader036/viewer/2022062323/568159a1550346895dc6f244/html5/thumbnails/63.jpg)
73
Para-phrase
Self-explain
Explan
No explan X
![Page 64: Studying and achieving robust learning with PSLC resources](https://reader036.fdocuments.in/reader036/viewer/2022062323/568159a1550346895dc6f244/html5/thumbnails/64.jpg)
74
Self-explanations => greater robust learning
0.670.670.90 0.980
0.2
0.4
0.6
0.8
1
1.2
Paraphrase,With just.
Paraphrase,Without just.
Self-explain,With just.
Self-explain,Without just.
(hints+errors) / steps
• Justifications: no effect!
• Immediate test on electricity problems:
• Transfer to new electricity homework problems
0.450.370.69 1.040
0.2
0.4
0.6
0.8
1
1.2
1.4
Paraphrase,With just.
Paraphrase,Without just.
Self-explain,With just.
Self-explain,Without just.
(hints+errors) / steps
• Instruction on electricity unit => accelerated future learning of magnetism!
0.911.17 0.75 0.680
0.2
0.4
0.6
0.8
1
1.2
1.4
Paraphrase,With just.
Paraphrase,Without just.
Self-explain,With just.
Self-explain,Without just.
(hints+errors) / steps
![Page 65: Studying and achieving robust learning with PSLC resources](https://reader036.fdocuments.in/reader036/viewer/2022062323/568159a1550346895dc6f244/html5/thumbnails/65.jpg)
75
Key features of H&V study
• In vivo experiment– Ran live in 4 physics sections at US Naval
Academy – Principle-focused: 2x2 single treatment
variations– Tight control manipulated through
technology
• Use of Andes tutor => repeated embedded assessment without
disrupting course
• Data in DataShop (more later)
![Page 66: Studying and achieving robust learning with PSLC resources](https://reader036.fdocuments.in/reader036/viewer/2022062323/568159a1550346895dc6f244/html5/thumbnails/66.jpg)
76
Develop a research-based, but practical framework• Theoretical framework key goals
– Support reliable generalization from empirical studies to guide design of effective ed practices
Two levels of theorizing: • Macro level
– What instructional principles explain how changes in the instructional environment cause changes in robust learning?
• Micro level– Can learning be explained in terms of what knowledge
components are acquired at individual learning events?
![Page 67: Studying and achieving robust learning with PSLC resources](https://reader036.fdocuments.in/reader036/viewer/2022062323/568159a1550346895dc6f244/html5/thumbnails/67.jpg)
77
Knowledge Components
• Knowledge Component– A mental structure or process that a learner uses,
alone or in combination with other knowledge components, to accomplish steps in a task or a problem-- PSLC Wiki
• Evidence that the Knowledge Component level functions in learning …
![Page 68: Studying and achieving robust learning with PSLC resources](https://reader036.fdocuments.in/reader036/viewer/2022062323/568159a1550346895dc6f244/html5/thumbnails/68.jpg)
![Page 69: Studying and achieving robust learning with PSLC resources](https://reader036.fdocuments.in/reader036/viewer/2022062323/568159a1550346895dc6f244/html5/thumbnails/69.jpg)
79
0.00
1.00
2.00
3.00
4.00
5.00
6.00
7.00
Problem1 Problem2 Problem3
(hints+errors)/steps
Instructionalexplanation
Self-explanation
Example1 Example2 Example3
Back to H&V study: Micro-analysisLearning curve for main KCSelf-explanation effect tapers but not to zero
![Page 70: Studying and achieving robust learning with PSLC resources](https://reader036.fdocuments.in/reader036/viewer/2022062323/568159a1550346895dc6f244/html5/thumbnails/70.jpg)
80
PSLC wiki: Principles & studies that support them
Points to Hausmann’s study page (and other studies too)
Instructional Principle pages unify across studies
![Page 71: Studying and achieving robust learning with PSLC resources](https://reader036.fdocuments.in/reader036/viewer/2022062323/568159a1550346895dc6f244/html5/thumbnails/71.jpg)
81
PSLC wiki: Principles & studies that support them
With links to concepts in glossary Hausmann’s study description:
![Page 72: Studying and achieving robust learning with PSLC resources](https://reader036.fdocuments.in/reader036/viewer/2022062323/568159a1550346895dc6f244/html5/thumbnails/72.jpg)
82
PSLC wiki: Principles & studies that support them
~200 concepts in glossarySelf-explanation glossary entry
![Page 73: Studying and achieving robust learning with PSLC resources](https://reader036.fdocuments.in/reader036/viewer/2022062323/568159a1550346895dc6f244/html5/thumbnails/73.jpg)
83
Research Highlights• Synthesizing worked examples &
self-explanation research – 10+ studies in multiple 4 math & science domains– New theory: It’s not just cognitive load!
• Examples for deep feature construction, problems & feedback for shallow feature elimination
This work inspired new question: Does self-explanation enhance language learning? Experiments in progress …
• Computational modeling of student Learning – Simulated learning benefits of examples/demonstrations vs.
problem solving (Masuda et al., 2008)• Theory outcome: problem solving practice is an important source of
negative examples • Engineering: “programming by tutoring” is more cost-effective than
“programming by demonstration”– Shallow vs. deep prior knowledge changes learning rate (Matsuda et
al., in press)
![Page 74: Studying and achieving robust learning with PSLC resources](https://reader036.fdocuments.in/reader036/viewer/2022062323/568159a1550346895dc6f244/html5/thumbnails/74.jpg)
84
Research Highlights (cont) Computational modeling of instructional assistance
Assistance formula: Optimal learning (L) depends on right level of assistance
Relevant to multiple experimental paradigms & dimensions of instructional assistance Direct instruction (worked examples) vs. constructivism (testing effect)
Concrete manipulatives vs. simple abstractions
Formula provides path to resolve hot debates
P*Sb+(1-P)FbP*Sc+(1-P)Fc
L =
Kirschner, Sweller, & Clark (2006). Why minimal guidance during instruction does not work: An analysis of the failure of constructivist, discovery, problem-based, experiential, and inquiry-based teaching. Educational Psychologist
Kaminski, Sloutsky, & Heckler (2008). The advantage of learning abstract examples in learning math. Science.
L
PAssistance
![Page 75: Studying and achieving robust learning with PSLC resources](https://reader036.fdocuments.in/reader036/viewer/2022062323/568159a1550346895dc6f244/html5/thumbnails/75.jpg)
85
Research Highlights (cont) Synthesis paper on computer tutoring of metacognition
Generalizes results across 7 studies, 3 domains, 4 populations Posed new questions about role of motivation
Lasting effects of metacognitive support Computer-based tutoring of self-regulatory learning
Technologically possible & can have a lasting effect Students who used help-seeking tutor demonstrated better learning
skills in later units after support was faded Spent 50% more time reading help messages
Data mining for factors that affect student motivation Machine learning to analyze vast student interaction data
from full year math courses (Baker et al., in press a & b) Students more engaged on “rich” story problems than standard Surprise: Also more engaged on abstract equation exercises!
Koedinger, Aleven, Roll, & Baker. (in press). In vivo experiments on whether supporting metacognition in intelligent tutoring systems yields robust learning. In Handbook of Metacognition in Education.
![Page 76: Studying and achieving robust learning with PSLC resources](https://reader036.fdocuments.in/reader036/viewer/2022062323/568159a1550346895dc6f244/html5/thumbnails/76.jpg)
86
Social Communication
Social context of classroom Teacher
Interaction
Thrusts investigate overlapping factors
Novice Expert
Learning
Instruction
Knowledge:Shallow, perceptual
Knowledge: Deep, conceptual, fluent
THRUSTSCognitive Factors
MetacognitionMotivation
MetacognitionMotivation
Metacognition & Motivation
Motivation Metacognition
Comp Modeling & Data Mining
Learning
Instruction
Knowledge Knowledge
![Page 77: Studying and achieving robust learning with PSLC resources](https://reader036.fdocuments.in/reader036/viewer/2022062323/568159a1550346895dc6f244/html5/thumbnails/77.jpg)
87
Thrust Research Questions• Cognitive Factors. How do instructional events affect
learning activities and thus the outcomes of learning?• Metacognition & Motivation. How do activities
initiated by the learner affect engagement with targeted content?
• Social Communication. How do interactions between learners and teachers and computer tutors affect learning?
• Computational Modeling & Data Mining. Which models are valid across which content domains, student populations, and learning settings?
![Page 78: Studying and achieving robust learning with PSLC resources](https://reader036.fdocuments.in/reader036/viewer/2022062323/568159a1550346895dc6f244/html5/thumbnails/78.jpg)
88
4th Measure of Robust Learning • Existing robust learning measures
– Transfer– Long-term retention– Acceleration of future learning
• New measure:– Desire for future learning
• Is student engaged in subject? • Do they chose to pursue further math, science, or
language?
![Page 79: Studying and achieving robust learning with PSLC resources](https://reader036.fdocuments.in/reader036/viewer/2022062323/568159a1550346895dc6f244/html5/thumbnails/79.jpg)
89
END