RELATE :
description
Transcript of RELATE :
1
REsearch in Learning, Assessing, and Tutoring Electronically
RELATE.mit.edu
Postdocs: Phil Dukes
Sofia MoroteRasil Warnakulasooriya
PI: Dave Pritchard$: MIT, NSF, DEP
RELATE:
2
an expert systembased on educational expertise, not AI
The most advanced tutorial and assessment system in the worldMade by Effective Educational Technologies, a Pritchard company.
Also known as
and CyberTutor.MIT in publications
by EET
3
Outline
• Objective
• Pedagogy that Works
• Feedback – Closed Loop Education
• Research from MIT
• Revolution in Assessment
• Closing Thoughts
4
Digital Education Future?!Broadcast Radio
Passive
Class
Uniform Style
Next Edition
Teacher
Author
High Stakes Tests
Two-way Radio
Interactive
Student
Stylized (e.g. Audio)
Next Day
Coach
Authors/Researchers
Embedded Assessment
Perspective
5
Why Homework??
Teachers’ Priorities:
1. Lectures
2. Exams
3. Notes and Demonstrations
4. Homework
Students spend most time and learn most from 1. Homework
6
TWO WAY LEARNING
Books, lectures, most WWW education
Students, Teachers, Authors, ResearchersLearn from each other
DATA, EXPERIMENT, ANALYSIS, CONCLUSIONS
7
SOCRATIC LEARNING
Authors, Researchers
Student System
Teacher
8
Outline
Objective
• Pedagogy that Works
• Feedback – Closed Loop Education
• Research from MIT
• Revolution in Assessment
• Closing Thoughts
9
PedagogyDesign Philosophy of myCyberTutor
Emulate the interaction between a human tutor and a student.
Results:an effective interactive learning toolyou can author, deliver and improve contentan expert program embodying your expertise
The tutor informs the teacher.
The process informs the author.
10
Pedagogy
Mastery LearningThe amount learned should be constantand time allowed
to vary
OthersSocratic pedagogy and learning styles
(to be implemented)
ConstructivistsAllow students to
construct knowledge in their own way
Student
Student-Centered Instruction
11
Pedagogy
Pedagogical Principles• Actively engage the student • Adapt problem to less skillful students with hints• Prompt feedback addresses wrong answers• Mastery Learning >90% get solution• Declarative and procedural knowledge are both
important hints and subproblems• Solidify and extend the solution followups• Free response answers reduce guessing
12
13
14
15
16
17
Gain on the MIT Final Exam
December 2000 to May 2001
-0.100
0.000
0.100
0.200
0.300
0.400
0.500
0.600
0.700
0.800
Group Problems WrittenHomework
ClassParticipation
CyberTutor
P-value 0.69 0.69 0.35 0.010
Results from MIT
18P-value 0.854 0.807 0.198 0.087 0.015
-0.2
-0.1
0
0.1
0.2
0.3
0.4
0.5
0.6
Small TutorialSessions
PIVOTMultimedia
WrittenHomework
Group Problems CyberTutor
FCI gain=0.41 for course
Gain on Force Concept Inventory
data C. Ogilvie 2000
Results from MIT
19
Outline
Objective
Pedagogy that Works
• Feedback – Closed Loop Education
• Research from MIT
• Revolution in Assessment
• Closing Thoughts
20
Two Way Learning - Feedback
• Typical student returns to server 10 times during course of each problem (cf. Web Assign ~4 times per assignment)
• Students achieve the correct answer 90% of the time (cf. ~60% first time right)
• Students comment on ~3% of all problems– More if problem has flaws
myCyberTutor interactions:
TWO WAY LEARNING
21
Student Comments
Feedback – Closed Loop Education
22
Wrong Answers
Feedback – Closed Loop Education
23
Feedback to AuthorImproves Problems
• < 90% correct: Need more hints
• Wrong Answers: Respond to common ones
• Comments: Revise wording, remove confusion, revise program
• Time: Is this problem worthwhile?
Feedback – Closed Loop Education
24
Feedback Enables Revisionsthat Improve Problems
91.6 %
83.5 %
PercentAnsweringCorrectly
6.0%
12.3 %
Percent Requesting Solutions
0.89
1.51
AverageWrong Answers/part
0.83
0.76
Average hints/part
1.5Spring 2002
1.5Spring 2001
Median Minutes/part
Room for even more improvement!!
Fall
2003
93.4%
25
Outline
Objective
Pedagogy that Works
Feedback – Closed Loop Education
• Research from MIT
• Revolution in Assessment
• Closing Thoughts
26
Pedagogy
RELATE:REsearch in Learning, Assessing, and
Tutoring ElectronicallyRELATE.mit.edu
Postdocs: Phil Dukes
Sofia MoroteRasil Warnakulasooriya
27
Data with resolution
Capability for Split Class Assignments
Your Data PackageLog of your students’ interactionsAnonymous student # vs name & ID for your classDatabase of your class’ assignments
Plus - General Data PackagePerformance Data - your class vs. standardSML skeleton each problem (subparts, hints, etc.)Format key
Attractions for ResearchersObjective
28
Inductive vs. Deductive Instruction
Inductive: Students learn by doing a problem from the hints, from the subproblemsby figuring it out from feedback
Transfer learning to tutorial questions??
Deductive: Students learn from a tutorial from the learning goal & text from the hintsfrom the self-assessment questions
Transfer learning to related problem??
Tutorials
• Tutorial problems in Mastering Physics are
carefully planned and sequenced
instruction with SAQ’s
• They are used as instructional material to
impart principles in deductive learning.
30
Problems
• Problems require a student to apply an already familiar concept, formula, or procedure
• Socratic help is available, including explanation of the concept, a formula that is needed, etc.
• Related Problems cover same topic as adjacent tutorial
Pedagogy
31
p=0.01*
p=0.06**
p=0.03*
Deductive: Related Problem Difficulty Reduced by
working Tutorial First
After working tutorial
Tor
que
New
ton
3rd
Law
Har
mon
ic O
s.
Results from MIT
32
Tor
que
New
ton
3rd
Law
Har
mon
ic O
s.Inductive:
Tutorial Difficulty Reduced by working Related Problem First?
Results from MIT
33
Conclusion: Deductive Works
• Interactive tutorials significantly increase performance on
subsequent related problems (~25% less difficult)
• Students don’t learn inductively from a multi-part example
• We recommend using online tutorials in the old fashioned
way - preparation for subsequent deductive exercises
Results from MIT
34
0.000
0.002
0.004
0.006
0.008
0.010
0.012
0.014
0.016
0.018
0.020
Imp
rove
me
nt
per
un
it t
ime
Tutorial-first
Problem-first
Torque Newton III SHM
Twice as much learning per unit time spent on the tutorial compared with time spent on the preparatory
problem
35
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
Cross-section for asteroid impact
fraction of students who request hints
Colliding cars
Unprepared group Prepared group by solving a related problem
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
Newton III
fraction of students who request hints
Finding Torque
Unprepared group Prepared group by solving a tutorial problem
p < 0. 1
p < 0. 005p < 0. 05
p < 0. 01
Prior tutorial reduces the hints requested on the related problem by ~19%
(based on 5 problems)
Prior related problem reduces the hints requested on the related problem by ~12%
(based on 6 problems)
36
Time to Completion
The real time environment allows us to study how long it takes students to work problems, whether good students do problems quicker or slower, etc.
We have discovered that there areThree groups of students in time:
37
When Students Finish: Three distinct groupsQuick solvers < 2.5 minutes
Real-time solvers 2.5 min – 2.2 hours
Interrupted solvers > 2.2 hours
2 4 6 8 10 12 14-0.05
0.00
0.05
0.10
0.15
0.20
0.25
0.30
0.35
0.40
7s 20s 55s 2.5m 6.7m 18m 50m 2.2h 6h 17h 2d 5d
Rate of completion
Ln(t)
Colliding cars
Quick solvers
Real-time solvers
Interrupted solvers
38
2 4 6 8 10 12 14
0.0
0.1
0.2
0.3
0.4
7s 20s 55s 2.5m 7m 18m 50m 2.2h 6h 17h 2d 5d
Rate of completion
Ln(t)
Finding torque Flywheel kinematics Parallel-axis theorem
Median time ~ 7min
2 4 6 8 10 12 14-0.05
0.00
0.05
0.10
0.15
0.20
0.25
0.30
0.35
0.40
Median time ~ 11min
7s 20s 55s 2.5m 7m 18m 50m 2.2h 6h 17h 2d 5d
Rate of completion
Ln(t)
Colliding cars Shooting a block up an incline
2 4 6 8 10 12 14-0.05
0.00
0.05
0.10
0.15
0.20
0.25
0.30
0.35
0.40
Median time ~ 18-30min
7s 20s 55s 2.5m 7m 18m 50m 2.2h 6h 17h 2d 5d
Rate of completion
Ln(t)
The parallel-axis theorem A person standing on a leaning ladder Collision at an angle
2 4 6 8 10 12 14
0.00
0.05
0.10
0.15
0.20
0.25
0.30
Median time ~ 30min
7s 20s 55s 2.5m 7m 18m 50m 2.2h 6h 17h 2d 5d
Rate of completion
Ln(t)
Cross-section for asteroid impact Post-collision orbit
39
2 4 6 8 10 12 14
0.00
0.05
0.10
0.15
0.20
0.25
Rate of completion
Ln(t)
total no hints, no wrong ans. no hints, at least one wrong at least one hint, one wrong
7s 20s 55s 2.5m 6.7m 18m 50m 2.2h 6h 17h 2d 5d
Real-time solvers make errors and ask for hints
40
2 4 6 8 10 12 14
0.00
0.05
0.10
0.15
0.20
0.25
Rate of completion
Ln(t)
total no hints, no wrong answers no hints, at least one wrong at least one hint, one wrong
Collision at an angle:Prepared group
7s 20s 55s 2.5m 6.7m 18m 50m 2.2h 6h 17h 2d 5d
2 4 6 8 10 12 14
0.00
0.05
0.10
0.15
0.20
0.25 total no hints, no wrong ans. no hints, at least one wrong at least one hint, one wrong
7s 20s 55s 2.5m 6.7m 18m 50m 2.2h 6h 17h 2d 5d
Rate of completion
Ln(t)
Collision at an angle: Unprepared group
Note that:1. The quick solvers do not make mistakes or ask for hints2. The real-time solvers make mistakes and ask for hints3. The interrupted solvers make mistakes and ask for hints4. Fewer real-time solvers in the prepared group ask for hints
41
Fraction Finished curves with hints & feedback
2 4 6 8 10 12 14
0.0
0.2
0.4
0.6
0.8
1.0
fraction
Ln(t)
Colliding cars
7s 20s 55s 2.5m 6.7m 18m 50m 2.2h 6h 17h 2d 5d
2 4 6 8 10 12 14
0.0
0.2
0.4
0.6
0.8
7s 20s 55s 2.5m 6.7m 18m 50m 2.2h 6h 17h 2d 5d
fraction
Ln(t)
Collision at an angle
For 14 problems: fraction of real-time solvers = 65 4%
64% 44%
42
Time to completion curves without hints & feedback
2 4 6 8 10 12 14
0.0
0.2
0.4
0.6
0.8
1.0
7s 20s 55s 2.5m 6.7m 18m 50m 2.2h 6h 17h 2d 5d
fraction
Ln(t)
End-of-chapter 10.46
For 3 typical homework problems:
fraction of real-time solvers = 29 3%
2 4 6 8 10 12 14
0.0
0.2
0.4
0.6
0.8
1.0
7s 20s 55s 2.5m 6.7m 18m 50m 2.2h 6h 17h 2d 5d
fraction
Ln(t)
End-of-chapter 10.38
35%
2 4 6 8 10 12 14
0.0
0.2
0.4
0.6
0.8
1.0
7s 20s 55s 2.5m 6.7m 18m 50m 2.2h 6h 17h 2d 5d
fraction
Ln(t)
End-of-chapter 10.40
28%
43
Outline
Objective
Pedagogy that Works
Feedback – Closed Loop Education
Research from MIT
• Revolution in Assessment
• Closing Thoughts
44
Imagine that a rich ship-owner has hired Socrates to tutor his children. At the end of the month he desires to assess the amount they have learned. Would you advise him to:
a) Administer a standardized hour-long test to the children?
b) Ask Socrates how much they have learned?
Low-Error Embedded Assessment
Future - Assessment
This Assessment gives ~6 times as reliable an assessment per unit of student time as a good final exam!
MyCyberTutor Assessment has ~100 times less variance due to error than a good final exam!
45
Embedded Assessment
myCyberTutor vs. Final ExamSocraticTutor
R2 = 0.986
0
0.1
0.2
0.3
0.4
0.5
0 0.1 0.2 0.3 0.4 0.5
Score (Even Problems)
Score (Odd Problems)
Final Exam
R2 = 0.4053
0
0.1
0.2
0.3
0.4
0.5
0 0.1 0.2 0.3 0.4 0.5
Score (Even Problems)
Score (Odd Problems)
•…is 6 times more reliable per unit time•…has ~100 times less error variance
myCyberTutor
Future - Assessment
46
Assessment: Detailed Skill Profile
Future - Assessment
47
R2 = 0.51
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
SocraticTutor Prediction
MIT Final Score
Implication: MyCyberTutor can replace tests
Predicting Final Exam Score
Future - Assessment
48
myCyberTutor Assessment Implies:
1) More Accurate
2) Fine Grained Assessment on Subtopics
3) Immediate Remediation
-Select Next Problem
4) JITT Guide for Teacher
5) Learning vs. Avoiding Lost Points
6) Predict Test Scores
-Eliminate Tests
7) Incredible Tool for Education Research
8) Replace High Stakes Tests
Assessment
49
Outline
Objective
Pedagogy that Works
Feedback – Closed Loop Education
Research from MIT
Revolution in Assessment
• Closing Thoughts
50
What you can gain
»Write interactive problems»Educational Research on them»Educational Research in general
What we can accomplish together»Partnerships in each - ideally all!!
51
You can write truly interactive material
Improve your problems from student data
Author in just your area of research expertise
Author tutorials, example problems, and problems involving applets
New education research tools improve your problems and exercises
Attractions for Problem Writers
52
New research tool with resolution of student’s capabilities and difficulties
Improve instructional material via Feedback
Experimentally Compare material and pedagogy
Develop texts, tutorials, traditional exercises, and more sophisticated problem sequences
Attractions for Curriculum Developers
53
Credits and ThanksEffective Educational Technologies, Inc.*
Alex Pritchard - sole programmer for 4 yearsAdam Morton - chief programmerDavid Kokorowski - content developmentAndrea Pritchard - president & treasurer & HR
Postdocs and UndergradsGabe Rockefeller - now at U. ArizonaPhil Dukes - now at UTBrownsvilleSofia Morote - now at Dennison College David Kokorowski - now at Effective Education TechRasil Warnakulasooriya - presentSupport:MIT - esp. Physics Dept. - support with TA’sNSF*DEP’s family has controlling financial interest
54
END
55
Student Opinion“How does the amount you learn per unit time with CyberTutor compare with time (including
checking solutions) spent on written homework?”
MIT Semesters
Much Less
Same
Much More
Results from MIT
56
Student Opinion“Would you recommend myCyberTutor for
8.01 next year?”Ratio of Yes to No
MIT Semesters
Results from MIT
57
Feedback – Closed Loop Education
58
Author training and manual
Pedagogy instructional material
Review first problems
Instruction on using feedback data to improve problems, guidelines for wrong answers
Help for Authors
59
85% Relilability is Unfair
1/4 of those students who failed had a passing true scoreAnother 1/4 should have failed, but passed
Future - Assessment
60
Attractions for Teachers
Tutoring available 24/7Better than written Homework
- not lower quality substitute
Shift grading effort into instruction
Teachers have instant access to detailed student performance data, facilitating Just-In-Time Teaching (JiTT)
Problem Library with Metadata on difficulty, time, student rating, topics involved
Objective
61
PedagogyWritten Homework
• No help for student when stuck
• Feedback to student takes 1 week
• Labor to grade• No feedback to
teacher• Copying is easy
Assign Do homework
Hand in
Grade
Learn from hw result?
62
Pedagogy
• Provides immediate feedback and help to the student
• Does the grading • Offers immediate
feedback to the teacher
• Supplies powerful insight into the student’s thinking
Assign
Learn whiledoing CyberTutor
homework
Detailed knowledgebase
myCyberTutor is a Web-Based Homework Tutorial System
Tutorials
• Tutorial problems in myCybertutor are carefully planned and sequenced instruction with SAQ’s
• They are used as instructional material to impart principles in deductive learning.
Pedagogy
64
Outline
ObjectivePedagogy
• Demo
• Results from MIT
• Feedback – Closed Loop Education
• Future - Assessment
• Closing Thoughts
65
Assessment/Testing Error
Any assessment has testing error - did you study problem that was on test?- careless mistakes?- lucky guess?
How to determine reliability (reproducibility)? - compare two equivalent tests
- split single test into two equivalent tests
If test is reliable (error-free), split grades will correlate
Future - Assessment
66Reliability 0.85 error =0.41
observed
Final Exam 2001-2002Future - Assessment