QA on “The behavior of tutoring systems”
description
Transcript of QA on “The behavior of tutoring systems”
![Page 1: QA on “The behavior of tutoring systems”](https://reader036.fdocuments.in/reader036/viewer/2022062520/56815d73550346895dcb7e3d/html5/thumbnails/1.jpg)
QA on “The behavior of tutoring systems”
CPI 494 Feb 3, 2009Kurt VanLehn
![Page 2: QA on “The behavior of tutoring systems”](https://reader036.fdocuments.in/reader036/viewer/2022062520/56815d73550346895dcb7e3d/html5/thumbnails/2.jpg)
What’s a “task domain”?
• Solving linear algebraic equations?• Solving 2x+3=5?• Solving math problems?• Reciting the Gettysburgh Address?• Painting color-by-numbers picture?• Dancing the Macrarena?• Folk dancing.• Writing a funny TV sit-com script
![Page 3: QA on “The behavior of tutoring systems”](https://reader036.fdocuments.in/reader036/viewer/2022062520/56815d73550346895dcb7e3d/html5/thumbnails/3.jpg)
What’s a “task”?
• Writing a hiaku on death of a loved one?• Solving 2x+3=5?• Writing the digit 5?• Reading • Reading an electronic schematic for a 2-
transitor amplifier.
![Page 4: QA on “The behavior of tutoring systems”](https://reader036.fdocuments.in/reader036/viewer/2022062520/56815d73550346895dcb7e3d/html5/thumbnails/4.jpg)
What is a “step”?
• What are the steps in– Solving 2x+5=11? 2x=11-5 2x=6 2x/2=6/2 – Writing the digit “4”– Dividing 10 by 2?– Phonic spelling of the last name of the current US
president?– Writing a hiaku on death of a loved one?– Painting a color-by-numbers picture?
![Page 5: QA on “The behavior of tutoring systems”](https://reader036.fdocuments.in/reader036/viewer/2022062520/56815d73550346895dcb7e3d/html5/thumbnails/5.jpg)
What is a “Knowledge component”?
• Definition (hint: Uses “independently”)?• What are choices of granularity in solving
2x+5=11?
![Page 6: QA on “The behavior of tutoring systems”](https://reader036.fdocuments.in/reader036/viewer/2022062520/56815d73550346895dcb7e3d/html5/thumbnails/6.jpg)
How to use transfer to check if KC’s are defined correctly
• Use KCs to represent task domain, e.g., multi-column addition (496+307) & subtraction
• Determine which KCs must be mastered for one set of tasks, e.g., addition A
• Determine which KCs must be mastered for a second set of tasks, e.g., subtraction S
• Experiment:– Measure time for group S to master subtraction – Measure time for group AS to master subtraction after mastering
addition– Group AS should take less time because some of the subtraction KCs
are also addition KCs, and thus have been mastered already.• |S-A| / |S| ≈ time(AS)/time(S)
– where |X| is the cardinality of set X– Where S-A is the set difference
![Page 7: QA on “The behavior of tutoring systems”](https://reader036.fdocuments.in/reader036/viewer/2022062520/56815d73550346895dcb7e3d/html5/thumbnails/7.jpg)
What is a learning event?
• Definition?• Give example from algebra• Give example from teaching your mom how to
enter a phone number into her cell phone.
![Page 8: QA on “The behavior of tutoring systems”](https://reader036.fdocuments.in/reader036/viewer/2022062520/56815d73550346895dcb7e3d/html5/thumbnails/8.jpg)
Fourth Third Second First Fifth
While studying an example, tries to self-explain; fails; looks in text; succeeds
While solving a problem, looks up example; recalls explanation; maps it to problem
Recalls explanation; slips; corrects
Solves without slips
5 sec.
10 sec.
15 sec.
25 sec.
20 sec.
Dura
tion
of th
e in
stru
ctio
nal e
vent
Chronological position of the learning event
![Page 9: QA on “The behavior of tutoring systems”](https://reader036.fdocuments.in/reader036/viewer/2022062520/56815d73550346895dcb7e3d/html5/thumbnails/9.jpg)
Using learning curves to check KCs
• Proper KCs should have smooth learning curves
1 2 3 4 5 6 7 …
Learning event s in chronological order
Dura
tion
of le
arni
ng e
vent
1 2 3 4 5 6 7 …
Learning event s in chronological orderDu
ratio
n of
lear
ning
eve
nt
![Page 10: QA on “The behavior of tutoring systems”](https://reader036.fdocuments.in/reader036/viewer/2022062520/56815d73550346895dcb7e3d/html5/thumbnails/10.jpg)
How to collect a learning curve
• Track learning of N students• At each of their learning events – Measure time (plot duration on y-axis)– Measure errors (plot error rate on y-axis)– Measure hint requests (plot hints+errors on y-axis)
• Aggregate over the N students– E.g., at the 3rd learning event of all N students,
count number of students who make error & divide by N plot on y-axis, with x=3.
![Page 11: QA on “The behavior of tutoring systems”](https://reader036.fdocuments.in/reader036/viewer/2022062520/56815d73550346895dcb7e3d/html5/thumbnails/11.jpg)
Two nested loops of an immediate feedback step-based tutoring system• Task (outer) loop:– Select task– Do task– Repeat until student has mastered task domain
• Step (inner) loop– Tutor and student do a step, somehow• T:hint; S: trys; T: wrong!; S: trys again; T: Good!
– Repeat until the task is finished
![Page 12: QA on “The behavior of tutoring systems”](https://reader036.fdocuments.in/reader036/viewer/2022062520/56815d73550346895dcb7e3d/html5/thumbnails/12.jpg)
What are the 2 loops for CTAT fraction addition tutor?
• Outer?
• Inner?
![Page 13: QA on “The behavior of tutoring systems”](https://reader036.fdocuments.in/reader036/viewer/2022062520/56815d73550346895dcb7e3d/html5/thumbnails/13.jpg)
What are two loops for Steve?
![Page 14: QA on “The behavior of tutoring systems”](https://reader036.fdocuments.in/reader036/viewer/2022062520/56815d73550346895dcb7e3d/html5/thumbnails/14.jpg)
What are two loops for Andes?
![Page 15: QA on “The behavior of tutoring systems”](https://reader036.fdocuments.in/reader036/viewer/2022062520/56815d73550346895dcb7e3d/html5/thumbnails/15.jpg)
What are two loops for Cognitive Algebra Tutor?
![Page 16: QA on “The behavior of tutoring systems”](https://reader036.fdocuments.in/reader036/viewer/2022062520/56815d73550346895dcb7e3d/html5/thumbnails/16.jpg)
What are two loops for AutoTutor?
![Page 17: QA on “The behavior of tutoring systems”](https://reader036.fdocuments.in/reader036/viewer/2022062520/56815d73550346895dcb7e3d/html5/thumbnails/17.jpg)
What are the two loops for SQL-Tutor?
![Page 18: QA on “The behavior of tutoring systems”](https://reader036.fdocuments.in/reader036/viewer/2022062520/56815d73550346895dcb7e3d/html5/thumbnails/18.jpg)
Main issue for the task (outer) loop?
• How to select the next task?• Where to get tasks to select from?
![Page 19: QA on “The behavior of tutoring systems”](https://reader036.fdocuments.in/reader036/viewer/2022062520/56815d73550346895dcb7e3d/html5/thumbnails/19.jpg)
5 common services for steps
• Minimal feedback• Hints on the next step• Error specific feedback• Assessment of knowledge• Review of solution
![Page 20: QA on “The behavior of tutoring systems”](https://reader036.fdocuments.in/reader036/viewer/2022062520/56815d73550346895dcb7e3d/html5/thumbnails/20.jpg)
Main issues for minimal feedback
• Correctness?• When?
![Page 21: QA on “The behavior of tutoring systems”](https://reader036.fdocuments.in/reader036/viewer/2022062520/56815d73550346895dcb7e3d/html5/thumbnails/21.jpg)
Main issues for next-step hints
• When?• What step to suggest?• How to give hints?
![Page 22: QA on “The behavior of tutoring systems”](https://reader036.fdocuments.in/reader036/viewer/2022062520/56815d73550346895dcb7e3d/html5/thumbnails/22.jpg)
Main issues in error-specific feedback
• How to classify errors?• How to give feedback? Debugging…• When/whether to give feedback?
![Page 23: QA on “The behavior of tutoring systems”](https://reader036.fdocuments.in/reader036/viewer/2022062520/56815d73550346895dcb7e3d/html5/thumbnails/23.jpg)
Main issues in assessment
• What should be assessed?• Grain size of assessment?• How to measure progress, hints & errors?• How to combine progress, hints & errors?
![Page 24: QA on “The behavior of tutoring systems”](https://reader036.fdocuments.in/reader036/viewer/2022062520/56815d73550346895dcb7e3d/html5/thumbnails/24.jpg)
Main issues in reviewing solutions
• Who controls the review?• What depth?• Anything suppressed?• Questions allowed?