1 Cognitive Tutor Development Kit (TDK) Tutorial Cognitive Modeling & Intelligent Tutoring Systems...
-
date post
20-Dec-2015 -
Category
Documents
-
view
217 -
download
0
Transcript of 1 Cognitive Tutor Development Kit (TDK) Tutorial Cognitive Modeling & Intelligent Tutoring Systems...
1
Cognitive Tutor Development Kit (TDK) Tutorial
Cognitive Modeling & Intelligent Tutoring Systems
Ken KoedingerVincent Aleven
2
Overview• Basics of TDK Notation• Matching: $ and = variables• Modeling alternative strategies• “Interface constraints” for model
tracing• Control flow and production chains• Writing hints and bug productions• Creating a curriculum
3
Production Rules in the Tutor Development Kit (TDK)
• How do you go about writing a production rule?
• The TDK Production System– Working Memory:
• Make up of “Working Memory Elements” (WMEs)
– Production Rules– Interpreter (Match, Conflict Resolution, Fire)
4
Implementing a Production Rule Model in TDK
• Simple example: a model for single-column addition without carrying!
• How would you define:– Working Memory representation for the
problem states– Production rules that transform working
memory
4+ 3 7
4+ 3
4+ 3 7 Done
5
A Notation for Working Memory Elements (“WMEs”) in TDK
PROBLEM4+3> ISA SINGLE-COLUMN-ADDITION-PROBLEM FIRST-ADDEND 4 SECOND-ADDEND 3 RESULT nil DONE nil
4+ 3 WME Name
Slot Values
Slots
WME TypeSeparator
6
Working Memory Transitions
PROBLEM4+3> ISA SINGLE-COLUMN-
ADDITION-PROBLEM
FIRST-ADDEND 4 SECOND-ADDEND 3 RESULT nil DONE nil
4+ 3 7
4+ 3
4+ 3 7 Done
PROBLEM4+3> ISA SINGLE-COLUMN-
ADDITION-PROBLEM
FIRST-ADDEND 4 SECOND-ADDEND 3 RESULT 7 DONE nil
PROBLEM4+3> ISA SINGLE-COLUMN-
ADDITION-PROBLEM
FIRST-ADDEND 4 SECOND-ADDEND 3 RESULT 7 DONE T
Production: ADD Production: DONE
7
TDK Production Rule Notation(defproduction add single-column-addition (=problem) =problem> isa single-column-addition-problem result NIL first-addend =num1 second-addend =num2
==>
!eval! =sum (+ =num1 =num2)
=problem> result =sum)
ADD (English version)IFThe goal is to do =problem, a
single-column addition problemAnd no result has been found yetAnd the first addend is =num1And the second added is =num2THENSet =sum to the sum of
=num1 and =num2
Write =sum as the result
8
TDK Production Rule Notation(defproduction add single-column-addition (=problem)
=problem> isa single-column-addition-problem result NIL first-addend =num1 second-addend =num2
==>
!eval! =sum (+ =num1 =num2)
=problem> result =sum)
If-part
Then-part
9
TDK Production Rule Notation(defproduction add single-column-addition (=problem)
=problem> isa single-column-addition-problem result NIL first-addend =num1 second-addend =num2
==>
!eval! =sum (+ =num1 =num2)
=problem> result =sum)
If-part
Then-part
WME pattern
WME pattern
Computation in Lisp
10
TDK Production Rule Notation(defproduction add single-column-addition (=problem)
=problem> isa single-column-addition-problem result NIL first-addend =num1 second-addend =num2
==>
!eval! =sum (+ =num1 =num2)
=problem> result =sum)
WME pattern
Variables(“bound” to values through matching) Separator
11
Matching a Production Rule to Working Memory—Find Values for Each Variable
Working Memoryproblem4+3> isa single-column- addition-
problem first-addend 4 second-addend 3 result NIL done NIL
Find value for each variable
Variable=problem=num1=num2=sum
ValueProblem4+3437
Match!
Modify WME
problem4+3> isa single-column-
addition-problem first-addend 4 second-addend 3 result 7 done NIL
Production Rule
add =problem> isa single-column-
addition-problem result NIL first-addend =num1 second-addend =num2==>!eval! =sum (+ =num1 =num2)=problem> result =sum)
This WME pattern in the THEN-part means change slot value of existing WME
What changes are made to Working Memory?
12
Summary—Components of a TDK Production Rule Model
• Working Memory is a collection of WMEswme-name>
isa wme-typeslot1 value1slot2 value2
• Production rules specify working memory transitions– WME patterns in the IF-part are matched against
Working Memory=wme-variable>
isa wme-typeslot1 value-pattern1slot2 value-pattern2
– WME patterns in the THEN-part specify changes to Working Memory
13
Overview• Basics of TDK Notation• Matching: $ and = variables• Modeling alternative strategies• “Interface constraints” for model
tracing• Control flow and production chains• Writing hints and bug productions• Creating a curriculum
14
WME Representation
26+ 51
26+ 51 7
26+ 51 77
26+ 51 77 Done
Value of a slot in a WME can be a list.
Value of a slot in a WME can be a list.
Initial StatePROBLEM26+51> ISA MULTI-COLUMN-ADDITION-PROBLEM COLUMNS (COLUMN2 COLUMN1) DONE nilCOLUMN1> ISA COLUMN FIRST-ADDEND 6 SECOND-ADDEND 1 RESULT nilCOLUMN2> ISA COLUMN FIRST-ADDEND 2 SECOND-ADDEND 5 RESULT nil
Final StatePROBLEM26+51> ISA MULTI-COLUMN-ADDITION-PROBLEM COLUMNS (COLUMN2 COLUMN1) DONE TCOLUMN1> ISA COLUMN FIRST-ADDEND 6 SECOND-ADDEND 1 RESULT 7COLUMN2> ISA COLUMN FIRST-ADDEND 2 SECOND-ADDEND 5 RESULT 7
15
Production Rules Set New Goals & Perform Actions
FOCUS-ON-FIRST-COLUMN,FOCUS-ON-NEXT-COLUMN
ADD
26+ 51 77
Goal: Solve the addition problem
Goal: Process column C
Action: Write the sum
Action: Mark as done
DONE
16
Goals in Production Rules
FOCUS-ON-FIRST-COLUMNIF The goal is to do a multi-column
addition problem And C is the rightmost column of
the problem And there are numbers to add in C
And there is no result yet in CTHEN Set a subgoal to process column C
FOCUS-ON-NEXT-COLUMNIF The goal is to do a multi-column
addition problem And C is the rightmost column with
numbers to add and no resultTHEN Set a subgoal to process column C
WRITE-RESULTIF There is a goal to process column CTHEN Set Sum to the sum of the
addends in column C Write Sum as the result in column C
DONEIF The goal is to do a multi-column
addition problem And there is no column left with
numbers to add and no result
THEN Mark the problem as done
17
Figuring Out Which ColumnFOCUS-ON-FIRST-COLUMNIF The goal is to do a multi-column
addition problem And C is the rightmost column of
the problem And there are numbers to add in C
And there is no result yet in CTHEN Set a subgoal to process column C
FOCUS-ON-NEXT-COLUMNIF The goal is to do a multi-column
addition problem And C is the rightmost column with
numbers to add and no resultTHEN Set a subgoal to process column C
WRITE-RESULTIF There is a goal to process column CTHEN Set Sum to the sum of the
addends in column C Write Sum as the result in column C
DONEIF The goal is to do a multi-column
addition problem And there is no column left with
numbers to add and no result
THEN Mark the problem as done
Example Production Rule(defproduction focus-on-next-column multi-column-addition (=problem) =problem> isa multi-column-addition-problem columns ($ =next-column =previous-column $) =previous-column> isa column - result NIL =next-column> isa column result NIL first-addend =num1 second-addend =num2 !eval! (or =num1 =num2) ==> =process-column-goal> isa process-column-goal column =next-column !chain! multi-column-addition
(=process-column-goal))
FOCUS-ON-NEXT-COLUMNIFThe goal is to do =problem, a multi-column
addition problemAnd =next-column and =previous-column are
columns next to each otherAnd there is a result in =previous-columnAnd there is no result in =next-column And =num1 is the 1st addend in =next-column And =num2 is the 2nd addend in =next-column And there is no result yet in =next-column And =num1 or =num2 is not blankTHENSet a subgoal to process =next-column
TDK Production Rule Notation(defproduction focus-on-next-column multi-column-addition
(=problem) =problem> isa multi-column-addition-problem columns ($ =next-column =previous-column $) =previous-column> isa column - result NIL =next-column> isa column result NIL first-addend =num1 second-addend =num2 !eval! (or =num1 =num2) ==> =process-column-goal> isa process-column-goal column =next-column !chain! multi-column-addition (=process-column-goal))
If-part
Then-partWME pattern
WME pattern
WME pattern
WME pattern
TDK Production Rule Notation(defproduction focus-on-next-column multi-column-addition
(=problem) =problem> isa multi-column-addition-problem columns ($ =next-column =previous-column $) =previous-column> isa column - result NIL =next-column> isa column result NIL first-addend =num1 second-addend =num2 !eval! (or =num1 =num2) ==> =process-column-goal> isa process-column-goal column =next-column !chain! multi-column-addition (=process-column-goal))
Variablesmultiple occurrences of same variable in IF-part must match the same value
TDK Production Rule Notation(defproduction focus-on-next-column multi-column-addition
(=problem) =problem> isa multi-column-addition-problem columns ($ =next-column =previous-column $) =previous-column> isa column - result NIL =next-column> isa column result NIL first-addend =num1 second-addend =num2 !eval! (or =num1 =num2) ==> =process-column-goal> isa process-column-goal column =next-column !chain! multi-column-addition (=process-column-goal))
Variables within a list whose name starts with $ match any number of items (0 or more).
22
Matching a Production Rule to Working Memory—Find Values for Each Variable
(1)Production Rulefocus-on-next-column =problem> isa multi-column-
addition-problem columns
($ =next-col =prev-col $) =prev-col> isa column - result NIL =next-col> isa column result NIL first-addend =num1 second-addend =num2 !eval! (or =num1 =num2) ==> =process-column-goal> isa process-column-goal column =next-col
Working Memoryproblem26+51> isa multi-column-
addition-problem columns (column2 column1) done nilcolumn1> isa column first-addend 6 second-addend 1 result 7column2> isa column first-addend 2 second-addend 5 result nil
Find value for each variable
Variable=problem1st $=next-col=prev-col2nd $=num1=num2
Valueproblem26+51()column2column1()61
Match!
Match!
Create new WME<process-column-goal 0> isa process-column-goal column column2
What changes are made to Working Memory?
When a WME pattern with “isa” is in the THEN-part of a rule, it means create a new WME rather than modify an existing one.
23
Value to be matched: (COLUMN4 COLUMN3 COLUMN2 COLUMN1) Pattern: ($ =next-column =previous-column $)
Match 1Left $ ()=next-column COLUMN4=previous-column COLUMN3Right $ (COLUMN2 COLUMN1)
Match 2Left $ (COLUMN4)=next-column COLUMN3=previous-column COLUMN2Right $ (COLUMN1)
Match 3Left $ (COLUMN4 COLUMN3)=next-column COLUMN2=previous-column COLUMN1Right $ ()
Pattern Matching with Lists—Patterns with$ variable(s) often create multiple matches
Common pattern to extract the next element of a list.
24
TDK Production Rule Notation
• “==>” separates if-part & then-part• If- & then-parts are made up of WME patterns of form:
=wme-variable>isa wme-typeslot1 value-pattern1slot2 value-pattern2
• An atom starting with either “=” or “$” is a variable– Ex: “=num” “$columns”– “=” vars match atoms– “$” variables match 0 or more elements of a list– “=” or “$” by itself is just an unnamed (dummy)
variable
25
The TDK Production System Interpreter
• Match phase:– Attempt to match each WME pattern with 1 or more WMEs in
working memory– Variables are “bound” to a value. Subsequent references to those
variables must have the same value.– A set of “bindings” for all variables in the if-part is called a “instance”
of a production.– If a consistent set of bindings is found, the production “matches”,
otherwise it “fails”.– A production can match working memory in more than one way, that
is, it can have multiple instances (or rule instantiations).
• Conflict resolution phase: – In stand-alone mode: Determined by hand or arbitrarily– In tutor mode: Production priorities & model tracing heuristics
• Act phase (fire a production):– Use the variable bindings to modify or create the WME indicated in
the then-part.– If “isa” slot is present, create a WME, otherwise modify.
26
Overview• Basics of TDK Notation• Matching: $ and = variables• Modeling alternative strategies• “Interface constraints” for model
tracing• Control flow and production chains• Writing hints and bug productions• Creating a curriculum
27
Modeling Alternative Strategies
• Consider multiple solution paths– Reordering of steps
• Write sum first or write carry in next column first
– Alternative strategies• Left to right strategy• Use calculator and write result strategy
– Shortcuts• Implicit carrying
• Which of these will the model cover?• Motivate choices pedagogically!
28
Execution Space Diagram Illustrating Flexibility
1 264 + 716 80
1 264 + 716 0
264 + 716
1 264 + 716 980
264 + 716 0
1 264 + 716
29
Production Rules Set New Goals & Perform Actions
Goal: Solve the addition problem
ADD-CARRY
264 + 716FOCUS-ON-FIRST-COLUMN,
FOCUS-ON-NEXT-COLUMN
Goal: Process column C
Goal: Write carry in next column
MUST-CARRY
WRITE-SUM
Action: Write the sum
Action: Write the carry
WRITE-CARRY
Goal: Write sum in column C
ADD-ADDENDS
30
Model Tracing 1 — Using a Cognitive Model to Evaluate Students’ Solution Steps
• Cognitive model is an expert system that– is psychologically plausible– accounts for all reasonable solutions– accounts for commonly occurring errors (“bugs”)
• To verify solution step by student– Use model to generate correct and buggy next steps– If student step matches
• Correct step: Accept– update Working Memory to reflect new problem state
• Buggy step: Give error-specific feedback• No step: Reject without feedback
31
Overview• Basics of TDK Notation• Matching: $ and = variables• Modeling alternative strategies• “Interface constraints” for model
tracing• Control flow and production chains• Writing hints and bug productions• Creating a curriculum
32
Communicating About Student Solution Steps between Interface and Tutor
• Represent each observable action in the user interface as a selection-action-input triple.
Selection = (column2)Action = write-carryInput = <whatever student typed>
Selection = (column1)Action = write-sumInput = <whatever student typed>
33
Execution Space Diagram with Selection-Action-Input Triples
1 264+ 716 80
1 264+ 716 0
264+ 716
1 264+ 716 980
264+ 716 0
1 264+ 716
Selection: Column1Action: Write-ResultInput: 0
Selection: Column2Action: Write-ResultInput: 8
Selection: Column3Action: Write-ResultInput: 9
Selection: Column2Action: Write-CarryInput: 1
Selection: Column1Action: Write-ResultInput: 0
Selection: Column2Action: Write-CarryInput: 1
1 264+ 716 980 Done.
Selection: -Action: DoneInput: -
34
Interface Constraints—Matching Student Steps Against Model Steps
• Production rule writer can attach “model-tracing” or “interface constraints” to rules.– Specify tests on the selection-action-input
triples.
• During model-tracing TDK will not consider productions whose constraints are violated.
35
CONSTRAINTS—consider rule only if:Selection is:Action is:Input is:
Example of Interface Constraints in English
WRITE-SUMIF There is a goal to write Sum as the result in column C And Sum < 10 And the carry into column C (if any) has been addedTHEN Write Sum as the result in column C And remove the goal
Same as value of variable CWrite-ResultEqual to the value of variable Sum
Q: Do we really need a constraint on the Selection AND a constraint on the Action AND a constraint on the Input? What if we left out one?
A: Yes, each is necessary. Without a constraint on the Selection, the rule will accept the student step if the number typed is correct, regardless of the column in which it was typed. Without a constraint on the Action, the the rule will accept the step if the right number was typed in the right column, regardless of whether it was in the carry spot or at the bottom of the column. Etc.
36
Example of Interface Constraints in English
WRITE-SUMIF There is a goal to write Sum as the result in column C And Sum < 10 And the carry into column C (if any) has been addedTHEN Write Sum as the result in column C And remove the goal
CONSTRAINTS—student step matches only if:Selection is:Action is:Input is:
Same as value of variable CWrite-ResultEqual to the value of variable Sum
37
More Detail on Interface Constraints
• In Tutor Mode– When evaluating student input
• TDK will not consider productions whose constraints are violated.
– When responding to a help request• Tries to find the production path with the least
severe constraint violation.– from least to most severe: input, action, selection.
• In Stand-Alone Mode, interface constraints are ignored
• Convenient for development—can run model without having to supply input.
TDK Notation for Interface Constraints(defproduction write-sum addition (=problem) =problem>
isa addition-problem . . . =subgoal> isa write-sum-goal . . .==>
=column>
result =sum =problem> subgoals ($sg1 $sg2) ; the remaining subgoals
:nth-selection 0 =column :action 'write-result :input =sum #'equal-value-p)
See Example 4 from the Examples menu.
The first item of Selections must be equal to the value of =column.
The Action must be write-result.
The value of Input must be equal to the value of =sum. Use function equal-value-p to compare.
39
Syntax of TDK Notation for Interface Constraints
:nth-selection n value [test-fn]:action value [test-fn]:input value [test-fn]
40
Overview• Basics of TDK Notation• Matching: $ and = variables• Modeling alternative strategies• “Interface constraints” for model
tracing• Control flow and production
chains• Writing hints and bug productions• Creating a curriculum
41
Creating a Control Flow Diagram
• Control Flow Diagram elements:– Boxes are goals or actions– Arrows are production rules– A “path” or “chain” through the diagram starts at a
goal and ends with an action
• Crucial idea:– Each chain should produce exactly one action– When you “cycle” the TDK production system a “chain”
of productions fire
42
Production Rules Set New Goals & Perform Actions
Goal: Solve the addition problem
ADD-CARRY
264 + 716FOCUS-ON-FIRST-COLUMN,
FOCUS-ON-NEXT-COLUMN
Goal: Process column C
Goal: Write carry in next column
MUST-CARRY
WRITE-SUM
Action: Write the sum
Action: Write the carry
WRITE-CARRY
Goal: Write sum in column C
ADD-ADDENDS
43
(defproduction focus-on-next-column multi-column-addition
(=problem) =problem> isa multi-column-addition-problem columns ($ =next-column =previous-column $) =previous-column> isa column - result NIL =next-column> isa column result NIL first-addend =num1 second-addend =num2 !eval! (or =num1 =num2) ==> =process-column-goal> isa process-column-goal column =next-column !chain! multi-column-addition (=process-column-goal))
Chaining of Production Rules—Multiple Productions per Step
(defproduction write-result multi-column-addition (=subgoal)
=subgoal> isa process-column-goal column =column =column> isa column first-addend =num1 second-addend =num2 ==> !eval! =sum (+ (or =num1 0) (or =num2 0)) =column> result =sum)
Parameters
!chain! specifies the production set and parameters and initiates another attempt to match productions to this parameter & WM.
Name of the production rule set
No !chain! so this rule ends the chain.
44
Important Design Principle
• Design your production rule model so that there is exactly one cycle per observable (student) action.
45
Overview• Basics of TDK Notation• Matching: $ and = variables• Modeling alternative strategies• “Interface constraints” for model
tracing• Control flow and production chains• Writing hints and bug productions• Creating a curriculum
46
Example Hint Sequences for Addition Tutor
Start with the column on the right. This is the "ones" column.You need to add the two digits in this column. Adding 4 and 6 gives 10.The sum that you have, 10, is greater than 9. You need to 10 of the 10 into the
second column. And you need to write the rest of the 10 at the bottom of the first column.
Write 0 at the bottom of the rightmost column.
You need to complete the work on the first column.Write the carry from the first to the next column.Write 1 at the top of the second column from the right.
Now move on to the second column from the right. This is the "tens" column.You need to add the two digits in this column. Adding 6 and 1 gives 7.There is a carry into this column, so you need to add the value carried in. This gives
7 + 1 equals 8.Write 8 at the bottom of the second column from the right
264+ 716
264+ 716 0
1 264+ 716
47
Design of Hint Messages
– What makes a good hint sequence?• Short• Clear (minimum of jargon, short sentences)• Tells student what to do• But also why/when that action is appropriate• Goes from the general to the specific
– Writing hints can help in designing the production rule model.
– Hint sequences often combine hints from multiple productions in a chain.
48
(defproduction focus-on-next-column addition (=problem) =problem> isa addition-problem . . . =previous-column> isa column . . . name =name position =pos !eval! (or =num1 =num2 =carry) ==> . . . !chain! addition (=problem)
:messages (help `(Now move on to the #\space ,=pos column from the right #\. ~n ~n This is the #\space ,=name column #\.)))
Example Hint Template
Attach list of hint templates to the rule.
49
• Example hints1. Now move on the the second column from the right.
This is the “tenths” column.
2. Next big event: the second column from the right!!!
• Each template is a Lisp form that evaluates to a list. :messages (help `(Now move on to the #\space ,=pos column from the right #\. ~n ~n This is the #\space ,=name column #\.)
`(Next big event #\: the #\space ,=pos column from the right #\! #\! #\! )
)
2nd Example Hint Template
50
• Example hints1. Now move on the the second column from the right.
This is the “tenths” column.
2. Next big event: the second column from the right!!!
• Each template is a Lisp form that evaluates to a list. :messages (help `(Now move on to the #\space ,=pos column from the right #\. ~n ~n This is the #\space ,=name column #\.)
`(Next big event #\: the #\space ,=pos column from the right #\! #\! #\! )
)
2nd Example Hint Template
51
• Example hints1. Now move on the the second column from the right.
This is the “tenths” column.
2. Next big event: the second column from the right!!!
• Each template is a Lisp form that evaluates to a list. :messages (help `(Now move on to the #\space ,=pos column from the right #\. ~n ~n This is the #\space ,=name column #\.)
`(Next big event #\: the #\space ,=pos column from the right #\! #\! #\! )
)
2nd Example Hint Template
52
• Example hints from this template1. Now move on the the second column from the right.
This is the “tenths” column.
2. Next big event: the second column from the right!!!
• Each template is a Lisp form that evaluates to a list. :messages (help `(Now move on to the #\space ,=pos column from the right #\. ~n ~n This is the #\space ,=name column #\.)
`(Next big event #\: the #\space ,=pos column from the right #\! #\! #\! )
)
2nd Example Hint Template
For punctuation and spaces, use #\ notation.
To insert value of variable: comma followed by variable name.
~n produces new line.
Backquote.
When inserting a string, put a space in front of it.
53
Sequences Combining Hints from Multiple Productions
Start with the column on the right. This is the "ones" column.You need to add the two digits in this column. Adding 4 and 6
gives 10.The sum that you have, 10, is greater than 9. You need to
move 10 of the 10 into the second column. And you need to write the rest of the 10 at the bottom of the first column.
Write 0 at the bottom of the rightmost column.
You need to complete the work on the first column.Write the carry from the first to the next column.Write 1 at the top of the second column from the right.
Now move on to the second column from the right. This is the "tens" column.
You need to add the two digits in this column. Adding 6 and 1 gives 7.
There is a carry into this column, so you need to add the value carried in. This gives 7 + 1 equals 8.
Write 8 at the bottom of the second column from the right
264+ 716
264+ 716 0
1 264+ 716
FOCUS-ON-FIRST-COLUMN
ADD-ADDENDS
MUST-CARRY
WRITE-SUM
WRITE-CARRY
FOCUS-ON-NEXT-COLUMN
ADD-ADDENDS
ADD-CARRY
WRITE-SUM
54
How could you model students who don’t carry?
• Instead of doing the addition correctly:
• Can you model a student who writes:
• Can you write a “bug rule” that models this error?
1 264+ 716 980
264+ 716 970
55
Production Rules Set New Goals & Perform Actions
Goal: Solve the addition problem
ADD-CARRY
264 + 716FOCUS-ON-FIRST-COLUMN,
FOCUS-ON-NEXT-COLUMN
Goal: Process column C
Goal: Write carry in next column
MUST-CARRY
WRITE-SUM
Action: Write the sum
Action: Write the carry
WRITE-CARRY
Goal: Write sum in column C
ADD-ADDENDS
56
How to come up with a bug rule?
• Intuition: Need to model the “forgetting” of the subgoal to carry into the next column.
• Normal processing:
• If you then
• You will pretty soon be in situation:
264+ 716
Subgoal 1: write-sum-goal write 0 as result in the first column.Subgoal 2: write-carry-goal write 1 as carry in second column
Forget Subgoal 2, and continue as if nothing has happened
264+ 716 70
Solution 1: Explicit Forgetting Rule;; BUG-FORGOT-WRITE-CARRY-GOAL;; IF The goal is to do an addition problem;; And there is a pending subgoal to write a carry;; THEN Delete that pending subgoal (w/o carrying it out).;; (And carry on with the addition problem in the usual manner.)
(defproduction-bug BUG-FORGOT-WRITE-CARRY-GOAL addition (=problem) =problem> isa addition-problem subgoals ($sg1 =write-carry-goal $sg2) =write-carry-goal> isa write-carry-goal ==> =problem> subgoals ($sg1 $sg2) ; Remove the write-carry-goal. !chain! addition (=problem) :messages (bug `(First #\, you need to write the carry from the previous column #\. ) ))
Use defproduction-bug instead of defproduction.
Use :messages (bug … ) instead of :messages (help … )
;; BUG-FORGOT-WRITE-CARRY-GOAL;; IF The goal is to do an addition problem;; And there is a pending subgoal to write a carry;; THEN Delete that pending subgoal (w/o carrying it out).;; (And carry on with the addition problem in the;; usual manner.)
(defproduction-bug BUG-FORGOT-WRITE-CARRY-GOALaddition (=problem)
=problem> isa addition-problem subgoals ($sg1 =write-carry-goal $sg2) =write-carry-goal> isa write-carry-goal ==> =problem> subgoals ($sg1 $sg2) !chain! addition (=problem) :messages (bug `(First #\, you need to write the carry from the previous column #\. ) ))
Solution 1: Explicit Forgetting RuleSome questions & details•Why does the rule chain?•Why does the rule fire only when the student forgets to add a carry, and not for any wrong answer?•Does this work when action “done” follows after 97+42=39?•Does this rule fire under different circumstances than expected?•YES - after MUST-CARRY in the same cycle. Not intended but not a problem.
Some questions & details•Why does the rule chain?•Why does the rule fire only when the student forgets to add a carry, and not for any wrong answer?•Does this work when action “done” follows after 97+42=39?•Does this rule fire under different circumstances than expected?•YES - after MUST-CARRY in the same cycle. Not intended but not a problem.
59
Overview• Basics of TDK Notation• Matching: $ and = variables• Modeling alternative strategies• “Interface constraints” for model
tracing• Control flow and production chains• Writing hints and bug productions• Creating a curriculum
60
Setting Up a Curriculum
• Organize curriculum early on during the development of your tutoring system.
• Organize the tutor curriculum– Sequencing of skills– Break down curriculum into lessons and
sections– Each section involves a handful of “skills”– Assign problem sets to each section
61
Setting Up Knowledge Tracing and Mastery Learning• Indicate how to group the crucial
productions into traced skills.– Each traced skill corresponds to one or more
productions.
• For each skill, provide values for the four knowledge tracing parameters.
• For each problem, indicate which productions are needed to solve it.– Enables tutor to select problems that
involve un-mastered skills.
Example: Addition Skills• Start — Start in the rightmost column
FOCUS-ON-FIRST-COLUMN
• Add — Add the addends in a columnADD-ADDENDS
• Write — Write the sum in a columnWRITE-SUM ADDITION
• Done — Determine that all columns are doneDONE ADDITION
• Next — Focus on the next columnFOCUS-ON-NEXT-COLUMN
• Regroup — Realize that 10 goes to next columnMUST-CARRY
• Write Carry — Write the carry
WRITE-CARRY • Add Carry — Add the carry to the sum
ADD-CARRY ADDITION
63
Example from Addition Tutor• Structure curriculum, based on skills
Section1 — one column, no carrynew skills: Start, Add, Write, Done
Section2 — two columns, no carrynew skills: Next
Section3 — two columns, possible carrynew skills: Regroup, Mark Carry, Add Carry
Section4 — three columns, possible carrynew skills: -
64
Excerpt from Lesson File
(Make-Section :Required-Problems '(Problem11 Problem12 Problem15) :Remedial-Problems '(Problem13 Problem14 Problem16 Problem17 Problem18 Problem19) :Skills '(((Focus-On-First-Column Addition) :Description "Start") ((Done Addition) :Description "Done") ((Add-Addends Addition) :Description "Add") ((Write-Sum Addition ) :Description "Write") ))
65
Set Knowledge-Tracing Parameters
Parameter(setf *p-initial* 0.33)
(setf *p-learn* 0.20)
(setf *p-guess* 0.20)
(setf *p-slip* 0.10)
(setf *p-known-p* 0.95)
Represents probability that
the student has learned the skill prior to the instruction
the student learns the skill when successfully applying it
the student gets the answer right even though the skill is not mastered
the student gets the answer wrong even though the skill is mastered
Represents mastery threshold
66
Provide Production Counts
For problem
(or any other 2-column problem without carrying)
(set-problem-production-counts problem26+51 '(((FOCUS-ON-FIRST-COLUMN ADDITION) . 1) ((FOCUS-ON-NEXT-COLUMN ADDITION) . 1) ((ADD-ADDENDS ADDITION) . 2) ((WRITE-SUM ADDITION) . 2) ((DONE ADDITION) . 1) ))))
26+ 51 77
67
Collecting Logs ofStudent-Tutor Interactions in
TDK• Create “protocol collectors” when
starting tutor (make-instance 'global protocol-collector :broadcast-station broadcast-station) (make-instance 'protocol-collector :broadcast-station broadcast-station) ; add to function ; make-addition-tutor
• Set directory parameters(setf *global-protocol-directory* "Home:Globals;")(setf *protocol-directory* "Home:Protocols;")
• Flip the switch(setf *collect-protocols* t)
Example Protocol from Addition Tutor
*000:00:01 START STUDENT; "Student3", USER-DATA; NIL, PROBLEM; PROBLEM15, NPRESENTED; (0 0), START-TIME; "07/18/01 21:57", FILE-NAME; "Macintosh HD:Desktop
Folder:Vincent:CIRCLE Summer School:TDK:Protocols:LESSON1:Student3:1%PROBLEM15%0.p-0",
LESSON; LESSON1, LESSON-PRINT-NAME; "", SECTION; SECTION155, SECTION-PRINT-NAME; "",.*000:00:24 INITIAL-PROBABILITIES p/p-1; (0.6762857412475228 (ADD-ADDENDS
ADDITION)), p/p-2; (0.9451767563766541 (DONE
ADDITION)), p/p-3; (0.6762857412475228 (FOCUS-ON-
FIRST-COLUMN ADDITION)), p/p-4; (0.6762857412475228 (WRITE-SUM
ADDITION)),.*000:00:40 FOCUS SELECTED-WMES; (COLUMN1),.*000:14:926 APPLY-ACTION WINDOW; ADDITION-WINDOW, CONTEXT; ROOT, SELECTIONS; (COLUMN1), ACTION; WRITE-RESULT, INPUT; "5",.
*000:14:939 UPDATE-P-KNOW META; NIL, PRODUCTION; (FOCUS-ON-FIRST-COLUMN
ADDITION), SUCCESS?; NIL, P-KNOW; 0.3656548145323001,.*000:14:945 UPDATE-P-KNOW META; NIL, PRODUCTION; (ADD-ADDENDS ADDITION), SUCCESS?; NIL, P-KNOW; 0.3656548145323001,.*000:14:952 UPDATE-P-KNOW META; NIL, PRODUCTION; (WRITE-SUM ADDITION), SUCCESS?; NIL, P-KNOW; 0.3656548145323001,.*000:14:958 NO-CONFLICT-SET .*000:14:962 READY .*000:17:41 SUGGESTED-SELECTION ITEMS; (WRITE-RESULT COLUMN1),.*000:17:46 HELP SELECTION; (COLUMN1), NHINT; 0,.#000:17:50 HELP-TEXT You need to add the two digits in this column.
Adding 3 and 3 gives 6..
*000:17:53 HELP-PATH P-1; (FOCUS-ON-FIRST-COLUMN ADDITION), P-2; (ADD-ADDENDS ADDITION), P-3; (WRITE-SUM ADDITION),.*000:17:68 READY .*000:19:246 SUGGESTED-SELECTION ITEMS; (WRITE-RESULT COLUMN1),.*000:19:252 HELP SELECTION; (COLUMN1), NHINT; 1,.#000:19:256 HELP-TEXT Write 6 at the bottom of the rightmost column..*000:19:259 HELP-PATH P-1; (FOCUS-ON-FIRST-COLUMN ADDITION), P-2; (ADD-ADDENDS ADDITION), P-3; (WRITE-SUM ADDITION),.*000:19:270 READY .*000:21:94 WINDOW-SELECT WINDOW-TITLE; ADDITION-WINDOW,.*000:23:765 APPLY-ACTION WINDOW; ADDITION-WINDOW, CONTEXT; ROOT, SELECTIONS; (COLUMN1), ACTION; WRITE-RESULT, INPUT; "6",.
*000:23:774 GOOD-PATH .*000:23:778 HISTORY P-1; (FOCUS-ON-FIRST-COLUMN ADDITION), P-2; (ADD-ADDENDS ADDITION), P-3; (WRITE-SUM ADDITION),.*000:23:782 READY .*000:25:84 APPLY-ACTION WINDOW; ADDITION-WINDOW, CONTEXT; ROOT, SELECTIONS; NIL, ACTION; DONE, INPUT; NIL,.*000:25:94 UPDATE-P-KNOW META; NIL, PRODUCTION; (DONE ADDITION), SUCCESS?; T, P-KNOW; 0.9898195480465753,.*000:25:99 GOOD-PATH .*000:25:102 HISTORY P-1; (DONE ADDITION),.*000:25:106 READY .*000:25:109 PROBLEM-COMPLETED .*000:25:113 FINAL-PROBABILITIES p/p-1; (0.3656548145323001 (ADD-ADDENDS
ADDITION)), p/p-2; (0.9898195480465753 (DONE ADDITION)), p/p-3; (0.3656548145323001 (FOCUS-ON-FIRST-COLUMN
ADDITION)), p/p-4; (0.3656548145323001 (WRITE-SUM ADDITION)),.
69
Example Uses of Log Data• To improve model and tutor
– Is skill decomposition accurate?• If “learning curves” for skills do not follow power law,
crucial difficulty factors may have been overlooked.
– Tune knowledge tracing parameters• Fit the parameters to the log data, use the new
parameter values next time around.
– What kind of errors do students make?• Any use for new bug messages?
– Do we have enough different problems?
• For research– Relation between help use and learning outcomes.– Compare learning rates in comparative studies.
70
END