Post on 01-Jan-2016
description
SimStudent:A computational model of learning for Intelligent Authoring and beyond
Noboru Matsuda
Human-Computer Interaction Institute
Carnegie Mellon University
Sim
Stu
den
t Pro
ject
PSLC Summer School 2010 :: SimStudent Demo :: Noboru Matsuda (CMU) 2
CTAT: Cognitive Tutor Authoring Tools
• Example-Tracing Tutor
– Zero programming
– A cognitive model specific to a particular problem
– Limited generalization by editing a behavior graph
• Model-Tracing Tutor
– Requires a cognitive model– Cognitive task analysis is hard– Writing production rules is even more challenging
• Performing a task is much easier…
Sim
Stu
den
t Pro
ject
PSLC Summer School 2010 :: SimStudent Demo :: Noboru Matsuda (CMU) 3
Next Generation AuthoringBuild a tutor GUI
Teaching a solution
SimSt. learning
Rule simplify-LHS:
IF is-equation( Eq ),
is-lhs( Eq, Lhs ),
polynomial( Lhs ),
all-var-terms( Lhs )
Then simplify( Lhs, S-lhs ),
enter( S-lhs )
Production Rules
Rule simplify-LHS:
IF is-equation( Eq ),
is-lhs( Eq, Lhs ),
polynomial( Lhs ),
all-var-terms( Lhs )
Then simplify( Lhs, S-lhs ),
enter( S-lhs )
Rule simplify-LHS:
IF is-equation( Eq ),
is-lhs( Eq, Lhs ),
polynomial( Lhs ),
all-var-terms( Lhs )
Then simplify( Lhs, S-lhs ),
enter( S-lhs )
Sim
Stu
den
t Pro
ject
PSLC Summer School 2010 :: SimStudent Demo :: Noboru Matsuda (CMU) 4
SimStudent
• Machine learning agent
– Learns problem-solving steps by …
– Observes model solutions / solving problems, and …
– Outputs a set of production rules
• Fundamental technology
– Programming by Demonstration
– Inductive Logic Programming
Lau & Weld (1998). Blessing (1997).
Sim
Stu
den
t Pro
ject
PSLC Summer School 2010 :: SimStudent Demo :: Noboru Matsuda (CMU)
Authoring Strategies
• Authoring by demonstration– Demonstrate whole solutions
– Learning from worked-out examples• Demonstrated solutions as positive examples
• Authoring by tutoring– Interactively tutor with immediate feedback and hint
– Learning by tutored problem-solving• Learning by generalizing hint with taking feedback into
account
5
Sim
Stu
den
t Pro
ject
PSLC Summer School 2010 :: SimStudent Demo :: Noboru Matsuda (CMU)
Demo!
6
Authoring by Tutoring
Sim
Stu
den
t Pro
ject
PSLC Summer School 2010 :: SimStudent Demo :: Noboru Matsuda (CMU)
Learning Production Rules in 3 parts: What-When-How
If
among this and that GUI elements
such and such constraints hold
Then
do actions with the GUI elements
When
What
How
Sim
Stu
den
t Pro
ject
PSLC Summer School 2010 :: SimStudent Demo :: Noboru Matsuda (CMU)
Production Rule in JESS
(defrule trans-lr-lhs
?problem <- (problem (interface-elements ?table1 ? ? ?))?table1 <- (table (columns ?column1))?column1 <- (column (cells $?m1 ?cell0 $?))?cell0 <- (cell (value ?val0&~nil))
?problem <- (problem (interface-elements ? ?table2 ? ?))?table2 <- (table (columns ?column2))?column2 <- (column (cells $?m2 ?cell1 $?))?cell1 <- (cell (value ?val1&~nil))
?column1 <- (column (cells $?m3 ?cell2 $?))?cell2 <- (cell (name ?selection) (value ?input))
(test (consecutive-row ?cell0 ?cell2))(test (same-column ?cell0 ?cell2))(test (distinctive ?cell0 ?cell2))(test (consecutive-row ?cell1 ?cell2))(test (same-column ?cell1 ?cell2))(test (distinctive ?cell1 ?cell2))
(test (polynomial ?val0))(test (not (has-var-term ?val1)))
=>
(bind ?input (first-var-term ?val0))(modify ?cell2 (value ?input)) )
LHS
RHS
WME path
Topologicalconstraints
Featureconstraints
WMEconditions
Working Memory Element (WME)(defrule trans-lr-lhs
?problem <- (problem (interface-elements ?table1 ? ? ?))?table1 <- (table (columns ?column1))?column1 <- (column (cells $?m1 ?cell0 $?))?cell0 <- (cell (value ?val0&~nil))
?problem <- (problem (interface-elements ? ?table2 ? ?))?table2 <- (table (columns ?column2))?column2 <- (column (cells $?m2 ?cell1 $?))?cell1 <- (cell (value ?val1&~nil))
?column1 <- (column (cells $?m3 ?cell2 $?))?cell2 <- (cell (name ?selection) (value ?input))
(test (consecutive-row ?cell0 ?cell2))(test (same-column ?cell0 ?cell2))(test (distinctive ?cell0 ?cell2))(test (consecutive-row ?cell1 ?cell2))(test (same-column ?cell1 ?cell2))(test (distinctive ?cell1 ?cell2))
(test (polynomial ?val0))(test (not (has-var-term ?val1)))
=>
(bind ?input (first-var-term ?val0))(modify ?cell2 (value ?input)) )
LHS
RHS
WME path
Topologicalconstraints
Featureconstraints
WMEconditions
Working Memory Element (WME) GUI elements
Constraints
Actions
Sim
Stu
den
t Pro
ject
PSLC Summer School 2010 :: SimStudent Demo :: Noboru Matsuda (CMU) 9
Background Knowledge
• Domain concepts to “explain” demonstrations– Operators
– Feature predicates
• External Jess function written in Java
(defrule multi-lhs…?var22140 <- (column (cells ? ? ?var22143 ? ? ? ? ?))?var22143 <- (cell (value ?val0&~nil))
(test (fraction-term ?val0 )) =>
(bind ?val2 (denominator ?val0))(bind ?input (mul-term-by ?val0 ?val2))…
)
Sim
Stu
den
t Pro
ject
PSLC Summer School 2010 :: SimStudent Demo :: Noboru Matsuda (CMU) 10
Example : Algebra domain•16 Feature predicates & 28 operators
Feature Predicates for LHS conditions Operators for RHS actions
HasCoefficient
HasConstTerm
HasVarTerm
Homogeneous
IsFractionTerm
IsConstant
IsDenominatorOf
IsNumeratorOf
IsPolynomial
Monomial
NotNull
VarTerm
IsSkillAdd
IsSkillSubtract
IsSkillDivide
IsSkillMultiply
AddTerm
AddTermBy
Coefficient
CopyTerm
Denominator
DivTerm
DivTermBy
EvalArithmetic
FirstTerm
FirstVarTerm
GetOperand
InverseTerm
LastConstTerm
LastTerm
LastVarTerm
MulTerm
MulTermBy
Numerator
ReverseSign
RipCoefficient
SkillAdd
SkillClt
SkillDivide
SkillMultiply
SkillRf
SkillMt
SkillSubtract
VarName
Sim
Stu
den
t Pro
ject
PSLC Summer School 2010 :: SimStudent Demo :: Noboru Matsuda (CMU) 11
Learning Results
€
StepScore =0, if C = 0
C
C + I
⎧ ⎨ ⎪
⎩ ⎪
# of training tasks
% C
orr
ect
rule
firi
ng
s (
10
test
tasks) Authoring by
tutoring
Authoring by demonstration
Betterthan
Sim
Stu
den
t Pro
ject
PSLC Summer School 2010 :: SimStudent Demo :: Noboru Matsuda (CMU) 12
Authoring TimeAuthoring by Tutoring
Authoring by Demonstration
• Authoring by tutoring took 86 minutes
• Authoring by demonstration took 238 minutes
• A 2.8x speed-up!
Sim
Stu
den
t Pro
ject
PSLC Summer School 2010 :: SimStudent Demo :: Noboru Matsuda (CMU) 13
Example: Stoichiometry Tutor
Approximate Time Distribution (hours)
55
30
15
40
Operator generation
Feature predicate generation
Debugging
Testing
Sim
Stu
den
t Pro
ject
PSLC Summer School 2010 :: SimStudent Demo :: Noboru Matsuda (CMU)
SimStudent Projects
• Intelligent Authoring– Building a Cognitive Tutor as a CTAT Plug-in
• Student Modeling and Simulation– Controlled educational studies
– Error formation study
– Prerequisite conceptual knowledge study
• Teachable Peer Learner– Learning by teaching
14
Sim
Stu
den
t Pro
ject
PSLC Summer School 2010 :: SimStudent Demo :: Noboru Matsuda (CMU)
SimStudent Projects
• Intelligent Authoring– Building a Cognitive Tutor as a CTAT Plug-in
• Student Modeling and Simulation– Controlled educational studies
– Error formation study
– Prerequisite conceptual knowledge study
• Teachable Peer Learner– Learning by teaching
15
Sim
Stu
den
t Pro
ject
PSLC Summer School 2010 :: SimStudent Demo :: Noboru Matsuda (CMU)
Model of Incorrect Learning
• Identify errors students commonly make
• Weaken SimStudent’s background knowledge
• Let SimStudent make an induction error
16
Sim
Stu
den
t Pro
ject
PSLC Summer School 2010 :: SimStudent Demo :: Noboru Matsuda (CMU)
Weak Prior Knowledge Hypothesis
• Multiple ways to make sense of examples
17
3x=5 “divide by 3”3x=5 “divide by 3”
Get a coefficient and divideGet a coefficient and divide
Get a number and divideGet a number and divide
Get a denominator and multiplyGet a denominator and multiply
4/x=5
strong prior knowledge
weak prior knowledge
“divide by 4”
“multiply by x”
Sim
Stu
den
t Pro
ject
PSLC Summer School 2010 :: SimStudent Demo :: Noboru Matsuda (CMU)
Target Common Errors
18
€
ax − b = c → "subtract b"
€
ax = b → "multiply by a"€
ax + b = c → "add b"
€
x
a= b → "divide by a"
Sim
Stu
den
t Pro
ject
PSLC Summer School 2010 :: SimStudent Demo :: Noboru Matsuda (CMU)
Results: Learning Rate
Steps Score = 0 (if there is no rule applicable)# correct rule applications / # all rule applications
# training problems
Step Score
Sim
Stu
den
t Pro
ject
PSLC Summer School 2010 :: SimStudent Demo :: Noboru Matsuda (CMU)
SimStudent Projects
• Intelligent Authoring– Building a Cognitive Tutor as a CTAT Plug-in
• Student Modeling and Simulation– Controlled educational studies
– Error formation study
– Prerequisite conceptual knowledge study
• Teachable Peer Learner– Learning by teaching
20
Sim
Stu
den
t Pro
ject
PSLC Summer School 2010 :: SimStudent Demo :: Noboru Matsuda (CMU)
Learning by Teaching SimStudent
Sim
Stu
den
t Pro
ject
PSLC Summer School 2010 :: SimStudent Demo :: Noboru Matsuda (CMU)
Demo!
22
Learning by Teaching
Sim
Stu
den
t Pro
ject
PSLC Summer School 2010 :: SimStudent Demo :: Noboru Matsuda (CMU)
Learn more about SimStudents
• Project Web– www.SimStudent.org
• Download & Tutorial– http://ctat.pact.cs.cmu.edu (linked from project web)
• Contact us!– mazda@cs.cmu.edu
23