SimStudent:A Computational Model of Learning as a Research Toolbox for the Sciences of Learning
Noboru Matsuda
Human-Computer Interaction Institute
Carnegie Mellon University
Sim
Stu
den
t Pro
ject
PSLC Summer School 2012 :: SimStudent :: Noboru Matsuda (CMU)
Research Questions
• Building a cognitive model is hard. Can machine-learning techniques help non-expert authors build a Cognitive Tutor?
• Would like to simulate students’ learning. Can machine-learning techniques help us build a computational model of learning with a cognitive fidelity?
• I heard that students learn by teaching others. Can we use the computational model of learning to study the theory of learning by teaching?
2
Sim
Stu
den
t Pro
ject
PSLC Summer School 2012 :: SimStudent :: Noboru Matsuda (CMU) 3
Solution: SimStudent
• Machine learning agent
– Learns procedural skills, by
– Observing model solutions & solving problems
• Fundamental technology
– Programming by Demonstration
– Inductive Logic Programming
• Knowledge representation
– Production rules (Jess)
Lau & Weld (1998). Blessing (1997).
Sim
Stu
den
t Pro
ject
PSLC Summer School 2012 :: SimStudent :: Noboru Matsuda (CMU)
SimStudent Projects
• Intelligent Authoring– Building a Cognitive Tutor as a CTAT Plug-in
• Student Modeling and Simulation– Controlled educational studies
– Error formation study
– Prerequisite conceptual knowledge study
• Teachable Peer Learner– Learning by teaching
4
Sim
Stu
den
t Pro
ject
PSLC Summer School 2012 :: SimStudent :: Noboru Matsuda (CMU) 5
Authoring a Cognitive Tutor
• Example-Tracing Tutor
– Little programming
– A cognitive model specific to a particular problem
• Limited generalization by editing a behavior graph
• Model-Tracing Tutor
– Powerful student model– Cognitive task analysis is hard– Writing production rules is even more challenging
• Performing a task is relatively easy…
Sim
Stu
den
t Pro
ject
PSLC Summer School 2012 :: SimStudent :: Noboru Matsuda (CMU) 6
Next Generation AuthoringBuild a tutor GUI
Teaching a solution
SimSt. learning
Rule simplify-LHS:
IF is-equation( Eq ),
is-lhs( Eq, Lhs ),
polynomial( Lhs ),
all-var-terms( Lhs )
Then simplify( Lhs, S-lhs ),
enter( S-lhs )
Production Rules
Rule simplify-LHS:
IF is-equation( Eq ),
is-lhs( Eq, Lhs ),
polynomial( Lhs ),
all-var-terms( Lhs )
Then simplify( Lhs, S-lhs ),
enter( S-lhs )
Rule simplify-LHS:
IF is-equation( Eq ),
is-lhs( Eq, Lhs ),
polynomial( Lhs ),
all-var-terms( Lhs )
Then simplify( Lhs, S-lhs ),
enter( S-lhs )
Sim
Stu
den
t Pro
ject
PSLC Summer School 2012 :: SimStudent :: Noboru Matsuda (CMU)
Demo
7
Authoring by Tutoring
Example: Learning to subtract a constant term
PSLC Summer School 2012 :: SimStudent :: Noboru Matsuda (CMU) 8
Learning to subtract a constant numberFirst example
subtract 1
Subtract the last term on the left-hand side…Subtract the last term on the left-hand side…
Subtract the coefficient of X…
Subtract the coefficient of X…
Subtract the difference between 4 and 3…
Subtract the difference between 4 and 3…
I see 3x, 1, x, and 4 in the equation.I wonder where the ‘1’ came from…
I see 3x, 1, x, and 4 in the equation.I wonder where the ‘1’ came from…
Example: Learning to subtract a constant termLearning to subtract a constant number
First example
subtract 1
Subtract the last term on the left-hand side…
Subtract the last term on the left-hand side…
Subtract the coefficient of X…Subtract the coefficient of X…
Subtract the difference between 4 and 3…Subtract the difference between 4 and 3…
I see 3x, 1, x, and 4 in the equation.I wonder where the ‘1’ came from…
I see 3x, 1, x, and 4 in the equation.I wonder where the ‘1’ came from…
PSLC Summer School 2012 :: SimStudent :: Noboru Matsuda (CMU) 9
Prior Knowledge• Feature predicates
– 18 predicates– isFractionTerm(X), isConstant(X), isPolynomial(X),…
• Operators– 42 operators– add(X,Y), coefficient(X), getFrstNumber(X),
…
10PSLC Summer School 2012 :: SimStudent :: Noboru Matsuda (CMU)
Sim
Stu
den
t Pro
ject
PSLC Summer School 2012 :: SimStudent :: Noboru Matsuda (CMU) 11
Example: Stoichiometry Tutor
Sim
Stu
den
t Pro
ject
PSLC Summer School 2012 :: SimStudent :: Noboru Matsuda (CMU)
SimStudent Projects
• Intelligent Authoring– Building a Cognitive Tutor as a CTAT Plug-in
• Student Modeling and Simulation– Controlled educational studies
– Error formation study
– Prerequisite conceptual knowledge study
• Teachable Peer Learner– Learning by teaching
12
Sim
Stu
den
t Pro
ject
PSLC Summer School 2012 :: SimStudent :: Noboru Matsuda (CMU)
Model of Incorrect Learning
• Identify errors students commonly make
• Weaken SimStudent’s background knowledge
• Let SimStudent make an induction error
13
Sim
Stu
den
t Pro
ject
PSLC Summer School 2012 :: SimStudent :: Noboru Matsuda (CMU)
Weak Prior Knowledge Hypothesis
• Multiple ways to make sense of examples
14
3x=5 “divide by 3”3x=5 “divide by 3”
Get a coefficient and divideGet a coefficient and divide
Get a number and divideGet a number and divide
Get a denominator and multiplyGet a denominator and multiply
4/x=5
“divide by 4”
“multiply by x ”
Sim
Stu
den
t Pro
ject
PSLC Summer School 2012 :: SimStudent :: Noboru Matsuda (CMU)
Results: Learning Rate
Steps Score = 0 (if there is no rule applicable)# correct rule applications / # all rule applications
# training problems
Step Score
Student Model A set of knowledge components (KCs) Encoded in intelligent tutors to model how
students solve problems E.g. How to proceed given problems of the form Nv=N
One of the key factors that affects automated tutoring systems in making instructional decisions
Previous Approach: Require expert input Highly subjective
Proposed Approach: Use a machine-learning agent, SimStudent, to acquire
knowledge 1 Skill 1 KC Skill application for that step KC for each step
(Li et al. EDM2011) A Machine Learning Approach for Automatic Student Model Discovery
Human-generated vs SimStudent KCsHuman-generated Model
SimStudent
Comment
Total # of KCs 12 21
# of Basic Arithmetic Operation KCs
4 13 Split into finer grain sizes based on different problem forms
# of Typein KCs 4 4 Approximately the same
# of Other Transformation Operation KCs (e.g. combine like terms)
4 4 Approximately the same
4x = 20 vs.
–x = 5
Results
Human-generated Model
SimStudent
AIC 6529 6448
3-Fold Cross Validation RMSE
0.4034 0.3997
Significance Test SimStudent outperforms the human-
generated model in 4260 out of 6494 steps p < 0.001
SimStudent outperforms the human-generated model across 20 runs of cross validation
p < 0.001
Sim
Stu
den
t Pro
ject
PSLC Summer School 2012 :: SimStudent :: Noboru Matsuda (CMU)
SimStudent Projects
• Intelligent Authoring– Building a Cognitive Tutor as a CTAT Plug-in
• Student Modeling and Simulation– Controlled educational studies
– Error formation study
– Prerequisite conceptual knowledge study
• Teachable Peer Learner– Learning by teaching
19
Sim
Stu
den
t Pro
ject
PSLC Summer School 2012 :: SimStudent :: Noboru Matsuda (CMU)
Learning by Teaching SimStudent
Sim
Stu
den
t Pro
ject
PSLC Summer School 2012 :: SimStudent :: Noboru Matsuda (CMU)
Learn more about SimStudents
• Project Web– www.SimStudent.org
• Contact us– Noboru Matsuda
21
Top Related