In-vivo Experimentation
description
Transcript of In-vivo Experimentation
![Page 1: In-vivo Experimentation](https://reader036.fdocuments.in/reader036/viewer/2022062305/568161b3550346895dd179af/html5/thumbnails/1.jpg)
©2012 Carnegie Learning, Inc.
In-vivo Experimentation
Steve RitterFounder and Chief Scientist
Carnegie Learning
![Page 2: In-vivo Experimentation](https://reader036.fdocuments.in/reader036/viewer/2022062305/568161b3550346895dd179af/html5/thumbnails/2.jpg)
©2012 Carnegie Learning, Inc.
An attempt to find meaning in three acts
• Design: Geometry Contiguity (Vincent Aleven, Kirsten Butcher)
• Modeling: Adjusting learning curve parameters (Cen, Koedinger, Junker)
• Personalization: Word problem content (Candace Walkington)
![Page 3: In-vivo Experimentation](https://reader036.fdocuments.in/reader036/viewer/2022062305/568161b3550346895dd179af/html5/thumbnails/3.jpg)
©2012 Carnegie Learning, Inc.
DESIGN
![Page 4: In-vivo Experimentation](https://reader036.fdocuments.in/reader036/viewer/2022062305/568161b3550346895dd179af/html5/thumbnails/4.jpg)
©2012 Carnegie Learning, Inc.
Geometry angles
![Page 5: In-vivo Experimentation](https://reader036.fdocuments.in/reader036/viewer/2022062305/568161b3550346895dd179af/html5/thumbnails/5.jpg)
©2012 Carnegie Learning, Inc.
ContiguityEarly Version Commercial Version
(Carnegie Learning)
Research Version
(Carnegie Mellon)
Butcher, K., & Aleven, V. (2008). Diagram interaction during intelligent tutoring in geometry: Support for knowledge retention and deep transfer. In C. Schunn (Ed.) Proceedings of the Annual Meeting of the Cognitive Science Society, CogSci 2008. New York, NY: Lawrence Earlbaum.
Hausmann, R.G.M. & Vuong, A. (2012) Testing the Split Attention Effect on Learning in a Natural Educational Setting Using an Intelligent Tutoring System for Geometry. In N. Miyake, D. Peebles, & R. P. Cooper (Eds.), Proceedings of the 34th Annual Conference of the Cognitive Science Society. (pp. 438-443). Austin, TX: Cognitive Science Society.
![Page 6: In-vivo Experimentation](https://reader036.fdocuments.in/reader036/viewer/2022062305/568161b3550346895dd179af/html5/thumbnails/6.jpg)
©2012 Carnegie Learning, Inc.
Early Tutor
![Page 7: In-vivo Experimentation](https://reader036.fdocuments.in/reader036/viewer/2022062305/568161b3550346895dd179af/html5/thumbnails/7.jpg)
©2012 Carnegie Learning, Inc.
Revised (commercial) tutor
![Page 8: In-vivo Experimentation](https://reader036.fdocuments.in/reader036/viewer/2022062305/568161b3550346895dd179af/html5/thumbnails/8.jpg)
©2012 Carnegie Learning, Inc.
Geometry Contiguity• Design and field experimentation– Butcher and Aleven (2008)• Diagram interaction led to better transfer
and retention• Analysis of impact– Hausmann and Vuong (2012)• Unit-level effects mixed• Advantage for harder skills
![Page 9: In-vivo Experimentation](https://reader036.fdocuments.in/reader036/viewer/2022062305/568161b3550346895dd179af/html5/thumbnails/9.jpg)
©2012 Carnegie Learning, Inc.
Geometry Angles
![Page 10: In-vivo Experimentation](https://reader036.fdocuments.in/reader036/viewer/2022062305/568161b3550346895dd179af/html5/thumbnails/10.jpg)
©2012 Carnegie Learning, Inc.
Lessons• Change is constant• Transition from research to
production always requires adaptation
![Page 11: In-vivo Experimentation](https://reader036.fdocuments.in/reader036/viewer/2022062305/568161b3550346895dd179af/html5/thumbnails/11.jpg)
©2012 Carnegie Learning, Inc.
MODELING
![Page 12: In-vivo Experimentation](https://reader036.fdocuments.in/reader036/viewer/2022062305/568161b3550346895dd179af/html5/thumbnails/12.jpg)
©2012 Carnegie Learning, Inc.
Skillometer
![Page 13: In-vivo Experimentation](https://reader036.fdocuments.in/reader036/viewer/2022062305/568161b3550346895dd179af/html5/thumbnails/13.jpg)
©2012 Carnegie Learning, Inc.
Expression Writing
![Page 14: In-vivo Experimentation](https://reader036.fdocuments.in/reader036/viewer/2022062305/568161b3550346895dd179af/html5/thumbnails/14.jpg)
©2012 Carnegie Learning, Inc.
What gets learned?
![Page 15: In-vivo Experimentation](https://reader036.fdocuments.in/reader036/viewer/2022062305/568161b3550346895dd179af/html5/thumbnails/15.jpg)
©2012 Carnegie Learning, Inc.
Bayesian Knowledge Tracing
Cognitive tutor traces these skills differently
![Page 16: In-vivo Experimentation](https://reader036.fdocuments.in/reader036/viewer/2022062305/568161b3550346895dd179af/html5/thumbnails/16.jpg)
©2012 Carnegie Learning, Inc.16
Learning Curve Parameter Fitting• Field study looking at learning area of geometric figures
– One group used adjusted learning parameters based on previous year’s data• Optimized group took 12% less time to reach same performance
• Significant learning gain in both groups• No difference in learning gain between groups (p = 0.772 )
0
20
40
60
80
100
120
Square
Parallelogram
Triangle
Trapezoid
Polygon
Circle
Optimized
Control
![Page 17: In-vivo Experimentation](https://reader036.fdocuments.in/reader036/viewer/2022062305/568161b3550346895dd179af/html5/thumbnails/17.jpg)
©2012 Carnegie Learning, Inc.
Lessons• Learning efficiency is a great
outcome• Small, systemic changes can have
big impact• Optimizing skills requires appropriate
skill model– Koedinger, McLaughlin and Stamper
(2012) - LFA
![Page 18: In-vivo Experimentation](https://reader036.fdocuments.in/reader036/viewer/2022062305/568161b3550346895dd179af/html5/thumbnails/18.jpg)
©2012 Carnegie Learning, Inc.
PERSONALIZATION
![Page 19: In-vivo Experimentation](https://reader036.fdocuments.in/reader036/viewer/2022062305/568161b3550346895dd179af/html5/thumbnails/19.jpg)
©2012 Carnegie Learning, Inc.
Word problem customization
![Page 20: In-vivo Experimentation](https://reader036.fdocuments.in/reader036/viewer/2022062305/568161b3550346895dd179af/html5/thumbnails/20.jpg)
©2012 Carnegie Learning, Inc.
Personalization field study• Students who got problems related
to their interests made fewer errors• Also affected subsequent unit• Interaction with readability
![Page 21: In-vivo Experimentation](https://reader036.fdocuments.in/reader036/viewer/2022062305/568161b3550346895dd179af/html5/thumbnails/21.jpg)
©2012 Carnegie Learning, Inc.
Lessons• Content matters– Challenge for knowledge component
modeling• Are we personalizing preferences,
reading level or both?
![Page 22: In-vivo Experimentation](https://reader036.fdocuments.in/reader036/viewer/2022062305/568161b3550346895dd179af/html5/thumbnails/22.jpg)
©2012 Carnegie Learning, Inc.
Summary• It’s not about whether A is better
than B– It’s about why A is better than B