Training in Experimental Design (TED):Developing Scalable and Adaptive Computer-based Science Instruction (Year 2)Stephanie Siler, Mari Strand Cary, Cressida Magaro, Kevin Willows, and David Klahr

Background

The ability to design and evaluate experiments is critical in both science classes and in the real world. Although elementary and middle school students often do not have these skills, they can be taught with relatively brief direct instruction (e.g., Chen & Klahr, 1999; Klahr & Nigam, 2004). However, such instruction is less successful in more challenging schools.

Instructional goals: • Significantly increase elementary and middle school children’s understanding of

scientific experimentation, including procedural (how to design good experiments) and conceptual (why experiments are/are not informative) understandings. Students will develop general schemas for designing and interpreting experiments that will enable them to correctly answer achievement test questions on experimental design.

• Close the achievement gap between high- and low-SES students in this specific, yet broadly applicable, area of science education.

Design goal: Develop a computer-based intelligent tutor for Training in Experimental Design (TED) that will provide adaptive instruction based on individuals’ reading level, knowledge state and mastery in real time across a variety of tasks and science content domains.

Participants: Fifth through seventh graders from five local Catholic schools serving a diverse student population in terms of SES, reading ability, and experience in experimental design.

Acknowledgments

This research was supported by the Department of Education,

Institute of Education Sciences (# R305H060034)

Version 2.2Human tutor-delivered

small group or one-to-one discussion-based instruction (computer assisted)

Adaptive: Groups divided based on pre-test diagnoses (rules/misconceptions); tutoring decisions adapted to students.

TED Tutor Evolution…

Year 2 Activity and Results

Our activities in Year 2 built on important results of our Year 1 work: • Learning of CVS was found to be highly correlated with reading ability. Thus, we sought to decrease

reading and processing demands in Version 2.1 to help lower-reading/knowledge students. Specifically, we presented the procedural rules prior to providing the reasoning for those rules.

• Students often misinterpreted the goal of instructional activities. They often thought that the primary instructional goal was to learn about the specific domain, rather than the more abstract, and domain-general goal of learning how to design good experiments. Thus, in Version 3.2, we will use variables with “nonsense” names as choices to help prevent students’ goal misconceptions.

• Identification of successful instructional techniques (e.g., “step-by-step” remedial instruction we will use in Version 3.3).

Was Version 2.1 an improvement over Version 1? • Immediate ramps post-test (near-transfer) scores suggest the answer is “no.” Although there was a trend

for V2.1 > V1, the difference was not significant.• Delayed TED post-test (far-transfer) scores also suggest the answer is “no”:

• There was a Version x Pre-test x Reading interaction:• V1 > V2.1 for high-reading, low-pre-test students• V1 > V2.1 for low-reading, medium-pre-test students• V1 = V2.1 for low-reading, low-pre-test students

• These results suggested a sequential focus on procedural before conceptual knowledge was not beneficial to the students we are most hoping to help.

Version 2.2 Focus and Implementation• Based on the comparisons between V2.1 and V1, we returned to the traditional Klahr lab approach

(Chen & Klahr, 1999) of focusing on conceptual knowledge (from which the procedural rules emerge). We retained, however, the discussion emphasis present since V1.

• To simulate our final Intelligent (knowledge-tracing) Tutor that tailors instruction to the individual student, we:

• Used human tutors to diagnose students’ initial knowledge state from their pre-test response patterns (i.e., which CVS rules and misconceptions students held). Members of the research team tutored students one-to-one or in small groups (where members shared a common diagnosis).

• Used a fully navigable V2.2 interface and a “database” of questions (practice and remedial) that corresponded to CVS rule(s). Tutors selected questions for individual students from this database based on their assessments of their students’ knowledge states.

Was Version 2.2 an improvement over Version 1?• Delayed TED post-test scores suggest the answer is “no.” (V1 = V2.2).

• No significant interactions between the Pre-test, Reading achievement scores and Version.• No differential effects of instruction based on reading, prior knowledge.

• The relationship between reading achievement and post-test was positive and highly significant (p < .001). (The relationship between pre-test and post-test was weaker, p = .09).

How can we help low-reading students and students with goal misconceptions?• Version 3.2 (in development) will address this question by incorporating explicit instruction on CVS

concepts prior to any other tutor interactions.• We adopted this in V3.2 because the heavy emphasis on discussion may have been confusing to low-

reading students.• To minimize elicitation of domain knowledge and in turn reduce goal misconceptions, two of the four

variables used in the explicit instruction have “mystery” values (i.e., nonsense names are used).• We will compare students’ CVS learning in V3.2 to analogous face-to-face learning outcomes (e.g., Klahr &

Nigam, 2004; Strand-Cary & Klahr (to appear)) as well as earlier versions of TED instruction.

Are computerized TED pre- and post-tests at least as good as paper/pencil tests (Version 3.1)?• Online assessment of student knowledge prior to instruction will allow automatic scoring and diagnoses of

student knowledge that will contribute to the tutor’s knowledge-tracing capabilities. • Computerized and paper/pencil test scores were not significantly different; thus, we are confident we can

incorporate the computerized tests into all future TED instruction. • Scoring trend was in favor of the computerized version. • All students who took the computerized test reported they preferred it to a paper version.

Instructional components of TED:• TED pre/post-tests (6 questions): evaluate and design experiments in three domains. • Ramps pre/post-test (4 questions): design experiments for different ramps variables.• Introduction: Purpose of lesson, term definitions, etc.• Explicit CVS instruction (e.g., Klahr & Chen, 1999).• Remedial CVS instruction (e.g., “step-by-step” walk-through of setting up informative

experiments) for students showing difficulty understanding explicit instruction.• CVS practice: Students apply their CVS knowledge to problems selected based on students’

knowledge states.

Refer to “TED Tutor Evolution” figure below for instructional components by TED version.

Year 1 Year 2 Year 3

• TED pre-test• Ramps pre-test• CVS instruction discussion of 3

ramps set-ups via:• Ask students if experiment

is good and why/why not.• Students generate and justify

ideas for improving the experiment.

• Ramps post-test• TED post-test

Version 1Teacher-delivered

whole-classroom discussion-basedinstruction (no computerization)

Inflexible(No differentiation)

Version 2.1Teacher-delivered

whole-classroom discussion-based instruction (computer-assisted)

Limited flexibility(Specified points for differentiation)

• TED pre-test• Introduction (e.g., definitions)• Conceptual/procedural (ramps, study)• CVS practice (many domains)

• Remedial: focus on individual “rules” of CVS (many domains)

• TED post-test

TED pre/post-test• “Design” questions and

“explain” prompts added to increase diagnostic info (below) .

• Server logging of student responses, including diagnostic information.

• Logging of user actions (during instruction and problem-solving) in place.

• Ramps pre-test,• Computerized explicit CVS

instruction• Ramps post-test.

• TED pre-test • Version 3.2 explicit instruction, followed by either:• “Step-by-step” instruction on designing good experiment if NO CVS mastery following explicit instruction, OR• Scaffolded CVS practice in problems (including problems similar to those on standardized achievement tests) if CVS mastery.• TED post-test

Future work (Year 3 goals):• Develop and test Wizard of Oz tutor (Version 3.3):

• The purpose of V3.3 is to test TED architecture, and provide further information about the pedagogical actions to take given the student’s current knowledge state.

• It will be a full TED tutor with computerized pre- and post-tests, introduction, CVS instruction, remediation, and problem-solving. Artificial intelligence (AI) component will not be present.

• Through a computer interface, a human tutor (acting as the AI component) will observe a student as he works through TED instruction. Based on the student’s current actions and knowledge state (visible to the tutor), the computer and human tutor will jointly determine instruction.

• Develop final intelligent TED tutor (Version 4) which will be identical in format to final Wizard of Oz versions, but will have the AI component in place of the human tutor (refer below for architecture).

Student UIStudent Student UIStudent

Input Stream

Automation Stream

StudentUI

Teacher UI

Server

Automation Stream

Student

LogLog Stream

SAI Events

State Events

Teacher

AutomationCommands

Ramps Discussion

Holistic 1Holistic Qj

Error1 Get it

Holistic Qj + 1

Error1

Alternate Remediation

Holistic Qn

Step by Step

Don’t Get

Holistic 1Scaffolded

Holistic Qn

Step repetition possible

AlternateRemediation(if necessary)

......

Fading Scaffolding

Sockets

Knowledge Tracing Knowledge

Tracing

Tutor Logic Overview

Data Streams Overview Server Architecture

Knowledge Tracing (per user)

User Threads

Teacher Audit mux

Stream

TutorState

Production System

User ThreadsUser

Threads

Logging

Playback Engine

Encryption

NTP

Version 3.2Computer-delivered “explicit

instruction”

Version 3.3Wizard of Oz (computer-delivered,

supplemented by human)

Flexible(Multiple branching paths)

Version 3.1Computerized TED

pre/post-tests.

• TED pre-test• Sequential CVS instruction via:

• Introduction• Procedural rules (Study habits),• Conceptual (Ramps figures and tables)

• Ramps post-test• TED post-test

Version 4Brief teacher-delivered

instructionIndividual computer use

Ongoing teacher facilitation

Adaptive / “Intelligent”(“Web” of branches)