Tracing behaviors associated with motivational states and learning outcomes when students learn with...
-
Upload
thomasine-little -
Category
Documents
-
view
217 -
download
0
Transcript of Tracing behaviors associated with motivational states and learning outcomes when students learn with...
Tracing behaviors associated with motivational states and learning outcomes
when students learn with the Cognitive Tutor
Team: Matthew Bernacki & Pranav Garg
Mentors: Erik Zawadzki & Ryan Baker
20
11
Sum
mer
Sch
ool
Overview
• We investigated relationships between motivation, learning behaviors and learning outcomes amongst high school students learning geometry using the Cognitive Tutor.
• We identified a series of 3 sequential behaviors (triplets) and plotted their frequency across the logs of 38 learners in one geometry unit.
• We conducted a factor analysis to reduce 147 triplets into 28 factors and examined their correlation with self-reports of affective state, self-efficacy for the unit and their achievement goals for mathematics.
METHOD: In the Classroom
Participants• 38 high school geometry students completing Unit 13 in the
Cognitive Tutor, which was a standard component of the their rural high school’s geometry curriculum.
Instruments• Cognitive Tutor for Geometry
– Unit 13: Circumference and Area of Circles
• Achievement Goal Questionnaire-Revised– Elliot & Murayama, 2008 (9 items, 3 per Mastery Approach, Performance
Approach, Performance Avoidance subscale)
• Academic Self Efficacy Survey– Midgely, et al., 2000; Patterns of Adaptive Learning Survey
• Affect (single items constructed for this project)– Boredom, Confusion, Frustration Engaged Concentration,
Positive Experience
METHOD: Data Mining Procedure
1. Exported transaction level log file from Cognitive Tutor2. Selected only those students who completed the Unit of interest;
cleaned data to remove any students who were missing self-report data or a complete log file
3. Calculated the duration (seconds) to complete each learner action in the OUTCOME column
– OK – answered problem step correctly– BUG – incorrectly answered the problem step (common error)– ERROR – incorrectly answered the problem step– HINT [1,2,3]– requested a hint – SWITCH – switched their window to consult a worked example
4. Recoded Duration by Quartile (1, middle 2&3, 4)1. Q1 = Short durations, typically 1-2 seconds; coded as “…_1”2. Q2&3 = Medium Durations, typically 2-10 seconds; coded as “…_2”3. Q4 = Longest durations, typically upward of 10 seconds; coded as “…_3”
5. Concatenated Outcome with Q(uartile version of) Duration.
METHOD: Data Mining Cont’d.
6. Ran a script in Python to move a sliding window over the OutcomeQDuration column and populated a column with a triplet: [FIRST TRANSACTION_DURATION_SECOND_D_THIRD_D].
7. Calculated the total number of unique triplets (n = 7,885) and, with a Pivot Table, determined the frequency each occurred per student.
8. Eliminated those that occurred less than 5 times and those that occurred in less than 2 students (n = 147)
9. Imported into SPSS, merged with a file of their self-reported motivational states and official record of learning outcomes
10. Ran a Principle Components Factor Analysis (unrotated) to determine a factor structure.
11. Correlated Factor Scores with motivation and performance data
RESULTSTHOSE WHO …TEND TO CONDUCT
BEHAVIORS THAT LOAD ON FACTOR
…EXPERIENCE …
Boredom 15, 28
Confusion 27
Frustration 12, 13, 27
Self-Efficacy 6,22
…PURSUE…
Mastery or Performance Approach Goals
27
Performance Avoidance Goals 17
… PERFORM WELL ACCORDING TO…
4th Quarter Grades 3
Academic Self Efficacy
Boredom Confusion Frustration Positive Experience
Engaged Concentration
Mastery Approach Goals
Performance Approach Goals
Performance Avoidance Goals
3rd Qtr Grade
4th Qtr Grade
FACTOR 1 -.004 .105 -.025 .083 .063 .069 .020 .079 -.114 .099 .032
2 .022 -.045 -.124 -.050 -.238 -.244 -.096 -.258 -.193 .117 .086
3 .220 .090 -.005 .006 .245 .222 .164 .110 -.135 .171 .321*
4 -.088 .125 .077 .082 .078 .127 -.065 .066 .152 -.023 -.166
5 .002 .117 .048 .073 .065 .132 -.108 -.090 -.147 .022 -.001
6 .378* -.086 -.152 -.097 .284 .279 -.126 -.245 -.158 .254 .315
7 .015 -.079 -.277 -.103 .006 -.063 -.108 .017 -.065 .106 .051
8 -.067 .196 .133 .177 -.148 -.165 -.210 -.174 -.156 .116 -.028
9 .054 .095 -.021 .026 -.148 -.116 -.133 .022 .111 .079 .149
10 -.256 -.317 -.007 -.086 .036 .032 .060 .213 .100 -.101 -.148
11 -.133 -.051 .224 .120 .032 .069 .009 -.037 .099 .068 .237
12 -.005 -.136 -.194 -.452** -.179 -.178 -.108 .202 .030 .212 .118
13 .113 -.203 .369* .398* .262 .159 -.003 -.187 -.165 -.249 -.197
14 -.057 -.199 -.316 -.131 .090 .002 -.267 -.056 -.011 -.083 -.099
15 -.146 .354* .119 .194 -.091 -.051 .219 -.053 -.094 -.168 -.073
16 .138 -.039 .189 .228 .086 .109 -.098 -.108 .158 -.030 -.133
17 .147 .076 .189 .111 .261 .204 .044 .045 .373* .194 .316
18 .084 -.034 -.035 -.057 .016 -.027 -.244 -.004 -.101 -.109 -.246
19 .145 -.056 .165 .095 .102 .070 -.092 -.120 .001 .152 .005
20 .098 -.052 -.153 -.178 -.275 -.217 .125 .064 .127 .162 .049
21 -.243 -.111 .107 .160 .002 .034 .141 -.041 .159 .213 .217
22 -.329* -.072 -.087 -.063 -.061 -.063 .161 -.240 -.160 -.114 -.109
23 .043 .178 -.212 -.191 -.216 -.054 -.134 .030 .100 .042 .059
24 -.221 .009 -.002 .150 -.261 -.264 .099 .029 -.111 .212 -.070
25 -.050 -.013 .040 -.021 -.104 -.195 -.101 -.068 .151 .068 .008
26 -.131 -.226 .126 .124 .030 .119 -.298 .281 .139 .008 .05627 .063 -.195 -.417** -.389* .020 -.042 -.599** -.519** -.092 .173 .179
28 .015 .456** .016 .048 -.232 -.193 .042 -.079 .115 .006 -.247
THE FACTORSfactor
Behavior Triplet with highest factor loading
2nd MAX 3rd MAX
1['ERROR_3_ERROR_3_OK_3{.858} , ', 'ERROR_2_OK_3_BUG_2{.858} , ', 'OK_2_BUG_1_OK_2{.858} , ']
['OK_3_ERROR_2_ERROR_2{.845} , ']
['ERROR_2_ERROR_1_ERROR_1{.835} , ', 'ERROR_1_ERROR_1_ERROR_1{.835} , ', 'ERROR_2_ERROR_2_ERROR_1{.835} , ']
2 ['OK_2_OK_3_OK_3{.825} , '] ['OK_3_OK_3_OK_2{.758} , '] ['OK_1_OK_3_OK_2{.746} , ']
3 ['OK_3_ERROR_3_OK_3{.623} , '] ['hint_2_HINT2_1_hint_3{.542} , ', 'OK_3_BUG_2_OK_2{.542} , ']
['OK_3_hint_3_HINT2_2{.520} , ']
4 ['OK_3_OK_3_BUG_3{.738} , '] ['OK_1_OK_2_OK_2{.616} , '] ['OK_1_OK_1_OK_2{.584} , ']
5 ['ERROR_2_HINT_2{.647} , ', 'ERROR_3_OK_3_BUG_3{.647} , ']
['ERROR_3_OK_2_OK_2{.600} , '] ['OK_2_OK_3_BUG_2{.591} , ']
6 ['OK_3_ERROR_3_ERROR_3{.540} , ']
['OK_3_OK_1_OK_3{.470} , '] ['OK_3_OK_3_ERROR_3{.467} , ']
7 ['OK_1_OK_1_BUG_3{.599} , '] ['HINT3_2_ERROR_3_ERROR_2{.582} , ']
['OK_2_OK_3_BUG_1{.551} , ']
8 ['BUG_2_OK_2_OK_2{.612} , '] ['OK_2_OK_2_BUG_2{.480} , '] ['OK_1_OK_2_BUG_1{.453} , ']
9 ['OK_1_OK_1_BUG_1{.603} , '] ['OK_2_BUG_2_OK_2{.500} , '] ['OK_3_OK_3_OK_1{.483} , ']
10 ['BUG_3_OK_3_OK_1{.465} , '] ['OK_2_BUG_1_OK_3{.457} , '] ['HINT3_3_OK_3_OK_2{.448} , ']
11 ['HINT_2 _HINT_1_HINT2_1{.560} , ']
['OK_3_ERROR_3_ERROR_3{.437} , ']
['ERROR_2_OK_3_ERROR_2{.433} , ']
12 ['BUG_1_OK_3_OK_2{.490} , '] ['BUG_3_OK_2_OK_3{.477} , '] ['ERROR_3_OK_3_OK_3{.382} , ']
13 ['OK_3_BUG_1_OK_2{.518} , '] ['HINT3_2_ERROR_3_ERROR_2{.476} , ']
['OK_1_OK_3_BUG_1{.430} , ']
14 ['OK_2_OK_3_ERROR_3{.437} , '] ['OK_3_ERROR_2_OK_2{.387} , '] ['BUG_3_OK_2_OK_2{.349} , ']
THE FACTORSfactor
Behavior Triplet with highest factor loading
2nd MAX 3rd MAX
15 ['HINT2_1_hint_3_HINT2_2{.575} , ']
['hint_3_HINT2_1_HINT3_3{.451} ']
['BUG_3_OK_3_OK_3{.401} , ']
16 ['ERROR_3_OK_3_OK_2{.463} , '] ['OK_3_ERROR_2_OK_2{.424} , '] ['OK_3_ERROR_2_ERROR_3{.372} , ']
17 ['OK_2_OK_3_hint_3{.501} , '] ['OK_2_ERROR_2_OK_3{.459} , '] ['OK_2_OK_2_BUG_2{.374} , ']
18 ['BUG_3_OK_3_OK_1{.396} , '] ['OK_1_ERROR_3_OK_3{.356} , '] ['HINT3_2_ERROR_3_ERROR_2{.327} , ']
19 ['OK_3_OK_1_OK_1{.377} , '] ['BUG_3_OK_2_OK_3{.350} , '] ['OK_2_OK_3_OK_2{.337} , ']
20 ['BUG_1_OK_3_OK_2{.373} , '] ['ERROR_3_OK_2_OK_3{.372} , '] ['OK_3_OK_2_OK_1{.319} , ']
21 ['OK_3_BUG_1_OK_3{.382} , '] ['BUG_3_OK_2_OK_3{.312} , '] ['BUG_1_OK_3_OK_2{-.395} , ']
22 ['ERROR_2_ERROR_1_OK_3{.337} , ']
['OK_3_BUG_1_OK_2{.330} , '] ['OK_1_OK_1_ERROR_2{.328} , ']
23 ['OK_1_BUG_3_OK_2{.488} , '] ['ERROR_3_OK_3_BUG_2{.430} , ']
['OK_2_OK_3_ERROR_3{.331} , ']
24 ['OK_2_OK_1_OK_3{.420} , '] ['OK_2_OK_3_BUG_3{.306} , '] ['HINT2_1_HINT3_3_OK_3{-.343} , ']
25 NULL* NULL NULL
26 NULL* NULL NULL
27 ['OK_1_OK_3_OK_1{.306} , '] NULL NULL
28 NULL* NULL NULL
THEORETICAL CONCLUSIONS
• Some behaviors associated with a factor can be interpretted somewhat easily. Factor 12:
• BUG_1_OK_3_OK_2{.490}• BUG_3_OK_2_OK_3{.477} • ERROR_3_OK_3_OK_3{.382}
– Student made errors, often after some perserveration, then correctly answered items after a medium to long period.
– Factor is associated with low frustration.
THEORETICAL CONCLUSIONS
• However, scores on one factor (#27) were significantly associated with self-reports of confusion and frustration and negatively associated with mastery and performance approach goals.– Only one triplet [OK_1_OK_3_OK_1] loaded
higher than .30 on the factor– Low factor loading and meaningfulness of a
short period prior to a correct, followed by a long and a short actually run counter to some conclusions based on self-reports.
METHODOLOGICAL CONCLUSIONS
• Triplets composed of behaviors and their durations can be meaningful measures of behavior
• They can also be insufficiently descriptive of a students’ behavior re: specificity – number of behaviors captured– Precision of duration when collapsed to
quartile– Meaningfulness of cuts between quartiles
NEXT STEPS
• Test factor structure across additional units– If not, may make sense to abandon factors and examine
relations one behavioral trace at a time
• Generate 4-lets and 5-lets to see if these behaviors provide more intuitive glimpses of student behaviors
• Once a set of behaviors has been found that associate with motivation – develop a flag for a behavior… and an intervention?
• Test structural models with paths from motivational state to behavior to learning outcomes