Effective Help-seeking Behavior Among Students Using an Intelligent Tutoring System

for Math: a Cross-Cultural ComparisonJose Carlo A. Soriano

Break down

Effective Help-seeking Behavior Among Students

Using an Intelligent Tutoring System for Math:

A Cross-Cultural Comparison

Intelligent Tutoring SystemsSeeks to simulate the effectiveness of a good

personal human tutorIndividual tutoring is more effective than

classroom instruction by 2 standard deviations (Bloom, 1984) Knows what specific skills the student is having trouble

with Able to offer appropriate help at an appropriate time

Objective: help the student learnITS are better by 1 standard deviation (Koedinger

et al, 1998; Corbett et al., 2001)

Intelligent Tutoring Systems

Help-Seeking in ITSThe kind of help, and how the help is offered,

affects learning (Aleven et al, 2003)Students who use High-level help most

frequently have the least learning(Matthews and Mitrovic, 2008)

Help-seeking is a Meta-cognitive skillMeta-cognition is “cognition about cognition”

One’s knowledge of the processes in play One’s active control of it during learning

Help-seeking Behavior1. Both the EU and UNESCO declared:

developing metacognitive skills, or ‘teaching students how to learn’ should be among the highest educational priorities (Louizidu and Kotselini, 2007)

2. Help-seeking behavior is an achievement-related behavior (Karabenick and Knapp, 1991)

3. Higher-achieving students were more likely to ask for help when encountering personal difficulties (Taplin et al, 2001)

Effective Help-SeekingStudent is more likely to learn when:

Student seeks for help when encountering personal difficulties Student knows what kind of help is needed such

that student can work effectively on his/her own Student knows how to ask for help

Student is not dependent on helpStudent spends time understanding help given

Help-seeking in ITSHowever, students generally do not know

when they need help (Aleven and Koedinger, 2000)

Students “game the system” (Baker et al)

Meta-cognitive tutors have been developed by Aleven et al

“Scooter the tutor” developed by Baker et alTo teach students “how to learn”

The Problem

Is “effective help-seeking” the same across cultures?Very few cross-cultural comparisons

Comparing ITS use between USA and Latin American students(Ogan et al, in press)

Comparing Disengaged behavior between USA and Filipino students (Rodrigo et al, 2010)

Implications on Meta-cognitive tutors

Method

Find out if effective help-seeking behavior is the transferrable across cultures, or are significantly differentMight encourage more cross-cultural

comparisonsImplications on future efforts on meta-

cognitive tutoring

Scatterplot tutor

Data

Costa Rica

Mexico

USA

Philippines

Feature Engineering1. Helpavoidance2. Nothelpavoidance3. Helpnonuse4. Unneededhelp5. BugmsgLongpause6. BugmsgShortpause7. HintmsgLongpause8. HintmsgShortpause

Feature Engineering9. HintmsgLongpauseCorrect10. HintmsgShortpauseCorrect11. NothelpavoidanceShortpause12. NothelpavoidanceLongpause13. UnneededhelpShortpause14. UnneededhelpLongpause15. ShortpauseHintmsg16. LongpauseHintmsg17. FirstattemptHintmsg

Feature OptimizationMost features require a threshold, either p-

know or a time threshold

Brute-force grid search:For p-know thresholds, grid-size is 0.05For pause thresholds, grid-size is 0.5 secondsSingle-parameter linear regression for each

threshold for each feature in grid

Feature SelectionCross-validated r was used as the goodness

criterionCorrelation between the predicted learning

values and the actual learning value

The threshold with the best cross-validated r becomes the threshold for each feature

As an additional control against over-fitting, features whose best threshold had negative cross-validated r is dropped from model creation

Model Creation and Evaluation

Models were created using Forward Selection

Models were evaluated by applying each country’s model to each country’s data set

A model were created after combining the four data sets

Brute-Force Grid SearchFeature cut-off r Feature cut-off RCR PH

Helpavoidance 0.15 0.081 Nothelpavoidance 0.4 0.087Helpnonuse 0.15 0.012 Helpnonuse 0.95 0.043

Unneededhelp 1 0.006 NothelpavoidanceShortpause 1 0.108BugmsgLongpause 25.5 0.06 NothelpavoidanceLongpause 0 0.075HintmsgLongpause 47.5 0.054 UnneededhelpShortpause 0.5 0.061HintmsgShortpause 0.5 0.017 US

HintmsgLongpauseCorrect 41.5 0.294 Helpavoidance 0.25 0.122NothelpavoidanceLongpause 45.5 0.284 Helpnonuse 1 0.014

UnneededhelpShortpause 13 0.019 Unneededhelp 1 0.117UnneededhelpLongpause 0 0.008 BugmsgLongpause 57 0.131

MX BugmsgShortpause 0.5 0.026Helpnonuse 1 0.201 HintmsgLongpause 0.5 0.003

BugmsgShortpause 2.5 0.044 HintmsgShortpause 6.5 0.02HintmsgShortpauseCorrect 0.5 0.025 HintmsgLongpauseCorrect 1 0.039

NothelpavoidanceLongpause 58.5 0.018 HintmsgShortpauseCorrect 4 0.073ALL NothelpavoidanceShortpause 0.5 0.265

Helpavoidance 0.05 0.023 NothelpavoidanceLongpause 0 0.096Nothelpavoidance 0.4 0.024 UnneededhelpShortpause 12 0.203

Helpnonuse 0 0.094 UnneededhelpLongpause 0 0.123BugmsgShortpause 2.5 0.071 ShortpauseHintmsg 32.5 0.026

HintmsgLongpauseCorrect 1 0.022 LongpauseHintmsg 37.5 0.04HintmsgShortpauseCorrect 3 0.062

NothelpavoidanceShortpause 20 0.027NothelpavoidanceLongpause 1 0.052

UnneededhelpShortpause 35 0.04UnneededhelpLongpause 1 0.062

AnalysisFirstattemptHintmsg – only feature that was not

able to pass in any countryDifferent nature of skills

Some can be hard at the very start, some can be easily understood from the start

Most skills may require a large number of actions

NothelpavoidanceLongpause – had positive cross-validated correlation in all five data setsIn contrast to theory, this feature has negative

directionality for most countries This might be because students who exhibit the behavior

simply do not understand the skill

Forward SelectionCountry Learning = Cross-validated r

CR

0.132 * Helpavoidance(0.15)+ 7.385 * HintmsgLongpause(47.5)

- 9.096 * HintmsgLongpauseCorrect(41.5)- 21.847 * NothelpavoidanceLongpause(0.25, 45.5)

+ 53.010

0.462

MX

- 0.147 * Helpnonuse(1) - 0.754 * BugmsgShortpause(2.5)

+ 1.187 * HintmsgShortpauseCorrect(0.5)+ 40.652

0.229

PH 0.021 * Helpnonuse (0.95)

- 0.763 * NothelpavoidanceShortpause(0.4, 1)+ 32.423

0.126

US

- 1.021 * Nothelpavoidance(0.25) - 2.870 * BugmsgLongpause(57)

- 6.680 * NothelpavoidanceShortpause(0.25, 19.5) + 5.605 *LongpauseHintmsg(37.5)

+ 12.086

0.350

ALL

0.036 * Helpavoidance(0.05) + 0.082 * Helpnonuse(0)

- 0.491 * BugmsgShortpause(2.5) - 46.861

0. 153

AnalysisHelpavoidance – negative directionality for

CR and PHreinforcces Aleven et al’s findings that avoiding

help is negatively correlated to learning (Aleven et al., 2006)

ButmsgShortpause and NothelpavoidanceShortpause also has negative directionality for MX and US, and PH and US.Possibly effect of not spending enough time to

understand bug or hint message provided

Cross-Cultural Evaluation

Country CR MX PH US ALL

CR 0.534 -0.285 0.051 0.151 0.08

MX 0.04 0.392 -0.086 -0.009 0.088

PH 0.004 -0.174 0.203 0.146 0.038

US -0.085 -0.164 0.228 0.476 0.057

All -0.032 0.165 0.047 0.142 0.215

Rows are models, columns are data sets applied to

AnalysisEighteen out of 25 model applications

produced positive correlation between model’s predicted learning and the actual learningNegative correlation means model did worse

than chance at predicting learningLOOCV r values are very high compared to r

when applied to other countriesReinforces hypothesis that help-seeking might

not transfer across countries

AnalysisMX and US – performance on each other’s

data sets are low (-0.009 and -0.164)This means that our model of effective help-

seeking is not effective when applied to the other country

Reinforces findings in (Ogan et al, in press) which compares differences in behavior of USA and Mexico students Collaborative nature of students from Mexico may

be the reason why help-seeking is different The help they ask from the tutor will only be help

that they do not get from other students

AnalysisCR and MX – did not perform well on each other’s data

sets (-0.285, 0.04)Though collaborative tendencies might be common

between Costa Rica and Mexico, help-seeking behavior with the ITS may differ

US and PH – performed very well on each other’s data set (0.146 and 0.228)Reinforces findings in (San Pedro, 2011) wherein a

carelessness detector is generalizable between the two countries

In contrast to (Rodrigo, 2010) which shows disengaged behavior is different between the two countries But Help-seeking and Disengaged behavior are two different sets

of behaviors

ConclusionResults did not expose that effective help-

seeking as a whole is very culture-specific (18 out of 25 applications returned positive r)

However, it is not apparent that effective help-seeking is transferrable across countriesBig difference between LOOCV r and cross-

cultural evaluations

Conclusion and RecommendationsThere are pairs of countries wherein effective help-

seeking fail to generalize to each other’s data set.Meaning effective help-seeking from one country may

not necessarily be effective in another

Single effective help-seeking models used by meta-cognitive tutors may be effective in one culture but not in othersFuture meta-cognitive tutors might have to use a

more generalizable modelMay have to switch models, depending on the culture

where the ITS is used