Artificial Intelligence as an Effective Classroom Assistant Benedict du BoulayUniversity of SussexUK

Bloom (1984) and the 2-sigma challenge

2-sigma

Contents1. What is an AIED system2. An example: not just about cognition3. A range of meta-reviews since 2011

What is an AIED system?

AIED systems: Adaptivity• Reason about Pedagogy

• What task to set next• How best to react to the leaner’s cognitive, affective and

motivational state• Reason about the domain of instruction

• Follow learner’s problem-solving step by step• Reason about how much is being learned

• Focused feedback, help and support • Adjust learning tasks dynamically

• Use interface to model domain• Both system and learner can reason in the interface

AIED (ITS) system: 4 components

Pedagogical model

Student model

Interface

Domain model

Betty’s Brain (Leelawong & Biswas, 2008)

Johnson (2010) Tactical Iraqi

Recent META-REVIEWSComparing AIED/ITSs to whole class teaching and one-to-one tutoring

VanLehn (2011)Various ITSs

No. of Studies

Effects And . . .

88 “just as effective as adult, one-to-one tutoring for increasing gains in STEM topics”

“ITSs should be used to replace homework, seatwork and perhaps other activities but not to replace a whole classroom experience”

Ma, Adesope, Nesbit, Liu (2014)Various ITSs

No. of Studies

Effects And . . .

107 Good effect sizes against teacher-led, large group instruction;textbooks;non-ITSs.

No difference against individualised human tutoring or small group instruction

Steenbergen-Hu & Cooper (2013)Various College level ITSs

No. of Studies

Effects But . . .

39 “Have demonstrated their ability to outperform many instructional methods or learning activities in facilitating college-level students’ learning of a wide range of subjects . . .

. . . although they are not as effective as human tutors”

Steenbergen-Hu & Cooper (2013)ITSs K-12 Maths

No. of Studies

Effects But . . .

34 Very small positive effect sizes < 0.1

Better for ordinary pupils than low achievers;Better if intervention lasted < 1 year

Kulik & Fletcher (2015)Various ITSs

No. of Studies

Effects But . . .

50 “ITSs can be very effective instructional tools”

Degree of improvement depended on whether standardised or local tests used

Nesbit, Adesope, Liu (2014)Computer Science ITSsNo. of Studies

Effects But . . .

22 “A significant advantage of ITS over teacher-led classroom instruction and over computer-based instruction”

No system combined procedural with conceptual tutoring

Cognitive Algebra Tutor: Koedinger et al.

Pane, McCaffrey & Karam (2014)Cognitive Tutor Algebra

No. of Schools

Comparison Effects But . . .

73 high 74 middle

Across 7 USA states

Post-tests using traditional teaching

First year not significant.Second year high schools significant. Effect size 0.20

No direct monitoring of how the system actually used

-0.19

EffectSize

0.47

Conclusions• Described AIED systems• Offered an example concerning Motivation and Metacognition with partially successful evaluation

• Summarised recent meta-reviews• Generally positive about ITSs vs most forms of

traditional teaching for STEM subjects• including vs CAI systems

• Blended learning model favoured• But degree of overlap in coverage between meta-

analyses not checked

An Example: not just about cognition

Alison Hull - SQL-Tutor* - 2014• Based on SQL-Tutor (Mitrovic)• +Metacognitive & Motivational help and feedback• Invites learner to assess his or her confidence • Tracks self-efficacy and its accuracy• Provides metacognitive and motivational feedback at start/end of session and at end of problem

• Refers back to similar previous situations

Accuracy of self-efficacy

Performance

High Mid LowPessimistic “Even though you have

not found the problems easy, you really have made good progress in successfully completing problems . . .”

. . . Effort Poor: “Think about focussing on problems with complexity rating of 1 or 2 at the beginning of the next session to firmly establish your core knowledge before expanding on it . . .”Effort Good: . . .

Realistic . . . . . . . . .

Optimistic “You have made really good progress this session, but have the problems challenged you enough? Have a think whether you are ready to try more difficult problems next time . . .”

. . . Effort Poor: “You have not made much progress this session, but that doesn't mean the next session has to be the same . . .”Effort Good: . . .

Accuracy of self-efficacy

Performance

MiddleRealistic You have made steady progress this session

Your performance matched your confidence this session. How confident do you feel about answering SQL queries? Please indicate below.

Evaluation• Northumbria University: one semester.• Basic SQL-Tutor vs enhanced SQL-Tutor* (n=39) [+ vs no ITS ]

• Log files, assessment scores, questionnaire.

Results1. Does providing M&M feedback

lead to any measurable learning gains?

1. Does providing M&M feedback lead to any measurable gains in learner focus?

2. Do the learners perceive any benefit from using an ITS to aid their learning?

3. What are the effects of including M&M feedback based on the learner’s past states and experiences?

11 Yes study group move slightly up class rankings; both groups solved same number of problems, but Control group took more attempts. Both groups better than no Tutor

11 Control group bit more persistent on unsolved problems study group made bit better use of the system

11 Yes for both groups, study group slightly more positive

11 Some positive trends study group students typically not follow up references to past similar situations