Evaluation of a Hybrid Self-improving Instructional Planner

Jon A. Elorriaga and Isabel Fernández-CastroComputer Languages and Systems Dept.

University of the Basque Country 649 Postakutxa, E-20080 Donostia.

e-mail: jipelarj@si.ehu.es 

Contents

• Introduction• Our approach: HSIIP

– Case-based Instructional Planner

– How it works

– Methodology of application

• Evaluation• Conclusions• Related and Future Work

Introduction

• Self-improving Vs. Adaptive ITSs– Self-improving ITSs: generalise the acquired knowledge and use it

in future instructional sessions (Dillenbourg, 1989)

• Few self-improving systems

• ITSs can learn in each of its modules – Tutor Module

– Student Model

– Domain Module

– Interface

• By using data– From the Student Model (ITS)

– Directly from the student

– From the teacher

Our Approach: HSIIP

HSIIP: Hybrid Self-Improving Instructional Planner• Objectives:

– To improve the adaptation ability

– To incorporate a learning ability to existing ITSs

• Focus: Tutor Module, Instructional Planning• Learning Techniques:

– Case-Based Reasoning

– Learning from memorization

– Statistical learning

The HSIIP Approach

Proposal: To incorporate a CBIP into existing ITSs

Aim: To enhance the ITS with learning ability

Result: SIITS

ITS+CBIP ===> SIITS

Case-Based Reasoning

Problem

REUSERETAIN

RETRIEVEFeatures

of the problem

Retrieved Cases

REVISE

Proposed Solution

New Case

Case Memory

(CM)

Case-Based Instructional Planner

InstructionalPlan Memory

(IPM)

CBIPGenerationComponent

LearningComponent

AssessmentComponent

CBIP: Case Structure

Application Context

<sequence of student related features>

<sequence of session related features>

<sequence of domain related features>

Instructional Plan

<sequence of plan items>

Results of the application

<sequence of result values>

CBIP: Detail of Instructional Plan Memory

I2

att = know-level (T2)(high GE2)

N1

(student-type novice)(learning-style learning-by-doing)

GE1

norm = N1indices = (I1,I2)

GE2

norm = N2indices = (...)

I1

att = know-level (T1)(low C1) (high C2)

GE1, GE2 : Generalised EpisodesN1, N2: NormsI1,I2: IndicesC1, C2: CasesT1, T2: Topics of a domain

N2

(student-type novice)(learning-style learning-by-doing)(know-level (T2) high)

C1

Application Context(student-type novice)(learning-style learning-by-doing)(know-level (T1) low)...Plan ...Results of App ---

C2

Application Context(student-type novice)(learning-style learning-by-doing)(know-level (T1) high)...Plan ...Results of App ---

Search• Hierarchical Organisation

of the IPM

• Exhaustive and Heuristic Search

• Heuristic function: Similarity function

• Similarity Threshold

CBIP: Generation Component (1)

Retrieval of CasesMatching

• Nearest neighbour matching

• Similarity function

SF (C1, C2)= w i sim (fi

1,fi2)

i=1

n

w ii=1

n

Adaptation of cases• Critic-Based Adaptation• Production System• Knowledge intensive task

CBIP: Generation Component (2)

CRITICIF <condition>+THEN <adaptation action>+Priority: <Integer>Adaptation-degree: (0 .. 1)

CBIP: Learning Component (1)

Revision of Cases• Evaluation items• Trace of the learning session• Two Dimensions:

– Educational• Beliefs of the ITS (Student Model)

• Beliefs of the Learner: Interaction

– Computational• Case Reuse level

CBIP: Learning Component (2)

Revision of Cases• Evaluation items:

– Knowledge Acquisition Levels

– Misconceptions

– Student Beliefs about KAL

– Student Beliefs about the session

– Replanning

– ...

• Result Object– Elementary results

– Collective results

• Normalisation of values (0 .. 1)

• Statistical Learning• Creation of new cases

CBIP: Learning Component (3)

Storage of new Cases• Find the appropriate GE

– Adapted Search Algorithms

• Generalisation– On-line

– Off-line

– Thresholds

CBIP: Heuristic Assessment Component

• Heuristic Formulae• Adaptable - Weights • Assessed objects

– Retrieved cases

– Built Instructional Plan

– Global result of the executed Instructional Plan

• Assessment Factors– Beliefs of the STI

(SM)

– Beliefs of the learner

– Similarity

– Reuse rating

– Adaptation level

– Influence level

Assesses some Instructional Planning Objects

TUTOR MODULE

InstructionalPlan

Memory DIDACTICDISPATCHER

LearningComponent

GenerationComponent

ClassicalInstructional

Planner

DIDACTICINSTRUCTOR

AssessmentComponent

Session Data

Estimates

Cases

InstructionalPlan

Estimates

Cas

es

Estimate > Threshold

SHIIP Working (1)

GenerationComponent

ClassicalInstructional

Planner

DIDACTICINSTRUCTOR

TUTOR MODULE

Session Data

Estimates

Cases

NIL

Estimates

Cas

es

Instructional

Plan

Estimate < Threshold

LearningComponent

AssessmentComponent

InstructionalPlan

Memory DIDACTICDISPATCHER

SHIIP Working (2)

HSIIP

• Working– Initially empty IPM

– Training Phase

– Co-operation Phase

• Kernel of CBIP– An object-oriented framework that facilitates the development of

Case-Based Instructional Planners

– Represent explicitly and separately the characteristics of the concrete ITS

• Methodology of Application– Procedure and Guidelines

HSIIP: Methodology of Application (1)

• Analysis of the ITS (levels, plan-items, attributes)** Most important task** Knowledge Engineering

• Adaptation of the framework

– Representing the ITS related knowledge

– Setting of the parameters: Thresholds and Weights

– Construction of the adaptation module

• Integration of the CBIP

• Test

HSIIP: Evaluation (1)

• Objective: Evaluate the performance of the HSIIP in terms of the changes in the student’s knowledge

• Design of the experiments:– Four classical instructional planners (CIP)

– Their corresponding HSIIP

– A Population of simulated students (4 groups)• To test isolated modules

• To perform a significative number of experiments in the same conditions

HSIIP: Evaluation (2)

Performance HSIIP-CIP

0

0,2

0,4

0,6

0,8

1

1 2 3 4 5 6 7 8 9 10 11

Stages

Performance

CIP-1

HSIIP-1

CIP-2

HSIIP-2

CIP-3

HSIIP-3

CIP-4

HSIIP-4

HSIIP: Evaluation (3)

 

Use of CIP-2 and CBIP-2

0

1000

2000

3000

4000

5000

1 2 3 4 5 6 7 8 9 10 11

Stages

Use

Total No. Of plans

Plans from CIP-2

Plans from CBIP-2

Result of the HSIIP-2

0

0,2

0,4

0,6

0,8

1

1 2 3 4 5 6 7 8 9 10 11

Stages

Result Performance

Conclusions

• HSIIP: An hybrid approach to enhance ITSs with learning capabilities based on a Case-Based Instructional Planner.– Case-Based Learning– Statistical Learning– Learning from Memorization

• Sound performance (combines two planners).• CBIP kernel: A framework for developing Case-Based

Instructional Planners.– Generic Module for Instructional Planning– Adaptable

• Positive evaluation results• Simulated student a useful tool for formative evaluation

Related and Further Work

• Related Work– Tool for interacting with the teacher (supervision of

plans and results)

– Data: System, Student, Teacher

• Further Work– Experimentation

– More aspects taken into account in revision

– Application to other planning problems

Result Object

Object Result (is-a Case-component Result-object)result-history: <list of Elementary-result instances>result-average: <Collective-result instance> Object Elementary-result (is-a Result-object)evaluation-item-list: <list of Evaluation-item instances>learner: <string>date: <date>learner-estimate: [0..1]global-estimate: [0..1] Object Collective-result (is-a Result-object)direct-use-number: <integer>use-number: <integer>evaluation-item-list: <list of Evaluation-item instances>learner-estimate: [0..1]global-estimate: [0..1] Object Evaluation-item (is-a Result-object)evaluation-attribute: <Attribute instance>final-value: [0..1]change: [0..1]learner-estimate: [0..1]

Elementary Result of a Subplan (ERS)

)()()( SWOBSAEISSERS L

NEI

jEI

NEI

jEI

LT

jLjT

j

j

W

WWW

SEIWBSEIWB

SAEIS

1

1

))(())((

)(

NA

SERSSARS

NA

ii

1

)()(

ERS = Elementary Result of a Subplan ARS = Average Result of a Subplan WBT = Weighted Belief of the Tutor (applied to each individual

feature) WBL = Weighted Belief of the Learner (applied to each individual

feature)WOBL= Weighted Overall Belief of the Learner (applied to a

subplan)AEIS= Average of the Evaluation Items of a SubplanNEI = Number of Evaluation Items NA = Number of Applications of the caseEIj = Evaluation Item j

S = SubplanWX = Weight related to factor X

Retrieved Case Estimate (RCE)

)())(()( CRCAFCSARSCRCE

LOBRFSF

L

WWW

CWOBCWRFCWSFCRCAF

)()()(

)(

RCE= Retrieved Case Estimate ARS = Average Result of a Subplan (see figure 6)RCAF = Retrieved Case Appropriateness FactorWOBL= Weighted Overall Belief of the Learner (applied to a case)

WSF =Weighted Similarity FactorWRF =Weighted Reuse FactorC = CaseS (C) =Subplan attached to the case CWX = Weight related to factor X

Created Plan Estimate (CPE)

NL

iL

NL

iLi

i

i

W

WIPLCLEIPCPE

1

1

))(()(

PCN

i

i

PCN

i

ii

LCCPF

LCCPFLCRCELCLE

L

1

1

))((

))(())(()(

IFTF WW

CWIFCWTFCCPF

)()(

)( CPE = Created Plan EstimateNL = Number of Levels in the Instructional PlanCLE= Created Level EstimatePCN= Number of Primary CasesRCE= Retrieved Case Estimate (see figure 7)CPF =Created Plan FactorWTF =Weighted Stability FactorWIF = Weighted Importance FactorIP = Instructional PlanL, Li = Levels of the Instructional Plan

C, Ci= Cases

WX = Weight related to factor X