INTELLIGENT TUTORIAL SYSTEM

& ITS ENHANCEMENT USING

EMOTIONAL FEEDBACK

Soumya Bose

11EC65R09, VIPES

1

SCOPE

● Emergence of Intelligent Tutoring System(ITS)

● Central framework of ITS.

● Different Modules

● NewGenerationITS (NGITS)● NewGenerationITS (NGITS)

● Need of emotional feedback

● Facial Emotion Analysis

● Conclusions

● Future Work2

EXISTING TUTORING SYSTEMS (CAI)

● E-learning – Web Based Tutorials using Audio/Video.

● Advantages:�Low Cost.�Learn when you need.�Not constrainedby geographicallocation.�Not constrainedby geographicallocation.

● Disadvantages:�Based on Simple Computer Aided Instructions�No student teacher feedback.�Lack of understanding.�Appears to be boring.

3

INTELLIGENT TUTORIAL SYSTEM

� Outgrowth of Computer Aided Instructionswith added intelligence.

� Systemwhere teacher-student communicationstrengthenslearningprocess.strengthenslearningprocess.

� Tracks student’s performance.

� Adaptive systemwhere next state of instructionis dependant upon student’s performance.

4

CENTRAL FRAMEWORK OF ITS

Traditionally ITS has the following components:

DOMAIN

MODULE

PEDAGOGICAL

MODULE

STUDENT

MODULE

TASK

ENVIRONMENT

Fig.1: ITS Framework 5

DOMAIN MODULE

� Knowledge database.

� Depends upon the domain in which the ITS is intended to instruct.

� Domain module is prepared with a view of cognitive psychology of human skill acquisition.

Knowledge DECLARATIVE or PROCEDURAL

6

Example

Q. How to evaluate an expression with different arithmetic operands?

Declarative:

Priority1. ‘( operation )’

Priority2. ‘/’

Procedural:

Evaluation: (3-4) x 18 / 9 + 5 - 8

Step1. A= (3-4) = -1

An ITS to tutor school mathematics.

Priority3. ‘x’

Priority4. ‘+’

Priority5. ‘-’

Step2. B= 18 / 9 = 2

Step3. C= A x B = -1 x 2 = -2

Step4. D= C+5 = -2+5 = 3

Step5. Ans.= 3-8 = -5

�Combination of both provides effective learning process7

TTTTASKASKASKASK EEEENVIRONMENTNVIRONMENTNVIRONMENTNVIRONMENT

Interface for student - teacher communication

GUI may serve the purpose

�Tutor displays instructions.

�Student gives input through a text editor.

Problem Statement Tutor Instructions

text editor.

�A simulation response to the student forms a feedback to the student.

�Feedback helps student in reasoning.

Output Window Students Worksheet

Fig.2: Typical Task Environment

8

PEDAGOGICAL / TUTORING MODEL

● Structuring the instructions.�At curriculum level it is sequence of information.�At problem solving level it can intervene to advise students.

● Next instructions should be on the basis of present state (can be modeled as a tree data structure).

● Present state includes (Logical decision):�Current stage of domain module.�Knowledge and Emotional state of student. 9

C

1

C

2

C

3

C

4

C

n

C

1

C

2

C

3

C

7

C

n

C

3

C

6

C

4

C

8

1. Primitive linear structure

2. Branched structure

IIIINSTRUCTIONNSTRUCTIONNSTRUCTIONNSTRUCTION STRUCTURESSTRUCTURESSTRUCTURESSTRUCTURES

C

6

C

7

C

8

C1

0

C

n

C

2

C

5

C

9

C

1

C

3

C

4

C1

1

C1

2

C1

3

C1

4

C

n

C

n3. Multilevel structure

Cn is quantum of domainknowledge / information inthe nth stage 10

SSSSTUDENTTUDENTTUDENTTUDENT MMMMODULEODULEODULEODULE

Record of students knowledge state.

Student module is dynamic: Knowledge state is changing (Modeling is complex).e.g: Identifying Operands

Identifying Operators

Doing operationsDoing operations

Evaluating Expressions

Solving Equations

Statistical methods used for estimating student’s knowledge state (by psychophysicist Green and Swets, 1973)

Intelligence of Domain Expert module assess student’s performance

11

ITS DITS DITS DITS DESIGNESIGNESIGNESIGN FFFFLOWLOWLOWLOW

Modeler

Expert Simulator Knowledge Base Student Model

TUTORING SYSTEM

Predicted and Preferred behavior Relations and

Student PrototypesUpdate Model

Tutor

Problem Student

ProblemInformation

Data Request

Problem Data

Explanation DataProblem Solving

Situation

Advice &Explanation

Students Current state

12

A TA TA TA TYPICALYPICALYPICALYPICAL ITSITSITSITS

Problem Statement Tutor Instructions

Output Window Students Worksheet

Hint: You have done an independentOperation. Calculate 1st priorityoperation

Hint: You have done 2nd priority operation which is also independent of the first operation. Calculate 1st

priority operation

Hint: 1st priority operation done.Write down the modified expressionHint: Calculate the 1st priorityOperation now on the modifiedexpression

Hint: Wrong! Now operations aredependant. Calculate 1st operationHint: Calculate 2nd operation.Problem Solved! ☺1.Evaluate the expression:

Z= (3-4) x 18 / 9 + 5 – 8Hint: Evaluate 1st priority operation

Output Window Students Worksheet

-1 x 2 = -2

1. A = -32. B = 2

4. Expression5. Wrong5. -26. -5

3. C= -1

5 - 8 = -318/9 = 23 - 4 = -1 -1 x 2 – 32 – 3 =1-2 – 3 = -5

13

PPPPRACTICALLYRACTICALLYRACTICALLYRACTICALLY IIIIMPLEMENTEDMPLEMENTEDMPLEMENTEDMPLEMENTED SSSSYSTEMSYSTEMSYSTEMSYSTEMS

1. The PUMP Algebra Tutor (PAT) (Anderson, Corbett, Koedinger and Pelletier, 1995):

Used for tutoring introductory algebra in Pittsburg Schools.

14

Fig.3.: The PAT GUI (Courtesy: Ref.4)

2. The SHERLOCK Project (Lesgold, Laioie, Bunzo and Eggan, 1992; Katz and Lesgold, 1993):

It is a practice environment for electronics troubleshooting commissioned by Air-Force.Fig.4.: The SHERLOCK interface

(Courtesy: Lawrence Elbaum Associates)

SSSSHORTCOMINGHORTCOMINGHORTCOMINGHORTCOMING OFOFOFOF CCCCURRENTURRENTURRENTURRENT SSSSYSTEMSYSTEMSYSTEMSYSTEMS

Q. What is the value of force acting on a body of mass 3Kg and moving with a retardation of 4m/s2?

Ans. : -12N

Ans. : -12N

Confident

Not-Sure

Evaluation by Current systems (Based on knowledge state):

In either of the above cases student will be assumed to have a knowledge of ‘FORCE’.

Ans. : -12N Not-Sure

15

NNNNEWEWEWEW GGGGENERATIONENERATIONENERATIONENERATION ITSITSITSITS

Aimed at developing more adaptive tutors

Expert Module evaluates both:�Knowledge state�Emotional / Mental state

Measuring mental state of learner by bio signals analysis:• Facial expressions

• Signals from Brain

• Electro-dermal signals

• ECG signals, etc.16

FFFFACIALACIALACIALACIAL EEEEMOTIONMOTIONMOTIONMOTION AAAANALYSISNALYSISNALYSISNALYSIS

● Involves lot of real time image processing.

● Time is the constraint.

● Should be processed in parallel with knowledge

Challenges:

● Should be processed in parallel with knowledge evaluation.

Hardware / Software co-design approach is adopted for fast processing.

17

BBBBASICASICASICASIC SSSSTEPSTEPSTEPSTEPS

IMAGE GRABBER PRE PROCESSING

FEATURE EXTRACTION

COARSE

REGION WAVELET

TRANSFORM

SOBEL /

LAPLACIAN

EDGE

GENETIC

ALGORITHM

MATCHINGINTERPRETATION

REGION

SEGMENTATION

EDGEDETECTION

TRANSFORMEDGE

18

● Labeling : Smoothing small variations in intensity.

● Segmentation : Finding edges or sharp transitions.

� Smoothening with linear resistive network blurs edges

Object Segmentation and LabelingObject Segmentation and LabelingObject Segmentation and LabelingObject Segmentation and Labeling

resistive network blurs edges of objects.

� Resistive Fuse networks are used to label and segmenting the image. 19

Resistive Fuse NetworkResistive Fuse NetworkResistive Fuse NetworkResistive Fuse Network

�The resistive fuse acts as a linear resistorfor |Vdiff| < Voff

�Acts as an open circuit for |Vdiff| > Voff

The change in voltage at each node

Fig.5.: Fuse Resistor

The change in voltage at each nodecan be calculated from Kirchoff’s current law:

Vout2,2(t+1) – Vout2,2(t) =

v[ ∑ G(Vout i,j – Vout2,2) + σ(Vin2,2- Vout2,2) ]i,j∈N

2,2

Fig.6.: Segmentation Circuit 20

v[ ∑ G(Vout i,j – Vout2,2) + σ(Vin2,2- Vout2,2) ]i,j∈N

2,2

The equation

can be realized in FPGA:

e = sample pixela – i = neighbor pixelSmem = pixel data Vin is stored Dmem = Output data Vout is storedLUT1 performs σ evaluationLUT2 performs G (fuse value)

per channel (RGB) 21

Fig.8.: FPGA implementation (Courtesy: Ref.7)

Fig.7.: Raster Scan of the image (Courtesy: Ref.7)

Gabor Wavelet Transform (GWT)Gabor Wavelet Transform (GWT)Gabor Wavelet Transform (GWT)Gabor Wavelet Transform (GWT)

Wavelet function is multiplication of a harmonic function and gaussian function

determines frequencyand the direction

GWT is performed overthe segmented object for feature extraction

22

Fig.9.: GWTs of the face (Courtesy: Ref.7)

Matching & InterpretationMatching & InterpretationMatching & InterpretationMatching & Interpretation

�Edges of features (eye / lips / nose): Sobel / Gaussian operator

•GA is an iterative process .

•In each step shapes arematched with known curves

Fig.10: Matching eye with an ellipse (Courtesy: Ref.8)

�Matching: Genetic Algorithm (GA)

•Termination occurs whenerror is minimized.

Final matched curve parameters (like major axis/ minor axis) are matchedwith known values to predict emotions:

HappySadFrustration, etc.

23

Fig.11.: Matching lips with an irregular ellipse (Courtesy: Ref.8)

CCCCONCLUSIONSONCLUSIONSONCLUSIONSONCLUSIONS

�ITS has been proved efficient and stronger than simple CAI�Involves student in sustained reasoning activity.�Problem solving tutor helps conceptual understanding as well as

solving real life problems related to a domain.�High level GUI attracts students for learning

�The ability to read the mental state of the learner through facial emotional analysis: Increases Adaptabilityemotional analysis: Increases Adaptability�Repeated instructions can be delivered on the basis of mental satisfaction�Helps student in sound understanding

However the accuracy of the emotional analysis can be improved significantly adding voice information

The main drawback is it is an one-to-one process 24

�Different bio-signals processing: Assess mental state of the student more correctly.

�Electro dermal signal(GSC): Human skin is a weak conductor of electricity.

Fig: Galvanic Skin Response (Courtesy: Springer Images)

FFFFUTUREUTUREUTUREUTURE DDDDIRECTIONSIRECTIONSIRECTIONSIRECTIONS

�Brain Signals (EEG): Higher frequency beta waves (15-25Hz) and low theta waves implies seriousness

�ECG can be analyzed to detect stress, low confidence of the student.

Real time processing several bio-signals will make the design complex.But even if half the ability of real human tutor is realized the payoff to the society will be substantial 25

1. Abdolhossein Sarrafzadeh, Hamid Gholam Hosseini, Chao Fan, Scott P. Overmyer; Facial Expression Analysis for Estimating Learner’s Emotional State in Intelligent Tutoring Systems; Proceedings of the The 3rd IEEE International Conference on Advanced Learning Technologies (ICALT’03); 2003

2. Morteza BIGLARI-ABHARI, Abbas BIGDELI; FPGA Implementation of Facial Expression Analysis For Intelligent Tutoring Systems; Proceedings of the II International Conference on Multimedia and Information & Communication Technologies in Education; ICTE ‘2003, Spain

3. Sunandan Chakraborty, Devshri Roy, Anupam Basu; Development of Knowledge Based Intelligent Tutoring System; Indian Institute of Technology, Kharagpur, India; 2001

4. M.Helander, T. K. Landauer, P. Prabhu (Eds), Elsevier Science B. V.;Intelligent Tutoring Systems, Handbook of Human-Computer Interaction, Second Edition; 1997

5. Arjen Hoekstra and JorisJanssen; Linking Bio-signals to Transfer of Knowledge Towards

RRRREFERENCESEFERENCESEFERENCESEFERENCES

5. Arjen Hoekstra and JorisJanssen; Linking Bio-signals to Transfer of Knowledge Towards Mind-reading ECAs; Faculty of Electrical Engineering, Mathematics and Computer Science University of Twente, The Netherlands

6. Teppei NAKANO, Hiroshi ANDO, Hideaki ISHIZU, Takashi MORIE, Atsushi IWATA; Coarse Image Region Segmentation Using Resistive-fuse Networks Implemented in FPGA; Graduate School of Life Science and Systems Engineering,Kyushu Institute of Technology; Graduate School of Advanced Sciences of Matter, Hiroshima University; Hiroshima Prefecture Industrial Research Institute

7. T. Nakano, T. Morie and A. Iwata; A Face/Object Recognition System Using FPGA Implementation of Coarse Region Segmentation; SICE Annual Conference in Fukui, Fukui University, Japan; 2003

8. M. Karthigayan, M. Rizon, R. Nagarajan and Sazali Yaacob; Genetic Algorithm and Neural Network for Face Emotion Recognition; School of Mechatronics Engineering, UniversitiMalaysia Perlis (UNIMAP); 2006

26