Tuteurs métacognitifs : Supporter la métacognition par la reflexion

Roger Nkambou

What is a “Cognitive Model”?

A simulation of human thinking & resulting behaviorUsually used to explain or predict data on human behavior Like error rates or solution time

Usually implemented as a computer program that can behave like humans Often using AI knowledge representations

like semantic nets, frames, schema, production rules

What are Cognitive Models used for?

Output of basic research Explain results of psychology experiments

Guide design of software systems Have cognitive model “use” the system

Model predicts people’s time & errors(VanLehn) Redesign system to reduce time or errors

Can derive predictions without full implementation (e.g., Ethan)

As a component in an intelligent system Player in a game or training simulation Part of expert system or intelligent tutor

What is an “Intelligent Tutoring System” (ITS)?

A kind of educational softwareUses artificial intelligence techniques to Provide human tutor-like behavior Be more flexible, diagnostic & adaptive Write more general code to get more

capabilities with less effort

Components of an ITS: Interface or problem solving environment,

domain knowledge, student model, pedagogical (tutoring) knowledge

Reflective thinking & tutoring meta-cognition

Cognitive Modeling and Intelligent Tutoring Systems

Ken KoedingerVincent Aleven

Overview

ACT-R background & declarative transfer

Two studies of tutoring meta-cognition

Future: 3rd generation tutors

Different Learning Goals

From: e-Learning and the Science of Instruction : Proven Guidelines for Consumers and Designers of Multimedia Learning by Ruth Colvin Clark & Richard E. Mayer, 2002.

Corresponding Instructional Approaches

ACT-R’s declarative-procedural distinction

Declarative knowledge Includes facts, procedures that people can describe Stores inputs of perception & includes visual memory

Procedural knowledge Performance knowledge, cannot be verbalized

Procedural k “runs on hardware” Efficient

Declarative k is interpreted by procedural k Can be flexibly adapted But requires associated interpretive procedural k

Calculus Study in Declarative Transfer chapter of Singley & Anderson

What’s the difference between operator selection & operator application?What are the four training conditions in the study? What’s the same in all 4?During test (day 2) the interface is like which training condition?Is there transfer from operator … application to selection? selection to application?

Declarative Transfer Summary

Declarative k is basis for transfer b/t different uses of same knowledgeMay be short-lived & sometimes overshadowed by extended practiceNeed to search for source of analogy Can be problematic (Gick & Holyoak) Requires world knowledge & can serve well

as a learning & transfer mechanism even as young as 3 yrs old (Brown & Kane)

Overview

ACT-R background & declarative transfer

Two studies of tutoring meta-cognition

Future: 3rd generation tutors

Meta-Cognition 1: Encourage Active Declarative Processing Through Self-Explanation

Aleven, V. & Koedinger, K. R. (2002). An effective metacognitive strategy: Learning by doing and explaining with a computer-based Cognitive Tutor. Cognitive Science, 26(2)

Problem: Shallow knowledge acquisition

Variations on shallow knowledge Over-general procedural knowledge

right for wrong reason No declarative k -- cannot explain, transfer

Geometry example “Looks-equal” production rule If the goal is to find angle A

and it looks equal to angle B and angle B is D degreesThen conclude that angle A is D degrees

Example of Shallow Reasoning

Hypothesized SolutionActive processing of declarative knowledge of problem-solving principles leads to: Better detection of relevant features behind

correct inference Provides dual code for enhanced memory Less error-prone implicit procedural learning

Instructional manipulation: Ss explain steps using principles

& get feedback on explanations

Explanation Condition

Problem solving answers

Explanation by reference

Problem Solving Condition

SE Study 1 MethodBetween subjects comparison: Problem Solving vs. Explanation

Run in a Geometry class at local HSParticipants 41 high school geometry students total 24 Ss provided complete data, pre-test,

tutor, & post-test

About 7 hours of instruction Ss done when they satisfy tutor’s mastery

criteria on problem solving skills

HypothesisRequiring students to explain steps results in deeper understanding: Less shallow procedural knowledge More general declarative knowledge

Consequences: Better reason giving Near transfer as good or better Better far transfer

Pre/Post Test Items

Problem-solving items Answer - Finding unknown quantities

Items associated with deeper understanding Reason - Explain answers by citing

geometry rule Not Enough Info - Transfer items where

students are asked to judge if there is enough information to find quantities, and the answer is “No”.

Assessing transfer: “Not Enough Info” item

Assessing transfer: Incorrect over-generalization

SE Study 1 Results

Answer Items Reason Items Not Enough Info Items

0

.1

.2

.3

.4

.5

.6

.7

.8

.9

Reason

Answer Only

%

Corr

ect

Condition

Possible confounding factors in study 1: time & S prior ability

Answer-OnlyCondition

ReasonCondition

Time on Tutor 383 mins 436 mins

Prior Score 82.3 87.3

Number of Problems Solved 135 102

Neither difference is statistically significant but ... Hard to rule out alternative explanation: Explanation condition had more time & higher prior ability

Self-Explanation Study 2 Motivation

Replicate the results of Study 1, while controlling for time on task

SE Study 2 MethodBetween subjects comparison: Problem Solving vs. Explanation

Run in a Geometry class at local HSParticipants 53 students total 41 provided complete data

7 hours of instruction Time fixed, so all students spent the same

time

No time differences in Study 2

Answer-OnlyCondition

ReasonCondition

Time on Tutor 501 mins 513 mins

Number of Problems Solved 111 76

Differences between conditions cannot be attributed to differences in time on task

SE Study 2 Results

Answer Items Reason Items Not Enough Info Items.2

.3

.4

.5

.6

.7

.8

Pro

port

ion C

orr

ect

on P

ost

-Test

ReasonAnswer Only

Numerical steps Explanations Transfer items

Problem SolvingExplanation

Condition

Different instruction => different kinds of knowledge acquisition

Shallow (over-general procedural) Right answers for wrong reason,

wrong answers when pressed

Procedural Right answers with correct knowledge Efficient, fluent, but inflexible

Declarative Principles interpreted & reflectively applied Flexible, but slow & may fail in high

cognitive load situations

Extra Practice in Problem Solving => More Shallow Learning

Easy to guessitems

Hard to guessitems

.3

.4

.5

.6

.7

.8

.9

1

% C

orr

ect Explanation

Problem Solving

Condition

Shallow Procedural Knowledge vs. “Frontal” Control

Commission errors / total errors

Easier-To-Guess Harder-To-Guess.1

.2

.3

.4

.5

.6

.7

.8

.9

Explanation

Problem Solving

Condition Explanation Problem Solving

Problem Solving group jumps to incorrect conclusions

Explanation group shows more control, reflects on sufficiency of knowledge

Student Performance During Instruction

ExplanationCondition

Problem -SolvingCondition

SuccessRate

# ofSteps

SuccessRate

# ofSteps

All problemsolvingsteps

51% 237 56% 457

First 237steps

51% 237 51% 237

Rest ofsteps

- - 62% 220

Explanationsteps

55% 236 - -Problem solving group appears better at end of tutoring. But, not better on post-test!

Shallow procedural knowledge acquisition => lack of transfer

Estimating Acquisition of Different Knowledge Types

Knowledge Type ExplanationCondition

ProblemSolving

Condition

Shallow ProceduralKnowledge

0.58 0.68

Correct ProceduralKnowledge

0.30 0.42

DeclarativeKnowledge

0.32 0.12

Predicts Performance on Different Test Items ...

Explanation Condition Problem-SolvingCondition

Type of Test Item Variable Actual Predicted Actual Predicted

Numeric, Easier-to-Guess E 0.76 0.80 0.84 0.84

Numeric, Harder-to-Guess H 0.54 0.52 0.49 0.49

Not Enough Information N 0.58 0.61 0.41 0.41

Explanation R 0.51 0.49 0.34 0.34

ImplicationsWhen Ss explain they learn more & learn with greater understanding: better explanations of answers better on harder-to-guess test items better on transfer questions

Possible to achieve benefits of self-explanation with simple manipulationFuture work: system with which students can explain in their own words

Meta-Cognition 2: Supporting Error Detection & Self-Correction

PhD student Santosh Mathan

Benefits of Immediate Feedback

Supports efficient skill acquisition Eliminates

floundering

LISP Tutor study Faster learning Same post-test

65432100

250

500

750

1000

1250

1500

Immediate Feedback

Error Flagging

Demand FeedbackNo Feedback

Tutor Lesson

Criticisms of Immediate Feedback

Qualitative Basis Human tutors may wait (Merrill, 1995) But, just because humans do it ...

Empirical basis Benefits of delayed feedback in motor learning

Schmidt et al., 1988

Some cognitive studies Transfer (Lee, 1992) Retention (Schooler & Anderson, 1985)

Recasting Delayed vs. Immediate Feedback Debate

Debate cast in terms of latencyAlternative: What is the “model of desired performance”?Expert Model

immediate error correction emphasizes generative skills

Intelligent Novice Model allows errors, guides students through error

detection & correction emphasizes generative & evaluative skills

Domain of study

Cell referencing in Excel spreadsheet programming

“Glass ceiling” in natural spreadsheet use & skill acquisition

Expert Feedback

Expert Feedback

Intelligent Novice Feedback

Intelligent Novice Feedback

Intelligent Novice Feedback

Participants

48 participants recruited from a temporary employment agency

All had general computer experience

No Excel experience

Instruction, transfer & retention testing

Day 1

Day 2

.

.

.

.

Day 3

Pre Test Declarative Procedural Post Test

Procedural Post Test

Transfer Pre Test Procedural Post Test

90 min

50 min

30 min

8 days later

Kinds of Pre & Post Tests

Prior experience tests Computer experience questionnaire Algebra word problems

Excel coding testExcel concept testTransfer coding task More complex with novel demands

Results

Students using intelligent novice model tutor significantly outperformed students using expert-model tutor on all measures Coding Concepts Retention Transfer

Coding Performance

F = 4.23, p < .05

75.90%

85.20%

0%

20%

40%

60%

80%

100%

Expert Intelligent Novice

Conceptual Performance

F =4.06, p < .05

66.60%72.90%

0%

20%

40%

60%

80%

100%

Expert Intelligent Novice

Retention Session Performance

F = 4.07, p < .05

72.50%

81.20%

0%

20%

40%

60%

80%

100%

Expert Intelligent Novice

Transfer Performance

F = 5.662, p < .03

59.79%

74.31%

0%

20%

40%

60%

80%

100%

Expert Intelligent Novice

Learning Curves: Difference Between Conditions Emerges Early

6 production rule model - surface feature model that separates surface differences

Opportunities to apply a production rule

Num

ber

of a

ttem

pts

at a

ste

p

Learning Curves: Deeper Generalizations

4 production rule model - deep feature model that merges surface feature differences

Opportunities to apply a production rule

Num

ber

of a

ttem

pts

at a

ste

p

ImplicationsIntelligent novice (IN) model feedback produces: better learning outcomes, retention, &

transfer

On-line data shows effects when coded by production rulesDifference in declarative encoding EX: shallow declarative encoding IN: more general declarative encoding

Future Contrast IN & delayed feedback

Summary: Cognitive Tutors & Meta-Cognition

Cognitive Tutors are experimental instrumentation Control instruction, long duration,

fine grain pre-coded learning data

Can support meta-cognitive processes to enhance transfer & retention Simple self-explanation by reference works Can model intelligent novice error detection

& self-correction skills Reinterpretation of feedback timing debate

Summary: Cognitive Tutors & Meta-Cognition

Can support development of both fluent domain experts & flexible "intelligent novices" Not just better, but different learning by supporting meta-cognitive processesDifferent knowledge is acquired: Less shallow procedural knowledge More declarative knowledge & interpretive

procedures

Overview

ACT-R background & declarative transfer

Two studies of tutoring meta-cognition

Future: 3rd generation tutors

3 Generations of Tutors

1st Generation -- Been on the market Underlying technology: Hypertext & Behaviorism Pedagogy: Didactic feedback on answers.

2nd Generation -- Emerging in the market Technology: Artificial Intelligence & Cognitive Psychology Pedagogy: Assistance on problem solving steps, not just

final answers

3rd Generation -- Emerging in the lab Technology: Natural language processing, reactive

planning, Cost-effective pedagogical-content assessment Pedagogy: Knowledge constructing dialogs

The first generation:Computer Aided Instruction (CAI)Underlying technology: Hypertext & BehaviorismPedagogy: Didactic feedback on answers.Example:

Solve 2+2x=12 Multiplication has ahigher precedencethan addition, so 2+2xis the same as 2+(2x),not (2+2)x. Try again.x=7

x=3

x=5

OK

OK

Excellent!

First Generation Tutor Example

Second generation:Intelligent Tutoring Systems (ITS)Underlying technology: Artificial intelligence & cognitive psyPedagogy: Assistance on problem solving steps, not just final answersExample:

Tutor: Solve 2+2x=12 Student: <enters 4x=12> Tutor: Not quite. Try again. Student: <clicks on “hint” button> Tutor: Think about operator

precedence. Student: <enters 2x=12-2> Tutor: Good!

2 + 2x = 12 4x = 122x = 12 - 2

Student’s workspace:

Tutor:

Good!

Hint

Algebra Cognitive Tutor (CL, Inc)

Technology: Reactive planning & Natural language processingPedagogy: Knowledge constructing dialogsExample:

Tutor: Solve 2+2x=12 Student: 4x=12 Tutor: Should this equation have the

same solution as the first one? Student: Yes. Tutor: The solution to 4x=12 is 3,

so let’s check for an error by trying x=3 in 2+2x=12.

Student: 2+2*3=2+6=8 oops! Tutor: Right! Now look at the

arithmetic steps you did …

Third generation

2+2x=124x=12

Student’s workspace:

Dialog:

S: 2+2*3=2+6=8 oops!T: Right! Now look...

Hint

The nested loops of CAI (1st generation tutors)

For each chapter in curriculumRead chapterFor each exercise Attempt answer Get feedback & hints on answer; try again If mastery is reached, exit loop

Take a test on chapter

The nested loops of ITS (2nd gen)

For each chapter in curriculumRead chapterFor each exercise For each step in solution

Student attempts step Get feedback & hints on step; try again

If mastery is reached, exit loop

Take a test on chapter

The nested loops of dialogue-based tutors (3rd gen)

For each chapter in curriculumRead chapterFor each exercise For each step in solution

Student attempts step If incorrect, for each inference in a directed

line of reasoning Elicit the inference from student Hint, prompt, pump; try again If S completes step, exit loop

If mastery has been reached, exit loop

Take a test on chapter

Limitations of 2nd-generation tutors

Better than classroom instruction, but not as good as human tutors! Human tutors 2 better (standard deviations or

“sigma”) than classroom instruction The best 2nd-generation tutors are 1 better

Do not always lead to deep understanding Symptoms of shallow learning

Lack of transfer to novel problems Inability to explain / carry on coherent abstract

conversation about the domain A problem for many instructional methods!

Third-Generation Tutors

Knowledge construction dialogs“there is something about conversational dialog

that plays an important role in learning”.

Better theory about how to get students to learn:“Good tutors tell less and ask more.”They guide students as they construct new

knowledge. Help them make abstractions, connections.

Third-generation tutors Research agenda

Empirical When and why is tutorial dialog effective?

Technical Develop systems that support students in

constructing knowledge through dialog Evaluate the effectiveness of the systems

Goal is to rival or surpass the effectiveness of human tutors

Examples of 3rd Generation Tutors

Andes/Atlas: Dialog instead of hintsWhy/Atlas: Dialog critiques of an essay-long physics explanationCIRCSIM: Dialog to correct errors in students’ predictions about physiological causalityAutoTutor: Dialog about computer literacyGeometry explanation: Expressed in dialog instead of menu choice & feedbackMs. Lindquist: Dialog about methods for analyzing algebra word problemsCATO: Courtroom dialog to teach heuristics for citing cases

Andes/Atlas: Dialogue replaces hint sequences

Andes: If you are moving in a straight line and accelerate in the same direction, does your velocity increase or decrease?

You: increase

Andes: You’ve drawn the acceleration of the elevator in the same direction as the velocity. Is the velocity of the elevator increasing?

Why/Atlas

CIRCSIMMartha Evens, Reva Freedman, Michael Glass, Yujian Zou, et al., Illinois Institute of Technology

Domain: physiology (negative feedback loop that controls blood pressure)Emphasis: dialog to correct errors in students’ predictions about physiological causalityShort answer questions, hint strategiesModeled after expert human tutors

CIRCSIM-Tutor User Interface

Problem: Pacemaker malfunctions, increasing to 120 beats/min.

T> What variable is affected by HR?S> Cardiac Output.T> But you predicted that HR increases and CO decreases.S>

DR RR SSCentral Venous Pressure -Inotropic State 0Stroke Volume -Heart Rate +Cardiac Output -Total Peripheral Resistance 0Mean Arterial Pressure +

MeanArterial

Pressure

TotalPeripheralResistance

ArteriolarResistance

ArteriolarMuscleTone

ArteriolarDiameter

StrokeVolume

CentralVenous

Pressure

InotropicState

Alpha-Blocker(drug)

Pacemaker Rate Up(or Down)

IntracellularCa++

Concentration

Beta-Blocker(drug)

Transfusion(or Hemorrhage)

Blood VolumeCentral Blood

Volume

CardiacOutput

HeartRate

Sino-AtrialNodeRate

NervousSystem

Response

BaroreceptorPressure

A Family of Possible Dialogs with CIRCSIM

I haveno idea

TPR isneurally

controlled

Sympatheticvasoconstriction

Nervoussystem

Radius ofarterioles

Right

Right.TPR isneural.

And whatcontrols

that?

Nervoussystem

Right

(3) So what must be the value of TPR?

(1) Can you tell me what controls TPR?

(2) Remember that we’re in DR now.

<wrong>

POP

AutoTutorArt Graesser, Derek Harter, Peter Wiemer-Hastings, Natalie Person, et al., University of Memphis

Domain: Computer LiteracyEmphasis: Simulating the average human tutor Mixed-initiative dialog with deep-reasoning questions Interface: Talking face, with gestures, intonation

Dialog Moves Pumping Prompt, hint, assert

Feedback Back channeling Pedagogical feedback (evaluation of answer) Corrective feedback

Responding to student initiative

AutoTutor with Marco Talking Head

An Example Tutorial Dialog with AutoTutorTUTOR-1: Alright, let's go on. You want to take photos and send them over the Internet. Consider this problem. What hardware do you need to take photos and send them over the Internet?

STUDENT-1: First of all you need to have access to the internet

TUTOR-2: What else?STUDENT-2: a cameraTUTOR-3: Yeah. For what type of connection do you need a network card?

STUDENT-3: network cardTUTOR-4: Okay. Now, if you have access to the Internet through a network card, then your connection is …

STUDENT-4: I don't know.TUTOR-5: Is direct. Okay. A network card is needed if you have a direct connection to the Internet. Okay. How does the user get hooked up to the internet?

STUDENT-5: by a plug.

TUTOR-6: Once again, a network card is needed if you have a direct connection to the Internet. Okay, what does a modem do?

STUDENT-6: connects network card and internet

TUTOR-7: Okay. Try this. A modem is required when there is …

STUDENT-7: a connectionTUTOR-8: a dial up connection. ...TUTOR-30: Let’s review. To send your photos on the Internet, you need either a digital camera or a regular camera to take the photos. If you use a regular camera, you need a scanner to scan them onto a computer disk. If you have a direct connection to the Internet, then you need a network card. A modem is needed if you have a dial up connection.

Why might natural language dialog enhance learning (or not)?Because it’s natural…

Draws student more into thinking into higher levels Leading student on to get more help whereas

student has to press more in 2nd gen

More tutor interventionWith speech processing Easier to speak than to write,

easier to listen than to read

S can express ideas informally in NL Reveals partial knowledge

Goal: Help students learn with greater understanding by tutoring at the explanation level

From: Guided learning by doingTo: Guided learning by doing and explaining

Motivated by Cognitive Science literature on the “Self-Explanation Effect”How can a computer tutor best support Self-Explanation? Have students explain in their own words? Is feedback on explanations necessary?

A 3rd-Generation Tutoring System: The Geometry Explanation Tutor

Study 1: Explanation ‘by reference’

Tutor feedback indicates whether explanation is correct.

Study 2:How about explaining in one’s own words, but without tutor feedback on explanations?

Students produce very few good explanations ...

Hypothesis: Students learn best when they explain in their own words and tutor help them to refine explanations, through dialog.

Geometry Explanation Tutor—A Tutorial Dialog System to Support Self-Explanation

Example Dialogue with the Geometry Explanation Tutor

Student: angles are equalTutor: Are any two angles congruent? What tells

you that there are two congruent angles here?Student: angles in a triangle are equalTutor: That is a bit imprecise. Are ALL angles in

ALL triangles congruent?Student: angles in an isosceles triangle are equalTutor: Correct, except for one thing: are all three

angles in an isosceles triangle congruent?Student: base angles in an isosceles triangle are

equalTutor: CORRECT

USER INTERFACE

LCFLEX PARSER

FEATURE STRUCTURE

UNIFIER

LOGIC SYSTEM (Loom)

PRODUCTION ENGINE

COGNITIVE MODEL

SEMANTIC REPRESENTATION

of Explanation

FEATURE STRUCTURES

KNOWLEDGE BASE —Ontology

& Explanation Hierarchy

GRAMMAR & LEXICON

COGNITIVE TUTOR

NLU COMPONENT

STATISTICAL CLASSIFIER

Student Explanation

Feedback or Help Message

Detailed Classification of

Explanation

Ballpark Classification of

Explanation

(Numerical)Answer or

Hint Request

STUDENT MODEL

Pedagogical Content Knowledge: Explanation Hierarchy

Hierarchy of Partial ExplanationsExcerpt — Isosceles Triangle TheoremUNKNOWN

CONGR-ANGLES“The angles are congruent.”BASE-ANGLES

“These are base angles.”

BASE-ANGLES-CONG“Base angles are

congruent.”

CONGR-ANGLES-IN-TRI“Angles in a triangle are

congruent.”

TRI-BASE-ANGLES“Base angles in a triangle

are congruent.”

CONGR-ANGLES-IN-ISOS-TRI

“Angles of an isosceles triangle are congruent.”

ISOS-TRI-BASE-ANGLES“Base angles in an isosceles

triangle are congruent.”

ANGLES-OPP-SIDES“Angles opposite the sides are

congruent.”

ANGLES-OPP-CONGR-SIDES“Angles opposite congruent

sides are congruent.”

ISOS-TRIANGLE“The angles opposite congruent sides in

an isosceles triangle are congruent.”

OPPOSITE-ANGLES“Opposite angles are

congruent.”

Why Might Natural Language Self-Explanations Assist Learning?“There is something about NL dialog that is right ...”It is good for students to explain in their own words …

But why?Natural language explanation requires recall, not recognition.Articulating forces attention to relevant features.Verbal learning and visual learning create “dual codes” in memory.Natural language allows for flexible expression of partial knowledge

Students can show what they do know Tutor can help student construct what they do not know

Help can come in smaller portions Tutor can support alternative developmental pathways to knowledge

construction

3rd Generation Tutor Summary

3 Generations of tutors differ in their underlying technology, psychological theory, and methods for developmentShallow learning can occur when students do not encode relevant features of the taskCIRSIM, AutoTutor, Ms. Lindquist, and the Geometry Explanation Tutor are examples of 3rd generation tutorsIntuitively, natural language dialog seems powerful for learning, but research is exploring when/why