G RAPHICS & I NTERACTIVE P ROGRAMMING Lecture 1 Introduction to Processing.
D ESIGNING AN I NTERACTIVE T EACHING T OOL WITH ABML K NOWLEDGE R EFINEMENT L OOP enabling arguing...
-
Upload
colt-pound -
Category
Documents
-
view
217 -
download
2
Transcript of D ESIGNING AN I NTERACTIVE T EACHING T OOL WITH ABML K NOWLEDGE R EFINEMENT L OOP enabling arguing...
![Page 1: D ESIGNING AN I NTERACTIVE T EACHING T OOL WITH ABML K NOWLEDGE R EFINEMENT L OOP enabling arguing to learn 1 Faculty of Education, University of Ljubljana,](https://reader038.fdocuments.in/reader038/viewer/2022103015/551c0a78550346a84f8b50b4/html5/thumbnails/1.jpg)
DESIGNING AN INTERACTIVE TEACHING TOOL WITH ABML
KNOWLEDGE REFINEMENT LOOPenabling arguing to learn
1 Faculty of Education, University of Ljubljana, Slovenia
2 Faculty of Computer and Information Science, University of Ljubljana, Slovenia
Matej Zapušek1, Martin Možina2, Ivan Bratko2, Jože Rugelj2, Matej Guid2
12th International Conference on Intelligent Tutoring SystemsITS 2014: Honolulu, Hawaii 2014
![Page 2: D ESIGNING AN I NTERACTIVE T EACHING T OOL WITH ABML K NOWLEDGE R EFINEMENT L OOP enabling arguing to learn 1 Faculty of Education, University of Ljubljana,](https://reader038.fdocuments.in/reader038/viewer/2022103015/551c0a78550346a84f8b50b4/html5/thumbnails/2.jpg)
SOME CONCEPTS ARE DIFFICULT TO EXPLAIN...
How to distinguish edible from toxic mushrooms?
![Page 3: D ESIGNING AN I NTERACTIVE T EACHING T OOL WITH ABML K NOWLEDGE R EFINEMENT L OOP enabling arguing to learn 1 Faculty of Education, University of Ljubljana,](https://reader038.fdocuments.in/reader038/viewer/2022103015/551c0a78550346a84f8b50b4/html5/thumbnails/3.jpg)
INTRODUCING DOMAIN EXPERTS
Even for domain experts it is hard to articulate their knowledge!
![Page 4: D ESIGNING AN I NTERACTIVE T EACHING T OOL WITH ABML K NOWLEDGE R EFINEMENT L OOP enabling arguing to learn 1 Faculty of Education, University of Ljubljana,](https://reader038.fdocuments.in/reader038/viewer/2022103015/551c0a78550346a84f8b50b4/html5/thumbnails/4.jpg)
MACHINE LEARNING: HOW TO INVOLVE A DOMAIN EXPERT?
The expert can state constraints and the domain knowledge in advance...
… verify, evaluate, and correct results of machine learning…
… or the expert and the computer iteratively improve the model.
ABMLargument-based machine learning
![Page 5: D ESIGNING AN I NTERACTIVE T EACHING T OOL WITH ABML K NOWLEDGE R EFINEMENT L OOP enabling arguing to learn 1 Faculty of Education, University of Ljubljana,](https://reader038.fdocuments.in/reader038/viewer/2022103015/551c0a78550346a84f8b50b4/html5/thumbnails/5.jpg)
ARGUMENT-BASED MACHINE LEARNING
given
set of labeled learning examples ei
described with attribute values Di
where Ci is classification of learning example ei goal
learn prediction model (hypoteshis) H
IF ... THEN ...IF ... THEN ......
HCiei: Di
ai
example ei may have argument ai
![Page 6: D ESIGNING AN I NTERACTIVE T EACHING T OOL WITH ABML K NOWLEDGE R EFINEMENT L OOP enabling arguing to learn 1 Faculty of Education, University of Ljubljana,](https://reader038.fdocuments.in/reader038/viewer/2022103015/551c0a78550346a84f8b50b4/html5/thumbnails/6.jpg)
IT IS MUCH EASIER TO EXPLAIN INDIVIDUAL CASES!
Is this mushroom toxic? Why?
![Page 7: D ESIGNING AN I NTERACTIVE T EACHING T OOL WITH ABML K NOWLEDGE R EFINEMENT L OOP enabling arguing to learn 1 Faculty of Education, University of Ljubljana,](https://reader038.fdocuments.in/reader038/viewer/2022103015/551c0a78550346a84f8b50b4/html5/thumbnails/7.jpg)
ABML KNOWLEDGE REFINEMENT LOOP
Step 1: Learn a hypothesis with ABML
Step 2: Find the “most critical” example (if none found, stop)
Step 3: Expert explains the example
Return to step 1
critical example
learn data set
ArgumentABML
![Page 8: D ESIGNING AN I NTERACTIVE T EACHING T OOL WITH ABML K NOWLEDGE R EFINEMENT L OOP enabling arguing to learn 1 Faculty of Education, University of Ljubljana,](https://reader038.fdocuments.in/reader038/viewer/2022103015/551c0a78550346a84f8b50b4/html5/thumbnails/8.jpg)
ABML KNOWLEDGE REFINEMENT LOOP
Step 1: Learn a hypothesis with ABML
Step 2: Find the “most critical” example (if none found, stop)
Step 3: Expert explains the example
Return to step 1 Step 3a: Explaining a critical example (in a natural language)
Step 3b: Adding arguments to the example
Step 3c: Discovering counter examples
Step 3d: Improving arguments
Return to step 3c if counter example found
![Page 9: D ESIGNING AN I NTERACTIVE T EACHING T OOL WITH ABML K NOWLEDGE R EFINEMENT L OOP enabling arguing to learn 1 Faculty of Education, University of Ljubljana,](https://reader038.fdocuments.in/reader038/viewer/2022103015/551c0a78550346a84f8b50b4/html5/thumbnails/9.jpg)
ILLUSTRATIVE EXAMPLE: LEARNING TO DIAGNOSE FLU
Pacient Temperature Vaccination Coughing Headache ... Flu
1 normal yes no no ... no
2 high no yes no ... yes
3 very high no no yes ... yes
4 high yes yes no ... no
... ... ... ... ... ... ...
The current model:
IF Temperature < very high THEN Flu = no
cannot explain well Pacient 2.
The question to the expert:
„What is the reason for Pacient 2 having the flu?“
![Page 10: D ESIGNING AN I NTERACTIVE T EACHING T OOL WITH ABML K NOWLEDGE R EFINEMENT L OOP enabling arguing to learn 1 Faculty of Education, University of Ljubljana,](https://reader038.fdocuments.in/reader038/viewer/2022103015/551c0a78550346a84f8b50b4/html5/thumbnails/10.jpg)
Pacient Temperature Vaccination Coughing Headache ... Flu
1 normal yes no no ... no
2 high no yes no ... yes
3 very high no no yes ... yes
4 high yes yes no ... no
... ... ... ... ... ... ...
EXPERT‘S EXPLANATION
Expert‘s explanation:
„Pacient #2 has the flue because of a high temperature.“
Expert‘s argument
is attached to learning example #2. New model is built.
Flu = yes BECAUSE Temperature > Normal
![Page 11: D ESIGNING AN I NTERACTIVE T EACHING T OOL WITH ABML K NOWLEDGE R EFINEMENT L OOP enabling arguing to learn 1 Faculty of Education, University of Ljubljana,](https://reader038.fdocuments.in/reader038/viewer/2022103015/551c0a78550346a84f8b50b4/html5/thumbnails/11.jpg)
Pacient Temperature Vaccination Coughing Headache ... Flu
1 normal yes no no ... no
2 high no yes no ... yes
3 very high no no yes ... yes
4 high yes yes no ... no
... ... ... ... ... ... ...
WHAT IF THE EXPERT‘S ARGUMENT IS NOT GOOD ENOUGH?
ML method now induced a rule consistent with argument:
IF Temperature > Normal THEN Flu = yes
The rule is inconsistent with data! Expert is presented with counter example:
„Compare pacients #2 and #4. Why Pacient #4 doesn‘t have the flu?“
![Page 12: D ESIGNING AN I NTERACTIVE T EACHING T OOL WITH ABML K NOWLEDGE R EFINEMENT L OOP enabling arguing to learn 1 Faculty of Education, University of Ljubljana,](https://reader038.fdocuments.in/reader038/viewer/2022103015/551c0a78550346a84f8b50b4/html5/thumbnails/12.jpg)
Pacient Temperature Vaccination Coughing Headache ... Flu
1 normal yes no no ... no
2 high no yes no ... yes
3 very high no no yes ... yes
4 high yes yes no ... no
... ... ... ... ... ... ...
THE EXPERT MAY IMPROVE THE ARGUMENT
Expert finds the crucial difference between Pacients #2 and #4:
„Pacient 2 didn‘t get vaccinated against the flu.“
![Page 13: D ESIGNING AN I NTERACTIVE T EACHING T OOL WITH ABML K NOWLEDGE R EFINEMENT L OOP enabling arguing to learn 1 Faculty of Education, University of Ljubljana,](https://reader038.fdocuments.in/reader038/viewer/2022103015/551c0a78550346a84f8b50b4/html5/thumbnails/13.jpg)
Pacient Temperature Vaccination Coughing Headache ... Flu
1 normal yes no no ... no
2 high no yes no ... yes
3 very high no no yes ... yes
4 high yes yes no ... no
... ... ... ... ... ... ...
IMPROVED RULES MAY EXPLAIN UNSEEN EXAMPLES AS WELL
ML method induces a new rule:
IF Temperature > Normal AND Vaccination = no
THEN Flue = yes
The new rule also explains diagnosis for Pacient #3:
„Has flu because of a high temperature and didn‘t get vaccinated against it.“
![Page 14: D ESIGNING AN I NTERACTIVE T EACHING T OOL WITH ABML K NOWLEDGE R EFINEMENT L OOP enabling arguing to learn 1 Faculty of Education, University of Ljubljana,](https://reader038.fdocuments.in/reader038/viewer/2022103015/551c0a78550346a84f8b50b4/html5/thumbnails/14.jpg)
ABML KNOWLEDGE REFINEMENT LOOP: THE INNER LOOP
Step 3a: Explaining a critical example (in a natural language)
„Pacient #2 has the flue because of a high temperature.“
Step 3b: Adding arguments to the example
Step 3c: Discovering counter examples
Step 3d: Improving arguments with counter examples
IF TEMPERATURE > NORMAL AND VACCINATION = NO
Temperature > Normal
![Page 15: D ESIGNING AN I NTERACTIVE T EACHING T OOL WITH ABML K NOWLEDGE R EFINEMENT L OOP enabling arguing to learn 1 Faculty of Education, University of Ljubljana,](https://reader038.fdocuments.in/reader038/viewer/2022103015/551c0a78550346a84f8b50b4/html5/thumbnails/15.jpg)
ABML REFINEMENT LOOP & KNOWLEDGE ELICITATION
IF ... THEN ...IF ... THEN ......
ABMLargument-based machine learning
explain single example easier for experts to articulate knowledge
“critical” examples expert provides only relevant knowledge
“counter” examples detect deficiencies in explanations
arguments
critical examplescounter examples
![Page 16: D ESIGNING AN I NTERACTIVE T EACHING T OOL WITH ABML K NOWLEDGE R EFINEMENT L OOP enabling arguing to learn 1 Faculty of Education, University of Ljubljana,](https://reader038.fdocuments.in/reader038/viewer/2022103015/551c0a78550346a84f8b50b4/html5/thumbnails/16.jpg)
EXPERT CAN INTRODUCE NEW CONCEPTS (ATTRIBUTES)
Pacient ... Headache Fatigue SoreThroat Appetite Flu
1 ... no no yes normal no
2 ... no yes yes low yes
3 ... yes yes no low yes
4 ... no no no normal no
... ... ... ... ... ... ...
FluSymptoms
no
yes
yes
no
Possible rule with the new attribute:
IF Temperature > Normal AND FluSymptoms = yes
THEN Flu = yes
...
![Page 17: D ESIGNING AN I NTERACTIVE T EACHING T OOL WITH ABML K NOWLEDGE R EFINEMENT L OOP enabling arguing to learn 1 Faculty of Education, University of Ljubljana,](https://reader038.fdocuments.in/reader038/viewer/2022103015/551c0a78550346a84f8b50b4/html5/thumbnails/17.jpg)
EXPERT CAN CORRECT CLASSIFICATION OF LEARNING EXAMPLE
... Headache Fatigue SoreThroat Appetite FluSymptoms Flu
... no no no normal no yes
Pacient Temperature Vaccination Coughing ...
37 normal no no ...
The question to the expert:
„What is the reason for Pacient 37 having the
flu?“
Expert corrects the classification of Pacient 37:
„Pacient 37 doesn‘t have the flu.“
no
![Page 18: D ESIGNING AN I NTERACTIVE T EACHING T OOL WITH ABML K NOWLEDGE R EFINEMENT L OOP enabling arguing to learn 1 Faculty of Education, University of Ljubljana,](https://reader038.fdocuments.in/reader038/viewer/2022103015/551c0a78550346a84f8b50b4/html5/thumbnails/18.jpg)
KNOWLEDGE ELICITATION WITH ABML
IF ... THEN ...IF ... THEN ......
ABMLargument-based machine learning
inconsistencies in labels
are detected automatically
misclassificated examples are
easily recognized and corrected
arguments
critical examplescounter examples
experts introduce
new attributes
human-understandable models
suitable for teaching
![Page 19: D ESIGNING AN I NTERACTIVE T EACHING T OOL WITH ABML K NOWLEDGE R EFINEMENT L OOP enabling arguing to learn 1 Faculty of Education, University of Ljubljana,](https://reader038.fdocuments.in/reader038/viewer/2022103015/551c0a78550346a84f8b50b4/html5/thumbnails/19.jpg)
IF ... THEN ...IF ... THEN ......
ABMLargument-based machine learning
arguments
critical examplescounter examples
How to use ABML in educational setting?
INTRODUCING STUDENTS
![Page 20: D ESIGNING AN I NTERACTIVE T EACHING T OOL WITH ABML K NOWLEDGE R EFINEMENT L OOP enabling arguing to learn 1 Faculty of Education, University of Ljubljana,](https://reader038.fdocuments.in/reader038/viewer/2022103015/551c0a78550346a84f8b50b4/html5/thumbnails/20.jpg)
THE OUTER LOOP
Step 1: Learn a hypothesis with ABML
Step 2: Find the “most critical” example (if none found, stop)
Step 3: Student explains the example
Return to step 1
![Page 21: D ESIGNING AN I NTERACTIVE T EACHING T OOL WITH ABML K NOWLEDGE R EFINEMENT L OOP enabling arguing to learn 1 Faculty of Education, University of Ljubljana,](https://reader038.fdocuments.in/reader038/viewer/2022103015/551c0a78550346a84f8b50b4/html5/thumbnails/21.jpg)
THE OUTER LOOP & THE INNER LOOP
Step 1: Learn a hypothesis with ABML
Step 2: Find the “most critical” example (if none found, stop)
Step 3: Student explains the example
Return to step 1 Step 3a: Explaining a critical example (in a natural language)
Step 3b: Adding arguments to the example
Step 3c: Discovering counter examples
Step 3d: Improving arguments with counter examples
Return to step 3c if counter example found
USING TEACHER‘S ATTRIBUTES!
![Page 22: D ESIGNING AN I NTERACTIVE T EACHING T OOL WITH ABML K NOWLEDGE R EFINEMENT L OOP enabling arguing to learn 1 Faculty of Education, University of Ljubljana,](https://reader038.fdocuments.in/reader038/viewer/2022103015/551c0a78550346a84f8b50b4/html5/thumbnails/22.jpg)
ARGUING TO LEARN
Argumentation involves elaboration, reasoning, and reflection. These activities have been shown to contribute to deeper conceptual learning (Bransford, Brown, & Cocking, 1999)
Participating in argumentation helps students learn about argumentative structures (Kuhn, 2001)
![Page 23: D ESIGNING AN I NTERACTIVE T EACHING T OOL WITH ABML K NOWLEDGE R EFINEMENT L OOP enabling arguing to learn 1 Faculty of Education, University of Ljubljana,](https://reader038.fdocuments.in/reader038/viewer/2022103015/551c0a78550346a84f8b50b4/html5/thumbnails/23.jpg)
A NEW PARADIGM
arguing to learn
with
argument-based machine learning
![Page 24: D ESIGNING AN I NTERACTIVE T EACHING T OOL WITH ABML K NOWLEDGE R EFINEMENT L OOP enabling arguing to learn 1 Faculty of Education, University of Ljubljana,](https://reader038.fdocuments.in/reader038/viewer/2022103015/551c0a78550346a84f8b50b4/html5/thumbnails/24.jpg)
Let‘s see how this works in practice
![Page 25: D ESIGNING AN I NTERACTIVE T EACHING T OOL WITH ABML K NOWLEDGE R EFINEMENT L OOP enabling arguing to learn 1 Faculty of Education, University of Ljubljana,](https://reader038.fdocuments.in/reader038/viewer/2022103015/551c0a78550346a84f8b50b4/html5/thumbnails/25.jpg)
BASIC OR ADVANCED?
„basic“
„advanced“
![Page 26: D ESIGNING AN I NTERACTIVE T EACHING T OOL WITH ABML K NOWLEDGE R EFINEMENT L OOP enabling arguing to learn 1 Faculty of Education, University of Ljubljana,](https://reader038.fdocuments.in/reader038/viewer/2022103015/551c0a78550346a84f8b50b4/html5/thumbnails/26.jpg)
LEARNING DATA SET
121 solutions of 62 different exercises
teacher labeled each solution
as „basic“ or „advanced“
learn data: 91 examples
test data: 30 examples
![Page 27: D ESIGNING AN I NTERACTIVE T EACHING T OOL WITH ABML K NOWLEDGE R EFINEMENT L OOP enabling arguing to learn 1 Faculty of Education, University of Ljubljana,](https://reader038.fdocuments.in/reader038/viewer/2022103015/551c0a78550346a84f8b50b4/html5/thumbnails/27.jpg)
KNOWLEDGE ELICITATION FROM THE TEACHER
1. relevant description language: new attributes
2. consistently labeled examples
TEACHER‘S GOALS:
![Page 28: D ESIGNING AN I NTERACTIVE T EACHING T OOL WITH ABML K NOWLEDGE R EFINEMENT L OOP enabling arguing to learn 1 Faculty of Education, University of Ljubljana,](https://reader038.fdocuments.in/reader038/viewer/2022103015/551c0a78550346a84f8b50b4/html5/thumbnails/28.jpg)
RESULTS OF KNOWLEDGE ELICITATION FROM THE TEACHER
9 iterations
9 new attributes
9 rules
only 1 out of 5 initial attributes remained
![Page 29: D ESIGNING AN I NTERACTIVE T EACHING T OOL WITH ABML K NOWLEDGE R EFINEMENT L OOP enabling arguing to learn 1 Faculty of Education, University of Ljubljana,](https://reader038.fdocuments.in/reader038/viewer/2022103015/551c0a78550346a84f8b50b4/html5/thumbnails/29.jpg)
A STUDENT-COMPUTER INTERACTIVE LEARNING SESSION
STUDENT‘S TASK
obtain rules for distinguishing „basic“ and „advanced“ solutions rules must consist of attributes in teacher‘s final model
use teacher‘s descriptive language
![Page 30: D ESIGNING AN I NTERACTIVE T EACHING T OOL WITH ABML K NOWLEDGE R EFINEMENT L OOP enabling arguing to learn 1 Faculty of Education, University of Ljubljana,](https://reader038.fdocuments.in/reader038/viewer/2022103015/551c0a78550346a84f8b50b4/html5/thumbnails/30.jpg)
A STUDENT-COMPUTER INTERACTIVE LEARNING SESSION
RECOMMENDATIONS TO THE STUDENT
use the most important features for explanations
use the smallest possible number of features in a single argument
try not to repeat the same arguments
![Page 31: D ESIGNING AN I NTERACTIVE T EACHING T OOL WITH ABML K NOWLEDGE R EFINEMENT L OOP enabling arguing to learn 1 Faculty of Education, University of Ljubljana,](https://reader038.fdocuments.in/reader038/viewer/2022103015/551c0a78550346a84f8b50b4/html5/thumbnails/31.jpg)
THE FIRST „CRITICAL“ EXAMPLE
The question to the student:
„Why is this solution advanced?“
Student‘s argument:
„Because function zip is present and the number of rows is low.“
Solution = advanced BECAUSE Zip = True AND cRows = low
![Page 32: D ESIGNING AN I NTERACTIVE T EACHING T OOL WITH ABML K NOWLEDGE R EFINEMENT L OOP enabling arguing to learn 1 Faculty of Education, University of Ljubljana,](https://reader038.fdocuments.in/reader038/viewer/2022103015/551c0a78550346a84f8b50b4/html5/thumbnails/32.jpg)
COUNTER EXAMPLE
IF Zip = True AND cRows = low THEN Solution = advanced
The rule is inconsistent with data! Student is presented with counter example:
„Compare these two solutions. Why is the second solution a basic one?“
![Page 33: D ESIGNING AN I NTERACTIVE T EACHING T OOL WITH ABML K NOWLEDGE R EFINEMENT L OOP enabling arguing to learn 1 Faculty of Education, University of Ljubljana,](https://reader038.fdocuments.in/reader038/viewer/2022103015/551c0a78550346a84f8b50b4/html5/thumbnails/33.jpg)
IMPROVING ARGUMENT
IF Zip = True AND cRows = low AND LiCom = True
THEN Solution = advanced
Student‘s extended argument:
„Because function zip is present, the number of rows is low,
and a list comprehension occurs. “
no more counter examples next iteration
![Page 34: D ESIGNING AN I NTERACTIVE T EACHING T OOL WITH ABML K NOWLEDGE R EFINEMENT L OOP enabling arguing to learn 1 Faculty of Education, University of Ljubljana,](https://reader038.fdocuments.in/reader038/viewer/2022103015/551c0a78550346a84f8b50b4/html5/thumbnails/34.jpg)
AT THE END OF THE INTERACTIVE SESSION
5 iterations
half an hour
90% accuracy on (previously unseen) testing data
several suggestions of new descriptive features
![Page 35: D ESIGNING AN I NTERACTIVE T EACHING T OOL WITH ABML K NOWLEDGE R EFINEMENT L OOP enabling arguing to learn 1 Faculty of Education, University of Ljubljana,](https://reader038.fdocuments.in/reader038/viewer/2022103015/551c0a78550346a84f8b50b4/html5/thumbnails/35.jpg)
ASSESMENT
Results: experiment with 7 students
7.1 iterations
87.1% classification accuracy of obtained rule model
86.7% correctly „manually“ classified (previously unseen) examples
very positive qualitative feedback from the students
![Page 36: D ESIGNING AN I NTERACTIVE T EACHING T OOL WITH ABML K NOWLEDGE R EFINEMENT L OOP enabling arguing to learn 1 Faculty of Education, University of Ljubljana,](https://reader038.fdocuments.in/reader038/viewer/2022103015/551c0a78550346a84f8b50b4/html5/thumbnails/36.jpg)
CONCLUSIONS
New paradigm:
arguing to learn with argument-based machine learning
Future work:
• applications in several domains
• assessing argument‘s quality for improved immediate feedback
• goal-oriented extension (see our ITS 2012 paper)
![Page 37: D ESIGNING AN I NTERACTIVE T EACHING T OOL WITH ABML K NOWLEDGE R EFINEMENT L OOP enabling arguing to learn 1 Faculty of Education, University of Ljubljana,](https://reader038.fdocuments.in/reader038/viewer/2022103015/551c0a78550346a84f8b50b4/html5/thumbnails/37.jpg)
QUESTIONS & DISCUSSION
Thank you!
slides: ailab.si/matej
enabling arguing to learn