Post on 16-Dec-2015
July 10, 2005 Educational Data Mining Workshop 20th AAAI-05 Conference
ITS Data Collection ITS Data Collection FrameworkFramework
Capturing data based on agent Capturing data based on agent communication standardcommunication standard
Olga Medvedeva, Center for Pathology Informatics,
University of Pittsburgh
July 10, 2005 Educational Data Mining Workshop 20th AAAI-05 Conference
OutlineOutline
• Need for communication standard for Intelligent Tutoring Systems
• Existing standard for multi-agent communication
• Implementation in SlideTutor – Communication protocol– Data collection– Database query tool– Lessons learned
• Comparison with recent standardization effort• Advantages of using the the existing standard
July 10, 2005 Educational Data Mining Workshop 20th AAAI-05 Conference
Intelligent Learning Environment Intelligent Learning Environment Common BaseCommon Base
• Underlying theory– Cognitive tutors (Anderson et al. 1995)– Adaptive hypermedia (Brusilovsky et al.
1996)– Constraint-based (Mitrovic et al. 2001)
• Modules– Expert, Student, Interface, Pedagogic
• “Single-purpose” development approach
July 10, 2005 Educational Data Mining Workshop 20th AAAI-05 Conference
Keystone – communication Keystone – communication standardstandard
• Previous efforts:– Inter-tutor communication (Ritter, Koedinger 1996;
Brusilovsky et al. 1997) one-to-one translators, strict channel, no real protocol
– Shared resources (Koedinger et al. 1999) – limited use: lack of standard
– DORMIN protocol (developed at CMU) – used in commercial product
• Our approach– Multi-agent technology– Use existing inter-agent communication standard
July 10, 2005 Educational Data Mining Workshop 20th AAAI-05 Conference
Foundation for Intelligent Physical Foundation for Intelligent Physical Agents (FIPA)Agents (FIPA)
FIPA (www.fipa.org) - collection of standards forinter-agent communication:• Agent Management System – manages an agent
life-cycle, maintains a registry with unique Agent Identifier (AID)
• Transport – describes message exchange protocol: transport type and specific address for an agent
• Agent Communication Language (ACL) –communication specifications
FIPA was officially accepted by the IEEE as one of its standards committees on 8 June 2005
July 10, 2005 Educational Data Mining Workshop 20th AAAI-05 Conference
FIPA Design PrincipalsFIPA Design Principals
• Forms abstract basis for concrete architecture
• Sets minimum required elements
• Permits introduction of new elements
• Permits arbitrary content language, uses Abstract Content Representation (ACR) for ACL as key-value pairs
Envelope:Sender (locator) Receiver (locator)Timestamp
Message (ACL):
Sender (AID)Receiver (AID)Performative (String)Content: ( ACR)
Reply-to(Message ID)
Message (ACL):
Sender (AID)Receiver (AID)Performative (String)Content: ( ACR)
Reply-to (Message ID)
July 10, 2005 Educational Data Mining Workshop 20th AAAI-05 Conference
FIPA ACL Message StructureFIPA ACL Message Structure
:sender – identity of the sender
:receiver – identity of the recipient
:content – the object of the action
:performative – the type of the communicative act
Optional:
:reply-with :replay-to :in-replay-to :replay-by– replay constraints
:language – encoding schema of the content of the message
:encoding – encoding identifier
:ontology – is used to give a meaning to symbols/concepts in the content
:protocol – gives additional context for the interpretation of the message
:conversation-id – identifies the ongoing sequence of communicative act, manages the conversation strategies
July 10, 2005 Educational Data Mining Workshop 20th AAAI-05 Conference
FIPA PerformativesFIPA Performatives
• Accept-proposal• Agree• Cancel• Call-for-proposal• Confirm• Disconfirm• Failure• Inform• Inform-if• Inform-ref• Not-understood
• Propagate• Propose• Proxy• Query-if• Query-ref• Refuse• Reject-proposal• Request• Request-when• Request-whenever• Subscribe
July 10, 2005 Educational Data Mining Workshop 20th AAAI-05 Conference
FIPA Implementation in JavaFIPA Implementation in Java
• Java Agent Services (JAS) (www.jcp.org) defines a set of objects and service interfaces to support the deployment and operation of the agents.
• Contains interfaces for building messages, directory services and a factory for message transfer services.
• JAS is a base for multi-agent communication in our system
July 10, 2005 Educational Data Mining Workshop 20th AAAI-05 Conference
SlideTutor ArchitectureSlideTutor Architecture http://slidetutor.upmc.eduhttp://slidetutor.upmc.edu
SlideTutor - an agent-based model tracing ITS for visual classification problem solving in surgical pathology
Tutor Servlet
Internet
Jess Protégé-2000
Dynamic Solution Graph
ProductionRules
DomainOntology
Pedagogic Production Rules
StudentModeling System
ExpertModule
Pedagogic Model
ProbabilisticStudent Model
WEB SERVER
2 Student Interfaces& 1 Authoring Interface
StudentFiles
Dow
nlo ad
Reasoning GUI-Tutor Communications
View
er GU
I-IDS
Reasoning G
UI-Tutor
Login Servlet
Protocol CollectionFilter
Client GUI Download withJava WebStart
Slide
Pedagogic Ontologies
InternetViewerServlet
ImagePump
Application
Whole-Slide
Images
IMA
GE
DE
LIV
ER
Y SY
STE
M
Project DB
Case DB
PedagogicOntology
Java WebstartDownload Manager
July 10, 2005 Educational Data Mining Workshop 20th AAAI-05 Conference
Generic Representation of Generic Representation of Problem-Solving SpaceProblem-Solving Space
Pedagogic Layer
Pedagogic TaskStructure
PedagogicTask
DermatologyKnowledge Base
DomainModel
VisualClassification
Task Structure
DomainTask
PedagogicKnowledge Base
PedagogicModel
Case Database
Interface
Expert Model
Student Model
StudentModelState
StudentModelStateStudent Model
Data
SlideRepresentation
Case Data
SlideRepresentation
Case Data
SlideRepresentation
Case Data
Student
Dynamic Solution Graph
Pedagogic Model
DomainBehaviorRefiner
ProblemSolvingMethods
PedagogicBehaviorRefiner
ProblemSolvingMethods
July 10, 2005 Educational Data Mining Workshop 20th AAAI-05 Conference
Collected DataCollected Data
• InterfaceEvent – low-level human-computer interactions
• ClientEvent – collection of InterfaceEvents that represents an elementary subgoal, understood by tutor
• TutorResponse – system response to a ClientEvent
Protocol Agent
InterfaceEvent(s)
ClientEvent
TutorResponse
ProblemEventstart problem AP_77
ProblemEventfinish problem AP_77
Tutor AgentClient Agent
ProjectDatabase
LogFiles
July 10, 2005 Educational Data Mining Workshop 20th AAAI-05 Conference
Message ExampleMessage Example
• Envelope indicates the locators of client and protocol agents
• 4 required key-value pairs for a message
• Performative defines a type of communicative act
• List of preceding InterfaceEvent Ids:– click on Finding button– Click on image– Selecting 3 times down a tree of
findings
Envelope:Sender: Client_1Receiver: PROTOCOLTimeStamp = 1114444377783Message: Sender: Concept2 Receiver: PROTOCOL Performative: X-Created In-reply-with: 1114444378242 Content: Type = Finding Label = blister Id = Concept2 ObjectDescription = Finding.blister.Concept2 Parent = null Input: name = text value = blister name = y value = 11808 name = x value = 38048 name = z value = 0.03 InterfaceEventIDS = [1114444374333, 1114444375546, 1114444376304,
1114444376798, 1114444377444]
ClientEvent
July 10, 2005 Educational Data Mining Workshop 20th AAAI-05 Conference
Message in DepthMessage in Depth
• Widget object (agent) description parameters– Type (“Button”, “Finding”)– Label (“Next”, “Blister”)– Id – unique within a session– ObjectDescription – combination of
Type+Label+Id (“Finding.blister.Concept2)– Parent – list of all parent ObjectDescriptions
for hierarchical structures• Common for ITS user action triplet
– Action = Performative– Selection = ObjectDescription+Parent– Input = list form Content Input
• Message encoded in XML is easy to translate into other languages (RDF, KIF, SL, etc.)
Envelope:Sender: Client_1Receiver: PROTOCOLTimeStamp = 1114444377783Message: Sender: Concept2 Receiver: PROTOCOL Performative: X-Created In-reply-with: 1114444378242 Content: Type = Finding Label = blister Id = Concept2 ObjectDescription = Finding.blister.Concept2 Parent = null Input: name = text value = blister name = y value = 11808 name = x value = 38048 name = z value = 0.03 InterfaceEventIDS = [1114444374333, 1114444375546, 1114444376304,
1114444376798, 1114444377444]
ClientEvent
July 10, 2005 Educational Data Mining Workshop 20th AAAI-05 Conference
TutorResponse ExampleTutorResponse Example
• Student performance data– Performative: FAILURE – user
took incorrect step– ErrorCode = 15 – user incorrectly
located existing finding– Input: - contains a description of
an error message to be presented to user
• Tutor performance data– Best possible next step – action
expert model would take in this problem state
Envelope Sender: TutorEngine0Receiver: PROTOCOLTimeStamp: 1114444379378Message: Sender: TutorEngine0 Receiver: PROTOCOL Performative: FAILURE Conversation_ID: 1114444378242 Content: ErrorCode = 15 NextStepType = Evidence NextStepLabel = blister NextStepID = Concept2 NextStepParent = null NextStepAction = DELETE Input: name = Messages value = "[TEXT:There is BLISTER present, but not where you have pointed in the image. See if you can find where. POINTERS:[PointTo:Concept2 IsPermanent:false Method:setFlash Args:[true]]]“ name= TutorAction value = "PointTo:Concept2 IsPermanent:false Method:setBackgroundColor Args:[RED]"
July 10, 2005 Educational Data Mining Workshop 20th AAAI-05 Conference
Database SchemaDatabase Schema
• High-level static tables similar to Mostow et al. 2002 contains Experiment, CaseList, Student, etc.• Low-level tables for captured events, including start/end of problem and session closely follow the FIPA standard, generic with any number of event parameters stored in corresponding Input tables
July 10, 2005 Educational Data Mining Workshop 20th AAAI-05 Conference
Web-Based Protocol Query ToolWeb-Based Protocol Query Tool
• Allows the user to obtain data sets specific to a wide range of constraints
• Outputs to HTML file that can be transferred to Excel
• Query can be saved and viewed in SQL
• Interface, Client and Tutor events data can be joined in different ways
July 10, 2005 Educational Data Mining Workshop 20th AAAI-05 Conference
Query Tool Results for Query Tool Results for Identifying BlisterIdentifying Blister
InterfaceEvents
ClientEvents
TutorResponses
July 10, 2005 Educational Data Mining Workshop 20th AAAI-05 Conference
Advantages of Event-Based Data Advantages of Event-Based Data RepresentationRepresentation
• Usability Perspective: InterfaceEvents linked to ClientEvents (Saadawi et al. 2005)– How many actions were performed– How much time was required to achieve a particular subgoal, such
as identification of Blister– How many InterfaceEvents were unrelated to any ClientEvent
• Student Performance over time: ClientEvents linked to TutorResponses– Number of hints requested– Depth of hints– Error frequency and distribution
• Tutor Performance: NextStep fields in TutorResponses– Compare next student actions to those predicted by tutor
July 10, 2005 Educational Data Mining Workshop 20th AAAI-05 Conference
SlideTutor Data Sharing SlideTutor Data Sharing LimitationLimitation
• This paper and presentation have been approved by Institutional Review Board (IRB)
• Researcher needs to sign a Limited Use Agreement
• There might be one agreement with consortiums
July 10, 2005 Educational Data Mining Workshop 20th AAAI-05 Conference
Lessons LearnedLessons Learned
For the past year our data collection framework was used in 4 small HCI studies and one large experiment with a total of 50 students.
• Keep data clean: ended up maintaining ‘raw’ and ‘clean’ copies of database
• Granularity of captured data: capturing of detailed data slows the system
• Separate database for assessment: no explicit mapping of performance on tests and in the tutoring system
July 10, 2005 Educational Data Mining Workshop 20th AAAI-05 Conference
Data Collection Framework Data Collection Framework AdvantagesAdvantages
• Advantages of relational database (Mostow et al. 2002)– Eases the analysis of the enormous volume of complex data
• Generic framework that might be adapted to other model-tracing ITS– Adapted in the extension of SlideTutor – ReportTutor that teaches
how to write the pathology reports• Flexibility of FIPA-based communication protocol
– Flexibility to describe interaction events– Extendable set of performatives– Multiple messages in one envelope, unrestricted number of input
parameters– Potential to reference ontologies within the message– Can be easily reused in the Data Shop
July 10, 2005 Educational Data Mining Workshop 20th AAAI-05 Conference
Data Shop Project, Data Shop Project, Pittsburgh Science of Learning Center (Pittsburgh Science of Learning Center (http://www.http://www.
learnlablearnlab.org.org ) )
• Logging and Analysis: Tools and reports to aid PSLC researchers and course developers– Log the activities of the experiments to a database– Provide the reports and queries on that
experiments
• Goal: Standardize the messaging format among tools, tutoring translators and agents– Message types: tool_message, tutor_message,
curriculum_message, message
• Data Shop Tutor Logging v3 released in June 2005
July 10, 2005 Educational Data Mining Workshop 20th AAAI-05 Conference
Data Shop Tool Message and SlideTutor Data Shop Tool Message and SlideTutor Interface/Client eventsInterface/Client events
tool_message attempt_id
event_descriptor (0+) event_id
selection (0+) id type
input (0+) id
action (0+) id
meta (0 or 1) user_id session_id time time_zone
problem_name (0 or 1)
ui_event id
(1+)
semantic_event id semantic_event_id name trigger
step (0+) probability
July 10, 2005 Educational Data Mining Workshop 20th AAAI-05 Conference
Data Shop Tutor Message Data Shop Tutor Message and SlideTutor TutorResponseand SlideTutor TutorResponse
action_evaluation (0+) current_hint_number total_hints_available classification
tutor_advice (0+)
skill (0+) probability
production (0+)
event_descriptor (0+) event_id
selection (0+) id type
input (0+) id
action (0+) id
name (1)
value (1)
custom_field (0+)
meta (0 or 1) user_id session_id time time_zone
problem_name (0 or 1)
ui_event id
semantic_event id semantic_event_id name trigger
step (0+) probability
name (1)
value (1)
step_interpretation (0+)
July 10, 2005 Educational Data Mining Workshop 20th AAAI-05 Conference
FIPA AdvantagesFIPA Advantages
• FIPA as a information exchange underlying standard– Develop a set of performatives – a controlled vocabulary for
ITS communication– Create sharable ontologies for domain knowledge, hint
content, error categories and use ‘:ontology’ FIPA parameter to give a meaning to the message content
– Use ‘:protocol’ parameter to identify the translator and to preserve the internal component structure
• Syntactically aligned systems – Ease meta-analysis for tutors with the identical performatives – Reuse data for simulations– Shared services for real-time interoperability
• Identifying particular help-seeking behavior• Calculating knowledge tracing probabilities
July 10, 2005 Educational Data Mining Workshop 20th AAAI-05 Conference
AcknowledgementsAcknowledgements
Grants:• National Library of Medicine• National Cancer Institute
People:• Rebecca Crowley• Girish Chavan• Eugene Tseytlin• Elizabeth Legowski• Katsura Fujita• Maria Bond
July 10, 2005 Educational Data Mining Workshop 20th AAAI-05 Conference
ReferencesReferences
• Anderson JR, Corbett AT, Koedinger KR, and Pelletier R. Cognitive Tutors: Lessons learned. Journal of the Learning Sciences 4(2): 167-207, 1995
• Brusilovsky, P., Kommers, P. & Streitz, N. (Eds.) (1996) Multimedia, Hypermedia, and Virtual Reality (LNCS Vol. 1077). Berlin: Springer-Verlag, 1996
• Mitrovic A, Mayo M, Suraweera, P and Martin, B. Constraint-Based Tutors: A Success Story . In Monostori, L. and Vancza, J. (Eds). Proceedings of the 14th International Conference on Industrial & Engineering Applications of Artificial Intelligence and Expert Systems, Budapest, Hungary, Springer, pp. 931-940, 2001
• Ritter, S. and Koedinger, K. R. (1996). An architecture for plug-in tutor agents. Journal of Artificial Intelligence in Education, 7, 315-347
• Brusilovsky, P., Ritter, S., & Schwarz, E. Distributed intelligent tutoring on the Web, Proceedings of AIEDâ97, the Eighth World Conference on Artificial Intelligence in Education. 1997
• Koedinger KR, Suthers DD, & Forbus KD. Component-based construction of a science learning space: A model and feasibility demonstration. International Journal of Artificial Intelligence in Education: 10, 392-31, 1999
• Mostow J, Beck J, Chalasani R, Cuneo A, and Jia P. Viewing and Analyzing Multimodal Human-computer Tutorial Dialogue: A Database Approach. Proceedings of the ITS 2002 Workshop on Empirical Methods for Tutorial Dialogue Systems, 75-84
• Saadawi G, Legowski E, Medvedeva O, Chavan G, and Crowley RS. A method for automated detection of usability problems from client user interface events. Accepted to Proceedings of the American Medical Informatics Association Symposium 2005