Argument structures for assessing life-long competence in

semantically rich-domains

Imran A. Zualkernan

American University of Sharjah

izualkernan@aus.edu

Overview

• Defining Competence

• Argument Structures

• Assessing competence based on these structures

• Conclusion

Competence

• Competence is defined as the “ability of an agent to generate specific types of arguments”

• This is especially true of professional fields such as Law, Medicine and Engineering

• In some sense, these argument structures serve as the “grammar” for acceptable behavior at various levels of competence.

Argument Fields

• Expertise in semantically rich domains can be characterized according to five different types of argument fields.

Arguments

• Each argument is tied to a goal or an objective. For example, diagnosis may be considered as a goal for a physician.

• Data represents the information or knowledge being considered in making an argument. The presenting symptoms, for example, constitute data for a physician engaged in the task of diagnosis.

• Claims represent end-statements, actions or products of problem solving. For example, diagnosis that a patient has pneumonia represents a valid claim for a physician.

Arguments

• A warrant indicates the reason that led from data to the claim. For example, existence of a particular pattern of symptoms such as blue lips and wheezing may serve as a warrant that led the physician to a specific diagnosis of pneumonia.

• Warrants themselves have a Backing that indicates why they are valid. For example, a pattern of symptoms may be a “valid” warrant because of an empirical correlation between particular patterns of symptoms and the diagnosis.

Argument FieldsBacking Philosophical Basis

Type – I Mathematical soundness, consistency and completeness

Leibnitzian

Type – II Agreement or consensus among practitioners

Lockean

Type- III Agreement among complimentary representations of a problem

Kantian

Type – IV Conflicts between alternative representation of a problem

Hegalian

Type – V Evaluation of alternative ways of knowing

Singerian

Competence in Professional Fields • Software engineering body of knowledge formalizes what a software

engineer needs to know. • This body of knowledge is a broad specification of competence required of

a software engineer. • The SWEBOK recognized multiple knowledge areas such as software

requirement, design, construction, testing, maintenance, configuration managements, engineering management, engineering process, tools and methods and quality.

• The competence of a software engineer can be judged based on assessing their knowledge of each of these areas.

• Each area is divided into sub-areas. For example, software design is divided into software design fundamentals, key issues in software design, software structures and architecture, software design quality analysis and evaluation, software design notations, and software design strategies and methods.

• Each area, in turn is sub-divided into further categories.

Software Patterns – one sub-area• Originally motivated from Alexander’s original thinking on

design patterns, software design patterns are abstractions or blue-print allowing a software engineer to construct high-level designs

• While there is a base-set of design patterns that every budding software engineer is expected to be familiar with, new patterns are continually emerging.

• For example, design patterns for the J2EE technology XML-processing, XML-schema design, web-languages and service-oriented architectures have been proposed.

• Special conferences dedicated to design patterns continually create new design patterns every year.

Goals• The first step in enumerating arguments in a

particular domain is to determine the goals.

• Two primary goals are associated with using design patterns in a professional setting; 1) recognizing an appropriate design pattern for a

particular design context or how to choose a design pattern

2) implementing a pattern in a particular design context or how to use a design pattern

Type I Argument

Type II Argument

Type III Argument

Type IV Argument

Type V Argument

• A Type V argument for design patterns would question the validity of design patterns as a viable tool for design practice.

Judging Competence

• If competence is characterized by an ability to generate different types of arguments, then judging competence consists of determining the levels and types of arguments a particular individual can generate.

• This is perhaps akin to judging whether a speaker is able to utter certain classes of sentences in a language.

• Hence, competence is tied to the “fitness” of an individual to generate particular types of arguments.

Semantic Fit

• Semantic fit is the familiar notion of fit that is tied to the ability of an agent to generate the “appropriate” action.

• That is, given certain information in the environment (e.g., presenting symptoms as data), arrive at an appropriate action (e.g., reasonable claim for a diagnosis).

Semantic Fit

• For example, given the goal of finding the square root the number four, if one arrives at the correct answer of two, one has demonstrated a competence of Type I argument for square-root functions.

• This is a Type I argument because the backing is based on mathematical truths.

Structural Fit

• Structural fit is tied to the ability of an agent to recognize a mismatch between their cognitive constraints, styles and abilities and the information presented in a specific context while trying to attain a goal.

• For example, given the goal of finding the square root of -1 in non-complex arithmetic should lead to the correct action of indicating that it is not possible to do so because -1 does not represent “valid” information for this goal.

• This behavior shows that the agent is structurally fit to generate Type I arguments for a square-root function.

Templates for Generating Assessments

Generating Assessments for Semantic Fit

Bloom’s Level Assessments Generation technique

Knowledge How many options are there for selecting creational patterns?

Enumerate options in claim template.

List conditions under which one would typically select a SINGLETON pattern.

Fix claim = SINGLETON & enumerate data template

Comprehension Compare the conditions under which one would use a BUILDER or a PROTOYPE pattern.

Fix claim to be a pair = (BUILDER, PROTOTYPE) & enumerate data.

Application Which pattern can be used for an application framework that can present multiple documents to a user?

Fix data (using intent) and ask to generate a claim.

Analysis Which pattern would you use if the ‘modifiability’ of a graphic tool is the primary design concern?

Fix warrant – exploit conflict between FACTORY METHOD, ABSTRACT FACTORY and PROTOTYPE pattern and backing Type IV. .

Synthesis Which variant of the FACTORY pattern is appropriate if classes require an implementation?

Fix claim – FACTORY METHOD and exploit variants.

Evaluation When would you choose a FACTORY METHOD pattern over a PROTOTYPE?

Fix claim (FACTORY METHOD) and Backing to Type IV.

Generating Assessments for Structural Fit

Bloom’s Level Assessments Generation technique

Knowledge Is PROXY a creational pattern? Enumerate illegal options in claim template.

Comprehension Compare the conditions under which one would use an ITERATOR or a PROTOYPE pattern.

Fix claim to be an un-reasonable pair = (ITERATOR, PROTOTYPE) & enumerate data.

Application Which pattern can be used to access to elements of an aggregate object sequentially.

Fix data (using intent) and ask to generate an illegal claim.

Analysis Which pattern would you use if the ‘modifiability’ of a collection class library is the primary design concern?

Use an unreasonable claim pattern and use backing Type IV.

Synthesis Which variant of the COMMAND pattern is appropriate if classes require an implementation?

Fix claim to be unreasonable and exploit variants.

Evaluation When would you choose a FACTORY METHOD pattern over a PROTOTYPE

Fix claim to be unreasonable and Backing to Type IV.

Proposed System

Conclusion

• Argument structures are a natural way to organize and think about competence assessment for life-long learning in semantically rich domains.

• Arguments allow one to no only model competence in a domain in structured way but can be combined with other methodologies such as Bloom to create authentic assessments in a structured manner.

• An effort is underway to formalize an argument-based structure in conjunction with the QTI 2.0 specification.