Defining a pragmatic QTI profile

2

Introduction

• Goals of a CETIS QTI profiling workgroup• Assessment system infrastructure• The role QTI plays in the infrastructure• Ways and methods to determine a profile• Means of testing a profile• The choice

3

Goals of a CETIS QTI profiling WG

In order of priority:1. Determine production profile(s) for UK education (HE? FE +

HE? All ?)2. Enabling IMS QTI 2.1 public release by producing a testable

profile3. Help determine a new QTI profile for IMS Common Cartridge

4

Assessment system infrastructure

5

Assessment system infrastructure

Institution owns everything

Easy coordination of all interoperability points (in theory)

Few resources to make that coordination happen (in practice)

Very difficult to meet all subject communities' needs

6

Assessment system infrastructure

Third party owns everything but content

Easy coordination of all interoperability points

Enough resources to make that coordination happen

Enough resources to meet all subject communities' needs

Politically impossible / unlikely

7

Assessment system infrastructure

Compromise: Subject centre bank Institutional learning

system 3d party delivery

Doable coordination of most interoperability points

Good spread of resource load

Still a few potential bottlenecks

8

Assessment system infrastructure

Therefore, for greatest interoperability: Inverse relation between the complexity of the data

exchanged, and the variation in applications that process that data

Hand responsibility for component to party with greatest interest

For profiling this means Subjects set requirements for rich profile (assuming

compromise or centralised infrastructure) Else: lowest common technical denominator profile

9

The role QTI plays in the infrastructure

QTI as exchange format across the system + Consistent semantics - Difficult profile coordination problem between systems and

over time

QTI as intermediary format between systems + Supports legacy systems now - Semantic roadblocks (unacceptable degradation between

authoring and use)

For profiling, this means: Intermediary format suits lowest common technical

denominator profile Exchange format suits rich subject profiles

10

Ways and methods to determine a profile

Community requirements led + surest means of achieving fitness for purpose + surest means of getting uptake - can lead to profiles that are technically very difficult to realise - size of group v. consensus building delicate

Implementation led + process of determining a profile is quick and easy + near instant implementation ± accurately reflects current practice - likely to be unfit for purpose (danger of balkanisation)

For profiling, this means Community led is better, but time-consuming and expensive Implementation led is quick, cheap but may not meet needs

11

Means of testing a profile

Formal testing + reliable - may well not be valid - v. expensive, continuously (particularly for content)

Self-service testing ± good enough reliability and validity + cheapish

Reference implementation + reliable + valid + cheapish - may cause political ructions

12

The choice

Compromise infrastructure

QTI as an exchange format

Community led profiling Reference

implementation Rich, subject specific

profiles IMS QTI test profile

(mostly) institutionally owned infrastructure

QTI as an intermediary format

Implementation led profiling

Self-service testing Small, lowest common

denominator profiles IMS CC profile

13

Thank you!

Wilbert Kraan w.g.kraan@ovod.net