A Practical Method for Courseware Evaluation

Aldo de Moor

CommunitySense

PragWeb 2007

The Pragmatic Web of what?

Communal effective use of

� Information

� Functionalities

Courseware and the Pragmatic Web

� Pragmatic Web = context� How to evaluate web functionalities in their context

of use?

� Courseware provides an interesting domain� technological environments consisting of multiple

functionality components, together offering a complete system of info/comm services required for supporting course needs

� Functionality evaluation needed� Too much (costly) functionality

� Gaps between required and available functionality

� Conflicting functionality requirements

Functionality

� Functionality� A set of functions and their specified properties that satisfy

stated or implied needs

� Levels of granularity

� Systems

� Courseware environments

� Tools

� Blackboard

� Modules

� Announcements

� Functions

� Post announcement

A context model of courseware evaluation

A practical courseware evaluation method

� Portfolio methods� Scores weighed by infrastructure/architecture of organization.

� Bedell’s method for ICT investment selection � Functionalities scored on both effectiveness and importance for the

activities to be supported.

� Practical method� Simplification of Bedell

� No higher-order analyses

� Actors (users in their roles) provide, interpret and decideupon scores

� Context: courseware evaluation:

� Actors: students, software manager

� Tool system level: module

� Two questions� How well are course activities supported by various functionality

components?

� To what extent are the functionality modules used?

Scores

� Elements� I(a) = importance of activity

� E.g. I(Information Collection) = 9

� I(f,a) = importance of functionality f in supporting

activity a

� E.g. I(Virtual Chat, Information Collection) = 4

� Q(f,a) = quality of functionality in supporting an

activity

� E.g. Q(File Transfer, Submission of Results) = 8

Activity and functionality scores

� Activity scores� Σ I(fi,a) * Q(fi,a), for all functionalities 1..i.

� Usefulness of the combined technologies for a

particular activity

� Relevant for technology users (lectures, students)

� Functionality scores� Σ I(aj) * I(f,aj) * Q(f,aj), for all activities 1..j

� Usefulness of a particular functionality component

for the combined activities

� Relevant for technology developers/maintainers

Experiment: evaluating group assignment functionality

� Two courseware tools: Blackboard, CourseFlow

� Goal: making group assignments

� Four activities, 11 functionality modules

� Actors: 2nd year Information Management students,

software manager� 2002: 62 students, 16 groups

� 2003: 46 students, 12 groups

� Questions� Quality of tools for various group assignment activities?

� Usefulness of various functionality modules?

Activity scores 2002/2003

Avg. Activity Scores in 2002 and 2003

0

50

100

150

200

250

300

350

400

450

Info coll. Discuss. Subm. Feedback

Activity

Avg. score

2002

2003

Functionality scores 2002/2003

Avg. Functionality Scores in 2002 and 2003

0200

400600

8001000

12001400

send

e-m

ail

disc

ussi

on b

oard

virtu

al c

hat

stud

ent r

oste

r

disc

ussi

on b

oard

(gro

up)

virtu

al c

hat (

grou

p)

file

trans

fer (

grou

p)

anno

unce

men

ts (i

nfo)

cour

se in

form

atio

n (in

fo)

cour

se d

ocum

ents

(inf

o)

assi

gnm

ents

(inf

o)

Functionality

Avg. score

2002

2003

Activity Scores BB/CF (2003)

Avg. Activity Scores Blackboard and CourseFlow (2003)

0

50

100

150

200

250

300

350

400

450

Info coll. Discuss. Subm. Feedback

Activity

Avg. score

Blackboard

CourseFlow

Functionality Scores BB/CF (2003)

Avg. Functionality Scores Blackboard and CourseFlow

(2003)

0

200

400

600

800

1000

1200

1400

1600

send

e-m

ail

disc

ussi

on b

oard

virtu

al c

hat

stud

ent r

oste

r

disc

ussi

on b

oard

(gro

up)

virtu

al c

hat (

grou

p)

file

trans

fer (

grou

p)

anno

unce

men

ts (i

nfo)

cour

se in

form

atio

n (in

fo)

cour

se d

ocum

ents

(inf

o)

assi

gnm

ents

(inf

o)

Functionality

Avg. score

Blackboard

CourseFlow

Conclusion

� Practical method, shown to be useful for initial

courseware functionality selection

� Context-bound, in toto, ex-post evaluation

� Simple measures useful for quickscan and

discussion purposes

� Open source can perform just as well

� “E-learning organization”: continuous evolution /

evaluation needed