Christos Katsanos* | ckatsanos@eap.grNikolaos Tselios# | nitse@ece.upatras.grMichalis Xenos* | xenos@eap.gr

Perceived usability evaluation of Learning Management Systems: a first step towards standardization of the System Usability Scale in Greek

PCI 2012 | Piraeus, Greece | 5-7 Oct 2012

*Hellenic Open University# Educational Science and Early Childhood Education, University of Patras

Perceived usability evaluation of educational software

Perceived usability is an important requirement of educational software, affecting greatly student’s learning effectiveness and overall learning experience (Kostaras & Xenos 2011; Tselios et al. 2008)

Numerous usability evaluation questionnaires designed to assess the perceived usability of a product and/or software are available at the moment (Brooke 1996; Lewis 1991, 1992, 1995; Tullis & Albert, 2008; Tullis & Stenson 2004)

2

Surprisingly, rather few studies report usage of questionnaires to assess perceived usability of educational software

Perceived usability evaluation of software: questionnaires

3

QUIS Questionnaire for User Interface Satisfaction Chin et al, 1988

27 questions

PUEU Perceived Usefulness and Ease of Use Davis, 1989 12 questions

NAU Nielsen's Attributes of Usability Nielsen, 1993 5 attributes

NHE Nielsen's Heuristic Evaluation Nielsen, 1993 10 heuristics

CSUQ Computer System Usability Questionnaire Lewis, 1995 19 questions

ASQ After Scenario Questionnaire Lewis, 1995 3 questions

PHUE Practical Heuristics for Usability Evaluation Perlman, 1997

13 heuristics

PUTQ Purdue Usability Testing Questionnaire Lin et al, 1997 100 questions

USE USE Questionnaire Lund, 2001 30 questions

SUS System Usability Scale Brooke, 1996).

10 questions

System Usability Scale (SUS)

The most widely adopted for perceived usability evaluation of software/products (Tullis & Albert, 2008; Sauro 2012)

Why? Quick and dirty (10 questions) Standardized (valid, reliable) Summarized score (0 to 100) Reliable even with small

sample (10~12 users) SUS norms available

4(Brooke, 1996)

Research Goals

Examine the applicability of the SUS method in the context of the usability evaluation of Learning Management Systems (LMSs)

Assess the validity and reliability of a Greek version of the SUS questionnaire Non-native English speakers can confront problems in

understanding SUS compared to native speakers (Finstad 2006)

5

Methodology 3 studies

280 university students LMS: Moodle Courses on ICT & Education Both blended and distance education Various study times (sem. start, mid, end)

6

SUS Between-subjects: Greek (N=221) vs English (N=59) versions Based on self-rated knowledge of English Greek version by 2 usability experts, native speakers

Perceived usability 7-point rating [worst-imaginable, best-imaginable]

Demographic questions (e.g. age, ICT skills, Moodle usage)

Studies Overview

7

Study Time N Gender Mean Age

Delivery method

1 Sem. End 45 45 Females 21.04 Blended

2 Sem. Start 191187 Females,4 Males

20.45 Blended

3 Sem. Mid 4415 Females,29 Males

37.20 Elearning

ALL {Sem. Start, Mid, End} 280247 Females,33 Males

23.18Blended (2)Elearning (1)

SUS English and Greek Version

Item Greek Version of SUS English Version of SUS

Q1 Νομίζω ότι θα ήθελα να χρησιμοποιώ αυτό το σύστημα Moodle του μαθήματος συχνά.

I think that I would like to use this lesson's Moodle system frequently.

Q2 Βρήκα αυτό το σύστημα Moodle του μαθήματος αδικαιολόγητα περίπλοκο.

I found the lesson's Moodle system unnecessarily complex.

Q3 Σκέφτηκα ότι αυτό το σύστημα Moodle του μαθήματος ήταν εύκολο στη χρήση. I thought the lesson's Moodle system was easy to use.

Q4Νομίζω ότι θα χρειαστώ βοήθεια από κάποιον τεχνικό για να είμαι σε θέση να χρησιμοποιήσω αυτό το σύστημα Moodle του μαθήματος.

I think that I would need the support of a technical person to be able to use this lesson's Moodle system.

Q5 Βρήκα τις διάφορες λειτουργίες σε αυτό το σύστημα Moodle του μαθήματος καλά ολοκληρωμένες.

I found the various functions in this lesson's Moodle system were well integrated.

Q6 Σκέφτηκα ότι υπήρχε μεγάλη ασυνέπεια σε αυτό το σύστημα Moodle του μαθήματος.

I thought there was too much inconsistency in this lesson's Moodle system.

Q7Φαντάζομαι ότι οι περισσότεροι άνθρωποι θα μάθουν να χρησιμοποιούν αυτό το σύστημα Moodle του μαθήματος πολύ γρήγορα.

I would imagine that most people would learn to use this lesson's Moodle system very quickly.

Q8 Βρήκα αυτό το σύστημα Moodle του μαθήματος πολύ περίπλοκο/δύσκολο στη χρήση.

I found the lesson's Moodle system very cumbersome to use.

Q9 Ένιωσα πολύ σίγουρος/η χρησιμοποιώντας αυτό το σύστημα Moodle του μαθήματος. I felt very confident using the lesson's Moodle system.

Q10Χρειάστηκε να μάθω πολλά πράγματα πριν να μπορέσω να ξεκινήσω με αυτό το σύστημα Moodle του μαθήματος.

I needed to learn a lot of things before I could get going with this lesson's Moodle system.

Moodle-based course on ICT & Education (screenshot)

9

Results (1/4) – Overview

10

StudyStudents

(N)

Mean SUS Score

(0-100)

Mean Perceived Usability Rating

(1-7)

1 45 74.50 ± 3.63 5.02 ± 0.20

2 191 70.48 ± 1.83 5.16 ± 0.11

3 44 71.48 ± 4.15 5.11 ± 0.27

All 280 71.29 ± 1.51 5.13 ± 0.09

ItemGreek SUS English SUS

Mean SD Mean SD

Q1 3.34 0.95 3.59 0.79

Q2 1.87 0.86 2.00 0.97

Q3 3.75 0.89 3.74 0.91

Q4 1.47 0.77 1.66 0.90

Q5 3.69 0.83 3.47 0.80

Q6 1.89 1.04 2.14 0.90

Q7 3.74 0.88 3.80 1.03

Q8 1.83 0.97 2.47 1.07

Q9 3.39 0.87 3.78 0.97

Q10 2.10 0.90 2.24 1.22

Note: Numbers after the ± symbol are 95% confidence intervals for the mean

Results (2/4) – Validity Analysis Is the Greek SUS valid for LMS perceived usability evaluation?

Validity = the extent to which an instrument, such as a questionnaire, measures what it is intended to measure (Nunnally, 1994)

11*0.30 ~ 0.40 justify validity of questionnaires (Nunnally, 1994)

rs=0.474, p<0.01

Concurrent Measure 1: Adjective Rating

t(278)=1.149, p=0.252

Concurrent Measure 2: SUS English

Results (3/4) – Reliability Analysis Does the Greek SUS measure LMS perceived usability reliably?

Reliability = the extent to which an instrument, such as a questionnaire, yields the same results under consistent conditions (Nunnally, 1994)

12Q6

Σκέφτηκα ότι υπήρχε μεγάλη ασυνέπεια σε αυτό το σύστημα Moodle του μαθήματος.

I thought there was too much inconsistency in this lesson's Moodle system.

Results (4/4) – Factor Analysis Does the Greek SUS measure a single construct?

English SUS was found to have two subscales: Learnability (Q4, Q10) and Usability (rest) (Lewis & Sauro, 2009)

Factor analyses using common verimax rotation

13

ItemComponent1 2

Q1 0.781 -0.096

Q2 0.485 0.356

Q3 0.652 0.339

Q4 0.124 0.648

Q5 0.642 0.293

Q6 0.032 0.537

Q7 0.648 0.107

Q8 0.280 0.598

Q9 0.812 0.078

Q10 0.072 0.765

SubscaleAlpha

No Items1 Q1,Q2,Q3,Q5,Q7,Q9 0.7892 Q4,Q6,Q8,Q10 0.573

ItemComponent

1 2 3Q1 0.782 -0.099 0.032Q2 0.462 0.184 0.388Q3 0.645 0.431 0.033Q4 0.108 0.736 0.099Q5 0.626 0.208 0.259Q6 -0.015 0.057 0.836Q7 0.654 0.291 -0.168Q8 0.244 0.341 0.567Q9 0.799 -0.053 0.267Q10 0.049 0.804 0.199

SubscaleAlpha

No Items1 Q1,Q2,Q3,Q5,Q7,Q9 0.7892 Q4,Q10 0.5313 Q6,Q8 0.382

ItemComponent

1 2 3 4Q1 0.668 -0.167 0.393 -0.136Q2 0.244 0.139 0.740 -0.004Q3 0.668 0.381 0.160 0.005Q4 0.143 0.726 0.115 0.036Q5 0.670 0.173 0.136 0.304Q6 0.060 0.098 0.101 0.953Q7 0.752 0.241 -0.100 -0.046Q8 0.056 0.321 0.719 0.197Q9 0.710 -0.112 0.422 0.134Q10 0.071 0.801 0.179 0.103

SubscaleAlpha

No Items1 Q1,Q2,Q3,Q5,Q7,Q9 0.7832 Q4,Q10 0.5313 Q2,Q8 0.5034 Q6 N/A

=> No reliable subscales

Summary & Questions Three studies involving 280 students assessing perceived

usability of an LMS were conducted in various settings: Education delivery method (blended, distance) Time of evaluation (start, mid, end of semester)

Results demonstrated the validity and reliability of the Greek SUS scale for LMS perceived usability evaluation

First medium-large scale study (N=280) of Moodle’s perceived usability in Greece (and among the few worldwide) Moodle SUS: 71.29 ± 1.51 Percentile rank: 60% (=> better than 60% of products in Sauro 2012

dataset of 446 studies and 5,000+ individual SUS responses)14

Future Work & Preliminary results

Effect of student’s characteristics (e.g. age, internet competence) on SUS scores (and score adjustments if required) Age & SUS

rs = 0.13, p<0.05 (but 80% of sample aged 18 to 25) r= –0.203, p<0.005 (Bangor et al, 2009)

Internet Competence & SUS rs=0.25, p<0.01) (but only 8% of sample low competence)

Effect of education delivery method on SUS scores (and score adjustments if required) Education delivery method & SUS

Mann-Whitney U=5164, p=ns (but 85% of sample blended)

More studies with different LMSs, course domain etc for generalization of results

15

Extra Slides

16

References (1/2) A. Bangor, P. T. Kortum, and J. T. Miller, “Determining what individual SUS scores mean: adding an

adjective rating scale,” Journal of Usability Studies, vol. 4, issue 3, 2009, pp.114–123. J. Brooke, “SUS: A quick and dirty usability scale,” in Usability evaluation in industry, P. W. Jordan, B.

Thomas, B. A. Weerdmeester, and I.L. McClelland, Eds. London, UK: Taylor and Francis, 1996, pp. 189–194.

N. Kostaras, and M. Xenos, "Assessing Educational Web-site Usability using Heuristic Evaluation Rules," Proc. PCI 2007 special session on Technological and Sociological Effects in the Use of e-Applications, Patras, 18-20 May, 2007, pp. 543-550.

J. R. Lewis, “Psychometric evaluation of an after-scenario questionnaire for computer usability studies: The ASQ,” SIGCHI Bulletin, vol. 23, issue 1, 1991, pp. 78–81.

J. R. Lewis, “Psychometric evaluation of the post-study system usability questionnaire: The PSSUQ,” Proc. of the Human Factors Society 36th Annual Meeting, Santa Monica, CA, 1992, pp. 1259-1263

J. R. Lewis, “IBM computer usability satisfaction questionnaires: psychometric evaluation and instructions for use,” International Journal of Human-Computer Interaction, vol. 7, issue 1, 1995, pp. 57–78.

J. R. Lewis, and J. Sauro, “The factor structure of the system usability scale”, Proc. Human Computer Interaction International Conference (HCII 2009), San Diego, CA, 2009, pp. 94–103

J. Nunnally, and I. Bernstein, Psychometric theory, McGraw-Hill Humanities/Social Sciences/Languages, 1994.

J. Sauro & J. R. Lewis, (2012). Quantifying the User Experience: Practical Statistics for User Research (1st ed.). Morgan Kaufmann.

References (1/2) N. Tselios, N. Avouris, and V. Komis, “The effective combination of hybrid usability methods in evaluating

educational applications of ICT: issues and challenges,” Education and Information Technologies Journal, vol. 13, issue 1, 2008, pp. 55–76.

T. Tullis, and W. Albert, Measuring the user experience: collecting, analyzing, and presenting usability metrics, CA: Morgan Kaufmann Publishers Inc., 2008.

T. Tullis, and J. Stetson, “A comparison of questionnaires for assessing website usability,” Proc. of the Usability Professionals Association (UPA) 2004 Conference, 2004, pp. 7–11.