User-Centered Evaluation of an On-Line Modern Greek

Language Course

Panayiotis ZaphirisThe Centre for HCI Design

City University

Giorgos ZachariaCenter for Biological and Computational Learning

Massachusetts Institute of Technology

Kypros-Net Inc. ( http://kypros.org/)

Formation– By a group of volunteers in 1996.– Composed mainly of Computer Science,

Information Technology and Human-Computer Interaction students.

Purpose:– “to utilize the Internet and particularly the

World Wide Web (WWW) system for the promotion of the world of Cyprus“

Learn Greek Online (http://kypros.org/Greek/)

Course Description– Independent study course to learn Greek

Online• 105 audio lessons, transcriptions,

vocabulary, spell checker, dictionary– Objectives

• Assist people in learning Greek language• Develop site content using Participatory

Design

Learn Greek Online Home Page

Participatory Design Methodology

(a) Building bridges with the intended users

(b) Map user needs and suggestions to the system

(c) Develop a prototype (d) Integrate feedback and continue the

iteration.

Design Methodology

Building bridges with the intended users

(Determine objectives, alternatives, constraints)

Mapping user needs and suggestions to the system

(Evaluate alternatives identify, resolve risks)

Developing a prototype (Develop, verify next level

product)

Integrating feedback & continuing the iteration

(Plan next phases)

STEP 1: Building bridges with the intended users Open lines of communication between

intended users and the development team.

– Initiate multidisciplinary development team

– Identify key groups of end users

– Create new methods of communication with users.

Intended Users

Bridges with users build through years of work at providing information about Cyprus through the web pages of Kypros-Net, Inc. who primarily attracts the same user population as the Greek language online course.– People of the Greek Diaspora– Travelers to Cyprus and Greece and other Greek

speaking areas – People who are generally interested in the Greek

culture and language or languages in general.

STEP 2: Map user needs and suggestions to the system Conceptual design model:

– “to design an online effective Greek language course that can build and sustain an online learning community of students”.

Tried to match user needs with our conceptual design model:– Easy to follow– Both Elementary and Advanced– Could attend at their own pace

STEP 3: Develop a prototype

The project consists of 105 audio files, which were originally recorded as Radio lessons in Modern Greek for English speakers back in 1960's.

The lessons were retrieved from the archives of the Cyprus Broadcasting Corporation, digitized in Real Audio 5.0 format and published on-line.

Prototype (Continued)

Although, an optional textbook accompanied the original Radio lessons, the lessons were designed as a complete standalone course.

We used several tools to assist the students of the lessons, including an online English-Greek-English dictionary, a Greek spell checker and a web-based discussion board for the students of the lessons.

STEP 4: Integrate feedback and continue the iteration. Feedback from our users and suggestions are

continuously incorporated into our design through a series of additions and corrections.

An important element in the participatory design methodology is the direct involvement of the user in all stages of the design process (including the evaluation).

Student Involvement

Users started exchanging through email written notes taken by the advanced users. – We provided the users with the capability to post their

notes on the project's site. Two months later, the students of the Audio lessons

have managed to transcribe 81 out of the 105 lessons, correct them through a peer review process among themselves and post them on the project's website.

Six months after that, the students had transcribed and peer reviewed all 105 lessons.

Evaluation Stages

1. Analysis of server logs 2. Analysis of discussion board postings 3. Questionnaires 4. Usability Testing

Key Evaluation Questions

How useful is the course? How functional is the course? How well does it meet on-line course

standards? What can be done to improve the course?

Analysis of server logsAccess Statistics

R2 = 0.7776

R2 = 0.6711

R2 = 0.606

0

5000

10000

15000

20000

25000

1 6 11 16 21 26 31 36 41 46

# of weeks

# o

f p

age

view

s

Total accesses

Audio accesses

Notes accesses

Board accesses

Boards+Notes

Expon. (Total accesses)

Expon. (Audioaccesses)

Expon. (Boards+Notes)

Questionnaires

Questionnaire 1 - ResultsQuestion Rating Q1 Overall, I am satisfied with how easy it is to use this online course. 1.40 (1.14) Q2 It was simple to use this online course. 1.40 (1.13) Q3 I can effectively complete my work using this online course. 1.20 (1.22) Q4 I am able to complete my work quickly using this online course. 1.05 (1.27) Q5 I am able to efficiently complete my work using this online course. 1.10 (1.24) Q6 I feel comfortable using this online course. 1.49 (1.07) Q7 It was easy to learn to use this online course. 1.37 (1.15) Q8 I believe I became productive quickly using this online course. 1.19 (1.17) Q9 The online course gives error messages that clearly tell me how to fix problems. 0.13 (1.60) Q10 Whenever I make a mistake using the online course, I recover easily and quickly. 0.81 (1.29) Q11 The information (such as help, on screen messages and other documentation) provided with

this online course is clear. 0.89 (1.34)

Q12 It is easy to find the information I needed. 0.83 (1.37) Q13 The information provided for the online course is easy to understand. 1.24 (1.07) Q14 The information is effective in helping me complete the tasks and scenarios. 1.19 (1.06) Q15 The organization of information on the online course screens is clear. 1.20 (1.23) Q16 The interface of this online course is pleasant. 1.30 (1.12) Q17 I like using the interface of this online course. 1.23 (1.17) Q18 This online course has all the functions and capabilities I expect it to have. 0.99 (1.30) Q19 Overall, I am satisfied with this online course. 1.40 (1.06)

Questionnaire 1: User Satisfaction

1.5 1.5 1.5 1.4

1.1 1.10.9 0.9

-2

-1.5

-1

-0.5

0

0.5

1

1.5

2

Overall Ease ofuse

Overallsatisfaction

Simple to use Easy to Learn Info Effective toComplete Tasks

Complete WorkEfficiently

Easy to Find Info Information Clear

Sca

le fr

om -2

to +

2

Questionnaire 1: Findings

User Expectations

– High satisfaction

– Positive comments• Easy to understand

• Fun

• It helped refresh my Greek

• Finally after surfing for a long time I found that you could learn Greek from the internet

Questionnaire 2: (Demographics)

Total of 16,848 registered students (6,098 completed the demographics questionnaire) – 49% Male, 51% Female– 65% found the page through a link from an

other page– Age 15-30 (44%), 31-50 (35%), 51- (10%)– Internet usage: 1-3 years (34%), 4-6 years

(37%), 7+ (16%)– Primary language: 72% English

Questionnaire 2: (Demographics)

– Education: High School (16%), Some College (20%), College Graduate (22%), Master (10%), PhD (3%)

– Location: Europe (38%), USA (43%)– Computer Platform: Windows 98 (54%),

Windows 2000 (20%), MAC (2%)

Analysis of Discussion Board (371 postings)

Category/Subject # % Total %TechnicalProblems with needing/installing keyboard fonts 56 15.1%Problems with/wanting to download lessons 49 13.2%Problems with installing/using RealPlayer 40 10.8% 39.1%

ContentPostings of on-line notes 73 19.7%Questions about availability of text, CDs or

including links to web sites 37 10.0% 29.7%

ComprehensionQuestions about vocabulary 28 7.5%Questions about grammar/spelling 17 4.6%Questions about dialect 2 5.4% 17.5%

Miscellaneous 69 18.6% 18.6%

Formal Usability Test

How functional is the course?– Do the operations work?– Is the architecture successful?

Findings– Site is subject to typical Internet problems– Learn Greek! architecture can be improved

Formal Usability Test - Findings Operations

– The site generally works (Translations, lessons, etc. function)

– Web operations less dependable (Server downtime, Slow lesson downloads, Intermittent transmission)

Architecture– Labeling

(Incorrect, Confusing, Out-of-order)

– Content (No directions, No descriptions, No “Help” or FAQ, Multiple translations)

Formal Usability Test - Site Architecture-Labeling

Formal Usability Test - Site Architecture-Formats

Discussion

Students created an open online community whose collaboration has boosted the learning experience of the whole community.

The web-based discussion board has proven to be the most constructive tool for the students learning experience and the main source of feedback for the maintainers of the project.

Discussion

The experiences shared on the discussion board include tricks and tips on how to record the audio files, installation of Greek fonts, learning methodologies and questions about the Greek language itself that arise from the lessons.

Also used for announcing the availability of their personal notes.

User Leaders

The advanced users (some of them retired teachers of foreign languages) have taken a lead role in the vast majority of the threads on the discussion board

Answering most of the questions and encouraging the beginners to study the lessons further.

Communication interface between the maintainers of the project and the community's requests. Maintain the online notes pages

Discussion

The accesses the audio lessons, the language tools, and the total access of the message board and the notes pages, all kept increasing exponentially.

Once we allowed our users to publish their own notes, there was a dramatic shift of traffic from the message board to the notes pages. Probably– Because it was no longer necessary to visit the message

board to find out where other users had posted their notes, since all were aggregated in a central location.