The Managed Learning Environment

at

De Montfort University

Report 6.2

“Managed Learning Environment – Usability Issues”

Anne Jelfs and Mary Thorpe

Institute of Educational Technology

Open UniversityWalton Hall

Milton Keynes MK7 6AA

July 2002

ContentsPage

1. Introduction............................................................................................. 3

2. Evaluation Strategy................................................................................. 3

2.1 Determine evaluation measures...................................................... 6

2.2 Pilot Groups.................................................................................... 7

2.3 Main Evaluation Study................................................................... 7

2.4 Review Student Performance Records............................................ 8

3. Findings................................................................................................... 8

3.1 Interviews with Academic, Library and Disability Unit staff......... 9

3.2 The Student focus group interview................................................. 12

3.3 A grouped list of findings from the interviews & focus group....... 13

4. Evaluation of MLE beyond academic year 2001/2................................. 15

5. Rationale for questions in the student and academic survey.................. 17

6. Conclusions and recommendations......................................................... 20

References................................................................................................... 22

Appendices

2

THE MANAGED LEARNING ENVIRONMENT AT DE

MONTFORT UNIVERSITY

(JISC 7/99)Evaluation Report 2

1. Introduction

This is the second formal report in completion of Work Package 6: User evaluation,

undertaken by the Open University in collaboration with the project team at De

Montfort University (DMU):

John Eyre, Project LeaderPat JefferiesMark Simpson

Open University Evaluation Team:

Professor Mary Thorpe, Director Institute of Educational Technology (IET)Anne Jelfs, Evaluation Project Officer, IET

Through a series of meetings in 2001/2, the development of the Managed Learning

Environment (MLE) and the strategy for its evaluation, both formative and

summative, was discussed at a number of project meetings involving Mary Thorpe as

the director of the Open University (OU) evaluation project, and since December

2001, Anne Jelfs as the project evaluator. At the meeting on 5th December 2001, a

draft evaluation outline developed by the OU in response to the JISC project proposal

and specifically WP6 in outline, was presented and agreed with DMU colleagues.

Details of this agreed approach and its subsequent adjustments are incorporated in the

evaluation strategy below.

2. Evaluation Strategy

Workpackage 6 requires four key tasks from the evaluation:

1. Determine evaluation measures with the OU and other interested parties2. Determine pilot groups3. Conduct surveys/interviews/focus groups4. Review student performance records

3

Two deliverables are specified from these studies – 6.1 Interim user evaluation report

and 6.2 ‘Managed Learning Environment – Usability issues’. Deliverable 6.1 was

completed in February 2002 and contains the results of interviews with pilot groups,

an initial evaluation of the CampusNet web pages and a report on accessibility issues

for MLE Demo pages, for users with a disability. The remainder of the planned

evaluation forms the second report presented here.

It should be emphasised that the evaluation plan in the project proposal was based on

the assumption that the MLE would be in general release for student usage during

2001/2, thus making user interviews and performance results feasible sources of data

for the evaluation. In the event the evaluation strategy has had to be adjusted to take

into account the changes in the timeframe for availability of the MLE to staff and

students. The MLE had not been generally released before the start of examinations in

2002 (and has not been widely promoted across the University at the time of writing),

and therefore it was not feasible to undertake extensive surveys on take-up and use.

Students’ study of their courses has not been influenced by the MLE since it has not

been made available to them, and performance data on 2001/2 is therefore irrelevant

to the evaluation.

The agreed evaluation goal was to provide an exploration of the usability issues for

students, teaching staff and support staff. Only those staff in departments that already

use technology in their teaching and in some cases have been involved in discussions

around the MLE, have been interviewed. One member of the Library staff was

interviewed and a representative from the disability unit. It would have been pointless

to interview outside this group since none of the staff has had access to the system and

therefore was not in a position to comment on it. The aim has been to find out as

much as possible in the circumstances about the perceptions of some of those in the

vanguard of usage of IT. Interviews have focused on the provision currently

available; their preferences about what they would like to have provided for support

of their teaching role, via the MLE and the University infrastructure; and their current

usages of IT for their own courses. The evaluation team were directed specifically to

focus on the MLE rather than details of VLE usage. However, the two cannot be

easily separated, since for teaching staff, their primary interests are likely to be in the

4

course-specific value-added of using a VLE in addition to existing teaching methods

and facilities. Further comments are made about this below.

The illuminative evaluation (see Parlett and Hamilton 1972) approach emphasises the

importance of the context within which innovations are introduced in education, and

this is still an appropriate approach in the early stages of introducing the MLE. The

use of semi-structured interviews conducted by the OU evaluator has enabled us to get

at both unexpected features of the infrastructure of the University as it affects IT use

for teaching, and to capture at least some of the preferences for what should be

available and how it should be supported.

Although it was agreed in meetings during March and April that a questionnaire

would not be a feasible strategy at this stage, the OU team agreed to prepare a draft

questionnaire that could be used in 2002/3 to evaluate the MLE at an appropriate

point after its general roll out. The draft questionnaires were discussed jointly with the

DMU team and the final amended version is set out in appendices B & C with a

commentary in section 5 below.

The heuristic evaluation (Nielsen 1994) was conducted by Mark Simpson and the

findings have been fed back to the project team as the basis for further developments

of the MLE.

Usability testing with participants from the target population of users is one

recommendation made by a number of researchers (Newman & Lamming, 1995,

Monk, Wright, Haber & Davenport, 1993). The value is that users can be observed as

they use the system and problems and queries identified. Computer users interact with

the computer system through a user interface, and for users that interface is the

system. It is knowing the user’s experience of the system which helps the designer to

understand the difficulties that users have and ways of improving the interface. Our

evaluation of the interface was conducted with the aim to identify how effective it was

in allowing the user to achieve their goals and how satisfied the user was with the

system. At the same time any problems that affect the usability are identified which

in turn lead to improvement of the interface.

5

The Managed Learning Environment incorporates a user-centred interface with a

single sign-on that provides a single site for students, teaching and support staff. It has

administrative, course specific and social elements, a calendar, email and personal

page http://severn.dmu.ac.uk:8080/mle. Staff who wish to use a VLE in their teaching

can do so, and in most cases staff use WebCT. The MLE is not currently designed to

include direct links to WebCT and staff and students access it via the University’s

intranet site

Details on additional information about building MLEs can be found on the JISC

website http://www.jisc.ac.uk/mle/7-99/. and a framework for Pedagogical Evaluation

of Virtual Learning Environments can be found on the JISC website

http://www.jisc.ac.uk/jtap/htm/jtap-041.html.

The specific data collection strategy was conducted under the four headings outlined

in Workpackage 6.

2.1 Determine evaluation measures

It was agreed that the basis for evaluation measures would be derived from semi-

structured interviews undertaken with both students and staff. A sample of staff and

students were to be provided by the DMU project team, from the areas with whom

participation in the development of the MLE in 2002 had already been negotiated,

including: Business, Law, Marketing, Pharmacy, and Computing. In the event this

proved difficult to arrange and the staff sample of seven academics were taken from

Marketing, Pharmacy, the Built Environment and Social Sciences. The five students

were recruited from one post-graduate course, M.A. in Design and Manufacture. One

interview was conducted with a member of the Library staff and a further interview

was conducted with a member of the Disability Unit (please see Appendix A for all

interview comments).

The content of these interviews included the following topics:

What has been your involvement in the MLE thus far?

What contribution to X (their course/s) do you expect the MLE to make this year?

6

What expectations do you have about how students will use the course related

elements of MLE for your course?

How often do you expect students to use these elements? How much time do you

think students might spend with these online elements?

In what ways if any do you think student learning will be helped through your

involvement in the project?

Do you have any goals for their development of the use of the MLE on your

course/area in the future?

These issues and the reasons for particular responses, were explored through semi-

structured interviews, and varied in the case of Library and Disability staff in line with

their roles and for students.

2.2 Pilot groups

The first stage of work with pilot groups included both interview questions and

observation of tasks, and the findings from the pilot studies was presented to the

project team in Report 6.1 Interim user evaluation report.

2.3. Main Evaluation Study

Conduct surveys/interviews/focus groups

Interviews

The expectation was that the MLE would be available at some point in March 2002

and data collection would be scheduled to take place after that, but before preparation

for the examinations became a major preoccupation for students. With the change to

the time frame of the MLE delivery, the evaluation study could not take place until

late May and early June. The interviews were based on demo versions of the MLE

made available to specific groups of staff and students for usability testing. The delay

in the MLE delivery did impact on the recruitment of both staff and students for

interview. Many students had returned home after their examinations and staff were

occupied with grading, exam boards etc..

The format of the student and staff interviews combined observations of usage

together with semi-structured interviews, similar to the approach reported on the

CampusNet interviews (Report 6.1 Interim user evaluation report). The names of all

those taking part in the interviews have been changed to preserve anonymity.

7

Focus Groups

Due to the impact of the later than anticipated completion of a fully developed MLE,

it was only possible to hold one student focus group. These students were post-

graduate computing students. Their reported comments can be found in Appendix A.

Surveys

As outlined above, the impact of the later than anticipated fully operational MLE

affected the proposed surveying of students. The work of the Evaluation Team has

therefore focused on creating questionnaires that can be used to evaluate the MLE and

its take-up over time (and beyond the end of the funded project - end of August 2002).

We were dependent on the delivery of the MLE within a relatively tight window of

opportunity, since until it was available, data collection could not begin. However,

the evaluation team has developed two questionnaires for the future evaluation of the

MLE. One questionnaire is for student data collection and the other is for staff data

collection. These questionnaires have been provided to the DMU team and copies are

presented in Appendices B and C.

The future evaluation and data collection from students will have to be sensitive to the

requirements of their timetables and availability before the examinations, particularly

since many are not available once these are completed.

2.4 Review Student Performance Records

Not relevant to the evaluation at this stage (see comments above).

8

3. Findings

3.1 Interviews with Academic, Library and Disability Unit staff

Seven academics were interviewed at the DeMontfort campus in Leicester using the

outline suggested. These were semi-structured interviews so that pertinent topics

would be covered although not always in the same order and with greater emphasis on

areas which related specifically to individuals and their concerns. A detailed account

of each interview is provided at Appendix A, and general themes summarised in the

commentary below.

The interface between MLE and VLEs in use

WebCT is the current learning environment in use at DeMontfort and most of the

academics interviewed used WebCT to differing degrees in their teaching. It proved

impossible to separate out only MLE-specific issues and this is not surprising given

the fact that most course-specific developments must occur in the context of the VLE

currently. Staff think about the MLE in relation to whether or not it enhances what

they can currently offer through the VLE, and whether or not the two environments

will be mutually supportive for students. The likelihood of students making frequent

and effective use of the MLE is seen in part as an issue of whether VLEs become

much more generally used by teaching staff across the institution, so that students

expect to be using computers as a regular and essential part of study, not just for

occasional social information or networking.

Some staff raised the issue of whether there should be a clearer University policy on

VLEs and whether a different platform was going to be introduced, such as

BlackBoard. The policy decision had not been made at the time of the interviews.

One academic said, “We’re waiting for the university to make clear what is happening

about learning environments. The faculty has been told not to make further

development until a decision is made.” When he asked when this decision might be he

was told September which he says is “actually when we will be short of time.”

9

MLE and its added value

One of the most active academic users of IT wanted to know what the MLE would

contribute and where it might duplicate other university provision. He would like to

have a greater understanding of what the students would view and how they would

use the MLE. He understands the MLE from his point of view, but not the students’

view. One recommendation by the evaluation team is that there needs to be

widespread workshops where academics, administrators and support staff are shown

the MLE, how it works and the benefits to all concerned. This includes giving

examples of student and other users viewing rights and specific pages.

The need for educational and technical support for academic users

One of the academics suggested by the MLE development team for interview was not

a user of any current learning environment systems. However he did conduct course

introductory sessions through distance education, therefore he was extremely

interested in the opportunity to use the MLE in his work. He is also a member of a

Leonardo project (European funding) for which he is the lead person. He wanted to

use a virtual learning environment such as WebCT in his teaching, but he feels he

needs support to do so. He wants not only technical support, but also pedagogical

support. He comments that he is appalled by the demise of the Centre for Education

Technology Development (CETD) as he needs that type of support particularly for the

Leonardo project and to enable him to put materials on WebCT.

All academics commented on the loss of the CETD support team at DMU. They all

thought that without the support they, as academics, were compromised and could not

dedicate time to the development of on-line materials, conferences etc for the MLE.

Reliability of the server and infrastructure essential

Academics also commented that the growth in student numbers and particularly

overseas provision and part-time students means that the MLE has to support these

students adequately and to do this it has to be available at all times. They were

concerned that without technical support and infrastructure the students would suffer

because of computer down-times.

10

Support infrastructure required in introducing the MLE effectively

Academics currently using WebCT as the learning environment were doing so to

differing levels. There were examples of academics fully integrating the use of the

learning environment such as in the marketing department and there were academics

who had no experience. As a random sample of the teaching staff this provides some

idea of the scale of the difficulties facing the MLE team. The anticipated problems

include the need for training, staff support in developing materials for presentation

within the MLE and the apathy of some staff in developing a strategy to include the

MLE in their teaching.

There are also reported policy difficulties. Academics report that there is no clear

university strategy for the development of managed and virtual learning environments

and how these should be incorporated in their teaching. As one of the academics said

“We have existing WebCT users and those keen to develop web-based learning. We

need school-based developments. We need to know what is going on.”

Staff acknowledge that the key to the successful uptake of the MLE is reliant on

academics incorporating the MLE in their teaching. However, there is concern that

they do not know how to go about this and the emphasis they should give to using the

MLE. Without full academic staff support and their input into the MLE it could

quickly become an unused resource.

Specific issues – email accounts, logging on, etc

Staff said that because students frequently attend university with current email

accounts such as Yahoo and Hotmail they have found that students do not use their

university email accounts. Therefore staff are concerned that the email via the MLE

might also fall into disuse. There were no suggestions how this could be overcome.

Academics also commented on a need for consistency of logging on to the system and

they questioned whether the initial log password for students should be their date of

birth, also the fact that the date of birth was in the US format of month, day, year.

Library staff would like to be able to develop the facility for students to renew and

request books and visit electronic databases from one single log-in. They are also

11

interested in knowing the actual figures of student log-ins to the MLE, and to provide

students in the future with notifications of a book’s status.

Specific issues for users with a disability and the Disability Unit

Staff at the Disability Unit would primarily need more lead in time to develop a fully

functional database of users with a disability. Disabled students need to be notified

that their information will be available to their tutors and given the opportunity to opt

out of this. The member of staff concerned would also wish to have liaison with the

MLE team about specific information for disabled students, such as banks with

wheelchair access. This type of information is not currently available on the DMU

web site, but would be useful to include on the MLE. She would need additional

support to both enter student data to the site and additional support to provide specific

pages for the MLE.

She would also like disabled students, particularly blind students to test out the MLE.

Some items she suggests may be unreadable by a blind student’s screen reader.

NB: A full analysis of the accessibility of the demo web pages has been completed by

the OU evaluation and detailed recommendations made, please see Report 6.1.

The Disabled Unit would also like a conference site to be set up specifically for

disabled students, although that would not preclude them from joining other

conferences.

3.2 The Student focus group interview

The student group interview was conducted after they had completed a session using

the demo version of the MLE. They liked most areas of the MLE but were concerned

about privacy and confidentiality. Part of the reason for these comments may have

arisen because they were post-graduate computer faculty students with an interest in

IT and design. They did suggest some design issues such as changing the font on the

MLE and moving the library site to the resource section and not the DMU location.

They saw the value of using only one site and one log-in, but didn’t fully understand

how it would be used and why the need to change from the current web site. More

use by the students and academics should alleviate this problem.

12

Both students and academics mention that they wanted something more magazine-like

and ‘punchy’. They felt that the interface at the moment was a little bland. As Trudy

from the Disability Unit commented, she wanted a “punchy, dynamic student

magazine design with punchy direct student language. As a magazine it would be

something to engage with rather than a text book when I have 80 other text books to

read.” At the moment the MLE portrays an image of a valuable administrative tool,

what it does not provide is the appearance of an interesting student-centred site.

3.3 A grouped list of findings from the interviews & focus group

In this section we discuss the issues which arose from the interviews. This includes

both staff and students’ concerns and these are grouped under the headings of

technology; teaching and student support; policy decisions; design; and general issues.

Technology

Academic staff in particular were concerned over possible computer

downtimes as they felt that downtimes would negatively influence student

interaction.

This also means that current problems with initial logging on to DMU’s

computing system need to be addressed immediately.

Both academics and students were concerned about student technology

support and problems with access to computer equipment.

Teaching staff wished to know if all current software programs used by the

University would be compatible with the MLE.

All interviewees to some extent referred to the requirement for technical

support.

Teaching and Student support

Staff recognised an increase in student numbers and an increasingly dispersed

student body required additional electronic support from the University and

the MLE was one such provision.

Academics value up to date class lists at the start of their courses and would

prefer the MLE to provide these earlier than the current system.

Teaching staff would value the opportunity to offer individual support and

diagnostic tests e.g. quizzes, to students via the MLE.

13

Academics had found that where WebCT has been used effectively there was

a reduction in student visits to academics for purely information needs and less

important items. A valuable point with the growth in student numbers.

It was hoped that the MLE would provide some of students’ pastoral needs

and therefore increase student contact with academics for subject-specific

reasons.

Interviewees needed training in the use of the MLE – workshops and

demonstrations.

There was a general feeling that the MLE would enhance student studies and

social support.

Academics were interested to find out how much students do engage in

discussion areas and what they can do to improve student engagement.

Academics think they would need additional training in the best use of the

medium for teaching.

Policy Decisions

A need for senior management leadership on the adoption of the MLE and

policy decisions to be made on the overall use of the MLE and VLE at DMU.

Academics commented on the time pressures that adopting the MLE places on

them without the support of CETD and requested staff to be made available

for ongoing support, both educational and technical.

Design

Interviewees commented that they would like a more magazine-like interface.

Library staff wanted to include library book renewal facility in the MLE.

General Issues

Monitoring of students’ and academics’ use of the MLE site – this enables

modification of sites through understanding of current use and is additional

date for the evaluation strategy.

One suggestion was made to have a standardisation of material presented for

publication on the MLE, although other members of staff did find the use of

templates unnecessarily restrictive.

There was a reported need for widespread publicity of the MLE to academic

staff, students and support areas.

14

4. Evaluation of MLE beyond academic year 2001/2

A number of options are available for evaluation beyond the end of the JISC funded

project. These should be clarified and reviewed in the light of decisions about the

specific goals of the MLE and how it will be implemented across the University.

Options include the following:

a) evidence of take up and usage by students of all areas of MLE, separately from

VLE usage

b) exploration of the experience ‘behind’ the data from a) i.e. reasons for usage,

whether high or low, in terms of performance of the interface, reliability of the

server(s), ‘fit’ with student behaviour and preferences, etc

c) exploration of staff usage and experience of MLE, in parallel with a) & b) above,

and including issues around the availability of support to them for use of the facilities

offered

d) evidence of usage of VLEs across all departments, and comparison with pre MLE

usage, for comparison, plus exploration of the reasons for this usage, including the

availability of support to staff for use of VLEs in their teaching

e) staff perceptions of changes in (i) how they teach their courses (ii) student

behaviour & performance, following on changes shown in d) above

f) Library and administrative staff usage and perceptions explored including their

perspective as key providers of data for the MLE system.

g) specific issues – qualitative evaluation allows for unexpected outcomes to become

apparent to the evaluators, and where these arise, to be followed up and incorporated

into the evaluation as it develops. It would be worth following up any indications

from staff that student performance or the quality of their learning was improving – or

indeed being damaged – by either the MLE or VLE developments. This seems a better

strategy than looking to find quantitative improvements in student performance as a

direct result of introducing either the MLE or VLEs. Performance data is so likely to

be influenced directly by a range of other issues, that it would be very misleading to

draw specific conclusions without extensive evaluation. This does not preclude in-

depth exploration of specific cases, as in the interview with ‘Tina’ (see appendix A),

who feels pass rates have improved through VLE use.

While the resources available for evaluation will obviously determine how much of

what is outlined above can be evaluated directly, we would in general recommend that

15

1) a system of collecting data about usage of both the MLE and WebCT (or

whichever VLE is used) is set up for the start of the autumn term, and covers

both staff and students

2) qualitative interviews and group discussions are undertaken before

questionnaire based surveys are done. Surveys are best undertaken when the

parameters and the content areas of problems and issues are broadly

understood and can be accurately ‘coded’ into clear and coherent questions.

Amendments to the draft questionnaires might then be made before usage.

The schedules that might be used in such interviews can be adapted from those

already used in the evaluations reported both here and in the earlier report.

If resource is available for providing a programme of support and staff development

around the introduction of the MLE, as recommended from the comments made in the

staff interviews, then it would be feasible for the project staff to follow an action

research model. An action research model is based on an integrated and planned

programme of data collection and development, in order to evaluate provision at key

stages, and to generate formative feedback that can be used to progressively improve

the MLE and the facilities and arrangements around its introduction.

In general, a long-term perspective should be adopted, looking ahead to a 3 to 5 year

period of development not just one year at a time. Change on the scale envisaged by

MLE or similar technology introduction is not completed in shorter time frames, and

the great value of data collection is to be able to review trend data over several years,

rather than one year only. A small amount of evaluation carried out consistently over

3 years might be just as, if not more useful, than a large amount that does not go

beyond 2002/3.

Among the many sources of information about evaluation in this area, Oliver and

Conole (1998) provide a very useful overview.

5. Rationale for questions in the student and academic survey.

The evaluation team agreed to produce draft questionnaires and rationale for each

question set, so that the Project Team could use these as the basis for subsequent

evaluation. This section sets out these requirements. It should be noted that the

16

questionnaires provided with this report (see appendices B can C) can be modified for

electronic use and the layout is only for report presentation purposes. The timing of

delivery of the survey is left to the Project Team, but as a longitudinal evaluation

there would be benefits in conducting the survey at differing points within the year to

look for variation in take up of the MLE.

Academic Survey

The first two questions in the survey refer to baseline data concerning the number of

modules where the respondent uses either the MLE or a VLE. The baseline data

provides information on staff use of the MLE and VLE, which can be compared with

each other, and these first two questions can also be correlated to further questions

such as acceptability of the system and technical needs.

Question 3 is a measure of the value placed on the use of computers in their teaching.

This acceptability measure can then be used to compare with adoption of the MLE

and VLE and also Question 10 on how the MLE has affected the academic’s teaching.

Question 4 looks at the locations where academics are accessing DMU systems. The

question is looking at whether academics are relying on systems based on- or off-

campus. This has relevance for supporting academics in differing situations.

Question 5 refers to the frequency that academics use the MLE. Again, this has

implications for support, especially if they are accessing the site off-campus and the

question also provides vital data on frequency of use. This question may be

something that could be used at differing points within the academic year. Teaching

staff may be using the site more frequently during the first few months of the new

academic year or the start of modules and less towards the end of the year.

Question 6 in the academic survey will provide evidence of the take up of specific

features offered by the MLE and a measure of the degree to which MLE activities are

in use across the University. Departmental differences can also be revealed through

comparative analysis across the responses for each feature.

17

Questions 7 and 8 on the academic survey provide open-ended free response

comments and therefore provide qualitative data on the perceptions of staff about the

MLE. Qualitative comments are a valuable addition to the questionnaire because it is

easy for survey designers to omit questions and issues which are pertinent to the user.

Requesting open-ended comments invites users to put issues into their own words and

to express issues important to them. Qualitative data provides respondents with the

opportunity to be reflective about their personal experiences and to address their own

needs.

Question 9 relates to their experiences of using the MLE interface and this offers the

project team information on areas for change or increased training for staff.

Question 10 refers to the ways the MLE has affected their teaching, plus the issue

they rate as having the most effect on their teaching. It is anticipated that this would

provide data on perceived changes to their pedagogical approaches.

There is also a final section where staff can again make a written response to any

aspect or issue arising from the MLE.

Student Survey

The first two questions in the student survey are the same for students as they are for

academics. These questions refer to baseline data concerning the number of modules

where the respondent uses either the MLE or a VLE or both. The baseline data

provides information on student use of the MLE and VLE which can be compared

with each other and these first two questions can also be correlated to further

questions such as acceptability of the system and technical needs. There can also be

some comparison between student usage and teaching staff usage of the MLE.

Question 3 is a measure of the value placed on the use of computers in their studying.

This acceptability measure can then be used to compare differing student based

cohorts: discipline/course differences; gender; part-time/full-time students.

18

Question 4 looks at the locations where students are accessing DMU systems. The

question is looking at whether students are relying on systems based on- or off-

campus. This has relevance for supporting students in differing situations.

Question 5 refers to the frequency that students use the MLE. Again, this has

implications for support, especially if they are accessing the site off-campus. The

question also provides vital data on frequency of use.

Question 6 for the student survey refers to how students are using the MLE and

question 7 refers to frequency of use of specific items within the MLE. Again,

departmental differences may become apparent from this question. As part of the

evaluation strategy this information could be fed back to course designers and module

leaders for the integration of specific uses and course design.

Questions 8 and 9 are open-ended free-response questions referring to their

perceptions of the MLE.

Question 10 is the same as question 9 in the academic survey and relates to their

experiences of using the MLE interface. This offers the project team information on

areas for change or increased training needs for students.

Finally students can make further free-response comments on the MLE.

There is a small bank of additional questions provided for each survey which the

project team may wish to include or to add at some later point. It is suggested that all

students and teaching staff are emailed the survey and the collected data fed back to

interested groups e.g. academic boards, student union bodies, disability and library

units and administrative departments.

The project team can gather qualitative data in the future either by face-to-face

interviews with staff and students or through focus groups. It is recommended that all

qualitative data is audiotape recorded or digitally recorded. With focus groups or

interviews a semi-structured interview approach is suggested, i.e. those topics of

importance are specified in an interview schedule, but that the delivery of the

19

schedule is responsive to the experience and identity of the interviewee. The

interviewer also has to attend to the responses so that a dialogue can develop whilst at

the same time ensuring that all the topics are covered. It is suggested that the topics to

be included are similar to those conducted for this evaluation, i.e. actual usage,

anticipated future use, effects on teaching and learning, and experience of the system

in use.

It is not anticipated that large numbers of qualitative interviews will be undertaken,

but in specific target areas or with discipline based cohorts of students. These may be

target groups selected from areas of concern identified from the survey and usage

statistics analysis.

6. Conclusions and recommendations

Evaluations of IT introduction into complex organizations such as universities and

colleges have systematically highlighted the impact of existing custom and practice of

staff and students as key in mediating the impact of the technology and its effects. The

project leaders are aware of the many challenges they face around these issues, as well

as those over which they have more direct control such as the delivery of the MLE

itself. They have succeeded in many of their immediate aims and a summary of some

of the key recommendations to arise from the work of the evaluation is as follows:

The evaluation of the MLE by the project team based at DMU should combine

quantitative and qualitative measures, and be designed according to the

resources available and with the long-term interests of developing high quality

student learning over a 3 to 5 year time scale.

Development of the support to academic staff is absolutely essential, for

effective use of both MLE and VLE. This was stressed by all staff, most of

whom are in the forefront of technology use and with expertise well beyond

the average. All wanted both pedagogy/educational and technical support.

Clear University policy on the adoption of the technology is called for.

Academic staff were unclear about strategy and felt that a University lead was

required.

There was an evident need for more lead-in time for the introduction of the

MLE. Staff also wanted a participative process, giving them opportunity to

20

come to terms with the facilities on offer and to see how they ‘fit’ with

existing practice and how that can be developed. There is a lack of awareness

and a programme of publicity and ‘hands on’ workshops would meet the

needs.

Training/workshops on the MLE as part of the above, are urgently required

A more magazine-like interface for MLE was wanted, to present a style that

students in particular could identify with – MLE needs to pull students in.

Library book renewal and book availability notices would be valued features.

Specific disability information requires careful handling, and the needs of

particular sections of the University such as the Disability Unit need to be

accommodated – it is unlikely that a ‘one size fits all’ strategy will be

successful.

Implementation of the accessibility recommendations should be done,

including achieving Bobby approval of the site, as set out in the earlier report

6.1.

Logs of student and academics’ use of the MLE site should be set up and

analysed as part of the evaluation, as outlined above

21

References

Monk, A., Wright, P., Haber, J. & Davenport, L. (1993) Improving Your Human-Computer Interface: A practical Technique. New York: Prentice Hall

Newman, W.M. & Lamming, M.G. (1995) Interactive System Design. New York: Addison Wesley

Nielsen, J. and R. L. Mack, Eds. (1994). Usability Inspection Methods. New York, Wiley.

Oliver, M. and G. Conole (1998). "Evaluating communication and information technologies: a toolkit for practitioners." Active Learning 8: 1-6.

Parlett, M. & Hamilton, D. (1981) “Evaluation as illumination: an new approach to the study of innovatory programs” (originally published as Occasional Paper Number Nine, University of Edinburgh Centre for Research in the Educational Sciences, 1972) in Parlett, M. and Dearden, G. (eds) Introduction to Illuminative Evaluation: Studies in Higher Education, SRHE, University of Surrey, Guildford

22

Appendix A

Managed Learning Environment Interviews

Interviewees:

Tina Lecturer in Marketing

Alan Interviewed together. Both working in the Built Environment department Charles

Dennis Head of Marketing Module. Interviewed with Mike

Mike Pharmacy – Teaching Fellow

Charles Follow-up interview

Gary Social Sciences

Adam Library

Trudy Disability unit

Student Focus Group

MA in Design Manufacture

Tina - Marketing

Tina has worked with CampusNet (the system developed at the Milton Keynes campus) for the last couple of years. This was primarily instigated by the size of student recruitment to the module. This course is delivered to over 10 sites, including franchised sites, to approx. 1000 students. This includes sites overseas. She wanted to help students in their studies and to get information to students effectively. The student body has a lot of people who are working and they need material available at all times. She does however say that WebCT is not used in isolation “The VLE is integrated with traditional ways. They still need to use a textbook and they can’t not come to tutorials and lectures. Its part and parcel of the delivery.”

According to Tina the key thing that it adds to the course are quizzes from the lecture notes, as formative assessment. Says it has made such a huge difference to the pass rate. Depending on which programme they came from it made between 7-20% difference in their overall grades. (She found this out by tracking students’ formative and summative scores). Its that which has interested her colleagues. The quizzes are only multiple choice and the increase in overall grade was irrespective of whether they attended tutorials or not.

Experienced user of WebCT.

Used to cope with increasing student numbers.

Facility to help students in their studies.

Formative assessment – linked to improved overall module grade.

23

Students say they really enjoyed it. The only problem Tina has is that of lack of student engagement with the discussion area. So much so that they have invented an imaginary student to start discussions and that still didn’t work. Students did use it to apologise for not attending tutorials. Therefore she says that when they did come to her, it was not to apologise for missing a tutorial, it was for subject-specific reasons – “it raised the game”.

Tina has left messages on the site for students but because the first part of the site is so ‘busy’ the important messages get lost. There needs to be something to raise students’ awareness of important messages on their modules.

WebCT helps students at franchised sites to feel part of the community – and that feeling helps the staff at those sites as well.

Says that they used to have tutorials every second week, but now there is a move to weekly tutorials. This means that if students miss three tutorials in succession, she can contact them via email and check why. In turn she needs to have accurate class lists, which aren’t always available. Students can’t miss tutorials because the end of module assessment is tied to the tutorials.

Tina is not sure about the statistics from WebCT on the number of visits people are supposed to have made to the module. She is unsure if students who go to another site and then return are counted as two visits or only one.

She uses it to see where students are looking and finds this useful for students who are having problems. Says you can see if the student is disengaged and not looking at the site. Plus she uses it as feedback on what students are using and improve the site accordingly. She also uses it for assessment and key skills teaching. Has found that students want web sites for other modules. However, she thinks that not all modules “lend themselves” to a web site.

She knows that colleagues have developed their own web sites but she needed someone to help her to set up what she wanted.

Provides students with training session on using WebCT in Week Two and not Week One, because she recognises that students just starting university are nervous and don’t take everything in. She stage releases information on module

Student feedback is positive.Difficult to get students to use discussion area. Students usually only visit staff to ask course specific questions.

A consideration for the MLE is that important messages need to be presented clearly and on the first page.

Helps staff & students at franchised sites feel part of the academic community.

Monitoring of student attendance. Need for accurate class lists.

Clarification of what the statistics really measure.

Quality of course – student feedback.

Technical support required to help develop own site.

Provides students with training. Stage

24

including key skills.

Has found that the use of WebCT has meant that she does not get as many questions on ‘where do I go to get this?’ and neither do her colleagues. It means that when a student comes to the member of staff with a problem it’s not trivial and they can be given more time because the trivial problems have gone away. However, she thinks that if students have a problem with ‘logging on’ in the first couple of weeks then it’s a barrier and it takes ages for them to come back. There are also problems when students keep changing their passwords. Passwords are given in tutorials and lectures. These usually include date of birth and sometimes the month and day are transposed (US version of dates) and at other times not, so there is no consistency.

One problem that Tina can foresee is that more use of on-line materials and sites will cause difficulties because the Faculty only employs one person half-time to provide both educational support and a technical service (other half of the time he is in another faculty). She says that there was support from an internal team but that this team has now been made redundant.

Tina does not give out handouts, so students have to go to the website and use the textbook and engage with the area of marketing. The number of students who don’t use the web site is so small that to her it’s no problem.

To encourage student use of the web site she gets students to work in peer groups because of the numbers and then has 3 un-staffed lab bookings so that students can go in at that time.

Tina thinks that a major barrier to the students using or not using the site is due to the staff and to counteract this she conducted a Module Staff Development day. The key themes for the day were based on assessment and dissemination and this included web use. Because of this the staff started to use it and she feels that impacts on how much students use it.

She also has a tutor-to-tutor group, so that private items for tutor use are put there. Has produced tutor guides and resources, so that tutors have material for sessions.

She also puts a lot of links to course related sites on WebCT.

If a lecture is cancelled the information is put on the WebCT site and if there is time then it is put on the door. Room changes are

releases information.

Releases tutors from answering trivial questions.

Problems with logging on are a deterrent to student interaction with the system. Hopefully this can be avoided with the single sign-on in the MLE.

Increased support from University infrastructure. Loss of internal support team has been a problem and will continue to be one.

All handout material is on the web site.

Understanding student needs and problems with access to computing equipment.

Staff Development and how staff use impacts on student use.

Tutor to tutor group. Tutor guide and resources.

Course materials

Cancellation notices and room changes

25

also put on the WebCT site.

She thinks the new MLE pages are very busy and needs to be more ‘staged in’ i.e. less information on each page and more ‘hot’ links to other pages.

Alan and Charles – Built Environment

One of the things they mention at the beginning of the conversation is the demise of the computer support department. The loss of which has not impacted on them as much as it could do with other academics.

They say that because they teach with ICT what they wanted in terms of support was not so much technical as someone who could help them make the correct decisions about the learning implications for students.

In their teaching they use a range of programs From Excel through to 3D Studio and CAD. They want to use systems the same way as the building industry and to use industry-wide programs in their teaching.

The school uses WebCT to support lectures. The material is usually produced off site and then hosted on WebCT. Alan and Charles bring to their lectures lap-tops so that their material is available electronically and only needs downloading to WebCT of the module site if they wish to make the material available to students. They have whole modules that are on-line and some formative assessment. Recently they had looked at WebCT data and found a number of students had not used it, which is a cause for concern as they present material and assessment for students. Normally Charles’s lectures are in a room with linked machines and a digital projector so using WebCT for him is no problem.

They say that their typical use is:- module guide on-line- project briefs- directed reading with links- class notes- on-line support material which is also available as hard

copy- guide to CAD and FrontPage

- used to replace overheads.

are put on the site.

Design of MLE

Loss of technical support

Need for educational technologist.

Specialised technical needs.

WebCT to support lectures. Number of modules are on-line.

Case studies and

26

Alan and Charles use WebCT for case studies and have a presentation area that students can also show potential employers.

Charles and Alan would like to be able to change student enrolments. Class lists are not received in a timely way.

Acknowledge that some academics would not see development of material for Intranet/WebCT as relevant to them. Charles and Alan say that what they personally need is a support system where there has been a dialogue between staff to develop the site. This is not always technical support, rather they want people who understand the needs of the teaching staff.

Alan and Charles tend not to use the module email as students have usually got Hotmail and University’s mail. They have part-time students who also probably have access from work (these are day release students.) Students provide their preferred email address.

There are important differences and variation of prior knowledge of computers by students.

They say the MLE calendar is tedious to use and has a tricky interface. They say that they could make a lot more use of the calendar and would like to be able to put important information directly on to the student’s calendar to remind them of important dates/deadlines etc..

Maintaining areas is difficult. If you use any sites then you need support to maintain them and keep up to date.

Charles develops his own materials and is a competent user of electronic communication. He fears that the MLE provides too uniform a site and that the over use of templates would inhibit innovation – so house styles might be prohibitive to him in developing his own module presentation on the MLE.

Dennis and Mike – Marketing and Pharmacy

Dennis says that a major problem for him is student support and making support easily accessible to students. Whether this is their results or module choices, he wants to make it easily available. He likes the idea that students have one central site to

presentation area.

The MLE would be one way to provide up-to-date class lists from registry.

Need for educational technologist.

An issue for the MLE is the email provision and a single site login to the email system.

Students have differing levels of skill.

Staff are interested in how to get the most from the calendar feature.

Technical support

Use of house styles may prohibit the innovative academic.

Student support

27

start from. Says there is a sense that it’s all to do with administration on some of the MLE but does not underestimate the value of administration in supporting students.

Dennis says that the more we can release staff from the burdens of making sure that ‘stuff’ is accurate on student details, the better. More time can then be used on academic input. This is important at the moment in getting marks back to students.

Academic use of home computers and internet are hidden costs that the academic picks up. Staff and students access from home as well as on site.

Too frequently the web-based material that academics put on is the text-based material they already have and this is not the best use of the system.

Dennis says that there are publishers offering on-line material or companion websites. His dept needs to look at the potential and the opportunities.

The ‘Principles of Marketing’ is the nearest they have to an on-line course (Tina’s course). It is actually a taught programme supported on-line. It is not intended to be an on-line course but it does have material available on WebCT. This course provides standardisation of materials for tutors and students.

There is very little on-line assessment in Marketing. What they do have are multi-choice testing part way through called ‘phase testing’. They have this type of formative assessment because they are concerned about the security of testing and would not want summative assessment unless they could be sure of security and policy etc. They also want to give good quick feedback. This is one way of addressing the challenges of large student numbers.

Dennis says he’s not sure if students want to send summative/formative assessment on-line but sure that staff want on-line assessment. The staff are concerned about plagiarism and in electronic form they could check for plagiarism, because of the programs available to do so.

Sharing material with students and staff is useful on-line.

including administrative support.

Staff supported by accurate student details and dissemination of exam results.

Academics use own hardware and phone. These are costs to the academic.

Poor use of the technology

Review and compare commercial on-line material which is available.

Use of WebCT to enhance an existing course and to support staff & students

Formative assessment

Fast feedback to students and computer based feed-back enabled scaling up to meet growth in student numbers

On-line assessment and plagiarism.

28

Staff are reticent to provide feed back on-line because it is felt that it increases the workload due to slow typing speeds etc.

Dennis says that the dept needs to find out if students are printing the material – the costs of the text is then passed to the students. Says ‘perhaps we are not using the technology in the right way and students print out because of that.’ Students can’t annotate on-line.

The faculty paid for the use of the Centre for Education Technology Development (CETD), which was closed down. Now the money is not given back to the faculty and so are trying to do the work without it (Dennis has only .5 of a person). Dennis illustrates his point through an analogy – when buying a car he doesn’t want to know how the engine ran or become a mechanic – wants to be able to use it – like going to a garage he wants to have it maintained and repaired.

They want educational technology expertise and how best to do things. Half a person is trying to support 170 staff in Business and Law.

CETD has gone and they ask ‘where do I go to when I have glitches with things e.g. WebCT? There is a technician who supports the server but not what I always want.’

Mike assumes that students come in with basic computer skills and uses WebCT to give students a discussion area. He has four years of evaluations of its use and students think its great.Dennis would like to know how students’ evaluations and impressions are made when some are on-line and others face to face.

For Mike his students say that they think its great and think it’s ‘sexy’ and want all modules to use on-line material.

Dennis says that this means that students may think that without on-line materials the course is not adequate. Also who should develop the skills to put material on-line?

Mike, very strongly criticises the decision to close down CETD. ‘The University unbelievably closed down the department that could help’.

Sharing of material online

Slow typing speeds impacts on on-line feedback

Need to understand student usage.

Loss of academic computing support causing major difficulties to the departments.

Inadequate educational technology support.

Students like using the on-line facilities of WebCT. What are student evaluations of on-line material based on?

Pharmacy student opinions of on-line materials very positive.

Whose responsibility is it to develop skills for on-line material?

Need for CETD or equivalent to return.

29

Mike says that he doesn’t think that putting material on the web saves time for students. He has one student who has looked at his site since the beginning of the semester @ 539 times. However, he thinks this is because the student is over compensating because she has come from the HND route.

They say that not all members of staff are up to speed on using computers and programs. Dennis asks what is the policy of what an academic should be. Says: we used to have the support for typing now we don’t, we do it ourselves. Are we saying that academics have to learn this type of technology? He means new software programs etc. They suggest that some people are frightened of trying. Which in turn has implications for resourcing and the policies surrounding resources and that is not just the institution. Mike says that compared to Coventry (University) it is ‘ad hoc here’. There are pockets of interested people, but not across the board

One person in the business department has produced their own web site, so that it can be on the Intranet, but this is someone who is technically able.

They comment that ‘we are not just expected to teach, but also to do QAA, RAE, funding’. They say that whereas 5 years ago they could look at web-based learning, now there are competing things for an academic’s time, so they appreciate that some people are reticent.

At the same time they say that more and more students are working and working more hours. There is a danger that full-time students are studying part-time. They question where the increase in student employment hours are to support their life styles, as nearly every student has a mobile phone.

The student list – Dennis says that ‘if we get one by day 21 we are happy.’ Whereas Mike says that pharmacy has access to their own electronic student records, but this is only due to their own efforts. One of the problems that Dennis says marketing has is how other faculties put their information on the system, and whether they do it at all. ‘We find sometimes 30 students there which a faculty hasn’t registered’.

When Dennis had seen some examples of how WebCT had been used they all looked different. If a student has a number of modules then it can cause problems. He would like consistency and at least consistency in terminology. ‘We need agreement or we end up confusing students.’ However as they say some people welcome conformity when they are starting out and need guidance.

In appropriate study strategies revealed – online learning strategies need more exploration.

What are the University’s policies on using technology and skills development?

Ad hoc development of on-line resources.

Time pressure on academics. QAA, RAE and external funding also make time demands

Student employment

Up-to-date student lists

Consistency of terminology and usage across modules.

30

They both feel that the loss of CETD was a retrograde step and that CETD could have provided guidance to the academics. Including design features e.g. use of an off-white background to cut down glare.

Mike has obtained from his Teacher Fellowship budget a hand held device. He is looking into ways of developing hand held delivery to students and student learning, instead of students having to buy laptops or pcs.

It is a Compaq on which he has put his own material and ebooks from the Internet. He is wondering if this is a viable way for student learning. It might not be at the moment because it costs approx £500, but he thinks costs will go down. His lecture notes are easy to read on the device.

The hand held Compaq uses Blue Tooth (a commercial product – wireless and enables links between mobile computers/mobile phones/connectivity to Internet) in a separate sleeve. It has a full screen and not a half screen which Palm has. Mike has put some PowerPoint it and has a separate VGA card, which at the moment he doesn’t know what it looks like linked to a projector.

He says that in pharmacy these wireless devices are used on the wards to bring up patients notes and to prescribe.

He thinks students would be attracted to these if they were £100. He would want the MLE to be able to interact with these types of devices.

Charles (second interview)

He wanted to complete a second interview 4 months later to comment on the developments of the MLE and the impacts that the MLE will make on his work.

He says he is waiting for the university to make clear what is happening about learning environments. The faculty has been told not to make further developments until a decision is made. When asked when this might happen, he replied that he was told September which is the point when the academics are short of time. He says the lack of information is difficult to deal with.

Charles says we have existing WebCT users and those keen to

Loss of educational technology support

Development of hand held materials.

Innovation of teaching.

Developing future employment skills.

MLE currently provides different pages for different devices – will hand helds be included?

What are policy decisions?

31

develop web-based learning. We need school-based developments. We need to know what is going on.

He would like to know what the MLE would contribute i.e. what will be duplicated and where do students go to get particular information. He feels he needs a greater understanding of what the students will have available from the MLE.

Charles can foresee lots of bottle necks at the moment. If the MLE adds to admin ‘hassles’ then he sees problems. “We just want to deliver additional course material and don’t want to have more hassle.”

He says he would like a ‘year’s grace’. He has no difficulty in using the MLE, but he thinks that if academics feel out of control then it will fall down.

Charles says that the interface is missing some interest. It feels like you go there if you must and is not a fun, attractive site.

Says the MLE needs to offer something to the academic for teaching and the student needs to find things easily. He says it is a case of “Does this bus go where you want it to go?”

He would like to know who is going to be updating the Social page – student union?

Gary – Social Science

Gary has no experience of the MLE. but refers to himself as an interested academic.

He works in the Social Work department and wishes to expand his introductory sessions which are currently held on Saturdays to a Distance Education Course.

He wants to use WebCT and needs support to do. Not only technical support but pedagogical support. He is appalled by the demise of the CETD department a couple of years ago. He says

Electronic teaching needs.

Training needs in what students will receive from the MLE.

MLE interface with academics needs to be sufficiently flexible to accommodate the sophisticated user, motivated to progress quickly to using the MLE/VLE for teaching.

Needs more time for implementation.

Layout and design issues.

Design and teaching issues need to be more obviously interesting and motivating.

Who updates the social page?

Development of distance teaching.

32

that he needs this form of support.

He is also part of a Leonardo project for deaf students and is the lead person for this project. Again he says this requires technical support for the project and the materials to be made available on WebCT. Without the support to do this work he feels frustrated.

Adam - Library

Currently the library provides web pages and they point students to electronic databases. He says the password might be different for each database. He sees the MLE as a ways that might help students, e.g. “If we know from the log in that this person is a student then we don’t need to ask again.”

He sees the MLE as a way of enhancing access. He thinks that students don’t realise that electronic databases are part of the library and not just the web. OPAC is available to everyone.

He would like students to have a self service option e.g. renewal of books. This would free up student as well as staff time involved in renewals.

Adam would like the testing of the site to ensure that there were no failures to provide a good service. This he sees as requiring a couple of months to set up and then testing so that it is reliable for students. In fact he then goes on to say that it would probably take six months.

Ideally he would like to be able to tell students that their books were overdue or that their requested book was now available. He thinks it would be a good idea to send messages to students. This he sees as requiring work from the library, but he feels there aren’t enough people in the library to support that.

Adam would like to know if people would use the MLE. He would like to know how the project staff aim to get students to use it. He says it needs to be led from the top – academics tell students. For example, if students don’t realise that there will be no postal reminder of overdue books, then they won’t renew them and never realise that there is a problem and fines accruing. He says students need to know these things and who is responsible for telling them?

Adam suggests that course designers could put the course books needed for the course on the MLE and then the library could eventually link to them and notify students of the course book status, e.g. out on loan, on order etc.. Instead of typing in the

Technical and educational technology support. CETD.

Educational technology support.

MLE could improve support to staff and students by enabling access to databases without using more passwords.

Library access to include renewal facility.

More lead in time to ensure quality service and complete testing before going live.

Future and further developments of the library based on the MLE usage.

Student and staff training needs.

Policy decisions.

Future

33

book reference it could be connected to OPAC.

Trudy – Disability Unit

Trudy has responsibility for disabilities and equal opportunities. She assesses students’ disability needs. She had not seen the MLE interface working at the time of the interview.

She is looking for something which is an enabler and feels at the moment it is almost better for staff use than students. For staff it’s a tool whereas students bring attitudes from the wider social context and initially associate it with game playing and social use only.

Because of the area that she has responsibility for she says that there would be a need for a disclosure policy. Also a way of removing data from the site. At the moment there are about 1000 students who are disabled. She would need to think through a letter to be mailed to students about putting their disability details on the MLE and then what would happen next if students didn’t respond to the letter. She would need to get a temp in who could put in the information. Trudy says that she would need 6 months to get things running.

For the social activities area she would like specialised information on the MLE for disabled students such as which banks have wheelchair access. This information in available from the unit, but it would need uploading, as would information on university policy on exams. Other things such as sport facilities which are available to disabled students, which halls of residence are available to a wheelchair user and maps showing where disabled toilets are in the university buildings. These things are not available on the website but would be useful to have for the MLE.

She is looking for “punchy, dynamic student magazine design with punchy direct student language” for the MLE, but no shooting stars etc which are difficult for screen readers to read. “As a magazine it would be something to engage with rather than a text book when I have 80 other text books to read.”

Trudy is keen to have a blind student to test usability.

She says that she would need someone with the technical knowledge to support the unit’s needs. “I would not contemplate starting this without someone to help me. I would want a bulletin board so I could put meetings etc. on it. I would need training in using bulletin boards too.” She feels that without the support and training then everyone would stop using the MLE.

developments for library links via the MLE.

Differing target audiences.

Permission from students required for disclosure of information and the high importance on needing more time to implement the new policies for the Disability Unit.

Specific disability information would need uploading.

Attractive magazine style for the MLE, but with reference to the needs of disabled students.

Blind student to test usability of MLE.

Need for technical support and training in bulletin board

34

Trudy would also like to have a conferencing facility for disabled students, “something like a chat room.” Other needs she reported included the need for administration and policies and information on things such as were the nurses were on campus.

Five Post-graduate Students

This part of the evaluation was conducted after the students had accessed the site and the heuristic evaluation had been conducted by Mark Simpson. A second group of students were to be interviewed, but due to technical problems which did not allow students to access the MLE, no interviews were conducted with the second group.

The five students consisted of one woman and four men, 2 were previous DMU undergraduates and two students were from overseas. One of the students was physically disabled.

For some of the students the MLE login did not recognise their names and P number. However they were able to access the site using a generic password. Students liked having the photograph of themselves and that it recognised who they were.

Students commented that they liked working from home and thought the MLE would be useful in their studying from home.

Students did not understand what the calendar was for and how it could support them. They wanted to be able to add their own things to the calendar and add alerts etc. “Is the calendar for the student or is it just posted course information?”

Some students said they would change the font, but not the colours.

They thought that linking the library and OPAC would be under ‘DMU’ and not under ‘Resources’.

They did think the Social Area would be useful.

They were a bit worried about personal details being on the page and would like to change their passwords immediately to make such details secure.

use.More technical support.

Conference area for disabled students, again specific information for students.

Approval of login photo recognition.

Positive attitude that studying from home the MLE would offer support.

The Calendar - lack of understanding of what the MLE offered – interest in personalising but would need support/training.

Design issues.

Design issues.

Security concerns.

35

They would like to be able to choose who has access to their personal details and wanted to be able to request privacy so that not everyone could read their information details.

They said that if their exam scores were on the site they would be concerned that anyone could read them.

Students said they had no experience of conferencing systems.

They said the MLE “would be good if it really worked”. They felt that everyone has to know about it to make it work well. “Otherwise people won’t go back to it.”

They would also like their module tutors photos available to know who was who at the start of their course.

On the whole a positive response and appreciation of the MLE’s potential.

Policy decisions on privacy should be made clear to students.

Privacy/Confidentiality issues about exam results.

Sceptical about the MLE’s ability to always work. More publicity about the MLE required so all will use it.

Tutor photographs would be useful.

36

Appendix BMLE Pilot Academic Questionnaire

The aim of this questionnaire is to help us to improve the MLE system for academics. The questionnaire will take only a few minutes to complete.

Your Department ……………………………………….

1. How many of your modules use DeMontfort’s MLE? (please cross one box only)

a) 1 xxb) 2 xxc) 3 - 5 xxd) Don’t know xxe) None xx

2. How many of your modules use DeMontfort’s VLE, e.g. WebCT? (please cross one box only)

a) 1 xxb) 2 xxc) 3 - 5 xxd) Don’t know xxe) None xx

3. To what extent do you value the use of the computer in teaching your discipline? (please cross one box only)

Of great value Somewhat valuable

Not very valuable

Of no value

xx xx xx xx

4. Please indicate where you use the following DMU systems? (please cross all that apply)

Home Campus Other Not useda) Mailspinner xx xx xx xx

b) Computer Science system

xx xx xx xx

c) Intranet xx xx xx xx

d) Library xx xx xx xx

5. Over the last month how frequently did you use the MLE? (please cross one box only)

a) More than once a day xxb) Once a day xxc) 2 – 3 times per week xxd) Once a week xxe) Less than once a week xx

37

6. Which of the following MLE activities do you use with your students? (please cross all that apply)

a) Email (group or individual) xxb) Updating course information xxc) WebCT activities xxd) Room changes xxe) Assignments xxf) External links xxg) Course calendar xxh) Add Lecture Notes/PowerPoint presentations xxi) Past examination papers xxj) Quizzes/Self assessment questionnaires xx

7. Please list up to three things that you like about using the MLE.a)–––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––b)–––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––c)–––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––

8. Please list up to three things that you dislike about using the MLE.a)–––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––b)–––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––c)–––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––

9. The following questions relate to your experience of using the system. For each question put a cross to the response that you agree with.

Strongly agree Agree Disagree

Strongly disagree

a) I found learning to use the MLE was easy. xx xx xx xx

b) I found sending messages to individuals using the MLE was straightforward. xx xx xx xx

c) I found the experience of using the MLE frustrating. xx xx xx xx

d) I could find the things I was looking for quickly. xx xx xx xx

e) I need more training in using the MLE. xx xx xx xx

g) I need more technical support. xx xx xx xx

h) I don’t know what I am supposed to do to help my students learn through the MLE. xx xx xx xx

i) I think the way the MLE was presented on screen was attractive. xx xx xx xx

j) I need a help function when using the MLE. xx xx xx xx

k) The different features of the MLE were obvious to me. xx xx xx xx

l) I found reading messages and information was easy. xx xx xx xx

m) I need tips to improve my teaching presentation on the MLE. xx xx xx xx

n) I found changing my preferences easy xx xx xx xx

o) I have found more students bring queries to the MLE rather than to my office. xx xx xx xx

38

10. How has using the MLE affected your teaching? Please put a cross against all that apply in column A and put a cross against the most important in column B.

A Ba) My basic approach to teaching seems to be

changing, in part because of the opportunities that the MLE provides

xx xx

b) My basic approach to teaching has not changed, but the MLE helps me to do certain things better

xx xx

c) I have found the MLE of some value in my teaching

xx xx

d) It has not helped my teaching at all. xx xxe) I have found it of no value in my teaching. xx xxf) I have found it increased my work time xx xx

Please add any further comments you would like to make about the impact that the MLE has had on your teaching.

–––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––

THANK YOU FOR TAKING THE TIME TO COMPLETE THE QUESTIONNAIRE.

39

Bank Questions

Additional options for Question 8

Use the search facility to find something xx xx xx xx

Send a message to someone in your address book xx xx xx xx

Check Who’s Online xx xx xx xx

Which of the following do you use the VLE/MLE for in your teaching? (please cross all that apply)

(NB the wording of this will need to be adapted according to whether any of these are a possibility on the MLE or

only the VLE)

a) Introductory materials xxb) Simulations xxc) Skill development xxd) Student feedback xxe) Online Case studies xxf) Online assessment xxg) Student interaction with outside ‘experts’ xxh) Updating of materials/modules xxi) Monitoring student performance xx

Which of the following do you use a computer for? (please cross all that apply)

a) To run general purpose software (e.g. word processing, spread sheets) and printing xx

b) To run specific commercial software xxc) Email/communications xxd) Instant messaging (e.g. ICQ, MSN Messenger) xxe) Internet/World Wide Web - Information seeking xxf) Internet Shopping xxg) Microsoft packages xxf) Games xxg) File Transfers xxh) Socialising xx

40

Appendix CMLE Pilot Student Questionnaire

The aim of this questionnaire is to help us to improve the MLE system through student evaluation. The questionnaire will take only a few minutes to complete.

Your P Number ……………………………………….

1. How many of your modules use DeMontfort’s MLE? (please cross one box only)

a) 1 xxb) 2 xxc) 3 - 5 xxd) Don’t know xxe) None xx

2. How many of your modules use DeMontfort’s VLE, e.g. WebCT? (please cross one box only)

a) 1 xxb) 2 xxc) 3 - 5 xxd) Don’t know xxe) None xx

3. To what extent do you value the use of the computer in studying your discipline? (please cross one box only)

Of great value Somewhat valuable

Not very valuable

Of no value

xx xx xx xx

4. Please indicate where you use the following DMU systems? (please cross all that apply)

Home Campus Other Not useda) Mailspinner xx xx xx xx

b) Computer Science system

xx xx xx xx

c) Intranet xx xx xx xx

d) Library xx xx xx xx

5. Over the last month how frequently did you use the MLE? (please cross one box only)a) More than once a day xxb) Once a day xxc) 2 – 3 times per week xxd) Once a week xxe) Less than once a week xx

6. What kind of activities do you mainly use the MLE for? (please cross all that apply)a) Course-related activities xxb) Social activities xxc) General University activities xx

41

7. How often do you do the following things when using the MLE? (please cross one box only in each row)Always Sometimes Rarely Never

a) Send private e-mail xx xx xx xx

b) Use the Private Chat facility xx xx xx xx

c) Start a discussion topic xx xx xx xx

d) Reply to a discussion topic xx xx xx xx

e) Read messages within a topic xx xx xx xx

f) View the history of a message xx xx xx xx

g) Add an entry to the calendar xx xx xx xx

8. Please list up to three things that you like about using the MLE.a)–––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––b)–––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––c)–––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––

9. Please list up to three things that you dislike about using the MLE.a)–––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––b)–––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––c)–––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––

10. The following questions relate to your experience of using the system. For each question put a cross to the response that you agree with.

Strongly agree Agree Disagree

Strongly disagree

a) I found learning to use the MLE was easy. xx xx xx xx

b) I found sending messages to individuals using the MLE was straightforward. xx xx xx xx

c) I found the experience of using the MLE frustrating. xx xx xx xx

d) I could find the things I was looking for quickly. xx xx xx xx

e) I thought that the system was secure. xx xx xx xx

f) I found that the interface responded quickly to my actions. xx xx xx xx

g) I believe that the MLE has been designed taking my needs as a student into account. xx xx xx xx

h) I don’t know what I am supposed to do in the MLE. xx xx xx xx

i) I think the way the MLE was presented on screen was attractive. xx xx xx xx

j) I need a help function when using the MLE. xx xx xx xx

k) The different features of the MLE were obvious to me. xx xx xx xx

l) I found reading messages and information was easy. xx xx xx xx

m) I found changing my preferences easy xx xx xx xx

Please add any further comments or elaborate on any of your responses in the space below.–––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––THANK YOU FOR TAKING THE TIME TO COMPLETE THE QUESTIONNAIRE.

42

Bank Questions

Optional additional items for Question 6.

Use the search facility to find something xx xx xx xx

Send a message to someone in your address book xx xx xx xx

Check Who’s Online xx xx xx xx

Which of the following do you use a computer for? (please cross all that apply)

a) To run general purpose software (e.g. word processing, spread sheets) and printing xx

b) To run specific commercial software xxc) Email/communications xxd) Instant messaging (e.g. ICQ, MSN Messenger) xxe) Internet/World Wide Web - Information seeking xxf) Internet Shopping xxg) Microsoft packages xxf) Games xxg) File Transfers xxh) Socialising xx

43