Post on 07-Jun-2019
The Managed Learning Environment
at
De Montfort University
Report 6.2
“Managed Learning Environment – Usability Issues”
Anne Jelfs and Mary Thorpe
Institute of Educational Technology
Open UniversityWalton Hall
Milton Keynes MK7 6AA
July 2002
ContentsPage
1. Introduction............................................................................................. 3
2. Evaluation Strategy................................................................................. 3
2.1 Determine evaluation measures...................................................... 6
2.2 Pilot Groups.................................................................................... 7
2.3 Main Evaluation Study................................................................... 7
2.4 Review Student Performance Records............................................ 8
3. Findings................................................................................................... 8
3.1 Interviews with Academic, Library and Disability Unit staff......... 9
3.2 The Student focus group interview................................................. 12
3.3 A grouped list of findings from the interviews & focus group....... 13
4. Evaluation of MLE beyond academic year 2001/2................................. 15
5. Rationale for questions in the student and academic survey.................. 17
6. Conclusions and recommendations......................................................... 20
References................................................................................................... 22
Appendices
2
THE MANAGED LEARNING ENVIRONMENT AT DE
MONTFORT UNIVERSITY
(JISC 7/99)Evaluation Report 2
1. Introduction
This is the second formal report in completion of Work Package 6: User evaluation,
undertaken by the Open University in collaboration with the project team at De
Montfort University (DMU):
John Eyre, Project LeaderPat JefferiesMark Simpson
Open University Evaluation Team:
Professor Mary Thorpe, Director Institute of Educational Technology (IET)Anne Jelfs, Evaluation Project Officer, IET
Through a series of meetings in 2001/2, the development of the Managed Learning
Environment (MLE) and the strategy for its evaluation, both formative and
summative, was discussed at a number of project meetings involving Mary Thorpe as
the director of the Open University (OU) evaluation project, and since December
2001, Anne Jelfs as the project evaluator. At the meeting on 5th December 2001, a
draft evaluation outline developed by the OU in response to the JISC project proposal
and specifically WP6 in outline, was presented and agreed with DMU colleagues.
Details of this agreed approach and its subsequent adjustments are incorporated in the
evaluation strategy below.
2. Evaluation Strategy
Workpackage 6 requires four key tasks from the evaluation:
1. Determine evaluation measures with the OU and other interested parties2. Determine pilot groups3. Conduct surveys/interviews/focus groups4. Review student performance records
3
Two deliverables are specified from these studies – 6.1 Interim user evaluation report
and 6.2 ‘Managed Learning Environment – Usability issues’. Deliverable 6.1 was
completed in February 2002 and contains the results of interviews with pilot groups,
an initial evaluation of the CampusNet web pages and a report on accessibility issues
for MLE Demo pages, for users with a disability. The remainder of the planned
evaluation forms the second report presented here.
It should be emphasised that the evaluation plan in the project proposal was based on
the assumption that the MLE would be in general release for student usage during
2001/2, thus making user interviews and performance results feasible sources of data
for the evaluation. In the event the evaluation strategy has had to be adjusted to take
into account the changes in the timeframe for availability of the MLE to staff and
students. The MLE had not been generally released before the start of examinations in
2002 (and has not been widely promoted across the University at the time of writing),
and therefore it was not feasible to undertake extensive surveys on take-up and use.
Students’ study of their courses has not been influenced by the MLE since it has not
been made available to them, and performance data on 2001/2 is therefore irrelevant
to the evaluation.
The agreed evaluation goal was to provide an exploration of the usability issues for
students, teaching staff and support staff. Only those staff in departments that already
use technology in their teaching and in some cases have been involved in discussions
around the MLE, have been interviewed. One member of the Library staff was
interviewed and a representative from the disability unit. It would have been pointless
to interview outside this group since none of the staff has had access to the system and
therefore was not in a position to comment on it. The aim has been to find out as
much as possible in the circumstances about the perceptions of some of those in the
vanguard of usage of IT. Interviews have focused on the provision currently
available; their preferences about what they would like to have provided for support
of their teaching role, via the MLE and the University infrastructure; and their current
usages of IT for their own courses. The evaluation team were directed specifically to
focus on the MLE rather than details of VLE usage. However, the two cannot be
easily separated, since for teaching staff, their primary interests are likely to be in the
4
course-specific value-added of using a VLE in addition to existing teaching methods
and facilities. Further comments are made about this below.
The illuminative evaluation (see Parlett and Hamilton 1972) approach emphasises the
importance of the context within which innovations are introduced in education, and
this is still an appropriate approach in the early stages of introducing the MLE. The
use of semi-structured interviews conducted by the OU evaluator has enabled us to get
at both unexpected features of the infrastructure of the University as it affects IT use
for teaching, and to capture at least some of the preferences for what should be
available and how it should be supported.
Although it was agreed in meetings during March and April that a questionnaire
would not be a feasible strategy at this stage, the OU team agreed to prepare a draft
questionnaire that could be used in 2002/3 to evaluate the MLE at an appropriate
point after its general roll out. The draft questionnaires were discussed jointly with the
DMU team and the final amended version is set out in appendices B & C with a
commentary in section 5 below.
The heuristic evaluation (Nielsen 1994) was conducted by Mark Simpson and the
findings have been fed back to the project team as the basis for further developments
of the MLE.
Usability testing with participants from the target population of users is one
recommendation made by a number of researchers (Newman & Lamming, 1995,
Monk, Wright, Haber & Davenport, 1993). The value is that users can be observed as
they use the system and problems and queries identified. Computer users interact with
the computer system through a user interface, and for users that interface is the
system. It is knowing the user’s experience of the system which helps the designer to
understand the difficulties that users have and ways of improving the interface. Our
evaluation of the interface was conducted with the aim to identify how effective it was
in allowing the user to achieve their goals and how satisfied the user was with the
system. At the same time any problems that affect the usability are identified which
in turn lead to improvement of the interface.
5
The Managed Learning Environment incorporates a user-centred interface with a
single sign-on that provides a single site for students, teaching and support staff. It has
administrative, course specific and social elements, a calendar, email and personal
page http://severn.dmu.ac.uk:8080/mle. Staff who wish to use a VLE in their teaching
can do so, and in most cases staff use WebCT. The MLE is not currently designed to
include direct links to WebCT and staff and students access it via the University’s
intranet site
Details on additional information about building MLEs can be found on the JISC
website http://www.jisc.ac.uk/mle/7-99/. and a framework for Pedagogical Evaluation
of Virtual Learning Environments can be found on the JISC website
http://www.jisc.ac.uk/jtap/htm/jtap-041.html.
The specific data collection strategy was conducted under the four headings outlined
in Workpackage 6.
2.1 Determine evaluation measures
It was agreed that the basis for evaluation measures would be derived from semi-
structured interviews undertaken with both students and staff. A sample of staff and
students were to be provided by the DMU project team, from the areas with whom
participation in the development of the MLE in 2002 had already been negotiated,
including: Business, Law, Marketing, Pharmacy, and Computing. In the event this
proved difficult to arrange and the staff sample of seven academics were taken from
Marketing, Pharmacy, the Built Environment and Social Sciences. The five students
were recruited from one post-graduate course, M.A. in Design and Manufacture. One
interview was conducted with a member of the Library staff and a further interview
was conducted with a member of the Disability Unit (please see Appendix A for all
interview comments).
The content of these interviews included the following topics:
What has been your involvement in the MLE thus far?
What contribution to X (their course/s) do you expect the MLE to make this year?
6
What expectations do you have about how students will use the course related
elements of MLE for your course?
How often do you expect students to use these elements? How much time do you
think students might spend with these online elements?
In what ways if any do you think student learning will be helped through your
involvement in the project?
Do you have any goals for their development of the use of the MLE on your
course/area in the future?
These issues and the reasons for particular responses, were explored through semi-
structured interviews, and varied in the case of Library and Disability staff in line with
their roles and for students.
2.2 Pilot groups
The first stage of work with pilot groups included both interview questions and
observation of tasks, and the findings from the pilot studies was presented to the
project team in Report 6.1 Interim user evaluation report.
2.3. Main Evaluation Study
Conduct surveys/interviews/focus groups
Interviews
The expectation was that the MLE would be available at some point in March 2002
and data collection would be scheduled to take place after that, but before preparation
for the examinations became a major preoccupation for students. With the change to
the time frame of the MLE delivery, the evaluation study could not take place until
late May and early June. The interviews were based on demo versions of the MLE
made available to specific groups of staff and students for usability testing. The delay
in the MLE delivery did impact on the recruitment of both staff and students for
interview. Many students had returned home after their examinations and staff were
occupied with grading, exam boards etc..
The format of the student and staff interviews combined observations of usage
together with semi-structured interviews, similar to the approach reported on the
CampusNet interviews (Report 6.1 Interim user evaluation report). The names of all
those taking part in the interviews have been changed to preserve anonymity.
7
Focus Groups
Due to the impact of the later than anticipated completion of a fully developed MLE,
it was only possible to hold one student focus group. These students were post-
graduate computing students. Their reported comments can be found in Appendix A.
Surveys
As outlined above, the impact of the later than anticipated fully operational MLE
affected the proposed surveying of students. The work of the Evaluation Team has
therefore focused on creating questionnaires that can be used to evaluate the MLE and
its take-up over time (and beyond the end of the funded project - end of August 2002).
We were dependent on the delivery of the MLE within a relatively tight window of
opportunity, since until it was available, data collection could not begin. However,
the evaluation team has developed two questionnaires for the future evaluation of the
MLE. One questionnaire is for student data collection and the other is for staff data
collection. These questionnaires have been provided to the DMU team and copies are
presented in Appendices B and C.
The future evaluation and data collection from students will have to be sensitive to the
requirements of their timetables and availability before the examinations, particularly
since many are not available once these are completed.
2.4 Review Student Performance Records
Not relevant to the evaluation at this stage (see comments above).
8
3. Findings
3.1 Interviews with Academic, Library and Disability Unit staff
Seven academics were interviewed at the DeMontfort campus in Leicester using the
outline suggested. These were semi-structured interviews so that pertinent topics
would be covered although not always in the same order and with greater emphasis on
areas which related specifically to individuals and their concerns. A detailed account
of each interview is provided at Appendix A, and general themes summarised in the
commentary below.
The interface between MLE and VLEs in use
WebCT is the current learning environment in use at DeMontfort and most of the
academics interviewed used WebCT to differing degrees in their teaching. It proved
impossible to separate out only MLE-specific issues and this is not surprising given
the fact that most course-specific developments must occur in the context of the VLE
currently. Staff think about the MLE in relation to whether or not it enhances what
they can currently offer through the VLE, and whether or not the two environments
will be mutually supportive for students. The likelihood of students making frequent
and effective use of the MLE is seen in part as an issue of whether VLEs become
much more generally used by teaching staff across the institution, so that students
expect to be using computers as a regular and essential part of study, not just for
occasional social information or networking.
Some staff raised the issue of whether there should be a clearer University policy on
VLEs and whether a different platform was going to be introduced, such as
BlackBoard. The policy decision had not been made at the time of the interviews.
One academic said, “We’re waiting for the university to make clear what is happening
about learning environments. The faculty has been told not to make further
development until a decision is made.” When he asked when this decision might be he
was told September which he says is “actually when we will be short of time.”
9
MLE and its added value
One of the most active academic users of IT wanted to know what the MLE would
contribute and where it might duplicate other university provision. He would like to
have a greater understanding of what the students would view and how they would
use the MLE. He understands the MLE from his point of view, but not the students’
view. One recommendation by the evaluation team is that there needs to be
widespread workshops where academics, administrators and support staff are shown
the MLE, how it works and the benefits to all concerned. This includes giving
examples of student and other users viewing rights and specific pages.
The need for educational and technical support for academic users
One of the academics suggested by the MLE development team for interview was not
a user of any current learning environment systems. However he did conduct course
introductory sessions through distance education, therefore he was extremely
interested in the opportunity to use the MLE in his work. He is also a member of a
Leonardo project (European funding) for which he is the lead person. He wanted to
use a virtual learning environment such as WebCT in his teaching, but he feels he
needs support to do so. He wants not only technical support, but also pedagogical
support. He comments that he is appalled by the demise of the Centre for Education
Technology Development (CETD) as he needs that type of support particularly for the
Leonardo project and to enable him to put materials on WebCT.
All academics commented on the loss of the CETD support team at DMU. They all
thought that without the support they, as academics, were compromised and could not
dedicate time to the development of on-line materials, conferences etc for the MLE.
Reliability of the server and infrastructure essential
Academics also commented that the growth in student numbers and particularly
overseas provision and part-time students means that the MLE has to support these
students adequately and to do this it has to be available at all times. They were
concerned that without technical support and infrastructure the students would suffer
because of computer down-times.
10
Support infrastructure required in introducing the MLE effectively
Academics currently using WebCT as the learning environment were doing so to
differing levels. There were examples of academics fully integrating the use of the
learning environment such as in the marketing department and there were academics
who had no experience. As a random sample of the teaching staff this provides some
idea of the scale of the difficulties facing the MLE team. The anticipated problems
include the need for training, staff support in developing materials for presentation
within the MLE and the apathy of some staff in developing a strategy to include the
MLE in their teaching.
There are also reported policy difficulties. Academics report that there is no clear
university strategy for the development of managed and virtual learning environments
and how these should be incorporated in their teaching. As one of the academics said
“We have existing WebCT users and those keen to develop web-based learning. We
need school-based developments. We need to know what is going on.”
Staff acknowledge that the key to the successful uptake of the MLE is reliant on
academics incorporating the MLE in their teaching. However, there is concern that
they do not know how to go about this and the emphasis they should give to using the
MLE. Without full academic staff support and their input into the MLE it could
quickly become an unused resource.
Specific issues – email accounts, logging on, etc
Staff said that because students frequently attend university with current email
accounts such as Yahoo and Hotmail they have found that students do not use their
university email accounts. Therefore staff are concerned that the email via the MLE
might also fall into disuse. There were no suggestions how this could be overcome.
Academics also commented on a need for consistency of logging on to the system and
they questioned whether the initial log password for students should be their date of
birth, also the fact that the date of birth was in the US format of month, day, year.
Library staff would like to be able to develop the facility for students to renew and
request books and visit electronic databases from one single log-in. They are also
11
interested in knowing the actual figures of student log-ins to the MLE, and to provide
students in the future with notifications of a book’s status.
Specific issues for users with a disability and the Disability Unit
Staff at the Disability Unit would primarily need more lead in time to develop a fully
functional database of users with a disability. Disabled students need to be notified
that their information will be available to their tutors and given the opportunity to opt
out of this. The member of staff concerned would also wish to have liaison with the
MLE team about specific information for disabled students, such as banks with
wheelchair access. This type of information is not currently available on the DMU
web site, but would be useful to include on the MLE. She would need additional
support to both enter student data to the site and additional support to provide specific
pages for the MLE.
She would also like disabled students, particularly blind students to test out the MLE.
Some items she suggests may be unreadable by a blind student’s screen reader.
NB: A full analysis of the accessibility of the demo web pages has been completed by
the OU evaluation and detailed recommendations made, please see Report 6.1.
The Disabled Unit would also like a conference site to be set up specifically for
disabled students, although that would not preclude them from joining other
conferences.
3.2 The Student focus group interview
The student group interview was conducted after they had completed a session using
the demo version of the MLE. They liked most areas of the MLE but were concerned
about privacy and confidentiality. Part of the reason for these comments may have
arisen because they were post-graduate computer faculty students with an interest in
IT and design. They did suggest some design issues such as changing the font on the
MLE and moving the library site to the resource section and not the DMU location.
They saw the value of using only one site and one log-in, but didn’t fully understand
how it would be used and why the need to change from the current web site. More
use by the students and academics should alleviate this problem.
12
Both students and academics mention that they wanted something more magazine-like
and ‘punchy’. They felt that the interface at the moment was a little bland. As Trudy
from the Disability Unit commented, she wanted a “punchy, dynamic student
magazine design with punchy direct student language. As a magazine it would be
something to engage with rather than a text book when I have 80 other text books to
read.” At the moment the MLE portrays an image of a valuable administrative tool,
what it does not provide is the appearance of an interesting student-centred site.
3.3 A grouped list of findings from the interviews & focus group
In this section we discuss the issues which arose from the interviews. This includes
both staff and students’ concerns and these are grouped under the headings of
technology; teaching and student support; policy decisions; design; and general issues.
Technology
Academic staff in particular were concerned over possible computer
downtimes as they felt that downtimes would negatively influence student
interaction.
This also means that current problems with initial logging on to DMU’s
computing system need to be addressed immediately.
Both academics and students were concerned about student technology
support and problems with access to computer equipment.
Teaching staff wished to know if all current software programs used by the
University would be compatible with the MLE.
All interviewees to some extent referred to the requirement for technical
support.
Teaching and Student support
Staff recognised an increase in student numbers and an increasingly dispersed
student body required additional electronic support from the University and
the MLE was one such provision.
Academics value up to date class lists at the start of their courses and would
prefer the MLE to provide these earlier than the current system.
Teaching staff would value the opportunity to offer individual support and
diagnostic tests e.g. quizzes, to students via the MLE.
13
Academics had found that where WebCT has been used effectively there was
a reduction in student visits to academics for purely information needs and less
important items. A valuable point with the growth in student numbers.
It was hoped that the MLE would provide some of students’ pastoral needs
and therefore increase student contact with academics for subject-specific
reasons.
Interviewees needed training in the use of the MLE – workshops and
demonstrations.
There was a general feeling that the MLE would enhance student studies and
social support.
Academics were interested to find out how much students do engage in
discussion areas and what they can do to improve student engagement.
Academics think they would need additional training in the best use of the
medium for teaching.
Policy Decisions
A need for senior management leadership on the adoption of the MLE and
policy decisions to be made on the overall use of the MLE and VLE at DMU.
Academics commented on the time pressures that adopting the MLE places on
them without the support of CETD and requested staff to be made available
for ongoing support, both educational and technical.
Design
Interviewees commented that they would like a more magazine-like interface.
Library staff wanted to include library book renewal facility in the MLE.
General Issues
Monitoring of students’ and academics’ use of the MLE site – this enables
modification of sites through understanding of current use and is additional
date for the evaluation strategy.
One suggestion was made to have a standardisation of material presented for
publication on the MLE, although other members of staff did find the use of
templates unnecessarily restrictive.
There was a reported need for widespread publicity of the MLE to academic
staff, students and support areas.
14
4. Evaluation of MLE beyond academic year 2001/2
A number of options are available for evaluation beyond the end of the JISC funded
project. These should be clarified and reviewed in the light of decisions about the
specific goals of the MLE and how it will be implemented across the University.
Options include the following:
a) evidence of take up and usage by students of all areas of MLE, separately from
VLE usage
b) exploration of the experience ‘behind’ the data from a) i.e. reasons for usage,
whether high or low, in terms of performance of the interface, reliability of the
server(s), ‘fit’ with student behaviour and preferences, etc
c) exploration of staff usage and experience of MLE, in parallel with a) & b) above,
and including issues around the availability of support to them for use of the facilities
offered
d) evidence of usage of VLEs across all departments, and comparison with pre MLE
usage, for comparison, plus exploration of the reasons for this usage, including the
availability of support to staff for use of VLEs in their teaching
e) staff perceptions of changes in (i) how they teach their courses (ii) student
behaviour & performance, following on changes shown in d) above
f) Library and administrative staff usage and perceptions explored including their
perspective as key providers of data for the MLE system.
g) specific issues – qualitative evaluation allows for unexpected outcomes to become
apparent to the evaluators, and where these arise, to be followed up and incorporated
into the evaluation as it develops. It would be worth following up any indications
from staff that student performance or the quality of their learning was improving – or
indeed being damaged – by either the MLE or VLE developments. This seems a better
strategy than looking to find quantitative improvements in student performance as a
direct result of introducing either the MLE or VLEs. Performance data is so likely to
be influenced directly by a range of other issues, that it would be very misleading to
draw specific conclusions without extensive evaluation. This does not preclude in-
depth exploration of specific cases, as in the interview with ‘Tina’ (see appendix A),
who feels pass rates have improved through VLE use.
While the resources available for evaluation will obviously determine how much of
what is outlined above can be evaluated directly, we would in general recommend that
15
1) a system of collecting data about usage of both the MLE and WebCT (or
whichever VLE is used) is set up for the start of the autumn term, and covers
both staff and students
2) qualitative interviews and group discussions are undertaken before
questionnaire based surveys are done. Surveys are best undertaken when the
parameters and the content areas of problems and issues are broadly
understood and can be accurately ‘coded’ into clear and coherent questions.
Amendments to the draft questionnaires might then be made before usage.
The schedules that might be used in such interviews can be adapted from those
already used in the evaluations reported both here and in the earlier report.
If resource is available for providing a programme of support and staff development
around the introduction of the MLE, as recommended from the comments made in the
staff interviews, then it would be feasible for the project staff to follow an action
research model. An action research model is based on an integrated and planned
programme of data collection and development, in order to evaluate provision at key
stages, and to generate formative feedback that can be used to progressively improve
the MLE and the facilities and arrangements around its introduction.
In general, a long-term perspective should be adopted, looking ahead to a 3 to 5 year
period of development not just one year at a time. Change on the scale envisaged by
MLE or similar technology introduction is not completed in shorter time frames, and
the great value of data collection is to be able to review trend data over several years,
rather than one year only. A small amount of evaluation carried out consistently over
3 years might be just as, if not more useful, than a large amount that does not go
beyond 2002/3.
Among the many sources of information about evaluation in this area, Oliver and
Conole (1998) provide a very useful overview.
5. Rationale for questions in the student and academic survey.
The evaluation team agreed to produce draft questionnaires and rationale for each
question set, so that the Project Team could use these as the basis for subsequent
evaluation. This section sets out these requirements. It should be noted that the
16
questionnaires provided with this report (see appendices B can C) can be modified for
electronic use and the layout is only for report presentation purposes. The timing of
delivery of the survey is left to the Project Team, but as a longitudinal evaluation
there would be benefits in conducting the survey at differing points within the year to
look for variation in take up of the MLE.
Academic Survey
The first two questions in the survey refer to baseline data concerning the number of
modules where the respondent uses either the MLE or a VLE. The baseline data
provides information on staff use of the MLE and VLE, which can be compared with
each other, and these first two questions can also be correlated to further questions
such as acceptability of the system and technical needs.
Question 3 is a measure of the value placed on the use of computers in their teaching.
This acceptability measure can then be used to compare with adoption of the MLE
and VLE and also Question 10 on how the MLE has affected the academic’s teaching.
Question 4 looks at the locations where academics are accessing DMU systems. The
question is looking at whether academics are relying on systems based on- or off-
campus. This has relevance for supporting academics in differing situations.
Question 5 refers to the frequency that academics use the MLE. Again, this has
implications for support, especially if they are accessing the site off-campus and the
question also provides vital data on frequency of use. This question may be
something that could be used at differing points within the academic year. Teaching
staff may be using the site more frequently during the first few months of the new
academic year or the start of modules and less towards the end of the year.
Question 6 in the academic survey will provide evidence of the take up of specific
features offered by the MLE and a measure of the degree to which MLE activities are
in use across the University. Departmental differences can also be revealed through
comparative analysis across the responses for each feature.
17
Questions 7 and 8 on the academic survey provide open-ended free response
comments and therefore provide qualitative data on the perceptions of staff about the
MLE. Qualitative comments are a valuable addition to the questionnaire because it is
easy for survey designers to omit questions and issues which are pertinent to the user.
Requesting open-ended comments invites users to put issues into their own words and
to express issues important to them. Qualitative data provides respondents with the
opportunity to be reflective about their personal experiences and to address their own
needs.
Question 9 relates to their experiences of using the MLE interface and this offers the
project team information on areas for change or increased training for staff.
Question 10 refers to the ways the MLE has affected their teaching, plus the issue
they rate as having the most effect on their teaching. It is anticipated that this would
provide data on perceived changes to their pedagogical approaches.
There is also a final section where staff can again make a written response to any
aspect or issue arising from the MLE.
Student Survey
The first two questions in the student survey are the same for students as they are for
academics. These questions refer to baseline data concerning the number of modules
where the respondent uses either the MLE or a VLE or both. The baseline data
provides information on student use of the MLE and VLE which can be compared
with each other and these first two questions can also be correlated to further
questions such as acceptability of the system and technical needs. There can also be
some comparison between student usage and teaching staff usage of the MLE.
Question 3 is a measure of the value placed on the use of computers in their studying.
This acceptability measure can then be used to compare differing student based
cohorts: discipline/course differences; gender; part-time/full-time students.
18
Question 4 looks at the locations where students are accessing DMU systems. The
question is looking at whether students are relying on systems based on- or off-
campus. This has relevance for supporting students in differing situations.
Question 5 refers to the frequency that students use the MLE. Again, this has
implications for support, especially if they are accessing the site off-campus. The
question also provides vital data on frequency of use.
Question 6 for the student survey refers to how students are using the MLE and
question 7 refers to frequency of use of specific items within the MLE. Again,
departmental differences may become apparent from this question. As part of the
evaluation strategy this information could be fed back to course designers and module
leaders for the integration of specific uses and course design.
Questions 8 and 9 are open-ended free-response questions referring to their
perceptions of the MLE.
Question 10 is the same as question 9 in the academic survey and relates to their
experiences of using the MLE interface. This offers the project team information on
areas for change or increased training needs for students.
Finally students can make further free-response comments on the MLE.
There is a small bank of additional questions provided for each survey which the
project team may wish to include or to add at some later point. It is suggested that all
students and teaching staff are emailed the survey and the collected data fed back to
interested groups e.g. academic boards, student union bodies, disability and library
units and administrative departments.
The project team can gather qualitative data in the future either by face-to-face
interviews with staff and students or through focus groups. It is recommended that all
qualitative data is audiotape recorded or digitally recorded. With focus groups or
interviews a semi-structured interview approach is suggested, i.e. those topics of
importance are specified in an interview schedule, but that the delivery of the
19
schedule is responsive to the experience and identity of the interviewee. The
interviewer also has to attend to the responses so that a dialogue can develop whilst at
the same time ensuring that all the topics are covered. It is suggested that the topics to
be included are similar to those conducted for this evaluation, i.e. actual usage,
anticipated future use, effects on teaching and learning, and experience of the system
in use.
It is not anticipated that large numbers of qualitative interviews will be undertaken,
but in specific target areas or with discipline based cohorts of students. These may be
target groups selected from areas of concern identified from the survey and usage
statistics analysis.
6. Conclusions and recommendations
Evaluations of IT introduction into complex organizations such as universities and
colleges have systematically highlighted the impact of existing custom and practice of
staff and students as key in mediating the impact of the technology and its effects. The
project leaders are aware of the many challenges they face around these issues, as well
as those over which they have more direct control such as the delivery of the MLE
itself. They have succeeded in many of their immediate aims and a summary of some
of the key recommendations to arise from the work of the evaluation is as follows:
The evaluation of the MLE by the project team based at DMU should combine
quantitative and qualitative measures, and be designed according to the
resources available and with the long-term interests of developing high quality
student learning over a 3 to 5 year time scale.
Development of the support to academic staff is absolutely essential, for
effective use of both MLE and VLE. This was stressed by all staff, most of
whom are in the forefront of technology use and with expertise well beyond
the average. All wanted both pedagogy/educational and technical support.
Clear University policy on the adoption of the technology is called for.
Academic staff were unclear about strategy and felt that a University lead was
required.
There was an evident need for more lead-in time for the introduction of the
MLE. Staff also wanted a participative process, giving them opportunity to
20
come to terms with the facilities on offer and to see how they ‘fit’ with
existing practice and how that can be developed. There is a lack of awareness
and a programme of publicity and ‘hands on’ workshops would meet the
needs.
Training/workshops on the MLE as part of the above, are urgently required
A more magazine-like interface for MLE was wanted, to present a style that
students in particular could identify with – MLE needs to pull students in.
Library book renewal and book availability notices would be valued features.
Specific disability information requires careful handling, and the needs of
particular sections of the University such as the Disability Unit need to be
accommodated – it is unlikely that a ‘one size fits all’ strategy will be
successful.
Implementation of the accessibility recommendations should be done,
including achieving Bobby approval of the site, as set out in the earlier report
6.1.
Logs of student and academics’ use of the MLE site should be set up and
analysed as part of the evaluation, as outlined above
21
References
Monk, A., Wright, P., Haber, J. & Davenport, L. (1993) Improving Your Human-Computer Interface: A practical Technique. New York: Prentice Hall
Newman, W.M. & Lamming, M.G. (1995) Interactive System Design. New York: Addison Wesley
Nielsen, J. and R. L. Mack, Eds. (1994). Usability Inspection Methods. New York, Wiley.
Oliver, M. and G. Conole (1998). "Evaluating communication and information technologies: a toolkit for practitioners." Active Learning 8: 1-6.
Parlett, M. & Hamilton, D. (1981) “Evaluation as illumination: an new approach to the study of innovatory programs” (originally published as Occasional Paper Number Nine, University of Edinburgh Centre for Research in the Educational Sciences, 1972) in Parlett, M. and Dearden, G. (eds) Introduction to Illuminative Evaluation: Studies in Higher Education, SRHE, University of Surrey, Guildford
22
Appendix A
Managed Learning Environment Interviews
Interviewees:
Tina Lecturer in Marketing
Alan Interviewed together. Both working in the Built Environment department Charles
Dennis Head of Marketing Module. Interviewed with Mike
Mike Pharmacy – Teaching Fellow
Charles Follow-up interview
Gary Social Sciences
Adam Library
Trudy Disability unit
Student Focus Group
MA in Design Manufacture
Tina - Marketing
Tina has worked with CampusNet (the system developed at the Milton Keynes campus) for the last couple of years. This was primarily instigated by the size of student recruitment to the module. This course is delivered to over 10 sites, including franchised sites, to approx. 1000 students. This includes sites overseas. She wanted to help students in their studies and to get information to students effectively. The student body has a lot of people who are working and they need material available at all times. She does however say that WebCT is not used in isolation “The VLE is integrated with traditional ways. They still need to use a textbook and they can’t not come to tutorials and lectures. Its part and parcel of the delivery.”
According to Tina the key thing that it adds to the course are quizzes from the lecture notes, as formative assessment. Says it has made such a huge difference to the pass rate. Depending on which programme they came from it made between 7-20% difference in their overall grades. (She found this out by tracking students’ formative and summative scores). Its that which has interested her colleagues. The quizzes are only multiple choice and the increase in overall grade was irrespective of whether they attended tutorials or not.
Experienced user of WebCT.
Used to cope with increasing student numbers.
Facility to help students in their studies.
Formative assessment – linked to improved overall module grade.
23
Students say they really enjoyed it. The only problem Tina has is that of lack of student engagement with the discussion area. So much so that they have invented an imaginary student to start discussions and that still didn’t work. Students did use it to apologise for not attending tutorials. Therefore she says that when they did come to her, it was not to apologise for missing a tutorial, it was for subject-specific reasons – “it raised the game”.
Tina has left messages on the site for students but because the first part of the site is so ‘busy’ the important messages get lost. There needs to be something to raise students’ awareness of important messages on their modules.
WebCT helps students at franchised sites to feel part of the community – and that feeling helps the staff at those sites as well.
Says that they used to have tutorials every second week, but now there is a move to weekly tutorials. This means that if students miss three tutorials in succession, she can contact them via email and check why. In turn she needs to have accurate class lists, which aren’t always available. Students can’t miss tutorials because the end of module assessment is tied to the tutorials.
Tina is not sure about the statistics from WebCT on the number of visits people are supposed to have made to the module. She is unsure if students who go to another site and then return are counted as two visits or only one.
She uses it to see where students are looking and finds this useful for students who are having problems. Says you can see if the student is disengaged and not looking at the site. Plus she uses it as feedback on what students are using and improve the site accordingly. She also uses it for assessment and key skills teaching. Has found that students want web sites for other modules. However, she thinks that not all modules “lend themselves” to a web site.
She knows that colleagues have developed their own web sites but she needed someone to help her to set up what she wanted.
Provides students with training session on using WebCT in Week Two and not Week One, because she recognises that students just starting university are nervous and don’t take everything in. She stage releases information on module
Student feedback is positive.Difficult to get students to use discussion area. Students usually only visit staff to ask course specific questions.
A consideration for the MLE is that important messages need to be presented clearly and on the first page.
Helps staff & students at franchised sites feel part of the academic community.
Monitoring of student attendance. Need for accurate class lists.
Clarification of what the statistics really measure.
Quality of course – student feedback.
Technical support required to help develop own site.
Provides students with training. Stage
24
including key skills.
Has found that the use of WebCT has meant that she does not get as many questions on ‘where do I go to get this?’ and neither do her colleagues. It means that when a student comes to the member of staff with a problem it’s not trivial and they can be given more time because the trivial problems have gone away. However, she thinks that if students have a problem with ‘logging on’ in the first couple of weeks then it’s a barrier and it takes ages for them to come back. There are also problems when students keep changing their passwords. Passwords are given in tutorials and lectures. These usually include date of birth and sometimes the month and day are transposed (US version of dates) and at other times not, so there is no consistency.
One problem that Tina can foresee is that more use of on-line materials and sites will cause difficulties because the Faculty only employs one person half-time to provide both educational support and a technical service (other half of the time he is in another faculty). She says that there was support from an internal team but that this team has now been made redundant.
Tina does not give out handouts, so students have to go to the website and use the textbook and engage with the area of marketing. The number of students who don’t use the web site is so small that to her it’s no problem.
To encourage student use of the web site she gets students to work in peer groups because of the numbers and then has 3 un-staffed lab bookings so that students can go in at that time.
Tina thinks that a major barrier to the students using or not using the site is due to the staff and to counteract this she conducted a Module Staff Development day. The key themes for the day were based on assessment and dissemination and this included web use. Because of this the staff started to use it and she feels that impacts on how much students use it.
She also has a tutor-to-tutor group, so that private items for tutor use are put there. Has produced tutor guides and resources, so that tutors have material for sessions.
She also puts a lot of links to course related sites on WebCT.
If a lecture is cancelled the information is put on the WebCT site and if there is time then it is put on the door. Room changes are
releases information.
Releases tutors from answering trivial questions.
Problems with logging on are a deterrent to student interaction with the system. Hopefully this can be avoided with the single sign-on in the MLE.
Increased support from University infrastructure. Loss of internal support team has been a problem and will continue to be one.
All handout material is on the web site.
Understanding student needs and problems with access to computing equipment.
Staff Development and how staff use impacts on student use.
Tutor to tutor group. Tutor guide and resources.
Course materials
Cancellation notices and room changes
25
also put on the WebCT site.
She thinks the new MLE pages are very busy and needs to be more ‘staged in’ i.e. less information on each page and more ‘hot’ links to other pages.
Alan and Charles – Built Environment
One of the things they mention at the beginning of the conversation is the demise of the computer support department. The loss of which has not impacted on them as much as it could do with other academics.
They say that because they teach with ICT what they wanted in terms of support was not so much technical as someone who could help them make the correct decisions about the learning implications for students.
In their teaching they use a range of programs From Excel through to 3D Studio and CAD. They want to use systems the same way as the building industry and to use industry-wide programs in their teaching.
The school uses WebCT to support lectures. The material is usually produced off site and then hosted on WebCT. Alan and Charles bring to their lectures lap-tops so that their material is available electronically and only needs downloading to WebCT of the module site if they wish to make the material available to students. They have whole modules that are on-line and some formative assessment. Recently they had looked at WebCT data and found a number of students had not used it, which is a cause for concern as they present material and assessment for students. Normally Charles’s lectures are in a room with linked machines and a digital projector so using WebCT for him is no problem.
They say that their typical use is:- module guide on-line- project briefs- directed reading with links- class notes- on-line support material which is also available as hard
copy- guide to CAD and FrontPage
- used to replace overheads.
are put on the site.
Design of MLE
Loss of technical support
Need for educational technologist.
Specialised technical needs.
WebCT to support lectures. Number of modules are on-line.
Case studies and
26
Alan and Charles use WebCT for case studies and have a presentation area that students can also show potential employers.
Charles and Alan would like to be able to change student enrolments. Class lists are not received in a timely way.
Acknowledge that some academics would not see development of material for Intranet/WebCT as relevant to them. Charles and Alan say that what they personally need is a support system where there has been a dialogue between staff to develop the site. This is not always technical support, rather they want people who understand the needs of the teaching staff.
Alan and Charles tend not to use the module email as students have usually got Hotmail and University’s mail. They have part-time students who also probably have access from work (these are day release students.) Students provide their preferred email address.
There are important differences and variation of prior knowledge of computers by students.
They say the MLE calendar is tedious to use and has a tricky interface. They say that they could make a lot more use of the calendar and would like to be able to put important information directly on to the student’s calendar to remind them of important dates/deadlines etc..
Maintaining areas is difficult. If you use any sites then you need support to maintain them and keep up to date.
Charles develops his own materials and is a competent user of electronic communication. He fears that the MLE provides too uniform a site and that the over use of templates would inhibit innovation – so house styles might be prohibitive to him in developing his own module presentation on the MLE.
Dennis and Mike – Marketing and Pharmacy
Dennis says that a major problem for him is student support and making support easily accessible to students. Whether this is their results or module choices, he wants to make it easily available. He likes the idea that students have one central site to
presentation area.
The MLE would be one way to provide up-to-date class lists from registry.
Need for educational technologist.
An issue for the MLE is the email provision and a single site login to the email system.
Students have differing levels of skill.
Staff are interested in how to get the most from the calendar feature.
Technical support
Use of house styles may prohibit the innovative academic.
Student support
27
start from. Says there is a sense that it’s all to do with administration on some of the MLE but does not underestimate the value of administration in supporting students.
Dennis says that the more we can release staff from the burdens of making sure that ‘stuff’ is accurate on student details, the better. More time can then be used on academic input. This is important at the moment in getting marks back to students.
Academic use of home computers and internet are hidden costs that the academic picks up. Staff and students access from home as well as on site.
Too frequently the web-based material that academics put on is the text-based material they already have and this is not the best use of the system.
Dennis says that there are publishers offering on-line material or companion websites. His dept needs to look at the potential and the opportunities.
The ‘Principles of Marketing’ is the nearest they have to an on-line course (Tina’s course). It is actually a taught programme supported on-line. It is not intended to be an on-line course but it does have material available on WebCT. This course provides standardisation of materials for tutors and students.
There is very little on-line assessment in Marketing. What they do have are multi-choice testing part way through called ‘phase testing’. They have this type of formative assessment because they are concerned about the security of testing and would not want summative assessment unless they could be sure of security and policy etc. They also want to give good quick feedback. This is one way of addressing the challenges of large student numbers.
Dennis says he’s not sure if students want to send summative/formative assessment on-line but sure that staff want on-line assessment. The staff are concerned about plagiarism and in electronic form they could check for plagiarism, because of the programs available to do so.
Sharing material with students and staff is useful on-line.
including administrative support.
Staff supported by accurate student details and dissemination of exam results.
Academics use own hardware and phone. These are costs to the academic.
Poor use of the technology
Review and compare commercial on-line material which is available.
Use of WebCT to enhance an existing course and to support staff & students
Formative assessment
Fast feedback to students and computer based feed-back enabled scaling up to meet growth in student numbers
On-line assessment and plagiarism.
28
Staff are reticent to provide feed back on-line because it is felt that it increases the workload due to slow typing speeds etc.
Dennis says that the dept needs to find out if students are printing the material – the costs of the text is then passed to the students. Says ‘perhaps we are not using the technology in the right way and students print out because of that.’ Students can’t annotate on-line.
The faculty paid for the use of the Centre for Education Technology Development (CETD), which was closed down. Now the money is not given back to the faculty and so are trying to do the work without it (Dennis has only .5 of a person). Dennis illustrates his point through an analogy – when buying a car he doesn’t want to know how the engine ran or become a mechanic – wants to be able to use it – like going to a garage he wants to have it maintained and repaired.
They want educational technology expertise and how best to do things. Half a person is trying to support 170 staff in Business and Law.
CETD has gone and they ask ‘where do I go to when I have glitches with things e.g. WebCT? There is a technician who supports the server but not what I always want.’
Mike assumes that students come in with basic computer skills and uses WebCT to give students a discussion area. He has four years of evaluations of its use and students think its great.Dennis would like to know how students’ evaluations and impressions are made when some are on-line and others face to face.
For Mike his students say that they think its great and think it’s ‘sexy’ and want all modules to use on-line material.
Dennis says that this means that students may think that without on-line materials the course is not adequate. Also who should develop the skills to put material on-line?
Mike, very strongly criticises the decision to close down CETD. ‘The University unbelievably closed down the department that could help’.
Sharing of material online
Slow typing speeds impacts on on-line feedback
Need to understand student usage.
Loss of academic computing support causing major difficulties to the departments.
Inadequate educational technology support.
Students like using the on-line facilities of WebCT. What are student evaluations of on-line material based on?
Pharmacy student opinions of on-line materials very positive.
Whose responsibility is it to develop skills for on-line material?
Need for CETD or equivalent to return.
29
Mike says that he doesn’t think that putting material on the web saves time for students. He has one student who has looked at his site since the beginning of the semester @ 539 times. However, he thinks this is because the student is over compensating because she has come from the HND route.
They say that not all members of staff are up to speed on using computers and programs. Dennis asks what is the policy of what an academic should be. Says: we used to have the support for typing now we don’t, we do it ourselves. Are we saying that academics have to learn this type of technology? He means new software programs etc. They suggest that some people are frightened of trying. Which in turn has implications for resourcing and the policies surrounding resources and that is not just the institution. Mike says that compared to Coventry (University) it is ‘ad hoc here’. There are pockets of interested people, but not across the board
One person in the business department has produced their own web site, so that it can be on the Intranet, but this is someone who is technically able.
They comment that ‘we are not just expected to teach, but also to do QAA, RAE, funding’. They say that whereas 5 years ago they could look at web-based learning, now there are competing things for an academic’s time, so they appreciate that some people are reticent.
At the same time they say that more and more students are working and working more hours. There is a danger that full-time students are studying part-time. They question where the increase in student employment hours are to support their life styles, as nearly every student has a mobile phone.
The student list – Dennis says that ‘if we get one by day 21 we are happy.’ Whereas Mike says that pharmacy has access to their own electronic student records, but this is only due to their own efforts. One of the problems that Dennis says marketing has is how other faculties put their information on the system, and whether they do it at all. ‘We find sometimes 30 students there which a faculty hasn’t registered’.
When Dennis had seen some examples of how WebCT had been used they all looked different. If a student has a number of modules then it can cause problems. He would like consistency and at least consistency in terminology. ‘We need agreement or we end up confusing students.’ However as they say some people welcome conformity when they are starting out and need guidance.
In appropriate study strategies revealed – online learning strategies need more exploration.
What are the University’s policies on using technology and skills development?
Ad hoc development of on-line resources.
Time pressure on academics. QAA, RAE and external funding also make time demands
Student employment
Up-to-date student lists
Consistency of terminology and usage across modules.
30
They both feel that the loss of CETD was a retrograde step and that CETD could have provided guidance to the academics. Including design features e.g. use of an off-white background to cut down glare.
Mike has obtained from his Teacher Fellowship budget a hand held device. He is looking into ways of developing hand held delivery to students and student learning, instead of students having to buy laptops or pcs.
It is a Compaq on which he has put his own material and ebooks from the Internet. He is wondering if this is a viable way for student learning. It might not be at the moment because it costs approx £500, but he thinks costs will go down. His lecture notes are easy to read on the device.
The hand held Compaq uses Blue Tooth (a commercial product – wireless and enables links between mobile computers/mobile phones/connectivity to Internet) in a separate sleeve. It has a full screen and not a half screen which Palm has. Mike has put some PowerPoint it and has a separate VGA card, which at the moment he doesn’t know what it looks like linked to a projector.
He says that in pharmacy these wireless devices are used on the wards to bring up patients notes and to prescribe.
He thinks students would be attracted to these if they were £100. He would want the MLE to be able to interact with these types of devices.
Charles (second interview)
He wanted to complete a second interview 4 months later to comment on the developments of the MLE and the impacts that the MLE will make on his work.
He says he is waiting for the university to make clear what is happening about learning environments. The faculty has been told not to make further developments until a decision is made. When asked when this might happen, he replied that he was told September which is the point when the academics are short of time. He says the lack of information is difficult to deal with.
Charles says we have existing WebCT users and those keen to
Loss of educational technology support
Development of hand held materials.
Innovation of teaching.
Developing future employment skills.
MLE currently provides different pages for different devices – will hand helds be included?
What are policy decisions?
31
develop web-based learning. We need school-based developments. We need to know what is going on.
He would like to know what the MLE would contribute i.e. what will be duplicated and where do students go to get particular information. He feels he needs a greater understanding of what the students will have available from the MLE.
Charles can foresee lots of bottle necks at the moment. If the MLE adds to admin ‘hassles’ then he sees problems. “We just want to deliver additional course material and don’t want to have more hassle.”
He says he would like a ‘year’s grace’. He has no difficulty in using the MLE, but he thinks that if academics feel out of control then it will fall down.
Charles says that the interface is missing some interest. It feels like you go there if you must and is not a fun, attractive site.
Says the MLE needs to offer something to the academic for teaching and the student needs to find things easily. He says it is a case of “Does this bus go where you want it to go?”
He would like to know who is going to be updating the Social page – student union?
Gary – Social Science
Gary has no experience of the MLE. but refers to himself as an interested academic.
He works in the Social Work department and wishes to expand his introductory sessions which are currently held on Saturdays to a Distance Education Course.
He wants to use WebCT and needs support to do. Not only technical support but pedagogical support. He is appalled by the demise of the CETD department a couple of years ago. He says
Electronic teaching needs.
Training needs in what students will receive from the MLE.
MLE interface with academics needs to be sufficiently flexible to accommodate the sophisticated user, motivated to progress quickly to using the MLE/VLE for teaching.
Needs more time for implementation.
Layout and design issues.
Design and teaching issues need to be more obviously interesting and motivating.
Who updates the social page?
Development of distance teaching.
32
that he needs this form of support.
He is also part of a Leonardo project for deaf students and is the lead person for this project. Again he says this requires technical support for the project and the materials to be made available on WebCT. Without the support to do this work he feels frustrated.
Adam - Library
Currently the library provides web pages and they point students to electronic databases. He says the password might be different for each database. He sees the MLE as a ways that might help students, e.g. “If we know from the log in that this person is a student then we don’t need to ask again.”
He sees the MLE as a way of enhancing access. He thinks that students don’t realise that electronic databases are part of the library and not just the web. OPAC is available to everyone.
He would like students to have a self service option e.g. renewal of books. This would free up student as well as staff time involved in renewals.
Adam would like the testing of the site to ensure that there were no failures to provide a good service. This he sees as requiring a couple of months to set up and then testing so that it is reliable for students. In fact he then goes on to say that it would probably take six months.
Ideally he would like to be able to tell students that their books were overdue or that their requested book was now available. He thinks it would be a good idea to send messages to students. This he sees as requiring work from the library, but he feels there aren’t enough people in the library to support that.
Adam would like to know if people would use the MLE. He would like to know how the project staff aim to get students to use it. He says it needs to be led from the top – academics tell students. For example, if students don’t realise that there will be no postal reminder of overdue books, then they won’t renew them and never realise that there is a problem and fines accruing. He says students need to know these things and who is responsible for telling them?
Adam suggests that course designers could put the course books needed for the course on the MLE and then the library could eventually link to them and notify students of the course book status, e.g. out on loan, on order etc.. Instead of typing in the
Technical and educational technology support. CETD.
Educational technology support.
MLE could improve support to staff and students by enabling access to databases without using more passwords.
Library access to include renewal facility.
More lead in time to ensure quality service and complete testing before going live.
Future and further developments of the library based on the MLE usage.
Student and staff training needs.
Policy decisions.
Future
33
book reference it could be connected to OPAC.
Trudy – Disability Unit
Trudy has responsibility for disabilities and equal opportunities. She assesses students’ disability needs. She had not seen the MLE interface working at the time of the interview.
She is looking for something which is an enabler and feels at the moment it is almost better for staff use than students. For staff it’s a tool whereas students bring attitudes from the wider social context and initially associate it with game playing and social use only.
Because of the area that she has responsibility for she says that there would be a need for a disclosure policy. Also a way of removing data from the site. At the moment there are about 1000 students who are disabled. She would need to think through a letter to be mailed to students about putting their disability details on the MLE and then what would happen next if students didn’t respond to the letter. She would need to get a temp in who could put in the information. Trudy says that she would need 6 months to get things running.
For the social activities area she would like specialised information on the MLE for disabled students such as which banks have wheelchair access. This information in available from the unit, but it would need uploading, as would information on university policy on exams. Other things such as sport facilities which are available to disabled students, which halls of residence are available to a wheelchair user and maps showing where disabled toilets are in the university buildings. These things are not available on the website but would be useful to have for the MLE.
She is looking for “punchy, dynamic student magazine design with punchy direct student language” for the MLE, but no shooting stars etc which are difficult for screen readers to read. “As a magazine it would be something to engage with rather than a text book when I have 80 other text books to read.”
Trudy is keen to have a blind student to test usability.
She says that she would need someone with the technical knowledge to support the unit’s needs. “I would not contemplate starting this without someone to help me. I would want a bulletin board so I could put meetings etc. on it. I would need training in using bulletin boards too.” She feels that without the support and training then everyone would stop using the MLE.
developments for library links via the MLE.
Differing target audiences.
Permission from students required for disclosure of information and the high importance on needing more time to implement the new policies for the Disability Unit.
Specific disability information would need uploading.
Attractive magazine style for the MLE, but with reference to the needs of disabled students.
Blind student to test usability of MLE.
Need for technical support and training in bulletin board
34
Trudy would also like to have a conferencing facility for disabled students, “something like a chat room.” Other needs she reported included the need for administration and policies and information on things such as were the nurses were on campus.
Five Post-graduate Students
This part of the evaluation was conducted after the students had accessed the site and the heuristic evaluation had been conducted by Mark Simpson. A second group of students were to be interviewed, but due to technical problems which did not allow students to access the MLE, no interviews were conducted with the second group.
The five students consisted of one woman and four men, 2 were previous DMU undergraduates and two students were from overseas. One of the students was physically disabled.
For some of the students the MLE login did not recognise their names and P number. However they were able to access the site using a generic password. Students liked having the photograph of themselves and that it recognised who they were.
Students commented that they liked working from home and thought the MLE would be useful in their studying from home.
Students did not understand what the calendar was for and how it could support them. They wanted to be able to add their own things to the calendar and add alerts etc. “Is the calendar for the student or is it just posted course information?”
Some students said they would change the font, but not the colours.
They thought that linking the library and OPAC would be under ‘DMU’ and not under ‘Resources’.
They did think the Social Area would be useful.
They were a bit worried about personal details being on the page and would like to change their passwords immediately to make such details secure.
use.More technical support.
Conference area for disabled students, again specific information for students.
Approval of login photo recognition.
Positive attitude that studying from home the MLE would offer support.
The Calendar - lack of understanding of what the MLE offered – interest in personalising but would need support/training.
Design issues.
Design issues.
Security concerns.
35
They would like to be able to choose who has access to their personal details and wanted to be able to request privacy so that not everyone could read their information details.
They said that if their exam scores were on the site they would be concerned that anyone could read them.
Students said they had no experience of conferencing systems.
They said the MLE “would be good if it really worked”. They felt that everyone has to know about it to make it work well. “Otherwise people won’t go back to it.”
They would also like their module tutors photos available to know who was who at the start of their course.
On the whole a positive response and appreciation of the MLE’s potential.
Policy decisions on privacy should be made clear to students.
Privacy/Confidentiality issues about exam results.
Sceptical about the MLE’s ability to always work. More publicity about the MLE required so all will use it.
Tutor photographs would be useful.
36
Appendix BMLE Pilot Academic Questionnaire
The aim of this questionnaire is to help us to improve the MLE system for academics. The questionnaire will take only a few minutes to complete.
Your Department ……………………………………….
1. How many of your modules use DeMontfort’s MLE? (please cross one box only)
a) 1 xxb) 2 xxc) 3 - 5 xxd) Don’t know xxe) None xx
2. How many of your modules use DeMontfort’s VLE, e.g. WebCT? (please cross one box only)
a) 1 xxb) 2 xxc) 3 - 5 xxd) Don’t know xxe) None xx
3. To what extent do you value the use of the computer in teaching your discipline? (please cross one box only)
Of great value Somewhat valuable
Not very valuable
Of no value
xx xx xx xx
4. Please indicate where you use the following DMU systems? (please cross all that apply)
Home Campus Other Not useda) Mailspinner xx xx xx xx
b) Computer Science system
xx xx xx xx
c) Intranet xx xx xx xx
d) Library xx xx xx xx
5. Over the last month how frequently did you use the MLE? (please cross one box only)
a) More than once a day xxb) Once a day xxc) 2 – 3 times per week xxd) Once a week xxe) Less than once a week xx
37
6. Which of the following MLE activities do you use with your students? (please cross all that apply)
a) Email (group or individual) xxb) Updating course information xxc) WebCT activities xxd) Room changes xxe) Assignments xxf) External links xxg) Course calendar xxh) Add Lecture Notes/PowerPoint presentations xxi) Past examination papers xxj) Quizzes/Self assessment questionnaires xx
7. Please list up to three things that you like about using the MLE.a)–––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––b)–––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––c)–––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––
8. Please list up to three things that you dislike about using the MLE.a)–––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––b)–––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––c)–––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––
9. The following questions relate to your experience of using the system. For each question put a cross to the response that you agree with.
Strongly agree Agree Disagree
Strongly disagree
a) I found learning to use the MLE was easy. xx xx xx xx
b) I found sending messages to individuals using the MLE was straightforward. xx xx xx xx
c) I found the experience of using the MLE frustrating. xx xx xx xx
d) I could find the things I was looking for quickly. xx xx xx xx
e) I need more training in using the MLE. xx xx xx xx
g) I need more technical support. xx xx xx xx
h) I don’t know what I am supposed to do to help my students learn through the MLE. xx xx xx xx
i) I think the way the MLE was presented on screen was attractive. xx xx xx xx
j) I need a help function when using the MLE. xx xx xx xx
k) The different features of the MLE were obvious to me. xx xx xx xx
l) I found reading messages and information was easy. xx xx xx xx
m) I need tips to improve my teaching presentation on the MLE. xx xx xx xx
n) I found changing my preferences easy xx xx xx xx
o) I have found more students bring queries to the MLE rather than to my office. xx xx xx xx
38
10. How has using the MLE affected your teaching? Please put a cross against all that apply in column A and put a cross against the most important in column B.
A Ba) My basic approach to teaching seems to be
changing, in part because of the opportunities that the MLE provides
xx xx
b) My basic approach to teaching has not changed, but the MLE helps me to do certain things better
xx xx
c) I have found the MLE of some value in my teaching
xx xx
d) It has not helped my teaching at all. xx xxe) I have found it of no value in my teaching. xx xxf) I have found it increased my work time xx xx
Please add any further comments you would like to make about the impact that the MLE has had on your teaching.
–––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––
THANK YOU FOR TAKING THE TIME TO COMPLETE THE QUESTIONNAIRE.
39
Bank Questions
Additional options for Question 8
Use the search facility to find something xx xx xx xx
Send a message to someone in your address book xx xx xx xx
Check Who’s Online xx xx xx xx
Which of the following do you use the VLE/MLE for in your teaching? (please cross all that apply)
(NB the wording of this will need to be adapted according to whether any of these are a possibility on the MLE or
only the VLE)
a) Introductory materials xxb) Simulations xxc) Skill development xxd) Student feedback xxe) Online Case studies xxf) Online assessment xxg) Student interaction with outside ‘experts’ xxh) Updating of materials/modules xxi) Monitoring student performance xx
Which of the following do you use a computer for? (please cross all that apply)
a) To run general purpose software (e.g. word processing, spread sheets) and printing xx
b) To run specific commercial software xxc) Email/communications xxd) Instant messaging (e.g. ICQ, MSN Messenger) xxe) Internet/World Wide Web - Information seeking xxf) Internet Shopping xxg) Microsoft packages xxf) Games xxg) File Transfers xxh) Socialising xx
40
Appendix CMLE Pilot Student Questionnaire
The aim of this questionnaire is to help us to improve the MLE system through student evaluation. The questionnaire will take only a few minutes to complete.
Your P Number ……………………………………….
1. How many of your modules use DeMontfort’s MLE? (please cross one box only)
a) 1 xxb) 2 xxc) 3 - 5 xxd) Don’t know xxe) None xx
2. How many of your modules use DeMontfort’s VLE, e.g. WebCT? (please cross one box only)
a) 1 xxb) 2 xxc) 3 - 5 xxd) Don’t know xxe) None xx
3. To what extent do you value the use of the computer in studying your discipline? (please cross one box only)
Of great value Somewhat valuable
Not very valuable
Of no value
xx xx xx xx
4. Please indicate where you use the following DMU systems? (please cross all that apply)
Home Campus Other Not useda) Mailspinner xx xx xx xx
b) Computer Science system
xx xx xx xx
c) Intranet xx xx xx xx
d) Library xx xx xx xx
5. Over the last month how frequently did you use the MLE? (please cross one box only)a) More than once a day xxb) Once a day xxc) 2 – 3 times per week xxd) Once a week xxe) Less than once a week xx
6. What kind of activities do you mainly use the MLE for? (please cross all that apply)a) Course-related activities xxb) Social activities xxc) General University activities xx
41
7. How often do you do the following things when using the MLE? (please cross one box only in each row)Always Sometimes Rarely Never
a) Send private e-mail xx xx xx xx
b) Use the Private Chat facility xx xx xx xx
c) Start a discussion topic xx xx xx xx
d) Reply to a discussion topic xx xx xx xx
e) Read messages within a topic xx xx xx xx
f) View the history of a message xx xx xx xx
g) Add an entry to the calendar xx xx xx xx
8. Please list up to three things that you like about using the MLE.a)–––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––b)–––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––c)–––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––
9. Please list up to three things that you dislike about using the MLE.a)–––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––b)–––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––c)–––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––
10. The following questions relate to your experience of using the system. For each question put a cross to the response that you agree with.
Strongly agree Agree Disagree
Strongly disagree
a) I found learning to use the MLE was easy. xx xx xx xx
b) I found sending messages to individuals using the MLE was straightforward. xx xx xx xx
c) I found the experience of using the MLE frustrating. xx xx xx xx
d) I could find the things I was looking for quickly. xx xx xx xx
e) I thought that the system was secure. xx xx xx xx
f) I found that the interface responded quickly to my actions. xx xx xx xx
g) I believe that the MLE has been designed taking my needs as a student into account. xx xx xx xx
h) I don’t know what I am supposed to do in the MLE. xx xx xx xx
i) I think the way the MLE was presented on screen was attractive. xx xx xx xx
j) I need a help function when using the MLE. xx xx xx xx
k) The different features of the MLE were obvious to me. xx xx xx xx
l) I found reading messages and information was easy. xx xx xx xx
m) I found changing my preferences easy xx xx xx xx
Please add any further comments or elaborate on any of your responses in the space below.–––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––THANK YOU FOR TAKING THE TIME TO COMPLETE THE QUESTIONNAIRE.
42
Bank Questions
Optional additional items for Question 6.
Use the search facility to find something xx xx xx xx
Send a message to someone in your address book xx xx xx xx
Check Who’s Online xx xx xx xx
Which of the following do you use a computer for? (please cross all that apply)
a) To run general purpose software (e.g. word processing, spread sheets) and printing xx
b) To run specific commercial software xxc) Email/communications xxd) Instant messaging (e.g. ICQ, MSN Messenger) xxe) Internet/World Wide Web - Information seeking xxf) Internet Shopping xxg) Microsoft packages xxf) Games xxg) File Transfers xxh) Socialising xx
43