Project Acronym: ITS4SEA Version: 1 Contact: s.j.eyre@bradford.ac.uk Date: 1 June 2009

PAGE 1 of 19 Document title: JISC Final Report

Project Document Cover Sheet

Project Information

Project Acronym ITS 4 SEA (Integrating Thin client Systems for Secure E-Assessment

Project Title Integrating thin client systems and smart card technology to provide integrated, flexible, accessible and secure E-assessment

Start Date October 2007 End Date March 2009

Lead Institution University of Bradford

Project Director Prof Peter Hartley

Project Manager & contact details

Mrs Sara Eyre, s.j.eyre@bradford.ac.uk 01274 233335

Partner Institutions None

Project Web URL http://www.brad.ac.uk/lss/jisc/

Programme Name (and number)

JISC Institutional Exemplars

Circular 1/07

Programme Manager Lawrie Phipps

Document Name

Document Title Final report

Reporting Period N/A

Author(s) & project role Sara Eyre, project manager

Date Filename

URL if document is posted on project web site

Access General dissemination

Document History

Version Date Comments

1 25 February 2009 First draft for approval by project board

2 1 June 2009 Final version for submission to JISC

Project Acronym: ITS4SEA Version: 2 Contact: s.j.eyre@bradford.ac.uk Date: 1 June 2009

PAGE 2 of 19

JISC Final Report

Table of ContentsJISC Final Report .................................................................................................................................... 2

Table of Contents............................................................................................................................. 2

Acknowledgements .......................................................................................................................... 2

Executive Summary ......................................................................................................................... 3

Background ...................................................................................................................................... 4

Aims and Objectives ........................................................................................................................ 4

Methodology .................................................................................................................................... 5

Implementation ................................................................................................................................ 5 Thin client activities 5 Word processed exams 6 Smart card technology 7 Examination Administration 7 Estates and E-coversity 9 Quality assurance and project management 10

Outputs and Results ...................................................................................................................... 10

Outcomes ....................................................................................................................................... 14 Major improvements to E-administration 15 Sustainable developments 16 Major Deliverables 16

Conclusions ................................................................................................................................... 17

Implications .................................................................................................................................... 17 Technology 17 E-assessments 17 Exam administration 18

Recommendations ......................................................................................................................... 18 Project management 18 For HEIs developing E-assessment facilities 18 For HEIs seeking to improve their E-assessment processes 19 For HEIs planning to improve their examination timetabling 19 Consider the technology options 19

Appendixes .................................................................................................................................... 19 Appendix A. Designing a new facility for Online Assessments 19 Appendix B. ExamAdmin Draft Functional Design Specification 19 Appendix C. Student exam evaluation 19

Acknowledgements

Gratefully acknowledgements are made to JISC for the funding of the development of this project under the Institutional Exemplars programme and for the support provided by the programme manager, Andy Dyson. Special thanks are also due to the members of the project board for their unfailing support and helpful advice, especially our two external evaluators Prof Mark Stiles from Staffordshire University and David Lewis from Glamorgan. Very special thanks to all the University of Bradford staff who worked on the project teams for their patience, particularly when things have not always gone as planned. Special thanks to John Dermo who has taken on the bulk of the dissemination work for the project at a wide range of conferences and been the main motivating force behind the increased uptake of online assessment at the University of Bradford.

Project Acronym: ITS4SEA Version: 2 Contact: s.j.eyre@bradford.ac.uk Date: 1 June 2009

PAGE 3 of 19

Also much appreciated has been the work of the external companies who worked so hard to ensure that the new E-assessment facility, which has been created at the University, was finished in time to be used for the January 2009 examinations, which were the only exams that could be scheduled during the life of this project. These include:-

Farrell and Clark, Architects, for the overall design based on a random list of general ideas

PSK Tilbrook, project managers, for ensuring the building work completed on time

Andy Hague, University project manager

Wood Mitchell, main contractors, for completing the work to a very tight timescale. Our thanks also need extending to Sun Microsystems for their matching grant programme which allowed the Sun Ray workstations to be obtained at the most competitive pricing.

Executive Summary

This project has demonstrated that thin client technology, allied to systematic institutional procedures and processes, can deliver effective and efficient summative e-assessment to learners in a flexible, secure, and accessible way. This technology has enabled major improvements to this University‟s delivery of e-assessment and offers a number of advantages over „traditional‟ PC clusters. Benefits to the sector include „proof of concept‟ in the application of this technology to e-assessment and guides and exemplars to assist institutions who wish to expand or improve their provision in e-assessment and examination timetabling. Major learning points for the HE community lie in the development of the infrastructure to support large numbers of thin client workstations using VDI technology and in the secure delivery of online assessments using Questionmark Perception, with Appsense used to control the facilities that students have access to during the examination. During the design phase of the thin client assessment facility, research was also conducted into best practice for layout, facilities required and other related features; a report detailing the findings was produced and forms part of this final report. The project has made a significant impact on our institutional use of e-assessment, with a major increase in uptake of online assessments following the creation of a new 100 seat thin client cluster and the demonstration that it can be used to deliver assessments in a secure manner. It has highlighted the need for high quality data on both student enrolments and module assessment recording. The student exam experience has been improved by the provision of personalised examination timetables, delivered by email, and the automatic allocation of facilities for students with special needs. The design specification for a new examination administration system forms part of this final report. The majority of students also found doing their exams online a major improvement over using Optical Mark Recognition forms; they were also considerably quicker for administrators to process the results from. The final learning outcome from this project relates to the environmental credentials of using thin client technology. The project has worked closely with the JISC funded SusteIT

1 project to try and

evaluate whether thin client technology provides a more environmentally friendly alternative to the standard PC. The tools developed as part of this collaboration will assist other Universities and Colleges to assess whether the total cost of ownership of thin clients will benefit their organisation in the long term.

1 http://www.jisc.ac.uk/publications/publications/sustainableictfinalreport.aspx

Project Acronym: ITS4SEA Version: 2 Contact: s.j.eyre@bradford.ac.uk Date: 1 June 2009

PAGE 4 of 19

Background

The project board believed at the outset that using thin client (Sun Ray) technology linked with smart card identification would allow a flexible, personalisable and scalable system for summative e-assessment. Thin client technology has already been successfully piloted at the University in other contexts and this project has built on work we have conducted with Sun Microsystems to evaluate and introduce their Sun Ray thin client technology as an environmentally friendly, secure and sustainable alternative to the PC. Some of the issues addressed by this project were highlighted in the 2007 JISC report „Effective Practice with e-Assessment‟

2; in particular the issues around „setting up viable physical and IT

infrastructure to support e-assessment to the required scale‟ and „cheating by digital means during examinations – for example, by accessing the internet...‟. The project also aimed to improve a number of processes around scheduling of examinations and integrating the procedures with key University systems, such as student records, disability information and module registrations. The project was designed to demonstrate that thin client technology can be used to provide an easy-to-manage, secure and reliable classroom resource for computer aided and computer-based assessment. To add further value to the project, we also proposed testing whether this system could provide easier to support technology than standalone PCs for use within lecture rooms. The use of online assessment at the University of Bradford had seen an exponential growth in the 12 months before this project began

3 and there was already a strong commitment to the use of

Questionmark Perception as the assessment tool of choice4. The difficulties encountered running

large scale assessments on standard PC clusters were deterring some staff from attempting to make use of online assessments, so a large number of modules were still being assessed using paper based Optical Mark Recognition forms, despite the time taken to check each form by eye and then scan them in; a figure of 10 hours work to vet 200 OMR forms has been quoted. This project also formed a natural follow on to the work that was done on the JISC and Higher Education Academy Pathfinder program, which investigated embedding support processes for e-assessment to ensure the reliable and secure large-scale implementation of CAA

5. The work done

under this programme had already streamlined the way for increasing the number of summative assessments conducted on line.

Aims and Objectives

At the start of the project the aims were to address the following:-

The need for secure and reliable systems to deliver computer aided assessments to large classes in a manageable and sustainable environment

The need for flexible systems which can deliver a wide range of assessment types and which support automated delivery of anonymised scripts (as recommended in current QAA guidance) for marking and for checking (e.g. testing for plagiarism through Turnitin etc.)

Full integration of Computer Aided Assessment (CAA) processes and marks with the student record system

Streamlining of examinations management by integrating exam scheduling software with student record systems

The need for effective strategies to manage and improve the learning and assessment process for inclusion, widening participation and accessibility.

Systems to allow use of IT facilities as part of the teaching process which are up to date, easily managed and responsive to academic staff needs.

2 http://www.jisc.ac.uk/media/documents/themes/elearning/effpraceassess.pdf

3 http://www.brad.ac.uk/elearning/pathfinder/documents/QMP_use_stats.pdf

4 http://www.questionmark.co.uk/uk/casestudies/bradford.aspx

5 http://www.brad.ac.uk/elearning/pathfinder/documents/Briefing_Papers.pdf

Project Acronym: ITS4SEA Version: 2 Contact: s.j.eyre@bradford.ac.uk Date: 1 June 2009

PAGE 5 of 19

The development of sustainable and environmentally responsible approaches to our development in learning, teaching and assessment, particularly in the deployment of educational technologies.

The aims did not alter as the project progressed, but some proved impossible to handle within the timescale of the project for a wide variety of reasons; these are explained in detail in the outcomes section below.

Methodology The various aspects of the project were split into separate work packages at the start of the project; these were managed separately with the project teams coming together when the integration became more important than the design processes. The Bradford project management methodology (based on Prince2) was used throughout the project with regular project team meetings, and project board meetings every 3 months where the risk and issue logs were considered and updated. The work packages were

1. Thin client related activities 2. Smart card technology 3. Integration of Examinations systems with Computer Aided Assessment and student record

systems 4. Estates and E-coversity 5. Evaluation of impact of the project on the Institution (see Output and Results section) 6. Dissemination to interested external parties (see project completion report) 7. Quality assurance of the project and its outcome 8. Overall project management

Implementation

Thin client activities

The thin client related activities commenced with a re-evaluation of whether Sun Rays were the best choice of thin client technology for this project. Other manufacturers‟ equipment was evaluated but the Sun Rays provided more flexibility than other „terminals‟ and the card reader that is incorporated into the design was thought to provide additional benefits for online assessments and potentially for staff use in teaching rooms. It would potentially allow a classroom session to be prepared in advance and then simply transferred to the teaching room by placing the smart card in the card reader. At the start of the project, Sun Rays were being slowly deployed into the University‟s student administration area known as the HUB. The aim was to allow staff to move rapidly from handling student queries at the front of house desks to their own desk in the back office while retaining details of the student‟s account. Initially desktops were supplied using Windows Terminal Services but this was causing increasing problems whereby a single failure with an application for one user had caused all other Sun Ray sessions to hang and require restarting. Around this time, VMware introduced a new system for handling Virtual Desktops, called VDI. VDI was selected instead of Terminal Services due to the increased flexibility it could provide for future use with the Sun Ray cluster. VDI should provide access to a wider range of applications software as well as providing increased security; VDI also builds more directly on existing expertise within the University which is based around managing PC desktops rather than round Windows servers. It wasn‟t until the large scale Sun Ray cluster became available for testing that problems integrating the working of VDI and Sun Rays became evident! We faced a number of issues surrounding storage, the amount of time required to clone the virtual images and coping with the load from such a high number of consecutive logins. Low end NFS based storage, as is commonly recommended for VDI, gave unsatisfactory clone times of up to a couple of days to provision all the work stations and proved unable to cope with the login load bringing the system to a halt. After trying various alternatives we found that iSCSI arrays configured for high performance were able to handle the number of logins required in the new facility. At the time of writing, there are still outstanding issues around clone times. An alternative hardware solution was investigated but proved not to be cost effective. As the Virtualisation/VDI technology develops, more software solutions from different suppliers are becoming

Project Acronym: ITS4SEA Version: 2 Contact: s.j.eyre@bradford.ac.uk Date: 1 June 2009

PAGE 6 of 19

available that claim to address the problems encountered during this project and we aim to implement one before the 2009-10 academic session. For normal student lab usage, the current image is 25Gb and each image takes approximately 10 minutes to clone, for online assessments the image is down to 6Gb and can be cloned in about 3 minutes. To overcome these problems for the January 2009 exams, a virtual desktop was assigned to each work station rather than to individual users or to each type of smart card. Since all the exams that were running at the same time were identical, this did not cause any problem for the exams but would for teaching using the Sun Rays since a new class of students would need to login and be assigned a new desk top within a short space of time. Another aim was to trial use of Sun Rays in General Teaching Areas of the University in rooms that do not have a fixed PC. Several staff have complained about having to carry laptops to these locations and there are frequent problems with the PCs that are in GTA becoming out of date, trying to install Windows updates at the start of a class etc. Tests were conducted to ascertain if the projectors fitted in classrooms were compatible with the screen resolution of Sun Rays and also with some multimedia applications to see if Sun Rays were suited to all potential software applications that may be required while teaching. The projectors worked correctly but some software applications struggle to display moving images and sound synchronised together which mean that, at the current time, some teaching applications may not be suited to Sun Rays. However, Sun Microsystems are aware of this shortcoming and have prioritised this for future firmware releases. However, the project encountered difficulties trying to get the single data connection in lecture rooms accessible to Sun Ray servers as well the University‟s roaming network, RoamNet. These have proved insurmountable due in part to the ongoing relocation of the main computer suite and network equipment. It is not possible to simply reconnect the data sockets to a different network router as they still need to be used with staff‟s own laptops in cases where a Sun Ray is unsuited to their teaching needs. Thus, the functionality and suitability of Sun Rays in teaching area has still to be tested.

Word processed exams

Another aim of the project was to move students who currently word process exams on PCs, often due to dyslexia, and currently store their work on floppy disks, to use Sun Rays instead for increased exam security. In association with this, a trial was also planned to convert a part written, part multiple choice examination into a format where it could be taken entirely online using the Sun Ray facility. Ideally, students should be able to use the word processing package with which they are most familiar for exams. Some investigation was conducted into what word processing features were actually used by the students who currently take their exams using a PC; this indicated that they were basically text processing only. However, it is not possible to tell from the exam scripts whether the students were also utilising features of Microsoft Word such as auto-correct and spell checking. In order to allow for these options, work was conducted into ways of controlling what aspects of a software package can be disabled to prevent access to the internet (possibly via the research and help options) and also to prevent access to files students had produced and stored previously either on USB devices or university provided files space. This investigation found a product called Appsense

6, which allows

many applications to have their menus and functionality to be exactly tailored to meet particular needs. The Appsense system was used for one of the trial examinations run on the new Sun Ray cluster in January 2009 to provide access to a file in Excel, which could not be saved and could not be used to access the internet. After much discussion, there were too many unresolved issues round word processing exams and the storing of files which need to be more fully investigated to risk running a summative assessment yet. The issues are detailed in the project Issue Log

7. Hopefully this investigation can be continued under

the proposed benefits realisation follow up project. The trial examination, which would have allowed students to word process their exam answers, failed at the last moment for two different reasons. The first was that the class size was so large it had to be

6 http://www.appsense.com/Files/Documents/University%20of%20Bradford%20(US).pdf

7 http://www.brad.ac.uk/lss/jisc/issuelog.pdf

Project Acronym: ITS4SEA Version: 2 Contact: s.j.eyre@bradford.ac.uk Date: 1 June 2009

PAGE 7 of 19

split into three cohorts for the exam; this would mean that effectively the last group would have well over an hour to research the exam questions after the first cohort had left the exam. This is obviously unacceptable. The second reason was that the students were in Life Sciences and the exam questions will often benefit from diagrams being included in the response; this is not easily possible with current word processing and drawing packages. The exam was therefore taken in two separate parts; the first online using Questionmark Perception and the second in a large hall where all the students could hand write their responses in a single sitting.

Smart card technology

The use of smart cards was another thorny issue that was not resolved in time for the January exams. The plan was to link the image for a particular exam to a smart card; this would allow for multiple exams to be held in a single location with student simply being told to use a workstation with a red triangle or blue square etc on the card. The University has been issuing Mifare based smart cards to all staff and students for several years; the chip was initially used for photocopying applications in the Library and more recently also by the access control system being introduced across the University. To find out more about smart cards and their uses within the HE community, the project manager tried to join the mailing list smartcards@jiscmail.ac.uk. However, the list was configured so that joiners had to be approved by the list owner, who seemed to have gone inactive. After contacting the JISCmail list manager, the list is now operative again and has increased from 12 to 54 members, with the author as list owner. Postings to the Smartcards list introduced the Higher Education Smart card Association (HESCA

8).

This has proved to be a very useful forum for staff looking at a wide range of services that can be provided with smart cards; however, the technology required by the card readers in Sun Rays does not appear to be in wide spread use for other applications, so deployment at Bradford will require dual technology cards with both the Mifare close proximity chip and the JCOP java enabled card; the new JCOP31 chip is alleged to provide both sets of functionality but it has not been tested as the price of a single JCOP31 card is higher than one with both Mifare and JCOP10 chips. Having discovered what was needed for the January exams and for staff who would potentially benefit from having both Mifare and JCOP in a single card, an initial order was placed. The design of the cards and the positioning of the chips on it to avoid areas that would need printing, took several weeks. The cards eventually arrived a few days before Christmas and didn‟t work with the Sun Rays! Subsequent investigations showed that the JCOP chip had not been initialised and the cards have since been sent to the JCOP manufacturing plant in Austria to have this done. Smart cards for use with the Sun Rays were therefore unavailable for use during the January 2009 examinations. However, dual technology chip smart cards have now been ordered and will become the standard for new students arriving at the University from summer 2009, so that this technology can be used in the future. The ability to mix different exams within a single room may become important as the number of modules assessed online increases further; linking a student‟s own card to the appropriate exam would allow this. Alternatively programming a smart card with a particular symbol on it to only link to a particular exam could enable the exam schedulers to tell students to sit at a „red triangle‟ or „blue square‟ workstation and distribute them appropriately round the available seats. Use of this smart card technology will only work with thin clients which have a smart card reader, eg Sun Rays.

Examination Administration

The integration of examinations scheduling with Computer Aided Assessment and student record systems has proved to be problematical. At the start of the project, the University had recently purchased a new module from the suppliers of the student record system, SITS, and a project board had been set up to oversee its implementation. The new module claimed to integrate information about students‟ disabilities with the exam schedules so that extra time and special facilities required were taken into consideration; since this was one of the goals of the ITS4SEA project, it should have made the project very easy indeed. After attending training courses and trying to use the new module, WASP, to schedule the May 2008 examinations, it was found that it was not fit for purpose. It did very little with the disabilities data and could not split classes into multiple cohorts so that they would fit into

8 http://www.hesca.com/

Project Acronym: ITS4SEA Version: 2 Contact: s.j.eyre@bradford.ac.uk Date: 1 June 2009

PAGE 8 of 19

the size of rooms available. This was a particular issue for online assessments that would eventually be held in the new Sun Ray facility as the class sizes were in excess of 300 in some instances. Some 6 months into the project a decision was made to revert to the original exam scheduling programme and develop the code necessary to integrate it with the module and student data information. A report on the deficiencies of the WASP exam scheduling software has been produced but not made generally available due to its commercial sensitivity As a result of the unsuitability of the WASP scheduling system, a system specification for a new Exam Admin system was drawn up and finalised in August 2008. Unfortunately this did not allow sufficient development and testing time for the new system before the January 2009 examinations had to be scheduled. The decision was taken to timetable the January exams based on the processes in the system design but to actually implement the new system during March 2009 when the May 2009 exams would have to be finalised. One major improvement for the January exams was the production of a personalised timetable for each student of the University which was sent out by email in the last week of term before Christmas. This has been repeated for the May examinations. The automatic allocation of additional time and separate locations for students with specific exam needs has seen an improved take up of special facilities since the introduction of the new exam administration system. The development of the system has proved demanding due to difficulties with the reliability of the data that has to be processed. After two attempts to automate the generation of an examination timetable during the course of the ITS4SEA project, the level of incorrect data related to type of assessment, length of assessment and students enrolments on modules has meant that far too many alterations are still required to be made manually. Correcting the data will take some considerable time and effort but tools have been developed to assist with this process and will be made available on the project web site. It is noticeable that some areas of the University who engaged with the data cleansing process had only 3% of modules requiring manual intervention whereas other areas required over 60% of their exams to be handled manually. As part of the investigation of examination processes, the possibility of linking the exam timetable directly with the Questionmark Perception system was evaluated. “Questionmark Web Integrated Services Environment, QMWISe, is a comprehensive series of web services methods that act as an Application Program Interfaces (APIs) that enable registration, management and reporting systems to tightly integrate with Questionmark Perception”

9. While QMWise provided the tools to allow tests to be

scheduled for a particular time on a particular date and also to extract the results files in order to process them directly into the student record system, it actually proved impossible to automate the online assessments in this way. The unreliability of the data of which students were enrolled on which module, which part of the assessment related to the online examination and which students were repeating the assessment proved to be insurmountable. However, a major reduction in the administrative burden of exams has been achieved in the transfer of some paper based multiple choice assessments into online assessments, where no post-exam administration is required to create the results file in a format that can be processed directly into the student record system for the majority of students. A simple expedient of modifying the results files produced by Questionmark Perception to include the student number as well as their computer user account name has also saved tedious manual processes importing the results into the student record system. The earlier Pathfinder project had developed strong working links between staff in the central examinations office and IT Services and streamlined processes around the booking of IT facilities during the exam period. It had also developed improved regulations on the conduct of online exams

10.

All these previous activities were essential to the success of the ITS4SEA project but that has highlighted yet more work that needs to be done in these areas; these are highlighted in the Risk Register and Issue Log for the project.

9 http://www.questionmark.co.uk/uk/perception/qmwise.aspx 10

http://www.brad.ac.uk/elearning/pathfinder/documents/CAA_policy.pdf

Project Acronym: ITS4SEA Version: 2 Contact: s.j.eyre@bradford.ac.uk Date: 1 June 2009

PAGE 9 of 19

Estates and E-coversity

The Estates issues have also proved interesting. There appeared to be little research into appropriate layouts for computer aided assessments, so an email requesting advice and information was circulated round various email lists. It generated a lot of interest and many useful replies with many photographs and pointers to other resources. These have been summarised in a separate guide to designing assessment facilities. The architects appointed to design the new assessment room took on board the information which had been received and designed a variety of different possible layouts based round furniture that was commercially available. These designs were discussed by the project board and agreement reached that having a room with the screens facing towards each other would minimise the possibility of viewing another student‟s screen during exams. This philosophy led to the eventual design that was constructed.

The room is split into two unequal sized spaces with a fully glazed divider between them. The smaller part can be used either for scheduling students who have been allocated extra time for exams or for small group teaching leaving the large part for ad-hoc use by students. The size and shape of the room mean that standard projection facilities for teaching are not appropriate, so screen management software has been investigated though not yet deployed; it is planned to make Net Support School available for teaching for the 2009-10 academic year. The University of Bradford is engaged in a University wide E-coversity Programme

11 which aims „to

embed the principles and practice of sustainable development across the entire institution by getting people in involved, taking the lead on issues, and encouraging and making it easier for people to adopt sustainable behaviours and lifestyles‟. It has also hosted another JISC funded project investigating Sustainable ICT in Further and Higher Education. The two JISC projects have met several times over the project‟s life to try and analyse whether thin client systems are more cost effective and more environmentally friendly than standard PCs. As part of the SusteIT project, two

11

http://www.brad.ac.uk/admin/ecoversity/

Project Acronym: ITS4SEA Version: 2 Contact: s.j.eyre@bradford.ac.uk Date: 1 June 2009

PAGE 10 of 19

tools12

have been developed to test this hypothesis, however, the server and disk configuration to run more than 100 fully configured Sun Ray thin clients has only just been finalised, so the output from these tools has still to be fully evaluated. Sun Microsystems have a strong commitment to minimising the environmental impact of its operations and they have published both a returns policy to ensure that all equipment purchased from Sun can be returned for recycling

13 and also a statement on the European Union Directive on Waste

Electrical and Electronic Equipment (WEEE), which took effect on August 13, 200514

. It is notable however that Sun Ray workstations do not have a power switch included in the design to allow them to be easily switched off when not in use. This is a part of the design to provide longevity of the equipment as it is often a manually operated switch which causes equipment to fail; Sun claim a life time in excess of 10 years for their thin client workstations. The decision was taken however that 100 Sun Rays left on 24 hours a day for 365 days a year would be an unacceptable increase in power consumption so, in an attempt to encourage student users to turn off the Sun Ray workstations when not in use, the power cables to each unit‟s transformer have been modified to include a power switch; as Sun have surmised, these have not proved very reliable to date.

Quality assurance and project management

The project board met 6 times during the course of the project, with two external members for the final 5 of these. Our grateful thanks go to Professor Mark Stiles from Staffordshire University and David Lewis from the University of Glamorgan for their enthusiasm and constructive criticism. The project board also contained members of the Teaching Quality Enhancement Group and Academic Standards Unit of the University to ensure that what was being developed fitted with University policies and processes. There were two project teams who met regularly during the project; one concentrating on exam timetabling and integration of administrative systems and the other on the technical development of the Sun Ray cluster and infrastructure. The building work was project managed by the University Estates department and their external project management partners, PSK Prout Tilbrook.

Outputs and Results

The following quotes exemplify the major achievement of this project.

I’ve already told the DVC that we will need another cluster like this within 2 years”, Dr Nigel Lindsey, Associate Dean, Life Sciences and lecturer in immunology “This is just what I always wanted but we couldn’t do on the PCs!”, Dr. Bob Lomas, Lecturer in Technological Management

They relate directly to the successful creation of a new 99 seat assessment facility utilising Sun Ray thin client technology and, in the second one, the use of Appsense to control the software which was made available for the examinations. The creation of the 100 seat assessment facility was major achievement in itself; finding a large enough empty space and also persuading Estates staff that an assessment facility was the best use for it was a major achievement and needed support from academic staff who would benefit from the development at a very early stage. Following the successful trials involving over 400 students in the January 2009 examination period, a further 9 new modules were timetabled to use online assessment for summative assessment for the first time at the end of semester 2, covering over 900 student assessments. Feedback from students taking their exams online has also been encouraging; in one group of 163 students, 128 agreed or strongly agreed with the statement „Would you like to have other summative assessments in this format‟. However, it must also be noted that 14 students disagreed or disagreed strongly. Comments ranged from „For most modules, summative and if possible, real tests / exams

12

http://www.susteit.org.uk/files/index.php 13

http://www.sun.com/aboutsun/environment/returns-intro.jsp 14

http://www.sun.com/aboutsun/ehs/ewaste.jsp#weee

Project Acronym: ITS4SEA Version: 2 Contact: s.j.eyre@bradford.ac.uk Date: 1 June 2009

PAGE 11 of 19

should be done on the computer, there's a lot less pressure and it helps to relax and maybe even do better‟ and „This format was perfect for me, it was much less hassle compared to the usual hassle during exam times‟ through to „I prefer it for shorter tests, however, for MCQs with over 50 or so questions, I prefer paper‟ and „Your eyes start hurting staring at a screen, you can't think. Your wrist starts to hurt too whilst using the mouse‟. Analysis of a second set of exam candidates is attached as Appendix C. Several students mentioned problems with the screen and the font size used. It is planned to make footrests and monitor stands available for the 2009-10 academic year so that the impact of these can be assessed. It is possible to change the contrast and brightness levels on the Sun Ray monitors, although how to do this is not obvious. The magnification of text on the screen can also be enlarged. For the 2009-10 assessment periods, we will provide all exam candidates with an instruction card on how to get comfortable and how to modify the screen to suit their personal preferences to assess whether this improves student feedback on taking exams online instead of on paper. An unexpected outcome from the development of the new 100 seat facility and the way the workstations have been arranged is a request to use the room for a new way of teaching. One lecturer whose traditional lectures have been poorly attended and the content poorly understood is planning to replace his lectures with small group working in the new facility; he will set exercises to be done in groups and go to groups who are having difficulty with an exercise to coach them in the problem. It will be interesting to see how this approach works. It could not have been accommodated in any of the existing PC clusters on campus where the layout would not encourage this type of working and the space between workstations would hinder staff moving between the groups. There were several major benefits to the examination processes during the January 2009 period

There was no need to check and scan 190 Optical Mark Recognition sheets thus saving approximately 9 hours work; the marks were available at the end of the afternoon for import into the student record system.

Students with extra time allowed were scheduled into the area adjacent to the main assessment room and started at the same time as the bulk of the class who could leave without disturbing them.

Students were enrolled faster against the appropriate modules in the student records system although major issues remain with the reliability of data for both module assessment modes and student enrolments.

There was an increased take up of specialist facilities by students with disabilities as they were informed individually where and when their assessment would occur compared with the previous system of having to look up each module and see where the assessment was scheduled.

Personalised examination timetables were emailed to all students. This highlighted to every individual whether they were entered for the correct assessments and led to many changes to module registrations at the last minute.

Students requiring extra time, or their own separate room for exams, were scheduled within Syllabus+ for the first time. Previously these students were scheduled manually.

ExamAdmin provided a flexible and easy to use interface that gave the examination Office staff control over which examination records were extracted from SAINT. Previously this was hard coded in a PERL script, and not under the user‟s control.

The automatic production of data quality reports so that academic schools could validate the accuracy of SAINT held examination data before any scheduling was performed.

Work on the examinations processes during the ITS4SEA project also resulted in a report on the shortcomings of the WASP exam scheduling product from SITS. These findings were validated given the increasing number of students with additional requirements and the proliferation of multi-session PC based exams which WASP would not have been able to cope with. An additional benefit of the exam administration work was the cleansing and enhancing of examination data in SAINT prior to scheduling, based on the results of Data Quality reports that were circulated to administrators in Schools before the exam timetable was published. However, there is a noticeable reluctance of academic schools to adopt changes in procedures or engage with the data quality process; there was a better response for the May exams than there was in January.

Project Acronym: ITS4SEA Version: 2 Contact: s.j.eyre@bradford.ac.uk Date: 1 June 2009

PAGE 12 of 19

A number of advances have also been made in the running of online assessments:

The ability to improve the security of Questionmark Perception assessments by controlling the IP addresses which can be used to access the server. This was thought desirable since Questionmark does not have an IP address checking facility currently. Although this has developed it has not so far been deployed due to last minute networking difficulties. It will be used to increase security for online assessments during 2009-10.

The ability to control what software and services are made available to the exam candidates by using Appsense has also encouraged some more sceptical academics to use computer based assessments for the first time in the May examination period.

The 10% failure proposed in the Bradford CAA exam guidelines for PC clusters has not proved necessary when using Sun Rays for online assessments. Two examinations were run successfully for 94 students in the 99 exam seat facility in order to avoid running them over 2 sittings.

The reputation and benefits of online assessment over Optical Mark Recognition based examinations have been greatly enhanced across the institution. Good news spreads quickly amongst the academic community!

Student feedback has also been encouraging but more assessment of their views and any associated issues is required. Although at the design stage the potential need for footrests, monitor stands etc was identified, these have not so far been made available or suggested to exam candidates.

Some potential difficulties with online assessments have also been identified:

In order to run exams using a word processor, if all students cannot take assessment in one sitting then a bank of questions may be needed so that students in cohort 3 cannot be told what the possible questions are by cohort 1. If only 2 sittings are required then the second cohort could be kept separate from the first by requiring them to be present at an alternative location until after the first sitting has finished and then supervised while they move to the exam location. This is also the case if a multiple choice exam is run online in multiple sittings but the questions cannot be randomised or chosen from a bank of questions of similar difficulty. One such exam was run during the May exams and the second cohort were required to arrive at a separate location shortly before the first cohort were due to leave the exam room; this worked successfully.

Some examinations may not be suited to word processing, for instance if they would benefit from diagrams being included.

The saving of word processed scripts and providing secure, anonymous delivery to the marker was not resolved. While it is administratively straight forward to ensure that the word processor package is configured to save the files created by the exam candidates to a location from where it can be automatically extracted (using Appsense), how to make it available for marking is less obvious. Other issues were around how to separate different answers to different exam questions; how to deliver different answers to different academics for marking; should marking be done online or on paper; should questions for answering be supplied on screen as a part of the word processor file or delivered on an exam paper; should students be allowed to opt to word process an exam instead of hand writing their answers; do word processed exams disadvantage those who cannot type well; should extra time be allocated for word processed examinations.

The environmental aspects of the project have also been challenging. Although assistance has been given to the SusteIT project to produce their carbon calculator and total cost of ownership calculator for thin clients, we have so far been unable to complete these to see the figures for the 100 seat Sun Ray cluster due to the problems finalising the server and disk configuration. It is planned to calculate these figures prior to a UCISA conference to Advisory and IT Support staff in July 2009

15 where they

will be presented alongside other green IT topics and the use of thin clients for secure e-assessment. An attempt has been made to save some power by including power switches in the mains leads for the Sun Ray workstations so they can be turned off easily. There is the potential to save enormous amounts of paper by converting OMR exam papers to online assessments and thus also saving on the results sheets; though at the current time both are still being produced to cover any failure of the IT systems

15

http://www.ucisa.ac.uk/groups/ssg/sdg/Events/2009/SDG2009.aspx

Project Acronym: ITS4SEA Version: 2 Contact: s.j.eyre@bradford.ac.uk Date: 1 June 2009

PAGE 13 of 19

The technical issues involved in the project have been a challenge due to the use of new virtualisation software. We have gained a greatly increased knowledge of Sun Ray server technology and increased experience with VMware Virtual Desktop Infrastructure (VDI) during the project. Whether the use of Sun Rays and the servers they require will prove to be financially advantageous or more eco-friendly than using standard PCs is unclear currently but use of Sun Microsystems own calculator shows that comparing traditional PCs with Sun‟s Virtual Desktop Solution for 100 Sun Ray clients results in a £183,932 advantage in total cost of ownership for the thin client solution over 3 years. Much of this is claimed in reduced support costs and less down time for users rather than in hardware or power consumption costs. The University of Bradford has also gained a much better understanding of creating virtual desktops and assigning them to users, smart cards or workstations during this project. While the use of smart cards has not so far been successfully introduced into the examinations process, the option will be available for future years. The ability to use Appsense to both control applications software packages and to manage virtual desktops has been an incredible benefit to both the security of the examinations and to the delivery of the virtual desktop for each exam. We are still investigating the use of NetSupport School to control screens on thin client workstations for teaching purposes and hope to deliver this for teaching in the 09/10 academic year. The use of smart cards within the University continues to evolve as it does at most other Higher Education sites. The impact of the recently announced requirement to report on international students to the UK Borders Agency has become highly relevant and the smartcards@jiscmail.ac.uk list has been revitalised after being apparently dormant at the start of this project. It now has 54 members compared to 12 initially. Dual technology MiFare and JCOP cards have been ordered to allow staff and students to be save Sun Ray sessions and reactivate them later, as well as for access control, printing and copying functionality.

Project Acronym: ITS4SEA Version: 2 Contact: s.j.eyre@bradford.ac.uk Date: 1 June 2009

PAGE 14 of 19

Outcomes

The table below gives the Outcomes expected from the original project plan. The final column shows what the project has or has not achieved towards the planned outcomes.

Outcome Impact Change Project achievements and progress

Improvements to exam facilities for disabled students

Automatic extension to time allowed for exams

Less stress for students and invigilators

Disabled students being correctly picked up from SAINT and allocated extra time and separate room automatically

Have not addressed issues with disabled students word processing their exams

Students simply requiring extra time allocated adjoining space to other students in new assessment cluster

Ability to timetable small classes for simultaneous online assessment

Less conflict of space needs by Exams Office

Reduced crush on exam timetable

Not needed for January trial exams since these were large class sizes

Likely to be needed for August re-sit period

Ability to more easily change questions than offered by OMR

Past exam questions need not be kept secret

Academics can alter questions easily from year to year

Existing 45 OMR questions for Immunology converted to QmP

Order of questions randomised

Correct answer to questions randomised

Ability to offer different types of questions other than „Select A to E‟

May need to engage reluctant academics. May need additional training and support

Students can attempt past exam papers for formative assessment

Provided Excel file for one exam without options to save alterations or access internet

Time needed to develop wider range of questions and question bank for new CAA exams

Ability to move traditional written exams to online assessment

Would require additional staff training and engagement with new technology

Less time consuming marking

Students may need to include diagrams

Three exam sittings mean final cohort will know what essay options are with enough time to revise topics

Exam topics need to be varied between cohorts

Much more work needed on this

Easy maintenance of demonstrators computer in lecture rooms

Lecturer has up to date software immediately available on arrival at lecture room

No Windows updates, anti-virus updates and software patches to install

Unable to link data socket in lecture rooms with both RoamNet and Sun Ray servers due to half completed machine room and network router moves

Dual data sockets now standard in refurbishment of teaching areas may allow deployment in 2009-10

Sun Rays not currently suited to running all teaching applications e.g. Flash. Firmware upgrade due summer 2009 may assist

Results from online assessments stored in student record system automatically

Exam results are accurate. Less time needed by administrative staff to process results

No transcribing of results between different systems and file formats

Student number included in output file from QmP as well as student name to allow easy data transfer into student record system

Done manually for May 08 and January 09 exams

Too many inconsistencies in assessment type codes to automate this. Data

Project Acronym: ITS4SEA Version: 2 Contact: s.j.eyre@bradford.ac.uk Date: 1 June 2009

PAGE 15 of 19

Outcome Impact Change Project achievements and progress

cleansing of module descriptors planned for summer 2009

Impossible to handle repeating students automatically. Unlikely this can be achieved.

Provision of a large assessment cluster

Will allow large groups to be assessed in a shorter time frame and use less individual small PC rooms

Fewer invigilators required

Space for 100 seats developed (inc. lecturer station)

Completed after end of term December 2008 for use with exams first week of 2009!!

Cluster development time scales prevented trials prior to exam period

Encouraged staff to think about different assessment types

Encouraged staff to also think about alternative teaching styles

Expansion in use of CAA

May increase requirements for large Sun Ray clusters

Fewer scripts to mark by hand

9 new modules timetabled for online assessment in May 2009 exams

Over 900 exam seats scheduled for May 2009

The original project plan proposed that the following deliverables would be made available at the end of the project. Many of these have been addressed in earlier parts of this final report but a summary is made following each target to show what was and wasn‟t achieved and the main reason behind this.

Major improvements to E-administration

Improved interfaces and processes within the Examinations Office

The initial stages of implementation of the Exam Admin system has achieved this aim

Improved delivery of individually tailored software configurations for computer based assessment

This has been achieved using Appsense to deliver different configurations for different exams but the process is still manually controlled

Controlled, reliable IT facilities for students with disabilities to take examinations with automatic extension to the time allowed for the online examination

Students with disabilities are scheduled into separate locations so they are not disturbed when other students finish. Exam length controlled by invigilators.

Replacement of existing Optical Mark Recognition based paper examinations with online assessments

Several OMR exams have been replaced during the January and May exam periods. We plan for this trend to continue by encouraging lecturers to think about alternate assessment methods.

Automatic delivery of appropriate assessment to each student if mixed cohorts of students are located in the same examination room

This has not been required so far but should be achievable if required

Automation of transfer of marks between testing software and student record system

Processes not automated due to inconsistencies in coding of assessment and problems with students repeating assessments

Reduced support requirements for PCs in examination and teaching rooms.

Sun Rays have proved MUCH easier to control and have lower failure rate during exams. Some examinations run successfully without allowing 10% failure rate.

Investigation of options for allowing anonymous submission of word processed scripts to examiners without resorting to printing

Not achieved due to unresolved problems related to word processing exams. Discussion with other sites to plan ways forward is required to take place at UCISA E-assessment conference in November 2009

16

16

http://www.ucisa.ac.uk/groups/ssg/asg/Events/2009/eAssessment.aspx

Project Acronym: ITS4SEA Version: 2 Contact: s.j.eyre@bradford.ac.uk Date: 1 June 2009

PAGE 16 of 19

Sustainable developments

Measurement of the resources required to run a large room of Sun Rays and compare it with the requirements of similar facilities based round networked PCs.

The final server infrastructure to support 100 Sun rays was only finalised as this report was being produced. It is planned to develop this aspect of the project by July 2009.

Replacement of a large number of tests which currently use paper based Optical Mark Recognition with online assessment, thus removing the need for large quantities of paper and the laborious process of scanning it.

This has been achieved for an additional 10 modules covering over 1000 student exam papers.

Major Deliverables

Comprehensive documentation on all aspects of the new system which will allow other universities to explore similar applications, e.g. full documentations on how to develop different server images for delivery to Sun Ray terminals for computer based assessments.

Following the successful use of the Sun Ray cluster during two exam periods, it is now vital that this work is documented to allow for the sickness or resignation of the member of staff who ran the processes and was indirectly funded by the project. This will be made available by July 2009 at http://www.brad.ac.uk/lss/jisc/

Scripts and APIs to allow other sites to develop similar systems integration.

Scripts to improve exam scheduling and student selection processes will be made available by July 2009 at http://www.brad.ac.uk/lss/jisc/ The scheduling of tests into Questionmark Perception and the extraction of results back into the student record system were not automated using QMWise due to the unreliability of data related to students taking modules and the coding of the assessment type.

Evaluation of the new e-assessment system against a range of criteria, including educational (e.g. effectiveness in delivery of a wide range of assessment types with improved feedback to students and staff), administrative (e.g. effectiveness and efficiency of procedures) and sustainability (e.g. comparison of the power consumption and management issues relating to use of a large cluster of thin client terminals against comparable PC clusters).

The educational benefits of the new assessment facility are clear, with immediate results being available at the end of each exam or test. Initial student feedback also rates the benefits highly. The benefits to exam admin staff have also been immediate with no paper forms to correct and scan and the results available for immediate transfer into the student record system. A report on the exam scheduling process will be available by July 2009 at http://www.brad.ac.uk/lss/jisc/ The sustainability aspects of using Sun Rays vs. PCs will be reviewed and data made available by July 2009.

Use cases documenting user and stakeholder needs, perspectives and evaluation feedback

To be disseminated at confererence http://www.ucisa.ac.uk/groups/ssg/asg/Events/2009/eAssessment.aspx QmP perspective http://www.questionmark.co.uk/uk/casestudies/bradford.aspx Appsense perspective http://www.appsense.com/Files/Documents/University%20of%20Bradford%20(US).pdf Student evaluation in Appendix C.

Report on handling the needs of disabled students as a part of large scale Computer Aided Assessments

Students with special needs who were scheduled to take online assessments were either timetabled automatically into a separate area of the new assessment facility if they simply had extra time allowed or into individual rooms equipped with a Sun Ray if other facilities were required. Due to a wide variety of different problems around word processing exam, work with students who word process their exam scripts has not been achieved.

Project Acronym: ITS4SEA Version: 2 Contact: s.j.eyre@bradford.ac.uk Date: 1 June 2009

PAGE 17 of 19

Integration of SMART Sympodium tablets with Sun Ray terminal technology.

This has not been achieved due to networking problems related to single sockets in teaching rooms requiring access to both the Sun Ray servers and the University roaming network.

Evaluation of issues of using thin client technology in teaching rooms instead of networked PCs.

As above.

Report on special factors affecting the design of a new cluster designed for online summative assessment.

Appendix A to this report

Conclusions

The project team firmly believes that using Sun Ray thin client technology to support E-assessment does provide a more secure and flexible solution that using standard PCs. We have recently received news that the University of Hanoi has installed a 100 seat Sun Ray cluster for conducting online assessments and plan to roll out duplicate facilities across Vietnam. They have obviously reached the same conclusion! Although smart cards were not actually used for the first trial examinations, they should also assist in providing a more flexible and accessible solution for future examination using the new Sun Ray facility. Integrating the information stored about student disabilities in the Student Record System with the exam scheduling systems improves the interfaces and processes for staff in the Exams Office. The risks and time associated with manually linking results of online assessments to individual students or double entry of marks into the student record system have been minimised by post processing the output file produced by Questionmark to include the student‟s number as well as their login name. Reducing, or better still removing, the use of paper based multiple choice examinations minimises the work required of exam administrators in academic Schools whilst increasing the security of the assessment since questions can both be randomised in the order of delivery and in the correct response. However, these processes rely on the original data being used by the Exam Admin system being correct. The issue log for the project highlights a large number of data errors that must be resolved before the new system can function anywhere near 100% of its capabilities. We faced a number of issues surrounding storage on the servers supporting the Sun Ray workstations, notably with the amount of time required to clone the virtual images and coping with the load from such a high number of consecutive logins. High performance disk arrays are essential to handle these problems in a realistic timescale.

Implications There are a number of issues logged which still require resolution. Some have local work rounds but others will need University procedures adapting in due course. When implementing any of the aspects of this project, sites should ask themselves the following questions.

Technology

Desktop virtualisation – what is the most appropriate system, what server infrastructure does it require, is it cost effective in the long term, does it reduce staff support costs?

Thin clients – which is the most appropriate platform, do they reduce your carbon footprint, do you need smart cards for them to work most flexibly, what applications are they suited for, what software may not run appropriately?

By changing the technology used to deliver online assessments, can the equipment failure rate allocated for exams be reduced?

E-assessments

Does controlling the services that students have access to during online exams using Appsense increase the opportunities for conducting an exam online instead of on paper?

Project Acronym: ITS4SEA Version: 2 Contact: s.j.eyre@bradford.ac.uk Date: 1 June 2009

PAGE 18 of 19

Can the expense of building a large thin client cluster be justified by the savings in staff time running assessments online instead of on paper?

Does doing assessments online affect the student experience?

Exam administration

How can you ensure that data held about students module enrolments are correct to allow for exam timetable accuracy?

How to ensure the correct allocation of appropriate facilities for students with special needs?

How can you ensure that data about module assessment are correct? Length of exam, time of year of assessment, part time vs. full time vs. overseas options all need consideration.

Is it possible to transfer results of online assessments directly into the student record if assessment types are coded consistently?

How can you layout workstations to maximise security during exams?

How best to disseminate individual examination timetables, e.g. personalised email, directly into personal diary, student portal, VLE, SMS?

Can the theories in the design of the ExamAdmin system be incorporated into the examination processes to provide a flexible real-time reporting system that can also be used to drive data cleansing?

Recommendations

Project management

Do not underestimate the amount of project management time that will be required for projects with multiple goals to achieve. Two days a week were needed towards the January 2009 exam period to handle building work, Sun Ray developments, smart card issues and data quality problems.

Writing final and completion reports is very time consuming after 18 months of continuous development and implementation work.

For HEIs developing E-assessment facilities

Get a realistic assessment of how many exams can reasonably be moved to an online format and assess how many students would take each of the exams in order to size the optimum number of workstations required in any new assessment facility.

Get the support of all the academic staff who would benefit from the introduction of a new assessment facility to back a bid for space and budget.

Do not start development of a new assessment facility with only 7 months to go until it is needed! The Bradford University facility was approved in mid–April and handed over in mid-December prior to exams starting on January 5

th; this did not leave sufficient time for testing

and student familiarisation even though the builders did not run a single day late!

Make sure all students have an opportunity to familiarise themselves with new examination formats and locations to minimise any additional stress during summative assessments.

Give students a formative assessment before any high stakes exams and ask them for their feedback on what facilities would help them adapt to the new online format more comfortably. Make sure you address what they tell you.

If existing PC facilities are being redeveloped, try to ensure there is room for invigilators to move between workstations; this may require the number of workstations to be reduced. Consider if using different shaped tables would minimise screen visibility between neighbouring workstations; this could require physically different PC configuration such as piggy-backed system units.

Project Acronym: ITS4SEA Version: 2 Contact: s.j.eyre@bradford.ac.uk Date: 1 June 2009

PAGE 19 of 19

For HEIs seeking to improve their E-assessment processes

Make sure there are robust guidelines about the running of PC based exams.

Ensure staff are properly trained in producing online assessments that are fair, secure, appropriate and accessible.

Ensure you have an appropriate assessment tool that is known to be reliable and flexible in its question formats.

Decide in advance how to handle multiple exam sittings. This will depend on the number of cohorts and whether the online assessment has to be identical for all candidates.

For HEIs planning to improve their examination timetabling

Engage examination administrators within academic schools with a data quality and cleansing exercise before any steps are taken to automate examination processes centrally.

Before trying to automate scheduling of exams, ensure that details of module assessment type, length and date are correct – MAJOR task! Tools which may assist with this for SITS and Syllabus Plus customers will be available from http://www.brad.ac.uk/lss/jisc/ by the end of June 2009.

Before sending personalised exam timetables to students, ensure that student enrolments against modules are correct – another MAJOR task.

Ensure students who are entitled to special arrangements for examinations have the correct entitlement recorded and that the details are available electronically so examination locations can be allocated to them automatically and appropriately.

Inform examination administrators within academic schools that they cannot tell students when their exams will be held before the exam timetable is produced.

Consider the technology options

Investigate the potential of using thin client workstations for examinations. They are much easier to re-image between exams than standard PCs so more exams can be scheduled into a location within the day.

Do not assume that a small, trial thin client cluster will scale upwards seamlessly.

Consider the use of thin client technology in teaching rooms as they should provide a more reliable, easier to support technology than standard PCs but may not currently be suited to all applications.

Use the tools developed by the SusteIT project to decide if the total cost of ownership and carbon emissions of thin clients are advantageous in your situation.

Investigate tools such as Appsense which could be used to make the online environment much more secure during exams than running standard web browsers and applications software.

If interested in smart cards and their uses, join the smartcards@jiscmail.ac.uk mailing list and the Higher Education Smart Card Association

17.

Appendixes

Appendix A. Designing a new facility for Online Assessments18

Appendix B. ExamAdmin Draft Functional Design Specification19

Appendix C. Student exam evaluation20

Thanks to Dr Liz Carpenter, FHEA, Lecturer in Medical Physiology, Clinical Sciences, School of Life Sciences for use of her student evaluation

17

http://www.hesca.com/ 18

http://www.brad.ac.uk/lss/jisc/layout.pdf 19

http://www.brad.ac.uk/lss/jisc/examadmin.pdf 20

http://www.brad.ac.uk/lss/jisc/evaluation.ppt