RAMADA Loughborough HOTEL Working experience in Loughborough Ilyasova Vladlena.
11th International CAA Conference, Research into e-Assessment, Burleigh Court International...
-
Upload
jasper-shelton -
Category
Documents
-
view
214 -
download
0
Transcript of 11th International CAA Conference, Research into e-Assessment, Burleigh Court International...
![Page 1: 11th International CAA Conference, Research into e-Assessment, Burleigh Court International Conference Centre, Loughborough University, 10th July 2007.](https://reader036.fdocuments.in/reader036/viewer/2022062717/56649e175503460f94b037f1/html5/thumbnails/1.jpg)
11th International CAA Conference, Research into e-Assessment, Burleigh Court International Conference Centre, Loughborough University, 10th July 2007
BENEFITS AND OBSTACLES: FACTORS AFFECTING THE UPTAKE OF CAA IN UNDERGRADUATE COURSES
John Dermo,
Pathfinder Project Team, University of Bradford
![Page 2: 11th International CAA Conference, Research into e-Assessment, Burleigh Court International Conference Centre, Loughborough University, 10th July 2007.](https://reader036.fdocuments.in/reader036/viewer/2022062717/56649e175503460f94b037f1/html5/thumbnails/2.jpg)
2
First, a plea for your help….
“The idea being that authors use the conference to air their work and receive feedback to enable judgements to be taken on progress to date and how things may be taken forward in future.”
(extract from Submissions: 11th CAA Conference)
http://www.caaconference.com/submissions/index.asp
![Page 3: 11th International CAA Conference, Research into e-Assessment, Burleigh Court International Conference Centre, Loughborough University, 10th July 2007.](https://reader036.fdocuments.in/reader036/viewer/2022062717/56649e175503460f94b037f1/html5/thumbnails/3.jpg)
3
Background to the Research Project
University of Bradford
• e-Strategy
• VLE = Blackboard ™
• CAA = QM Perception™
![Page 4: 11th International CAA Conference, Research into e-Assessment, Burleigh Court International Conference Centre, Loughborough University, 10th July 2007.](https://reader036.fdocuments.in/reader036/viewer/2022062717/56649e175503460f94b037f1/html5/thumbnails/4.jpg)
4
Developments in CAA @ UoB so far include:
• Deciding on Questionmark™ Perception™ as our supported enterprise level software for online summative assessment
• Encouraging and supporting its use in formative assessment and feedback
• Centralising the administrative support for all summative assessments in our Examinations Office
• Implementing Questionmark™ Perception™ version 4.3 with a server configuration to ensure security and reliability
![Page 5: 11th International CAA Conference, Research into e-Assessment, Burleigh Court International Conference Centre, Loughborough University, 10th July 2007.](https://reader036.fdocuments.in/reader036/viewer/2022062717/56649e175503460f94b037f1/html5/thumbnails/5.jpg)
5
Pathfinder Project @ UoB
Investment in
1. CAA
2. Enhancing FYE
March 2007-April 2008
![Page 6: 11th International CAA Conference, Research into e-Assessment, Burleigh Court International Conference Centre, Loughborough University, 10th July 2007.](https://reader036.fdocuments.in/reader036/viewer/2022062717/56649e175503460f94b037f1/html5/thumbnails/6.jpg)
6
Pathfinder Project: CAA Aims and Objectives
to embed support processes for e-assessment
to develop the administrative and support systems to ensure reliable and secure large-scale implementation of CAA with first year students so that we can subsequently roll this out to all students
![Page 7: 11th International CAA Conference, Research into e-Assessment, Burleigh Court International Conference Centre, Loughborough University, 10th July 2007.](https://reader036.fdocuments.in/reader036/viewer/2022062717/56649e175503460f94b037f1/html5/thumbnails/7.jpg)
7
Pathfinder Project: CAA Deliverables
A system to support the effective implementation of formative and summative computer-aided assessment across the university, consisting specifically of:
• baseline study and needs analysis report on use of CAA at UoB• research data (quantitative and qualitative) on use of CAA in undergraduate
courses• official guidelines for CAA: admin and support functions to ensure resilient and
manageable online assessment routes• pedagogical guidelines (“best practice”) document for recommended use of
software• training sessions for instructors (Sept/Jan/May)• induction and training sessions for invigilators• in-house training materials and web-based resources for use of software• case studies of institutional implementation of CAA• banks of online assessment materials
![Page 8: 11th International CAA Conference, Research into e-Assessment, Burleigh Court International Conference Centre, Loughborough University, 10th July 2007.](https://reader036.fdocuments.in/reader036/viewer/2022062717/56649e175503460f94b037f1/html5/thumbnails/8.jpg)
8
Aims of my research project
• gather data on attitudes towards computer-assisted assessment
• to identify drivers and obstacles to the uptake of CAA
• test various widely-held assumptions about attitudes towards e-learning
• inform Pathfinder project• anticipate risks
![Page 9: 11th International CAA Conference, Research into e-Assessment, Burleigh Court International Conference Centre, Loughborough University, 10th July 2007.](https://reader036.fdocuments.in/reader036/viewer/2022062717/56649e175503460f94b037f1/html5/thumbnails/9.jpg)
9
Methodology
• descriptive
• survey opinions of comprehensive range of interested stakeholders:– students, academic staff, administrators,
technical and support staff, management and financial departments.
• primarily a qualitative approach
![Page 10: 11th International CAA Conference, Research into e-Assessment, Burleigh Court International Conference Centre, Loughborough University, 10th July 2007.](https://reader036.fdocuments.in/reader036/viewer/2022062717/56649e175503460f94b037f1/html5/thumbnails/10.jpg)
10
Phases of research
• Informal interviews, desktop research to identify key areas of interest
• Online questionnaires
• Follow-up interviews to investigate issues arising
![Page 11: 11th International CAA Conference, Research into e-Assessment, Burleigh Court International Conference Centre, Loughborough University, 10th July 2007.](https://reader036.fdocuments.in/reader036/viewer/2022062717/56649e175503460f94b037f1/html5/thumbnails/11.jpg)
11
Challenges: methodology
• Quantitative vs Qualitative
• Positivist vs Anti-positivist
• Survey vs Case study
How can we draw generalisable conclusions based on individual personal opinions?
![Page 12: 11th International CAA Conference, Research into e-Assessment, Burleigh Court International Conference Centre, Loughborough University, 10th July 2007.](https://reader036.fdocuments.in/reader036/viewer/2022062717/56649e175503460f94b037f1/html5/thumbnails/12.jpg)
12
Challenges: methods
• Mode of delivery (online vs paper)• Item design (open vs closed)• Subject groups and sampling• Piloting• Assuring confidentiality• How to survey opinion on something never
experienced?
![Page 13: 11th International CAA Conference, Research into e-Assessment, Burleigh Court International Conference Centre, Loughborough University, 10th July 2007.](https://reader036.fdocuments.in/reader036/viewer/2022062717/56649e175503460f94b037f1/html5/thumbnails/13.jpg)
13
Important Note:
Spend time at the outset to make sure that the research tools are reliable and valid before moving to data collection.
All too often researchers rush into the data collection stage, only to find that the data they are collecting are useless and any findings based on it equally so.
![Page 14: 11th International CAA Conference, Research into e-Assessment, Burleigh Court International Conference Centre, Loughborough University, 10th July 2007.](https://reader036.fdocuments.in/reader036/viewer/2022062717/56649e175503460f94b037f1/html5/thumbnails/14.jpg)
14
Initial findings and looking ahead
• research still at initial phase• key areas of interest being put to subjects to
inform the main survey questions to come• reviewing the literature• It is hoped that feedback from conference
delegates will be able to feed into this process!!
![Page 15: 11th International CAA Conference, Research into e-Assessment, Burleigh Court International Conference Centre, Loughborough University, 10th July 2007.](https://reader036.fdocuments.in/reader036/viewer/2022062717/56649e175503460f94b037f1/html5/thumbnails/15.jpg)
15
Drivers
savings in human and financial resources
improved reliability in marking
ease of production of results and item analysis data
ease of creation of different versions and randomised assessments
recycling assessments positive backwash effect on teaching and learning
appeal to “digital native” students possible benefits for recruitment and retention
potential of portfolio assessment accessibility issues
encouraging good assessment practise concerning item banking and item analysis
![Page 16: 11th International CAA Conference, Research into e-Assessment, Burleigh Court International Conference Centre, Loughborough University, 10th July 2007.](https://reader036.fdocuments.in/reader036/viewer/2022062717/56649e175503460f94b037f1/html5/thumbnails/16.jpg)
16
Obstacles 1
limited suitable task types inability to assess higher level skills in a valid way
high risk of technical failure initial outlay of time
steep learning curve for instructors high cost of software licenses and support plans
difficulty in convincing examination boards and QAA concerning issues of quality
anonymous submission of assignments
security issues – e.g. passwords / collaboration / collusion / cheating / impersonation
item banking requiring more effort and time
technical expertise required of instructors
lack of immediate technical support
![Page 17: 11th International CAA Conference, Research into e-Assessment, Burleigh Court International Conference Centre, Loughborough University, 10th July 2007.](https://reader036.fdocuments.in/reader036/viewer/2022062717/56649e175503460f94b037f1/html5/thumbnails/17.jpg)
17
Obstacles 2
difficulties for administrators difficulties for invigilators
training implications accessibility issues
health and safety issues difficulty of instructors in moving away from traditional task types
issues of task design threat that CAA will be used to justify increased class sizes or staff reductions
lack of an agreed and enforced institutional policy
discrimination against “non-digital native” students
limited availability of Internet-connected computers at home, in halls of residences, on campus
availability of large computer rooms for examinations
![Page 18: 11th International CAA Conference, Research into e-Assessment, Burleigh Court International Conference Centre, Loughborough University, 10th July 2007.](https://reader036.fdocuments.in/reader036/viewer/2022062717/56649e175503460f94b037f1/html5/thumbnails/18.jpg)
18
Issues to consider
• Does the cumulative effect of the barriers outweigh that of the drivers?
• How to interpret the data in a meaningful way?• To what degree can e-assessment add value to
the learning experience? • Do some factors work as drivers under some
circumstances but as obstacles in others?
![Page 19: 11th International CAA Conference, Research into e-Assessment, Burleigh Court International Conference Centre, Loughborough University, 10th July 2007.](https://reader036.fdocuments.in/reader036/viewer/2022062717/56649e175503460f94b037f1/html5/thumbnails/19.jpg)
19
Closing comment…
The researcher is keen to involve the input of other experienced practitioners and researchers in the field of Computer Assisted Assessment by means of this conference, and hopes to work this short paper up to a full paper submission for the next event in 2008.
![Page 20: 11th International CAA Conference, Research into e-Assessment, Burleigh Court International Conference Centre, Loughborough University, 10th July 2007.](https://reader036.fdocuments.in/reader036/viewer/2022062717/56649e175503460f94b037f1/html5/thumbnails/20.jpg)
20
References
Cohen, L., Manion, L., Morrison, K. (2003) Research Methods in Education. London: RoutledgeFalmer.
Conole, G. (2003) Understanding Enthusiasm and Implementation: e-Learning Research Questions and Methodological Issues in JK. Seale (ed) Learning Technology in Transition: from Individual Enthusiasm to Institutional Implementation, Abingdon: Swets and Zeitlinger, pp. 129-146.
Dillman, DA. (2007) Mail and Internet Surveys: the Tailored Design Method. Hoboken, New Jersey: John Wiley & Sons.
Frankfort-Nachmias, C. and Nachmias, D. (1996) Research Methods in the Social Sciences. London: Edward Arnold.
Oppenheim, AN. (2000) Questionnaire Design: Interviewing and Attitude Measurement. London: Continuum International.
Verma, GK. and Mallick, K. (1999) Researching Education: Perspectives and Techniques. London: Falmer Press.
![Page 21: 11th International CAA Conference, Research into e-Assessment, Burleigh Court International Conference Centre, Loughborough University, 10th July 2007.](https://reader036.fdocuments.in/reader036/viewer/2022062717/56649e175503460f94b037f1/html5/thumbnails/21.jpg)
21
Thank you
Contact: John DermoPathfinder Project,
Learner Support Services,
University of Bradford
Email : [email protected]