Customisation - Functional Specification Enhancements in Debrief
Anonymous assignments: Core product or customisation? University of York.
-
Upload
lydia-waters -
Category
Documents
-
view
219 -
download
0
description
Transcript of Anonymous assignments: Core product or customisation? University of York.
![Page 1: Anonymous assignments: Core product or customisation? University of York.](https://reader036.fdocuments.in/reader036/viewer/2022062317/5a4d1af27f8b9ab05997eb6a/html5/thumbnails/1.jpg)
Anonymous assignments: Core product or customisation?
University of York
![Page 2: Anonymous assignments: Core product or customisation? University of York.](https://reader036.fdocuments.in/reader036/viewer/2022062317/5a4d1af27f8b9ab05997eb6a/html5/thumbnails/2.jpg)
2
Session overview
1. EMA in the UK
2. EMA workflow requirements
3. Anonymous assignment submission @ York
4. Blackboard’s anonymous assignment submission
5. Ways forward: core product or bespoke development?
![Page 3: Anonymous assignments: Core product or customisation? University of York.](https://reader036.fdocuments.in/reader036/viewer/2022062317/5a4d1af27f8b9ab05997eb6a/html5/thumbnails/3.jpg)
3
Session outcomes
• You– what we do at York– what BB does themselves– where do you need to be?
• Me– benchmark York within the sector– inform use of BB anon assignment – sanity check
• BlackBoard– anon assignment user feedback
![Page 4: Anonymous assignments: Core product or customisation? University of York.](https://reader036.fdocuments.in/reader036/viewer/2022062317/5a4d1af27f8b9ab05997eb6a/html5/thumbnails/4.jpg)
4
Electronic Management of Assessment (EMA) in the UK
• Growing adoption and interest across the sector (UCISA 2014)
• Strategic priority for JISC / UCISA / HELF identified benefits for– Students (convenience)– Admin (efficiencies)– Academics (pedagogic)
• Turnitin Grademark leading e-submission solution, 63%
• End to end solution remains elusive– JISC, Southampton, Northumbria, BB/SITS integration?
![Page 5: Anonymous assignments: Core product or customisation? University of York.](https://reader036.fdocuments.in/reader036/viewer/2022062317/5a4d1af27f8b9ab05997eb6a/html5/thumbnails/5.jpg)
5
Assessment and feedback lifecycle
1 - Specifying
2 - Setting
3 - Supporting
4 - Submitting
5 - Harvesting
the work
6 - Marking and
feedback
7 - Returning marks and feedback
8 Reflecting / evaluation
Lifecycle model adapted from Manchester University TRAFFIC Project
![Page 7: Anonymous assignments: Core product or customisation? University of York.](https://reader036.fdocuments.in/reader036/viewer/2022062317/5a4d1af27f8b9ab05997eb6a/html5/thumbnails/7.jpg)
7
EMA @ York: Contextual factors
• Anonymity mandatory “except where unfeasible”– Assessment principles: Equity, Openness, Clarity, Consistency
• No current institutional EMA policy /mandate, The “York Way”
• ~14,000 students
• Change is coming…– New L&T strategy and faculties– Growing DL provision
![Page 8: Anonymous assignments: Core product or customisation? University of York.](https://reader036.fdocuments.in/reader036/viewer/2022062317/5a4d1af27f8b9ab05997eb6a/html5/thumbnails/8.jpg)
8
Anonymous assignment submission at York
• Bespoke building block deployed since 2007 for file based workflow
• Iterative development to support submission and feedback return (2014/15)
• Complete interdependence on SITS data; student exam number and assessment records
• Student work anonymised on submission and tracked throughout the rest of the lifecycle by exam number
• Flexible “barebones” toolset maps onto familiar admin processes
• 20,558 files submitted, 11,267 feedback files returned online this academic year (Oct 14 - March 2015)
![Page 9: Anonymous assignments: Core product or customisation? University of York.](https://reader036.fdocuments.in/reader036/viewer/2022062317/5a4d1af27f8b9ab05997eb6a/html5/thumbnails/9.jpg)
9
York Anonymous Assignment: Workflow & features
• Deadlines and late submission alerts• Restrict file types (access / annotation)Setting
• Submitted files (30MB max) labelled with exam no & timestamped• E-mail receipt and alertsSubmitting
• Associate submissions with SITS assessment record • Download options; submissions, feedback (blank / template / annotated
work) marksheet, non-submission and tracking info
Harvesting the work
• Create feedback in standalone form / annotate work• Marks recorded in marksheet
Marking / feedback
• Completed feedback and marks batch uploaded and made available to students through SITS e:vision
Returning marks / feedback
VLE
Dept defined (shared drive
/ Google…)
SITS e:vision
![Page 10: Anonymous assignments: Core product or customisation? University of York.](https://reader036.fdocuments.in/reader036/viewer/2022062317/5a4d1af27f8b9ab05997eb6a/html5/thumbnails/10.jpg)
10
York Anonymous Assignment: Evaluation Highlights
• Institution: Need to fill gaps in policy and guidance
• Admin (n=18): Easy to use and saves time
• Students (n=261): Generally +ive, FB quality, confidence issues
• Markers (n=48): Mixed reaction to all aspects of workflow– Most negative response to reading work on screen– Broadly positive response to feedback production – Areas for development:
• Improved workflow and speed / More managed system• Annotations – awareness, workflow, reusable, workload implications• Resistance to greater use of technology
![Page 11: Anonymous assignments: Core product or customisation? University of York.](https://reader036.fdocuments.in/reader036/viewer/2022062317/5a4d1af27f8b9ab05997eb6a/html5/thumbnails/11.jpg)
11
Polarised response
“I can see no merit in this system. I have other experience of on-line marking from my external examining and have found it very difficult, time consuming and problematic. There are some parts of the pedagogic process that are not amenable to on-line systems and marking essays is one of these. The student sget a worse experience, worse feedback and and I have found it tiresome, unnecessarily complicated and much much slower. I conclude that it is inefficient use of time, makes turning the essays round longer, and pedagogically worse outcome.”
“This was my first year using this system and it was revolutionary. I could mark anywhere (just using a USB stick and a backup) and there was no copying of disks. Also there was no printing out of forms at the Dept Office. The whole thing was slick and easy.”
![Page 12: Anonymous assignments: Core product or customisation? University of York.](https://reader036.fdocuments.in/reader036/viewer/2022062317/5a4d1af27f8b9ab05997eb6a/html5/thumbnails/12.jpg)
12
York Anonymous Assignment: Pros / Cons
• Pros– Familiar and evolutionary
• In use since 2007• File based – closely aligned with paper based workflows• Significant “organic” opt-in from Departments
– Batch upload of feedback created offline– Low entry point – markers do not need to engage with VLE– Technology imposed where it counts– No possibility of anonymity being circumvented
• Cons– Feedback production cumbersome for some– Anonymity is permanent; sharing work with supervisors etc…– End to end workflow– Missed opportunity for greater use of TEL?
![Page 13: Anonymous assignments: Core product or customisation? University of York.](https://reader036.fdocuments.in/reader036/viewer/2022062317/5a4d1af27f8b9ab05997eb6a/html5/thumbnails/13.jpg)
13
BlackBoard Anonymous assignment – step by step
1 – Activate / deactivate when editing an assignment
![Page 14: Anonymous assignments: Core product or customisation? University of York.](https://reader036.fdocuments.in/reader036/viewer/2022062317/5a4d1af27f8b9ab05997eb6a/html5/thumbnails/14.jpg)
14
BlackBoard Anonymous assignment – step by step
2 – Students submit and advised of anonymity
![Page 15: Anonymous assignments: Core product or customisation? University of York.](https://reader036.fdocuments.in/reader036/viewer/2022062317/5a4d1af27f8b9ab05997eb6a/html5/thumbnails/15.jpg)
15
BlackBoard Anonymous assignment – step by step
3 – Student submissions obfuscated in the GradeCentre
![Page 16: Anonymous assignments: Core product or customisation? University of York.](https://reader036.fdocuments.in/reader036/viewer/2022062317/5a4d1af27f8b9ab05997eb6a/html5/thumbnails/16.jpg)
16
BlackBoard Anonymous assignment – step by step
4 – Read and provide feedback anonymously
![Page 17: Anonymous assignments: Core product or customisation? University of York.](https://reader036.fdocuments.in/reader036/viewer/2022062317/5a4d1af27f8b9ab05997eb6a/html5/thumbnails/17.jpg)
17
BB Anon Assignment @ York?
“I think that having the materials submitted online is good, and I prefer marking digitally. I found the VLE comments section less useful. I downloaded the essays and used the track changes function of Word, which worked great for me.”
“I've also been trialling non-anonymous marking through the VLE for one module - where we annotated on screen. I really liked providing feedback this way.”
• Marker preference for working offline– Batch upload of feedback– Downloading doesn’t respect Smart Views / Delegation– No delegation without reconciliation
• Interface– Learning a new system– Limited display area – is it as good as reading / annotating in Word?
• Integration with SITS e:vision – untested @ York
• Anonymity via randomly generated number– Tracking non submissions etc– All or nothing
![Page 18: Anonymous assignments: Core product or customisation? University of York.](https://reader036.fdocuments.in/reader036/viewer/2022062317/5a4d1af27f8b9ab05997eb6a/html5/thumbnails/18.jpg)
18
Comparing York and BB Anonymous submission
Feature York BB
Specifying Limited to 30MB No limit
Setting File restrictionsRequires SITS
Formative / summative
Supporting Staff for more advanced use Greater staff to get started
Submitting Anonymous tracking Tracking difficulty
Harvesting the work
Admin download filesFile management issues
Direct access by markersDownload issues
Marking and feedback
Offline onlyFamiliar feedback approachesSimple paper readingDisability stickers
Paper logistically trickyFlexible Feedback toolsManual entry into VLE – no batch upload
Returning marks / FB
Pass to AdminBatch upload to SITS
Direct through VLEUntested SITS integration
Reflecting / evaluation
Anonymous identifier cannot be shared
Possibility within the VLELongitudinal view problems
![Page 19: Anonymous assignments: Core product or customisation? University of York.](https://reader036.fdocuments.in/reader036/viewer/2022062317/5a4d1af27f8b9ab05997eb6a/html5/thumbnails/19.jpg)
19
Does BB Anon Assignment meet our workflow requirements?
• http://goo.gl/hMfx5q
![Page 20: Anonymous assignments: Core product or customisation? University of York.](https://reader036.fdocuments.in/reader036/viewer/2022062317/5a4d1af27f8b9ab05997eb6a/html5/thumbnails/20.jpg)
20
Links and resources
• JISC EMA guidance - http://www.jisc.ac.uk/guides/electronic-assessment-management
![Page 21: Anonymous assignments: Core product or customisation? University of York.](https://reader036.fdocuments.in/reader036/viewer/2022062317/5a4d1af27f8b9ab05997eb6a/html5/thumbnails/21.jpg)
21
Detail on Evaluation outcomes follows
![Page 22: Anonymous assignments: Core product or customisation? University of York.](https://reader036.fdocuments.in/reader036/viewer/2022062317/5a4d1af27f8b9ab05997eb6a/html5/thumbnails/22.jpg)
22
Evaluation: Administrators (n=18 from 11 depts)
• Easy to use, adoption simplified by fit with current practices
• Saves time – Student submission – 4.67 / 5– Sharing files with markers – 4.25 / 5– Adding marks to e:vision – 4.43 / 5– Returning feedback to students 5 / 5– Complete workflow 4.69 / 5
• Minor enhancement requests
• Highlighted policy and guidance blindspots
![Page 23: Anonymous assignments: Core product or customisation? University of York.](https://reader036.fdocuments.in/reader036/viewer/2022062317/5a4d1af27f8b9ab05997eb6a/html5/thumbnails/23.jpg)
23
Evaluation: Students (n=261 from 15 depts)
• Clear preference for “easy” online submission– Positive impact on student experience
• Preference for online feedback (delivery mode)– Feedback “utility” not linked to delivery mode– Correlation between feedback utility and feedback type (annotations
preferred)
• General lack of confidence in IT systems– Potential to address with guidance / training
![Page 24: Anonymous assignments: Core product or customisation? University of York.](https://reader036.fdocuments.in/reader036/viewer/2022062317/5a4d1af27f8b9ab05997eb6a/html5/thumbnails/24.jpg)
24
Markers: overall reaction
• Mixed reactions across all aspects of the workflow; – access to files– reading and marking– feedback production & quality– recording marks
• Broad acceptance from most markers, vitriolic resistance from some opponents
Neutral38%
Postive34%
Negative28%
![Page 25: Anonymous assignments: Core product or customisation? University of York.](https://reader036.fdocuments.in/reader036/viewer/2022062317/5a4d1af27f8b9ab05997eb6a/html5/thumbnails/25.jpg)
25
Markers: reading and marking work
• 83% read all work on screen despite options to print– 41% - “about the same as reading on paper”– 40% - “generally worse than paper”– 19% - “generally better than on paper”
• Polarised / negative response to marking on tablets– 33% - “Not at all – I hate the idea”– 29% - “Not bothered either way”– 23% - “Yes I really want to be able to do this”– 15% - “Other” (mainly qualified enthusiasm)
![Page 26: Anonymous assignments: Core product or customisation? University of York.](https://reader036.fdocuments.in/reader036/viewer/2022062317/5a4d1af27f8b9ab05997eb6a/html5/thumbnails/26.jpg)
26
Markers: feedback provision and quality
• “Easier to give more tailored (and legible) feedback to students. Also was possible to use "boilerplate" text to multiple students who had made similar errors.”
![Page 27: Anonymous assignments: Core product or customisation? University of York.](https://reader036.fdocuments.in/reader036/viewer/2022062317/5a4d1af27f8b9ab05997eb6a/html5/thumbnails/27.jpg)
27
Markers: Top suggested areas for development
• Improved workflow and speed– “The whole system is very cumbersome and clunky.”
• Annotations– Awareness– Workflow– Reusable– Workload implications / strategic design of feedback
• More managed system; VLE tools / Grademark– “Many other institutions use more stable systems where scripts and marks are linked
and can be accessed and viewed more easily.”
• Resistance to greater use of technology– “I would much prefer to leave things as they are and continue with the paper-based
system. I work on the computer all day and do not want to mark on the screen as well.”
![Page 28: Anonymous assignments: Core product or customisation? University of York.](https://reader036.fdocuments.in/reader036/viewer/2022062317/5a4d1af27f8b9ab05997eb6a/html5/thumbnails/28.jpg)
28
![Page 29: Anonymous assignments: Core product or customisation? University of York.](https://reader036.fdocuments.in/reader036/viewer/2022062317/5a4d1af27f8b9ab05997eb6a/html5/thumbnails/29.jpg)
29