Increasing learner success with technology supported assessment:

September 2007 JISC CETIS Assessment SIG

Increasing learner success with Increasing learner success with technology supported assessment: technology supported assessment:

Findings from the REAP projectFindings from the REAP project

Martin HawkseyTeaching & Learning Technology Adviser

www.reap.ac.uk

[email protected]


PlanPlan

• Defining assessment and feedback• Outline the REAP Project and its philosophy

towards a ‘principled’ approach to course redesign

• Some examples of REAP pilots• Questions and discussion


Defining assessment and feedbackDefining assessment and feedback

• Assessment – the narrow meaning is exam for external accreditation

• Feedback – the narrow sense is what the tutor writes about/on a finished piece of student work

• In REAP we are are also interested in:

• Self-assessment and reflection - a form of internally generated feedback

• Peer dialogue as feedback and peer assessment


Definitions (2)Definitions (2)

• Who is involved in formative assessment and feedback• Tutor• Peers• External (e.g. placement supervisor)• Computer generated• Self


Why take formative assessment and feedback seriously?Why take formative assessment and feedback seriously?

• Assessment is a key driver of student learning

• Assessment is a major cost in HE (staff time)

• Widely reported that students don’t read the feedback

• Dropout/retention – linked to academic experience

• First year experience – students need regular and structured feedback opportunities.

• National student survey (NSS) – students are dissatisfied with feedback.

• QAA reports – main area of criticism in England


NSS 2007 - Assessment and NSS 2007 - Assessment and Feedback ResultsFeedback Results

• Nationally only 55% of students think feedback is prompt and had helped to clarify things they did not understand [Scotland: 48%]

• Nationally only 63% of students agree that have received detailed comments on their work [Scotland: 49%]


Key messagesKey messages

• Formative assessment and feedback by others can only have an impact on learning when it influences a student’s own self-regulatory processes (adapted from Boud, 1995)

• Students are already self-assessing and generating feedback so we should build on this capacity (Nicol and Macfarlane-Dick, 2004)


• 3 HEIs (Strathclyde, Glasgow Caledonian Business School, Glasgow University)

• Targeting large 1st year classes• Multi-disciplinary as well as faculty-wide (19 pilots, ~6000 students)• Range of technologies: online tests, simulations, discussion boards,

e-voting, e-portfolios, peer/feedback software, admin systems, VLEs, offline-online

• Goals: Learning quality and teaching efficiencies• Outputs: case studies of redesign: advice to support strategic

change in institutions (transformation).• A ‘principled’ approach to designing and embedding assessment

practices• Underlying pedagogy: assessment for learner self-regulation

REAPREAP


Scaffolding self regulation: 7 principles of Scaffolding self regulation: 7 principles of good feedback (assessment design)good feedback (assessment design)

1. Clarify what good performance is (goals, criteria, standards).

2. Facilitate reflection and self-assessment in learning

3. Deliver high quality feedback to students: feedback that enables students to monitor and self-correct

4. Encourage peer and tutor dialogue around learning

5. Encourage positive motivational beliefs & self esteem through assessment

6. Provide opportunities to close the feedback loop

7. Use feedback information to shape teachingSource: Nicol and Macfarlane-Dick (2006)


1. Help clarify what good performance is1. Help clarify what good performance is

Q: Do students actively engage with assessment criteria and standards?

• Documents with printed criteria/ published online• Tutor explains/discusses criteria with students• Frequently asked questions database (FAQs)• Exemplars of performance with feedback (e.g. essays - - made

available online)• Students discuss good and bad examples of assignments and

identify criteria• Students rephrase criteria in own words• Students generate their own criteria for a task• Students construct own MCQs and post online


2. Facilitates self assessment 2. Facilitates self assessment and reflection in learningand reflection in learning

Q: Are there formal/informal opportunities for self and peer assessment processes?

• Online quizzes (e.g. MCQs/short answers) with feedback (self monitoring)

• Self-assessment at submission (strengths/weaknesses)• Open book use of online MCQs• Peer processes where students judge work of others against

standards and give each other feedback• Simulations – dynamic feedback shows effects of actions• PDP involving ongoing reflection on learning (e-portfolios)• Confidence ratings with MCQs increases reflection


3. Deliver high quality feedback to students3. Deliver high quality feedback to students

Q: What kind of feedback is provided - does it help students to self-assess, self-correct?

• Link feedback to pre-defined criteria (comment databanks)

• Provide feedback soon after submission (clarify expectations)

• Point to resources where answers can be found (hyper-links)

• Have students request feedback • Reader-response theory and feedback


4. Encourage teacher and student 4. Encourage teacher and student dialogue around learningdialogue around learning

Q: Are there opportunities for dialogue around assessment tasks?

• Break out discussions in tutorial to discuss written feedback and plan strategies for improvement

• Students give descriptive feedback to peers before submission

• Electronic voting technology – three levels of feedback (computer, peer and teacher)

• Unique weekly assessed tutorial sheets • Online discussion as peer feedback (psychology

example)


5. Encourage positive motivational beliefs 5. Encourage positive motivational beliefs (self-efficacy) and self-esteem(self-efficacy) and self-esteem

Q: Does feedback focus students on learning not just on their marks?

• Online objective tests (repeatable, private, comparisons with own learning goals)

• Using authentic ‘real life’ tasks that require the kinds of thinking and skills required in employment

• Structured milestones with early low stakes tasks and feedback • Focus on learning goals not just performance goals (e.g. rewarding

effort and improvement)• Two-stage assignments (drafts-feedback-marks) • Supporting formation of learning communities around tasks

(discussion boards)


6. Provide opportunities to close 6. Provide opportunities to close the feedback loopthe feedback loop

Q: Is feedback attended to and acted upon by students?

• Increase opportunities for resubmission• Two-stage assignments (drafts-feedback-marks) • Action points as feedback strategy• Personal development planning – students keep records of work at

different stages• Initiate online peer review processes as available to academics

when writing journal articles• Not releasing the grade until students have commented on the

feedback provided.


7. Provides information that can be used to 7. Provides information that can be used to shape the teachingshape the teaching

Q: How is feedback used to inform and shape teaching?

• Student requested feedback (questions worth asking)

• Electronic voting systems

• Just-in time-teaching using MCQs

• Monitoring online discussions

• Monitoring project work within shared workspaces

• Blackboard/WebCT have some built in reporting functionality about student activity


Two super principlesTwo super principles

Super-principle 1: developing learner self-regulation (empowerment) i.e steers to encourage ownership of learning – the seven principles discussed above.

Super-principle 2: time on task and effort (engagement) i.e. steers on how much work to do and when – Gibbs and Simpson 4 conditions

Case examples from REAP – applying these principles/conditions


REAP Pilots (1)REAP Pilots (1)

university of strathclyde

psychology foundation pharmacy

mechanical engineering pharmacy practice

3 modern languages hospitality & tourism management marketing

educational & professional studies childhood & primary studies

management development programme glasgow

caledonian university

consumer buyer behaviour economics, markets & enterprise marketing

fundamentals managerial finance

strategic management

university of glasgow

computing science 1st year programming institute of biomedical

& life sciences computing science accelerator course

• Department of Hospitality & Tourism Management• Students: 200• Technology: Podcasts, electronic voting system • Assessment Activities: Regular podcast releases

support changes to lectures to include interactive discussion using EVS.

• Efficiencies: 50% reduction in lectures (however, lectures delivered twice to smaller groups).Licensing and development costs associated with podcasts (c. £1k per 30 minutes podcast)

• Learning Gains: Significant gain in overall exam mark in 2006/07 diet (+12.2%) compared with 2005/06.

Significant reduction (-25%) in students receiving non-qualification.
















• Department of Mechanical Engineering• Students: 250• Technology: Commercial online homework

packages, electronic voting system (EVS)• Assessment Activities: Weekly tests provide on-

demand feedback on student performance for both students and tutors. Just-in-time teaching supported by interactive classes using EVS.

• Efficiencies: 60% reduction in assessment workload saving 102 staff hours. License cost of commercial packages £12.95 and £7 per student per annum.

• Learning Gains: Results from 2006/7 diet indicate strong class attainment maintained (90% pass rate, 65% average)

"I think it’s managed to save a lot of time for ourselves and the tutors and given them more time to develop what they are going to talk about and give more time for them to speak to people individually if they need it." Student comment
















• Department of Psychology• Students: 560• Technology: Online collaborative group tasks

supported by VLE message-board• Assessment Activities: Regular collaborative

tasks support peer feedback processes and student engagement.

• Efficiencies: 50% of lectures replaced with online tasks. Staff time re-directed to support online tasks.

• Learning Gains: Significant overall improvement in average exam pass mark (51.1% in 2005/06 diet rising to 57.4% in 2006/07).

Exam failure rate reduced from 13% to 5%. Course failure rate reduced from 12.1% to 2.8%.
















• Department of Modern Languages• Students: 200• Technology: Online homework and tests supported by VLE,

electronic voting system (EVS)• Assessment Activities: Diagnostic tests of student

knowledge at start of year to inform teaching.

Formative feedback from regular online testing and EVS classroom use.

• Efficiencies: Effective delivery of course made possible despite significant cuts in funding and staffing.

Tutorials reduced by 50% and replaced with online tasks. Listening classes reduced from 360 to 160 hours. Saving of 200 staff hours.

• Learning Gains: Failure rate in final exam reduced from 24% to 4.6% compared with 2005/06 diet.

"Having almost immediate feedback on marks was very useful as I was aware at every point as to how well I was coping" Student comment
















• School of Pharmacy - Pharmacy Practice 3• Students: 240• Technology: Online simulation tutorial • Assessment Activities: Regular online tasks to

improve student engagement, multiple opportunities for self-testing and regular feedback.

• Efficiencies: Savings in staff time due to reduced need for remedial revision work with individual students.

• Learning Gains: Significant gain in overall exam mark in 2006/07 (+16%) compared to 2005/06 diet.

“The tutorial was an excellent resource and learning tool to supplement our class”

Simulated exactly what it would be like to carry out a check on a prescription, allowing us to experience the difficulties involved and discover where we needed improvement” Student comments


What nextWhat next

• REAP will continue to be a source of advice on assessment for the HE and FE sectors

• the working models are already being applied in other institutions at local and strategic levels

• and the findings, archived at www.reap.ac.uk, are being disseminated worldwide


Questions & DiscussionQuestions & Discussion

Any questions?

www.reap.ac.uk


Relevant papersRelevant papers• Nicol, D (in press), Laying the foundation for lifelong learning: cases studies of technology

supported assessment processes in large first year classes, British Journal of Educational Technology (to be published July 2007).

• Nicol, D (2007) E-assessment by design: using multiple-choice tests to good effect, Journal of Further and Higher Education.

• Nicol, D. & Milligan, C. (2006), Rethinking technology-supported assessment in relation to the seven principles of good feedback practice. In C. Bryan and K. Clegg, Innovations in Assessment, Routledge.

• Nicol, D (2006), Increasing success in first year courses: assessment redesign, self-regulation and learning technologies, Paper prepared for ASCILITE conference, Sydney, Australia, Dec 3-6.

• Nicol, D, J. & Macfarlane-Dick (2006), Formative assessment and self-regulated learning: A model and seven principles of good feedback practice, Studies in Higher Education, 31(2), 199-218.

• Nicol, D & Boyle, J. (2003), Peer Instruction versus Class-wide discussion in large classes. Studies in Higher Education, 28(4), 457-473

• Boyle, J.T. and Nicol, D. J. (2003). Using classroom communication systems to support interaction and discussion in large class settings, Association for Learning Technology Journal, 11(3), 43-57

Increasing learner success with technology supported assessment:

Education

Transcript of Increasing learner success with technology supported assessment: