Assessment for learning: the benefits of generating feedback

David Nicol Professor of Higher Education

Centre for Academic Practice and Learning Enhancement (CAPLE]

Director, Peer project (www.reap.ac.uk)University of Strathclyde, Scotland

University of Cadiz, March 9-11, 2011

Plan

BackgroundRe-engineering Assessment Practices (REAP) project www.reap.ac.uk/reap/index.html Concepts and example of practiceInstitutional considerationsPEER project www.reap.ac.uk/peer.aspx Discussion

Background

Departments and faculties: educational improvement projects, including REAP projectPolicy/strategy: development of educational policy and principles of assessment & feedback (based on REAP)Students: ‘Feedback as dialogue’ campaignQuality procedures:– redesigning course documentation and review proceduresPEER project – developing students ability to evaluate the quality of each other’s work

See www.reap.ac.uk

REAP : Re-engineering Assessment Practices

Scottish Funding Council for Universities (£1m)3 Universities - Strathclyde, Glasgow & Glasgow CaledonianLarge 1st year classes (160-600 students)A range of disciplines (19 modules ~6000 students)Many technologies: online tests, simulations, discussion boards, e-portfolios, e-voting, peer/feedback software, VLE, online-offlineLearning quality and teaching efficienciesAssessment for learner self-regulationwww.reap.ac.uk

Background (1)

Gibbs, G. & Simpson, C (2004) Conditions under which assessment supports students’ learning, Learning and Teaching in Higher Education, 1, 3-31.

See:

Formative Assessment in Science Teaching (FAST) project at: http://www.open.ac.uk/science/fdtl

Gibbs and Simpson (2004)

Assessment tasks [Conditions 1-4]1. Capture sufficient study time (in and out of

class)2. Are spread out evenly across timeline of study3. Lead to productive activity (deep vs surface)4. Communicate clear and high expectations

i.e concern here is with ‘time on task’ how much work students do - their active engagement in study

Background (2)Literature Review

Nicol, D. & Macfarlane-Dick, D. (2006). Formative assessment and self-regulated learning: A model and seven principles of good feedback practice. Studies in Higher Education, 34 (1), 199-218

BackgroundStudent Enhanced Learning through Effective Feedback [SENLEF] project funded by HE AcademyREAP project: www.reap.ac.uk

Rethinking assessment and feedback1. Consider self and peers as much as the teacher as sources of assessment and feedback

Tap into different qualities than teacher can provide Saves time Provides considerable learning benefits (lifelong

learning)

2. Focus on every step of the cycle: Understanding the task criteriaApplying what was learned in action

3. Not just written feedback: Also verbal, computer, vicarious, formal and informal

Seven principles of good feedback

Good feedback:1. Clarifies what good performance is (goals, criteria,

standards).2. Facilitates the development of reflection and self-

assessment in learning 3. Delivers high quality information to students: that

enables them to self-correct4. Encourages student-teacher and peer dialogue

around learning5. Encourages positive motivational beliefs & self

esteem6. Provides opportunities to act on feedback7. Provides information to teachers that can be used

to help shape their teaching (making learning visible)

Source: Nicol and Macfarlane-Dick (2006)

EMPOWERMENT/SELF-REGULATION

ENGAGEMENT

Principle 1: Clarify what good performance is (the context of dialogue)

• Students create criteria• Students add own criteria • Students identify criteria

from samples of work• Exemplars of different

performance levels provided

• Students rephrase criteria in own words

• Provide document with criteria

EMPOWERMENT/SELF-REGULATION

ENGAGEMENT

Principle 2: Facilitate reflection and self-assessment in learning

• Students create MCQs including feedback for right and wrong (distractors) answers

• Students self-assess using MCQs and confidence ratings

• Students self-assess own performance using online MCQs

Two meta principles

1. Meta-PRINCIPLE 1: time and effort on task (structured engagement) i.e. steers on how much work to do and when – Gibbs and Simpson 4 conditions

2. Meta-PRINCIPLE 2: developing learner self-regulation (empowerment/self-regulation) i.e steers to encourage ownership of learning – the seven principles discussed above.

Key task for teacher is to balance 1 and 2

Example:Psychology

Psychology

560 first year students6 topic areas (e.g. personality, classical conditioning), 48 lectures, 4 tutorials, 12 practicalsAssessment; 2 x MCQs (25%), tutorial attendance (4%), taking part in experiment (5%), essay exam (66%)

Problems identified

No practice in writing skills but required in the examMore detail provided in lectures than mentioned in exams (not enough independent reading)No feedback except on Multiple Choice Questions (percent correct)Didn’t want to increase staff workload

Psychology Redesign

Discussion board in Learning Management SystemStudents in 85 discussion groups of 7-8, same groups throughout yearAlso open discussion board for classFriday lectures cancelled – discover material themselvesSeries of online tasks

Structure of group tasks

6 cycles of 3 weeks (one cycle x major course topic)

• First week: ‘light’ written task (e.g. define terms) = 7 short answers (all answer)

• Second week = guided reading • Week three: ‘deep’ written task: students

collaborate in writing a 700-800 word essay on the same topic.

Within each week:• The Monday lecture – introducing material• Immediately after lecture, task posted online –

for delivery the following Monday• Model answers (selected from students) posted

for previous week’s task

The teaching role

Participation in the discussions was compulsory but not marked (in subsequent years there was 2% mark for participation)Course leader provided general feedback to the whole class – often motivational He encouraged students to give each other feedbackThe group discussions were not moderated but monitored for participation

An example of ‘deep’ task

The Task – 800 word essay:

Assess the strengths and weaknesses of Freud’s and Eysenck’s theories of personality. Are the theories incompatible?readings suggestedquestions provided which all students should try

Relation to the Gibbs & Simpson’s four assessment conditions

1. Tasks require significant study out of class (condition 1)

2. Tasks are distributed across topics and weeks (condition 2)

3. They move students progressively to deeper levels of understanding (condition 3)

4. There are explicit goals and progressive increase in challenge (condition 4)

Relation to 7 feedback principles

1. Standard format and model answers provide progressive clarification of expectations (principle 1)

2. Students encouraged to self-assess against model answer (principle 2)

3. Course leader provides motivational and meta-level feedback and selects model answers (principle 3)

4. Online peer discussion aimed at reaching consensus is core feature of design about response (principle 4)

5. Focus on learning not just marks, sense of control/challenge enhanced motivation (principle5)

6. Repeated cycle of topics and tasks provide opportunities to act on feedback (principle 6)

7. VLE captures all interactions allowing course leader to monitor progress and adapt teaching (principle 7)

Benefits

Students worked exceptionally hard Written responses of exceedingly high standard Students took responsibility for learningHigh levels of motivation: atmosphere in class improvedOnline interactions showed powerful ‘scaffolding’ and community buildingFeedback with 560 students through peer and self-feedback (model answers)Easy for tutors to monitor participationImproved mean exam performance (up from 51-59%, p<0.01) weaker students benefit most

Has it worked?

I read more about Psychology and read it earlier in each semester than I would have done without

the online projects

0

10

20

30

40

50

60

1 2 3 4 5

agreed-disagreed

Nu

mb

ers

I found that reading other people's contributions helped me understand Psychology

010203040

50607080

1 2 3 4 5

agreed-disagreed

nu

mb

ers

Series1

I learned more in Psychology because of online projects than I did in my other subjects

0

10

20

30

40

50

60

1 2 3 4 5

agreed-disagreed

nu

mb

ers

Series1

Guidelines for Implementation

1. A single principle or many?2. Tight-loose – maintain fidelity to the principles

(tight) but encourage disciplines to develop their own techniques of implementation (loose)

3. Balance teacher feedback with peer and self-generated feedback

4. Focus on developing students’ own ability for critical evaluation

5. Create opportunities for ‘learning communities’ to emerge

6. The more actively engaged students are, the better the course design

Developments since REAP

1. Principles of Assessment and Feedback approved by University Senate and embedded in policy (2008)

2. Use of principles to inform curriculum renewal and Quality Assurance processes

3. ‘Feedback as Dialogue’ campaign to gain commitment of students

4. PEER Project (Peer Evaluation in Education Review)5. Work on the way we document our courses and

programmes

Peer Review in Education Evaluation [PEER]

The aims of the PEER project are to:Review evidence base for peer reviewDevelop educational designs for peer review (and self-review)Identify software support for peer reviewPilot implementations of peer review with large student numbersProduce guidelines for higher education – why do it, how to do it, pitfalls and solutions and software possibilities.

see http://www.reap.ac.uk/peer.aspx

PEER Project: Core Ideas

All graduate development in higher education requires that:

1. Students learn to evaluate critically the quality and impact of their own work (e.g. academic texts, problem solutions, designs)

2. Students learn to evaluate critically the work of others (e.g. peer review and peer feedback)

Ref:Nicol, D (2010) The foundation for graduate attributes: developing self-regulation through self and peer assessment. www.reap.ac.uk/peer.aspx

The research on peer feedback

Mainly about peers marking each others workWhere peer feedback is the focus it is usually about peers augmenting teacher feedback by increasing the quantity and variety of feedback

The focus of PEER project

Scenarios where students make evaluative judgements about the work of peers and provide a feedback commentary, usually writtenNot talking about .....collaborative tasks where students give each other informal feedback.....scenarios where the focus is on students evaluating each other’s performance in group working.....scenarios where the focus is on students grading/marking each other’s work although some rating of performance might be part of the peer design

Benefits of feedback construction (1)

Constructivist rather than transmission perspective

Producing is cognitively more demanding than reading feedback from others: students cannot be passiveStudents actively exercise assessment criteria (process/reprocess) from many different perspectivesSee how others tackle assignments and learn that quality can be produced in different waysDevelop writing skills through commentariesDeepens critical thinking in the discipline

Benefits of feedback construction (2)

Learn to assess own work, to monitor and evaluate their own productions - same skills involvedDevelops self-regulation – for life beyond university.Changes the power relationship in class (shares responsibility across students).Can develop learning communities

Giving and receiving feedback

Giving and receiving feedback within same domain enriches the whole process.Students are able to compare the feedback they receive with the feedback they produceAnd learn how different reviewers perceive their work – a single source of feedback is never satisfactoryCan develop a sense of shared responsibility across students for learning

Example: peer feedback

• Students write essay on one topic from three • Each student provides feedback on three essays in another topic

anonymously using rubric • The rubric: write a short summary of the essay, comment on and

rate (four point scale) the structure, arguments, evidence, writing, suggest ways of improving the essay.

• Students receive peer reviews of their own essays• Then review, comment on and rate their own essay using same

rubric.• Graded: for participating in the task, for their own essay and for

their review of it.• Finally students rate 3 reviews (on others’ work) and comment on

how useful they think they would be to author.• Software used to support administration

Six learning opportunities in peer review

1. Producing the target assignment2. Reviewing and producing peer feedback3. Receiving feedback from peers4. Responding to peer reviews (on own reviews of

those given to others)5. Self-reviewing one’s own work (e.g. after peer

review)6. Receiving feedback from the teacher – e.g. on

the assignment, peer review or self-review

1-3 comprise the essentials of peer review while 4-6 are further options.

Principles of effective peer reviewPeer review should

1. Engage students in active use of criteria and standards 2. Involve students in constructing commentaries in relation

to peer judgements, not just marks3. Lead to practice in both analytic (componential) and

holistic (configurational) judgements about quality4. Facilitate dialogue around the object and quality of the

review5. Ensure an atmosphere of mutual trust and respect6. Integrate self-review activities within peer review designs7. Ensure that making peer judgements is a regular activity

and not a one-off event

www.reap.ac.uk/PEER.aspx

Some of my PublicationsNicol, D (2010) The foundation for graduate attributes: developing self-regulation thorugh self and peer assessment, QAA Enhancement Themes, ScotlandNicol, D (2010) From monologue to dialogue: Improving written feedback in mass higher education. Assessment and Evaluation in Higher Education. 35(5), 501 -517 Nicol, D and Draper, S (2010) A blueprint for transformational organisational change in HE: REAP as a case study, Published by the Higher Education Academy, UK (see reap.ac.uk website)Nicol, D (2009) Transforming assessment and feedback: Enhancing integration and empowerment in the first year, Published by Quality Assurance Agency, Scotland (http://www.enhancementthemes.ac.uk/documents/firstyear/FirstYear_TransformingAssess.pdfNicol, D (2009), Assessment for learner self-regulation: Enhancing achievement in the first year using learning technologies, Assessment and Evaluation in Higher Education, 34(3), 335-352Nicol, D (2007), Laying the foundation for lifelong learning: cases studies of technology supported assessment processes in large first year classes, British Journal of Educational Technology, 38(4), 668-678Nicol, D (2007) E-assessment by design: using multiple-choice tests to good effect, Journal of Further and Higher Education.31(1), 53-64.Nicol, D. & Milligan, C. (2006), Rethinking technology-supported assessment in relation to the seven principles of good feedback practice. In C. Bryan and K. Clegg, Innovations in Assessment, Routledge.Nicol, D, J. & Macfarlane-Dick (2006), Formative assessment and self-regulated learning: A model and seven principles of good feedback practice, Studies in Higher Education, 31(2), 199-218.See www.reap.ac.uk/resources.aspx for copies.