Feedback, efficiency and learning pay-off: new designs for formative

assessment

Professor David Nicol, Deputy-Director,

Centre for Academic Practice and Learning Enhancement (CAPLE)

Director, REAP project (www.reap.ac.uk)University of Strathclyde

Queens University Belfast, 15th February 2008

Plan

Describe backgroundIntroduce researchCase studies of practice from REAPDiscussionThinking about your practice Developments and GuidelinesSources of information and advice

Background

ProjectsStudent Enhanced Learning through Effective Feedback (SENLEF) project funded by HE AcademyLiterature review: model of formative assessment and feedback: 7 principles in relation to development of learner self-regulationThe Reengineering Assessment Practices (REAP) project funded by SFC £1m) www.reap.ac.uk QAA Scotland – first year experience and formative assessment

Re-engineering Assessment Practices project

3 HEIs (Strathclyde, Glasgow Caledonian Business School, Glasgow University)Large 1st year classes (160-900 students)A range of disciplines (19 modules ~6000 students)Many technologies: online tests, simulations, discussion boards, e-portfolios, e-voting, peer/feedback software, VLE, online-offlineGoals: learning quality and teaching efficienciesOutputs: case studies, advice for strategic changeAssessment for learner self-regulation

Why take assessment and feedback seriously?

A key driver of student learningMajor cost in higher education Widely reported that students don’t read feedback providedDrop-out and retention linked to academic experienceQAA reports – main area of criticism in England

• Nationally only 55% of students think feedback is prompt and had helped to clarify things they did not understand [Scotland: 48%]

• Nationally only 63% of students agree that have received detailed comments on their work [Scotland: 49%]

A key research paper

Black and Wiliam (1998) 250 studiesReal classroom situations – tutor, peer and self-assessment and feedbackPositive benefits for learning and achievement across all content areas/skills and sectors…. the lowest effect size would move a student from the average into the top third in a class test…

Big impact in schools but what about HE?

First Year: The academic experience

What is important?

Coping with transitionUnderstanding what is requiredEngagement with academic programmesSupport and feedback Experiences of successAgents in own learningBelief in self (ability) and motivationSocial dynamics of learning (belonging)

Based on research by Yorke (UK) and Tinto (US)

Background (1)

Gibbs, G. & Simpson, C (2004) Conditions under which assessment supports students learning, Learning and Teaching in Higher Education, 1, 3-31.

See: Formative Assessment in Science Teaching (FAST) project at: http://www.open.ac.uk/science/fdtl/

Gibbs and Simpson (2004)

Assessment tasks [Conditions 1-4]1. Capture enough study time (in and out of class)2. Are spread out evenly across timeline of study3. Lead to productive activity (deep vs surface)4. Communicate clear and high expectations

i.e concern here is with ‘steers’ about how much work to do

Background (1)

Literature ReviewNicol, D. & Macfarlane-Dick, D. (2006). Formative assessment and self-regulated learning: A model and seven principles of good feedback practice. Studies in Higher Education, 34 (1), 199-218Nicol, D & Milligan, C. (2006), Rethinking technology-supported assessment practices in relation to the seven principles of good feedback practice. In C. Bryan & K. Clegg, Innovative assessment in higher education, Routledge.

BackgroundStudent Enhanced Learning through Effective Feedback [SENLEF] project funded by HE AcademyREAP project: www.reap.ac.uk

Current thinking

Students are always engaged in self-assessment/self-regulation of their own learning (Winne, 2005: Black & Wiliam, 2005). Logically entailed by constructivist thinkingThe act of using teacher feedback implies that self-assessment must be presentFeedback in HE is being reduced so how are students still learningThe question is: how can we scaffold students’ learning so they become better at self-regulation (Lajoie, 2005)

Ideas in Practice

Consider self and peers as much as the teacher as sources of Assessment and Feedback

Tap into different qualities than teacher can provide

Saves time Provides considerable learning benefits (LLL)

Don’t focus just on written feedback but every step of the cycle:

Understanding the task criteria (Sadler, 1983) Applying what was learned in action

Scaffolding self regulation: 7 principles of

good feedback (assessment design)1. Clarify what good performance is (goals, criteria,

standards).2. Facilitate the development of reflection and self-

assessment in learning 3. Deliver high quality feedback to students: that enables

them to self-correct4. Encourage peer and student-teacher and peer dialogue

around learning5. Encourage positive motivational beliefs & self esteem

through assessment6. Provide opportunities to act on feedback7. Provide information to teachers that can be used to help

shape their teaching Source: Nicol and Macfarlane-Dick (2006)

Two super principles

Super-principle 1: time on task and effort (engagement) i.e. steers on how much work to do and when – Gibbs and Simpson 4 conditions

Super-principle 2: developing learner self-regulation (empowerment/self-regulation) i.e steers to encourage ownership of learning – the seven principles discussed above.

Case examples from REAP – applying these conditions/ principles

REAP: Example 1: Mechanical Engineering(personal response systems)

Problems identified

Conceptual misunderstandings even after graduation (e.g. concept of force)Passive learning in classroom due to larger numbersEvidence of low levels of student motivation –attendanceDifficult to develop a sense of community amongst learnersRetention issues (20%)

Course Redesign

Looked worldwide for the best solutionFocus on teaching core conceptsCarefully constructed student workgroups Introduced personal response systems in lecture sessions (to facilitate peer discussion) as promoted by Mazur at HarvardLater linked this to online testing And to online homework system

Personal Response System (PRS)

PRS was developed by Professor Nelson Cue at Hong Kong University of Technology and has now been adopted by hundreds of educators worldwide

Benefits

PRS promotes reflection on and peer discussion of reasoning behind difficult conceptsSelf, peer and tutor feedback in a single classroom sessionImproved understanding in standardised engineering tests (e.g. force concept inventory)Culture of collaborative learning establishedStudents report enhanced satisfaction (fun!) compared to traditional lecture classes.High levels of ‘time on task’ in classDropout reduced from 20% to 3%

Some resultsSurvey Statement Students

agree Students disagree

Using the PRS helps me to develop a better understanding of the subject matter compared to traditional lectures

74% 4%

Using the PRS helps me to understand the concepts behind the problems

75% 6%

I am more actively involved during PRS classes than in traditional lectures

95% 1%

I have to think more in PRS classes than in traditional lecture classes

91% 0%

I study less outside of PRS classes than for traditional classes

24% 24%

I remember less after a PRS class than after other classes

12% 63%

The student experience

Some student comments:“…in this class everybody’s involved, you have to

think about what’s being said…”“…you are learning from people around you… it’s a

language you can understand…”“…you feel you are keeping pace with the class and

that everyone is learning together…”

Relation to Gibbs & Simpson’s four assessment conditions

1. Web-based assessment tasks (MCQs and problem solving exercises) keep students engaged in out-of-class activities and EVS encourages engagement in class (condition 1)

2. Activities are distributed across topics and weeks (condition 2)

3. EVS tasks are designed to deepen learning as conceptual understanding increases (condition 3)

4. EVS activities clearly communicate requirements and there is a progressive increase in challenge (communicates clear and high expectations, condition 4)

Relationship to seven principles

1. Learning goals clarified through iterative cycles of tutor presentation, tests and retests using MCQs in class (Principle 1)

2. Reflection/self-assessment triggered through bar-chart presentation and by online tests (Principle 2)

3. Teachers provide feedback at end of EVS concept-test sequence (Principle 3)

4. Both teacher-student and peer dialogue occur in EVS interactive sessions (Principle 4)

5. The focus on learning goals rather than performance goals in class and the staged difficulty of concept tests encourage motivational (Principle 5)

6. The continuous cycle of tests, retests and feedback ensures that students can use the feedback immediately (Principle 6)

7. Online MCQ tests and student performance in EVS lectures provide a range of feedback information that tutors can use adjust teaching to student needs (Principle 7)

Other uses of PRS

Diagnostic testing at beginning of yearCommunity building – induction activities and sharing data about student cohort Revision of lecture materials in classFormal examinations using multiple choice questionsEvaluation data – reactions to courseExperiments where human responses being tested (voting in politics, visual illusions, questionnaires)Been used across a range of disciplines worldwide

See resources section of www.reap.ac.uk website and D. Banks ‘Audience Response Systems in Higher Education: Applications and Cases’, Information  Science Publishing, Hershey 2006

Psychology

560 first year studentsMixture of psychology majors (130) and those taking psychology only for one year (430)6 topic areas, 48 lectures, 4 tutorials, 12 practicalsAssessment; 2 x MCQs (25%), tutorial attendance (4%), taking part in experiment (5%), essay exam (66%)

Problems identified

No practice in writing skills but required in the examMore detail provided in lectures than mentioned in exams (not enough independent reading)No feedback except on MCQs (percent correct)Didn’t want to increase staff workloadWanted to improve overall exam marksAnd standard of entrant to second year

Discussion point

What would you do to improve the student experience in first year psychology?

You can use any technology (or combination of technologies) but you must consider costs and staff time constraints…

Psychology Redesign

Discussion board in WebCTStudents in 85 discussion groups of 7-8, same groups throughout yearSeries of online tasksLecture on Monday, Friday lecture abolished Students to discover for themselves in collaboration with their group what would have been the topic of the Friday lecture

Structure of group tasks

6 cycles of 3 weeks (one cycle x major course topic)

• First week: ‘light’ written task (e.g. define terms) = 7 short answers (all answer)

• Second week = guided reading • Week three: ‘heavy’ written task: students

answer guided questions and then collaborate in writing a 700-800 word essay.

Within each week:• The Monday lecture – introducing material• Immediately after lecture, task posted online –

for delivery the following Monday• Model answers (selected from students) posted

for previous week’s task

The teaching role

Participation in the discussions was compulsory but not marked (this year there is 2% mark for participation)The course leader provided general feedback to the whole class – often motivational He encouraged students to give each other feedbackAnd he selected the model answersThe group discussions were not moderatedAround 8 teaching assistants monitored the discussions and reported non-participation to the teacher

 

Online Project 1 – Classical Conditioning Phenomena. 

 Each Group Member should read the Passer chapter from the beginning to at least as far the section which begins ‘Applications of Classical Conditioning’. Satisfy yourself that you can answer EACH of the questions below. Then agree as a group who will answer what.

 Project 1 is to answer these questions as fully as you can: 

1) What type of response is susceptible to Classical Conditioning? 2) Why does Extinction occur? 3) What is Spontaneous Recovery? 4) What does the phenomenon of Spontaneous Recovery tell us about the nature of Extinction in Classical Conditioning? 5) What is Generalisation? 6) What is Discrimination? 7) What is Higher Order Conditioning? 

Project 9: An example of ‘heavy’ task

The Task – 800 word essay:

Assess the strengths and weaknesses of Freud’s and Eysenck’s theories of personality. Are the theories incompatible?readings suggestedquestions provided – all should tryand advice on how to divide task given

BenefitsWritten responses of an exceedingly high standard (sometimes surpassing 3rd year)Spontaneous online discussions about learning and leaner responsibilityHigh levels of motivation, atmosphere in class improvedSome students burdened by workload – easily detectedSome requested to move groups (5 groups)Online interactions showed powerful ‘scaffolding’Interaction and feedback possible with 560 studentsEasy for tutors to monitor participationPeer feedback and self feedback (model answers) harnessedImproved mean exam performance (up from 51-57%)

Has it worked?

I read more about Psychology and read it earlier in each semester than I would have done without

the online projects

0

10

20

30

40

50

60

1 2 3 4 5

agreed-disagreed

Nu

mb

ers

I learned more in Psychology because of online projects than I did in my other subjects

0

10

20

30

40

50

60

1 2 3 4 5

agreed-disagreed

nu

mb

ers

Series1

I found that reading other people's contributions helped me understand Psychology

010203040

50607080

1 2 3 4 5

agreed-disagreed

nu

mb

ers

Series1

The feedback based on other students' work helped me understand how to improve my own

answers

0

10

20

30

40

50

60

1 2 3 4 5

agreed-disagreed

nu

mb

ers

Series1

Online postings/interaction

24,362 messages posted by groups Average number of postings per student 44.3Most messages posted by single student 3241067 postings to general open discussion forumAccumulated time students spent in psychology site was 1 year 37days 11 hours 56 minsStudents set up online study groups for other subjects

Structured tasks online triggered important social-cognitive processes

Relation to the Gibbs & Simpson’s four assessment conditions

1. Tasks require significant study out of class (capture sufficient study time, condition 1)

2. They are distributed across topics and weeks (are spread out evenly, condition 2)

3. They move students progressively to deeper levels of understanding (productive/ deep learning, condition 3)

4. There are explicit goals and progressive increase in challenge (communicates clear and high expectations, condition 4)

Relation to 7 feedback principles

• Standard format and model answers provide progressive clarification of expectations (clear goals, principle 1)

• Students encouraged to self-assess against model answer (self-assessment, principle 2)

• Course leader provides motivational and meta-level feedback and selects model answers (teacher feedback, principle 3)

• Online peer discussion aimed at reaching consensus about response (dialogue, principle 4)

• Staged complexity, focus on learning not just marks, use of students model answers (motivation, principle 5)

• Repeated cycle of topics and tasks (closing gap, principle 6)

• VLE captures all interactions allowing course leader to monitor progress and adapt teaching (shaping teaching, principle 7)

What can we learn from these case studies?

Use of simple technologies (PRS, quiz/survey tool, discussion board)Considerable thought gone into the learning design [which is transferable]The drivers were learning improvements rather than technology (context of use)Key finding across studies was need to balance structure and learner controlAlso important is the way that the social and the academic processes are shown to be mutually supportive

Discussion point

Consider your own classes/modules/programmes: Identify one or two ideas that you might use from today’s presentation in the redesign of your teaching.Identify any questions raised in your discussion groups

Plenary report back:Be prepared to share an idea with others in the plenary and/orA ‘question worth asking’ that might illuminate some important issue

QAA Scotland Project: Assessment and the First Year

ExperienceUse assessment and feedback as lever for transformation of the student experience

Develop in students the ability to monitor, evaluate and regulate their own learning (engagement-empowerment)

Foster learning groups and communities in the first year and beyond (academic-social integration)

The Concepts

Balancing engagement (time & effort on task) and empowerment (opportunities to take control and manage own learning)

Academic and social integration into the university: use the academic to support the social and vice versa

EMPOWERMENT/SELF-REGULATION

SOCIALEXPERIENCE

ENGAGEMENT

Figure 1: Assessment and feedback practices: dimensions of implementation

ACADEMICEXPERIENCE

Good formative assessment and feedback practices should:

1. Help clarify what good performance is (goals, criteria, standards)

2. Encourage ‘time an effort’ on challenging learning tasks3. Deliver high quality feedback information that helps learners

self-correct4. Provide opportunities to act on feedback 5. Ensure that summative assessment supports formative

learning processes6. Encourage interaction and dialogue around learning (peer,

teacher-student)7. Facilitate the development of self-assessment and reflection in

learning8. Give choice in the topic, method, criteria, weighting or timing of

assessments.9. Involve students in decision-making about assessment policy

and practice10. Support the development of learning groups and communities11. Encourage positive motivational beliefs and self-esteem12. Provide information to teachers that can be used to help shape

their teaching

Guidelines for Implementation (1)

1. Implement the principles2. A single principle or many3. Active involvement of students4. Tight-loose – maintain fidelity to the

principles (tight) but encourage disciplines develop their own techniques of implementation (loose)

EMPOWERMENT/SELF-REGULATION

SOCIALEXPERIENCE

ENGAGEMENT

Figure 2: Assessment and feedback practices: dimensions of implementation

ACADEMICEXPERIENCE

• Students create criteria• Students add own criteria • Students identify criteria

from samples of work• Exemplars of different

performance levels provided

• Students rephrase criteria in own words

• Provide document with criteria

EMPOWERMENT/SELF-REGULATION

SOCIALEXPERIENCE

ENGAGEMENT

ACADEMICEXPERIENCE

6.Encourage interaction and dialogue around learning (peer and teacher-student)

2. Encourage time & effort on challenging learning tasks

+

EMPOWERMENT/SELF-REGULATION

SOCIALEXPERIENCE

ENGAGEMENT

ACADEMICEXPERIENCE

Students self-assess own performance using MCQs

Students self-assess using MCQ and provide confidence ratings

Students create MCQs and feedback for wrong and right answers 6.Encourage

interaction and dialogue around learning (peer and teacher-student)

+

Guidelines for Implementation (2)

5. Determine where ICT can add value6. Clarify students’ responsibilities 7. Align responses to National Student Survey

to the assessment principles8. Alternate solo and group work9. Evaluate change (PIs)10. Consider programme coherence 11. Use principles as quality enhancement tool12. Share your learning and designs13. Quality enhancement

Transforming AssessmentA ‘blueprint’ for change in the first year

TABLE OF CONTENTS1. Recommendations: how to implement at module,

course and institutional level 2. A literature review: how assessment and feedback

can support empowerment and integration

3. 12 principles of assessment and feedback practice (+ questions to enhance quality)

4. 100+ examples and case studies illustrating implementation across the disciplines

(see also www.reap.ac.uk)

Some publicationsNicol, D (2008), Transforming assessment and feedback: Enhancing integration and empowerment in the first year, to be published by Quality Assurance Agency, Scotland (June)Nicol, D (in press), Assessment for learner self-regulation: Enhancing achievement in the first year using learning technologies, Assessment and Evaluation in Higher Education,Nicol, D (2007), Laying the foundation for lifelong learning: cases studies of technology supported assessment processes in large first year classes, British Journal of Educational Technology, 38(4), 668-678Nicol, D (2007) E-assessment by design: using multiple-choice tests to good effect, Journal of Further and Higher Education.31(1), 53-64.Nicol, D. & Milligan, C. (2006), Rethinking technology-supported assessment in relation to the seven principles of good feedback practice. In C. Bryan and K. Clegg, Innovations in Assessment, Routledge.Nicol, D, J. & Macfarlane-Dick (2006), Formative assessment and self-regulated learning: A model and seven principles of good feedback practice, Studies in Higher Education, 31(2), 199-218.Boyle, J.T. and Nicol, D.J. (2003) Using classroom communication systems to support interaction and discussion in large class settings, Association for Learning Technology Journal, 11(3), 43-57Nicol, D.J. and Boyle, J.T. (2003), Peer interaction and class-wide discussion: a comparison of two interaction methods in the wired classroom, Studies in Higher Education, 28(4), 457-73See also www.reap.ac.uk