University of YorkMay 2008

Using assessment to support student learning

Graham Gibbs

Personal Background

Open University 1975-80 and 1997-2003– Top in National Student Survey, especially assessment

and feedback ratings

Oxford Brookes University 1980-1997– Most ‘coursework’ assessment– Systematic course design

University of Oxford 2004-– Least ‘coursework’ assessment– Top in National Student Survey, especially assessment

and feedback ratings– ‘No course design’

Personal Background

Practical books and articles about assessment ‘53 Interesting ways to assess your students’

‘Assessing Student Centred Courses’

‘Assessing Large Classes’

Consultancy to universities on strategic decisions about assessment policy

Research into the impact of assessment on student learning

 

Student experience of assessment

“I just don’t bother doing the homework now. I approach the courses so I can get an ‘A’ in the easiest manner, and its amazing how little work you have to do if you really don’t like the course.”

“I am positive there is an examination game. You don’t learn certain facts, for instance, you don’t take the whole course, you go and look at the examination papers and you say ‘looks as though there have been four questions on a certain theme this year, last year the professor said that the examination would be much the same as before’, so you excise a good bit of the course immediately…”

“The feedback on my assignments comes back so slowly that we are already on the topic after next and I’ve already submitted the next assignment. I just look at the mark and throw it in the bin”

“The tutor likes to see the right answer circled in red at the bottom of the problem sheet. He likes to think you’ve got it right first time. You don’t include any workings or corrections – you make it look perfect. The trouble is when you go back to it later you can’t work out how you did it and you make the same mistakes all over again”

“One course I tried to understand the material and failed the exam. When I took the resit I just concentrated on passing and got 98%. My tutor couldn’t understand how I failed the first time. I still don’t understand the subject so it defeated the object, in a way”

“I do not like the on-line assessment method…it was too easy to only study to answer the questions and still get a good mark … the wrong reasoning can still result in the right answer so the student can be misled into thinking she understands something … I think there should have been a tutor-marked assessment part way through the course so someone could comment on methods of working, layout etc.”

“We were told this course was going to be an opportunity to be creative, to take risks. Then in week five we were hit with a multiple choice question test and we realised what it was really all about.”

Summative assessment that is redundant

• Most students can, in their first year, predict their final results with some accuracy

• As few as 5% of assessments are necessary to produce the same overall grades

Assessment that improves learning

• The case of the Engineer• The case of the Manager• The case of the Pharmacist• The case of the Psychologist• The case of the Accountant

The case of the engineer• Weekly lectures, problem sheets and classes• Marking impossible• Problem classes large enough to hide in• Students didn’t tackle the problems• Exam marks: 45%

The case of the engineer

• Course requirement to complete 50 problems• Peer assessed in six ‘lecture’ slots• Marks do not count• Lectures, problems, classes, exams unchanged

The case of the engineer

• Course requirement to complete 50 problems• Peer assessed in six ‘lecture’ slots• Marks do not count• Lectures, problems, classes, exams unchanged

• Exam marks increased from 45% to 85%

Why did it work?

The case of the engineer• time on task• social learning and peer pressure• timely and influential feedback• learning by assessing

– error spotting – developing judgement (internalisation of standards)– self-supervision (meta-cognitive awareness)

Assessment that improves learning

• The case of the Engineer• The case of the Manager• The case of the Pharmacist• The case of the Psychologist• The case of the Accountant

“Conditions under which assessment supports student learning”

Quantity and distribution of student effort

1 Assessed tasks capture sufficient student time and effort

2 These tasks distribute student effort evenly across topics and weeks

Quality and level of student effort

3 These tasks engage students in productive learning activity

4 Assessment communicates clear and high expectations to students

Quantity and timing of feedback

5 Sufficient feedback is provided, both often enough and in enough detail

6 The feedback is provided quickly enough to be useful to students

Quality of feedback

7 Feedback focuses on learning rather than on marks or students themselves

8 Feedback is understandable to students, given their sophistication

Student response to feedback

9 Feedback is received by students and attended to, and is acted upon by students to improve their work or their learning

Effective assessment tactics• Bioscience: poster reports• Engineering: sampling lab reports + cheap feedback• Law: essay requirements + sampling + models• Estates management: project exams• French Literature: critiquing texts under examination conditions

Assessment Experience Questionnaire

Measures extent to which the ‘conditions’ are perceived to be met– Quantity and distribution of effort– Quality, quantity and timeliness of feedback– Use of feedback– Impact of exams on quality of learning– Deep approach– Surface approach– Clarity of goals and standards– Appropriateness of assessment

University A University B

Time demands and distribution of student effort

% agree or strongly agree

I only study things that are going to be covered in the assignments

8%

27%

% disagree or strongly disagree On this course it is possible to do quite well without studying much

64%

33%

University A University B

Feedback % agree On this course I get plenty of feedback on how I am doing

68% 26%

The feedback comes back very quickly

65% 26%

Whatever feedback I get comes too late to be useful

11% 42%

‘University A’ is the Open University

• 8 assignments per course• Detailed written feedback on every assignment• Quality assurance of feedback• Less funding per student than any other university, mainly

spent on feedback• Best student ratings nationally• Much pedagogic research• Formative-only early assignments improve retention• Computer-based assignments reduce retention and

performance

…then I went to Oxford• Oxford responds in a limited way to most national quality assurance guidelines

– learning outcomes– assessment criteria– alignment of assessment with aims

• Oxford has not ‘modernised’ its assessment– reliance on examinations, little assessed coursework, little summative assessment of any kind, no modularisation

…then I went to Oxford• Oxford responds in a limited way to most national quality assurance guidelines

– learning outcomes– assessment criteria– alignment of assessment with aims

• Oxford has not ‘modernised’ its assessment– reliance on examinations, little assessed coursework, little assessment of any kind, no modularisation

• Outstanding quality of student experience at Oxford– student retention of 98% (1 st in UK)– Oxford ranked 1st for teaching in UK (Times, Guardian)– better CEQ scores than elsewhere in world– better NSS ratings than the Open University

Research questions

• What are the characteristics of programme level assessment environments that are associated with positive student learning responses?

• Are the characteristics of programme level assessment environments that are most closely associated with positive student learning responses those that quality assurance regulations emphasise?

Research design• Three contrasting universities (Oxford, pre-1992, post 1992)• Three contrasting programmes in each (Humanities, Science, Applied Social Science)• Characterise assessment environments

– Read documentation (all modules)– Interview programme leader, lecturers and students

• Administer AEQ• Explore relationships between characteristics of programme level assessment design and qualities of student learning

- with Harriet Dunbar-Goddet, Chris Rust and Sue Law- funded by the Higher Education Academy

Coding characteristics of programme level assessment environments• % marks from examinations• Volume of summative assessment• Volume of formative only assessment• Volume of (formal) oral feedback • Volume of written feedback• Timeliness: days after submission before feedback provided • Explicitness of criteria and standards • Alignment of goals and assessment

Coding characteristics of programme level assessment environments• % marks from examinations

– High: more than 70%– Med: between 40% and 70%– Low: less than 40%

Coding characteristics of programme level assessment environments• Explicitness of criteria and standards

– High: clear criteria for most assignments & exams; link to grades; effort made to enable students to internalise criteria & standards

– Low: explicit criteria and standards rare and/or nebulously formulated; marks/grades arrived at through global judgment in tacit way; no effort to enable students to internalise criteria and standards

Range of characteristics of programme level assessment environments

Range of characteristics of programme level assessment environments

• % marks from exams: 17% - 100%

Range of characteristics of programme level assessment environments• % marks from exams: 17% - 100%• number of times work marked: 11 - 95

Range of characteristics of programme level assessment environments• % marks from exams: 17% - 100%• number of times work marked: 11 - 95• number of times formative-only assessment: 2 - 134

Range of characteristics of programme level assessment environments

• % marks from exams: 17% - 100%• number of times work marked: 11 - 95• number of times formative-only assessment: 2 - 134• number of hours of oral feedback: 3 - 68

Institutional assessment environments 

Aspect of assessment Oxbridge Post-1992

Pre-1992

Volume/frequency of formative only assessment

High Low Med

% marks from exams High Low Med

% marks from coursework Low High Med

Alignment of learning activity with assessment

Low High Med

Explicitness of goals/outcomes and criteria

Low High Low

Patterns of assessment features within programmes

• every programme that is high on the volume of formative assessment is low on the volume of summative assessment • no examples of high volume of summative assessment and high volume of feedback

Patterns of assessment features within programmes

• every programme that is low on the volume of summative assessment is high on the volume of formative assessment• no examples of high volume of summative assessment and high volume of feedback

• there may be enough resources to mark student work many times, or to give feedback many times, but not enough resources to do both

Relationships between assessment characteristics and student learning

Assessment characteristics and student learning response: 1When the level of explicitness of criteria and standards is high, students’ experience is characterised by:

Less coverage of the syllabusLess and poorer quality feedbackLess use of feedbackLess learning from the examinationLess deep approach

Assessment characteristics and student learning response: 2When the level of alignment of goals and standards is high, students’ experience is characterised by:

Less coverage of the syllabusLess and poorer quality feedbackLess use of feedbackLess appropriate assessmentLess clear goals and standardsLess learning from the examinationLess deep approach

Assessment characteristics and student learning response: 3When the level variety of assessment methods is high, students’ experience is characterised by:

Less and poorer quality feedbackLess use of feedbackLess appropriate assessmentLess clear goals and standardsLess learning from the examinationLess deep approachMore surface approachLess overall satisfaction

Assessment characteristics and student learning response: 4When the volume of formative-only assessment is high, students’ experience is characterised by:

More coverage of the syllabusMore and better quality feedbackMore use of feedbackMore appropriate assessmentMore clear goals and standardsMore learning from the examinationMore deep approachMore overall satisfaction

Assessment characteristics and student learning response: 5When the volume of oral feedback is high, students’ experience is characterised by:

More coverage of the syllabusMore and better quality feedbackMore use of feedbackMore appropriate assessmentMore clear goals and standardsMore learning from the examinationMore deep approachMore overall satisfaction

Assessment characteristics and student learning response: 6When the timeliness of feedback is high, students’ experience is characterised by:

More effortMore coverage of the syllabusMore and better quality feedbackMore use of feedbackMore appropriate assessmentMore clear goals and standardsMore learning from the examination

Summary

• Explicitness of criteria and standards, alignment of goals and assessment and variety of assessment are all associated with a negative learning experience

…they are also associated with more summative and less formative-only assessment, less oral feedback and less prompt feedback

Summary• Explicitness of criteria and standards, alignment of goals and assessment and variety of assessment are all associated with a negative learning experience

…they are also associated with more summative and less formative-only assessment, less oral feedback and less prompt feedback

• Formative only assessment, oral feedback and prompt feedback are all associated with positive learning experience…even when they are also associated with lack of explicitness of criteria and standards, lack of alignment of goals and assessment and a narrow range of assessment.

Why?• being explicit does not result in students being clear about what they are supposed to be doing or what counts as high quality• ‘legitimate peripheral engagement in a community of practice’ (Lave and Wenger; Price et al)

Why?• Students experience very varied forms of assessment as confusing: ambiguity and anxiety are associated with a surface approach• Feedback improves learning most when there are no marks• Possible to turn feedback round quickly when there are no QA worries about marks

…alternative explanation A• The features of assessment environments identified here that appear to have negative consequences for student learning are also the features that are associated with modular courses in which each separate module has to have ‘self-contained’ assessment within a short time frame.

Conclusion

• It may be modularity, rather than QA regimes, that have caused some of the problems. Oxbridge is not modular (the Open University is…but has huge and long modules that are usually studied one at a time)

…alternative explanation B• High volumes of assessed ‘coursework’ have been introduced in part to increase student engagement• Student engagement improves learning outcomes• High % of marks from coursework is associated with higher marks and better degrees (at post ’92 universities)

However…• The effect of innovations that enhance engagement on learning outcomes only holds for low ability students (high ability students engage themselves) (Carini et al, in press)• The present study did not control for student ability

Conclusions

• Assessment has more impact on how students go about studying, on the quantity and quality of their effort, and on their performance – than does teaching

• It is relatively easy (and often cheap) to change student learning by changing assessment, provided the ‘conditions’ are met effectively

• Whole universities have implicit conventions about what is ‘acceptable’ in terms of assessment practice

• Some of these conventions are ill-informed and damaging … and are built in to QA systems

• Local contexts are likely to require different assessment strategies to engage their students.

ReferencesCarini, R.M., Kuh, G.D. & Klein, S.P. (in press) Student engagement and student learning: testing the outcomes. Research in Higher Education Dunbar-Goddet, H. & Gibbs, G. (under review) A methodology for evaluating the effects of programme assessment environments on student learning. European Association for Research into Learning and Instruction, Assessment Conference, Northumbria.Gibbs, G. (2002) Evaluation of the impact of formative assessment on student learning behaviour. European Association for Research into Learning and Instruction. Newcastle: Northumbria University. August 2002.Gibbs, G. & Simpson, C. (2003) Measuring the response of students to assessment: the Assessment Experience Questionnaire. 11th International Improving Student Learning Symposium, Hinckley.Gibbs, G. & Simpson, C. (2004) Conditions under which assessment supports student learning. Learning and Teaching in Higher Education 1, pp3-31.Gibbs, G., Simpson, C. & Macdonald, R. (2003) Improving student learning through changing assessment – a conceptual and practical framework. European Association for Research into Learning and Instruction Conference, Padova, Italy.