e-delivery of Formative Assessment and Feedback: How well does it work?

Michael J Cox, Bradford School of Optometry & Vision Science

An interactive poster presentation for: “Bradford’s Got TALENT”

The 9th Annual Learning, Teaching and Assessment Conference, June 2010

Click to begin

e-delivery of Formative Assessment and Feedback: How well does it work?

Michael J Cox, Bradford School of Optometry & Vision Choose an Option

The Aims The ActionsQMP – Pros

and Cons Formatives in Action:Choose a Demonstration

Evaluating Effectiveness:Facts and Figures

Evaluating Effectiveness :User CommentsConclusions

e-delivery of Formative Assessment and Feedback: What are we trying to do?

The Problem• A large (110+) cohort of

students• A level 3, 20 credit module

covering a broad and challenging area: Ocular & Systemic Disease

• Formative assessments existed but providing individual feedback to these was too resource hungry to be practical

The Aims• Develop and extend the

breadth and depth of formative assessment available to each student

• Provide individual feedback to students

• Enhance the effectiveness of the formative assessment in assisting the learning of each student

The Module and the New Formative Assessment Structure

Assessment Date Delivery Feedback

Slides in class QMP through Blackboard

1 Sem 1, Wk 5 70% 30% (in class) Class session

2 Sem 1, Wk 9 0% 100% Class session and QMP

3 Xmas Vacation 0% 100% QMP only

4 Sem 2, Wk 5 0% 100% Class session and QMP

5 Sem 2, Wk 9 0% 100% QMP only

•A 20 credit module, taught over 24 weeks and divided into 4-week blocks for the purposes of formative assessment•An assessment covers material from the latest block and revises some earlier material•An assessment takes 30 minutes, although students can take longer, if they wish as there is no limit to the number of attempts•The attainment on the assessments is entirely for the use of the student – it does not influence summative assessment marking

What can QMP do?

• Present questions to students• Provide a rich variety of question types• Record their answers• Mark many of their answers and archive results• Provide feedback, dependent upon the student’s answer, to

explain why a right answer is right and a wrong answer, wrong• Provide model answers, with a marking scheme, for questions

with paragraph length answers so that students can self-mark• Provide a grading scheme and instructions for students so

that they can determine their total mark and find out where this places them academically

Benefits when using QMP for Formative Assessment

• Good integration with Blackboard• Flexible (well – bendable) in operation and

availability• QMP can mark single word or multiple word

answers for you• QMP will deliver feedback for you. It is immediate

and can be answer (not just right/wrong) driven• Simple presentation of images/video in questions

and feedback

Benefits when using QMP for Formative Assessment

• “Sequencing” questions and “Hotspot” questions can be particularly powerful and automatically marked

• Archiving of students’ answers allows the quality control loop to be closed

• It makes you think hard about your assessments

• Watch the demo to find out more

Challenges in using QMP for Formative Assessment

• Not everyone is comfortable using a computer• Developing your design skills takes time• It can be an awkward tool to use• Development time can be extensive• You’ll want to make things perfect, so spend more

time• UoB server version doesn’t yet work with Win 7• It makes you think hard about your assessments• Watch the demo to find out more

Find out more:Choose a Demonstration Video

Full resolution versions available from the author on request. Best viewed in Full Screen

Click me to return to the main navigation screen

Click me to watch the YouTube video

demonstration about Feedback (5 Minutes)

Click me to watch the YouTube video

demonstration about Accessing the Assessment

(2 Minutes)

Click me to watch the YouTube video

demonstration about Using different Question Types

(7 Minutes)

Initial Evaluation

• All planned delivery took place• Uptake of the VLE delivered material was 80%

+ (following the assessments with friends could mean this is even higher)

• Summative assessments, also delivered through QMP, showed a small, but definite, improvement in the average student performance

Course Unit EvaluationWritten Comments

• In “major strengths”, 25 out of the 33 individuals who commented on major strengths identified the formative assessment

• In “how to improve the unit” three comments were made about formative assessment– More time to go through the

formatives together in class– More formative assessments– Formatives not to appear online, but

in classes, to force students to revise for them

• 53% return rate• 4.15 score (1->5 point scale, 1 bad,

5 good) for “Assessment of my work was helpful in my learning”

• Previous scoring for this area was 3.06 (2008) and 3.41 (2005), despite equally well received “overall satisfaction” at ~4.3

• “Learning objectives were clear to me” has also increased over this time, but less sharply

Although not directed towards specifically evaluating this work, the CUEQs gave valuable information on the success of the work

User Survey Results

• Mean satisfaction scores from 23 students that completed the survey. All except two had completed all four QMP formative assessments. Two had completed three of the four assessments

• Error bars show 95% confidence interval• 0 – Strongly Disagree, 4- Strongly Agree

1 2 3 4 5 60.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

Survey Questions1. Formative Assessments in OSD were

useful in my learning2. QMP was easy to use3. The variety of question styles used was

helpful4. Feedback from the questions was

helpful5. Image quality in QMP was better than

in paper based examinations6. I wish to see QMP continue to be used

in formative assessment in the future

All students registered on the module were invited to complete an online survey about the use of QMP for formative assessment

The Best

The Worst

User Survey Comments (Likes)Instant feedback and explanations for the correct answers were very

helpful. It was useful to have access to all the assessments so

you can re-do the assessments at a later date and to recap all the

information covered in the module.

The variety of subjects was excellent. The questions were superb in the main, although one or two of the questions

seemed a bit irrelevent to my actual understanding of OSD.

The varied content of MCQ and short answer and photo recognition work very well.

Mimicked exams taken in Jan very well. Good feedback points.

User Survey Comments (Likes)

Formative assessments were extremely helpful in preparing for the osd exam. Was useful to get

used to the layout and the feedback helped in letting us know how much detail to go into with the

answers.

An absolute brilliant way of revision, and it really helped

me in preparation for the exams in way of the format of

the exam, it also made the exam enjoyable if that can at all be possible.It is a pity that at the moment not all exams are possible in this format.

immediate feedback aided revision and the chance to repeat tests also

allowed me to go back and test myself again on certain areas where i

struggled.

User Survey Comments (Likes)

the feed back! the variety of questions, extremly useful in

testing my knowledge/ revision, and refelctes how the summative

assesment will be

Very close to the exam format that we had in January and therefore prepares

you well for what to expect. Liked variety of question styles rather than

purely text based. Demonstrating knowledge through the identification of

signs on the photos themselves ( remember a glaucoma one

particularly) by placing marker over sign seemed useful and relevant!

please make available every topic like this as class work

conjunction with on line lecture. for every subj if

possible

User Survey Comments (Dislikes)

More feedback with wrong answers.

Increase the number of available questions.

I did not feel that getting together and going over the assessments was always a

good use of my time. I think gaining the feedback via my PC

good enoughNot being the best speller (and from an e-mail in the

first semester i think i know your views on this one)some

lee way on spelling or perhaps auto correction

would help so long as it is not miles out. ps have used

spell check on this.

User Survey Comments (Dislikes)

Allow feedback on unanswered questions! Otherwise, they were so

useful the only way to improve them would be to offer more!!!

The questions which involved scrolling up and down were

not the easiest to follow, probably due to lack

concentration more than anything.

The automatic marking seemed a bit fussy, in that; if I had the correct

answer but another word also it was marked as wrong. In every

other way I think that the formatives were extremely useful.

the quality of the images!, I dont like how our exams are done on computers, as takes alot longer to type and think

using a p.c esp for people who do no not use a p.c for revision

but a book instead however overall EXCELLENT WORK ! love

it

Conclusions

• QMP was used with Blackboard to deliver re-designed and extended formative assessments for the module Ocular & Systemic Disease

• Assessments were successfully run and accessed throughout the academic year

• Question design encompasses a broad range of different tasks

• Automatic marking, self-marking and directed individual feedback were all used

• The student response to the new delivery was very positive