Experiences of audio assessment feedback: Staff and Student experiences Carol Ekinsmyth Department...
-
Upload
kaylee-cruz -
Category
Documents
-
view
215 -
download
1
Transcript of Experiences of audio assessment feedback: Staff and Student experiences Carol Ekinsmyth Department...
Experiences of audio assessment feedback: Staff and Student experiences
Carol EkinsmythDepartment of GeographyUniversity of Portsmouth
Why audio feedback might be important
Teaching and learning in higher education is often confounded by dominant cultures and quality audit frameworks that prevent the achievement of these desirable goals:
The sharing between ‘teachers’ and ‘learners’ of integrated knowledge structures (Kinchin, Lygo-Baker and Hay (2008)
The breaking down of “pedagogic resonance” (Trigwell and Shale (2004), Kinchin et al (2008)
Helping students arrive at and move beyond “liminal space” and develop understanding of “threshold concepts” (Meyer and Land, 2005)
Assessment and feedback
Researchers agree that the majority of student effort (learning?) is geared towards assessment (Carless et al, 2006; Brown, 2005; Gibbs, 2006)
Carless et al (2006) encourage us to engage in “learning orientated assessment”
Smith (2007, 11) argues: “If assessment is for learning, then feedback needs to focus on what the student could do differently and how to apply that learning to future work.”
Price (2007) and Baumard (1999) argue that the trend towards over-reliance on written feedback in HE crushes the ability to effectively encourage tacit knowledge formation and concentrates instead only on explicit knowledge
Assessment and feedback
“Good feedback comprises not just commentary about what has been done, but suggestions for what can be done next” (Brown, 2007)
And she argues;
“I believe that concentrating on giving students detailed and developmental formative feedback is the single most useful thing we can do for our students” (Brown, 2007)
Race (1999) argues “...feedback quality and quantity are probably the most important factors in enhancing student learning”
And yet, my experience suggests that Staff are reluctant to spend a lot of time on feedback ......
Why the reluctance over feedback?
Time pressures Uncertainty about what feedback is for:
Kinchin et al (2008) talk about more fundamental barriers – complicity between teachers and students in “non-learning outcomes”
“Correcting errors, providing guidance for future work, justifying the mark, satisfying quality
assurance processes, diagnosing difficulties, the list goes on.” – Price (2007)
How can audio feedback help?
It can’t - if teachers are unwilling to use it We perhaps need to concentrate on overcoming the
barriers to teacher-engagement Audio feedback seems especially suited to feed-
forward as a supplement to assessment proformas that often concentrate on feedback
Used in carefully identified places in the curriculum, it may be a powerful tool in breaking down pedagogic resonance
Why?
Because it enables the students to get closer to the thought processes (and thus community of practice) of the academic than written feedback does
Face-to-face contact might not be better as it can be intimidating and the details often cannot be recalled by students (my research findings)
The Project
GEES/LTSN (JISC ‘Sounds Good’ project)
Aims:
To evaluate the utility, efficiency, popularity and effectiveness (amongst students and staff) of audio feedback on assignments (range of types) for undergraduate students within the discipline of Geography.
Other reasons to bother
UK National Student Survey (2005-2008) shows that the area that students are least satisfied with is ‘assessment and feedback’ (Race and Pickford 2007)
The centrality of assessment to the student experience is widely acknowledged (Ribchester et al (2008), Brown 2005)
Research has shown that on average, 90% of all student study activity is directly related to assessment (Ribchester et al (2008))
Feedback that academics provide may not be effectively used by all students (Higgins et al (2002))
Academics and students have been shown to differ on their perception of what makes good feedback (Carless (2006))
“…the literature on student experiences of feedback tells a sorry tale” (Handley et al (2007, p1) quoted by Ribchester et al (2008))
Shriver (1992) has shown that hearing feedback helps writers appreciate the effects of their writing on the reader (Merry and Orsmond (2007))
The technologies exist so why not experiment!
An example
My feedback – A , B
Probably easier to leave feedback when
the work is good rather than poor.
Reactions: Student experiences
Student audio files - Response D Response A
Response B
Response D
Student Reactions: Positive thoughts
Key aspects of audio feedback that students liked were; Its more detailed nature Tone of voice (enriches experience) Clarity – “You can take it in easier” Not reliant on handwriting Its more personal nature – “I listened
carefully because it is more personal” Its potential for feed-forward
Students taking more notice?
“I think it is more articulate than written text. I think it allows you as a marker to articulate your points a lot better than a written line of text does - and it allows you as a student to take the feedback on
board more. I mean, I normally dismiss the written feedback form if I’ve done really badly”
“And you can’t read ahead. You have to listen to it all. You don’t know what’s coming (general
laughs). When you’ve got a sheet you can skim-read it ….”
Student Reactions: Positive thoughts
It is definitely
more memorable because
it is a different
way”
“Written feedback is often too brief and difficult to read”
“I take information in better when it is in a spoken form”
“I find it interesting to hear how you are criticising my work (tone)”
“Overall I think it is extremely useful”
“I really liked it”
“Very easy to use”
“It was the most feedback I have ever had on a single piece of work”
Student Reactions: Negative thoughts
It doesn’t replace the written feedback
Nothing visual to look at Can be too personal for comfort You can’t answer back/ engage in
dialogue
Student Reactions: Negative thoughts
Strange at first – “felt
like my lecturer
was in the room with
me”
“..perhaps made you more nervous.. Builds a bit of tension”
“I found that some of the feedback was very critical and it upset me more as an audio feedback
than a written feedback”
“Personally I do prefer hard copy –
I’m a bit old fashioned like that … so I do
think you shouldn’t loose
hard copy of feedback – you
can keep that for life then”
“My main dislike was that I had nothing visual to look at“
“It was a bit more personal, therefore I feel harder to take the criticism”
“It would wear off. It
can’t be used all the time – or it won’t go in it won’t be registered at
all well”
Staff Reactions: Focus group discussion
There was a reluctance to like this method – why?
Reactions focused on issues of: Exposure and feedback quality Feasibility/practicality Appropriateness Best applications Context limitations and litigation Language Skill acquisition
Staff Reactions: Exposure and feedback quality
Do we ‘hide’ behind written feedback conventions?
Staff Reactions: Exposure and feedback quality
Do we ‘hide’ behind written feedback conventions?
“When I was doing audio, I couldn’t hide anything… it made me think more about
what good feedback constitutes”“You can hide
more in written feedback – you are much more explicit when
speaking something so you need to think much
more carefully”
This isn’t to suggest that teachers are intellectually lazy, but under inevitable
time pressures during summative marking
processes, it was felt that feedback short-hand
comments require less time and effort than
comprehensive spoken feedback.
Do feedback proformas ease or exacerbate this?
Staff Reactions: Exposure and feedback quality
Written feedback, especially that coded onto generic feedback sheets perhaps leads us to write feedback in a form of short-hand that needs to be deciphered by the students.
Do we teach students to decipher this short-hand well enough?
Staff can ‘hide behind’ this short-hand which, for reasons of lack of time or the repetition of the process, reduces the need for great qualification of opinions or development of comments.
In audio feedback, you are able to develop this feedback with “for example.............”
“You do not develop your arguments thoroughly enough”
“You need to contextualise
these findings by referring to
debates in the academic literature”
“Your discussion is too anecdotal”
Staff reactions: A consensus
Feed-forward rather than feed-back is where the greatest strength of this method lies
Not everyone however is convinced of the need for thorough feed-forward – why? Unitised curricula structures Goal-oriented teaching and learning structures “Economies of practice” for teachers and
students (Kinchin and Hay 2007, 98) Teachers and students complicit in a cycle of
“non-learning” (Kinchin and Hay 2007, 98)
Staff reactions: Feasibility and practicality
It takes longer if you are going to do it properly (i.e. Use the method to its greatest potential)
“I think that if had undertaken better preparation for my 5
minutes of audio feedback, by preparing the comments in
advance, then the work necessary (and time taken)
would have been much greater than what I do when I’m giving written feedback,
and we do need to keep coming back to that“
““My gut feeling will still be … phew…. it’s a lot of work”“I don’t think it will ever be
widely applicable”“At the moment based on
my own experiences, I think the same, its beyond us but it may be because it is new, its learning a new language
and new skills””
Staff reactions: Practicality
“You do need a quiet room to sit down … we
work in a fragmented way and slot in marking … to make it efficient. Sitting
down to record a narrative was
problematic. To have a continuous narrative for 5
minutes was difficult.”
Staff reactions: Practicality and disembodiment
“The practicality of it, to me, is my main barrier – I think ideologically and pedagogically there are some benefits but the shear practicality, the fact that we almost disembody the feedback from the written text and then to reconnect – I’m not sure there is anything lost but I expect there is… but simply the recording of it, thinking of what you are going to record, it is a two-stage process
whereas when writing, it is all in one”
Staff reactions: Language and Litigation
“It is much more difficult if you are being natural to control the language that you are using. Now in our feedback we have to work to language criteria – it’s a very controlled environment – excellent is a first, very good is a 2.1 – so audio feedback, it’s almost impossible to work within those limitations - and if you do work to them, it is pointless.”
“The audio feedback has to be different from the written in that it can display some emotion or passion”
Do we need to accompany audio feedback with written guidance as a disclaimer?
Staff reactions: Applications
“I’d be happier giving feedback in a formative sense released from the
constraints of grade point
criteria and the associated language”
“We have sanitised our written feedback so it is probably less memorable. There’s clearly a
role for it, you can motivate and stimulate much more easily it is
much more memorable.”
Formative feedbackFeed-forward
But what of the novelty value?
“I’d be a little wary of replacing written comments with audio feedback – not least from a
quality perspective, records for External Examiners and so on”
Staff reactions: Re-accessibility
“The re-accessibility aspect is important. They can go back to parts of the written feedback but would
they listen to a 5 minute file again if they only needed a bit? Also, they have to listen to the
negative criticism again if there is some, and they might not want to do this and thus may not be able
to bear to listen again”
Will students keep it and listen to it again?
How do they keep it with the work?
Do they need instructions?
Conclusions/ Recommendations I
Excellent for formative feedback/ feed-forward – perhaps not so useful (or necessary) for summative feedback.
Staff were more cautious than students, remaining unconvinced overall of the cost-benefits but recognising great strengths and potentials if used judiciously.
For summative feedback, neither staff nor students recognise audio feedback as a replacement for the rule-governed written feedback.
Students in particular felt that audio feedback provided a much more detailed and richer account of the strengths and weaknesses of the work.
Conclusions/ Recommendations II
Students felt that hearing feedback was more effective and memorable than reading it but both staff and students felt that the sporadic use of audio feedback was important in this respect.
Thus the judicious use of use of the audio medium at the most appropriate moments would be the best approach.
Staff had concerns about the need to be very careful about language and tone in audio feedback, and felt that a separation of intent between written feedback and audio feedback would be useful.
In a nutshell
Excellent for feed-forward Saves time in delivering detailed feed-forward Not time-efficient for vast piles of scripts Best times to use method in the curriculum/degree
programme need identifying A key problem is breaking through established
economies of practice of teachers with regard to feedback
Central to this is a re-evaluation amongst practitioners of the goals, possibilities and importance of feedback
Thank you
Carol EkinsmythPrincipal Lecturer
Department of GeographyUniversity of Portsmouth
References
Baumard, P (1999) Tacit Knowledge in Organizations, London, Sage Brown, S (2007) Feedback and feed-forward, Centre for Biosciences Bulletin, 22, Autumn 2007 Brown, S. (2005) Assessment for learning. Learning and Teaching in Higher Education 1, 81-89. Carless, D. (2006). Differing perceptions in the feedback process. Studies in Higher Education,
31, 219-233. Carless D, Joughin G and Mok M (2006) Learning-orientated assessment: principles and
practice. Assessment and Evaluation in Higher Education 31, 4, 396 Gibbs G (2006) How assessment frames student learning. In C Bryan and K Clegg (Eds.),
Innovative Assessment in Higher Education, Routledge, London. Handley, K., Szwelnik, A., Ujma, D., Lawrence, L., Millar, J., Price, M. (2007). When less is more:
Student’s experiences of assessment feedback. Paper presented at the Higher Education Academy Conference July 2007. http://www.heacademy.ac.uk/assets/York/documents/events/conference/E5.doc
Higgins, R., Hartley, P., Skelton, A. (2002).The conscientious consumer: Reconsidering the role of assessment feedback in student learning. Studies in Higher Education, 27(1), 53-64
Handley, K., Szwelnik, A., Ujma, D., Lawrence, L., Millar, J., Price, M. (2007). When less is more: Student’s experiences of assessment feedback. Paper presented at the Higher Education Academy Conference July 2007. http://www.heacademy.ac.uk/assets/York/documents/events/conference/E5.doc
References
Kinchin I, lygo-Baker S and Hay D (2008) Universities as centres of non-learning, Studies in Higher Education 33, 1, 89-103
Merry, S., Orsmond, P. (2007) Feedback via MP3 audio files. Bioscience HEA Academy Centre for Excellence Bulletin, Autumn 2007
Meyer J and Land R (2005) Threshold concepts and troublesome knowledge (2) – epistemological considerations and a conceptual framework for teaching and learning, Higher Education, May
Price M (2007) Should we be giving less written feedback?, Centre for Biosciences Bulletin, 22, Autumn, 9
Race, P., Pickford, L. (2007). Making Teaching Work: Teaching Smarter in Post-compulsory Education. Sage, London.
Race, P (1999) Enhancing Student Learning, Birmingham: SEDA Ribchester, C., France, D., Wakefield, K. (2008) ‘It was just like a personal tutorial’: Using
podcasts to provide assessment feedback, paper presented at the Higher Education Academy Conference July 2008
Shriver, K. (1992) Teaching writers to anticipate reader’s needs. Written Communication 9(2) 179-208
Smith B (2007) Is assessment really for learning?, Centre for Biosciences Bulletin, 22, Autumn, 11
Trigwell K and Shale S (2004) Student learning and the scholarship of University teaching, , Studies in Higher Education 29, 4, 523-36
Appendix
Practicalities
Phases and progress
Phase one – Experimentation
1. Using audio feedback in a range of ways (individual feedback, group feedback, generic feedback, ‘bite-sized’ feedback
2. Trialling WIMBA as the enabling platform
Phases and progress
Phase 2
1. Staff experiences of using audio feedback recorded in a videoed focus group. Video to be podcasted for use in staff-training workshops.
2. Student experiences and evaluations of the method elicited through Wimba voice email, focus groups and questionnaires.
Phases and progress: Future plans
Phase 3Investigation into where in the curriculum audio feedback can be most effective
Phase 4Evaluation and dissemination (including production of staff-training materials)
Constraints
With limited time and large student numbers, any method will need to be measured in terms of a cost-benefit analysis
Giving more detailed feedback takes time – so what methods are available that might help in this respect?
Phil Race (2008) – on feedback(www.phil-race.co.uk retrieved 12.12.08)
Two of his key recommendations are:
Making sure they get feedback quickly enough so that they still care about it
Ensuring that they get plenty of feed-forward, so they can make their next piece of work better
Audio feedback – my method
Read work, make notes against the assessment criteria.
Also make notes on each section of the work (perhaps paragraph by paragraph for a traditional essay). This might be in the margins of the work.
Take a note of overall feedback. Make a voice recording of this feedback. Send to student. Provide summary notes on a conventional
marking sheet for return to the student.
Staff Reactions: My experiences
Can be daunting at first leaving the recording but you quickly gain confidence and do not feel the need to listen to the message
Despite worries about tone of voice or in authenticity of sound, if you are natural and speak as you normally would, the recordings are well-received
You get quicker at the whole process – at first you think it will never work. Having said this, it is more time-consuming than methods I am used to (but the quality of feedback is much greater)
In spoken form, I could go into depth about issues and make suggestions that I simply wouldn’t have time (or inclination) to do in written-form.
Minute for minute therefore, it is much quicker to speak than to write.
You need to record the feedback after you have read an
Audio feedback- Colleague A’s method
Read work and record feedback instead of writing it on marking sheet – trying to take no longer than he would normally take in conventional marking/ feedback production
Audio feedback – Colleague B’s method
Provide feedback in a PowerPoint presentation – scanning in sections of the student work and using ‘narration record’ to provide explanation of points.
Hardware & Software The recordings could be made using any
MP3 or digital recording devise but sending files then takes time.
VLE/Wimba Voicemail can be used and for this, only a headset is needed.
PowerPoint can be used for audio-visual feedback
Wimba audio archive