Extract from the AQSM (last approved Senate 2008) Student ...

Quality Assurance General Guidance

V2, Nov 2012

Extract from the AQSM (last approved Senate 2008)

Student evaluation of modules

Student evaluation of modules is used as part of the University's quality assurance procedures and is

intended to inform decisions on programme development and the overall process of monitoring the

effectiveness of programmes. The purpose of the student evaluation process is to improve the

quality of learning and teaching. Evaluations are not intended to monitor the individual

performance of members of academic staff, nor to be channels for student complaints, for which

other procedures exist.

Module evaluation may be either qualitative or quantitative in emphasis. The University does not

dictate the form or content of questionnaires and other methods used in module evaluation.

However, evaluation should be informed by the advice available in Annex A.

All modules offered by a School must be evaluated on every occasion the module is taught.

All forms of evaluation, including questionnaires, should be made available to and collected from

students in such a way as to permit students sufficient time for a considered response,

proportionate to the nature of the evaluation. Schools should aim to achieve an optimal rate of

return.

The anonymity of respondents must be guaranteed, although this does not preclude making use of

student evaluation volunteered non-anonymously.

The member of staff responsible for the module should incorporate a summary statement giving the

number of responses and per cent response rate and a numerical summary of gradings and also an

evaluative summary of the responses into the annual module report. In all cases both the summary

statement and the module report should be reviewed annually by the relevant Programme

Committee and Staff Student Liaison Committee.

A report on the overall outcome of student evaluation of modules in each programme should be

made at least annually to the Programme Committees responsible for the running of the relevant

programmes. The report should contain at least the following information:

the number of students in each module;

the number of responses;

the response rate as a percentage;

an overall appraisal of student views in each module;

consideration of the main issues raised.

Programme Committees should report on their consideration of the outcome of module evaluation

by students to the School's Learning and Teaching Committee, with recommendations for action.

Reports must be made to the School's Staff-Student Liaison Committee(s) on the results of student

evaluations, and the School's proposed response. Comments and recommendations made by the

Staff-Student Liaison Committee(s) must be reported to the committee responsible for managing the


V2, Nov 2012

programme, which must report its response back to students, even if the decision is to take no

action. It is important that this process should be recorded in full in the minutes of both committees.

Completed questionnaires and a record of the School's response should be kept for at least three

years and should, subject to requirements of confidentiality for matters relating to individuals, be

made available for internal quality audit and external review.


V2, Nov 2012

Annex A

Guidance on Good Practice in Module Evaluation

Matthew Wyman, Stephen Bostock and Chris Pike

Approved by Learning and Teaching Committee 14 October 2009 with a minor edit approved

afterwards by the Chair.

This document is about evaluating the student learning experience in a module so as to be

able to improve it. Much of that experience is determined by the module design and our

teaching, supporting learning, and assessing student performance in it. This is evaluation for

formative purposes, to help us improve the module, not for summative purposes - making

overall judgements about the module and, possibly, the teaching on it for management

purposes. To repeat, this is not for judging the quality of teaching, it is to improve the student

learning experience including, possibly, the quality of teaching as students perceive it.

Designing, teaching and assessing are reflective practices and we need to gather

information, both informally and formally, to reflect on.

The student learning experience is complex and multifaceted and understanding it better

involves multiple methods of gathering information and then making reflective judgements

from that information, against broad criteria. There is no single method that will provide all

the relevant information at a single time (e.g. a questionnaire), so we should employ a

variety of methods to gather information from a variety of sources. Multiple sources and

perspectives will provide the basis for more secure and valuable inferences. The most

appropriate methods to use will vary with the context – the discipline, the history of the

module, the level, and so on. Some techniques are listed below. Above all we should provide

risk-free contexts for our students to “tell” us honestly about their experiences, and “listen” to

them (but not necessarily in a verbal conversation).

Current practices at Keele are varied. In most programmes, teaching is systematically

evaluated, primarily for summative purposes. Individual teachers are free, if they wish, to use

these data for formative purposes, but it may often not be the most appropriate. It is the

responsibility of the module leader to formatively evaluate their module each time it runs.

Evaluation of the student experience of learning on a particular module can be carried out at

many points, in the classroom or virtual environment, to compare the perceptions of the

teacher with those of the students. The purpose of this evaluation is for teachers to find out

what changes they might make in teaching methods or style, course organization or content,

evaluation and marking procedures, use of resources etc., in order to improve student

learning.

Evaluation is initiated by the teacher and information and feedback can be solicited from

many sources (for example, oneself, students, colleagues, experts from the central

university) using a variety of methods, and form the basis for incremental, ongoing

improvement. The information gathered belongs to the module leader but it may also be

productively shared with colleagues and others. Generally, students should be given a

summary of any information they have provided as a group.

Student performance is also valuable information about the success of the module,

obviously. As well as final grades to coursework and examinations, more frequent simple


V2, Nov 2012

checks on student learning will keep a teacher in touch with student progress, or lack of it,

and identify gaps between what has been ‘taught’ and what students have actually learned.

The timing of collecting information – such asking the students about the module – is crucial.

If you leave it until after the module ends the students will have experienced the whole

module including the assessment. This may be best for summative evaluation but not for

current purposes. It will be difficult to get many students to respond and, worse, they have

no incentive to tell the whole truth – it is too late for their experience of the module to be

improved. Therefore, gathering information at least once in the middle of the module, or at

one-third and two-third points in the module, will get more reliable information and,

importantly, you can respond to issues quickly to improve the module for those students. If

students see you doing this it will improve their motivation and engagement. Quick and

simple evaluations can be done every week or two, for example on slips of paper left after a

lecture. ‘Little and often’ opportunities for students to tell you their enthusiasms and concerns

will give you timely information about how the module is going, from their point of view.

One way of viewing module evaluation by students is that it is an additional channel of

communication between you and your students. Use it to frequently keep in touch with their

experience of the module, inevitably very different from your experience as the teacher.


V2, Nov 2012

Example methods of evaluation

1. Classroom evaluations

Classroom evaluation discovers the effects of teaching on student learning and/or the

opinions of students about the module. It involves methods designed to give teachers

ongoing feedback about the effect their teaching is having on the level and quality of student

learning. Unlike routine tests and quizzes, classroom assessment can be used in a timely

way to help instructors identify gaps between what they teach and what students learn and

enable them to adjust the module or their teaching to make learning more efficient and

effective. The information should always be shared with students to help them improve their

own learning strategies and become more successful self-directed learners.

Classroom evaluation can be integrated into teaching in a graduated way, starting out with a

simple assessment technique in one class involving five to ten minutes of class time, less

than an hour for analysis of the results, and a few minutes during a subsequent class to let

students know what was learned from the assessment and how the teacher and students

can use that information to improve learning. After conducting one or two quick evaluations

the teacher can decide whether this approach is worth further investment of time and

energy.

1.1 The One Minute Paper

The One-Minute Paper, a brief written reflection, is a technique that is used to provide

teachers with feedback on what students are learning in a particular class. It may be

introduced in small seminars or in large lectures, at any level, or electronically via the KLE.

The One-Minute Paper asks students to respond anonymously to the following questions:

a. What is the most important thing you learned today?

b. What question remains uppermost in your mind?

Depending upon the structure and format of the learning environment, the One-Minute Paper

may be used in a variety of ways:

During a lecture, to break up the period into shorter segments enabling students

to reflect on the material just covered

At the end of a class, to inform your planning for the next session

In a module comprising lectures and tutorials, the information gained can be

passed to seminar tutors giving them advance notice of issues that they may

wish to explore with students.

You need to collect these One-Minute Papers on slips of paper or post-its, in the session or

after it.

An alternative formulation is The Muddiest Point, particularly useful in gauging how well

students understand the course material. Ask students:

What was the ‘muddiest point’ for you today?


V2, Nov 2012

1.2. One Sentence Summaries

One Sentence Summaries can be used to find out how concisely, completely and creatively

students can summarize a given topic within the grammatical constraints of a single

sentence. It is also effective for helping students break down material into smaller units that

are more easily recalled. This strategy is most effective for any material that can be

represented in declarative form – historical events, story lines, chemical reactions and

mechanical processes.

The One Sentence Summary technique involves asking students to consider the topic you

are discussing in terms of Who Does/Did What to Whom, How, When, Where and Why, and

then to synthesize those answers into a single informative, grammatical sentence. These

sentences can then be analyzed to determine strengths and weaknesses in the students’

understanding of the topic, or to pinpoint specific elements of the topic that require further

elaboration. Before using this strategy it is important to make sure the topic can be

summarized coherently. It is best to impose the technique on oneself first to determine its

appropriateness or feasibility for given material.

A modern alternative your students may be familiar with is Twitter: a Tweet message is no

more than 140 characters, but it is not likely to use standard grammar.

Methods 1 and 2 above concern what students have learnt; the next two ask them for their

opinions on the module.

1.3. Critical Incident Questionnaires

A more detailed alternative to the above is The Critical Incident Questionnaire, a technique

that can be used to find out what and how students are learning, and to identify areas where

adjustments in the module are necessary (e.g., the pace of the course, confusion with

respect to assignments or expectations).

On a single sheet of paper, students are asked five questions which focus on critical

moments for learning in a course. The questionnaire is handed out about ten minutes before

the end of a class and collected at the end.

Critical Incident Questionnaire

1. At what moment this week were you most engaged as a learner?

2. At what moment this week were you most disengaged as a learner?

3. What action or contribution taken this week by anyone on the module did you find

most beneficial or helpful?

4. What action or contribution taken this week by anyone on the module did you find

most puzzling or confusing?

5. What surprised you most about the course this week?


V2, Nov 2012

Critical Incident Questionnaires provide substantive feedback on student engagement and

may also reveal dynamics in the classroom that may not initially be evident to the teacher.

1.4. Quick questionnaires

A single, anonymous sheet, given out 5 minutes before the end of a class can ask three

questions:

1. What aspects of this module are most helpful to your learning?

2. What aspects of this module are least helpful to your learning?

3. How should we improve the module for you?

1.5 The Helping Game

Aims/Rationale

To encourage students to put into words their reflections and concerns about the

module.

To use the group and the teacher as a source of advice and help.

Resources: An empty shoe box

Method

1. Ask students to write down individually one or more aspects of the module, or

studying generally, that they have found problematic to date. Fold and put into the

shoe box.

2. Shake the box and ask students to draw out one piece of paper each, or until the

box is empty, and to think about how they would overcome the ‘problem’ described. If

by chance they pull out their own, ask them to re-draw.

3. Students take it in turns to read out the problems that they have drawn, and make

suggestions about how they would deal with it. Discuss in the whole group and offer

appropriate advice.

2. Student Evaluation Questionnaires

Using a questionnaire with some standard questions facilitates comparison of the student

experience over time and across levels. Structured questionnaires are particularly

appropriate where there are relatively large numbers of students involved. Provided they are

well designed, questionnaires are relatively economical to administer, summarize and

interpret. However, if they take time to analyse, any changes you make because of them

may be too late.

Provided that students are asked to comment only on items with which they have direct

experience, student responses to questionnaires have been found to be valid. Closed

questions obviously limit what students can tell you to what you are asking about – there

may be other concerns you haven’t expected. Open-ended questions are more difficult to

analyse but they often provide more useful sources of information. Also, open-ended


V2, Nov 2012

questions provide insight into the numerical ratings you also collect, and provide suggestions

for making changes. In general, questionnaires should always ask students what aspects of

the module are going well for them, what aspects cause problems, and what suggestions

they have for changes that would improve their experience. The simplest questionnaire asks

only that.

Don’t ask students to directly rate the teaching, you cannot expect honest or helpful

answers. Other aspects of the module that students are probably not in a position to make

informed comment about include the appropriateness of the intended learning outcomes and

the selection of content.

As with all sources of evaluation information, care is needed in interpreting it, in light of other

sources of information and in comparison with other modules and previous runs of this

module.

3. Peer Observations

See Keele University Guidelines on Peer Review of Teaching at

http://www.keele.ac.uk/depts/aa/landt/links/PVTguidelines.htm

4. Letters, Structured Interviews and Focus Groups

Individual and group interviews, and anonymous ‘letters’, can elicit information not readily

available through questionnaire ratings. Insights, success stories, and thoughtful analyses

are often the outcomes of an interview or a request for a written impression of the student

experience on a particular module. Students who are reluctant to give information on a rating

scale or in written form often respond well to a skilled interviewer. They can provide

information on:

Experience of the module through reflection

The module administration

The effectiveness of teaching

The usefulness of online resources

However, interviews and focus groups cannot be run by the module teacher(s); students

cannot realistically be asked to provide in-depth information on their experiences and

feelings by those assessing them. You will need help from an experienced colleague(s) who

is not teaching the module or from a central support service like the LDU.

Disadvantages are that letters can have a low response rate, interviews are very time-

consuming, and both are possibly unrepresentative.

For further information on Critical Incident Questionnaires see Brookfield, S. J. and Preskill, S. (1999)

Discussion as a Way of Teaching: Tools and Techniques for a Democratic Classroom. (CA: Jossey

Bass), page 49

http://www.keele.ac.uk/depts/aa/landt/links/PVTguidelines.htm


V2, Nov 2012

This guidance used freely a document at York University, Canada, Guide to Teaching Assessment

and Evaluation http://www.yorku.ca/univsec/senate/committees/scotl/tevguide.pdf

http://www.yorku.ca/univsec/senate/committees/scotl/tevguide.pdf


V2, Nov 2012

Appendix: ideas for evaluation questionnaires (formative or summative)

The Use of Qualitative Approaches to Student Feedback

Research evidence suggests that questionnaires which make use of qualitative feedback on modules

tend to be particularly useful in giving teachers information about ways to develop and improve

modules. Where problems arise, this approach allows staff more easily to identify the underlying

reason for these problems.

Qualitative feedback is time-consuming to analyse, and a major weakness is that it struggles to

illustrate what views are typical /representative. However the University strongly endorses a mixed

approach, combining qualitative and quantitative indicators, or using qualitative approaches to

follow up problems when they arise. The following is an example of how qualitative feedback can be

generated in written form. This is ideally combined with group discussion of the points that arise

from individual comments.

MODULE QUESTIONNAIRE

We are interested in anything that affected the way you learned on this module.

1. POSITIVE POINTS

What would you say were the best features of this module? What did you enjoy most, and what

should be retained in future years?

(You might like to consider in your answer any of the following, or anything else you think is an important issue: module content and organisation; quality of the teaching; anything that helped or hindered your ability to learn; workload; the way you were assessed; usefulness of feedback on your work; availability of teaching staff)

2. NEGATIVE POINTS

What aspects of the module were less successful? What problems did you encounter in the course

of your studies? What needs to be changed in future?

(You might like to consider in your answer any of the following, or anything else you think is an important issue: module content and organisation; quality of the teaching; anything that helped or hindered your ability to learn; workload; the way you were assessed; usefulness of feedback on your work; availability of teaching staff)


V2, Nov 2012

Possible indicators for Quantitative Questionnaires

1. Questionnaires should as a suggested minimum obtain feedback on the following areas (sample

indicators follow):

Learning / academic value of module

Quality of teaching

Clarity and achievement of goals and expectations

Availability of advice and guidance

Workload

Suitability of assessment

2. Suggested additional factors, which will not be appropriate for every module

Usefulness of learning resources

Availability of learning resources

Encouragement of independent learning

Group interaction

Quality of feedback (required where timing of evaluation permits)

Skills development

3. For dissertation modules or equivalent, questionnaires might also obtain feedback on the

following areas:

Academic value of module

Availability of supervisor

Quality of supervision

Availability of resources

Skills development

4. Separate questionnaires will be needed for placement learning, fieldwork and other distinctive

forms of teaching as appropriate.


V2, Nov 2012

Sample indicators

Please note that Schools, programmes and individual teachers are encouraged to develop these

questions as appropriate for local needs. Nor should this list be considered restrictive. Answers may

be qualitative or quantitative, and if the latter, with answer codes in any form.

Learning / academic value of module

The module was intellectually stimulating

I have learned something that I consider valuable

My interest in the subject has increased as a consequence of this module

I have understood the subject materials in this module

Good teaching

Students were motivated to do their best work

Staff were good at explaining things

Staff have made the subject interesting

Clarity and achievement of goals and expectations

Proposed objectives agreed with those actually taught so you knew where the class was going

It’s always easy to know the standard of work expected on this module

You usually have a clear idea of where you are going and what’s expected of you

It’s often hard to discover what’s expected of you

The aims and objectives of this course are not made very clear

Staff made it clear right from the start what was expected of students

The module outline made it clear what I was expected to do on this module

Availability of staff

Staff made students feel welcome in seeking advice in or outside of class

Staff were adequately accessible to students during office hours or after class

Staff made a real effort to understand the difficulties students were having with their work

Workload

The workload for this module is too heavy

It seems to me that the module tries to cover too many topics

We are generally given enough time to understand the things we have to learn

The sheer volume of work on this module means you can’t comprehend it all thoroughly

Average numbers of hours work per week required outside class


V2, Nov 2012

Usefulness of learning resources

Required readings / texts were valuable

Required readings contributed to my appreciation and understanding of the subject

Availability of learning resources

Required readings / texts I needed for this course were easily available

I had sufficient access to the materials /equipment I needed

The classes were too big

Encouragement of independent learning

There are few opportunities to choose the particular topics you want to study

The module has encouraged me to develop my own academic interests as far as possible

Students have a great deal of choice over how they are going to learn in this module

Students are given a lot of choice in the work they have to do

Group interaction

Students were encouraged to participate in class discussions

Students were invited to share their ideas and knowledge

Students were encouraged to ask questions

Students were given meaningful answers to questions asked

Students were encouraged to express their own ideas and / or question the staff member

The tutor frequently gives the impression he/she has nothing to learn from students

Suitability of assessment

Assessment on this module was too difficult

The methods of assessment on this module gave me a fair opportunity to demonstrate what I had

learned

Assessment deadlines were appropriate

There was too much assessment on this module

Quality of feedback

Feedback on assessed work was comprehensive and useful

Feedback on assessed work was received in good time

Skills development

This course has helped me to develop my problem solving skills

This course has sharpened my analytic skills

This course has helped develop my ability to work as a team member

As a result of doing this course I feel more confident about tackling unfamiliar problems


V2, Nov 2012

This course has improved my written communication skills

This course has helped me develop the ability to plan my own work

This module has helped me to develop my research skills

Indicators suitable for feedback on dissertations

Advice on how to choose a topic of the right size was available

Advice on choosing a research question was available

Advice on how to research my chosen topic was available

My supervisor was sufficiently available for advice on my dissertation

The advice I received on my dissertation was helpful

There were sufficient resources available for me to research my topic

Feedback on draft written work was helpful

I knew what was expected of me on this module

This module has helped me to develop my research skills

Extract from the AQSM (last approved Senate 2008) Student ...

Documents

Transcript of Extract from the AQSM (last approved Senate 2008) Student ...