Student Evaluations: How Do They Influence Teacher

Thinking And Behaviour?

In search of the evidence

Project Team

Back row:Stuart Terry (OP)Sarah Stein (OU)Lynley Deaker (OU)

Front row:Jo Kennedy (OU)Dorothy Spiller (WU)Trudy Harris (WU)

Research Question

How do the current evaluation processes and practices influence teachers’ thinking and behaviours in relation to student learning at all stages of the teaching and learning cycle?

Overview of Project• What are teachers’ perceptions and how do these perceptions

influence teacher thinking and behaviour?• Does institutional use constrain teaching innovation?• Why do teachers undertake evaluations?• Do teachers use student evaluation feedback to improve

teaching?• Do they engage in dialogue with students about evaluation

feedback and their responses to it?• How can we use our findings to enhance the use of evaluations to

improve teaching and learning?

Progress to DateBy the 2010 AKO Colloquium:• Literature and environmental scan around evaluations• Preparation and writing of a literature review• Questionnaire sent out to academic staff (2445 – 45%

response rate).

Since the 2010 AKO Colloquium:• Quantitative & qualitative data analysis of the survey data• Interviews conducted at each institution• Thematic analysis of the qualitative data (survey & interview

data)

Quote of the Week“I’m a lapsed Irish Catholic with massive guilt and chips on my shoulders so whenever I get an evaluation faster than any computer software you have I will gravitate towards the negative comments about what I am doing and I will, my parents and grandparents have taught me very very well to ignore compliments, so any thing that is remotely complimentary I ignore and assume that it is just someone trying to be nice and just focus on the negative things, whether it’s a number or a comment. So my in my mind I think gosh I have really let someone down and them completely ignore the other 50 odd students who have good comments.That’s the other thing about how we receive them. It’s a bit of paper, it’s a set of numbers, and we know stats are like a bikini, they show something but often hide the best bits!”

(from an interview with an Otago Polytechnic staff member)

Initial Findings - Quantitative• Frequencies• Chi Square and cross tabulations• ANOVA• Factor analysis• Cluster analysis• Regression

Teachers at Otago University are more constrained by the possible negative effects on their students evaluations when looking at trying new teaching approaches than the other two intuitions.(P<0.01).

15. My willingness to try new teaching approaches is constrained by the possible negative effects on my student evaluations/appraisals.

A great deal 1 2 3 4 5 Not at all

15. My willingness to try new teaching approaches is constrained by the possible negative effects on my student evaluations/appraisals.

A great deal 1 2 3 4 5 Not at all

  1 & 2 ratings % 3 rating % 4 & 5 ratings % Nil responses %

Waikato University 

16% 18% 63% 3%

Otago Polytechnic 

8% 11% 71% 10%

Otago University 

15% 20% 49% 16%

All Institutions 

15% 18% 56% 12%

26. How long have you been teaching in the tertiary sector?0-5 years6-10 years11-15 years16-20 years21 years or more

15. My willingness to try new teaching approaches is constrained by the possible negative effects on my student evaluations/appraisals. A great deal 1 2 3 4 5 Not at all

The only statistically significant difference found was at Otago University where the 11-15 years teaching experience group identified as being more constrained than the other groups (p<0.01)

Additional Quantitative Analysis – questions 15 & 26

Initial Findings – Qualitative (Survey Comment Data)

15. My willingness to try new teaching approaches is constrained by the possible negative effects on my student evaluations/appraisals.

A great deal 1 2 3 4 5 Not at all

16. Please explain your answer to q15 (WU & OU) / 16. Any comments about question 15 (OP)

Total of 595 comments for q16 (56% of respondents made comments)

0%

5%

10%

15%

20%

25%

30%

35%

40%

5%

14%

38%

17%

14%

4%

8%

q16 Comment Data - Theme Coding - All Institutions

Percentage of Comments

q16 - Please explain your answer to q15: My willingness to try new teaching approaches is constrained by the possible negative effects on my student evaluations/appraisals. 1=A great deal, 5=Not at all

Q16 - Example QuotesCode Ref1 - constrained – absolutely - “Absolutely! If my career advancement will suffer due to lower evaluations; why would I take a chance on innovative; demanding; or difficult course material?????” (OU)

Code Ref2 - constrained - cautious about evaluations - “There appears to be greater monitoring teaching appraisals these days and I feel greater pressure to get good appraisal results than try new teaching approaches.” (WU)

Code Ref3 - not constrained – absolutely - “No, I think that by being creative in the classroom and making changes that students actually respond to this and like it - I would not hold back” (OP) “I don't really care about getting sacked or a pay rise; if I was in the job for the money I would do a different job and earn 2* as much!” (OU)

Code Ref4 - not constrained - need evaluations to learn from them - “Appraisals are good servants but bad masters. The point is that they should benefit the students, not (or only secondarily) me.” (WU)

Next steps

• Completion of the coding of the qualitative data from the survey

• Thematic analysis of interviews to be completed at all institutions

• Combination of the qualitative and quantitative data to develop a fuller understanding of teacher thinking

• Comparison of findings with literature• Writing of the AKO research and summary reports

Are There

AnyQuestions?

Are There

AnyQuestions?