Evaluating a Student Rating of Teaching Form
John Ogier
Survey & Testing Unit (STU)
University Centre for Teaching & Learning (UCTL)
Ph 64-3-364-2850
A case of quality not being assured!
• “No single student rating item, nor set of related items will be useful for all purposes”
Cashin (1995)
•“Student ratings should be used to make only crude judgements of instructional effectiveness (exceptional, adequate, and unacceptable).”
D’Apollonia & Abrami (1997)
UOC Student Surveys• Course
– Organisation• a well organised course
– Stimulation• helped to stimulate my
interest in the course area
– Workload• The overall workload in this
course was reasonable
– Difficulty• The level of difficulty of this
course was reasonable
– Overall• Overall, this was a good
quality course
• Teaching– Organisation
• classes were well organised
– Communication• able to communicate ideas
and information clearly
– Interest• stimulated my interest in the
subject
– Attitude• attitude towards students has
been good
– Overall• Overall, the lecturer is an
effective teacher
Effective?
• How appropriate are these 5 Likert scale questions?– Do they define a “good
teacher”?– Is that measure reliable?
• 1-5 scale appropriate?– Some cultures are use to
“1” being a good score– Is 3 really “neutral”?
• Students circle 2 no’s!
• Students circle a column!!
• Is it correct to use the mean of a discontinuous scale?
• Median rather than mean?
Other survey “quality” issues
• Was the class told about it in advance?• What extra questions were asked?• When in the semester was the survey held?• What time was the survey held?• How many other surveys were given out?• Who administered the survey?• How long were they given to complete it?• Is it really anonymous?• Ad hominem comments!
Possible factors
• Course
Organisation
Stimulation
Workload
Difficulty
Overall
• Teaching
Organisation
Communication
Interest
Attitude
Overall
Correlation of Q5 (Overall) mean against the means of Q1, Q2, Q3 & Q4
weighted by responses (capped at 50)
Organisation Communication Interest Attitude
R2 = 0.7441 R2 = 0.9222 R2 = 0.8310 R2 = 0.6580
The lecturer is able to communicate ideas and information clearly
Overall, the lecturer is an effective teacher
Scatterplot & Regression Line- Overall v Communication
Where to now?• Difficulties faced by tertiary NESB students in English language lectures
• Cosmopolitan UOC academics – both NESB & ESB students must cope!
• Finegan, T.A., & Siegfried, J.J. (2000). Are student ratings of teaching
effectiveness influenced by instructors’ English language proficiency?
– they found in an introductory Economics course that “the student
ratings of ESL instructors are, on average, about 0.4 points lower,
on a scale of 1 to 5, than the student ratings of native English
speaking instructors.” The American Economist, 44, 17-29.
NESB by Subjective Classification
• Surveys (UGrad, 2 or more responses) - 7072 from ESB lecturers and 524 from NESB lecturers
Means with 95% Conf.
ESB NESB
Q2 - Comm 3.810 ± 0.002 3.216 ± 0.009
Q5 - Overall 3.922 ± 0.002 3.496 ± 0.009
Q5 Difference ~ 0.43, Q2 ~ 0.60
A similar difference, but…
• What about across
– Faculties?
– Levels?
– Departments?
Breakdown of surveys
Number of Cases within grouping variables (and no. of lecturers surveyed)
Arts Commerce Engineering Science Law
Level Y ESL N Y ESL N Y ESL N Y ESL N Y ESL N
1(Lect)
14(8)
476(153)
12(5)
98(31)
0 38(15)
42(12)
585(167)
0 29(10)
2 (Lect)
40(16)
678(196)
24(9)
213(62)
22(6)
245(81)
89(23)
713(206)
0 88(18)
3 (Lect)
31(13)
517(151)
46(14)
267(83)
49(15)
387(96)
98(24)
820(218)
0 193(29)
4 (Lect)
0 0 0 0 57(17)
537(148)
0 0 0 0
Faculty Means
Why the faculty effect?
• Cashin (1990) mentions a hypothesis that– students’ quantitative skills are less well
developed than their verbal skills so that quantitative based courses are more difficult for students and are more difficult for lecturers to teach.
By Level
Some relevant student commentsThe following student comments from relevant courses put ‘quantitative skills’ into perspective:
~ “He did the best he could but his accent was still very strong, and made a complex subject even harder to understand.”Ratings: ‘Comms’ = 2, ‘Overall’ = 3~ “.. we should have loud, clearly spoken English from the lecturer as otherwise you have to concentrate too hard on just understanding the lecturer.”Ratings: ‘Comms’ = 1, ‘Overall’ = 1
Compare a NESB student rating an ESB lecturer ~ a strong accent, ~ “It’ll be very interesting if the lecturer can speak in Thai.” Ratings: ‘Comms’ = 2, ‘Overall’ = 2.
Implications
• For a “Transmission model” of teaching, do we need anything more than a Communication question!!! ?– Need questions that cover the important aspects of
teaching (or courses) with an appropriate weighting.• NESB lecturers may have a tougher time gaining
promotion and tenure.– Assistance with language?– Alternative communication methods?
• Assisting ESB & NESB students to understand and cope?
The NESB student’s view
• Econ104 ~ 33% NESB Students• Online survey – 353 responses ESB, 165 NESB• Overall – ESB 4.58 ± 0.07, NESB 4.28 ± 0.10
– “... And think about the language he use in the test paper some words I even do not know and if he want to use it then use it in his lecture. ECON test is not a word game its aim should be test people how well they learn for ECON!”
Ratings: ‘Comms’ = 3, ‘Overall’ = 4
Top Related