TUTOR SELECTION PROCESSES AND THE STUDENT LEARNING …

UTS TEACHING & LEARNING COMMITTEEVolume 15, Number 1, 2018, pp.8-28
TUTOR SELECTION PROCESSES AND THE
STUDENT LEARNING EXPERIENCE*
School of Economics,
University of Queensland
Le Hoa Phan
University of Queensland
ABSTRACT
The selection process for tutors in higher education is somewhat opaque and is
largely unexplored. We compare two methods of tutor selection. The first,
traditionally used in many universities, is based on the academic performance of
applicants. The second is based on both academic performance and a group
interview process that focuses on the applicants’ communication and interpersonal
skills. Data from teaching evaluations suggests that while most tutors selected
under the traditional process perform well, some must be regarded as
‘underperforming’. We provide teaching evaluation data showing that tutors
chosen under the new selection process improved by nearly 20 percent on average,
and that the proportion of ‘underperforming’ tutors fell from 19 to less than 7
percent. These findings suggest that the group interview process can complement
and improve on the traditional process of tutor selection based on academic
performance alone.
JEL classifications: A22.
* Correspondence: Kam Ki Tang, School of Economics, University of Queensland, QLD
4072, Australia. E-mail: [email protected]. Thanks to two anonymous referees for
comments and suggestions.
Tutor Selection Processes 9
The hiring process typically requires candidates to submit resumes,
research papers and reference letters, to present seminars, to meet with
incumbent staff and to undergo one or more interviews. These multiple
hurdles aim to ensure that the best person is hired. Yet as far as teaching
is concerned, full-time faculty members are not the only ones employed
for this task. Casual teaching staff, sessional academics and tutors (also
referred to as sessional teachers in Australia, adjunct faculty in North
America, or part-time teachers in the UK) are also hired for teaching
duties. Many tutors are hired by universities from their undergraduate
or postgraduate cohorts. Yet the processes used to hire these tutors are
not always clear, sometimes even to those academics working with
them (Gilbert 2017).
In U.S. and European institutions, teaching assistant appointments
are reportedly based on academic merit alone with no assessment of
teaching ability (Fumasoli, Goastellec & Kehm 2015; and Sutherland
2009). In Australia, the recommended minimum standards for the
recruitment of sessional staff within the Benchmarking Leadership and
Advancement of Standards for Sessional Teaching (BLASST 2013)
states that universities should at least specify minimum qualification
requirements for the recruitment of sessional staff. But no other advice
is provided within the BLASST guidelines for the selection process.
Anecdotal evidence suggests that tutor selection in many Australian
universities is mostly based on the academic achievement of applicants
as indicated by such measures as their Grade Point Average (GPA), or
whether they are enrolled in a postgraduate program, especially a Ph.D.
program. The effectiveness of such practices depends on the
assumption that students with higher academic achievement are more
knowledgeable and will, therefore, make better tutors. Such practices
also have administrative benefits such as being inexpensive and easy to
implement, and being arguably objective and equitable. They do,
however, have limitations. Students, universities and society-at-large
expect casual tutors to contribute to quality learning environments
(Sutherland 2009) and the narrow focus on academic performance in
tutor selection may not necessarily meet this expectation if being smart
is not enough to be effective in the classroom. The selection process
may, therefore, need to identify applicants that, when provided with
minimal tutor training, will be able to teach, manage and facilitate
10 C. Sherwood, K.K. Tang, Z. Yin & L.H. Phan
effective learning environments (Knott, Crane, Heslop & Glass 2015).
A good hiring process does not eliminate the need for training, but the
training will be more efficient if the trainees have the attributes for the
job in the first place.
This paper describes and evaluates an alternative tutor selection
process developed at the University of Queensland (denoted as UQ
hereafter) in Australia, to identify tutors with the right attributes for
effective teaching as well as the intellectual ability to work in a
university department. More specifically, we ask and answer the
following series of questions: Is academic performance a useful
criterion for selecting tutors? Is academic performance a sufficient
criterion for selecting tutors? What importance should be placed on
applicants’ communication and collaboration skills compared to their
academic performance in the selection process? Does the new selection
process developed at UQ result in higher tutor evaluations from
students, suggesting an enhanced learning experience? If the new
selection process does improve the student learning experience, is this
improvement significant enough to justify the additional costs that it
entails? These questions will be answered by comparing data from
student feedback surveys for tutors selected under the old and new
reqimes at UQ. Before considering the approach taken to this analysis,
however, we review relevant liteature.
2. LITERATURE REVIEW
In Australia since around 1990, the growth of casual teaching staff in
universities has outpaced the growth of full-time academics as a result
of the “massification” of higher education (Bryson 2013; Kreber 2007;
and Matthews, Duck & Bartle 2017). From 1989 to 2007, Australian
tertiary students increased from 441,000 to over one million. Casual
teaching staff have been deemed a solution by university administrators
to the problem of meeting significantly increased demand for teaching
services while simultaneously addressing issues relating to staff costs,
redundancy and ensuring that full-time teaching academics have
sufficient time also to conduct research (Bryson 2013; Coates, Dobson,
Goedegebuure & Meek 2009; Lama & Joullié 2015; Percy & Beaumont
2008; and Sutherland 2009). In his introduction to the RED Report
(Percy et al. 2008, p.6), the University of Wollongong Vice-Chancellor,
suggested that “between 40 to 50 percent of teaching in Australian
higher education is currently done by sessional staff”. Likewise,
Harvey, Fraser & Bowes (2005, p.2) observed that up to 85 percent of
teaching staff in one department in their Australian university were
sessional, while Sutherland & Gilbert (2013, p.1) reported that 40
percent of teaching staff in New Zealand universities were casual.
Clearly, as for the last 30 years, casual teaching staff today continue to
make a significant contribution to teaching in higher education and
tutors are one of the largest segments of causal teaching staff. This is
particularly evident in large undergraduate courses.
For these courses, lecturers may take only a few or no tutorials,
leaving this task largely to tutors. Assuming a teaching model where a
one hour tutorial accompanies a two hour lecture, tutors occupy about
one third of the academic contact hours with university students.
Furthermore, tutorials are important in contributing toward enhancing
student learning outcomes (Baderin 2005). Milliken & Barnes (2002,
p.17) argue that tutorials can provide “an effective arena for teaching
and learning through immediate, interpersonal dynamic exchange”.
Tutors are, therefore, a major contributor to the overall quality of a
university’s education programs. However, how much thought have
universities given toward the selection of tutors?
With quality of teaching and learning being at the heart of the student
experience, and with tutors increasingly and actively contributing
towards this experience, surprisingly little has been written on exactly
how tutors are being recruited. There is a large body of literature on
graduate teaching assistants (see, for example, Park (2004) and Muzaka
(2009)), yet this literature rarely touches on the practice of their
recruitment. Park & Ramos (2002) suggested that in North America,
teaching assistant jobs are commonly offered to graduate students
because of their financial needs, while in the UK, graduate teaching
assistants are recruited largely based on their potential to undertake
research. Since casual appointments are typically on a short-term basis,
there is a general belief that the stringent recruitment processes
designed for continuous appointments is unwarranted (Lama & Joullié
2015). This very much reflects the nature of tutor appointments. For
instance, Kwiek & Antonowicz (2015, p. 44) reported that students
were informally invited to apply for tutor positions by “the coordinating
professor”. Such an informal approach often means selecting someone
locally available, such as PhD students (Bryson 2013). Walstad &
Becker (2010, p. 209) warned that this approach can result in appointing
international students having “limited English-language skills for
teaching”. This problem presents real risks for both teaching quality and
students’ learning outcomes (Lama & Joullié 2015). It is, therefore, not
surprising that the Australian Learning and Teaching Council (ALTC)
reported “quality assurance of sessional teaching in many institutions is
inadequate” (Percy & Beaumont 2008).
Having a less rigorous selection process could also put pressure on
the subsequent training programs for recruits. Regarding the training of
casual tutors, the overall picture is mixed. Percy & Beaumont (2008, p.
11) found the “support of sessional teachers is still largely ad hoc”, with
quality assurance measures for sessional teachers being inadequate and
having the ability to compromise institutional risk management
strategies. Harvey (2017) also stressed that despite a continued increase
and expected reliance on casual staff into the future, there is no
systematic approach toward their academic training. On the other hand,
Knott et al. (2015) reckoned that training programs for sessional staff
are increasing worldwide. One example is the institution-wide tutor
training program at the University of Queensland (Matthews, Duck &
Bartle 2017).
With the student experience being an imperative for many Australian
universities in the current educational environment (Australian
Government 2016), there has been a shift towards an active rather than
passive approach to teaching and learning. The literature provides us
insights as to what requirements might be needed in active learning
tutorials. For example, essential features of good teaching require
abilities such as being able to stand in the shoes of students (Ramsden
2003), enthusiasm, providing timely, consistent and relevant feedback,
while actively promoting collaborative learning and sharing
experiences with peers (Duarte 2013). Accordingly, the role of tutors
has recently changed from being a transmitter of knowledge to one that
offers guidance and facilitates learning, particularly for problem-based
learning approaches (Azer 2005; and Dolmans, De Grave, Wolfhagen
& Van Der Vleuten 2005). Kane, Taylor, Tyler & Wooten (2011) found
that classroom behaviours are useful parameters to identify the practices
of effective teachers. This suggests that appointing tutors exhibiting
these behaviours would support the student learning experience through
harnessing tutors’ motivation and enthusiasm (Sutherland 2009). A
selection process based on the principles of effective teaching and
learning practices would therefore be an improvement on the informal
practices.
This paper aims to fill this void around tutor recruitment by
expanding on the preliminary analysis conducted by Sherwood &
Littleboy (2016).
3. THE TUTOR SELECTION AND EVALUATION PROCESS
In this section we describe the old tutor selection process at UQ, the
new process, and provide a preliminary qualitative evaluation of the
new approach.
(a) The Old Process
Prior to 2013, tutor applications at UQ were initially filtered using
applicants’ overall GPA scores for their undergraduate studies. The
grading scale for courses at UQ ranges from 1 to 7, with 4 being a
passing grade, a grade of 6 designated as a “Distinction” grade, and a
grade of 7 designated as a “High Distinction” grade. Students with a
GPA of between 6 and 7 would automatically be shortlisted for an
interview. Doctoral and fourth year Honours students were also
automatically shortlisted because of the demanding academic entry
requirements for these programs. This process typically identified about
half the number of tutors required. A second stage was then used to look
for tutors for specific economics courses, especially large, first year
courses. Applicants who had a grade of 6 or 7 in these courses were
then also shortlisted for interview. Using this two-stage approach,
typically 50 to 60 applicants were selected for interview each year.
During a 15 minute interview, applicants were then asked standard
questions such as “Why do you want to be a tutor?”. Almost every
applicant passed this benign interview process and was offered a
position. In effect, the selection had been determined largely by
applicants’ overall GPAs or grades in specific courses.
(b) The New Regime
Since 2013, a new tutor selection approach has been implemented. The
new screening process reduced the overall GPA and course specific
GPA from 6 to 5.5. This had the effect of increasing the pool of potential
applicants. Since 2013, around 120 students apply each year and up to
90 of these are shortlisted for interview. To test the ability of applicants
to deliver teaching which the UQ Economics Department regards as
effective and which is regularly evaluated using surveys completed by
students, a 20-minute group interview was designed.
Each group interview is conducted in the following way. Three
applicants are randomly assigned to a group, but they do not know who
will be in their group until the interview. At the interview, applicants
are required to collaboratively work together to create a tutorial
question in 10 minutes. Their tutorial question must be based on a
newspaper article assigned to them on the spot. As many as nine articles
are used to reduce the chances of applicants in later interviews knowing
what articles they will be given. Examples of article titles include:
“Hospital parking fees enough to make you sick” (Mickelburough
2012); and “Easter holidays to deliver fuel price hikes for motorists”
(Kelly 2012). In essence, the applicants need to apply economic
theories to dissect daily life events, to communicate their ideas to each
other, and to work together to complete the task (writing a tutorial
question). As such, this process aims to identify natural collaborators
and facilitators who can work individually and as part of a team. During
the interview, a chief academic interviewer and several observers
(typically two other academic staff and one administrative staff) are
present. In the last 10 minutes of the interview, applicants individually
answer questions from the chief academic interviewer.
Each staff member scores each applicant against five criteria: (i)
appropriateness of their questions and answers; (ii) communication
skills; (iii) interpersonal skills; (iv) provides evidence of encouraging
student participation; and (v) potential for being a tutor. These criteria
are closely related to the aspects on which tutors are evaluated by their
prospective students. Each criterion is scored out of 5, with a total score
independently arrived at by each observer out of 25 for each applicant.
The observers and chief interviewer then discuss all three applicants for
about five minutes, with the chief interviewer determining a final score.
Typically, a minimum score of 21 is required for an applicant to be
offered a position. The group setting in itself is not a competition
because it is possible for all applicants in the same group to obtain a
high score. Using this approach, 40 to 50 applicants (out of about 90)
are appointed as new tutors.
This new selection process has resulted in some applicants with high
(to very high) GPA scores failing to be appointed. This contrasts to the
high probability that such applicants would have been appointed prior
to 2013. Further, some applicants with slightly lower GPA scores who
might not have been selected before 2013, are now being recruited. As
a case in point, less Ph.D. students have been appointed under the new
selection process compared to the old process.
(c)Preliminary Feedback on the New Regime
Each year, observers during the interviews have been invited to provide
written feedback on the process. Examples of this feedback include the
following:
The way it is structured provides for an equitable playing field for all
involved which is so important in selecting tutors.
Teaching and Learning Awards and Grants Officer, 2014
I thought by running these interviews three students at a time was a great
way to assess the strengths of the students in a way that could not have been
done by their grades alone or even a one-on-one interview.
Senior Lecturer, Business School, 2015
I found the format provided valuable insight into the candidates which is
difficult to achieve through the typical interview process of questions and
answers. I was able to assess the candidate’s skills across a number of areas
including time management, leadership, teamwork, and communication.
Human Resource Staff, 2015
Feedback from the tutors appointed under the new process indicates
that 70 percent view the process favourably. These tutors have indicated
that the process allowed them to adequately demonstrate their
knowledge, personality and abilities. The materials relating to the
interview and associated training processes, have been shared with
colleagues at two other Australian Universities. Their feedback notes
that:
We are following most of your interview strategy, with only a few changes.
University of New South Wales, 2016
I have put forward the proposal for our school to start with the initial tutor
training workshop and it has been very positively received. This could put
us on track to build towards a more comprehensive program such as what
you have established at UQ.
Royal Melbourne Institute of Technology, 2016
This qualitative feedback complements the quantitative analysis
presented below in that it reflects an assessment of the recruitment
process by tutors and staff while the quantitative analysis reflects
evaluation of the process by students. Furthermore, to the extent that
both the qualitative and quantitative evidence point to the same
conclusions, they reinforce and validate each other.
4. METHODOLOGY AND DATA
Performance of tutors selected under both the old and the new regimes
was measured using data from student feedback surveys each semester.
In each semester, we collect tutor evaluation results on a tutor-course
basis. Tutorial class sizes are typically capped at 25 students. When
evaluating tutors at the end of each semester, the average number of
student responses for a tutor are approximately 12 for both regimes.
Beginning from 2010 onwards, the evaluation form asked students to
rate tutors by expressing a level of agreement with the propositions
outlined in Table 1.
For each proposition, students were able to select a level of
agreement on a 5-point scale, where 1 indicated strong disagreement
and 5 indicated strong agreement. Thus, the higher the score, the better
the evaluation. The scores on these questions are designed to measure
students’ perceptions of a tutor’s performance. Besides these eight
questions, students also have the opportunity to provide written
comments.
Question Number Question Wording
Q2 The tutor communicated clearly.
Q3 The tutor was approachable.
Q4 The tutor inspired me to learn.
Q5 The tutor encouraged student input.
Q6 The tutor treated students with respect.
Q7 The tutor gave helpful advice.
Q8 Overall, how would you rate this tutor?
We do not engage with the debate about the efficacy of student
evaluations, which is a subject in its own right (see, for example, Biggs
(2011) and Nulty (2008)). We simply make the assumption that student
evaluations of tutors are a valid and reliable source of data to evidence
teaching quality. The evaluation process has been tested on multiple
occasions at UQ to ensure that responses to evaluation propositions are
not too highly correlated with each other and are reliable in that they
have reproducible outcomes across class size, course levels, locations
and modes of teaching. The tutor survey from 2010 to 2014 was
administered via paper surveys. From 2015 onwards, this was switched
to an online format via single use Quick Response (QR) codes.
However, the response propositions remained the same and tutors were
still required to hand out QR codes to students for the surveys to be
completed in class. Kordts-Freudinger & Geithner (2013) argue that the
evaluation situation (in-class versus after-class) has more of an impact
on survey results than evaluation mode (paper versus online). We find
no significant shifts in the data from 2014 to 2015 and, therefore, we
assume that the change in survey administration method did not impact
the results of the present study.
For evaluation purposes, our dataset includes only tutors who had
completed UQ courses as students in previous years. This included
current undergraduate students, course work masters students, and
Ph.D. students. The majority were undergraduate and coursework
masters students, with only two out of 231 tutors evaluated being Ph.D.
students.
We only used evaluation data for each tutor’s first semester of
teaching. The School of Economics monitored tutors’ performances
during each semester and only reappointed those with good evaluations
in subsequent semesters. So, selection bias would likely be introduced
if incumbent tutors were included in the analysis. Data were de-
identified and ethics clearance was granted for this research.
It should also be stressed that we focused only on the impact of the
new tutor selection process on the student learning experience and not
on student academic performance because data to control for factors
affecting academic performance other than the selection process, such
as, for example, lecture and class attendance (Stanca 2006), were not
available.
Using data from these student surveys we regressed tutor performance
(as measured by student satisfaction on these surveys) against a number
of explanatory variables. Specifically, the dependent variable was the
logarithmic value of student responses to Questions 1 to 8 on the
questionnaire and the explanatory variables included: a dummy variable
(NEW ) set equal to one if the tutor was selected under the new regime,
and zero if the tutor was selected under the old regime; the log value of
a tutor’s GPA score at the time of interview (ln(GPA)); and the log
value of a tutor’s interview score under the new selection regime
(ln(Score)) - there were no interview scores under the old regime). In
addition, a set of control variables were included to differentiate tutors
teaching in first, second, or third year undergraduate courses, or in
master’s courses. These variables accounted for the possibility that tutor
evaluation results may vary across courses of different levels. The
analysis was conducted using cross-sectional ordinary least square
(OLS) regression.
We did not include tutor characteristics such as gender or program of
study. This was because if the selection process successfully identified
applicants with ‘good’ teaching characteristics (for example, females
because women are hypothetically better communicators), then
controlling for things such as gender would lead to an under-estimation
of the effects associated with the new selection process.
Our dataset had 231 observations, which span across two years of the
old regime (2011 and 2012) and four years of the new regime (2013-
2016). With 97 percent of the tutors only tutoring in one course, 231
observations can be considered to represent 231 tutors. Of these tutors,
64 percent were selected under the new regime and the remaining 36
percent under the old regime. These tutors were deployed amongst six
first year, six second year, and eight third year undergraduate courses,
and 13 master’s courses.
Table 2 provides summary statistics of the key variables used in the
regression. GPA scores for the 231 tutors were very high, indicated by
a mean value of 6.14 and a median value of 6.20 on a 7-point scale. Yet
not all tutors were academic ‘superstars’ as there were instances where
tutors had a GPA score between about 4.5 and 5 under the new regime.
Tutors selected under the two regimes had average GPA scores that
were almost identical at 6.16 for the old and 6.12 for the new. A t-test
of this difference indicated a lack of statistical significance with p =
0.57. Furthermore, tutors under the two regimes had almost the same
GPA score spread (as measured by the standard deviation to mean
ratio). Interview scores under the new regime had a rather narrow
distribution with high average scores (22.5 out of a maximum possible
25). This was, of course, to be expected since this score was used as a
selection criterion.
A key point of comparison between the old and new selection
regimes was indicated by the median Q8 score (a measure of overall
tutor effectiveness) which increased from 4.39 under the old regime to
4.54 under the new regime. In addition, the proportion of tutors with
Mean Median Standard
GPA (Old Regime) 6.16 6.23 0.51 4.86 7.00 84
GPA (New Regime) 6.12 6.15 0.56 4.45 7.00 147
Interview Score
(New Regime) 22.46 22.50 1.73 17.00 25.00 152
scores below 4 on Q8, considered as a sign of underperformance, fell
noticeably from 19.1 percent under the old regime to only 6.6 percent
under the new regime. Similarly, the distribution of scores on Q2 (a
measure of tutor communication skills) increased from 4.31 under the
old regime to 4.51 under the new regime. The proportion of
underperforming tutors based on Q2 also fell from 23.6 percent under
the old regime to 11.2 percent under the new regime.
4. RESULTS
Under the old selection process, tutors were chosen on the basis of their
GPA scores. We, therefore, first examined whether GPA was a good
predictor of tutor performance as measured by student feedback. To that
end, we regressed each of the 8 evaluation outcomes on the student
survey against ln (GPA) for tutors appointed under the old regime. The
results are reported in Table 3. In all regressions reported in this and
other tables, course level dummies are included as controls. The dummy
for first year undergraduate courses is not included because tutors of
first year undergraduate courses constitute a baseline group. For
example, for Q1, after controlling for ln (GPA), the score for tutors of
second year undergraduate courses is 0.057 lower than that for the
baseline group on average, though the difference is not statistically
significant.
Table 3 indicates that ln (GPA) is only statistically significant for Q1
(“was well prepared”) at p < 0.05. This result shows that a one percent
increase in the GPA of a tutor is on average associated with a 0.743 (out
of 5) point increase in the evaluation score on Q1 of the student
feedback survey. This effect is large in magnitude and the largest
amongst all eight evaluation items. The magnitudes of the coefficients
for all other regressions are in general quite sizable as well, with the
exception of that for Q2. But none of these coefficients are statistically
significant at the standard levels. This may be due to an insufficient
number of observations and associated large standard errors.
Regardless of this significance issue, the result for Q2 (“communicated
clearly”) is revealing in that its coefficient is the only one that has a
negative sign. These results suggest that while tutors with stronger
academic background are likely to be more competent with course
material (Q1), they are not necessarily better communicators (Q2).
Table 3: Regression Results for Old Tutor Selection Regime.
Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8
Log 0.743** -0.058 0.560 0.438 0.490 0.339 0.301 0.648
GPA (0.367) (0.68 5) (0.445) (0.621) (0.513) (0.245) (0.632) (0.594)
2nd year Course
3nd year Course
Masters
Course
Const 3.188*** 4.435*** 3.446*** 3.150*** 3.471*** 4.063*** 3.766*** 3.178***
R2 0.070 0.028 0.041 0.041 0.076 0.065 0.020 0.019
No. Obs 84
Note: Figures in parentheses are robust standard errors. All regressions include course level
dummies as controls. ***, **, * denote significance at the 1%, 5% and 10% levels respectively.
Although the coefficients of seven out of the eight evaluation
outcomes were not significant at the standard levels and one of them
was negative, it may be unwarranted to conclude that GPA has little
value in the tutor selection process. This is because our sample does not
include those applicants that failed the selection. Therefore, a more
cautious interpretation of the results is that, among those recruited as
tutors under the old regime, there is insufficient evidence that those with
higher GPAs perform better on most evaluation criteria.
Next, we turned to the key question of whether the new selection
process successfully identifies ‘better’ tutors in the sense that students
reported higher levels of satisfaction and more positive learning
experiences under these teachers compared to the old regime. To
answer this question, we considered observations from both the old and
Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8
Old 0.132*** 0.204*** 0.158*** 0.255*** 0.186*** 0.134*** 0.215*** 0.185***
(0.039) (0.061) (0.042) (0.061) (0.048) (0.026) (0.053) (0.054)
2nd year 0.070* 0.035 0.096** 0.105 -0.004 0.017 0.040 0.077
(0.041) (0.069) (0.042) (0.064) (0.052) (0.027) (0.056) (0.059)
3rd year -0.084 -0.100 -0.032 -0.016 -0.216** -0.079 -0.077 -0.008
(0.115) (0.136) (0.093) (0.117) (0.102) (0.070) (0.119) (0.125)
Masters 0.030 0.120* 0.101** 0.341*** 0.065 -0.005 0.152*** 0.118**
(0.048) (0.064) (0.048) (0.066) (0.055) (0.035) (0.056) (0.057)
Constant 4.484*** 4.237*** 4.439*** 3.878*** 4.304*** 4.644*** 4.275*** 4.305***
(0.038) (0.061) (0.044) (0.064) (0.048) (0.025) (0.054) (0.056)
R2s 0.0641 0.065 0.099 0.158 0.100 0.128 0.110 0.074
New 0.136*** 0.204*** 0.159*** 0.257*** 0.187*** 0.135*** 0.217*** 0.188***
(0.038) (0.061) (0.041) (0.061) (0.048) (0.026) (0.052) (0.053)
Ln(GPA) 0.499** 0.078 0.303 0.236 0.270 0.252** 0.228 0.493*
(0.207) (0.293) (0.195) (0.289) (0.223) (0.118) (0.264) (0.257)
2nd year 0.060 0.034 0.090** 0.101 -0.009 0.012 0.035 0.067
(0.041) (0.070) (0.043) (0.065) (0.052) (0.027) (0.056) (0.059)
3rd year -0.103 -0.103 -0.043 -0.025 -0.226** -0.088 -0.086 -0.026
(0.114) (0.137) (0.092) (0.116) (0.100) (0.069) (0.119) (0.124)
Masters 0.021 0.118* 0.096** 0.337*** 0.060 -0.009 0.148*** 0.110*
(0.047) (0.065) (0.048) (0.066) (0.055) (0.034) (0.057) (0.057)
Constant 3.585*** 4.095*** 3.893*** 3.453*** 3.818*** 4.190*** 3.862*** 3.416***
(0.374) (0.512) (0.352) (0.522) (0.407) (0.212) (0.469) (0.461)
R2s 0.090 0.066 0.109 0.160 0.106 0.143 0.114 0.089
#obs = 231
Note: Figures in parentheses are robust standard errors. All regressions include course
level dummies as controls. ***, **, * denote significance at the 1%, 5% and 10% levels
respectively.
new regimes, and the results are reported in Table 4. The upper panel
of Table 4 shows the results obtained from regressing each of the
evaluation outcomes against the dummy variable NEW, which
identifies whether a tutor was selected under the new regime. The
coefficient for the variable is positive and significant at p < 0.01 for all
eight questions. This implies that tutors selected under the new regime
performed better across all eight questions in the evaluation survey than
their counterparts under the old regime. The coefficient for Q2 suggests
that the communication skills of tutors selected under the new regime
were about 20.4 percent better. These results are in line with the design
of the new regime to better identify communicators and facilitators. An
unexpected result, however, was that the coefficient for Q4 (“inspired
me to learn”) was the largest amongst all eight questions at p < 0.01.
This is interesting because it is not an easy task to identify an inspiring
Table 5: Results for New Regime Conditioned on Interview Score.
Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8
Ln(Score) 0.815*** 1.650*** 0.606*** 0.921*** 0.460* 0.270 0.986*** 1.177***
(0.295) (0.357) (0.222) (0.334) (0.260) (0.164) (0.275) (0.291)
2nd year 0.143*** 0.189*** 0.099** 0.201*** 0.002 0.045 0.096* 0.141**
(0.049) (0.066) (0.045) (0.064) (0.058) (0.029) (0.050) (0.056)
3rd year 0.007 -0.002 0.016 0.092 -0.087 -0.044 0.027 0.091
(0.134) (0.146) (0.131) (0.157) (0.101) (0.072) (0.152) (0.137)
Masters 0.090 0.268*** 0.164*** 0.429*** 0.206*** 0.060 0.215*** 0.215***
(0.061) (0.079) (0.052) (0.071) (0.057) (0.041) (0.062) (0.068)
Constant 2.051** -0.755 2.697*** 1.232 3.031*** 3.921*** 1.395 0.795
(0.928) (1.123) (0.700) (1.043) (0.813) (0.516) (0.867) (0.913)
R2s 0.094 0.172 0.102 0.200 0.092 0.045 0.132 0.141
No. obs 152
(0.301) (0.362) (0.228) (0.340) (0.270) (0.170) (0.282) (0.294)
Ln(GPA) 0.434* 0.248 0.194 0.179 0.099 0.21 0.242 0.486*
(0.252) (0.294) (0.199) (0.312) (0.227) (0.128) (0.256) (0.266)
2nd year 0.131*** 0.167** 0.089* 0.184*** 0.003 0.040 0.079 0.120**
(0.050) (0.068) (0.047) (0.066) (0.061) (0.030) (0.051) (0.057)
3rd year -0.022 -0.015 0.004 0.085 -0.091 -0.058 0.012 0.059
(0.135) (0.150) (0.131) (0.158) (0.101) (0.070) (0.153) (0.139)
Masters 0.109* 0.297*** 0.192*** 0.459*** 0.217*** 0.065 0.236 0.232***
(0.061) (0.082) (0.052) (0.074) (0.060) (0.043) (0.064) (0.070)
Constant 0.986 -1.539 2.189** 0.602 2.781*** 3.407*** 0.671*** -0.461
(1.143) (1.299) (0.844) (1.251) (0.987) (0.618) (1.097) (1.122)
R2s 0.125 0.191 0.132 0.222 0.099 0.067 0.159 0.181
No.obs 147
Note: Figures in parentheses are robust standard errors. All regressions include course
level dummies as controls. ***, **, * denote significance at the 1%, 5% and 10% level
respectively.
teacher within a brief 20-minute interview. Yet the results show that
under the new regime, which places emphasis on an applicant’s ability
to collaborate with others, more inspiring teachers appear to have been
identified. Quantitatively, the results for Q8 suggest that tutors selected
under the new regime were on average 18.5 percent ‘better’ than their
old regime counterparts on this characteristic.
The lower panel of Table 4 reports the results of tests that included
ln (GPA) as an explanatory variable. Both the qualitative and
quantitative results regarding the NEW variable remained the same in
this specification of the regressions, suggesting that the effectiveness of
the new regime was robust. More importantly, the results indicate that
the new process is able to identify attributes of good tutors about which
GPA data is not informative.
Under the new regime, prospective tutors are given a score based on
the communication and collaboration skills they demonstrate during the
interview. This information allowed us to further test the effectiveness
of the new selection process using a subsample of tutors from the new
regime only. In particular, if the new process is working effectively,
tutors with a higher interview score should have higher evaluation
scores. The results from testing this hypothesis are reported in Table 5.
The upper panel of Table 5 reports results from regressing each of
the eight valuation outcomes against ln (Score). Coefficients of this
variable for all but Q6 were significant at the standard levels, with 6 out
of 8 evaluation outcomes significant at p < 0.01. The effect of ln (Score)
on performance against Q2 was particularly strong and large. In the
lower panel, we report results when we further controlled for ln (GPA).
These results are largely the same with performance against Q2
remaining higher than that against all other questions. Overall, the
results in Table 5 confirm that the design of the new selection process
has been very effective in identifying good tutors who receive very
positive evaluations from their students.
Lastly, we used GPA scores for only economics courses instead of
overall GPA score in the estimations. All our previous conclusions
remained the same and we do not report these results.
5. DISCUSSION AND CONCLUSION
The selection of tutors in a higher education setting has attracted little
research attention. Unlike the very formal, in depth approach taken to
appointing faculty members, the selection of tutors has tended to be
more informal, generally identifying tutors who are immediately and
locally available. This paper has investigated how the selection of tutors
might be done more rigorously. It has presented student evaluation data
of tutor performance under two tutor recruitment methods in the School
of Economics at UQ. The first method largely relied on filtering
applicants using only GPA scores, with the second method relaxing the
GPA requirement slightly and supplementing selection with a new
group interview process. The new group interview approach sought to
identify additional tutor attributes that could further enhance students’
learning experiences. By collecting and analysing data from the two
regimes, we aimed to answer several research questions listed in the
introduction.
Is academic performance a useful criterion for selecting tutors?
The answer to this question is a qualified ‘yes’. Results from Table 3
do not provide a definite answer to the question. However, the results
presented in Tables 4 and 5 indicate that GPA was still informative even
under the new group interview selection process. This suggests that the
new process could be further strengthened by placing more emphasis
on both the applicants’ GPA and communication skills. To some extent,
this is already happening since applicants with a low GPA are unlikely
to be shortlisted for interview.
Is academic performance a sufficient criterion for selecting tutors?
The answer to this question is a definite ‘no’ based on the results from
Tables 4 and 5.
What importance should be placed on a tutor’s communication and
collaboration skills compared to their academic performance in the
selection process?
The results from Tables 4 and 5 clearly demonstrate that
communication and collaboration skills are as important, if not more so,
than academic performance for selecting tutors.
Did the new group interview selection process result in higher tutor
evaluations from students, suggesting an enhanced learning
experience?
The answer to this question is a definite ‘yes’. The results from Tables
4 and 5 provide encouraging evidence that the new selection process
has been very successful. The aim of the new process was to identify
‘better’ tutors, in the sense that they improve the student learning
experience, as well as see a reduction in the number of underperforming
tutors, defined as those having a Q8 score below a threshold of 4 out of
5. Under the old scheme most tutors in the sample could have been
classified as “good” with a median score for Q8 of 4.39, almost 10
percent above the threshold. In comparison, under the new scheme,
tutors could be considered to have been “excellent” with a median score
for Q8 equal to 4.54.
If the new selection process does improve students’ learning
experience, is this improvement significant to justify the cost of
implementing it?
On the cost side, the new system requires about the same amount of
time to interview applicants as the old system, but requires more input
time from staff. On the benefit side, an immediate demonstrable reward
is an improvement in the student experience evidenced by increases in
the mean and median student evaluation scores. But there are also
implicit down-stream benefits, including: (1) reducing the need for
students to consult directly with the course instructor; (2) lessening the
chance of students failing their courses; (3) an extension of benefit (2)
in the form of a reduced need to organise supplementary exams or for
students to repeat courses; (4) decreasing the incidence of student
complaints, and thus, resources needed to handle these complaints; and
(5) lowering tutor training costs by selecting applicants that have
greater potential, and thus, require less training and support. Although
our analysis does not yield a dollar value for the benefits from the new
process (and the costs for that matter), our view from the experience of
operating it is that its combined benefits easily outweigh its additional
costs.
A limitation of the current study is that student evaluations only
indicate students’ learning experiences, not their learning outcomes.
Therefore, a natural extension of the study would be to examine
whether and how the new tutor selection process might impact students’
learning outcomes such as their overall scores or grades for particular
courses. This extension is feasible given students’ academic outcomes
are readily available. Another possible extension is to invite a panel of
educational experts to come and observe the selection process and then
gather their feedback through a systematic evaluation survey. A
qualitative analysis of their systematic feedback would be an
improvement on the anecdotal evidence presented in Section 3.
To close the discussion, it is worth noting that the new process is not
discipline specific. Although our analysis is based on the data from an
economics department, the emphasis on communication and
collaboration skills is universal. This suggests that in selecting tutors
for any discipline, not only should their academic performance be
evaluated, but also their skills to work in a collaborative environment
using a realistic student learning setting. In doing so, a combination of
both strong academic ability, communication and collaborative
working skills can lead to improved chances of identifying high
performing tutor applicants, thereby helping to further enhance
students’ learning experiences and learning outcomes. Likewise,
although the majority of tutors in our analysis were undergraduate-
student tutors, the lesson is not confined to their recruitment.
Communication and interpersonal skills in conducting tutorials or
teaching in general, is equally important for graduate-student tutors or
full-time tutors and, therefore, their recruitment as well.
REFERENCES
in Australian Higher Education (TRIM Ref No. ED16/000060), View at
https://docs.education.gov.au/documents/driving-innovation-fairness-
and-excellence-australian-education.
Azer, S.A. (2005) “Challenges Facing PBL Tutors: 12 Tips for Successful
Group Facilitation”, Medical Teacher, 27 (8), pp.676-681.
Baderin, M.A. (2005) “Towards Improving Students’ Attendance and
Quality of Undergraduate Tutorials: A Case Study on Law”, Teaching in
Higher Education, 10 (1), pp.99-116.
Benchmarking Leadership and Advancement of Standards for Sessional
Teaching (2013) The Sessional Staff Standards Framework, View at
http://www.blasst.edu.au/docs/BLASST_ framework_WEB.pdf.
Biggs, J.B. (2011) Teaching for Quality Learning at University: What the
Student Does, Maidenhead, UK: McGraw-Hill Education.
Bryson, C. (2013) “Supporting Sessional Teaching Staff in the UK – To
What Extent is There Real Progress?” Journal of University Teaching &
Learning Practice, 10 (3), pp.1-17.
Coates, H., Dobson, I. R., Goedegebuure, L., and Meek, L. (2009)
“Australia's Casual Approach to its Academic Teaching Workforce”,
People and Place, 17 (4), pp.47-54.
Dolmans, D.H., De Grave, W., Wolfhagen, I. H. and Van Der Vleuten, C. P.
(2005) “Problembased Learning: Future Challenges for Educational
Practice and Research”, Medical Education, 39 (7), pp.732-741.
Duarte, F.P. (2013) “Conceptions of Good Teaching by Good Teachers:
Case Studies from an Australian University”, Journal of University
Teaching & Learning Practice, 10 (1), pp.1-15.
Fumasoli, T., Goastellec, G., and Kehm, B. M. (eds.) (2015) Academic Work
and Careers in Europe: Trends, Challenges, Perspectives, London, UK:
Springer International Publishing.
Gilbert, A. (2017) “Using Activity Theory to Inform Sessional Teacher
Development: What Lessons Can Be learned from Tutor Training
Models?”, International Journal for Academic Development, 22 (1),
pp.56-69.
Harvey, M. (2017) “Quality Learning and Teaching with Sessional Staff:
Systematising Good Practice for Academic Development”, International
Journal for Academic Development, 22 (1), pp.1-6.
Harvey, M., Fraser, S. and Bowes, J. (2005) “Quality Teaching and Sessional
Staff”, Paper presented at the 28th Annual Higher Education Research
and Development Society of Australasia Conference, 3-6 July, Sydney,
Australia.
Kane, T.J., Taylor, E.S., Tyler, J.H. and Wooten, A.J. (2011) “Identifying
Effective Classroom Practices Using Student Achievement Data”, Journal
of Human Resources, 46 (3), pp.587-613.
Kelly, A. (2012) “Easter Holidays to Deliver Fuel Price Hikes for Motorists”,
The Sunday Mail, April 1.
Kordts-Freudinger, R. and Geithner, E. (2013) “When Mode Does Not
Matter: Evaluation in Class versus out of Class”, Educational Research
and Evaluation, 19 (7), pp.605-614.
Knott, G., Crane, L., Heslop, I. and Glass, B.D. (2015) “Training and
Support of Sessional Staff to Improve Quality of Teaching and Learning
at Universities”, American Journal of Pharmaceutical Education, 79 (5),
pp.1-8.
Kreber, C. (2007) “What‘s It Really All About? The Scholarship of Teaching
and Learning as an Authentic Practice”, International Journal for the
Scholarship of Teaching and Learning, 1 (1), p.3.
Kwiek, M. and Antonowicz, D. (2015) “The Changing Paths in Academic
Careers in European Universities: Minor Steps and Major Milestones”, in
Fumasoli, T., Goastellec, G. and Kehm, B.M. (eds.), Academic Work
and Careers in Europe: Trends, Challenges, Perspectives, London, UK:
Springer International Publishing, pp. 41-68.
Lama, T. and Joullié, J.E. (2015) “Casualization of Academics in the
Australian Higher Education: Is Teaching Quality at Risk?”, Research in
Higher Education, 28, pp.1-11.
Matthews, K. E., Duck, J. M. and Bartle, E. (2017) “Sustaining Institution-
wide Induction for Sessional Staff in a Research-intensive University: The
Strength of Shared Ownership”, International Journal for Academic
Development, 22 (1), pp.43-55.
Mickelburough, P. (2012) “Hospital Parking Fees Enough to Make You
Sick”, Herald Sun, March 17.
Milliken, J. and Barnes, L.P. (2002) “Teaching and Technology in Higher
Education: Student Perceptions and Personal Reflections”, Computers
and Education, 39 (3), pp.223-235.
Muzaka, V. (2009) “The Niche of Graduate Teaching Assistants (GTAs):
Perceptions and Reflections”, Teaching in Higher Education, 14 (1), pp.1-
12.
Nulty, D.D. (2008) “The Adequacy of Response Rates to Online and Paper
Surveys: What Can be Done?”, Assessment & Evaluation in Higher
Education, 33 (3), pp.301-314.
Park, C. (2004) “The Graduate Teaching Assistant (GTA): Lessons from
North American Experience”, Teaching in Higher Education, 9 (3),
pp.349-361.
Park, C. and Ramos, M. (2002) “The Donkey in the Department? Insights
into the Graduate Teaching Assistant (GTA) Experience in the UK”,
Journal of Graduate Education, 3 (2), pp.47-53.
Percy, A. and Beaumont, R. (2008) “The Casualisation of Teaching and the
Subject at Risk”, Studies in Continuing Education, 30 (2), pp.145-157.
Percy, A., Scoufis, M., Parry, S., Goody, A. and Hicks, M. (2008) The RED
Report, Recognition - Enhancement - Development: The Contribution of
Sessional Teachers to Higher Education, Sydney: Australian Learning
and Teaching Council.
Ramsden, P. (2003) Learning to Teach in Higher Education, 2nd edition,
New York, NY: Routledge Falmer.
Sherwood, C.W. and Littleboy, B. (2016) “Selecting Sessional Tutors: A
New and Effective Process”, Australasian Journal of Economics
Education, 13 (1), pp.30-48.
Stanca, L. (2006) “The Effects of Attendance on Academic Performance:
Panel Data Evidence for Introductory Microeconomics”, Journal of
Economic Education, 37 (3), pp.251-266.
Sutherland, K.A. (2009) “Nurturing Undergraduate Tutors’ Role in the
University Teaching Community”, Mentoring & Tutoring: Partnership in
Learning, 17 (2), pp.147-164.
Sessional Tutors in a New Zealand University”, Journal of University
Teaching & Learning Practice, 10 (3), pp.1-11.
Walstad, W.B. and Becker, W.E. (2010) “Preparing Graduate Students in
Economics for Teaching: Survey Findings and Recommendations”,
Journal of Economic Education, 41 (2), pp.202-210.

TUTOR SELECTION PROCESSES AND THE STUDENT LEARNING …

Documents

Transcript of TUTOR SELECTION PROCESSES AND THE STUDENT LEARNING …