Postgraduate Research Experience (PREQ) Report...Figure 4: Levels of satisfaction for JCU recent...
Transcript of Postgraduate Research Experience (PREQ) Report...Figure 4: Levels of satisfaction for JCU recent...
1
Postgraduate Research Experience (PREQ) Background Report
2012
Prepared April 2012 James Cook University
Office of
Corporate
Planning &
Performance In collaboration with the
Graduate
Research
School
2
Introduction
The Survey Instrument
The Postgraduate Research Experience Questionnaire (PREQ) is part of the nation-wide
Australian Graduate Survey (AGS) owned by Graduate Careers Australia (GCA), and
administered annually by JCU’s Careers Advisory Centre and the Teaching and Learning
Directorate (TLD). The data are stored and managed by Information, Technology and
Resources (IT&R).
The PREQ component of the survey instrument measures the student satisfaction with their
research experience, taken after students have completed their postgraduate research degrees.
The 2012 PREQ survey evaluates students’ learning experiences on seven different scales,
they are:
Supervision;
Intellectual Climate;
Examination;
Infrastructure;
Skills
Goals and Expectations; and
Overall Satisfaction
The PREQ survey includes 28 five-point Likert scale questions, and two open ended
questions. The open ended questions request responses on ‘Best Aspects’ and ‘Needs
Improvement’.
In 2012, the Graduate Research School (GRS) administered an internal postgraduate
satisfaction survey. This survey collects the same satisfaction data as the PREQ and will be
used to gauge any shifts in student satisfaction within the current cohort. The internal survey
also incorporates questions relating to access to library resources, timely feedback on work,
fortnightly meetings with principal supervisor and if the research takes advantage of JCU’s
tropical location.
Caution is advised in interpreting the PREQ results as many factors can contribute to the
results, including response rate, method of data collection, and other broader University
factors. However, all attempts have been made to ensure data integrity and statistical rigour.
Corporate Planning and Performance would also value feedback about the content of this
report and any suggestions should be forwarded to [email protected]
3
Overview of the Report Data
This report includes analyses of the responses of JCU graduands to the national PREQ survey
for the years 2007 to 2012, and includes sector benchmark analysis for 2012. A further
comparison of current student satisfaction with the PREQ survey (2010 and 2012) and the
sector result for 2012 is also presented within the report.
The year represents the year the survey was conducted, not the graduating year. For example,
2012 data was collected in early 2012 for students completing their awards in 2011.
Background
Data Sources
The data set is the survey results from the 2010-2012 PREQ, internal survey results from the
current HDR candidate cohort of 2012 and sector results published in the Postgraduate
Research Experience Report 2012 (Graduate Careers Australia).
Methodology
The twenty eight PREQ survey questions are designed to measure a student’s perception of
the learning community and conditions provided by the institution (Intellectual Climate
Scale), the accessibility and quality of the supervision (Supervision Scale), the extent of
generic analytical and communication skill development (Skill Development Scale), the
quality of the learning infrastructure (Infrastructure Scale), whether the examination process
was timely, fair and satisfactory (Thesis Examination Scale), the clarity of learning
structures, requirements and standards (Goals and Expectations) and the overall satisfaction
level of the degree (Overall Satisfaction) (Table 1). Each respondent is asked to express their
degree of acceptance on a five-point Likert scale ranging from ‘strongly disagree’ to
‘strongly agree’. While the data are captured at the degree level, this report combines the JCU
data at the Faculty and Campus level from the student system with historical records being
attributed to the current Faculty. The discipline groups are further classified based on the
Field of Education (FoE) categories.
The PREQ results are reported in Mean Precent Agreement (MAP) and calculated in the
following manner:
1. The ‘percentage agreement’ is calculated by recoding the responses 1, 2, and 3 as 0,
while the responses of 4 and 5 on the reporting scale are recoded as 1 for each
question within the scale. Each respondent obtains a ‘percentage agreement’ score for
each scale depending on the proportion of 1’s within the scale.
2. The Mean Agreement Percent (MAP) is the average ‘percentage agreement’ score
from all respondents within the scale of interest.
Because the JCU sample sizes for each survey year are small (typically < 50) a decision was
made in consultation with the Graduate Research School (GRS) to pool the results of all
survey years (2010-2012) and compare them to the sector results from 2012. This decision
4
was statistically validated as the year to year variation within the scales at JCU was found to
be statistically non significant.
The qualitative responses to the PREQ survey have been analysed using word clouds. A word
cloud is a list of commonly used words where the size of the word is correlated with the word
frequency in the respondents answer. The use of this technique nicely summarises the type of
descriptions used when answering the ‘Best Aspects’ and ‘Needs Improvement’ questions.
Table 1: PREQ scales and survey questions
Scale Questions Supervision Scale Questions Accessible PREQ01 Supervision was available when I needed it. Understand difficulties PREQ07 My supervisor(s) made a real effort to understand the
difficulties I faced. Additional information PREQ13 My supervisor(s) provided additional information relevant to
my topic. Topic guidance PREQ17 I was given good guidance in topic selection and refinement. Helpful feedback PREQ21 My supervisor(s) provided helpful feedback on my progress. Literature guidance PREQ24 I received good guidance in my literature search. Intellectual Climate Scale Questions Postgrad contact PREQ05 The department provided opportunities for social contact with
other postgraduate students. Part of community PREQ09 I was integrated into the department’s community. Involved in research PREQ16 The department provided opportunities for me to become
involved in the broader research culture. Seminar program PREQ22 A good seminar program for postgraduate students was
provided. Stimulating PREQ23 The research ambience in the department or faculty stimulated
my work. Thesis Examination Scale Questions Fair process PREQ02 The thesis examination process was fair. Satisfied process PREQ15 I was satisfied with the thesis examination process. Reasonable timeframe PREQ25 The examination of my thesis was completed in a reasonable
time. Skill Development Scale Questions Problem solving PREQ06 My research further developed my problem-solving skills. Develop ideas PREQ10 I learned to develop my ideas and present them in my written
work. Analytical skills PREQ14 My research sharpened my analytical skills. Planning skill PREQ20 Doing my research helped me to develop my ability to plan my
own work. Tackling unfamiliar
problems PREQ26 As a result of my research, I feel confident about tackling
unfamiliar problems. Infrastructure Scale Questions Access space PREQ03 I had access to a suitable working space. Technical support PREQ08 I had good access to the technical support I needed. Equipment PREQ12 I was able to organise good access to necessary equipment. Computing PREQ18 I had good access to computing facilities and services. Financial support PREQ27 There was appropriate financial support for research activities. Goals and Expectation Scale Questions Know standards PREQ04 I developed an understanding of the standard of work expected. Understand standards PREQ11 I understood the required standard for the thesis. Understand Requirements PREQ19 I understood the requirements of thesis examination.
Overall Satisfaction Indicator
5
Scale Questions Overall satisfaction PREQ28 Overall, I was satisfied with the quality of my higher degree
research experience. Further Questions Relating to Internal PREQ
Tropical Location NoPREQ01 My research takes advantage of the universities tropical
theme. Library Resources NoPREQ02 I have access to all library resources I need.
Supervision meetings NoPREQ03 I generally meet with my principal supervisor at least once
per fortnight. Timely feedback NoPREQ04 My supervisors provide timely feedback on my written work.
Report Structure
This report is hierarchical in nature taking a top down approach from the University to the
Faculty, Discipline and Demographic levels. The sections are outlined below:
Section A describes the trends at the University level from 2006 to 2012 and compares JCU
to the sector result for 2012.
Section B explores the difference in respondents’ attitudes within the Faculties and Fields of
Education and compares them to JCU, the broad sector and the sector Fields of Education.
Section C focuses on the results for each demographic group. In this case the survey
population is analysed by the qualification, attendance, gender, age group and mode of study.
These groups are then compared to their sector counterparts. This section also analyses the
open ended qualitative responses.
Section D compares the results of the current student satisfaction 2012 with the PREQ (2010-
2012) and sector results for 2012.
6
Section A – University Overview
University
Analysis of each scale compared to the sector across time reveals that JCU does not differ
from the sector. Further analysis reveals that the variation from year to year within the scales
at JCU does not vary significantly1 (Figure 1). Therefore, we are able to treat the data from
2010-2012 as one sample thereby increasing samples sizes at the Faculty, Discipline and
Demographic level.
Because the sector remains fairly constant across time and JCU’s responses vary
insignificantly, it is possible to take long term averages of JCU’s responses and compare
them to the sector at the different levels of interest. This approach has the effect of removing
insignificant year to year variation in order to uncover new information. Therefore, this report
combines the historical PREQ data and compares them to the latest published sector results
(2012) as a proxy of long term satisfaction within the sector.
1 Each survey year represents an independent sample, one from each population. We assume the ith population has a
normal distribution and the populations have the same standard deviation. In the event that one or all of these assumptions
are violated, non-parametric tests were used to draw conclusions.
7
Figure 1: Time series of JCU satisfaction levels (% satisfied y axis) for each of the PREQ
scales by year for recent graduands compared to the sector averages. The JCU
results are presented as means + standard errors. The Y axis on each graph is
% satisfied.
a. b.
c. d.
e. f.
g.
JCU
Sector
8
Figure 2: Levels of satisfaction for HDR candidates (% satisfied y axis) from JCU
(2010-2012) compared with the National PREQ results (2012) for recent
graduands. The JCU results are presented as means + standard errors.
The national mean is presented as the standard.
9
Section B – Faculty Overview
Faculty
In this section, each of the Faculties within JCU is compared to the overall University
average and the overall sector average for each scale.
The Faculty of Law, Business and Creative Arts has the fewest respondents with eighteen
students answering the survey since 2008, whereas the Faculty of Science and Engineering
had the most respondents with 145.
Figure 3: Levels of satisfaction of recent graduands (% satisfied y axis) from each
of the JCU Faculties (2010-2012) compared to the corresponding JCU and
National PREQ results (2012) for recent graduands. The JCU results for
each faculty are presented as means + standard errors. Significantly
different results occur when the national and overall JCU means are
outside the error bars.
a. b.
c. d.
10
Broad Field of Research
Figure 4: Levels of satisfaction for JCU recent graduands (2010-12) (% satisfied y
axis) compared to the corresponding National PREQ results (2012) by
Aggregated Broad Field of Research Categories. The JCU results for each
Field of Education are presented as means + standard errors. Significantly
different results occur when the national mean is outside the error bars.
\
a. b.
c. d.
e.
11
Section C – Demographic Overview
This section compares the following groups within JCU to their sector counterparts.
Figure 5: Levels of satisfaction (% satisfied y axis) for recent graduands from JCU
(2010-2012) compared with the corresponding national PREQ results
(2012) by level of qualification. Significantly different results occur when
the national mean is outside the error bars.
Figure 6: Levels of satisfaction (% satisfied y axis) for recent graduands from JCU
(2010-2012) compared with the corresponding national PREQ results
(2012) by Attendance Type. Significantly different results occur when the
national mean is outside the error bars.
a. b.
a. b.
12
Figure 7: Levels of satisfaction (% satisfied y axis) for recent graduands from JCU
(2010-2012) compared with the corresponding national PREQ results
(2012) by Mode of Study. Significantly different results occur when the
national mean is outside the error bars.
Figure 8: Levels of satisfaction (% satisfied y axis) for recent graduands from JCU
(2010-2012) compared with the corresponding national PREQ results
(2012) by of each gender. Significantly different results occur when the
national mean is outside the error bars.
a. b.
a. b.
c.
13
Figure 9: Levels of satisfaction (% satisfied y axis) for recent graduands from JCU
(2010-2012) compared with the corresponding national PREQ results
(2012) by Age Group. Significantly different results occur when the
national mean is outside the error bars.
Figure 10: Levels of satisfaction (% satisfied y axis) for recent graduands from JCU
(2010-2012) compared with the corresponding national PREQ results
(2012) by Resident Status. Significantly different results occur when the
national mean is outside the error bars.
a. b.
a. b.
14
Figure 11: Levels of satisfaction (% satisfied y axis) for recent graduands from JCU
(2010-2012) compared with the corresponding national PREQ results
(2012) by Language Background. Significantly different results occur
when the national mean is outside the error bars.
a. b.
15
Qualitative Analysis
The qualitative analysis is undertaken using the word cloud technology in conjunction with
reading through the responses to the qualitative questions.
Figure 12: Word Cloud depicting the best aspects of the course of study for JCU
graduands (2010-2012) . The size of the word is correlated with the word
frequency in the respondent’s answer.
Figure 13: Word Cloud depicting the worst aspects of the course of study for JCU
graduands (2010-12). The size of the word is correlated with the word
frequency in the respondent’s answer.
16
Section D – Internal Survey 2012
Comparison with internal surveys of current students
In addition to the PREQ, JCU has two internal methods of getting feedback from current
research students. See https://www.jcu.edu.au/graduate-research-school for the results
of recent surveys. It is important to note that the survey population is different from that
targeted by the PREQ.
The internal PREQ was conducted for the second time in 2012. This survey allows the JCU
internal results for current students to be benchmarked against the national results for
graduating students. The internal surveys are administered in even-numbered years, with
qualitative focus groups to be repeated in odd-numbered years.
Figure 14: Levels of satisfaction (% satisfied y axis) for HDR candidates
from JCU (2012) compared with the JCU and national PREQ results (2010-2012) for
recent graduands. The JCU results are presented as means + standard
errors.
17
Figure 15: Levels of satisfaction (% satisfied y axis) for JCU HDR candidates in 2012
and 2010 for each of the PREQ satisfaction scales. The JCU results are
presented as means + standard errors.
Figure 16: Levels of satisfaction (% satisfied y axis) for HDR candidates for each
faculty of JCU (2012) compared with: (1) JCU graduands (2010-2012) for
each faculty, and (2) the overall sector results for graduands (2010-12).
The JCU results are presented as means + standard errors. Significantly
different results occur when the national mean is outside the error bars.
18
Figure 17: Levels of satisfaction (% satisfied y axis) for JCU candidates compared for
2012 and 2010 by Faculty. The JCU results are presented as means +
standard errors. Significantly different results occur when the national
mean is outside the error bars.
19
Figure 18: Levels of satisfaction (% satisfied y axis) for HDR candidates from JCU
(2012) compared with JCU and sector results (2010-12) for graduands
for field of research. The JCU results are presented as means + standard
errors. Significantly different results occur when the national mean is
outside the error bars.
20
Figure 19: Levels of satisfaction (% satisfied y axis) for HDR candidates from JCU
(2012) compared with the corresponding results for 2010 by Field of
Research. The JCU results are presented as means + standard errors.
Significantly different results occur when the national mean is outside the
error bars.
21
Figure 20: Levels of satisfaction (% satisfied y axis) for HDR candidates from JCU
(2012) compared with, JCU and sector results (2010-12) for graduands by
attendance type. The JCU results are presented as means + standard
errors.
Figure 21: Levels of satisfaction for HDR candidates from JCU (2012) compared with
the corresponding results for 2010 by attendance type. The JCU results
are presented as means + standard errors.
Figure 22: Levels of satisfaction (% satisfied y axis) for HDR candidates from JCU
(2012) compared with JCU and sector results (2010-12) for graduands by
mode of study. The JCU results are presented as means + standard errors.
22
Figure 23: Levels of satisfaction (% satisfied y axis) for HDR candidates from JCU
(2012) compared with the corresponding results for 2010 by mode of
study. The JCU results are presented as means + standard errors.
23
Figure 24: Levels of satisfaction (% satisfied y axis) for HDR candidates from JCU
(2012) compared with JCU and sector results (2010-12) for graduands by
resident status. The JCU results are presented as means + standard
errors.
Figure 25: Levels of satisfaction (% satisfied y axis) for HDR candidates from JCU
(2012) compared with the corresponding results for 2010) by resident
status. The JCU results are presented as means + standard errors.
24
Figure 26: Levels of satisfaction (% satisfied y axis) for HDR candidates from JCU
(2012) compared with, JCU and sector results (2010-12) for graduands by
age group. The JCU results are presented as means + standard errors.
Figure 27: Levels of satisfaction (% satisfied y axis) for HDR candidates from JCU
(2012) compared with the corresponding results for 2010 by age group.
The JCU results are presented as means + standard errors.
25
Figure 28: Levels of satisfaction (% satisfied y axis) for HDR candidates from JCU
(2012) compared with JCU and sector results (2010-12) for graduands by
gender. The JCU results are presented as means + standard errors.
Figure 29: Levels of satisfaction (% satisfied y axis) for HDR candidates from JCU
(2012) compared with the corresponding results for 2010 for each
gender. The JCU results are presented as means + standard errors.
Figure 30: Levels of satisfaction (% satisfied y axis) for HDR candidates from JCU
(2012) compared with JCU and sector results (2010-12) for graduands by
working status. The JCU results are presented as means + standard
errors.
26
Figure 31:
Comparison of the satisfaction levels (% satisfied y axis) of JCU current
HDR candidates in 2012 and 2010 by work status for each of the PREQ
satisfaction scales. The JCU results are presented as means + standard
errors.
Figure 32: Levels of satisfaction (% satisfied y axis) for HDR candidates from JCU
(2012) compared with JCU and sector results (2010-12) for graduands by
qualification level. The JCU results are presented as means + standard
errors.
27
Figure 33: Levels of satisfaction (% satisfied y axis) for HDR candidates from JCU
(2012) compared with the corresponding results for 2010 by
qualification level. The JCU results are presented as means + standard
errors.
Figure 34: Levels of satisfaction (% satisfied y axis) for HDR candidates from JCU
(2012) compared with JCU and sector results (2010-12) for graduands by
employment type. The JCU results are presented as means + standard
errors.
Figure 35: Levels of satisfaction (% satisfied y axis) for HDR candidates from JCU
(2012) compared with the corresponding results for 2010 by employment
type. The JCU results are presented as means + standard errors.
28
Figure 36: The proportions of HDR candidates in agreement with the additional
statements in the internal PREQ surveys in 2012 and 2010. The JCU
results are presented as means + standard errors.
29
Figure 37: The proportion of HDR candidates (% y axis) who agreed that their
research takes advantage of the University’s tropical location by Faculty
and aggregated field of research and the satisfaction levels (% satisfied y
axis) on the PREQ satisfaction scales (2012 and 2010) of such candidates.
The JCU results are presented as means + standard errors.
a. b.
c.
30
Figure 38: Comparison of the satisfaction levels (% satisfied y axis) of HDR
candidates whose research does not take advantage of the University’s
tropical location by PREQ satisfaction scale for 2012 and 2010. The JCU
results are presented as means + standard errors.
Figure 39: Comparison of the satisfaction levels (% satisfied y axis) of HDR
candidates whose research does and does not take advantage of the
University’s tropical location by PREQ satisfaction scale for 2012. The JCU
results are presented as means + standard errors.
31
Figure 40: The proportion of HDR candidates who agreed that their Access to
Library Resources met their needs by Faculty and aggregated field of
research and the satisfaction levels (% satisfied y axis) on the PREQ
satisfaction scales (2012 and 2010) of such candidates. The JCU results
are presented as means + standard errors.
a. b.
c.
32
Figure 41: The satisfaction levels of HDR candidates who considered that their
access to Library Resources was not satisfactory for each PREQ
satisfaction scale (2012 and 2010). The JCU results are presented as
means + standard errors.
Figure 42: Comparison of the satisfaction levels (% satisfied y axis) of HDR candidates
whose were and were not satisfied by their access to library resources by
PREQ satisfaction scale for 2012. The JCU results are presented as means
+ standard errors.
33
Figure 43: The proportion of HDR candidates (% y axis) who had a least Fortnightly
Meetings with Principal Supervisor by Faculty and aggregated field of
research and the satisfaction levels (% satisfied y axis) on the PREQ
satisfaction scales (2012 and 2010) of such candidates. The JCU results
are presented as means + standard errors.
a. b.
c.
34
Figure 44: The satisfaction levels for the PREQ satisfaction scales (2012 and
2010) for the HDR candidates who did not meet wither their primary
supervisor on at least a fortnightly basis . The JCU results are presented
as means + standard errors.
\
Figure 45 : Comparison of the satisfaction levels (% satisfied y axis) of HDR
candidates who did and did not have at least fortnightly meetings with their
principal supervisor by PREQ satisfaction scale for 2012. The JCU results are
presented as means + standard errors.
35
Figure 46: The proportion of HDR candidates (% y axis) whose supervisor provides
timely feedback to written work by Faculty and aggregated field of
research and the satisfaction levels (% satisfied y axis) on the PREQ
satisfaction scales (2012 and 2010) of such candidates. The JCU results
are presented as means + standard errors.
a. b.
c.
36
Figure 47: The satisfaction levels for the PREQ satisfaction scales (2012 and
2010) for the HDR candidates who did not receive timely feedback from
their supervisor. The JCU results are presented as means + standard
errors.
\
Figure 48: Comparison of the satisfaction levels (% satisfied y axis) of HDR
candidates who did and did not receive timely feedback from their supervisor by
PREQ satisfaction scale for 2012. The JCU results are presented as means +
standard errors.
37
Qualitative Analysis
The qualitative analysis is undertaken using the word cloud technology in conjunction with
reading through the responses to the qualitative questions.
Figure 46: Word Cloud depicting the best aspects of the course of study as identified
by JCU HDR candidates in 2012. The size of the word is correlated with the
word frequency in the respondent’s answer.
Figure 41: Word Cloud depicting the worst aspects of the course of study as
identified by JCU HDR candidates in 2012. The size of the word is
correlated with the word frequency in the respondent’s answer.