Student readiness Supervision challenges New examiner challenges Challenges of format Towards a...
-
Upload
shavonne-horn -
Category
Documents
-
view
217 -
download
0
Transcript of Student readiness Supervision challenges New examiner challenges Challenges of format Towards a...
Student readiness Supervision challenges
New examiner challenges Challenges of format
Towards a ThesisAssessment Matrix: Anaction research project
Michelle Picard & Lalitha Velautham
Literature on thesis examination
• ‘There seems to be a tacit understanding of what constitutes
a well-prepared Ph.D. student’, and that ‘in the complete
absence any central repository of rules or a cosmic
accrediting agency’ there is ‘extraordinary stability’
(Kwiram, 2006 p. 142 cited in Holbrook et al, 2008).
• Experienced examiners generally give feedback
consistently. Inexperienced examiners find it difficult to
define acceptable thesis standards and are inconsistent in
giving feedback (Kiley & Mullins, 2004; Kiley et al, 2011).
• Need for indicators to distinguish between theses of
threshold quality and those of highest quality
(Holbrook et al, 2008)
• The global research/ the global thesis standard? RED, School of Education
Literature on assessment matrices
• Tools to negotiate expectations between
supervisors and students (Cadman & Cargill,
2007, Kumar & Strache, 2007)
• Taxonomies for reviewing literature (Cooper,
1988; Boote and Beile, 2005; Randolph, 2009)
• Thesis rating scale (Albertyn et al, 2011)
Areas for extension
• No clear empirical basis
• Lack descriptive language and quality indicators
commonly employed by experienced examiners
• Disciplinarily slanted towards Sciences
• Focussed on traditional thesis formats
Research aim
• To create thesis assessment discussion tools for
supervisors, students and (potentially) examiners
in order to clarify expectations and assist in
assessing research documents across formats
and disciplines
RED, School of Education
Theoretical Framework and Methods
RED, School of Education
Identification of issue
Development of theory
Development of matrices
Deployment & evaluation of matrices
(Drummond & Themessl-Huber 2007)
Action research spiral 1
1. Research proposal assessment matrix• Developed and refined based on discussions with
EAL students and their supervisors• Final product developed from RSD7 & Literature
Review (Willison, 2008 & Boote & Beile 2011)
• Used with over 600 research students from March 2010 and their supervisors as:
- Negotiation tool - Assessment tool • Overwhelmingly positive evaluation • Wordiness
RED, School of Education
Action research spiral 2
2. First draft thesis assessment matrix
• Imperatives from our experience & the literature as well as
work on Research Proposal Matrix
• Draft matrix developed from RSD7 & Literature Review (e.g.
Willison, 2009; Randolph, 2009; & Boote & Beile 2011) as well as insights
from spiral 1.
• Deployment to experienced supervisors at the
University of Adelaide via an online survey, by students in a
thesis writing group and at a postgraduate research
conference in South Africa (April, 2013).
RED, School of Education
Student comments Spiral 2
HDR students
• ‘It is good to have something to take in to discuss my writing with my
supervisor’ (HDR 15).
• ‘It helps me to know what I must do to do research in a PhD’
• ‘It is scary to see how much I still have to learn, I thought I knew all about
research in honours’ (HDR1
• ‘It is a relief to see I am ok, at a stage of development just like my
supervisor’ (HDR 21)
• ‘This grid really helped us to see what is expected of a thesis...you
know you say it is not a Nobel Prize, but sometimes my supervisors seem to
expect that... so it helps to know what a thesis really should do’.
• A bit ‘wordy’ (HDRs 1,12 & 16)
Conference comments
• ‘Too focussed on literature review and writing
skills’
• ‘Needs more focus on original contribution’
• ‘Good to consider thesis by publication’
• ‘Likely to be less useful in creative disciplines’
• ‘Does not really address argument-based thesis’
• ‘It still does not encapsulate the true work of the
PhD’
Experienced supervisors
Statements
1. Strongly agree 2. 3. 4. 5. Strongly disagree
A Thesis Assessment Tool could potentially 0help clarify thesis standards
with research students
16.7% (3)
41.2%* (7)
11.1% (2)
16.7% (3)
11.1% (2)
A Thesis Assessment Tool could potentially help new supervisors clarify
expectations of the thesis
16.7% (3)
52.9% (9)
0.0 % (0)
16.7% (3)
11.1% (2)
A Thesis Assessment Tool could potentially help new examiners clarify
thesis standards
29.4 % (5)
38.9 % (7)
0.0 % (0)
5.6 % (1)
22.2 % (4)
A Thesis Assessment Tool could potentially be useful to clarify standards
with international examiners
25.0 % (4)
31.3 % (5)
5.9 % (1)
18.8 % (3)
12.5 % (2)
A Thesis Assessment Tool could potentially be useful to clarify standards
in other thesis formats (e.g. by publication or exegesis)
11.1% (2)
35.3 % (6)
16.7% (3)
27.8 % (5)
5.6 %
(1)
A Thesis Assessment Tool could potentially differentiate clearly between
MPhil, PhD & Doctorate requirements
11.1% (2)
35.3 % (6)
33.3 % (6)
11.1% (2)
5.6 %
(1)
A Thesis Assessment Tool could potentially aid in researcher training
11.1% (2)
52.9 % (9)
5.6 %
(1)
11.1% (2)
16.7% (3)
Positive responses in….
• Clarify expectations for research students
• Clarifying expectations for new supervisors
• Clarifying expectations for new examiners (but
also high strongly disagree)
• Aid in researcher training
Positive comments
• ...this would be very helpful for examiners - provides clear differentiation and progression, and what is suitable for each level.(ID 12)
• ...any template which gives students, supervisors, etc. an understanding of levels of competency will assist them to distinguish between competent and excellent work. (ID 12)
• the criteria listed coincide with the long established measures of adequacy in our discipline. (ID 17)
Less favourable responses in…
• Assist new examiners (22.2 strongly disagree)
• Other thesis formats (27.8 disagree)
• Differentiate between MPhil, Doctorate, PhD (33.3 neutral)
Examiners & Supervisors
• ...it introduces thresholds that do not correspond with the Pass/Fail approach to research degree examination. There is absolutely no point in differentiating between 'Pass' and 'Excellence' at the PhD level. (ID 10)
• New students, supervisors and even examiners may still lack the knowledge or confidence, even with these criteria, to work out where their work sits... (ID 12)
• If a supervisor doesn't know what is entailed in a thesis of particular kind at a particular level, he/she should not be a supervisor. (ID 16)
Formats & disciplines
• Please note that creative theses (creative writing
PhD and MPhil) would need an adapted assessment
tool - to be developed by the relevant discipline(s).
One size does not fill all.(ID 3)
• fails to address metrics related to thesis writing
specifically in the Sciences related to areas of poor
experimental design, data analysis, and data
interpretation
Differentiation
• ‘It wasn't clear to me that the draft assessment tool provided these metrics, but rather was couching inadequacies within more general and rather heavily semantic terminologies - the RSD has its own lingua franca that may not be widely understood or appreciated, and certainly I'd suggest is beyond the interpretation of a busy international examiner. (ID 8)
• ‘Experienced examiners will dismiss it. Inexperienced examiners will be misled by it.’ (ID 10)
• ‘...it is implicitly language rich, and with 50% of our students coming to UA from o/seas, to introduce them to such refinement of language early in their research careers, is surely asking a bit much.(ID 8)
Differentiation
• If it is to have 'currency' it would need to have
greater clarity through simplicity. The essential feature
for a PhD, for example, is originality. This is the crucial
ingredient. For the MPhil, however, the originality looms less
large. (ID 10)
• ‘excellent research outcomes’ often overshadows ‘poor
thesis construction’. (ID 8)
• A PhD degree is examined on a Pass/Fail basis. There is no
recognised international concept of a 'Pass' PhD as opposed
to a 'Better than Pass' PhD. (ID 10)
Action research spiral 3
3. Refinement of thesis assessment matrix
- Audit of rubrics provided to Australian and
international thesis examiners by institutions
- An audit of recurring phrases indicating quality
and rigor in examiners reports
- Combination of all data into revised thesis
assessment matrix
- Thus far, audit of rubrics completed
RED, School of Education
Categories of phrases
Knowledge Contribution
Synthesis and Analysis of the Literature
Methodology Innovativeness and Critical Thinking
Results/Findings
Vocabulary/Presentation
Other Attributes
The expectation that a thesis displays new perspectives and/or advanced knowledge of research
The student shows familiarity with, and understanding of, the relevant literature
The techniques adopted are appropriate to the subject matter and are properly applied
Originality of approach orinterpretation
The results are suitably set out, and accompanied by adequate exposition
The thesis contains material worthy of publication in a form appropriate to the discipline
Research apprenticeship iscomplete and holder isadmitted to thecommunity of scholars inthe discipline
University of Adelaide 21
Rubrics Across GO8 Universities
• Knowledge contribution PhD
“Thesis as a whole is substantial and original contribution to
knowledge of the subject…” (ANU; UWA)
“Significant contribution to knowledge and understanding in the
relevant field” (MU)
“Makes a distinct contribution to knowledge” (UOS)
“Thesis displays new perspectives and/or advanced knowledge…”
(UNSW)
“makes a distinct contribution to knowledge because of the
originality of the approach and/or interpretation of the findings and
in some cases the discovery of new facts” (UM)
Rubrics Across GO8
• Knowledge contribution Masters
“Shows a sound knowledge of the subject…evidence of some
independence of thought” (UWA)
“Makes a contribution…” (MU)
“Appreciation of relationship of topic to wider field of knowledge”
(UOS)
“Displays new perspectives” (UNSW)
“Demonstrates advanced learning in research skills” (UM)
Potential matrix advantages
• Supervisors: Clear articulation of expectations and standards
• Examiners: Fair, transparent and consistent evaluation,
assistance in articulating concerns
• Students: Clear goals and direction leading to better self-
management techniques in thesis writing
RED, School of Education
A word of caution
• Hijacking by rampant managerialists/ administrators
• Difficulties of pinning down the ephemeral
• Danger of ‘marking’ rather than ‘examining’ (Ward, 2013).
• Matrix/grid implies grades – the PhD is not graded per se
• Disciplinary peer review element of becoming ‘part of the
club’, entering into an in-depth discussion could potentially
be lost
• Personally now with two PhD examinations – still need for
assessment and comments, but helped in examining
process
Next step in Action Research Cycle
• Audit of reoccurring phrases/ language used by
experienced examiners
• Refining current matrix (in handout)
• Trial of current matrix with examiner volunteers
Questions & Feedback
• Please contact:[email protected] Or [email protected]
RED, School of Education