How do we achieve accuracy in the content, presentation and
instructions of examination papers?
Sonya StewartRichard Greatbanks
University of Otago
Overview
• Introductions
• Context – the examination experience
• Process – the exam paper system
• Analysis – data collection and analysis approaches
• Future improvement actions and further analysis
Introductions
• Sonya Stewart, Group Leader – Student Administration
• Richard Greatbanks, Associate Dean (Academic) – OBS
Context - the Examination Experience
• Examinations are integral to the university experience– Academically essential in our current university system
– Can account for up to 70 percent of a programme’s assessment• Often considered the only true reflection of a student’s individual capability
– Might become even more prominent in the future• Emergence of ghost writing sites and increased access to technology
– Exams are therefore a ‘key business process’ for a university
Context - the Examination Experience
• From a student’s perspective they are integral too– They are often a ‘rite of passage’
– Examinations mark the ‘end of study’• A semester, a paper, an academic year, or even their final degree
– Examinations often the last interaction before graduating university• They can be stressful or emotional
• They are meant to test the student’s capability
– Will directly influence the class of degree a student obtains
– Will influence early career choices and opportunities
Context - the Examination Experience
• As a university community, we therefore have an obligation to offer the best examination experience for our students
– Provide examination papers which are • Appropriate and relevant to the subject of study
• Marked fairly and consistently across the class
• Set out with clear accurate instructions
• Offered complete and without errors
• To any student, these would be considered basic expectations
Process - the Examination Paper System
Department
Academic responsible
for exam
ExaminationsOffice
(PE errors)
Student examination(ED errors)
Exams office query
Department rework
Day of exam query to Dept/ Academic and response
Analysis - the Examination Paper System
• Data collected by Sonya Stewart
– Number of errors in each examination paper submitted to the examinations office between S1 2015 and S1 2019• Number of papers with at least one error (defective papers)
• Number of errors per paper (defects)
• Type of error detected – admin or content
• Time when error detected – pre-exam (PE) or exam day (ED)
Exam paper data analysis - 1
0
100
200
300
400
500
600
2015 2015 2016 2016 2017 2017 2018 2018 2019
Number of exam paper in each semester
0
20
40
60
80
100
120
140
160
180
200
2015 2015 2016 2016 2017 2017 2018 2018 2019
Number of exam papers - PE and ED errors
Note different scales
Exam paper data analysis - 2
0
20
40
60
80
100
120
140
160
180
200
2015 2015 2016 2016 2017 2017 2018 2018 2019
S1 2015 to S1 2019 time based variationPE and ED errors – Number of papers
S2 2018 indicates 176 papers were identified with at least
one error at the PE stage
S2 2018 indicates 26 papers were identified with at least
one error at the ED stage
Exam paper data analysis - 3
0
50
100
150
200
250
300
2015 2015 2016 2016 2017 2017 2018 2018 2019
PE Error type - Admin or ContentNumber of errors (Defects)
admin content
0
5
10
15
20
25
2015 2015 2016 2016 2017 2017 2018 2018 2019
ED Error type - Admin or ContentNumber of errors (defects)
admin content
Note different scales
Exam paper errors - mean system performance
P1 Department
Academic responsible
for exam
P2 Examinations
Office
P3Student
examination
PE Error query
Department Rework 31%(145 papers)
ED Errors 3% (16 papers),of which 2% (9 papers) were initiallyconsidered error free at P2 PE check
100% all papers
Day of exam query to Dept/ Academic and response
Process analysis – Rolled Throughput Yield
P1 Department100% Yield
P2 Exam Office69.3% Yield
P3 Examination96.7% Yield
Total Process RTY = 1.0 x 0.693 x 0.967
Total Process RTY = 0.670 or 67% yield
Where as for S1 2016 the RTY is only 66.4%. We could expect 66.4% of all papers to pass through the process
without failing at any stage
Process analysis – Rolled Throughput Yield
2015 2015 2016 2016 2017 2017 2018 2018 2019
Rolled Throughput Yield S1 2015 to S1 2019
So for S2 2017 the RTY is 72.7% - that is we could expect 72.7% of all
papers to pass through the process without failing at any stage
RTY represents the capability of a system to produce good product through all process steps
Future actions and opportunities
• Advanced Quality Planning (AQP)• Analysis of key business processes
– Ideally in advance of product or service delivery
– Develop measures which indicate system capability
– Ability to identify and minimise high risk factors
• Focus on placing responsibility for quality at source– Commination of PE and ED performance across divisions
Development of Risk Priority Number• An RPN defines the likely levels of risk given known factors
– Severity of error • Missing page number = 1, incorrect exam duration = 5, missing formula = 10
– Ability to detect error • Spelling mistake = 1, incorrect formula = 10• Qualitative or discursive questions = 3, quantitative or science questions = 9
– Consequential effect - This could be class size or final year exams• Small class < 20 = 3, large class > 500 = 10• First year papers = 3, Final year papers = 10
– Each factor assessed out of 10 and multiplied to derive a total RPN out of 1000
• Calculation of RPN can then be used to identify higher risk combinations in advance of the exam paper being produced– Effectively offering a alternative process for high risk papers, subjects or departments
Future improvement actions
• Improvement actions must emphasize placing responsibility with the academic team– Develop policy and standards to define expectations
– Offer training to academic staff to support better practice
– Establish an audit regime for high RPN papers, subjects and departments
– Continue to collect data and disseminate across departments
– Use RTY each semester as a measure of system capability
Department
Academic responsible
for exam
ExaminationsOffice
Student examination
Exams office query
Department rework
Day of exam query to Dept/ Academic and response
Conclusions
• Ultimately, this requires a cultural change• Place responsibility with academic team
• Further analysis by division, department and subject
• Continue to collect data and measure system capability
• Potential to benchmark with other departments, divisions or institutions
• Voice of the Customer (VOC) data on exam experience
Thank you for your attention
Questions?
Top Related