How do we achieve accuracy in the content, presentation and

instructions of examination papers?

Sonya StewartRichard Greatbanks

University of Otago

Overview

• Introductions

• Context – the examination experience

• Process – the exam paper system

• Analysis – data collection and analysis approaches

• Future improvement actions and further analysis

Introductions

• Sonya Stewart, Group Leader – Student Administration

• Richard Greatbanks, Associate Dean (Academic) – OBS

Context - the Examination Experience

• Examinations are integral to the university experience– Academically essential in our current university system

– Can account for up to 70 percent of a programme’s assessment• Often considered the only true reflection of a student’s individual capability

– Might become even more prominent in the future• Emergence of ghost writing sites and increased access to technology

– Exams are therefore a ‘key business process’ for a university

Context - the Examination Experience

• From a student’s perspective they are integral too– They are often a ‘rite of passage’

– Examinations mark the ‘end of study’• A semester, a paper, an academic year, or even their final degree

– Examinations often the last interaction before graduating university• They can be stressful or emotional

• They are meant to test the student’s capability

– Will directly influence the class of degree a student obtains

– Will influence early career choices and opportunities

Context - the Examination Experience

• As a university community, we therefore have an obligation to offer the best examination experience for our students

– Provide examination papers which are • Appropriate and relevant to the subject of study

• Marked fairly and consistently across the class

• Set out with clear accurate instructions

• Offered complete and without errors

• To any student, these would be considered basic expectations

Process - the Examination Paper System

Department

Academic responsible

for exam

ExaminationsOffice

(PE errors)

Student examination(ED errors)

Exams office query

Department rework

Day of exam query to Dept/ Academic and response

Analysis - the Examination Paper System

• Data collected by Sonya Stewart

– Number of errors in each examination paper submitted to the examinations office between S1 2015 and S1 2019• Number of papers with at least one error (defective papers)

• Number of errors per paper (defects)

• Type of error detected – admin or content

• Time when error detected – pre-exam (PE) or exam day (ED)

Exam paper data analysis - 1

0

100

200

300

400

500

600

2015 2015 2016 2016 2017 2017 2018 2018 2019

Number of exam paper in each semester

0

20

40

60

80

100

120

140

160

180

200

2015 2015 2016 2016 2017 2017 2018 2018 2019

Number of exam papers - PE and ED errors

Note different scales

Exam paper data analysis - 2

0

20

40

60

80

100

120

140

160

180

200

2015 2015 2016 2016 2017 2017 2018 2018 2019

S1 2015 to S1 2019 time based variationPE and ED errors – Number of papers

S2 2018 indicates 176 papers were identified with at least

one error at the PE stage

S2 2018 indicates 26 papers were identified with at least

one error at the ED stage

Exam paper data analysis - 3

0

50

100

150

200

250

300

2015 2015 2016 2016 2017 2017 2018 2018 2019

PE Error type - Admin or ContentNumber of errors (Defects)

admin content

0

5

10

15

20

25

2015 2015 2016 2016 2017 2017 2018 2018 2019

ED Error type - Admin or ContentNumber of errors (defects)

admin content

Note different scales

Exam paper errors - mean system performance

P1 Department

Academic responsible

for exam

P2 Examinations

Office

P3Student

examination

PE Error query

Department Rework 31%(145 papers)

ED Errors 3% (16 papers),of which 2% (9 papers) were initiallyconsidered error free at P2 PE check

100% all papers

Day of exam query to Dept/ Academic and response

Process analysis – Rolled Throughput Yield

P1 Department100% Yield

P2 Exam Office69.3% Yield

P3 Examination96.7% Yield

Total Process RTY = 1.0 x 0.693 x 0.967

Total Process RTY = 0.670 or 67% yield

Where as for S1 2016 the RTY is only 66.4%. We could expect 66.4% of all papers to pass through the process

without failing at any stage

Process analysis – Rolled Throughput Yield

2015 2015 2016 2016 2017 2017 2018 2018 2019

Rolled Throughput Yield S1 2015 to S1 2019

So for S2 2017 the RTY is 72.7% - that is we could expect 72.7% of all

papers to pass through the process without failing at any stage

RTY represents the capability of a system to produce good product through all process steps

Future actions and opportunities

• Advanced Quality Planning (AQP)• Analysis of key business processes

– Ideally in advance of product or service delivery

– Develop measures which indicate system capability

– Ability to identify and minimise high risk factors

• Focus on placing responsibility for quality at source– Commination of PE and ED performance across divisions

Development of Risk Priority Number• An RPN defines the likely levels of risk given known factors

– Severity of error • Missing page number = 1, incorrect exam duration = 5, missing formula = 10

– Ability to detect error • Spelling mistake = 1, incorrect formula = 10• Qualitative or discursive questions = 3, quantitative or science questions = 9

– Consequential effect - This could be class size or final year exams• Small class < 20 = 3, large class > 500 = 10• First year papers = 3, Final year papers = 10

– Each factor assessed out of 10 and multiplied to derive a total RPN out of 1000

• Calculation of RPN can then be used to identify higher risk combinations in advance of the exam paper being produced– Effectively offering a alternative process for high risk papers, subjects or departments

Future improvement actions

• Improvement actions must emphasize placing responsibility with the academic team– Develop policy and standards to define expectations

– Offer training to academic staff to support better practice

– Establish an audit regime for high RPN papers, subjects and departments

– Continue to collect data and disseminate across departments

– Use RTY each semester as a measure of system capability

Department

Academic responsible

for exam

ExaminationsOffice

Student examination

Exams office query

Department rework

Day of exam query to Dept/ Academic and response

Conclusions

• Ultimately, this requires a cultural change• Place responsibility with academic team

• Further analysis by division, department and subject

• Continue to collect data and measure system capability

• Potential to benchmark with other departments, divisions or institutions

• Voice of the Customer (VOC) data on exam experience

Thank you for your attention

Questions?