Ottawa Final
-
Upload
chris-hebbes -
Category
Documents
-
view
216 -
download
0
Transcript of Ottawa Final
-
8/8/2019 Ottawa Final
1/16
Student detection of errors in
examinationsDr Christopher Hebbes ([email protected])
Dr Jenny Briggs
Dr Emmanuel DavidsonDr Ricky Bell
Dr Heather Crick
-
8/8/2019 Ottawa Final
2/16
Introduction
10% of the UK medical student populationstruggle to pass exams
Multiple reasons
Implications for patient safety
Yates J, James D. Risk factors at medical school for subsequent professional misconduct: multicentreretrospective case-control study. BMJ 2010;340:c2040
-
8/8/2019 Ottawa Final
3/16
Self assessment
Self assessment often inaccurate
Why?
Do students give the same error attributionas their tutors?
I. Colthart, et al. (2008). `The effectiveness of self-assessment on the identification of learner needs, learneractivity, and impact on clinical practice: BEME Guide no. 10'. Medical Teacher30(2):124-145.
= +Score True score Error component
-
8/8/2019 Ottawa Final
4/16
Aims
Establish a taxonomy of common types oferrors
Determine ability to detect gold standard
errors
Encourage reflection in learners
Student engagement in assessment
-
8/8/2019 Ottawa Final
5/16
Methods
Pilot study
67 students includedof 84
Failed one or bothsemester exams
Student consent
Mock exam
10 SAQ questions
SAQ Exam
Self-marked Tutor-marked
Errors coded Errors coded
Comparison
Standardisedcoding sheet
-
8/8/2019 Ottawa Final
6/16
Errors (overall)
Tutor mark = max mark? Student mark = max mark?N
Student mark = max mark
Y
True PositiveFalse Positiv
False Negative
True Negativ
Error No Error
Error
No Error
Tutor
Stud
ent
False Positive
True Negative
N
Y
False Negative
True Positive
Y
N
-
8/8/2019 Ottawa Final
7/16
Errors (specific)
=Tutor mark max mark? Tutor code = X*N
Exclude
Y
An error has occurred
* Where X is the test
Y
Student code = X*
N
Student code = X*
True Positive
False Negative
Y
N
False Positive
True Negative
Y
N
-
8/8/2019 Ottawa Final
8/16
Results
-
8/8/2019 Ottawa Final
9/16
Results
.Cronbach s Alpha 0 81
: .Tutor 29 45: .Student 44 29
< .p 001Eta2 .608
-
8/8/2019 Ottawa Final
10/16
Error Detection (overall)
2107 Errors*
Students detected74%
Error prevalence 0.88 Sensitivity 0.74
True Positive1560
False Positive102
False Negative547
True Negative173
Error No Error
Error
No Error
Tutor
tudent
*Instances where full marks were not achieved for a part of a question
-
8/8/2019 Ottawa Final
11/16
Error Type DetectionComparison of tutor and student error attribution
Sensitivity Specificity
A 0.016 0.919
B 0.012 0.908
C 0.166 0.723
D 0.032 0.837
-
8/8/2019 Ottawa Final
12/16
Summary
Students are able to identify errors withhigh sensitivity
There is a difference in error attribution
between students and their tutors
-
8/8/2019 Ottawa Final
13/16
Future work
Is there a difference in error attributionbetween first, second and third yearstudents?
What factors are associated withcorrelation between student and tutorerror attribution?
Do interventions to help students attributeerrors improve correlation and improveacademic performance?
-
8/8/2019 Ottawa Final
14/16
Acknowledgements
Professor Stewart Petersen, Professor of MedicalEducation, University of Leicester
Dr Heather Crick, Lecturer, University of Leicester
Dr Gordon French, Deputy Postgraduate Dean,East Midlands Healthcare Workforce Deanery
Dr Adrian Hastings, Senior Clinical Educator,University of Leicester
Dr David Matheson, Lecturer in MedicalEducation, Medical Education unit, University ofNottingham & East Midlands HealthcareWorkforce Deanery
-
8/8/2019 Ottawa Final
15/16
?: / / . /ttp tinyurl com hebbes
-
8/8/2019 Ottawa Final
16/16
References