ESMEA online
2017
+
To assure standards at graduation from
medical school or postgraduate certification
To award the MBBS to students who meet the
University’s standards (University interest)
To certify that graduates are suitable for provisional
registration (GMC interest)
To ensure graduates are fit to undertake residency
posts (NHS/employer interest)
To award PG certification to those who have met
required specialist standards
To distinguish between the competent and the
insufficiently competent (Public interest)
Standard setting – why bother?
+
Test
Result
Pass
Fail
Competent Incompetent
+Definition of Standards
A standard is a statement about whether an examination performance is good enough for a particular purpose
Standards are based on judgments about examinees’performances against a social or educational construct
TerminologyPassing standard
Pass mark
Cut score
A special score that serves as the boundarybetween passing and failing
+ Types of Standard Setting
Methods
Relative standards: norm referenced methods Candidates are compared to others in the group
A set proportion of candidates fails regardless of how well they perform
Competitive situation eg selection
Absolute standards: criterion referenced methods Candidates are compared to a set standard
Define the outcomes of course
Competence eg tests of knowledge/skills
Compromise methods
+ What does this mean in
practice?
‘Historical precedent’ cut scores are no longer credible for use in high stakes summative tests Even if your HE institution requires it
We are dealing with health professional competence
Setting passing standards should be determined by careful analysis and judgement of what acceptable performance is This may vary from test to test
Decisions around how to set cut scores should draw on the best available evidence Defensibility
+ Standards
Standards are based on judgments about
examinees’ performances against a social or
educational construct
Requires judgment of experts
Dependent on context
e.g. Competent practitioner or student ready for
graduation
+
Standard setting at graduation level
Pass/Fail: distinguish between
the competent and the not yet
sufficiently competent
Level: what a candidate knows/is able to do to
become a FY1 doctor/ Day 1vet /
Day 1 dentist/ graduate nurse
What is the purpose of standard
setting at Finals/graduation?
+
Pass/Fail: distinguish between
the competent and the not yet
sufficiently competent at the
specified level
Level: what a candidate knows/is able to do to
become a specialist/enter a speciality
training programme
What is the purpose of standard
setting at postgraduate
certification levels?
+
Standard Setting Methods
+Principles of standard setting
Method used should be:
Credible
Defensible
Feasible
Acceptable to all stakeholders
Choice of method less important than whether it:
Relies on informed expert judgement
Demonstrates due diligence
Produces standards consistent with purpose of the test
Is easy to explain and implement
Supported by body of evidence in the literature
Norcini, J. J. (2003). Setting standards on educational tests. Medical Education, 37, 464-469.
+Setting the pass mark
Who should decide on the standards?
Teachers of the students
Academics and clinicians
Familiar with the curriculum
Familiar with levels of student knowledge
Understand the purpose and level of examination
Familiar with ‘what is good enough’ for this particular test
+Defining the ‘Borderline
Candidate’
Group of judges should discuss:
Nature and purpose of examination
Nature and level of examinees
What is good enough for this particular test
Thinking about the JUST GOOD enough
candidate
Characteristics of their knowledge base
+ Criterion-referenced Methods
Panel Judgments About Individual
Test Items
Angoff’s method
Ebel’s method
+Angoff method
Each judge estimates the probability of the
‘just good enough candidate’ getting the
correct answer to each item
Between 0 and 100 % likelihood
Each judge allocates estimates to every item
Discussion among judges about differences
Adjustments
Final pass mark is summated
Angoff Method
Items Judge1 Judge2 Judge3 Judge4 Judge5 Mean
1 60 70 55 75 65 65
2 80 90 85 95 90 88
3 70 75 80 75 40 68
4 45 55 50 60 55 53
5 90 95 85 95 90 91
6 30 40 30 50 40 38
7 60 60 50 70 60 60
Total
463
62% 70% 74%62% 62%Pass mark 463/7 = 66%
Ebel method - 1
Easy Medium Hard
Essential
Important
Acceptable
RELEVANCE
DIFFICULTY
Judges make “Difficulty-Relevance” decisions
Ebel method - 2
Weighting Easy Medium Hard
Essential 90% 80% 70%
Important 60% 50% 40%
Acceptable 30% 20% 10%
Ebel method - 3
Weighting Easy Medium Hard
Essential 90% x 2 =
1.8
80% x 3 =
2.4
70% x 1 =
0.7
Important 60% x 3 =
1.8
50% x 3 =
1.5
40% x 1 =
0.4
Acceptable 30% x 1 =
0.3
20% x 1 =
0.2
10% x 1 =
0.1
16 items
{Sum/ no of items} x
100 = {9.2/16}x100 =
57.5%
Weighting x
number of items
+ Criterion referenced methods: Angoff
and Ebel
Advantages
They focus attention on item content
They are relatively easy to use
There is a considerable body of published work supporting their use
They are used frequently in high stakes testing
Disadvantages
The concept of a "borderline group" is sometimes difficult to define
Judges sometimes feel they are "pulling numbers out of the air” (Angoff)
The methods can be tedious & time-consuming
+Standard setting methods
for OSCEs
Methods specifically
developed for OSCEs
Borderline Group Method
Contrasting group method
Borderline Regression Method
Boulet, J. R.; De Champlain, A.F.; McKinley, D.W. (2003) Setting defensible performance
standards on OSCEs and standardized patient examinations. Medical Teacher, 3, 245-249.
+ The examiner’s role in
standard setting
Examiner’s clinical expertise is used to judge the candidate’s performance overall (separate from scoring rubric)
Examiner allocates a global judgement based on the candidate’s performance at that station
Remembering the level of the examination
FailBorderline
Pass
Checklist1. Hs shjs sjnhss sjhs sjs sj
2. Ksks sksmsiqopql qlqmq q q qkl
3. Lalka kdm ddkk dlkl dlld
4. Keyw dd e r rrmt tmk
5. Jfjfk dd
6. Hskl;s skj sls ska ak akl ald
7. Hdhhddh shs ahhakk as
TOTAL
Passing score
Borderline score distribution
Pass, Fail, Borderline
Test score distribution
Borderline Group Method
Checklist
1. Hs shjs sjnhss sjhs sjs sj
2. Ksks sksmsiqopql qlqmq q q qkl
3. Lalka kdm ddkk dlkl dlld
4. Keyw dd e r rrmt tmk
5. Jfjfk dd
6. Hskl;s skj sls ska ak akl ald
7. Hdhhddh shs ahhakk as
TOTAL
Overall rating 1 2 3 4 5
1 2 3 4 5
Checklist
Score
X
X = passing score
1 = Clear fail2 = Borderline3 = Clear pass4 = Very good5 = Outstanding
Clear Borderline Clear Very Outstanding
fail pass good
Borderline Group Method
+ Performance-based standard setting
Advantages
Utilises the expertise of
clinical examiners
Real-time observation
Large number of examiners
set a collective standard while
observing the candidates –
not just an academic exercise
Reliable: cut-off score based
on large sample of judgments
There is a considerable body
of published work supporting
their use
Credible and defensible:
based on expert judgment in
direct observation
Disadvantages
• Requires large cohort of candidates to achieve enough numbers in the‘borderline’group
• Passing score not known in advance
• Judgments not independent of checklist scoring
• Examiners require training
• Methods require expert processing of marks immediately after the exam
• Delay in producing results
+ Implementation Guidelines for
Setting Standards
There is no substitute for a considered assessment strategy, high quality items and careful test construction
Choose the method which suits your context best Consider resources: time, individuals
Select the judges
Assign an appropriate number (at least 6-8 for high stakes testing)
Select the characteristics the group should possess
Develop an efficient design for the process
+ The choices
Different standard setting methods lead to
different cut off scores – no gold standard exists
Robust method should be credible and acceptable
and produce realistic outcomes
Process is more important than method in overall
validity of your assessment – collect and analyse
your own data
Not using a recognised method is NOT a viable
choice for assessing the competence of this
century’s health professionals
+
Thank you!
Email:
Our website: http://hpac.sg
Top Related