Using Rasch Analysis to Develop an Using Rasch Analysis to Develop an Extended Matching Question (EMQ) Extended Matching Question (EMQ)

Item Bank for Undergraduate Medical Item Bank for Undergraduate Medical EducationEducation

Mike HortonMike HortonBipin BhaktaBipin BhaktaAlan TennantAlan Tennant

Medical student trainingMedical student training

Miller (1990) identified that no single assessment Miller (1990) identified that no single assessment method can provide all the data required for judging method can provide all the data required for judging anything as complex as the delivery of professional anything as complex as the delivery of professional services by a competent physician services by a competent physician

• KnowledgeKnowledge• SkillsSkills• AttitudesAttitudes

Miller’s pyramid of competenceMiller’s pyramid of competence

knows

knows how

shows how

does

OSCE

EMQ, FRQ, Essays

MCQ

case presentationslog booksdirect observation of clinical activity

What are assessments used for?What are assessments used for?

Primary aimPrimary aim• To identify the student who is deemed to be safe To identify the student who is deemed to be safe

and has achieved the minimal acceptable standard and has achieved the minimal acceptable standard of competenceof competence

Secondary aimSecondary aim• To identify students who excelTo identify students who excel

Written AssessmentWritten Assessment

““True or False” questionsTrue or False” questions ““Single, best option” multiple choice questionsSingle, best option” multiple choice questions Multiple true or false questionsMultiple true or false questions ““Short answer” open ended questionsShort answer” open ended questions EssaysEssays ““Key feature” questions Key feature” questions Extended matching questionsExtended matching questions

Free response questionsFree response questions

Broadly, the free-response type questions are commonly Broadly, the free-response type questions are commonly believed to test important higher-order reasoning skills believed to test important higher-order reasoning skills

Validity is high as examinees have to generate their own Validity is high as examinees have to generate their own responses, rather than selecting from a list of optionsresponses, rather than selecting from a list of options

However…However…

Only a narrow range of subject matter can be assessed in a Only a narrow range of subject matter can be assessed in a given amount of time given amount of time

They are administratively resource-sappingThey are administratively resource-sapping Due to their nature, the reliability is limited.Due to their nature, the reliability is limited.

Multiple choice questionsMultiple choice questions

Multiple choice type questions have been widely used and are Multiple choice type questions have been widely used and are popular as:popular as:

They generally have a high reliability They generally have a high reliability They can test a wide range of themes in a relatively short period They can test a wide range of themes in a relatively short period

of timeof time

However… However…

They only assess the knowledge of isolated factsThey only assess the knowledge of isolated facts By giving an option list, examinees are cued to respond and the By giving an option list, examinees are cued to respond and the

active generation of knowledge is avoided active generation of knowledge is avoided

What are Extended Matching Questions (EMQs)?What are Extended Matching Questions (EMQs)?

EMQs are used as part of the undergraduate medical course EMQs are used as part of the undergraduate medical course assessment programmeassessment programme

EMQs are used to assess factual knowledge, clinical decision EMQs are used to assess factual knowledge, clinical decision making and problem solvingmaking and problem solving

They are a variant of multiple choice questions (MCQs)They are a variant of multiple choice questions (MCQs)

EMQs are made up of 4 componentsEMQs are made up of 4 components

Example of Extended Matching Question (EMQ) format Example of Extended Matching Question (EMQ) format (Taken from Schuwirth & van der Vleuten, 2003)(Taken from Schuwirth & van der Vleuten, 2003)

Theme:Theme: Micro-organismsMicro-organisms Answer options:Answer options:

Instructions:Instructions: For each of the following cases, select (from the list above) the micro-For each of the following cases, select (from the list above) the micro-organism organism most likely to be responsible. Each option may most likely to be responsible. Each option may

be used once, be used once, more than once or not at all.more than once or not at all. A 48 year old man with a chronic complaint of dyspepsia suddenly develops severe A 48 year old man with a chronic complaint of dyspepsia suddenly develops severe

abdominal pain. On physical examination there is general tenderness to palpitation with abdominal pain. On physical examination there is general tenderness to palpitation with rigidity and rebound tenderness. Abdominal radiography shows free air under the rigidity and rebound tenderness. Abdominal radiography shows free air under the diaphragm.diaphragm.

A 45 year old woman is treated with antibiotics for recurring respiratory tract infections. She A 45 year old woman is treated with antibiotics for recurring respiratory tract infections. She develops a severe abdominal pain with haemorrhagic diarrhoea. Endoscopically a develops a severe abdominal pain with haemorrhagic diarrhoea. Endoscopically a pseudomembranous colitis is seen.pseudomembranous colitis is seen.

A Campylobacter jejuni I Proteus mirabilisB Candida Albicans J Pseudomonas aeruginosaC Clostridium difficile K RotavirusD Clostridium perfringens L Salmonella typhiE Escherichia coli M Shigella flexneriF Giardia lamblia N Tropheryma whippeliiG Helicobacter pylori O Vibrio choleraeH Mycobacterium tuberculosis P Yersinia enterocolitica

Item PoolsItem Pools

Currently, a lot of institutions formulate their tests from year to year by Currently, a lot of institutions formulate their tests from year to year by selecting items from a pre-existing pool of questions. selecting items from a pre-existing pool of questions. • Questions are pre-existingQuestions are pre-existing• Time and resources are saved by employing this method. Time and resources are saved by employing this method.

However…However…

It has been widely recognised that if tests are made up of items from a It has been widely recognised that if tests are made up of items from a pre-existing item pool, then the relative difficulty of the exam paper will pre-existing item pool, then the relative difficulty of the exam paper will vary from year to year. [McHarg et al (2005), Muijtens et al (1998), vary from year to year. [McHarg et al (2005), Muijtens et al (1998), McManus et al (2005)]McManus et al (2005)]

Item PoolsItem Pools

If the questions have been set, used and assessed using traditional If the questions have been set, used and assessed using traditional approaches, then this will provide a certain amount of information about each approaches, then this will provide a certain amount of information about each of the individual questions, however, there are also drawbacks surrounding of the individual questions, however, there are also drawbacks surrounding the traditional approach.the traditional approach.

It has been recognised [Downing (2003)] that there are certain limitations to It has been recognised [Downing (2003)] that there are certain limitations to Classical Measurement Theory (CMT) in that it is sample dependent. Classical Measurement Theory (CMT) in that it is sample dependent.

Thus, the comparability of examination results from year to year will be Thus, the comparability of examination results from year to year will be confounded by the overall difficulty of the exam and the relative ability of the confounded by the overall difficulty of the exam and the relative ability of the examinees, rendering a comparison invalid. examinees, rendering a comparison invalid.

This is particularly troublesome when we wish to This is particularly troublesome when we wish to • compare different cohorts of students compare different cohorts of students • maintain a consistent level of assessment difficulty over subsequent administrations. maintain a consistent level of assessment difficulty over subsequent administrations.

The problemThe problem

What is the best way to ensure that all passing students are reaching the What is the best way to ensure that all passing students are reaching the required level of expertise??required level of expertise??

2 forms of pass mark selection2 forms of pass mark selection• Criterion referencedCriterion referenced• Norm referencedNorm referenced

Criterion referenced refers to when a specific pass mark has been Criterion referenced refers to when a specific pass mark has been designated prior to the exam as a pass/fail point.designated prior to the exam as a pass/fail point.

Norm referenced refers to when a specific proportion of the sample will be Norm referenced refers to when a specific proportion of the sample will be designated to pass the exam. i.e. the highest scoring 75% of students will designated to pass the exam. i.e. the highest scoring 75% of students will passpass

Norm referenced or Criterion Testing?Norm referenced or Criterion Testing?

Norm-referenced testingNorm-referenced testing

Whatever the ability of the students taking the test, a fixed portion of them Whatever the ability of the students taking the test, a fixed portion of them will pass/failwill pass/fail

The standard needed to pass the test is not known in advanceThe standard needed to pass the test is not known in advance The validity of the norm-referencing method relies on a homogenous The validity of the norm-referencing method relies on a homogenous

sample, which may not necessarily be the case sample, which may not necessarily be the case There is also the risk that with a less able group of students, a student There is also the risk that with a less able group of students, a student

could pass the exam without reaching the desirable acceptable level of could pass the exam without reaching the desirable acceptable level of clinical knowledge. clinical knowledge.

Norm referencing is not appropriateNorm referencing is not appropriate

Norm referenced or Criterion Testing?Norm referenced or Criterion Testing?

Criterion testingCriterion testing

Relative difficulty of Exam could change depending on the items that are Relative difficulty of Exam could change depending on the items that are in the exam, therefore a pre determined pass mark could be easier or in the exam, therefore a pre determined pass mark could be easier or harder to obtain depending upon the items in the testharder to obtain depending upon the items in the test

Although having their own disadvantages, it has been recognised [Wass Although having their own disadvantages, it has been recognised [Wass et al (2001)] that Criterion referenced tests are the only acceptable means et al (2001)] that Criterion referenced tests are the only acceptable means of assessing that a pre-defined clinical competency has been reached of assessing that a pre-defined clinical competency has been reached

Solution?Solution?

It has been identified [Muijtjens (1998)] that a criterion referenced test It has been identified [Muijtjens (1998)] that a criterion referenced test could be utilised if a test was constructed by selecting items from a bank could be utilised if a test was constructed by selecting items from a bank of items of known difficulty, which would then enable measurement and of items of known difficulty, which would then enable measurement and control of the test difficulty.control of the test difficulty.

Difficulty estimates, as defined by classical test theory, are sample Difficulty estimates, as defined by classical test theory, are sample dependent!dependent!

Item BankingItem Banking

Item bankingItem banking

Item Banking is a process whereby all EMQ Items that have been Item Banking is a process whereby all EMQ Items that have been used over the past few years are ‘banked’ and calibrated onto the used over the past few years are ‘banked’ and calibrated onto the same metric scalesame metric scale

Previously used EMQ ItemsPreviously used EMQ Items

Psychometrically calibrated using Rasch AnalysisPsychometrically calibrated using Rasch Analysis

Data is linked by common items that have been used between the Data is linked by common items that have been used between the examsexams

• ““Common Item Equating”Common Item Equating”

Rasch AnalysisRasch Analysis

When data fit the model, generalisability of EMQ difficulties When data fit the model, generalisability of EMQ difficulties beyond the specific conditions under which they were observed beyond the specific conditions under which they were observed occurs (occurs (specific objectivityspecific objectivity ) )

In other words…In other words…

Item Difficulties are not sample dependent as they are in Item Difficulties are not sample dependent as they are in Classical Test TheoryClassical Test Theory

Item bankingItem banking

Term 1 Term 2 Term 3 Term 4

ITEM 1

ITEM 2

ITEM 3

ITEM 4

These Items cannot be directly compared as there are no These Items cannot be directly compared as there are no common linkscommon links

Item bankingItem banking

Term 1 Term 2 Term 3 Term 4

ITEM 1

ITEM 2

ITEM 3

ITEM 4

ITEM 5

These Items can be directly compared via the common link item These Items can be directly compared via the common link item across all termsacross all terms

Item bankingItem banking

Following calibration, the set of items within the bank will define Following calibration, the set of items within the bank will define the common domain that that they are measuring (in this case, the common domain that that they are measuring (in this case, medical knowledge). medical knowledge).

These items, therefore, will provide an operational definition of These items, therefore, will provide an operational definition of this unobservable (latent) trait. this unobservable (latent) trait.

When all EMQ Items are calibrated onto the same scale, then it When all EMQ Items are calibrated onto the same scale, then it will be possible to compare the performance of students across will be possible to compare the performance of students across terms, despite the fact that the EMQ exam content was made up terms, despite the fact that the EMQ exam content was made up of different items across each term.of different items across each term.

Sufficient Linkage?Sufficient Linkage?

What is classed as sufficient linkage between item sets?What is classed as sufficient linkage between item sets? There has been some variation in the literature regarding this.There has been some variation in the literature regarding this. Three differing viewpoints suggest that:Three differing viewpoints suggest that:

• linking items should be the larger of 20 items or 20% of the total number of linking items should be the larger of 20 items or 20% of the total number of items [Angoff (1971)]items [Angoff (1971)]

• 5 to 10 items are sufficient to form the common link [Wright & Bell (1984)]5 to 10 items are sufficient to form the common link [Wright & Bell (1984)]• one single common item one single common item couldcould provide a valid link in co-calibrating datasets provide a valid link in co-calibrating datasets

[Smith (1992)]. [Smith (1992)].

However, it has also been suggested [Smith (1992)] that the larger the However, it has also been suggested [Smith (1992)] that the larger the number of common items across datasets, the greater degree of number of common items across datasets, the greater degree of precision and stability for the item bank. precision and stability for the item bank.

Potential Problems?Potential Problems?

Sparse Data MatrixSparse Data Matrix

Limited LinkageLimited Linkage• Data overlap reducedData overlap reduced

Potential Misfit or DIF on Link Potential Misfit or DIF on Link ItemsItems

Term

1

Term

2

Term

3

Term

4

Term

5

Term

6

Term

7

Term

8

Q1Q2Q3Q4Q5Q6Q7Q8Q9Q10Q11Q12Q13Q14Q15Q16Q17Q18Q19Q20Q21Q22Q23Q24Q25

SampleSample

Data was collected from 550 4Data was collected from 550 4thth year medical students over 8 year medical students over 8 termsterms

All EMQ data was read in to a single analysis to establish the All EMQ data was read in to a single analysis to establish the Item BankItem Bank

RUMM2020 Rasch Analysis software was usedRUMM2020 Rasch Analysis software was used

Over the 8 terms, 6 different test forms were used (the test Over the 8 terms, 6 different test forms were used (the test remained the same over the first 3 terms)remained the same over the first 3 terms)

EMQ Item BankEMQ Item Bank

Th

eme

Co

de

Qu

esti

on

C

od

e

1 2 3 4 5 6

Th

eme

Co

de

Qu

esti

on

C

od

e

1 2 3 4 5 6

Th

eme

Co

de

Qu

esti

on

C

od

e

1 2 3 4 5 6

1 1 10 97 23 2281 2 10 98 24 2311 3 10 99 24 2321 4 11 101 24 2331 5 11 102 24 2341 6 11 103 24 2352 11 11 104 24 2362 12 12 111 24 2372 13 12 112 24 2382 14 12 113 24 2392 15 12 114 24 2402 16 13 121 25 2413 21 13 122 25 2423 22 13 123 25 2433 23 13 124 25 2443 24 14 131 25 2453 25 14 132 25 2463 26 14 133 25 2473 27 14 134 25 2483 28 15 141 25 2493 29 15 142 25 2504 31 15 143 26 2514 32 15 144 26 2524 33 15 145 26 2534 34 15 146 26 2544 35 16 151 26 2554 36 16 152 26 2564 37 16 153 27 2614 38 16 154 27 2624 39 17 161 27 2634 40 17 162 27 2645 41 17 163 27 2655 42 17 164 27 2665 43 17 165 27 2675 44 18 171 28 2715 45 18 172 28 2725 46 18 173 28 2735 47 18 174 28 2745 48 18 175 29 2816 51 18 176 29 2826 52 18 177 29 2836 53 19 181 29 2846 54 19 182 30 2916 55 19 183 30 2926 56 19 184 30 2936 57 19 185 30 2946 58 19 186 31 3016 59 20 191 31 3026 60 20 192 31 3036 1051 20 193 31 3047 61 20 194 31 3057 62 20 195 31 3067 63 20 196 31 3078 71 21 201 31 3088 72 21 202 32 3118 73 21 203 32 3128 74 21 204 33 3218 75 21 205 33 3229 81 22 211 33 3239 82 22 212 33 3249 83 22 213 34 3319 84 22 214 34 3329 85 22 215 34 3339 86 23 221 34 334

10 91 23 222 35 34110 92 23 223 35 34210 93 23 224 35 34310 94 23 22510 95 23 22610 96 23 227

• Approximately 25% of the items were changed from exam form-to-exam form

• This provides good linkage

EMQ Item BankEMQ Item Bank

Pretty Good!!

To be expected Low. But…

Low Person Separation IndexLow Person Separation Index

Person Separation Index is fairly lowPerson Separation Index is fairly low We would expect this to a certain degree due to the highly focussed sampleWe would expect this to a certain degree due to the highly focussed sample

Problematic ItemsProblematic Items

Approximately 12.5% (26/205) of Individual Approximately 12.5% (26/205) of Individual Items were found to display some sort of misfit or Items were found to display some sort of misfit or DIFDIF

MisfitMisfit

What does misfit tell us?What does misfit tell us?

Misfitting items are flagged up for non-inclusion and will either Misfitting items are flagged up for non-inclusion and will either be amended or removedbe amended or removed

Differential Item FunctioningDifferential Item Functioning

Misfit & DIFMisfit & DIF

What does misfit tell us?What does misfit tell us? Misfitting items are flagged up for non-inclusion and will either be Misfitting items are flagged up for non-inclusion and will either be

amended or removedamended or removed

DIF could be due to:DIF could be due to:• Curriculum changesCurriculum changes• Teaching RotationsTeaching Rotations• TestwisenessTestwiseness

Does Exam Difficulty remain equal over Does Exam Difficulty remain equal over different exam forms?different exam forms?

Example across item set using Exam Form 1 as Example across item set using Exam Form 1 as the Baselinethe Baseline

Measurement and the pass markMeasurement and the pass mark

Pass mark

EMQ Exam

Increasing student ability

60%

0 100

Minimum standard

The latent trait

The “real” assessment

Equivalent standard?Equivalent standard?

Exam Form 1 was based on 98 ItemsExam Form 1 was based on 98 Items Pass Mark was set at 60%Pass Mark was set at 60% 60% of 98 = Raw Score pass mark of 58.860% of 98 = Raw Score pass mark of 58.8

Equivalent pass mark?Equivalent pass mark?

Raw score of 58.8 on Exam Form 1 = 0.561 logitsRaw score of 58.8 on Exam Form 1 = 0.561 logits

Equating ProblemsEquating Problems

We had to remove 2 extreme itemsWe had to remove 2 extreme items

Exam form 1 was out of 98 anywayExam form 1 was out of 98 anyway

Exam FormExam Form Maximum Maximum obtainable obtainable scorescore

11 9898

22 9999

33 100100

44 100100

55 100100

66 9999

Equivalent standard?Equivalent standard?

Exam form 1 had 98 itemsExam form 1 had 98 items 60% of 98 = 58.860% of 98 = 58.8 58.8 score on Exam form 1 = 0.561 logits58.8 score on Exam form 1 = 0.561 logits

Exam Form 1 2 3 4 5 6

0.561 logit equated score 58.7 57.7 58.2 59.6 62.4 59.9

out of 98 99 100 100 100 99

59.90% 58.30% 58.20% 59.60% 62.40% 60.50%

62.4% - 58.2% = 4.2%62.4% - 58.2% = 4.2%

ConclusionConclusion

Item Banking is a way of assessing the psychometric properties of EMQs Item Banking is a way of assessing the psychometric properties of EMQs that have been administered over different test formsthat have been administered over different test forms

Can identify and adapt poor questionsCan identify and adapt poor questions

Can perform a comparative analysis of relative test form difficultyCan perform a comparative analysis of relative test form difficulty

Should the pass mark be amended every term?Should the pass mark be amended every term?

ReferencesReferences

1.1. Miller GE. The assessment of clinical skills/competence/performance. Academic Medicine Miller GE. The assessment of clinical skills/competence/performance. Academic Medicine 1990; 65: S63-71990; 65: S63-7

2.2. McHarg J, Bradley P, Chamberlain S, Ricketts C, Searle J, McLachlan JC. Assessment of McHarg J, Bradley P, Chamberlain S, Ricketts C, Searle J, McLachlan JC. Assessment of progress tests. Medical Education 2005; 39: 221-227.progress tests. Medical Education 2005; 39: 221-227.

3.3. Muijtjens AMM, Hoogenboom RJI, Verwijnen GM, Van der Vleuten CPM. Relative or absolute Muijtjens AMM, Hoogenboom RJI, Verwijnen GM, Van der Vleuten CPM. Relative or absolute standards in assessing medical knowledge using progress tests. Advances in Health Sciences standards in assessing medical knowledge using progress tests. Advances in Health Sciences Education 1998; 3: 81-87.Education 1998; 3: 81-87.

4.4. McManus IC, Mollon J, Duke OL, Vale JA. Changes in standard of candidates taking the McManus IC, Mollon J, Duke OL, Vale JA. Changes in standard of candidates taking the MRCP(UK) Part 1 examination, 1985 to 2002: Analysis of marker questions. BMC Medicine MRCP(UK) Part 1 examination, 1985 to 2002: Analysis of marker questions. BMC Medicine 2005; 3 (13). 2005; 3 (13).

5.5. Downing S M. Item response theory: applications of modern test theory in medical education. Downing S M. Item response theory: applications of modern test theory in medical education. Medical Education 2003; 37: 739-745Medical Education 2003; 37: 739-745

6.6. Wass V, Van der Vleuten C, Shatzer J, Jones R: Assessment of clinical competence. Wass V, Van der Vleuten C, Shatzer J, Jones R: Assessment of clinical competence. LancetLancet 2001, 357(9620): 945-942001, 357(9620): 945-94

7.7. Angoff W H. Scales, norming, and equivalent scores. In: Thorndike R L, editor. Educational Angoff W H. Scales, norming, and equivalent scores. In: Thorndike R L, editor. Educational Measurement. 2nd ed. Washington (DC): American Council on Education; 1971. p508-600Measurement. 2nd ed. Washington (DC): American Council on Education; 1971. p508-600

8.8. Wright B D, Bell S R. Item Banks: what, why, how. Journal of Educational Measurement 1984; Wright B D, Bell S R. Item Banks: what, why, how. Journal of Educational Measurement 1984; 21(4): 331-34521(4): 331-345

9.9. Smith R M. Applications of Rasch Measurement. Chicago: Mesa Press; 1992Smith R M. Applications of Rasch Measurement. Chicago: Mesa Press; 1992

New BookNew Book

Smith EV Jr. & Stone GE (Eds.). Criterion Referenced Testing: Smith EV Jr. & Stone GE (Eds.). Criterion Referenced Testing: Practice Analysis to Score Reporting Using Rasch Measurement Practice Analysis to Score Reporting Using Rasch Measurement Models. Maple Grove, Minnesota. JAM Press; 2009Models. Maple Grove, Minnesota. JAM Press; 2009

Contact DetailsContact Details

Mike Horton:Mike Horton: M.C.Horton@leeds.ac.uk

Matt Homer:Matt Homer: m.s.homer@leeds.ac.uk

Alan Tennant:Alan Tennant: A.Tennant@leeds.ac.uk

Website:Website: http://www.leeds.ac.uk/medicine/rehabmed/psychometric/

Course Date

Introductory

March 10-12 2010May 12-14 2010Sept 15-17 2010Dec 1-3 2010March 23-25 2011May 18-20 2011Sept 14-16 2011Nov 30-Dec 2 2011

Intermediate

May 17-19 2010Sept 20-22 2010Dec 6-8 2010May 23-25 2011Sept 19-21 2011Dec 5-7 2011

AdvancedSept 23-24 2010Sept 22-23 2011