MCQ’s 3: Aligning with Course Objectives
description
Transcript of MCQ’s 3: Aligning with Course Objectives
MCQ’s 3:MCQ’s 3:Aligning with Course Aligning with Course
ObjectivesObjectives
MCQ Check-listMCQ Check-list
This presentation will focus on:This presentation will focus on: Why we write themWhy we write them What type is bestWhat type is best What flaws are to be avoidedWhat flaws are to be avoided What content to testWhat content to test How to construct itHow to construct it
Why do we test?Why do we test?
Formative assessment Formative assessment Communicate to the students what material is Communicate to the students what material is
importantimportant Identify areas of deficiency in need of remediation Identify areas of deficiency in need of remediation
or further learningor further learning Summative assessmentSummative assessment
Determine final gradesDetermine final grades Make promotion decisionsMake promotion decisions
BothBoth Motivate students to studyMotivate students to study Identify areas where the course/curriculum is weakIdentify areas where the course/curriculum is weak
What Should Be Tested?What Should Be Tested?
Exam content should match course Exam content should match course objectivesobjectives
Important topics should be weighted Important topics should be weighted more heavily than less important more heavily than less important topicstopics
The testing time devoted to each The testing time devoted to each topic should reflect the relative topic should reflect the relative importance of the topicsimportance of the topics
What Should Be Tested?What Should Be Tested?
Sampling - Performance on the sample Sampling - Performance on the sample provides a basis for estimating provides a basis for estimating achievement in the broader domain that achievement in the broader domain that is actually of interestis actually of interest
Questions sample: Questions sample: TOPICS – eg MCC presentationsTOPICS – eg MCC presentations SKILLS – eg Course Objectives – determining SKILLS – eg Course Objectives – determining
diagnosis, next step in management, etc.diagnosis, next step in management, etc.
Term 4A, Jan 2012Term 4A, Jan 2012
Course Blueprint – Course Blueprint – ‘Alignment’‘Alignment’
Types of MCQ’sTypes of MCQ’s
True/False familyTrue/False family Assess recall of isolated factsAssess recall of isolated facts
Stems must be clear and unambiguousStems must be clear and unambiguous Options must be absolutely true or false or Options must be absolutely true or false or
examinee must decide examinee must decide how truehow true options are options are
Generally not recommended – usually Generally not recommended – usually have subtle flawshave subtle flaws
Types of MCQ’s: True/FalseTypes of MCQ’s: True/FalseWhich of the following statements is true regarding Which of the following statements is true regarding
contraception?contraception? A)A) oral contraceptives should not be initiated until oral contraceptives should not be initiated until
six weeks after therapeutic abortion to avoid the six weeks after therapeutic abortion to avoid the risk of hypercoagulabilityrisk of hypercoagulability
B)B) an intrauterine device can provide good an intrauterine device can provide good contraceptive protection 5 years after insertioncontraceptive protection 5 years after insertion
C)C) oral contraceptives should not be prescribed to oral contraceptives should not be prescribed to individuals under the age of 15 without parental individuals under the age of 15 without parental consentconsent
D)D) lesbian women do not require counseling about lesbian women do not require counseling about contraception or sexually transmitted diseasescontraception or sexually transmitted diseases
E)E) oral contraceptives should not be prescribed for oral contraceptives should not be prescribed for women overwomen over the age of 35the age of 35
Types of MCQ’sTypes of MCQ’sOne-Best-Answer family (A type)One-Best-Answer family (A type) Options may be diagrammed:Options may be diagrammed:
D C A E BD C A E B
Correct answer is the Correct answer is the most likelymost likely correct correct Better application of knowledge, Better application of knowledge,
integration, synthesis and judgmentintegration, synthesis and judgment
Least correct Most correct
Types of MCQ’s: Types of MCQ’s: One-Best-Answer familyOne-Best-Answer family
A 24 year old university student came to see you A 24 year old university student came to see you because she is missing several days of school each because she is missing several days of school each month because of severe dysmenorrhea unresponsive month because of severe dysmenorrhea unresponsive to the OCP and NSAIDS. She also complains of deep to the OCP and NSAIDS. She also complains of deep dyspareunia and has a tender uterosacral nodule on dyspareunia and has a tender uterosacral nodule on pelvic examination. Which one of the following pelvic examination. Which one of the following conditions is most likely responsible for her symptoms?conditions is most likely responsible for her symptoms?
A)A) Chronic pelvic inflammatory diseaseChronic pelvic inflammatory diseaseB)B) EndometriosisEndometriosisC)C) AdenomyosisAdenomyosisD)D) FibroidsFibroids
MCQ Check-listMCQ Check-list
Why we write themWhy we write them What type is bestWhat type is best What flaws are to be avoidedWhat flaws are to be avoided What content to testWhat content to test How to construct itHow to construct it
What to test?What to test?
Items can be classified byItems can be classified by cognitive processescognitive processes required to answer the question:required to answer the question:
Recall Recall (Memory)– tests knowledge of isolated (Memory)– tests knowledge of isolated factsfacts
InterpretationInterpretation (Comprehension)– review (Comprehension)– review information to reach conclusion, eg diagnosisinformation to reach conclusion, eg diagnosis
Problem solvingProblem solving (Reasoning) – situation (Reasoning) – situation requires action, eg next step in managementrequires action, eg next step in management Process depends on background of Process depends on background of
traineetrainee
What to test?What to test?Simpler classification based on taskSimpler classification based on task:: Application of knowledgeApplication of knowledge item item ****
Reach conclusionsReach conclusions Make predictionsMake predictions Select course of actionSelect course of action
Recall Recall itemitem Test knowledge of isolated factsTest knowledge of isolated facts
What to test?What to test?
Test application of knowledge using Test application of knowledge using clinical vignettesclinical vignettes Adds ‘face-validity’Adds ‘face-validity’ Usually selects more important, less trivialUsually selects more important, less trivial Identifies student who’ve memorized but Identifies student who’ve memorized but
do not understanddo not understand
BUT experience in patient care should BUT experience in patient care should not be necessarynot be necessary
What to test - summaryWhat to test - summary Focus item on Focus item on key conceptskey concepts and principles and principles
that are essential information (without that are essential information (without access to references) for all examinees to access to references) for all examinees to understandunderstand
Test material that is Test material that is relevantrelevant to learning in to learning in pre-clinical courses and beyondpre-clinical courses and beyond
AvoidAvoid items that items that onlyonly require require recall of recall of isolated factsisolated facts
Avoid esotericAvoid esoteric or interesting topics that are or interesting topics that are not essentialnot essential
What to test - summaryWhat to test - summary Test application of knowledge using Test application of knowledge using
clinical vignettesclinical vignettes to pose medical to pose medical decisions in patient care situationsdecisions in patient care situations
Focus items on Focus items on common or common or potentially catastrophic problemspotentially catastrophic problems
Pose clinical decisions that would be Pose clinical decisions that would be expected of a successful examinee expected of a successful examinee ie Avoid clinical situations that would be ie Avoid clinical situations that would be
handled by a (sub)specialisthandled by a (sub)specialist
Writing the QuestionsWriting the Questions
Construct stem –include all Construct stem –include all necessary information to arrive at necessary information to arrive at the right answerthe right answer
Choose lead-in – pose a clear Choose lead-in – pose a clear questionquestion
Choose distractorsChoose distractors
Stem*Stem*
Usually a clinical scenario Usually a clinical scenario Clear, unambiguousClear, unambiguous Should be long relative to optionsShould be long relative to options Includes all pertinent informationIncludes all pertinent information
Patient’s age/genderPatient’s age/gender Clinical settingClinical setting ComplaintComplaint Other important info – history, physical Other important info – history, physical
findings, test resultsfindings, test results
Lead-in*Lead-in*
Follows stemFollows stem In the form of a questionIn the form of a question Should relate to the stemShould relate to the stem Should be answerable without looking at Should be answerable without looking at
optionsoptions ““Cover the Options test”Cover the Options test”
Should clearly communicate the learning Should clearly communicate the learning objective objective
DistractorsDistractors
Each should be selected by some, Each should be selected by some, therefore all plausible, none therefore all plausible, none obviously incorrectobviously incorrect
3 or 4 choices3 or 4 choices Common misconceptionsCommon misconceptions Faulty reasoningFaulty reasoning
DistractorsDistractors
Homogeneous in content to the correct answerHomogeneous in content to the correct answer Should be in same category as correct Should be in same category as correct
answer– diagnosis, treatment, diagnostic testanswer– diagnosis, treatment, diagnostic test Clearly incorrect or inferior to the correct Clearly incorrect or inferior to the correct
answeranswer Plausible and attractive to the uninformedPlausible and attractive to the uninformed Similar to the correct answer in construction Similar to the correct answer in construction
and length and length Grammatically consistent and logically Grammatically consistent and logically
compatible with the stemcompatible with the stem
General Guidelines for Item General Guidelines for Item ConstructionConstruction Include as much of the item as possible in the Include as much of the item as possible in the
stems: Long stems, short optionsstems: Long stems, short options Avoid negatively phrased items (Avoid negatively phrased items (exceptexcept or or
notnot in the lead-in). If you must, use only with in the lead-in). If you must, use only with short optionsshort options
Avoid writing questions of the form:Avoid writing questions of the form: Which of the following statements is correct?Which of the following statements is correct? Each of the following statements is correct EXCEPTEach of the following statements is correct EXCEPT
Unfocussed Unfocussed Heterogeneous optionsHeterogeneous options
““Top-down” versus “Bottom-Top-down” versus “Bottom-up”up”
Top-DownTop-Down Knowledge or recall question Knowledge or recall question Cite a disease and then ask what Cite a disease and then ask what
patient findings are expectedpatient findings are expected Structured like textbooksStructured like textbooks
Clinically backwardClinically backward
““Top-down” versus “Bottom-Top-down” versus “Bottom-up”up”
Bottom-upBottom-up Application of knowledgeApplication of knowledge Gives findings and asks examinee to Gives findings and asks examinee to
indicate the diseaseindicate the disease Examinees need to be able to synthesize Examinees need to be able to synthesize
information from several pages of a information from several pages of a textbook (or notes) to answer these textbook (or notes) to answer these questionsquestionsClinically realisticClinically realistic
Basic Rules for MCQ’s (A type)*Basic Rules for MCQ’s (A type)*
Focus item on important conceptFocus item on important concept Item should assess application of Item should assess application of
knowledge, not recall of an isolated factknowledge, not recall of an isolated fact Stem must pose clear question – ‘Cover Stem must pose clear question – ‘Cover
the options’ testthe options’ test All distractors should be homogeneousAll distractors should be homogeneous Avoid technical item flawsAvoid technical item flaws
ReferenceReferenceConstructing Written Test Questions Constructing Written Test Questions
For the Basic and Clinical SciencesFor the Basic and Clinical Sciences Case & Swanson, NBME 1996Case & Swanson, NBME 1996
Clinical PresentationClinical Presentation
A 28 year old female presents to the ER with a 2 A 28 year old female presents to the ER with a 2 hour history of severe right sided abdominal hour history of severe right sided abdominal pain and vomiting. pain and vomiting.
Clinical PresentationClinical Presentation
A 28 year old female presents to the ER with a 2 A 28 year old female presents to the ER with a 2 hour history of severe right sided abdominal hour history of severe right sided abdominal pain and vomiting. pain and vomiting.
Clinical presentationClinical presentation
A 28 year old female presents to the ER with a 2 A 28 year old female presents to the ER with a 2 hour history of severe right sided abdominal hour history of severe right sided abdominal pain and vomiting. This was preceded by a 1 pain and vomiting. This was preceded by a 1 week history of urinary frequency and urgency. week history of urinary frequency and urgency. On physical examination, VSS are stable and On physical examination, VSS are stable and she is afebrile. Abdomen is non-tender. She is she is afebrile. Abdomen is non-tender. She is noted to have mild Right CVA tenderness. noted to have mild Right CVA tenderness.
Clinical PresentationClinical Presentation
1) A 28 year old female presents to the ER with a 2 hour 1) A 28 year old female presents to the ER with a 2 hour history of severe right sided abdominal pain and history of severe right sided abdominal pain and vomiting. This was preceded by a 1 week history of vomiting. This was preceded by a 1 week history of urinary frequency and urgency. On physical urinary frequency and urgency. On physical examination, VSS are stable and she is afebrile. examination, VSS are stable and she is afebrile. Abdomen is non-tender. She is noted to have mild Abdomen is non-tender. She is noted to have mild Right CVA tenderness. Right CVA tenderness. Which of the following Which of the following investigations would be the next step to make a investigations would be the next step to make a diagnosis?diagnosis?a) abdominal X-raya) abdominal X-rayb) abdo/pelvic USSb) abdo/pelvic USSc) urinalysisc) urinalysisd) CBCd) CBC
PresentationPresentation
2) A 28 year old female presents to the ER with a 2 2) A 28 year old female presents to the ER with a 2 hour history of severe right sided abdominal pain hour history of severe right sided abdominal pain and vomiting. This was preceded by a 1 week and vomiting. This was preceded by a 1 week history of urinary frequency and urgency. On history of urinary frequency and urgency. On physical examination, VSS are stable and she is physical examination, VSS are stable and she is afebrile. Abdomen is non-tender. She is noted to afebrile. Abdomen is non-tender. She is noted to have mild Right CVA tenderness. have mild Right CVA tenderness. Urinalysis Urinalysis reveals +++ blood, + WBC and nitrites neg. CBC reveals +++ blood, + WBC and nitrites neg. CBC is normal. Which of the following is the most likely is normal. Which of the following is the most likely diagnosis?diagnosis?
a) appendicitis a) appendicitis b) pyelonephritisb) pyelonephritisc) nephrolithiasisc) nephrolithiasisd) cystitisd) cystitis
PresentationPresentation
3) A 28 year old female presents to the ER with a 2 hour 3) A 28 year old female presents to the ER with a 2 hour history of severe right sided abdominal pain, preceded history of severe right sided abdominal pain, preceded by urinary frequency and urgency. VSS are stable and by urinary frequency and urgency. VSS are stable and she is afebrile. She is noted to have mild Right CVA she is afebrile. She is noted to have mild Right CVA tenderness. Urinalysis reveals +++ blood, + WBC, tenderness. Urinalysis reveals +++ blood, + WBC, nitrites neg. nitrites neg. You suspect nephrolithiasis. Which of the You suspect nephrolithiasis. Which of the following investigations is the best choice to confirm the following investigations is the best choice to confirm the diagnosis?diagnosis?
a) Abdominal and pelvic USSa) Abdominal and pelvic USSb) abdominal x-rayb) abdominal x-rayc) CT scanc) CT scand) IVPd) IVP
PresentationPresentation
4) A 28 year old female presents to the ER with a 4) A 28 year old female presents to the ER with a 2 hour history of severe ride sided abdominal 2 hour history of severe ride sided abdominal pain with R CVA tenderness. A CT scan confirms pain with R CVA tenderness. A CT scan confirms the presence of a R UVJ stone measuring the presence of a R UVJ stone measuring approximately 3 mm with complete ureteric approximately 3 mm with complete ureteric obstruction. There are no signs of Urinary Tract obstruction. There are no signs of Urinary Tract infection. You provide analgesia. infection. You provide analgesia. Which of the Which of the following is the next step in managing this following is the next step in managing this problem?problem?a) Hydration a) Hydration b) Percutaneous nephrostomyb) Percutaneous nephrostomyc) Cystoscopy and basket retrieval of stonec) Cystoscopy and basket retrieval of stoned) Lithotripsyd) Lithotripsy
PresentationPresentation
5) A 28 year old female has recently recovered 5) A 28 year old female has recently recovered from her third bout of renal colic. She is from her third bout of renal colic. She is otherwise healthy. otherwise healthy. Which of the following Which of the following investigations would be most helpful in finding investigations would be most helpful in finding the cause of her recurrent nephrolithiasis?the cause of her recurrent nephrolithiasis?
a) IVPa) IVPb) CT scanb) CT scanc) serum Calciumc) serum Calciumd) 24 hour urine calciumd) 24 hour urine calcium
PresentationPresentation
6) A 28 year old female has recently recovered from 6) A 28 year old female has recently recovered from her third bout of renal colic. She is otherwise her third bout of renal colic. She is otherwise healthy. Investigations reveal no evidence of healthy. Investigations reveal no evidence of hyperparathyroidism, urinary tract anomaly or hyperparathyroidism, urinary tract anomaly or other underlying condition. other underlying condition. Which of the following Which of the following would be most helpful in preventing this from would be most helpful in preventing this from recurring?recurring?
a) hydrochlorothiazidea) hydrochlorothiazideb) Flomaxb) Flomaxc) cranberry juicec) cranberry juiced) 8 glasses of water dailyd) 8 glasses of water daily
Basic Rules for MCQ’s (A type)*Basic Rules for MCQ’s (A type)*
Focus item on important conceptFocus item on important concept Item should assess application of Item should assess application of
knowledge, not recall of an isolated factknowledge, not recall of an isolated fact Stem must pose clear question – ‘Cover Stem must pose clear question – ‘Cover
the options’ testthe options’ test All distractors should be homogeneousAll distractors should be homogeneous Avoid technical item flawsAvoid technical item flaws
One last step…One last step… Get a colleague to read over your Get a colleague to read over your
questions questions Detects more than one right answerDetects more than one right answer Discovers the ‘what am I thinking?’ Discovers the ‘what am I thinking?’
questionquestion Identifies not enough information in the Identifies not enough information in the
stemstem ……