Effects of Question Formats on Student and Item Performance
Transcript of Effects of Question Formats on Student and Item Performance
The fundamental problem in shock is:a. Decreased diastolic blood pressureb. Decreased systolic blood pressurec. Decreased tissue perfusiond. Increased serum lactic acid
Example
If you find a patient in shock, your first action should be:a. Begin CPRb. Call 911c. Circulation, airway, breathing assessmentd. Find an AED
Example
Haladyna, Downing, & Rodriguez
• Developed from an analysis of 27 textbooks and 27 research studies
• Purpose was to validate each guideline based on agreement in studied sources
Haladyna TM, Downing SM, Rodriguez SM. Applied Measurement in Education. 2002; 15(3):309-334.
Which of the following is the capital of Texas?a. Austin*b. Dallasc. El Pasod. Houston
Anatomy of a MC question
Guidelines: a closer look
Guideline For (%) Uncited (%) Against (%)
Use positives, no negatives 63 19 18
Write as many plausible distractors as you can 70 26 4
Use None of the Above carefully 44 7 48
Avoid All of the Above 70 7 22
All of the following drugs are metabolized by the P450 enzyme system except:a. Atazanavirb. Darunavirc. Raltegravird. Tipranavir
Use positives, no negatives
Haladyna and Downing (1993)• Two-thirds of all items had only 1-2
effectively performing distractors• Only 1-8% of items had three
effective distractors• No measured effect on difficulty, but
more effective distractors = more discrimination
Write as many plausible distractors as possible
Haladyna TM, Downing SM. Education and Psychological Measurement. 1993; 53:999-1010.
Which of the following is the capital of Texas?A. DallasB. El PasoC. HoustonD. LubbockE. None of the above
Use “none of the above” carefully
Which of the following is true regarding transdermal NRT?A. These products may be more appropriate in patients who anticipate
adherence problemsB. The 16-hr patch reduces the risk of insomnia and vivid dreaming
compared to the 24-hrC. Currently there is insufficient data to directly compare transdermal
NRT to oral NRTD. Transdermal NRT patches should never be cutE. All of the above
Avoid “all of the above”
50
60
70
80
90
100
Difficulty (p)
StandardFlawed
A retrospective study
0
0.2
0.4
0.6
0.8
1
Discrimination (rpb)
p=0.01 p=0.64
!
48%
52%
Guideline # Times broken % Total
Minimize reading 24 16.9
All of the above 24 16.9
Complex item format 19 13.38
Central idea in stem 17 11.97
Edit and proof 10 7.04
Choices homogenous 8 5.63
Plausible distractors 8 5.63
Trivial Content 6 4.23Length of choices equal 6 4.23
None of the above 6 4.23
Other 13 9.86
Ratio Standard:Flawed
Pate AN, Caldwell DJ. CPTL. 2014; 6(1):130-134.
A prospective study5 authors
Student randomizationstandard, n=55
flawed, n=54
Test administration
Analysis: one-sidedindependent t-tests
between the two scalesMean
discrimination, rpbMean itemdifficulty, p
40 exam items
25
35
45
55
65
75
Difficulty (p)
Standard Flawed
p=0.040
0.1
0.2
0.3
0.4
0.5
Discrimination (rpb)
p=0.44
Caldwell DJ, Pate AN. J Pharm Ed. 2013; 77(4):Article 71.
Investigator review
Test form 120 standard items
Test form 220 flawed items