Achieve your 2017 Assessment Resolutions: How to Improve Your Item Writing Skills and Create Better...
Transcript of Achieve your 2017 Assessment Resolutions: How to Improve Your Item Writing Skills and Create Better...
![Page 1: Achieve your 2017 Assessment Resolutions: How to Improve Your Item Writing Skills and Create Better Exams](https://reader035.fdocuments.in/reader035/viewer/2022070512/589dc0661a28abf7288b6399/html5/thumbnails/1.jpg)
Achieve Your 2017 Resolution to Improve Your Test Item Writing Skills and Create Better Exams Ainslie T. Nibert, PhD, RN, FAAN
Consultant
Email – [email protected]
![Page 2: Achieve your 2017 Assessment Resolutions: How to Improve Your Item Writing Skills and Create Better Exams](https://reader035.fdocuments.in/reader035/viewer/2022070512/589dc0661a28abf7288b6399/html5/thumbnails/2.jpg)
Objectives Create/Edit test items that assess for application and analysis.
Critique exams for alignment with the NCLEX-RN® Test Plan.
Evaluate the relevance and effectiveness of current testing policies.
![Page 3: Achieve your 2017 Assessment Resolutions: How to Improve Your Item Writing Skills and Create Better Exams](https://reader035.fdocuments.in/reader035/viewer/2022070512/589dc0661a28abf7288b6399/html5/thumbnails/3.jpg)
Resources for Developing Critical Thinking Test Items and Alternate
Format Items:National Council Website
www.nscbn.org◦ NCLEX Test Plans
◦ 2016 RN◦ Candidate FAQ◦ Alternate item formats FAQ◦ Exam Development FAQ
Source: https://www.ncsbn.org/9010.htm
![Page 4: Achieve your 2017 Assessment Resolutions: How to Improve Your Item Writing Skills and Create Better Exams](https://reader035.fdocuments.in/reader035/viewer/2022070512/589dc0661a28abf7288b6399/html5/thumbnails/4.jpg)
Relationship betweenTesting & the Curriculum
Focus today: INTERNAL Curriculum
Evaluation(Teacher-made Tests)
![Page 5: Achieve your 2017 Assessment Resolutions: How to Improve Your Item Writing Skills and Create Better Exams](https://reader035.fdocuments.in/reader035/viewer/2022070512/589dc0661a28abf7288b6399/html5/thumbnails/5.jpg)
Writing Critical Thinking Test Items Item Analysis Software & Blueprinting Test Item Banking & Exam Delivery
Internal EvaluationEvaluation of course objectives (faculty designed)
![Page 6: Achieve your 2017 Assessment Resolutions: How to Improve Your Item Writing Skills and Create Better Exams](https://reader035.fdocuments.in/reader035/viewer/2022070512/589dc0661a28abf7288b6399/html5/thumbnails/6.jpg)
Five Guidelines to Developing Effective Critical Thinking Exams
Assemble the “basics.” Write critical thinking test items. Pay attention to housekeeping duties. Develop a test blueprint. Scientifically analyze all exams.
![Page 7: Achieve your 2017 Assessment Resolutions: How to Improve Your Item Writing Skills and Create Better Exams](https://reader035.fdocuments.in/reader035/viewer/2022070512/589dc0661a28abf7288b6399/html5/thumbnails/7.jpg)
Critical Thinking Test Items Contain Rationale Written at the Application Level or Above Require Multilogical Thinking to Answer Ask for High Level of Discrimination
Source: Morrison, Nibert, & Flick (2006)
![Page 8: Achieve your 2017 Assessment Resolutions: How to Improve Your Item Writing Skills and Create Better Exams](https://reader035.fdocuments.in/reader035/viewer/2022070512/589dc0661a28abf7288b6399/html5/thumbnails/8.jpg)
Housekeeping Tips
![Page 9: Achieve your 2017 Assessment Resolutions: How to Improve Your Item Writing Skills and Create Better Exams](https://reader035.fdocuments.in/reader035/viewer/2022070512/589dc0661a28abf7288b6399/html5/thumbnails/9.jpg)
Rules Get rid of names Get rid of ‘multiple’ multiples Use non-sexist writing style Develop parsimonious writing style
Eliminate Delete scenarios
Write items independent of each other
“of the following…”X
![Page 10: Achieve your 2017 Assessment Resolutions: How to Improve Your Item Writing Skills and Create Better Exams](https://reader035.fdocuments.in/reader035/viewer/2022070512/589dc0661a28abf7288b6399/html5/thumbnails/10.jpg)
… and More Rules
Use a question format when possible Make distracters plausible and homogeneous Include in stem words repeated in responses
![Page 11: Achieve your 2017 Assessment Resolutions: How to Improve Your Item Writing Skills and Create Better Exams](https://reader035.fdocuments.in/reader035/viewer/2022070512/589dc0661a28abf7288b6399/html5/thumbnails/11.jpg)
… and More Rules Eliminate “all of the above” and “none of the above” Rewrite any “all except” questions Ensure that alternatives do not overlap Present choices in a logical order Vary correct answer
![Page 12: Achieve your 2017 Assessment Resolutions: How to Improve Your Item Writing Skills and Create Better Exams](https://reader035.fdocuments.in/reader035/viewer/2022070512/589dc0661a28abf7288b6399/html5/thumbnails/12.jpg)
… and the MOST IMPORTANT Rule
Develop written testing policy Writing style Format
![Page 13: Achieve your 2017 Assessment Resolutions: How to Improve Your Item Writing Skills and Create Better Exams](https://reader035.fdocuments.in/reader035/viewer/2022070512/589dc0661a28abf7288b6399/html5/thumbnails/13.jpg)
Does the test measure what it claims to
measure?
Content ValidityContent Validity
![Page 14: Achieve your 2017 Assessment Resolutions: How to Improve Your Item Writing Skills and Create Better Exams](https://reader035.fdocuments.in/reader035/viewer/2022070512/589dc0661a28abf7288b6399/html5/thumbnails/14.jpg)
Use a Blueprint to Assess a Test’s Validity
Test Blueprint Reflects Course Objectives Rational/Logical Tool
Testing Software Program Storage of item analysis data (Last & Cumulative) Storage of test item categories
![Page 15: Achieve your 2017 Assessment Resolutions: How to Improve Your Item Writing Skills and Create Better Exams](https://reader035.fdocuments.in/reader035/viewer/2022070512/589dc0661a28abf7288b6399/html5/thumbnails/15.jpg)
![Page 16: Achieve your 2017 Assessment Resolutions: How to Improve Your Item Writing Skills and Create Better Exams](https://reader035.fdocuments.in/reader035/viewer/2022070512/589dc0661a28abf7288b6399/html5/thumbnails/16.jpg)
![Page 17: Achieve your 2017 Assessment Resolutions: How to Improve Your Item Writing Skills and Create Better Exams](https://reader035.fdocuments.in/reader035/viewer/2022070512/589dc0661a28abf7288b6399/html5/thumbnails/17.jpg)
![Page 18: Achieve your 2017 Assessment Resolutions: How to Improve Your Item Writing Skills and Create Better Exams](https://reader035.fdocuments.in/reader035/viewer/2022070512/589dc0661a28abf7288b6399/html5/thumbnails/18.jpg)
Consistency of Scores
![Page 19: Achieve your 2017 Assessment Resolutions: How to Improve Your Item Writing Skills and Create Better Exams](https://reader035.fdocuments.in/reader035/viewer/2022070512/589dc0661a28abf7288b6399/html5/thumbnails/19.jpg)
Reliability Tools
Kuder-Richardson Formula 20 (KR20)—EXAMRange from –1 to + 1
Point Biserial Correlation Coefficient (PBCC)—TEST ITEMS
Range from – 1 to + 1
![Page 20: Achieve your 2017 Assessment Resolutions: How to Improve Your Item Writing Skills and Create Better Exams](https://reader035.fdocuments.in/reader035/viewer/2022070512/589dc0661a28abf7288b6399/html5/thumbnails/20.jpg)
Item difficulty 30% - 90%
Item Discrimination Ratio 25% and Above
PBCC 0.20 and Above
KR20 0.70 and Above
Standards of Acceptance
![Page 21: Achieve your 2017 Assessment Resolutions: How to Improve Your Item Writing Skills and Create Better Exams](https://reader035.fdocuments.in/reader035/viewer/2022070512/589dc0661a28abf7288b6399/html5/thumbnails/21.jpg)
one “absolute” rule about item difficulty
TEST ITEMS ANSWERED CORRECTLY BY 30% or LESS of the examinees should always be considered too difficult, and the instructor must take action.
![Page 22: Achieve your 2017 Assessment Resolutions: How to Improve Your Item Writing Skills and Create Better Exams](https://reader035.fdocuments.in/reader035/viewer/2022070512/589dc0661a28abf7288b6399/html5/thumbnails/22.jpg)
…but what about high difficulty levels?Test items with high difficulty levels (>90%) often yield poor discrimination values.Is there a situation where faculty can legitimately expect that 100% of the class will answer a test item correctly, and be pleased when this happens?RULE OF THUMB ABOUT MASTERY ITEMS: Due to their negative impact on test discrimination and reliability, they should comprise no more than 10% of the test.
![Page 23: Achieve your 2017 Assessment Resolutions: How to Improve Your Item Writing Skills and Create Better Exams](https://reader035.fdocuments.in/reader035/viewer/2022070512/589dc0661a28abf7288b6399/html5/thumbnails/23.jpg)
Item difficulty 30% - 90%
Item Discrimination Ratio 25% and Above
PBCC 0.20 and Above
KR20 0.70 and Above
Standards of Acceptance
![Page 24: Achieve your 2017 Assessment Resolutions: How to Improve Your Item Writing Skills and Create Better Exams](https://reader035.fdocuments.in/reader035/viewer/2022070512/589dc0661a28abf7288b6399/html5/thumbnails/24.jpg)
Thinking more about item discrimination statistics on teacher-made tests…IDR can be calculated quickly, but doesn’t effectively consider variance of the entire group. Use it to quickly identify items that have zero/negative discrimination values, since these need to be edited before using again.PBCC is a more powerful measure discrimination.
Correlates the correct answer to a single test items with the total test score of the student.
Considers the variance of the entire student group, not just the lower and upper 27% groups.
For a small ‘n,’ consider referencing the cumulative value.
![Page 25: Achieve your 2017 Assessment Resolutions: How to Improve Your Item Writing Skills and Create Better Exams](https://reader035.fdocuments.in/reader035/viewer/2022070512/589dc0661a28abf7288b6399/html5/thumbnails/25.jpg)
… what decisions need to be made about Test items?
When a test item has poor difficulty and/or discrimination values, action is needed.All of these actions require that the exam be
rescored.Credit can be given for more than one choice.Test item can be nullified.Test item can be deleted.
REMEMBER: Each of these actions has a consequence, so faculty need to carefully consider these when choosing an action. Faculty judgment is crucial when determining actions affecting test scores.
![Page 26: Achieve your 2017 Assessment Resolutions: How to Improve Your Item Writing Skills and Create Better Exams](https://reader035.fdocuments.in/reader035/viewer/2022070512/589dc0661a28abf7288b6399/html5/thumbnails/26.jpg)
Standards of Acceptance Nursing
Nursing-PBCC 0.15 and Above
Nursing-KR20 0.60 - 0.65 and Above
![Page 27: Achieve your 2017 Assessment Resolutions: How to Improve Your Item Writing Skills and Create Better Exams](https://reader035.fdocuments.in/reader035/viewer/2022070512/589dc0661a28abf7288b6399/html5/thumbnails/27.jpg)
3-Step Method forItem Analysis
1. Review Difficulty Level2. Review Discrimination Data
Item Discrimination Ratio (IDR) Point Biserial Correlation Coefficient
(PBCC)3. Review Effectiveness of Alternatives
Response Frequencies Non-distracters
![Page 28: Achieve your 2017 Assessment Resolutions: How to Improve Your Item Writing Skills and Create Better Exams](https://reader035.fdocuments.in/reader035/viewer/2022070512/589dc0661a28abf7288b6399/html5/thumbnails/28.jpg)
..and a word about using Response Frequencies
A review of the response frequency data can focus your editing.
For items where 100% of students answer correctly, and no other options were chosen, make sure that this is indeed intentional (MASTERY ITEM), and not just reflective of an item that is too easy (>90% DIFFICULTY.)
Target re-writing the “zero” distracters – those options that are ignored by students. Replacing “zeros” with plausible options will immediately improve item DISCRIMINATION.
![Page 29: Achieve your 2017 Assessment Resolutions: How to Improve Your Item Writing Skills and Create Better Exams](https://reader035.fdocuments.in/reader035/viewer/2022070512/589dc0661a28abf7288b6399/html5/thumbnails/29.jpg)
![Page 30: Achieve your 2017 Assessment Resolutions: How to Improve Your Item Writing Skills and Create Better Exams](https://reader035.fdocuments.in/reader035/viewer/2022070512/589dc0661a28abf7288b6399/html5/thumbnails/30.jpg)
![Page 31: Achieve your 2017 Assessment Resolutions: How to Improve Your Item Writing Skills and Create Better Exams](https://reader035.fdocuments.in/reader035/viewer/2022070512/589dc0661a28abf7288b6399/html5/thumbnails/31.jpg)
![Page 32: Achieve your 2017 Assessment Resolutions: How to Improve Your Item Writing Skills and Create Better Exams](https://reader035.fdocuments.in/reader035/viewer/2022070512/589dc0661a28abf7288b6399/html5/thumbnails/32.jpg)
Critically-thinking Questions
Which intervention is most important?Which intervention, plan, assessment data is/are most critical to developing a plan of care?
Which intervention should be done first?
What action should the nurse take first?Which intervention, plan, nursing action has the highest priority?
What response is best?
![Page 33: Achieve your 2017 Assessment Resolutions: How to Improve Your Item Writing Skills and Create Better Exams](https://reader035.fdocuments.in/reader035/viewer/2022070512/589dc0661a28abf7288b6399/html5/thumbnails/33.jpg)
33
![Page 34: Achieve your 2017 Assessment Resolutions: How to Improve Your Item Writing Skills and Create Better Exams](https://reader035.fdocuments.in/reader035/viewer/2022070512/589dc0661a28abf7288b6399/html5/thumbnails/34.jpg)
Fair/Common Universal Language
The client is running late for an appointment. The client understands Buddhist practices are peaceful. The client is on five different medications. The client ate a submarine sandwich. The alcoholic client with delirium tremens is agitated. After the client sneezed, the nurse said “bless you.” The nurse is giving a report on the client. The nursing unit is working shorthanded.
Source: Bristol, T. 2016) NCLEX® Updates (webinar series) Available: http://nursetim.com/webinars/nclex
![Page 35: Achieve your 2017 Assessment Resolutions: How to Improve Your Item Writing Skills and Create Better Exams](https://reader035.fdocuments.in/reader035/viewer/2022070512/589dc0661a28abf7288b6399/html5/thumbnails/35.jpg)
![Page 36: Achieve your 2017 Assessment Resolutions: How to Improve Your Item Writing Skills and Create Better Exams](https://reader035.fdocuments.in/reader035/viewer/2022070512/589dc0661a28abf7288b6399/html5/thumbnails/36.jpg)
Latest NCLEX® Test Item Format
Considerations Units of Measure•International Systems of Units (SI)•Metric•Imperial Measurement
Generic vs. Trade Names for Medications•Generic names only in most cases•References to general classifications of medications
![Page 37: Achieve your 2017 Assessment Resolutions: How to Improve Your Item Writing Skills and Create Better Exams](https://reader035.fdocuments.in/reader035/viewer/2022070512/589dc0661a28abf7288b6399/html5/thumbnails/37.jpg)
Item Writing Tools for Success …
Knowledge
Test Blueprint
Testing Software
![Page 38: Achieve your 2017 Assessment Resolutions: How to Improve Your Item Writing Skills and Create Better Exams](https://reader035.fdocuments.in/reader035/viewer/2022070512/589dc0661a28abf7288b6399/html5/thumbnails/38.jpg)
ReferencesMorrison, S., Nibert, A., & Flick, J. (2006). Critical thinking and test item writing (2nd ed.). Houston, TX: Health Education Systems, Inc.
National Council of State Boards of Nursing. (2016) 2016 NCLEX-RN test plan. Chicago, IL: National Council of State Boards of Nursing. https://www.ncsbn.org/RN_Test_Plan_2016_Final.pdf
Nibert, A. (2010) Benchmarking for student progression throughout a nursing program: Implications for students, faculty, and administrators. In Caputi, L. (Ed.), Teaching nursing: The art and science, 2nd ed. (Vol. 3). (pp.45-64). Chicago: College of DuPage Press.