MC Item Writing Workshop

Nursing Workshop 2/12/2013

Educational

TESSERACT

Multiple Choice Item Writing Workshop

Doug Mauger EdDSchool Of Instructor Educationdmauger@vcc.ca

Workshop Agenda

• Introductions

• Overview

• Perspectives and Attitudes

• Stakeholders

• Validity and Reliability

• Alignment

• Classifying skills and outcomes

• Best Practices in MC item writing

• Table of Specification or Blueprint

• Take Away

• Next Workshop

Introductions

• Name:

• What topics do you teach?

• How long have you been teaching?

• One thing you know about Item writing

• What do you want to get out of the workshop?

Workshop Agenda

• Introductions

• Overview

• Perspectives and Attitudes• Stakeholders

• Valdity and Reliability

• Alignment

• Take Away

• Next Workshop

Put each rider on the top

of a horse

What are your attitudes to creating exams?

Workshop Agenda

• Introductions

• Overview

• Stakeholders• Validity and Reliability

• Alignment

• Take Away

• Next Workshop

Stakeholders Gathering

Workshop Agenda

• Introductions

• Overview

• Stakeholders

• Validity and Reliability• Alignment

• Take Away

• Next Workshop

Instrument……Confidence

There were flaws in 46% of the total questions, with the number of flaws per test ranging from 36% to 65%. In a sample of examinations distributed to first and second year medical students(Downing, 2005)

Downing, S. M., 2005. The effects of violating standard item writing principles on tests and students: the consequences of using flawed test items on achievement examinations in medical education. Advances in Health Science Education 10 (2), 133–143.

Points to Ponder

Process ValidityActivity

Validity and Reliability

Reliable but Not Valid

NOT Valid and NOT Reliable

Valid and Reliable

Validity Content

Predictive

Construct

Consequences

Process

Reliable Test – Retest

Alternate Form

Inter Rater

Types of ValidityActivity

1. Process, Content Validity (inadequate sample)

2. Process Validity (stress)

3. Content Validity (inadequate preparation)

4. Predictive Validity (inconsistent results)

5. Consequence Validity (induced anxiety)

6. Reliability/Process Validity (possible bias)

7. Process Validity (students not able to provide best performance)

8. Content Validity (different instructor’s test reflects different instruction)

9. Construct Validity (improper form of assessment)

10. Content Validity/Alignment (instruction and assessment do not match)

Why is Validity and Reliability important to you as a professional?

Workshop Agenda

• Introductions

• Overview

• Stakeholders

• Validity and Reliability

• Alignment• Classifying skills and outcomes

• Take Away

• Next Workshop

Alignment

Content and Assessment

Alignment

Are Critical

For Valid

Assessment

Intended

Curriculum

WHAT we plan

on teaching

Implemented

Curriculum

HOW we teach

the curriculum

Achieved

CurriculumAssessment

Evidence of

Student

Learning

Assessment

Perfect Alignment

HowWhat

Outcomes: Intended Implemented Achieved

Assessment

Scenario 1

tHowWhat

Assessment

Scenario 2

tHowWhat

Assessment

Scenario 3

tHowWhat

Assessment

Scenario 4

tHowWhat

Assessment

Scenario 5

tHowWhat

Assessment

Scenario 6

tHowWhat

Assessment

Scenario 7

tHowWhat

Assessment

Scenario 8

tHowWhat

Assessment

Perfect Alignment

tHowWhat

How has Alignment

Impacted You?

Workshop Agenda

• Introductions

• Overview

• Stakeholders

• Alignment

• Classifying skills and outcomes• Best Practices in MC item writing

• Take Away

• Next Workshop

Blooms Taxonomy revised

Sorting skills/outcomes

fromSimple to Complex

Blooms Taxonomy revised

Remembering/Understanding Level

Describes the circulation through the heart

Defines the term systole

Application (AP)theories in providing care to clients (e.g., a

Application Level

Apply the concepts of aging in developing interventions for the elderly

Blooms Taxonomy revisedCritical Thinking Level

Develop a plan for delivering services to elderly persons who are homebound

Evaluate nursing research based on predetermined criteria

Analyzes the organizational structure of the community health agency and its impact on client services.

Sorting skills/outcomes

into 3 classifications

Factual KnowledgeThe basic elements students must know to be acquainted with a discipline or solve problems in it.

Conceptual KnowledgeThe interrelationships among the basic elements within a larger structure that enable them to function together.

Procedural KnowledgeHow to do something, methods of inquiry, and criteria for using skills, algorithms, techniques, and methods.

Meta-Cognitive KnowledgeKnowledge of cognition in general as well as awareness and knowledge of one's own cognition

Four Knowledge Dimensions

Identify skills/outcomes

Knowledge Dimension & Cognitive Process

Writing a MC ItemWhat is the biggest challenge for you in sorting outcomes?

Workshop Agenda

• Introductions

• Overview

• Stakeholders

• Alignment

• Best Practices in MC item writing• Table of Specification or Blueprint

• Take Away

• Next Workshop

Writing a MC Item

MC Item Anatomy

1. Selecting the competency/outcome

2. Construct the stem

3. Write plausible distractors

4. Peer edit

5. Item analysis

Steps to writing a MC Item item

ATOA and NOTA

Results showed that when “all-of-the-above” is used as the correct answer, the item is more difficult for all students, despite the literature assumption that it provides a cueing effect to students. Research findings corroborate literature assumptions that high ability students score significantly higher than other ability students in this type of option.

Huang, Yi Min, Trevisan, Michae lThe Impact of the “all-of-the-above” Option and Student Ability on Multiple Choice Tests International Journal for the Scholarship of Teaching and Learninghttp://www.georgiasouthern.edu/ijsotl Vol. 1, No. 2 (July 2007) ISSN 1931-4744 © Georgia Southern University

Constructing a MC ItemPoints to Ponder

Huang, Yi Min, Trevisan, Michae lThe Impact of the “all-of-the-above” Option and Student Ability on Multiple Choice Tests International Journal for the Scholarship of Teaching and Learninghttp://www.georgiasouthern.edu/ijsotl Vol. 1, No. 2 (July 2007) ISSN 1931-4744 © Georgia Southern University

Constructing a MC Item

2010 Matthew V. Pachaia, David DiBattistab, Joseph: The effect of ’None of the Above’ on multiple choice questions in a first year classroom A Preprint submitted to Advances in Health Science Education

The present study examined the effect of NOTA on multiple choice questions in a large classroom setting. Specifically, we manipulated the place- ment of NOTA across versions of the test such that, for each of 20 experimental questions, NOTA was either not present, replaced the key, or replaced the high, medium, or low frequency distractors respectively. We found that NOTA sig-nificantly increased the difficulty of multiple choice questions only when it was used as the key. We found no effect of NOTA replacing any of the distractors, regardless of their effectiveness. Addi-tionally, we found no significant effect of NOTA on discrimination scores, regardless of its place- ment.

Constructing a MC ItemPoints to Ponder

Constructing a MC Item

Correct Answer

Best Answer

Correct or Best

Content Free Test

Varieties of MC Items

1. Single Correct Answer

2. Multiple Response

3. Combined Response

4. Multiple Response

Writing a MC Item

1. Design each item to measure an important learning outcome

2. Present a single clearly formulated problem in the stem of the item

3. State the stem of the item in simple, clear language

4. State the stem in positive form wherever possible

5. Word the alternatives clearly and concisely

6. Keep the alternatives mutually exclusive

7. Keep the alternatives homogeneous in content

8. Keep the alternatives free from clues as to which response is correct

9. Avoid the alternatives AOTA and NOTA

10. Use as many functional distractors as are feasible.

11. Include one and only one correct or clearly best answer for each item

Writing a MC Item

12. Present the answer in each of the alternative positions approximately an equal number of times, in a random order

13. Layout the items in a clear and consistent manner

14. Use proper grammar, punctuation, and spelling

15. Avoid using unnecessarily difficult vocabulary

16. Analyze the effectiveness of each item after each administration of the test.

Writing a MC Item

Rules Broken in the

Content Free Test

Writing a MC Item

Item writing……

Workshop Agenda

• Introductions

• Overview

• Stakeholders

• Alignment

• Table of Specification or Blueprint • Take Away

• Next Workshop

Blueprint or TOS

Remembering/Understanding Level

Application (AP)theories in providing care to clients (e.g., a

Application Level

Blooms Taxonomy revisedCritical Thinking Level

Blueprint or TOS

Workshop Agenda

• Introductions

• Overview

• Stakeholders

• Alignment

• Take Away• Next Workshop

Writing a MC Item

WhatSo what Now what

Writing a MC Item

Workshop Agenda

• Introductions

• Overview

• Stakeholders

• Alignment

• Take Away

• Next Workshop

Writing a MC ItemThank You

References

http://shpliteracy.pbworks.com/w/file/fetch/54533674/NEW%20Blooms%20Taxonomy%20Question%20Stems.pdf

Anderson, Lorin W. & Krathwohl, David R. (2001). A Taxonomy for Learning, Teaching and Assessing: a Revision of Bloom’s Taxonomy. New York. Longman Publishing.

MC Item Writing Workshop

Documents

Transcript of MC Item Writing Workshop

The Validity of Writing Two-Item Rules

I024 writing commentary brian mc neal

MC Writing Sample

Item writing in literature

1.Test Design 2.Item Specifications 3.Item Writing 4.Item Review.

ITEM WRITING GUIDELINES - TIMSS and PIRLS Home · ITEM WRITING GUIDELINES ... good item and test development practices in general, ... translated from English into the languages of

P A D I Research on Item Writing ExpertiseI CONTENTS Abstract IV 1.0 PADI Design Patterns and a Cognitive Model of Item Writing Expertise 1 2.0 A Preliminary Theory of Item Writing

I024 writing commentary mc neal

A Review of Multiple-Choice Item-Writing Guidelines for · PDF file · 2012-10-18A Review of Multiple-Choice Item-Writing Guidelines for Classroom Assessment ... ent types of MC item

MCQs - Item Writing & Item analysis

ONLINE ABVP ITEM WRITING GUIDE - Microsoftabvpprod.blob.core.windows.net/sitedocs/Item Writing/07 Online Item... · • All offered answers to the item ... B. chronic hypoglycemic

Test item writing and analysis

Item Writing Workshop

ITEM WRITING FOR SOUTHEAST ASIAN LANGUAGES. Goals Improve item writing skills of all participants Begin training of writers for national assessments Enhance.

Guide to Item Writing - abpsus.org

English Language Arts (ELA) Item Writing Guide and Item ...

TIMSS 2015 Item Writing Guidelines

What Item Writers Think When Writing items: Towards a ... · PDF fileWHAT ITEM WRITERS THINK WHEN WRITING ITEMS: TOWARDS A THEORY OF ITEM WRITING EXPERTISE ... The eggs are laid in

Item Writing Techniques for Surveys, Quizzes and Tests Writing... · Item Writing Techniques for Surveys, Quizzes ... When replacing strings on a bass guitar: ... Copyright © 1995-2014

Grade 4 Reading MCA Item Sampler Key Guidetest-minnesota.pearsonaccessnext.com/resources/item...Item # Correct Answer Item Type Skill Domain SSB Depth of Knowledge 1 D MC Key Ideas