MCQ_Guidlines

8/8/2019 MCQ_Guidlines

1/36


2/36

2

Assessment generally

Examinations are formidable

even to the best prepared, for the

greatest fool may ask more than

the wisest man can answer.

Charles Caleb Colton (1780 - 1832)


3/36

3

General thoughts

MCQs can be a useful element of an overall

assessment design

Cant be abstracted from the contextNo marks / diagnostic / formative / interactive

Few marks / semi-formative / assignment

High stakes / summative / examinationDesign and evaluation approach should be

appropriate for the context


4/36

4

No need to look too far from home

Considine, J; Botti, M and Thomas, S. Design, format, validity and

reliability of multiple choice questions for use in nursing research and

education. Collegian, 2005 Jan; 12 (1): 19-24.


5/36

5

A proposed MCQ process

From: Considine, J; Botti, M and Thomas, S. Design, format, validity and reliability of multiple choice questions

for use in nursing research and education. Collegian, 2005 Jan; 12 (1): 19-24.


6/36

6

An alternative/pragmatic process

Design

Content/Face validity

Use

Construct validity

Reliability

Keep / Replace


7/36

7

Design unit assessment plan

Assignment No. Assignment details Marks

1 Introduction to DSO 3 %

2 Information literacy & the Library 5 %

3 Referencing 10 %

4 Professional ethics report 15 %

5 Multi-choice test 1 6 %

6 Multi-choice test 2 6 %

7 Major report and presentation 25 %

Exam Must pass! 30 %


8/368

Design MCQ test plan

ContentAreas

Processes

1. Knowledge 2. Comprehension 3. Application %

A 3.75% 5.25% 6% 15

B 6.25% 8.75% 10% 25

C 3.75% 5.25% 6% 15

D 7.5% 10.5% 12% 30

E 3.75% 5.25% 6% 15

% 25 35 40 100


9/369

Design NCLEX-RN MCQ test plan

Source: 2010 NCLEX-RN Detailed Test Plan


10/3610

Face validity

A sub-set of content validity

Use an expert panel to :assess item clarity

assess item readability

assess consistency of stylecheck for errors


11/3611

Construct validity

Key check do experts agree on the correct

answer?

Item difficulty the proportion of examineesthat get the correct answer

Distracter evaluation

Item discrimination the ability of an item to

distinguish between knowledgeable studentsand others


12/3612

Correlation with final unit mark

r5= 0.584 r

6= 0.569

(p < 5.3x10-10) (p < 1.8x10-9)


13/3613


Ass 1

Ass 2Ass 3

Ass 4

Ass 5Ass 6

Ass 7

Exam

y =0.1854ln(x) +0.2209

R =0.8698

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

0 5 10 15 20 25 30 35

Pearsoncorrelationc

oefficient

Percentage of unit marks


14/36

Ass 1

Ass 2Ass 3

Ass 4

Ass 5

Ass 6

Ass 7

Exam

y =0.0139x +0.4626

R =0.7434

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

0 5 10 15 20 25 30 35

Pearsoncorrelationc

oefficient

Percentage of unit marks

14



15/3615

Reliability

Really internal consistency

Typically, Cronbachs alpha or KR-20

Provides a measure of the average correlation

among items within a test scale

If a test examines a range of themes, may

need to divide the test data into scales


16/3616

Review of items

Where items have been developed based on a

systematic test plan and good content validity,

they should only be discarded on the basis of

clearly poor construct validity and reliability


17/3617

Negative marking / formula scoring

Be the reasons for test-takers reluctance to answer under

formula scoring what they may be, this reluctance is a

psychometric Pyrrhic victory. Neither test-takers risk attitudes

nor their strategic reasoning are what multiple-choice tests

purport to measure. From the perspective of educationalmeasurement, individual differences in peoples risk attitudes are

nothing but unwanted noise: they add another source of

variance, which is likely to reduce the tests reliability and

validity. Furthermore, the testmakers persisting inability to

distinguish between a guessed correct answer and a knowncorrect answer is compounded by their inability to distinguish

between an omission deriving from ignorance and one deriving

from risk aversion. (Bar-Hillel, Budescu & Attali, 2005, p. 7)


18/3618

Negative marking discontinued by:

The Royal College of General Practitioners from

1992 discriminated against female candidates

The Royal College of Obstetricians andGynaecologists prior to 2004

benefited bold and test-wise candidates; inhibited reasoning

Examination Committee of the European Society of

Anesthesiology from 2008 does not reflect real practice where decisions must be made on

incomplete knowledge; no effect on rank ordering of results


19/3619

A useful

resource?

Source: https://www.ncsbn.org/1287.htm


20/3620

Development of the NCLEX-RN Test Plan

A total of 12,000 newly licensed RNs are asked

about the frequency and importance of performing

155 nursing care activities. Nursing care activities

are then analyzed in relation to the frequency ofperformance, impact on maintaining client safety

and client care settings where the activities are

performed. This analysis guides the development of

a framework for entry-level nursing practice thatincorporates specific client needs, as well as

processes fundamental to the practice of nursing.


21/3621

NCLEX-RN test plan structure

Safe and Effective Care EnvironmentManagement of Care

Safety and Infection Control

Health Promotion and Maintenance

Psychosocial Integrity

Physiological Integrity

Basic Care and ComfortPharmacological and Parenteral Therapies

Reduction of Risk Potential

Physiological Adaptation


22/3622

Design NCLEX-RN MCQ test plan


23/3623

Detailed

test plan


24/3624

Detailed

test plan


25/3625

Sample items management of care

For each of the eight content areas a sample

item is provided


26/3626

Steps in item writing

1. Select an area of the test plan for the focus of the item

2. Select a subcategory from the chosen area of the test

plan

3. Select an important concept within that subcategory4. Use the concept selected and write the stem

5. Write a key to represent important information the entry-

level nurse should know

6. Identify common errors, misconceptions or irrelevantinformation

7. Use the previous information and write the distracters

8. Complete the item using the stem, key and distracters


27/36

27

Steps in item writing - example

Step 1. Select an area of the test plan for the focus of the item

Safety and Infection Control

Step 2. Select a subcategory from the chosen area of the test planStandard Precautions/Transmission-Based

Precautions/Surgical Asepsis

Step 3. Select an important concept within that subcategory

Evaluate infection control precautions implemented by staffmembers


28/36

28


Step 4. Use the concept selected and write the stem

The nurse and nursing assistant are caring for a client with

vancomycin-resistant enterococci (VRE). Which of the

following activities by the nursing assistant would require

immediate follow-up?

Step 5. Write a key to represent important information the entry-

level nurse should know

Contact Isolation: Assisting the client to ambulate in thehallway


29/36

29


Step 6. Identify common errors, misconceptions or irrelevant

information

Lack of understanding of isolation precautions

Uncertainty related to specific diagnosis

Step 7. Use the previous information and write the distracters

Leaving a blood pressure cuff in the clients room to be

used by the client onlyPutting on a protective gown to assist the client to sit in

a chair

Taking the gloves off before leaving the clients room


30/36

30


Step 8. Complete the item using the stem, key and distracters

The nurse and nursing assistant are caring for a client with

vancomycin-resistant enterococci (VRE). Which of the

following actions performed by the nursing assistant wouldrequire immediate follow-up from the nurse?

a.Leaving a blood pressure cuff in the clients room to be used

by that client only

b.Putting on a protective gown to assist the client to sit in achair

c.Taking the gloves off before leaving the clients room

d.Assisting the client to ambulate in the hallway (key)


31/36

31

Case scenario exercise

Reduction of Risk Potential

The nurse is caring for client who had a

procedure three hours ago. Write an item thatincludes assessment data the nurse would

observe in this client and which data should the

nurse respond to first.


32/36

32

Reduction of Risk Potential model

The nurse is caring for a client who had a cardiac

catherization 3 hours ago. Which of the following

findings would be essential for the nurse to follow-up?

a.blood pressure increase from 103/68 to 110/70 over

the past one hour

b.blood urea nitrogen (BUN), 22 mg/dL (key)

c.pulse, 101

d.decrease in respiratory rate from 18 to 16 over the

past one hour


33/36

DSO assessment tools - resources

Online assessment strategies:

http://www.deakin.edu.au/itl/dso/strategies-teaching/assess

Blackboard assessment tools guides:

http://www.deakin.edu.au/itl/dso/guides/guide-list-assessm

Respondus guides:

http://www.deakin.edu.au/itl/dso/guides/guide-list-respondu

33
http://www.deakin.edu.au/itl/dso/strategies-teaching/assessment.phphttp://www.deakin.edu.au/itl/dso/guides/guide-list-assessments.phphttp://www.deakin.edu.au/itl/dso/guides/guide-list-respondus.phphttp://www.deakin.edu.au/itl/dso/guides/guide-list-respondus.phphttp://www.deakin.edu.au/itl/dso/guides/guide-list-assessments.phphttp://www.deakin.edu.au/itl/dso/strategies-teaching/assessment.php


34/36

34

Case scenario exercise

Psychosocial Integrity

A nurse on an inpatient psychiatric unit

observes a client pacing the hallway, mumblingand occasionally yelling aloud Stop it! Write an

item describing the action the nurse should take

in this situation.


35/36

35

Psychosocial Integrity model MCQ

The nurse is caring for a client who is mumbling, pacing

in the hallway, and occasionally yelling Stop it! Which

of the following actions should the nurse take?

a.Remove other clients from the area.

b.Escort the client back to the clients room.

c.Request that the client be quiet and not disrupt others.

d.Use distraction to re-focus the client to reality. (key)


36/36

Thank you for your time

MCQ_Guidlines

Documents

Transcript of MCQ_Guidlines