8. Setting and Maintaining Examination Standards

7/28/2019 8. Setting and Maintaining Examination Standards

1/20

SETTING & MAINTAINING

EXAM STANDARDS

Raja C. Bandaranayake


2/20

DEFINITIONS

Standard setting is a process of determining howmuch is good enough.

The standard or criterion level of performance is apoint on the scale of measurement at whichseparation of competence and incompetenceoccurs.

Cut-score, cut-off score or passing score representsthis standard on a given test for making decisionspertaining to the purpose for which the test wasconducted, e.g., to certify competence.


3/20

ERROR IN MEASUREMENT

True score is a conceptual measure indicating trueextent of competence in a given subject, e.g.,Anatomy.

Observed score is the score assigned as a result oftaking a test, say in Anatomy.

The difference between true and observed scores isindicative of the amount of error in the

measurement.

The reliability of a test and the associated standarderror of measurement are estimates of theamount of error in the measurement.


4/20

DECISION ERRORS

False positive:

passing an incompetent examinee

False negative:

failing a competent examinee


5/20

NORM- & CRITERION-REFERENCED STANDARDS

NORM-REFERENCED

Relative

Based on peer-performance

Varies with each group

Cut-off point notrelated to competence

CRITERION-REFERENCED

Absolute

Not related to peerperformance

Standard set prior toexam

Referenced to adefinedlevel ofperformance


6/20

METHODS OF STANDARD SETTING

1. Test-centred methodsStandards derived from hypotheticaldecisions based on test content before thetest is answered.

2. Examinee-centred methodsStandards derived from reviewing examineesperformance before deciding cut-off score.

3. Compromise methodsProvide flexibility for adjusting thestandard based on the examineesperformance on the test.


7/20

NEDELSKY (1954) METHOD: Example

Consider N judges and n MCQ items of 1 in 5 type

Judge A identifies 2 options in item 1 as those which aminimally competent examinee should eliminate as incorrect.

MPL for that item for Judge A [MPLA1] = 1/(5-2) = 1/3

Similarly, in item 2 he identifies 3 options, giving an MPLA2 =1/(5-3) = 1/2He repeats this process for each item.

The exam MPL for Judge A [MPLA] = MPLA1 +MPLA2 + MPLA3 +

.MPLAn

Similarly, Judge Bs MPL [MPLB] is determined

The MPL for the exam (= cut-off score) is: (MPLA +

MPLB + MPLC+.MPLN) / N


8/20

ANGOFF (1971) METHODExample

N judges consider 100 minimally competent examineestaking an MCQ exam of n items.

Judge A estimates that, of these examinees, 50 shouldanswer item 1 correctly, 20 item 2 correctly, 70 item 3

correctly, and so on to item n.

The MPL for Judge A [MPLA] = (0.5 + 0.2 + 0.7 + . xn) / n X100 = (say) A%.

Similarly, for Judges B, C, D, E, ..N, the MPLs would be B%,C%, D%, E% N%, respectively.

The MPL (cut-off score) for the exam is: (A% + B% + C% +D% + E% +....N%) / N


9/20

EBEL (1972) METHODExample

Assume that Judge A assigns items in a 200-item MCQ test to the cells of arelevance-by-difficulty matrix, as follows.

He then estimates the percentage of items in each cell of the matrix that aminimally competent examinee should be able to answer correctly (as indicatedwithin the cell).

Each cell also includes the products of these two values.

EASY MEDIUM HARDESSENTIAL 15 x100% = 1500 25 x80% =2000 10 x60% = 600

IMPORTANT 20 x 80% = 1600 40 x60% =2400 20 x50% =1000

ACCEPTABLE 10 x 50% = 500 25 x40% = 1000 5 x10% = 50

QUESTIONABLE 10 x30% = 300 15 x20% = 300 5 x0% = 0


10/20

EBEL (1972) METHOD - contd.Example

The MPL for Judge A [MPLA] is then:

(1500 + 1600 + 500 + 300 + 2000 + 1000 + 300 + 600 + 1000

+ 50 + 0) / 200 = 56.25 %

Similarly, the MPL for Judges B [MPLB], C [MPLc], D [MPLD]..N [MPLN] are determined.

The MPL for the exam (cut-off score) is:

(MPLA+MPLB+ MPLc+ MPLD+ ..MPLN) / N


11/20

PROPOSED EBEL MODIFICATION

EASY MEDIUM HARD

ESSENT. 6x100% = 600 12 x80% = 960 7 x50% = 350

IMPORT. 12 x80% = 960 24 x60% = 1440 19 x40% = 760

ACCEPT. 5 x60% = 300 12 x50% = 600 3 x10% = 30

MPL: =600 + 960 + 350 + 960 + 1440 + 760 + 300 + 600 + 30

=6000/100

= 60


12/20

Failure

Rate%

Cut-offscore(%)

10

15

20

35 40 45 50

fmin

fmax

cmin cmax

A

B

HOFSTEE METHOD


13/20

HOFSTEE METHODExample

A plot of cut-off scores for a given examagainst resulting failure rates is given

cmin= 40%

cmax= 45%fmin = 10%fmax = 20%

A = point representing cmin,fmax

B = point representing cmax,fminLine AB intersects the curve at a cut-offpoint of 42.5%

Thus, operational cut-off score = 42.5%


14/20


15/20

CUT-OFF SCORE FOR 1 IN 5 MCQ[FRACS PART 1]

Probability of guessing (=1 in 5) = 20%

Total ignorance score = 20%

Maximum possible score =100%

Effective range of scores = 20% to100%

Mid-point of this range = 60%

Additional factor (as PG exam) = 5%

Nominal cut-off score (60%+5%) = 65%


16/20

CUT-OFF SCORES:MARKER QUESTIONS

1. Comparison of exam scores

Mean score in this exam: 56.7%

Average exam mean score over last 4 years: 59.4%

Thus mean score in this exam is: 2.7% lower

Assuming this candidate group is of samestandard as in last 4 yrs, this exam is: 2.7% harder


17/20

CUT-OFF SCORES:MARKER QUESTIONS - contd.2. Comparison of marker scores

Mean score in this exam on previously usedquestions (N=162): 62.5%

Mean score on same questions when theywere each last used: 60.5%

Thus, when compared with previous

candidates, this group of candidates, onthese items, scored (62.5-60.5)% = 2.0% higher

Thus this group of candidates is: 2.0% betterthan previous groups


18/20

CUT-OFF SCORES:MARKER QUESTIONS contd.3. Estimating examination difficulty

Thus it is expected that their mean scorein this exam would be: 2.0% higher

But their mean score in this exam is: 2.7% lower

Thus this exam is really: 4.7% harder


19/20

CUT-OFF SCORES:MARKER QUESTIONS contd.4. Determining cut-off score

The cut-off level for an average exam is: 65.0%

Thus the cut-off level for this exam

should be (65 4.7)% = 60.3%

Cut-off score = 60.3%


20/20

Failure

Rate%

Cut-offscore(%)

10

15

20

55 60 65 70

HOFSTEE CURVE

8. Setting and Maintaining Examination Standards

Documents

Transcript of 8. Setting and Maintaining Examination Standards