Improving Content Validity: A Confidence Interval

for Small Sample Expert Agreement

Jeffrey M. Miller & Randall D. PenfieldNCME, San Diego

April 13, 2004University of Floridamillerjm@ufl.edu & penfield@coe.ufl.edu

“Validity refers to the degree to which evidence and theory support the interpretations of test scores entailed by proposed uses of tests (AERA/APA/NCME, 1999)

Content validity refers to the degree to which the content of the items reflects the content domain of interest (APA, 1954)

INTRODUCING CONTENT VALIDITY

Content is a precursor to drawing a score-based inference. It is evidence-in-waiting (Shepard, 1993; Yalow & Popham, 1983)

“Unfortunately, in many technical manuals, content representation is dealt with in a paragraph, indicating that selected panels of subject matter experts (SMEs) reviewed the test content, or mapped the items to the content standards…(Crocker, 2003)”

THE NEED FOR IMPROVED REPORTING

Several indices for quantifying expert agreement have been proposed

The mean rating across raters is often used in calculations

However, the mean alone does not provide information regarding its proximity to the unknown population mean.

We need a usable inferential procedure go gain insight into the accuracy of the sample mean as an estimate of the population mean.

QUANTIFYING CONTENT VALIDITY

A simple method is to calculate the traditional Waldconfidence interval

However, this interval is inappropriate for rating scales.

THE CONFIDENCE INTERVAL

df

sX t

n

1. Too few raters and response categories to assume population normality has not been violated.

2. No reason to believe the distribution should be normal.

3. The rating scale is bounded with categories that are discrete.

Penfield (2003) demonstrated that the Score method outperformed the Wald interval especially when The number of raters was small (e.g., ≤ 10) The number of categories was small (e.g., ≤ 5)

AN ALTERNATIVE IS THE

Furthermore, this interval is asymmetricIt is based on the actual distribution for the mean rating of concern.Further, the limits cannot extend below or above the actual limits of the categories.

SCORE CONFIDENCE INTERVAL FOR RATING SCALES

1. Obtain values for n, k, and z

n = the number of raters

K = the highest possible rating

z = the standard normal variate associated with the confidence level (e.g., +/- 1.96 at 95% confidence)

STEPS TO CALCULATING THE SCORE CONFIDENCE INTERVAL

2. Calculate the mean item rating

The sum of the ratings for an item divided by the number of raters

X

1

n

ii

X

n

3. Calculate p

p =

Or if scale begins with 1 then

p =

1

n

ii

X

nk

1 1

n

ii

X

nk k

4. Use p to calculate the upper and lower limits for a confidence interval for population proportion (Wilson, 1927)

2 2

2

2 4 (1 )

2( )L

pnk z z nkp p z

nk z

2 2

2

2 4 (1 )

2( )U

pnk z z nkp p z

nk z

5. Calculate the upper and lower limits of the Score confidence intervalfor the population mean rating

(1 )L LkLower X z

n

(1 )U UkUpper X z

n

Shorthand Example

Item: 3 + ? = 8

The content of this item represents the ability to add single-digit numbers.

1 2 3 4Strongly Disagree Disagree Agree Strongly Agree

Suppose the expert review session includes 10 raters.

The responses are 3, 3, 3, 3, 3, 3, 3, 3, 3, 4

Shorthand Example

n = 10

k = 4

z = 1.96

the sum of the items = 31

= 31/10 = 3.10 p = so, p = 31 / (10*4) = 0.775

1

n

ii

X

nk

X

Shorthand Example (cont.)

= (65.842 – 11.042) / 87.683 = 0.625

= (65.842 + 11.042) / 87.683 = 0.877

2 2

2

2 4 (1 )

2( )L

pnk z z nkp p z

nk z

2 2

2

2 4 (1 )

2( )U

pnk z z nkp p z

nk z

Shorthand Example (cont.)

= 3.100 – 1.96*sqrt(0.938/10) = 2.500

= 3.100 + 1.96*sqrt(0.421/10) = 3.507

(1 )L LkLower X z

n

(1 )U UkUpper X z

n

We are 95% confident that the

population mean rating falls somewhere

between 2.500 and 3.507

Content Validation

1. Method 1: Retain only items with a Score

interval of a particular width based on

a. A priori determination of appropriatenessb. An empirical standard (25th and 75th percentiles of all

widths)

2. Method 2: Retain items based on hypothesis test that the lower limit is above a particular value

EXAMPLE WITH 4 ITEMS  

 

  

 

  Rating Frequency for 10 Raters   95% Score CI

       

Item 0 1 2 3 4 Mean Lower Upper

                 

1 0 0 0 4 6 3.60 3.08 3.84

2 0 0 2 5 3 3.10 2.50 3.51

3 2 0 2 6 0 2.20 1.59 2.77

4 1 2 3 3 1 2.10 1.50 2.68

Conclusions 1. Score method provides a confidence interval that is

not dependent on the normality assumption

2. Outperforms the Wald interval when the number of raters and scale categories is small

3. Provides a decision-making method for the fate of items in expert review sessions.

4. Computational complexity can be eased through simple programming in Excel, SPSS, and SAS

For further reading,

Penfield, R. D. (2003). A score method for constructing asymmetric confidence intervals for the mean of a rating scale item. Psychological Methods, 8, 149-163.

Penfield, R. D., & Miller, J. M. (in press). Improving content validation studies using an asymmetric confidence interval for the mean of expert ratings. Applied Measurement in Education.