Establishing Performance Levels for the Stanford … Stanford English Language Proficiency Test...

.........

. . . . . . . . . .

technical report

Establishing Performance Levels for the Stanford English Language Proficiency Test (Stanford ELP)

Agnes Stephenson, Ph.D.

October 2003 (Revision 1, December 2003)

Copyright © 2003 by Pearson Education, Inc. or its affiliate(s). All rights reserved.Pearson and the Pearson logo are trademarks of Pearson Education, Inc. or its affiliate(s).

.........TECHNICAL REPORT

Establishing Performance Levels for Stanford ELP

2

Establishing Performance Levels for the Stanford English Language Proficiency Test

The Stanford English Language Proficiency Test (Stanford ELP) was first published by Pearson Inc. in 2003.

Introduction

The Stanford English Language Proficiency Test (Stanford ELP) measures the English proficiency of students in kindergarten through grade 12 whose first language is not English. The test directly fulfills the requirements of the No Child Left Behind Act of 2001 (NCLB), which states that beginning in the 2002 school year, an annual assessment of English language proficiency is required of each student who is limited English proficient (LEP). These English language learners (ELL students) must be assessed in the areas of oral language, reading and writing skills, and comprehension. The Stanford ELP was developed by experts in the field of English as a second language (ESL) and assesses the important aspects of literacy and oral competency required by the legislation.

The Stanford ELP assesses students’ general English language abilities by measuring the following:

• Listening, Writing Conventions, and Reading, using multiple-choice measures

• Writing, using an open-ended direct writing assessment

• Speaking, using a performance test

The topics and vocabulary used in each of the four Stanford ELP test levels are age-appropriate for the corresponding grade range:

Stanford ELP Test Level Grade Range

Primary K–2

Elementary 3–5

Middle Grades 6–8

High School 9–12


. . . . . . .. . TECHNICAL REPORT


3

To ensure that ELL students with varying degrees of English proficiency are reliably evaluated, each Stanford ELP test level includes items targeted to the range of possible performance levels. The performance levels were initially defined as Emergent, Basic, Intermediate, and Proficient. As will be shown later in this report, a fifth performance level, “Pre-Emergent,” was defined after an evaluation of the standard setting committees’ final recommendations.

Performance Levels

A performance level represents a level of mastery based upon the judgment of teachers and provides information about what students at a particular grade and proficiency level should know and be able to do in a particular language skill. The performance level associated with each possible score on a subtest is determined by a standard setting procedure that results in the identification of “cut scores.” For example, given a subtest with a maximum possible score of 20, what minimum score would a student have to earn for his or her performance to be classified as Basic, Intermediate, or Proficient? Note that a distribution across four performance levels requires the identification of only three cut scores.

The modified Angoff procedure (Angoff, 1984) was used to produce the recommended cut scores for the Stanford ELP. The modified Angoff procedure has a long and respected history in similar applications for both educational and professional certification assessments. The use of the procedure by Pearson provided a systematic technique for eliciting judgments from panels of experts (i.e., standard setting committees), producing consensus among these experts, and quantifying the results of the judgments. The modified Angoff procedure is widely recognized as the simplest method to use for setting performance level cut scores (Norcini, et al., 1987; Shepard, 1980). Moreover, research has shown that it produces ratings with better reliability and smaller variability among the ratings of judges than other standard setting procedures (Andrew and Hecht, 1976; Brennan and Lockwood, 1980; Cross, et al., 1984; Poggio, Glasnapp, and Eros, 1981; Skakun and Kling, 1980). The modified Angoff procedure incorporates an appropriate balance between statistical rigor and informed opinion.

Standard Setting Committees

The training and experience of the members of the standard setting committees is essential in establishing the validity of their ratings (American Educational Research Association, et al., 1999). The standard setting committees assembled by Pearson included outstanding ESL teachers of kindergarten through grade 12 from across the United States.




4

Five standard setting committees were established to identify cut scores for grades K–12. Each committee was responsible for establishing cut scores for each grade within an assigned group of grades as shown in Table 1. Each committee member was asked to set standards for the Stanford ELP.

Table 1. Standards Setting Committees, Grade Levels Assigned, and Stanford ELP Level(s) Reviewed

Committee Grades Assigned Stanford ELP Level(s) Reviewed K

A 1 2

Primary

B 3

4 C

5

Elementary

6

7 D 8

Middle Grades

9

10

11 E

12

High School

The Standard Setting Process

Pearson conducted the standard setting procedure by having the committee members use an Internet-based application. The standard setting activity was conducted over a 10-day period from January 16, to January 25, 2003. The committee members were permitted access to the online application from home or work at any time during the 10-day window. However, they were required to meet certain deadlines and had to participate in two scheduled online chat sessions. The resources available to the committee members included a user’s manual as well as guidance from their group facilitator (a Pearson psychometrician) and system administrator.

Orientation The committee members were informed that their task was to review all items in the Stanford ELP test level(s) that covered their assigned grades (see Table 1). They were also informed that the items contained within each of the Stanford ELP


. . . . . . .. .

TECHNICAL REPORT


5

levels were intended to assess the range of defined performance levels from Basic to Proficient. The orientation concentrated on helping the committee members (judges) become familiar with two substantive aspects of the standard setting procedure: (1) item rating and (2) confidentiality of recommendations.

Item Rating

The judges’ primary task was to “rate” each item on the Stanford ELP in terms of how a Basic, Intermediate, or Proficient student should perform on it, rather than how they do perform or will perform.

For a given multiple-choice item (Listening, Writing Conventions, and Reading subtests), the rating process worked by answering the following questions:

1. What percentage of “Basic” performing students in the grade should correctly answer the item?

2. What percentage of “Intermediate” performing students in the grade should correctly answer the item?

3. What percentage of “Proficient” performing students in the grade should correctly answer the item?

For a given open-ended or performance-based item (Writing and Speaking subtests, respectively), the rating process worked by answering the following questions:

1. What average number of rubric points should a “Basic” performing student earn on this item?

2. What average number of rubric points should an “Intermediate” performing student earn on this item?

3. What average number of rubric points should a “Proficient” performing student earn on this item?

Judges were reminded that they were to rate an item based on only the group of students within a performance level stratum, rather than the total group. In other words, for a Basic rating, judges were to identify the percentage that should correctly answer the item based on only the group of students whose best performance is Basic; not the percentage based on the total group’s performance.

Confidentiality of Recommendations

The judges were assured that their performance level recommendations (ratings) would remain confidential. Although an important goal of the process was for judges to approach consensus or convergence with their ratings, it was integral to the process for them to feel free to maintain a rating they personally believed was correct, whether or not it was consistent with ratings made by other committee members.




6

Round 1 Using the online standard setting application, the panelists recorded their ratings for each item in their assigned grades. After the panelists finished, item level data was calculated automatically by the online application in the form of frequency distributions of the ratings for each panelist for each item. Panelists then had the opportunity to examine each frequency distribution and evaluate the practical effect of their standard setting efforts. Providing the committee members with this information enabled them to see how their individual ratings related to those of other committee members. The panelists then had the opportunity to discuss their ratings with their respective committees in an online chat session.

Round 2 Once discussion of the results of the Round 1 ratings concluded, panelists were asked to review the entire set of items they rated in Round 1, reconsider the ratings, and revise any of their ratings, if necessary. Once all panelists completed their Round 2 ratings, they were again able to view a frequency distribution of their Round 2 ratings for each panelist for each item. Panelists were again able to discuss their Round 2 ratings in an online chat session.

Final Rating Following the Round 2 online discussion, the panelists by committee provided final cut scores for the test as a whole. These final cut scores became the committee’s recommended cut scores for achievement of each performance level above Emergent.

Comparison of Cut Scores Across the Grades

In order to verify that cut scores ascended with the grade level, the raw cut scores were first converted to a proportion of the total possible points. The total possible points for grades K through 5 was 102, while that for grades 6 through 12 was 110.


. . . . . . .. .

TECHNICAL REPORT


7

Smoothed Cut Scores

The recommended cut scores in raw score units were smoothed using a quadratic function. The result of this process is presented in Figure 1.

Figure 1. Recommended Proportion Cut Scores Across Grades––Smoothed

Developing Performance Levels from the Judges’ Recommendations

The results provided by the panel of judges served as a basis for Pearson to identify final cut scores for the Stanford ELP. Other considerations were brought to bear by Pearson in establishing the final cut score points and performance levels for the Stanford ELP. Ganopole (1980) and Millman (1973) have noted that the determination of the final performance levels must be made in terms of consequences including the instructional, legal, financial, and societal costs of alternative performance levels.

0.00

0.10

0.20

0.30

0.40

0.50

0.60

0.70

0.80

0.90

1.00

000 001 002 003 004 005 006 007 008 009 010 011 012

Grade

Prop

ortio

n

Basic Intermediate Proficient


.........

TECHNICAL REPORT


8

Based in part in the final recommendations of the standard setting committees, Pearson identified a recommended set of three cut score points for the overall test at each grade.

An impact analysis was conducted after the standard setting procedure. The data comprised about 38,000 ELL students from kindergarten through grade 12. The percentage of students falling into the Pre-Emergent, Emergent, Basic, Intermediate, and Proficient performance levels were calculated for each grade. Smoothing procedures were applied to ensure that the cut scores were depicting an upward trend within each level of the test.

The final recommended cut scores in raw score units for each of the grades on the total test are presented in Table 2.

Table 2. Pearson’s Recommended Cut Scores Based on Total Test

Performance Levels Grade Emergent Basic Intermediate Proficient

K 15 26 56 82 1 18 28 59 84 2 20 31 61 87 3 15 26 56 82 4 18 28 59 84 5 20 31 61 87 6 17 28 61 88 7 19 30 63 91 8 22 33 66 94 9 17 28 61 88

10 18 29 62 90 11 20 31 64 92 12 22 33 66 94

Addition of Fourth Performance Threshold

After close inspection of these recommended cut score points, it was decided that a fourth cut score point would be mathematically calculated to differentiate between the Emergent performance level and a lower level to be called Pre-Emergent.


. . . . . . .. .

TECHNICAL REPORT


9

Cut Scores for Subtests

Tables 3a through 3l present the raw cut scores for the different Stanford ELP subtests.

Table 3a. Raw Cut Scores––Listening


K 3 5 11 16 1 4 6 12 17 2 4 6 12 17 3 3 5 11 16 4 4 6 12 17 5 4 6 12 17 6 3 5 11 16 7 4 6 12 17 8 4 6 12 17 9 3 5 11 16

10 3 5 11 16 11 4 6 12 17 12 4 6 12 17

Table 3b. Raw Cut Scores––Writing Conventions


K 3 5 11 16 1 4 6 12 17 2 4 6 12 17 3 3 5 11 16 4 4 6 12 17 5 4 6 12 17 6 4 6 13 19 7 4 7 14 20 8 5 7 14 20 9 4 6 13 19

10 4 6 14 20 11 4 7 14 20 12 5 7 14 20




10

Table 3c. Raw Cut Scores––Reading


K 3 5 11 16 1 4 6 12 17 2 4 6 12 17 3 3 5 11 16 4 4 6 12 17 5 4 6 12 17 6 4 6 13 19 7 4 7 14 20 8 5 7 14 20 9 4 6 13 19

10 4 6 14 20 11 4 7 14 20 12 5 7 14 20

Table 3d. Raw Cut Scores––Writing


K 1 2 4 6 1 1 2 5 7 2 1 2 5 7 3 1 2 4 6 4 1 2 5 7 5 1 2 5 7 6 1 2 4 6 7 1 2 5 7 8 1 2 5 7 9 1 2 4 6

10 1 2 5 7 11 1 2 5 7 12 1 2 5 7


. . . . . . .. .

TECHNICAL REPORT


11

Table 3e. Raw Cut Scores––Speaking


K 5 9 19 27 1 6 9 20 28 2 7 10 20 29 3 5 9 19 27 4 6 9 20 28 5 7 10 20 29 6 5 9 19 27 7 6 9 20 28 8 7 10 20 29 9 5 9 19 27

10 6 9 19 28 11 6 10 20 28 12 7 10 20 29

Table 3f. Raw Cut Scores––Listening + Writing Conventions + Reading + Speaking


K 14 24 52 75 1 16 26 54 78 2 19 28 56 80 3 14 24 52 75 4 16 26 54 78 5 19 28 56 80 6 15 26 56 82 7 18 28 59 84 8 20 31 61 87 9 15 26 56 82

10 17 27 58 83 11 19 29 60 85 12 20 31 61 87




12

Table 3g. Raw Cut Scores––Writing + Speaking (Productive)


K 6 11 23 34 1 7 12 24 35 2 8 13 25 36 3 6 11 23 34 4 7 12 24 35 5 8 13 25 36 6 6 11 23 34 7 7 12 24 35 8 8 13 25 36 9 6 11 23 34

10 7 11 24 34 11 8 12 25 35 12 8 13 25 36

Tables 3h. Raw Cut Scores––Listening + Reading (Comprehension)


K 6 10 22 32 1 7 11 23 33 2 8 12 24 34 3 6 10 22 32 4 7 11 23 33 5 8 12 24 34 6 7 11 24 35 7 8 12 25 36 8 9 13 26 37 9 7 11 24 35

10 7 12 25 36 11 8 12 26 37 12 9 13 26 37


. . . . . . .. .

TECHNICAL REPORT


13

Table 3i. Raw Cut Scores––Listening + Speaking (Social)


K 8 14 30 43 1 9 15 31 45 2 11 16 32 46 3 8 14 30 43 4 9 15 31 45 5 11 16 32 46 6 8 14 30 43 7 9 15 31 45 8 11 16 32 46 9 8 14 30 43

10 9 14 31 44 11 10 15 32 45 12 11 16 32 46

Table 3j. Raw Cut Scores––Writing Conventions + Reading + Writing (Academic)


K 7 12 26 38 1 8 13 28 40 2 10 14 29 41 3 7 12 26 38 4 8 13 28 40 5 10 14 29 41 6 8 14 31 45 7 10 15 32 46 8 11 17 34 48 9 8 14 31 45

10 9 15 32 46 11 10 16 33 47 12 11 17 34 48




14

Table 3k. Raw Cut Scores––Total Test


K 15 26 56 82 1 18 28 59 84 2 20 31 61 87 3 15 26 56 82 4 18 28 59 84 5 20 31 61 87 6 17 28 61 88 7 19 30 63 91 8 22 33 66 94 9 17 28 61 88

10 18 29 62 90 11 20 31 64 92 12 22 33 66 94

Table 3l. Raw Cut Scores––Writing Conventions + Writing (Total Writing)


K 4 7 15 22 1 5 8 16 23 2 6 8 17 24 3 4 7 15 22 4 5 8 16 23 5 6 8 17 24 6 5 8 18 26 7 6 9 18 26 8 6 10 19 27 9 5 8 18 26

10 5 9 18 26 11 6 9 19 27 12 6 10 19 27


. . . . . . .. . TECHNICAL REPORT


15

Conclusion Establishing cut scores and performance level thresholds was part of the test development process for Stanford ELP. A rigorous standard setting method, the modified Angoff procedure, was applied to this undertaking. The involvement of highly qualified and experienced ESL teachers of kindergarten through grade 12, as well as the impact analysis conducted by Pearson, ensured the validity of the ratings and resulting cut scores.

References

American Educational Research Association, American Psychological Association, and National Council on Measurement in Education (1999). Standards for educational and psychological testing. Washington, DC: American Psychological Association.

Angoff, W. H. (1984). Scales, norms, and equivalent scores. Princeton, NJ: Educational Testing Service. (Reprint of chapter in R. L. Thorndike (Ed.). (1971). Educational Measurement (2nd ed.). Washington, DC: American Council on Education.

Andrew, B. J., & Hecht, J. T. (1976). A preliminary investigation of two procedures for setting examination standards. Educational and Psychological Measurement, 36, 45-50.

Brennan, R. L., & Lockwood, R. E. (1980). A comparison of the Nedelsky and Angoff cutting score procedures using generalizability theory. Applied Psychological Measurement, 4, 219-240.

Cross, L. H., Impara, J. C., Frary, R. B., & Jaeger, R. M. (1984). A comparison of three methods for establishing minimum standards on the National Teacher Examination. Journal of Educational Measurement, 21, 113-129.

Fitzpatrick, A. R. (1989). Social influences in standard-setting: The effects of social interaction on group judgments. Review of Educational Research, 59, 315-328.

Ganopole, S. J. (1980). “Using performance and preference data in setting standards for minimum competency assessment programs,” in R. M. Jaeger and C. K. Tittle (Eds.), Minimum competency achievement testing. Berkley, CA: McCutchan, 406-418.

Hambleton, R. K., Powell, S., & Eignor, D. R. (1979). “Issues and methods for standard setting” in A practitioner’s guide to criterion-referenced test development, validation, and test score use: Laboratory of Psychometric and Evaluative Research Report No. 70 (2nd ed.). Amherst, MA: School of Education, University of Massachusetts.




16

Jaeger, R. M. (1982). An iterative structures judgment process for establishing standards on competency tests: Theory and applications. Educational Evaluation and Policy Analysis, 4, 461-475.

Jaeger, R. M. (1991). Selection of judges for standard-setting. Educational Measurement: Issues and Practices, 10, 3-10.

Meskauskas, J. A. (1983). An assessment of the state of the art of standard setting methodology: Characteristics of improved approaches and two new methods. Paper presented at the Annual Meeting of the American Educational Research Association.

Millman, J. (1973). Passing scores and test lengths for domain-referenced measures. Review of Educational Research, 43, 205-216.

Millman, J. (1989). If at first you don’t succeed: Setting passing scores when more than one attempt is permitted. Educational Researcher, 18, 5-9.

Mills, C. N., & Melican, G. J. (1988). Estimating and adjusting cutoff scores: Features of selected methods. Applied Measurement in Education, 1, 261-275.

Norcini, J. J., Shea, J. A., & Kanya, D. J. (1988). The effect of various factors on standard setting. Journal of Educational Measurement, 25 (1), 57-65.

Poggio, J. P., Glasnapp, D. R., & Eros, D. S. (1981). An empirical investigation of the Angoff, Ebel, and Nedelsky standard setting methods. Paper presented at the annual meeting of the American Educational Research Association.

Shepard, L. A. (1980). Standard setting issues and methods. Applied Psychological Measurement, 4, 447-467.

Skakun, E. N., & Kling, S. (1980). Comparability of methods for setting standards. Journal of Educational Measurement, 17, 229-235.

Zeiky, M., & Fremer, J. (1979). Guidelines for developing and using minimum competency tests. Paper presented at the Annual Meeting of the American Educational Research Association.

Additional copies of this and related documents are available from: Pearson Inc. 19500 Bulverde Rd. San Antonio, TX 78259 1-800-211-8378 1-877-576-1816 (fax) http://www.hemweb.com/library/researchreports/index.htm


Establishing Performance Levels for the Stanford … Stanford English Language Proficiency Test...

Documents

Transcript of Establishing Performance Levels for the Stanford … Stanford English Language Proficiency Test...