REA Brown Bag – SAT10 Growth Methods and Reporting

Tom Watkins, Ph.D

Director of Research, Evaluation and Assessment, SPPS

Goals for Brown Bag (2 or more of the following):

1. To describe some strengths and weaknesses of various approaches for calculating achievement growth on the SAT10 in SPPS.

2. To describe some strengths and weaknesses of various approaches for reporting achievement growth on the SAT10 in SPPS.

3. To examine the degree of correlation between different growth measures at the student score level

4. To examine the degree to which different growth measures produce the same growth categories (average, hi or low) at the grade level

Goals for Brown Bag

1. To describe some strengths and weaknesses of various approaches for calculating achievement growth on the SAT10 in SPPS.

4. To examine the degree to which different growth measures produce the same growth categories (average, hi or low) at the grade level within school, and at the group level within the district

Part 1: Strengths and Weaknesses of Approaches for Calculating Achievement Growth

Illustrations and Basis for SAT10 Growth Scaled Score Improvement (SSI – OLS Reg) Percent Gaining (NCE change 0 or more) Index Rate Growth (SAT10 to MCA) Heirarchical Linear Modeling (HLM)

General Psychometric ConsiderationsGeneral Practical ConsiderationsSPPS Considerations

90%

63%

48%

40%

31% 31%25% 23%

0%

20%

40%

60%

80%

100%

1. NCLB/MCAIndex (A)

2. TEAE - SOLOM (A)

3. SAT-10 (E/J) 4. Teacher Prof.Dev. (A)

5. NCLB Attend.(E/J)

6. NCLB/MCAPart.(A)

7. MCA 2yrGrowth (E/J)

8. MCA Ann.Growth (E/J)

Figure 1: Top Eight Goals Chosen for School Continuous Improvement Plans in District Elementary Schools

Percent of Elementary Schools Using the Goal in their 2005-06 Plans

Which Indicators Matter to SPPS Elementary Schools?

The method for calculating growth for student groups from MAT7 to SAT10 (similar to what has been used since):

1. For each grade level and subject area, run a simple linear regression of SAT10 scaled scores on MAT7 scaled scores.

2. Save the standardized residuals (unstandardized residuals were used for the MAT7 SSI).

3. For groups with N's of 10-39, a mean standardized residual at or below -.20 is "low growth", and residuals at or above +.20 is "high growth".

4. For groups with N's of 40-79, a mean standardized residual at or below -.15 is "low growth", and residuals at or above +.15 is "high growth".

5. For groups with N's of 80+, a mean standardized residual at or below -.10 is "low growth", and residuals at or above +.10 is "high growth".

SAT10 Scaled Score Improvement (SSI) Illustration READING SSI from Grade 2 in 2003 to Grade 3 in 2004

45 Groups With 10 to 19 Students per Group (Each Dot is One Group)

580

590

600

610

620

630

640

650

660

670

680

550 560 570 580 590 600 610 620 630

SAT10 Scale Score 2003

SAT1

0 Sca

le Sc

ore 2

004

LOW GROWTH

AVG GROWTH

HIGH GROWTH

Example of Group w ith Low Growth (SSI = -.69)

Threshold for Low Growth (SSI = -.30)

Threshold for High Growth (SSI = +.30)

Example of Group w ith High Growth (SSI = +1.52)

MCA Grade CorrelationPercent

CorrectlyNCE Range at Each MCA

Achievement Level

and Subjectto 2004 SAT10b Projectedc I II III IV V

Grade 3 Reading .77 83% 1 - 25 26 - 38 39 - 46 47 - 69 70 +

Grade 3 Math .68 79% 1 - 20 21 - 41 42 - 50 51 - 71 72 +

               

Grade 5 Reading .82 86% 1 - 24 25 - 37 38 - 43 44 - 65 66 +

Grade 5 Math .81 84% 1 - 23 24 - 43 44 - 52 53 - 75 76 +

               

Grade 7 Reading .82 83% 1 - 19 20 - 42 43 - 61 62 - 85 86 +

Grade 7 Math .83 82% 1 - 20 21 - 43 44 - 64 65 - 80 81 +

Basis of Index Growth from SAT10 to MCA

Basis for MLM Approach (HLM one type) (Steenbergen, 2002)

Uses “Nested” Data - Distinguishing Student-Level from Teacher, Grade, School and District Levels

Data IntensiveTheory IntensiveRequires More Assumptions

DISTRICT GROWTH OR GAIN MEASURE

Index RateSAT10 Percent Gaining

SAT10 Scaled Score Improvement

MCA Index Growth

HLMEVAAS

GROWTH CATEGORY (O’Malley, et al., 2005)

Performance Category Count Models

OLS Regression Equipercentile Equating Method

Mixed Models

Can Use Stu/sch Demographic Variables

X X

Explicitly Incorporates Standards

X [X]

Simple to Apply and Explain

X X [X?]

Students Must Have Unique ID Numbers

X X X

Applied with Tests in Consecutive Years

[X] X

Can Use Scores of Multiple Years

X

General Practical Considerations in Growth and Gain Models

DISTRICT GROWTH OR GAIN MEASURE

Index RateSAT10 Percent Gaining

SAT10 Scaled Score Improvement

MCA Index Growth

HLMEVAAS

GROWTH CATEGORY (O’Malley, et al., 2005)

Performance Category Count Models

OLS Regression Equipercentile Equating Method

Mixed Models

Requires Scores on Same Scale

X

Results in Formula X X

Missing Data OK X

Accounts for Measurement Error

X (if CI’s used) X

Regr to Mean OK X X

Models Nesting Effect X

Can Include Nonlinear Parameters

X X

General Psychometric Considerations in Growth and Gain Models

Additional SPPS Growth Method Considerations Reflects local input (Administrator’s Academy,

Accountability Framework) Live or dead growth norm Local, national or state growth comparisons Provides estimates of growth of student groups (ethnic,

FRL, ELL, Spec Ed) Multiple measures/methods preferred As simple as appropriate, but no simpler. Acknowledges real student growth patterns Descriptive vs. VAM approach (Davison, Thum) Consider link to reported growth history, and likely future Consider link to formative assessment system

DISTRICT GROWTH OR GAIN MEASURE

Index RateSAT10 Percent Gaining

SAT10 Scaled Score Improvement

MCA Index Growth

HLMEVAAS

GROWTH CATEGORY Performance Category Count Models

OLS Regression Equipercentile Equating Method

Mixed Models

Provides Results for Student Groups

[X] [X] [X]

Live Growth Norm [X] [X]

Local Growth Comparison

[X] [X]

(centering)

National Growth Comparison

X

(dead)

Link to Reported Growth History

X [X]

Most or All Groups in SPPS Average Range

[X] [X]

Additional SPPS Considerations in Growth and Gain Models

Part 2: In what ways do different growth measures produce the same results?

Growth categories (average, hi or low):at the grade level within the school, at the group level within the district

Percent of AFRICAN-AMERICAN Students Making One Year's Growth or More in READING on the SAT10 03-04 Correlated With Percent Low Income in School

30.0

40.0

50.0

60.0

70.0

80.0

90.0

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Percent Low Income

Per

cen

t M

akin

g O

ne

Yea

r's

Gro

wth

in

RE

AD

ING

Percent of ALL STUDENTS Making One-Year's Growth in MATH on the SAT10 03-04 is Not Closely Related to Percent Low Income

30.0

40.0

50.0

60.0

70.0

80.0

90.0

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Percent Low Income

Per

cen

t M

akin

g 1

+ Y

ear'

s G

row

th in

MA

TH

Percent of ALL STUDENTS Making One Year's Growth or More in READING Compared to MOBILITY PERCENTAGE IN SCHOOL - SAT10 2003-04

45.0

50.0

55.0

60.0

65.0

70.0

75.0

80.0

0% 10% 20% 30% 40% 50% 60%

School Mobility Percentage

Per

cen

t o

f A

fric

an-A

mer

ican

Stu

den

ts

Mak

ing

On

e-Y

ear'

s G

row

th in

Rea

din

g

  Reading Growth 03-04  

Stanine Category in 2003 SSI NCE change % Gain

1. Below Avg (Sta 1-3) 0.03 4.59 0.69

2. Average (Sta 4-6) -0.07 0.76 0.54

3. Above Avg (Sta 7-9) 0.15 -1.28 0.48

Total 0.00 1.60 0.58

Some Relevant Items in CRESST’s Standards for

Educational Accountability Systems (Baker, et al., 2002)

(A1) Accountability expectations should be made public and understandable to all participants in the system.

(A2) Accountability systems should employ different types of data from multiple sources.

(A4) Accountability systems should include the performance of all students, including subgroups that historically have been difficult to assess.

(B9) The validity of a measure that has been administered as a part of an accountability system should be documented for the various purposes of the system.

(B10) If tests are to help improve system performance, there should be information provided to document that test results are modifiable by quality instruction and effort.

To what degree do these standards apply to School Improvement Systems?

Buried Treasure – Implications (Celio & Harvey, 2005)

Less may be more Parsimony and power should be respected Status data necessary but not sufficient Potential to change board-supt relations Cost-benefit analysis Professional development and tech assistance

will be required State leaders have a significant role to play

Buried Treasure – Indicators (Celio & Harvey, 2005)

Achievement (reading and math)Elimination of the achievement gapStudent attraction (school ability to attract

students)Student engagement with the schoolStudent retention/completionTeacher Attraction and retentionFunding Equity

Four Quadrants of School Improvement Indicators (Dale Carlson, 2002)

Effectiveness –Current Level

Improvement – Change Over Time

Status Quadrant 1 (Q1)

Quadrant 2 (Q2)

Longitudinal Quadrant 3 (Q3)

Quadrant 4 (Q4)

So, Where Are We in SPPS?

Put growth models in perspective (only part of one piece of the puzzle)

Transition from SAT10 to MCA-II growthNeed to dedicate resources to solid cost-

benefit analyses, using multiple outcomesHave to prioritize and commit to indicators

so solid professional development and technical assistance is possible.

Partial List of References Baker, E. L., Linn, R. L., Herman, J. L., & Koretz, D. (Winter, 2002). Standards for Educational Accountability Systems. Policy Brief 5: Center for Research on Evaluation, Standards and Student Testing. Celio.M.B & Harvey, J. (2005). Buried treasure: Developing a management guide from mountains of school data. http://www.crpe.org/pubs.shtml#leadership Olson, L. (2005, July 13). Education Department convenes working group on ‘growth’ models. Education Week. Available at http://www.edweek.org/ew/articles/2005/07/13/42growth.h24.html . O’Malley, K., Vansickle, T. & Housson, S. (2005, June). Measuring growth: Where policy makers and psychometricians meet. Paper presented at the annual national conference on Large-Scale Assessment. San Antonio, TX. Thum, Y.M (2005, April). Designing school productivity indicators: Connecting goals, data & models. Paper presented at the annual meeting of the American Educational Research Association, Montreal.