School Accountability and the Distribution of Student

Achievement

Randall RebackBarnard College Economics Department

and Teachers College,Columbia University

No Child Left Behind• States must adopt accountability systems that

assign ratings to schools based on student pass rates on exams in elementary, middle school, and high school grades

• School is not making ‘Adequate Yearly Progress’ if a pass rate is not sufficiently high, where the required pass rate increases each year

• Consequences– Stigma, financial rewards/penalties, loss of

local control, changes in property values– Intra-district public school choice provision– Tutoring for economically disadvantaged

children

Texas Accountability Program

• Precursor to No Child Left Behind• Assigns schools one of four ratings based on

– Dropout Rates– Attendance Rates– Fraction of students who pass exams (overall

and within subgroups by race and family income)

• Testing Incentives are based on Pass Rates, not value-added measures of student-achievement

Key Provisions of the Texas Accountability System

Minimum TAAS Pass Rate

Maximum Dropout Rate

Minimum Attendance Rate

E R A E R A E R A 1993 90.0% 65.0% 20.0% 1.0% 3.5% N/A 97.0% 95.0% N/A

1994 90.0% 65.0% 25.0% 1.0% 3.5% N/A 94.0% 94.0% N/A

1995 90.0% 70.0% 25.0% 1.0% 3.5% 6.0% 94.0% 94.0% N/A

1996 90.0% 70.0% 30.0% 1.0% 3.5% 6.0% 94.0% 94.0% N/A

1997 90.0% 75.0% 35.0% 1.0% 3.5% 6.0% 94.0% 94.0% N/A

1998 90.0% 80.0% 40.0% 1.0% 3.5% 6.0% 94.0% 94.0% N/A

1999 90.0% 80.0% 45.0% 1.0% 3.5% 6.0% 94.0% 94.0% N/A

Previous Research Related to School Accountability/Minimum Proficiency

• Relative performance of students at different points in distribution (Holmes, 2004; Deere & Strayer, 2001)

• Achievement trends– Grissmer & Flanagan (98)- Math NAEP in TX– Hanushek & Raymond- Math NAEP – Carnoy, Loeb, Smith (2002)- TX improvements didn’t correspond

with improved 10-12th grade transitions, SAT participation, SAT performance

• Low-performing versus high-performing schools (Jacob, forthcoming• States with or without HS graduation exams (Jacobsen, 1993)• Gaming

– Exemptions: Figlio & Getzler, Cullen & Reback– School meals: Figlio & Winicki– Disciplinary practices: Figlio

Theoretical Framework• Subject Specific but not Student Specific Inputs (as)• Not Subject Specific but Student Specific Inputs (bi)• Subject Specific and Student Specific Inputs (cs)• Assume only campus-wide Math (m) and Reading (r) pass

rates count. Call all other subjects (z). Schools want to maximize:

TcbafPTcbafPN

iimimimim

N

iiriririr

~)),,((Prob*

~),,((Prob

11

N

iimirizmr Kccbaaa

1

)(

subject to

The Data• Texas Assessment of Academic Skills

– Math Tested Grades 3-8 and 10– Writing Tested Grades 4, 8, and 10– Test Documents Submitted for Every Student– Includes Student Descriptors

• Campus Level Data on Attendance/Dropouts• Texas Learning Index

– Measures How Student Performs Compared to Grade Level

– I Do Not Measure Test Score Gains for Observations with Prior Year’s Scores Below 30 or Above 84

Pass Rate Probabilities Based on Prior Year Test Score Range

Previous Year’s Scoring

Range

Probability of Passing Math based on Previous

Math Score Range

Probability of Passing Reading based on Previous

Reading Score Range 30-44 7.3% 14.9% 45-54 17.7% 26.3% 55-64 39.2% 45.8% 65-74 69.0% 69.1% 75-84 92.3% 88.8%

Passing Score=70

Estimating the Marginal Benefit to the School from a Moderate Increase in a

Student’s Expected Performance (1) estimate the probability that each student

passes by grouping students based on their performance during other years

(2) use these student-level pass probabilities to compute the probability that the school will obtain each rating

(3) find the marginal effect of a moderate improvement in the expected achievement of a particular student on the probability that the school obtains the various ratings….

How a ‘moderate improvement’ in a student’s achievement is determined

• hypothetical pass probability by re-estimating the student’s pass probability after dropping the bottom X% of the current year score distribution among students with identical prior year scores

• For example: distribution of this year’s Math scores for students scoring 53 last year in Math

0%: 36 Actual pass probability= .20 20%: 49

40%: 55 60%: 59 Pass probability with X% set at 20%: 80%: 70 .2 / .8=.25 100%: 86

Dependent Variable

• Year-to-year variation in test scores might be greater at certain points of achievement distribution

• Value-added models examining distributional effects SHOULD NOT simply look at changes in levels or in relative place in distribution

• Instead, use conditional Z-score… Z-score among students with similar prior year scores

• This way, results are compared to typical progress at that place in the test score distribution

Empirical Model #1: Campus-Year Fixed Effects

Student i in grade g during year y at school s

stjittjti

stistjistisi

tjstgi

YCS

RPEERRx

vY

,,,5,4,

3,1,2,1,,,1,,

,1,,,

log

Si,t includes control variables for student characteristics:• Cubic terms for prior year scores in other subject• Racial dummy variables, Low-income family dummy, and Race-income interactions

Achievement Gains and Marginal Accountability Incentives within

Schools and within the Same Year MATH READING

Model Also Includes Peer Ability Controls?

(1) (2)

YES NO

(3) (4)

YES NO Point Estimate for Student-level Accountability Incentive Variable, ( in equation 6)

.00090 (.00013)

.00099 (.00013)

-.00007 (.00015)

.00009 (.00015)

Effect sizes: Approximate standard deviation change in statewide distribution associated with a 1 standard deviation increase in relative importance of student’s score within the school

.005 .005 -.0004 .0005

Model #2: Response to Infra-marginal Incentives (Cross-sectional comparisons)

• Schools might consider the impact of improving the expected performance of 5% of the students

• Define as the marginal change in the

schools’ probability of a higher rating if all students in the ‘group’ are expected to do better

stjisttti

stistjistisi

tjstgi

SYX

RPEERRg

vY

,,,6,54,

3,1,2,1,,,1,,

,1,,,

log

si

tj

g

v

,

,

Achievement Gains and Infra-marginal Accountability Incentives

MATH READING

Includes Peer Ability Controls?

YES NO

YES NO Point Estimate for Student-level Accountability Incentive Variable, (in equation 7)

.00092

(.00006)

.00119

(.00006)

.00005

(.00008)

.00057

(.00007)

Effect sizes: Approximate standard deviation change in student’s place in the statewide distribution associated with a 1 standard deviation increase in the infra-marginal incentive to help the student’s performance

.0111 .0143 .0007 .0074

Model #3: Incentive to Improve Performance within a Grade-level

• Schools might use inputs that simultaneously affect multiple students

• Define as change in school’s probability of receiving a higher rating if all students in student i‘s grade at the school improve

.

log

,,,7,

,,1,6,5

4,3,1,2,1,,,1,,

,1,,,

stjisi

tjstistt

tististjistisi

tjstgi

h

vRSY

XRPEERRg

vY

si

tj

h

v

,

,

Approximate Effect Sizes (SD change in Statewide Achievement Distribution from 1 SD increase in

Accountability Incentive

MATH READING

Student-level Incentive

.009 .010 -.001 .002

Grade-level Incentives Interacted with Prior Year

Achievement Level Lowest 0.082 0.086 0.010 0.018 Very Low 0.074 0.077 0.014 0.023 Low 0.068 0.071 0.014 0.020 Marginal 0.060 0.065 0.013 0.017 Proficient

0.061 0.067 0.004 0.005

Peer ability Controls?

YES NO YES NO

Effects of Sample Selection Due to

Student Exemptions & Grade Repetition • Exemptions

– negative relationship between student-level accountability incentive and likelihood that student is exempted from accountability pool

– suggests that estimated effect of student-level accountability incentive may understate the true effect

– Also suggests that estimated effect of grade-level incentives overstate the effects for lowest achievers and understate for others

• Grade Retention– small, positive relationship between the student-level

accountability incentives and the probability of grade retention

– effect on main results is unclear, but likely small

Conclusions

• Schools respond to specific incentives of a rating system

• Appear to respond with broad changes in teaching or resource allocation rather than narrowly-targeted changes

• Current findings may understate distributional effects– High achievers (top 50% Reading, top 33%

math) are not included– May be permanent changes rather than

response to short-run incentives• NCLB-style ratings. Are they good? bad?...

depends on one’s preferences.