Sensitivity of Teacher Value-Added Estimates to Student and Peer Control

VariablesOctober 2013

Matthew Johnson Stephen Lipscomb Brian Gill

Value-Added Models (VAMs) Used Today Differ in Their Specifications

2

Value-Added ModelStudent

CharacteristicsPeer

Characteristics

Multiple Years of

Prior Scores

Chicago Public Schools Yes No No

DC IMPACT Yes Yes No

Florida Yes Yes Yes

Pittsburgh Public Schools Yes Yes No

SAS EVAAS No No Yes

How sensitive are teacher VAM estimates to choice of control variables?– Are estimates for teachers with more students from

disadvantaged backgrounds affected by this choice?

Does the substitution of teacher-year level average student characteristics in place of classroom averages impact teacher VAM estimates?

Does allowing for relationship between current and lagged achievement to vary based on student demographic characteristics matter for teacher VAM estimates?

Research Questions

3

Data from a northern state and a medium-sized urban district in that state– District has more minority and low-income students

than state average

Estimate separate VAMs using state data and district data– More control variables available in district VAMs– For peer characteristics, use teacher-year level

averages in state VAMs, classroom averages in district VAMs

Each VAM uses three years of teaching data from 2008-2009 through 2010-2011

Data

4

Explore sensitivity to several specifications:– Exclude peer average characteristics (X̅; i,t)

– Exclude student characteristics (X̅i,t) and peer characteristics (X̅; i,t)

– Add scores from two prior years (Yi,t-2)

– Interact free/reduced lunch status with baseline scores

Estimate all models using the same set of student observations

Control for measurement error in prior test scores using an errors-in-variables approach

Empirical Bayes (shrinkage) adjusted estimates

Baseline Model

5

Student Controls(State)

Student Controls(District)

Peer Averages

(State)

Peer Averages (District)

Free or Reduced-Price Meals x x x x

Disability x x x x

Race/Ethnicity x x x x

Gender x x

English Language Learner x x x x

Age/Behind Grade Level x x

Gifted Program Participation x x

Lagged Rate of Attendance x x

Lagged Fraction of Year Suspended x x

Average Lagged Achievement x x

SD of Lagged Achievement x x

Number of Students in Class x

Student and Peer Characteristics

6

Correlation of 8th-Grade State Teacher VAM Estimates Relative to Baseline Specification

7

Math (N = 2,778)

Reading (N = 3,347)

Exclude peer characteristics 0.970 0.982

Exclude student and peer characteristics 0.964 0.979

Add scores from t-2 0.977 0.958

Add scores from t-2 and exclude student/peer characteristics 0.946 0.946

Baseline: Student characteristics, peer characteristics, and prior scores from t-1

Findings are based on VAM estimates from 2008–2009 to 2010–2011 on the same sample of students.

Remove Student/Peer Controls and Add t-2 Scores

1st (Lowest) 2nd 3rd 4th

5th

(Highest)

Baseline Model

1st (Lowest) 81 17 1 1 0

2nd 18 57 23 3 0

3rd 1 23 53 22 1

4th 0 3 22 59 16

5th (Highest) 0 0 1 16 83

Percentage of 8th-Grade Reading Teachers in Effectiveness Quintiles, by VAM Specification

8

Findings are based on VAM estimates for 3,347 reading teachers in grade 8 from 2008–2009 to 2010–2011. Correlation with baseline = 0.946.

How Are Teachers in One District Affected?

9

District has relatively large fraction poor and minority students

Math Grade 8 Reading Grade 8

District Percentile Rank 15 50 85 15 50 85

State Percentile Rank:

Baseline 21 62 86 17 66 90

Exclude peer characteristics 23 59 86 13 59 87

Exclude student and peer characteristics 20 58 85 13 57 89

Percentile Rank of District Teachers in State Distribution

Using Additional Controls in District Data

10

Math Grades 6-8 (N = 164)

Reading Grades 6-8(N = 215)

Exclude peer characteristics 0.955 0.963

Exclude student and peer characteristics 0.918 0.949

Add scores from t-2 0.987 0.967

Add scores from t-2 and exclude student/peer characteristics 0.927 0.909

Baseline: Student characteristics, peer characteristics, and prior scores from t-1

Findings are based on VAM estimates from 2008–2009 to 2010–2011 on the same sample of students.

Teacher-Year Average Student Characteristics vs. Classroom Average

11

Math Grades 6-8 (N = 164)

Reading Grades 6-8(N = 215)

Correlation Between Effect Estimates 0.956 0.975

Average Standard Error (Classroom) 0.065 0.073

Average Standard Error (Teacher) 0.069 0.075

Correlation of teacher effect estimates with baseline model above 0.99 for both subjects

Different Relationship Current and Prior Test Scores for FRL Students

12

Math Grade 8

(N = 2,778)

Reading Grade 8

(N = 3,347)

Non-FRL Student Coefficient on Prior Year Math Score (SE)

0.847(0.002)

0.199(0.003)

FRL student coefficient on Prior Year Math Score (SE)

0.896(0.003)

0.154(0.004)

Non-FRL Student Coefficient on Prior Year Reading Score (SE)

0.086(0.002)

0.641(0.003)

FRL student coefficient on Prior Year Reading Score (SE)

0.031(0.003)

0.728(0.004)

Teacher VAM estimates highly correlated across specifications– Choice of control variables– Use of teacher-year level averages in place of

classroom averages– Interaction between FRL status and prior scores

Choice of control variables can impact estimates for teachers of disadvantaged students

Conclusions

13

Other researchers have examined correlations in teacher effect estimates when different same-subject assessments are used as outcomes for teacher VAMs

The highest correlations these authors found are:– Lockwood et al. (2007): 0.46– Sass (2008): 0.48– Concoran et al. (2011): 0.62– Lipscomb et al. (2010): 0.61– Papay (2011): 0.54

Context for Results

14

Mathematica® is a registered trademark of Mathematica Policy Research.

Please contact– Matthew Johnson

• MJohnson@mathematica-mpr.com

– Stephen Lipscomb• SLipscomb@mathematica-mpr.com

– Brian Gill• BGill@mathematica-mpr.com

For More Information

15