Sensitivity of Teacher Value-Added Estimates to Student and Peer Control
VariablesOctober 2013
Matthew Johnson Stephen Lipscomb Brian Gill
Value-Added Models (VAMs) Used Today Differ in Their Specifications
2
Value-Added ModelStudent
CharacteristicsPeer
Characteristics
Multiple Years of
Prior Scores
Chicago Public Schools Yes No No
DC IMPACT Yes Yes No
Florida Yes Yes Yes
Pittsburgh Public Schools Yes Yes No
SAS EVAAS No No Yes
How sensitive are teacher VAM estimates to choice of control variables?– Are estimates for teachers with more students from
disadvantaged backgrounds affected by this choice?
Does the substitution of teacher-year level average student characteristics in place of classroom averages impact teacher VAM estimates?
Does allowing for relationship between current and lagged achievement to vary based on student demographic characteristics matter for teacher VAM estimates?
Research Questions
3
Data from a northern state and a medium-sized urban district in that state– District has more minority and low-income students
than state average
Estimate separate VAMs using state data and district data– More control variables available in district VAMs– For peer characteristics, use teacher-year level
averages in state VAMs, classroom averages in district VAMs
Each VAM uses three years of teaching data from 2008-2009 through 2010-2011
Data
4
Explore sensitivity to several specifications:– Exclude peer average characteristics (X̅; i,t)
– Exclude student characteristics (X̅i,t) and peer characteristics (X̅; i,t)
– Add scores from two prior years (Yi,t-2)
– Interact free/reduced lunch status with baseline scores
Estimate all models using the same set of student observations
Control for measurement error in prior test scores using an errors-in-variables approach
Empirical Bayes (shrinkage) adjusted estimates
Baseline Model
5
Student Controls(State)
Student Controls(District)
Peer Averages
(State)
Peer Averages (District)
Free or Reduced-Price Meals x x x x
Disability x x x x
Race/Ethnicity x x x x
Gender x x
English Language Learner x x x x
Age/Behind Grade Level x x
Gifted Program Participation x x
Lagged Rate of Attendance x x
Lagged Fraction of Year Suspended x x
Average Lagged Achievement x x
SD of Lagged Achievement x x
Number of Students in Class x
Student and Peer Characteristics
6
Correlation of 8th-Grade State Teacher VAM Estimates Relative to Baseline Specification
7
Math (N = 2,778)
Reading (N = 3,347)
Exclude peer characteristics 0.970 0.982
Exclude student and peer characteristics 0.964 0.979
Add scores from t-2 0.977 0.958
Add scores from t-2 and exclude student/peer characteristics 0.946 0.946
Baseline: Student characteristics, peer characteristics, and prior scores from t-1
Findings are based on VAM estimates from 2008–2009 to 2010–2011 on the same sample of students.
Remove Student/Peer Controls and Add t-2 Scores
1st (Lowest) 2nd 3rd 4th
5th
(Highest)
Baseline Model
1st (Lowest) 81 17 1 1 0
2nd 18 57 23 3 0
3rd 1 23 53 22 1
4th 0 3 22 59 16
5th (Highest) 0 0 1 16 83
Percentage of 8th-Grade Reading Teachers in Effectiveness Quintiles, by VAM Specification
8
Findings are based on VAM estimates for 3,347 reading teachers in grade 8 from 2008–2009 to 2010–2011. Correlation with baseline = 0.946.
How Are Teachers in One District Affected?
9
District has relatively large fraction poor and minority students
Math Grade 8 Reading Grade 8
District Percentile Rank 15 50 85 15 50 85
State Percentile Rank:
Baseline 21 62 86 17 66 90
Exclude peer characteristics 23 59 86 13 59 87
Exclude student and peer characteristics 20 58 85 13 57 89
Percentile Rank of District Teachers in State Distribution
Using Additional Controls in District Data
10
Math Grades 6-8 (N = 164)
Reading Grades 6-8(N = 215)
Exclude peer characteristics 0.955 0.963
Exclude student and peer characteristics 0.918 0.949
Add scores from t-2 0.987 0.967
Add scores from t-2 and exclude student/peer characteristics 0.927 0.909
Baseline: Student characteristics, peer characteristics, and prior scores from t-1
Findings are based on VAM estimates from 2008–2009 to 2010–2011 on the same sample of students.
Teacher-Year Average Student Characteristics vs. Classroom Average
11
Math Grades 6-8 (N = 164)
Reading Grades 6-8(N = 215)
Correlation Between Effect Estimates 0.956 0.975
Average Standard Error (Classroom) 0.065 0.073
Average Standard Error (Teacher) 0.069 0.075
Correlation of teacher effect estimates with baseline model above 0.99 for both subjects
Different Relationship Current and Prior Test Scores for FRL Students
12
Math Grade 8
(N = 2,778)
Reading Grade 8
(N = 3,347)
Non-FRL Student Coefficient on Prior Year Math Score (SE)
0.847(0.002)
0.199(0.003)
FRL student coefficient on Prior Year Math Score (SE)
0.896(0.003)
0.154(0.004)
Non-FRL Student Coefficient on Prior Year Reading Score (SE)
0.086(0.002)
0.641(0.003)
FRL student coefficient on Prior Year Reading Score (SE)
0.031(0.003)
0.728(0.004)
Teacher VAM estimates highly correlated across specifications– Choice of control variables– Use of teacher-year level averages in place of
classroom averages– Interaction between FRL status and prior scores
Choice of control variables can impact estimates for teachers of disadvantaged students
Conclusions
13
Other researchers have examined correlations in teacher effect estimates when different same-subject assessments are used as outcomes for teacher VAMs
The highest correlations these authors found are:– Lockwood et al. (2007): 0.46– Sass (2008): 0.48– Concoran et al. (2011): 0.62– Lipscomb et al. (2010): 0.61– Papay (2011): 0.54
Context for Results
14
Mathematica® is a registered trademark of Mathematica Policy Research.
Please contact– Matthew Johnson
– Stephen Lipscomb• [email protected]
– Brian Gill• [email protected]
For More Information
15
Top Related