Adaptive Design for the New National Teacher and Principal ... · • Given a short field period,...
Transcript of Adaptive Design for the New National Teacher and Principal ... · • Given a short field period,...
Adaptive Design for the NewNational Teacher andPrincipal Survey
Minsun Riddles, David Marker, Lou Rizzo, Erin Wiley, WestatAndrew Zukerberg, National Center for Education Statistics
• This presentation is intended to promote ideas. The viewsexpressed are part of ongoing research and do notnecessarily reflect the position of Westat or the U.S.Department of Education.
2
Disclaimer
• Starts in school year 2015-16
• Redesign of Schools and Staffing Survey (SASS)
3
National Teacher and Principal Survey (NTPS)
SASS NTPSCycle Every four years Every two yearsEstimating National and state NationalSamples School districts
Public/private schoolsPublic/private school principalsPublic/private school teachersPublic school libraries
Public schoolsPublic school principalsPublic school teachers
SASS Response Rates by Date
59% 62% 66% 70% 73%
2/7 3/6 4/3 5/1 7/10
Principals
58%67% 72%
77% 78%
4/3 5/1 5/15 6/5 7/10
Teachers
• Data collection every two years.
• Faster release of data.
• Integration with other ED collections.
• Reduce response burden.
5
Goals for NTPS
• Given a short field period, responsive design approach maynot be realistic.
• Conflict between “Go the Hard Road” vs. “Pick the LowHanging Fruit”
6
NTPS Adaptive Design Approach
“Go the Hard Road” “Pick the Low Hanging Fruit”Groves (2006)Groves & Peytcheva (2008)Schouten et al (2009)Schouten et al (2011)Särndal and Lundström (2008)
Groves and Heeringa (2006)
Low-propensity cases High-propensity casesReduce differences in responsepropensity
Maximize response rate
• Can we publish in a more timely fashion by cutting off datacollection early?
• What are the implications on estimates?
• Can we use paradata to identify methods that can reducepotential nonresponse bias?
7
NTPS Adaptive Design Research Questions
8
SASS Principal Response Rates by Cutoff Dates
Student enrollmentFree, reduced
price lunchCommunity typeSchool levelSchool type
0
20
40
60
80
100
2/7 3/6 4/3 5/1 Final (7/10)
9
SASS Teacher Response Rates by Cutoff Dates
Student enrollmentFree, reduced
price lunchCommunity typeSchool levelSchool type
0
20
40
60
80
100
4/3 5/1 5/15 6/5 Final (7/10)
10
SASS Teacher Response Ratesby Teacher Wave Group
Significantly different (p<0.005), Rao-Scott chi-square test
0%
20%
40%
60%
80%
100%
All Early Middle Late
11
SASS Follow-up Effort for Principal ResponsePovertyCommunity typeSchool type
Significantly different (p<0.05), Rao-Scott chi-square test
0%
20%
40%
60%
80%
100%
All Charter Traditional City Suburban Town Rural Highpoverty
Low/medpoverty
No follow-up One follow-up Multiple follow-ups Field Follow-up
12
SASS Follow-up Effort for Teacher ResponsePovertyCommunity typeSchool type School level
Significantly different (p<0.05), Rao-Scott chi-square test
0%
20%
40%
60%
80%
100%
No follow-up One follow-up Multiple follow-ups Field follow-up
13
SASS Follow-up Effort for Teacher Responseby Teacher Wave Group
Significantly different (p<0.05), Rao-Scott chi-square test
0%
20%
40%
60%
80%
100%
All Early Middle Late
No follow-up One follow-up Multiple follow-ups Field follow-up
• Create 8 data sets, 4 for principals, 4 for teachers, usingdifferent cutoff dates for the field period.
• Reproduce initial public release tables from each data set.
• Compare estimates to the final estimates
– Do some of the interim data sets produce similar estimates to thefinal data set?
– Ideally identify a response level at which similar results areachieved, but earlier time frame
14
SASS Counterfactual Scenarios with Early Cutoff
15
SASS Counterfactual Scenarios with Early Cutoff
*Significantly different (p<0.05), t-test
0%
2%
4%
6%
8%
10%
2/7* 3/6* 4/3* 5/1* Final (7/10)
Principal Estimates
Hispanic Non-Hispanic African American
16
SASS Counterfactual Scenarios with Early Cutoff
*Significantly different (p<0.05), t-test
0%
2%
4%
6%
8%
4/3* 5/1* 5/15* 6/5 Final (7/10)
Teacher Estimates
Non-Hispanic African American
17
SASS Survey Coordinator
• In sampled public schools, survey coordinators wererecruited.
Survey coordinator Principal response% Teacher response%No survey coordinator 26.4% 58.6%With survey coordinator 81.1% 76.4%
18
SASS Follow-up by Survey Coordinator Status
Significantly different (p<0.05), Rao-Scott chi-square test
0%
20%
40%
60%
80%
100%
All principals No surveycoordinator
With surveycoordinator
All teachers No surveycoordinator
With surveycoordinator
No follow-up One follow-up
Multiple follow-ups Field Follow-up
19
SASS Experimentation with Survey Coordinator
• Among sampled teachers in public schools with a surveycoordinator, an experiment was conducted.
– Teacher letter through coordinator (treatment group)
– Teacher letter not through coordinator (control group)
Teacher experimental group Teacher response%Teacher letter through coordinator 78.1%Teacher letter not through coordinator 74.7%
Significantly different (p<0.05), Rao-Scott chi-square test
20
SASS Follow-up by Teacher Experimental Group
Significantly different (p<0.05), Rao-Scott chi-square test
0%
20%
40%
60%
80%
100%
With survey coordinator Letter through surveycoordinator
Letter not through surveycoordinator
No follow-up One follow-up Multiple follow-ups Field Follow-up
• Field period length can be shortened with small loss inresponse rate and minimal change in estimates.
• Need to prepare for short field period for teacherquestionnaires with teacher listing form received late.
• Recruiting and making contact through a survey coordinatormay help to increase response rates and to reduce follow-up efforts.
21
Conclusions from SASS Results
• Paradata to be collected and analyzed.
– Modes of contact, dates for each contact, mail out dates, …
• Variable field efforts from the beginning for classes ofschools, principals, and teachers with possibly lowerresponse propensities (city schools, high poverty schools,minority or less experienced principals and teachers).
• Reduce response rate differences (“Go the hard road”).
• Ongoing use of experimentation and simulation.
22
NTPS Planned Initiatives for 2015-16 Cycle
23
Thank [email protected]