North Carolina New Teacher Support Program › files › 2015 › 07 › North...implement the North...
Transcript of North Carolina New Teacher Support Program › files › 2015 › 07 › North...implement the North...
North Carolina New Teacher Support
Program
Final Race to the Top Evaluation Report
Authors:
Kevin Bastian and Julie Marks
Education Policy Initiative at Carolina, The University of North
Carolina at Chapel Hill
August 2015
Consortium for Educational Research and Evaluation–North Carolina
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina
Table of Contents
Executive Summary ........................................................................................................................ 3
Introduction ..................................................................................................................................... 6
Data and Analyses........................................................................................................................... 8
Evaluation Sample ...................................................................................................................... 8
Data Sources ............................................................................................................................... 9
Analysis..................................................................................................................................... 10
Findings......................................................................................................................................... 11
Implementation ......................................................................................................................... 11
To What Extent was the NC NTSP Implemented as Intended? ........................................... 11
To What Extent did the NC NTSP Reach its Target Population? ........................................ 11
Knowledge, Skills, and Attitudes.............................................................................................. 13
How do Teachers Perceive the Impact of NC NTSP Components on their Knowledge,
Skills, and Attitudes about Teaching? .................................................................................. 13
How do Teachers Perceive the Impact of NC NTSP Components on Self-Efficacy and Job
Satisfaction? .......................................................................................................................... 15
Teacher Effectiveness ............................................................................................................... 16
To What Extent does the NC NTSP Impact Teacher Effectiveness, as Measured by Teacher
Value Added to Student Achievement? ................................................................................ 16
To What Extent does the NC NTSP Impact Teacher Effectiveness, as Measured by the
North Carolina Educator Evaluation System (NCEES) Teacher Evaluation Ratings? ........ 18
Retention ................................................................................................................................... 20
To What Extent does the NC NTSP Impact the Retention of Novice Teachers a) in the Same
Schools or LEAs, and b) in the State? .................................................................................. 20
Conclusions ................................................................................................................................... 22
Summary of Findings ................................................................................................................ 22
Implementation ..................................................................................................................... 22
Teacher Knowledge, Skills, and Attitudes ............................................................................ 22
Teacher Effectiveness ........................................................................................................... 22
Teacher Retention ................................................................................................................. 22
Limitations ................................................................................................................................ 23
Recommendations and Next Steps............................................................................................ 23
Appendix A. Evaluation Sample................................................................................................... 25
Appendix B. Data Sources ............................................................................................................ 30
Appendix C. To What Extent was the NC NTSP Implemented as it was Intended? ................... 42
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 2
Appendix D. To What Extent did the NC NTSP Reach its Target Population? ........................... 44
Appendix E. How do Teachers Perceive the Impact of the NC NTSP Components on their
Confidence, Knowledge, and Skills in Teaching? ........................................................................ 52
Appendix F. How do Teachers Perceive the Impact of the NC NTSP on their Self-Efficacy and
Job Satisfaction? ........................................................................................................................... 58
Appendix G. To What Extent does the NC NTSP Impact Teacher Effectiveness as Measured by
Teacher Value-Added to Student Achievement? .......................................................................... 61
Appendix H. To What Extent does the NC NTSP Impact Teacher Effectiveness as Measured by
the North Carolina Educator Evaluation System (NCEES) Teacher Evaluation Ratings? .......... 82
Appendix I. To What Extent does the NC NTSP Impact the Retention of Novice Teachers to the
Same School, LEA, and the State? ............................................................................................... 90
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 3
NORTH CAROLINA NEW TEACHER SUPPORT PROGRAM: FINAL RACE TO THE
TOP EVALUATION REPORT
Executive Summary
Overview
The North Carolina New Teacher Support Program (NC NTSP) was developed to provide
induction supports to beginning teachers in North Carolina’s lowest-achieving schools and to
meet two high-priority needs identified by the state’s Race to the Top (RttT) grant: (1) helping
teachers to succeed during their initial years in teaching; and (2) retaining qualified teachers,
particularly in high-need schools. The NC NTSP aims to improve the instructional knowledge,
skills, attitudes, effectiveness, and retention of participating teachers through the provision of
three support components: institutes (multi-day trainings); instructional coaching; and
professional development. This report reflects findings from an independent external evaluation
of the NC NTSP conducted by the Education Policy Initiative at Carolina as part of the statewide
RttT evaluation undertaken by the Consortium for Educational Research and Evaluation–North
Carolina (CERE–NC).
Data and Methods
This final evaluation report draws upon the following data sources: (1) participation records from
each of the components of the NC NTSP; (2) Perception of Success Inventory for Beginning
Teachers (PSI-BT) survey responses; and (3) student demographics and test scores, classroom
rosters, teacher evaluation ratings, EVAAS scores, certified salaries, school personnel, and
school characteristics files provided by the North Carolina Department of Public Instruction.
With these data sources, the Evaluation Team used a comparison group design to examine: levels
of participation in NC NTSP components; NC NTSP teachers’ perceptions of the impact of
program components on their confidence, knowledge, and skills for teaching; teachers’
perceptions of their self-efficacy and job satisfaction; and the impact of the NC NTSP on teacher
value-added, teacher evaluation ratings, and teacher retention.
Summary of Findings
This report reflects evaluation findings for the two full years of program implementation
spanning the 2012-13 and 2013-14 school years. The study design and analysis were developed
to address evaluation questions across four overarching areas: implementation; teacher
knowledge, skills, and attitudes; teacher effectiveness; and teacher retention.
Implementation
To what extent was the NC NTSP implemented as intended, and to what extent did it reach its
target population? Over the course of the four-year grant period, the NC NTSP was developed,
staffed, and implemented from the ground up and has grown to scale, serving over 1,100 teachers
in 114 schools as of the 2013-14 school year. There were substantial differences in program
participation and implementation across the four NC NTSP regions. As the NC NTSP doubled in
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 4
size in 2013-14, there was a drop in attendance at institutes and professional development
sessions and fewer instructional coach visits per teacher.
Teacher Knowledge, Skills, and Attitudes
How do teachers perceive the impact of NC NTSP components on their confidence, knowledge,
skills, and attitudes toward teaching? There was a statistically significant difference in the
proportion of NC NTSP evaluation sample respondents who felt the program components had a
positive impact on their teaching, compared with similar services provided by their own school.
This significant difference also existed between NC NTSP respondents’ perceptions of program
utility and comparison sample perceptions of analogous school-provided services.
Teacher Effectiveness
To what extent does the NC NTSP impact teacher effectiveness as measured by teacher value-
added to student achievement (EVAAS) and the North Carolina Educator Evaluation System
(NCEES) teacher evaluation ratings? Overall, NC NTSP evaluation sample teachers had
significantly higher EVAAS estimates than comparison sample teachers in fifth and eighth grade
science. When assessing results by cohort, positive and significant EVAAS results were
concentrated within NC NTSP Cohort 1 teachers, while NC NTSP Cohort 2 teachers were
generally no more or less effective. Regarding teacher evaluation ratings, there were no
significant differences between NC NTSP evaluation sample teachers and comparison teachers
in overall models. By cohort, NC NTSP Cohort 1 teachers had significantly higher evaluation
ratings on four standards in 2013-14.
Teacher Retention
To what extent does the NC NTSP impact the retention of novice teachers in the same schools or
local education agencies (LEAs) and in the state? Overall, NC NTSP evaluation sample teachers
were significantly more likely to return to teaching in North Carolina public schools, to the same
LEA, and to the same low-performing school. NC NTSP teachers from both cohorts were
significantly more likely than comparison sample teachers to return to the same low-performing
school.
Limitations
Two primary limitations should be taken into account when interpreting the findings in this
report: (1) the ability to isolate the impact of the NC NTSP is diminished due to the lack of a
comparison group of teachers working in schools similar to those served by the NC NTSP in all
ways other than participating in the program; and (2) given the time required for program
development and scaling up, the length of the evaluation period is not adequate to assess the
effectiveness of a consistent program model implemented as intended for first-, second-, and
third-year teachers over the course of three years.
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 5
Recommendations and Next Steps
There are three overarching recommendations for the NC NTSP after the close of the RttT grant:
1. The impacts of the NC NTSP on teacher value-added to student achievement and teacher
retention support a recommendation to sustain the program beyond the end of the RttT grant;
2. Findings from this evaluation should be used to explore the disparities in program
implementation by region, and moving forward, implementation fidelity should be formally
monitored; and
3. Strategies should be explored to counter the decline in program participation and
effectiveness seen in 2013-14, such as requiring participating schools and Local Education
Agencies (districts) to enforce mandatory participant attendance, and/or concentrating
resources toward instructional coaching (the most intensive program component).
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 6
Introduction
North Carolina’s $400 million, four-year Race to the Top (RttT) grant was built upon a
comprehensive plan to strengthen the education workforce, with the goal of having a great
teacher in every classroom and a great principal leading every school. With this end in mind, the
North Carolina Department of Public Instruction (NCDPI) identified the state’s highest-priority
needs, including the recruitment and preparation, equitable distribution, professional
development, and induction and retention of high-quality teachers.1
Data from the 2010 Teacher Working Conditions Survey illustrated the need for induction
supports prior to RttT, showing that while nearly all new teachers (93%) are assigned a mentor,
almost half (47%) do not have time during the day to meet with their mentors, and one in eight
indicate that they never received any additional support as new teachers.1,2
These supports are
particularly needed in high-need schools, where novice teachers are concentrated and where the
teacher turnover rate is often greatest.
To help teachers succeed during their early-career years and persist in the state’s highest-need
schools, NCDPI partnered with the UNC General Administration (UNC-GA) to develop and
implement the North Carolina New Teacher Support Program (NC NTSP). The NC NTSP is a
comprehensive induction program for early-career teachers designed to increase competency in
goal-setting, backwards planning and assessment, data-driven decision-making, classroom
management, and strategies for success in the school and community. The goal of the NC NTSP
is to help participating novice teachers acquire the knowledge and skills necessary to increase the
quality of their instruction, raise student achievement, and persist in teaching.
The NC NTSP drew from the successful induction model utilized by Teach For America to
create a three part program comprised of 1) institutes (multi-day training sessions); 2) intensive
face-to-face and virtual instructional coaching; and 3) professional development sessions held
throughout the academic year. Implementation of these beginning teacher supports is organized
through a central NC NTSP office and regional anchor sites located at four UNC system
institutions—East Carolina University (ECU), the UNC Center for School Leadership
Development (UNC-CSLD), UNC Charlotte (UNCC), and UNCG Greensboro (UNCG).
As part of the statewide evaluation of RttT conducted by the Consortium for Education Research
and Evaluation–North Carolina (CERE–NC),3 the Education Policy Initiative at Carolina (EPIC)
was tasked with conducting an independent external evaluation of the NC NTSP from the 2011-
12 to the 2013-14 school years. Over this time period, EPIC has collected and analyzed data on
program implementation and participation, short-term and intermediate outcomes, and
summative impact findings to address the following evaluation questions:
1 NC RttT Proposal, 2010
2 Teacher Working Conditions Survey, 2010
3 The Consortium for Educational Research and Evaluation–North Carolina (http://cerenc.org) is a partnership of:
the SERVE Center, University of North Carolina at Greensboro; the Education Policy Initiative at Carolina,
University of North Carolina at Chapel Hill; and the Friday Institute for Educational Innovation, North Carolina
State University.
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 7
Implementation
1. To what extent was the NC NTSP implemented as it was intended?
2. To what extent did the NC NTSP reach its target population?
Teacher Knowledge, Skills, and Attitudes
3. How do teachers perceive the impact of NC NTSP components on their confidence,
knowledge, and skills in teaching?
4. How do teachers perceive the impact of NC NTSP components on their self-efficacy and job
satisfaction?
Teacher Effectiveness
5. To what extent does the NC NTSP impact teacher effectiveness as measured by teacher
value-added to student achievement?
6. To what extent does the NC NTSP impact teacher effectiveness as measured by the North
Carolina Educator Evaluation System (NCEES) teacher evaluation ratings?
Teacher Retention
7. To what extent does the NC NTSP impact the retention of novice teachers in the same
schools or LEAs and in the state?
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 8
Data and Analyses
Evaluation Sample
For the 2012-13 academic year, the inclusion criteria for the NC NTSP evaluation sample was all
first- and second-year teachers who began receiving NC NTSP services by December 2012 and
who worked in schools that were eligible for and agreed to participate in the NC NTSP.
The exclusion criteria for the 2012-13 NC NTSP evaluation sample were as follows:
Teachers who began receiving NC NTSP supports in January 2013 or later;
Novice teachers receiving NC NTSP supports in non-RttT schools;
The small number of third-year teachers served by the program in 2012-13; and
Teach for America (TFA) corps members.
For the 2013-14 academic year, the Evaluation Team defined the NC NTSP evaluation sample as
all first-, second-, and third-year teachers who began receiving NC NTSP supports by December
2013 and who worked in schools that were eligible for and agreed to participate in the NC NTSP.
While this sample includes third-year teachers who were fully served by the program in 2013-14,
it again excludes teachers who began receiving NC NTSP supports in January 2014 or later,
novice teachers who received NC NTSP supports in non-RttT schools, and TFA corps members.
In the 2012-13 academic year, the NC NTSP evaluation sample consisted of 344 teachers
working in 59 schools and 16 LEAs, with a large majority (72%) of this treatment sample being
first-year teachers. In the 2013-14 academic year, the NC NTSP evaluation sample consisted of
808 teachers working in 91 schools and 25 LEAs. Approximately 50% of the treatment sample
were first-year teachers, 36% were second-year teachers, and 15% were in their third year of
teaching.
To best isolate the impact of the NC NTSP, the Evaluation Team used a comparison group
design to compare findings between NC NTSP program participants and comparison teachers
who did not receive the intervention. An optimal comparison group would be comprised of
teachers who were identical to the evaluation sample in every way other than participating in the
NC NTSP; however, identifying such a group poses a particular challenge for evaluations of
programs that are universally offered within the eligible target population, leaving two primary
options for comparison groups: (1) those who were not eligible for the program but are thought
to be similar due to characteristics that place them narrowly outside the program criteria, and (2)
those who are eligible for the program but did not participate.
To assess the impact of the NC NTSP, the Evaluation Team created two comparison groups. The
group included in the main body of this final evaluation report meets the criteria of Option One
above—schools that were in the bottom decile of school performance in 2011-12 but that were
not eligible for and did not receive NC NTSP services. The key limitation of this sample is that it
is comprised of schools that did not receive any RttT services, meaning comparisons to this
group do not allow for isolation of the impact of the NC NTSP from other RttT programs.
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 9
In the 2012-13 school year, this comparison group (labeled Non-RttT Comparison throughout the
report) consisted of all of the first- and second-year teachers who began working in these schools
by December 2012. In the 2013-14 school year, this comparison group consisted of all of the
first-, second-, and third-year teachers who began working in these schools by December 2013.4
Overall, the Non-RttT Comparison evaluation sample included 1,033 teachers working in 147
schools and 48 LEAs in the 2012-13 school year and 1,507 teachers working in 149 schools and
48 LEAs in the 2013-14 school year.
Table 1 presents school characteristics for the NC NTSP evaluation sample and the Non-RttT
Comparison evaluation sample. Across 2012-13 and 2013-14, NC NTSP teachers worked in
schools with more students qualifying for subsidized school meals, more racial and ethnic
minority students, a higher short-term suspension rate, and a lower school performance
composite than did comparison teachers. In 2012-13, NC NTSP schools also had a significantly
higher percentage of novice teachers. Please see Appendix A for more details on the NC NTSP
evaluation sample, both comparison samples,5 and additional characteristics of the treatment and
comparison sample schools.
Table 1. School Characteristics for the NC NTSP and Non-RttT Comparison Group
Characteristic
2012-13 School Year 2013-14 School Year
NC NTSP
Non-RttT
Comparison NC NTSP
Non-RttT
Comparison
Free and Reduced-Price
Lunch Percentage 93.58 85.93
** 91.78 87.40**
Racial/Ethnic
Minority Percentage 90.87 78.37
** 89.63 78.67**
Short-Term Suspension Rate
(Per 100 students) 48.86 33.52
** 38.14 25.50**
Performance Composite 20.04 23.25** 31.07 34.29
**
Novice Teacher Percentage 32.99 27.19** 32.93 30.29
Note: +, *, and ** indicate statistically significant differences at the 0.10, 0.05, and 0.01 levels, respectively.
Data Sources
The Evaluation Team collected the data presented in this final evaluation report from the UNC-
GA implementation team, the NC NTSP evaluation sample teachers and comparison sample
teachers, and administrative datasets supplied by NCDPI. Specifically, this final evaluation
report draws upon the following data sources: (1) participation records from each of the
components of the NC NTSP (institutes, instructional coaching, and professional development);
(2) the Perception of Success Inventory for Beginning Teachers (PSI-BT) survey responses of
NC NTSP evaluation sample teachers and Non-RttT Comparison sample teachers; and (3)
student test scores, student demographics, classroom rosters, teacher evaluation ratings, EVAAS
scores, certified salaries, school personnel, and school characteristics files provided by NCDPI.
4 Like the NC NTSP evaluation sample, the Non-RttT Comparison evaluation sample excludes TFA corps members.
5 The second sample is identified as the NC NTSP Eligible Comparison sample.
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 10
With these data sources, the Evaluation Team examined: levels of participation in NC NTSP
components; NC NTSP teachers’ perceptions of the impact of program components on their
confidence in, knowledge of, and skills for teaching; their self-efficacy and job satisfaction; and
the impact of the NC NTSP on teacher value-added measures, teacher evaluation ratings, and
teacher retention. Please see Appendix B for more details on program documents provided by the
UNC-GA implementation team and for PSI-BT survey items, response rates, and respondent
characteristics.
Analysis
This summative report utilizes data collected over a two-year period from two cohorts and across
four regional sites. While the NC NTSP was implemented using a structured program model
across regions and over time, natural variation is to be expected based upon the different needs
and contextual factors within each region. Furthermore, due to the annual scaling-up of the
program, different cohorts were exposed to different levels of program intensity. To account for
these differences, the Evaluation Team will present impact results for teacher value-added
measures, teacher evaluation ratings, and teacher retention in the following groupings:
Overall: To determine the overall impact of the NC NTSP, the Evaluation Team will present
results for the NC NTSP and Non-RttT Comparison evaluation sample teachers using pooled
data from the 2012-13 and 2013-14 school-years.
By Region: To examine any regional differences in outcomes, the Evaluation Team will
present results, separately, for each of the four regions served by the anchor institutions at
East Carolina University (ECU), the University of North Carolina Center for School
Leadership Development (UNC-CSLD), the University of North Carolina at Charlotte
(UNCC), and the University of North Carolina at Greensboro (UNCG). These results use
pooled data from 2012-13 and 2013-14.
By Cohort: To examine the impact of the NC NTSP based on when a teacher entered the
program, the Evaluation Team will present results, separately, for teachers who entered the
program in 2012-13 (Cohort 1) and those who entered the program in 2013-14 (Cohort 2).
Appendices G through I show value-added, evaluation rating, and retention results in reference
to a second comparison group (the NC NTSP Eligible Comparison sample), display results for
first-year teachers only, and include results from analyses with a more comprehensive set of NC
NTSP and comparison sample teachers.
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 11
Findings
Implementation
To What Extent was the NC NTSP Implemented as Intended?
After the receipt of RttT grant funds, the NC NTSP was developed from the ground up in the
2010-11 school year. As such, the model was scaled up over time as the program increased
capacity and there was increased buy-in from Local Education Agencies (LEAs)6 and schools
across the state.
As shown in Table 2, there was a 30-fold increase in the number of teachers served by the NC
NTSP over the course of the grant period. By 2013-14, the NC NTSP provided induction
services to 1,108 teachers in 114 schools and 27 LEAs. Overall, the NC NTSP provided
induction services to novice teachers in 73 RttT schools in 2012-13 and 91 RttT schools in 2013-
14.7 By the end of the RttT funding period, this coverage includes over three-quarters of the
lowest-achieving schools the program was tasked to serve.
Please see Appendix C for more details on the implementation of the NC NTSP.
Table 2. NC NTSP Program Implementation
Year Teachers Served Schools Served LEAs Served
2010-11* N/A N/A N/A
2011-12 35 13 5
2012-13 542 77 19
2013-14 1108 114 27
* Note: The 2010-11 school year was used for program development, and focused on program
planning, staffing, identification of anchor institutes, and school recruitment.
To What Extent did the NC NTSP Reach its Target Population?
The NC NTSP is comprised of three components: institutes (multi-day trainings); instructional
coaching; and professional development. In 2012-13, institutes were held at one centralized
location at the University of North Carolina at Chapel Hill; in 2013-14, there were regional
institutes held prior to the start of the school year, followed by a centralized institute in
September. All instructional coaching interactions and professional development opportunities
were organized and implemented at the regional level.
6 LEA is North Carolina’s term for traditional school districts and charter schools.
7 There were a small number of schools served by the NC NTSP that were not in the RttT sample. Please see
Appendix C for more details.
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 12
As shown in Table 3, the attendance and reach for all three NC NTSP components notably
declined between the 2012-13 and 2013-14 school years. In addition, there was notable
variability across regions in program attendance and reach.
In the 2012-13 school year, nearly 55% of all NC NTSP evaluation sample participants attended
an institute. There was considerable variation between regions, however, with almost 75% of
teachers from the ECU region attending an institute, compared with only 40% of teachers in the
UNCG region. In 2013-14, institute attendance decreased by over 50% across all regions and fell
to 21% overall.
Table 3. NC NTSP Evaluation Sample Participation
NC NTSP
Overall
ECU
Region
UNC-CSLD
Region
UNCC
Region
UNCG
Region
NC NTSP Institute
Attended a
2012-13 Institute 54.94% 74.24% 66.25% 46.67% 39.74%
Attended a
2013-14 Institute 20.54% 27.51% 18.99% 16.48% 19.50%
NC NTSP Instructional Coaching
Average Number of
In-Person Coaching
Visits Per Month in
2012-13
3.73 6.02 2.10 3.39 4.01
Average Number of
In-Person Coaching
Visits Per Month in
2013-14
2.39 3.82 1.65 2.56 1.77
NC NTSP Professional Development
Average Number of
PD Sessions Attended
in 2012-13
2.22 2.41 3.59 1.17 2.26
Average Number of
PD Sessions Attended
in 2013-14
0.76 1.03 1.22 0.14 0.52
The middle panel of Table 3 reflects a decrease in the frequency of instructional coaching visits
between the 2012-13 and 2013-14 school years and shows substantial variation in the average
number of in-person coaching visits between regions. Program-wide, teachers averaged 1.3
fewer in-person instructional coaching visits per month in 2013-14.8 Coaching visits declined
across all regions, with the largest decreases in the UNCG and ECU regions. In the 2012-13 and
2013-14 school years, the ECU region had the highest average number of instructional coaching
visits, while the UNC-CSLD had the lowest average number of instructional coaching visits.
8 The drop in the average number of in-person coaching visits was not due to the NC NTSP serving third-year
teachers in 2013-14 (a teacher group that averaged fewer coaching visits overall); rather, the average number of in-
person instructional coaching visits declined for first- and second-year teachers.
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 13
To further illustrate the variation in program intensity across regions, Figure 1 displays the
average number of total contact hours (in-person and virtual) between instructional coaches and
NC NTSP evaluation sample teachers in the 2013-14 school-year.9 Overall, teachers in the ECU
and UNCC regions averaged 41 and 34 contact hours, respectively, while teachers in the UNC-
CSLD and UNCG regions averaged approximately 10 contact hours.
Finally, the bottom panel of Table 3 (above) reports the average number of NC NTSP
professional development sessions participants attended. Overall, professional development
attendance was low, with a marked decline in the 2013-14 school year. In 2012-13, NC NTSP
teachers averaged 2.2 professional development sessions (out of six), with a high of 3.59
sessions in the UNC-CSLD region.10
In 2013-14, NC NTSP teachers averaged 0.76 professional
development sessions (out of six), with a high of 1.22 in the UNC-CSLD region and a low of
0.14 in the UNCC region. Please see Appendix D for more detailed information on participation
in NC NTSP components.
Figure 1. Average Total Contact Hours with Instructional Coaches (2013-14)
Knowledge, Skills, and Attitudes
How do Teachers Perceive the Impact of NC NTSP Components on their Knowledge, Skills, and
Attitudes about Teaching?
A survey assessing teachers’ perceptions of their knowledge, skills, and attitudes was
administered to the NC NTSP evaluation sample and the Non-RttT Comparison evaluation
sample teachers. The Evaluation Team asked questions to reflect items specific to NC NTSP
participants alone (e.g., related to institutes or instructional coaching), as well as items that
allowed NC NTSP and comparison sample teachers to respond (e.g., concerning analogous
support services provided by their schools or LEAs). Respondents were asked the extent to
which they agreed that each NC NTSP component (or analogous school-/LEA-provided
9 The UNC-GA program implementers only tracked and reported contact hours for the 2013-14 school year.
10 In 2012-13, several LEAs and schools in the UNC-CSLD region required their NC NTSP teachers to attend the
NC NTSP professional development sessions.
41.16
11.35
33.61
10.20
0
5
10
15
20
25
30
35
40
45
ECU Region UNC-CSLD Region UNCC Region UNCG Region
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 14
component) had been helpful in developing their confidence, knowledge, and skills in teaching.
Given the potential for response bias, due in part to lower than desirable response rates (see
Appendix B), survey results should be interpreted with caution.
The top panel of Table 4 presents the summative survey item for NC NTSP institutes. In 2012-13
and 2013-14, approximately 80% of NC NTSP survey respondents agreed or strongly agreed that
the NC NTSP institute was helpful in developing their confidence, knowledge, and skills in
teaching. Within years, however, there was considerable variation in responses by region,
ranging from 59% to 94% in 2012-13 and from 65% to 100% in 2013-14.
The middle panels of Table 4 display responses to two items related to NC NTSP instructional
coaching. In both the 2012-13 and 2013-14 school years, 77% of the NC NTSP evaluation
sample respondents agreed or strongly agreed that the program’s instructional coaching was
helpful in developing their confidence, knowledge, and skills in teaching. In 2012-13 and 2013-
14, approximately 60% of NC NTSP evaluation sample respondents attributed “quite a bit” or a
“great deal” of their teaching success to help from their NC NTSP instructional coaches.
Table 4. NC NTSP Summative Survey Items
NC NTSP
Overall
ECU
Region
UNC-CSLD
Region
UNCC
Region
UNCG
Region
NC NTSP Institute was helpful in developing my confidence, knowledge, and skills in teaching
Percentage Agree or
Strongly Agree in 2012-13 80.00 81.58 59.09 93.93 76.47
Percentage Agree or
Strongly Agree in 2013-14 82.48 83.72 64.70 75.00 100.00
NC NTSP Instructional Coaching was helpful in developing my confidence, knowledge, and skills in
teaching
Percentage Agree or
Strongly Agree in 2012-13 77.58 70.21 72.72 75.00 94.60
Percentage Agree or
Strongly Agree in 2013-14 77.05 81.25 66.21 78.16 72.12
Of the success you have had as a beginning teacher, what amount would you attribute to help from
your NC NTSP Instructional Coach
Percentage Responding
Quite a Bit or a Great Deal
in 2012-13
62.43 53.19 45.45 66.67 83.78
Percentage Responding
Quite a Bit or a Great Deal
in 2013-14
58.20 65.98 52.71 52.88 55.21
NC NTSP Professional Development was helpful in developing my confidence, knowledge, and skills
in teaching
Percentage Agree or
Strongly Agree in 2012-13 87.12 87.18 80.77 87.50 91.43
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 15
As a basis of comparison, Figure 2 displays the responses of NC NTSP evaluation sample
teachers to comparable items about their school-provided mentors. While 77% of NC NTSP
respondents indicated that their NC NTSP instructional coaches were helpful in developing their
confidence, knowledge, and skills in teaching, only 60% of NC NTSP respondents answered
similarly about their school-provided mentors. Likewise, while approximately 60% of NC NTSP
respondents attributed “quite a bit” or a “great deal” of their teaching success to their NC NTSP
instructional coaches, less than half of the NC NTSP respondents answered similarly for their
school-provided mentors. Both of these differences were statistically significant.
The last panel of Table 4 (above) presents the summative survey item for NC NTSP professional
development. In 2012-13, 87% of NC NTSP evaluation sample respondents agreed or strongly
agreed that the program’s professional development sessions were helpful in developing their
confidence, knowledge, and skills in teaching.11
Please see Appendix E for more details on NC NTSP teachers’ perceptions of program utility.
Figure 2. Teacher Perceptions of NC NTSP Quality Relative to School-Provided Support
How do Teachers Perceive the Impact of NC NTSP Components on Self-Efficacy and Job
Satisfaction?
The Evaluation Team assessed teachers’ perceptions of self-efficacy and job satisfaction with
five previously-validated items on the PSI-BT. As shown in Table 5 (following page), in both
2012-13 and 2013-14, the percentages of NC NTSP evaluation sample respondents who agreed
or strongly agreed with the self-efficacy and job satisfaction items were higher than the
percentages of Non-RttT Comparison evaluation sample respondents who did so. In 2012-13, NC
11
Due to an error in the response categories recorded for the 2013-14 summative professional development
question, the Evaluation Team is unable to present 2013-14 results.
49.09
60.10
45.92
60.38
58.10
77.05
62.43
77.58
0 10 20 30 40 50 60 70 80 90
Attributes "Quite a Bit" or "A Great Deal" of
Success to Coach/Mentor
Agrees/Strongly Agrees that Coach/Mentor Helpful
in Developing Confidence, Knowledge, and Skills in
Teaching
NC NTSP-Provided 2012-13 NC NTSP-Provided 2013-14
School-Provided 2012-13 School-Provided 2013-14
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 16
NTSP evaluation sample respondents registered significantly higher levels of agreement (84% to
78%) with the self-efficacy item, “I feel inspired to instruct students to the best of my ability.” In
2013-14, NC NTSP evaluation sample survey respondents rated their self-efficacy and job
satisfaction significantly higher than did comparison sample respondents on all five items. These
differences should be interpreted carefully, however, due to the low response rate for Non-RttT
Comparison teachers and the potential for bias in the sample of teachers who responded.
See Appendix B for more details on these response rates and differences between NC NTSP
teachers who did and did not respond to the PSI-BT, and see Appendix F for more details on the
NC NTSP and Non-RttT Comparison evaluation sample responses to the self-efficacy and job
satisfaction items.
Table 5. Self-Efficacy and Job Satisfaction Survey Items
Survey Items
Percentage Agree or Strongly Agree
NC NTSP
2012-13
Comparison
2012-13
NC NTSP
2013-14
Comparison
2013-14
I am able to successfully
teach students with a
variety of ability levels
78.92% 72.64% 84.07% 74.16%
I am able to motivate all
students 60.24% 54.07% 68.87% 54.50%
I feel inspired to instruct
students to the
best of my ability
84.15%
77.92% 81.52% 75.14%
In general, I am satisfied
with my current job 59.15% 54.55% 56.76% 47.75%
I consider teaching to be my
ideal career 70.73% 67.20% 68.07% 60.68%
Teacher Effectiveness
To What Extent does the NC NTSP Impact Teacher Effectiveness, as Measured by Teacher Value
Added to Student Achievement?
To assess the contributions of NC NTSP teachers to student achievement, the Evaluation Team
analyzed EVAAS teacher effectiveness estimates. The Evaluation Team combined EVAAS data
from elementary and middle grades to run separate models for mathematics, reading, and science
(End-of-Grade [EOG] science exams in grades five and eight), and combined EVAAS data from
End-of-Course (EOC) exams (Biology, English II, and Math I) to run a single EOC model. For
these models, the Evaluation Team made teacher EVAAS estimates the outcome variable and
regressed this measure of value added to student achievement on a set of school characteristics
and teacher experience. The Evaluation Team chose to control for these variables due to the
significant differences in school characteristics between NC NTSP and comparison sample
schools, and because these contextual variables are not accounted for in EVAAS models. Results
from these models express the adjusted-average differences in student achievement—in normal
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 17
curve equivalency units—between students taught by NC NTSP teachers and students taught by
Non-RttT Comparison sample teachers.
The top portion of Table 6 displays overall EVAAS results for NC NTSP evaluation sample
teachers in comparison to Non-RttT Comparison evaluation sample teachers. Over the 2012-13
and 2013-14 school years, students taught by NC NTSP evaluation sample teachers in fifth and
eighth grade science made significantly larger achievement gains than did students taught by
Non-RttT Comparison evaluation sample teachers. There were no significant differences for
EOG mathematics, EOG reading, and all EOC exams.
The bottom portion of Table 6 illustrates the heterogeneity of value-added results across regions.
Students taught by NC NTSP evaluation sample teachers in the ECU region made significantly
larger achievement gains in mathematics, reading, and fifth and eighth grade science than did
students taught by Non-RttT Comparison evaluation sample teachers. Conversely, students
taught by NC NTSP evaluation sample teachers in the ECU region made significantly smaller
achievement gains on EOC exams. NC NTSP evaluation sample teachers in the UNCC region
were significantly more effective in elementary and middle grades mathematics, and NC NTSP
evaluation sample teachers in the UNCG region were significantly more effective in courses with
EOC exams.
Table 6. NC NTSP Overall and Regional EVAAS Results (2012-13 and 2013-14)
Elementary
and Middle
Grades
Mathematics
Elementary
and Middle
Grades
Reading
5th
and 8th
Grade
Science
EOC
Exams
NC
NT
SP
vs.
Non
-
Rtt
T C
om
pari
son
Gro
up (
2012-1
3 a
nd
2013-1
4)
NC NTSP Overall 0.828 0.259 1.011+ 0.083
ECU Region 2.219**
1.116**
2.772*
-0.652*
UNC-CSLD Region -0.007 -0.161 1.052 -0.727
UNCC Region 1.915+ 0.236 0.234 -0.370
UNCG Region -1.033 -0.336 -0.051 1.214*
Note: +, *, and ** indicate statistically significant differences at the 0.10, 0.05, and 0.01 levels, respectively.
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 18
Table 7 presents considerable variability in teacher value added to student achievement by NC
NTSP cohort. NC NTSP Cohort 1 teachers—first served by the program in 2012-13—were
significantly more effective than Non-RttT Comparison evaluation sample teachers in elementary
and middle grades mathematics and reading in 2012-13 and when pooling data for two years
(2012-13 and 2013-14). Teachers from both NC NTSP cohorts were more effective in fifth and
eighth grade science in 2013-14. There were no other significant differences for NC NTSP
Cohort 2 teachers.
Table 7. NC NTSP EVAAS Results by Cohort
Elementary and Middle Grades
Math
Elementary and Middle Grades
Reading
Cohort 2012-13 2013-14
2012-13
2013-14 2012-13 2013-14
2012-13
2013-14
NC NTSP
Cohort 1 3.054
** 0.639 1.483+
1.670** 0.307 0.759
*
NC NTSP
Cohort 2 --- -0.189 -0.011 --- -0.183 -0.273
5th
and 8th
Grade Science EOC Exams
NC NTSP
Cohort 1 0.971 1.208
+ 1.127 0.693 -0.088 0.029
NC NTSP
Cohort 2 --- 1.261
+ 0.862 --- -0.122 0.202
Note: +, *, and ** indicate statistically significant differences at the 0.10, 0.05, and 0.01 levels, respectively.
Please see Appendix G for: (1) more details on the value-added methodology; (2) a complete set
of EVAAS results, including both comparison groups and models focusing on first-year teachers
only; (3) an alternative set of value-added models with student-level data; and (4) value-added
results with a more comprehensive set of NC NTSP and comparison sample teachers.
To What Extent does the NC NTSP Impact Teacher Effectiveness, as Measured by the North
Carolina Educator Evaluation System (NCEES) Teacher Evaluation Ratings?
Since many important aspects of teaching will not be fully captured by measures of teachers’
value added to student achievement, the Evaluation Team analyzed teachers’ evaluation ratings
on the five North Carolina Professional Teaching Standards (NCPTS), all of which are directly
assessed by school administrators. For these analyses, the Evaluation Team estimated models for
which the outcome variable was a teacher’s evaluation rating on a one-to-five scale (where one
was Not Demonstrated and five was Distinguished) and controlled for teacher experience and
school characteristics. Results from these models estimate the odds of NC NTSP evaluation
sample teachers receiving higher evaluation ratings than their Non-RttT Comparison evaluation
sample peers. Statistically significant results greater than one indicate higher evaluation ratings;
statistically significant results less than one indicate lower evaluation ratings.
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 19
Over the 2012-13 and 2013-14 school years, the top panel of Table 8 shows no significant
evaluation rating differences between NC NTSP evaluation sample teachers and Non-RttT
Comparison evaluation sample teachers. When analyzing the data by region, the bottom panel of
Table 8 shows that NC NTSP teachers in the UNCC region had significantly higher evaluation
ratings on Leadership (Standard 1) and Reflecting on Practice (Standard 5).
Table 8. Overall and Regional NC NTSP Evaluation Rating Results (2012-13 and 2013-14)
Standard
1
Standard
2
Standard
3
Standard
4
Standard
5
NC
NT
SP
vs
Non
-Rtt
T
Com
pari
son
(2012-1
3 a
nd
2013
-14)
NC NTSP Overall 1.072 0.960 0.993 0.975 0.996
ECU Region 0.878 0.921 0.666 0.655 0.941
UNC-CSLD Region 0.834 0.719 0.906 0.896 0.886
UNCC Region 1.822+ 1.601 1.613 1.588 1.735
+
UNCG Region 0.928 0.771 0.931 0.913 0.675
Note: +, *, and ** indicate statistically significant differences at the 0.10, 0.05, and 0.01 levels, respectively.
To determine whether evaluation ratings differ by NC NTSP cohort, Table 9 presents separate
evaluation rating results for Cohorts 1 and 2. For NC NTSP Cohort 1, there were no significant
differences in 2012-13; however, in their second year in the program (2013-14), NC NTSP
Cohort 1 teachers had significantly higher evaluation ratings on Leadership (Standard 1),
Content Knowledge (Standard 3), Facilitating Student Learning (Standard 4), and Reflecting on
Practice (Standard 5). In contrast, NC NTSP Cohort 2 teachers had significantly lower evaluation
ratings for Classroom Environment (Standard 2) in 2013-14 and when pooling data for two years
(2012-13 and 2013-14).
Please see Appendix H for: (1) more details on the evaluation rating models; (2) a complete set
of evaluation rating results, including both comparison groups and models focusing on first-year
teachers only; and (3) evaluation rating results with a more comprehensive set of NC NTSP and
comparison sample teachers.
Table 9. NC NTSP Evaluation Rating Results by Cohort
Cohort
Standard
1
Standard
2
Standard
3
Standard
4
Standard
5
2012-13 School Year
NC NTSP Cohort 1 1.082 1.128 0.997 0.842 0.980
NC NTSP Cohort 2 --- --- --- --- ---
2013-14 School Year
NC NTSP Cohort 1 1.722* 1.388 1.709
+ 1.844
* 1.572
+
NC NTSP Cohort 2 0.845 0.700+ 0.748 0.811 0.801
2012-13 and 2013-14 School Years
NC NTSP Cohort 1 1.376 1.267 1.323 1.203 1.257
NC NTSP Cohort 2 0.779 0.658* 0.694 0.752 0.730
Note: +, *, and ** indicate statistically significant differences at the 0.10, 0.05, and 0.01 levels, respectively.
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 20
Retention
To What Extent does the NC NTSP Impact the Retention of Novice Teachers a) in the Same
Schools or LEAs, and b) in the State?
To determine whether NC NTSP evaluation sample teachers were more likely to remain in
teaching than their Non-RttT Comparison evaluation sample peers, the Evaluation Team
estimated models for three types of retention: (1) returning to any North Carolina public school
in the following school year (2013-14 and 2014-15); (2) returning to the same LEA in the
following school year; and (3) returning to the same low-performing school in the following
school year. For these analyses, the outcome variable was a “1” if the teacher returned in the
following school year and a “0” if the teacher did not return. In these models, the Evaluation
Team controlled for teacher experience and school characteristics and, post-estimation,
converted the results to predicted retention probabilities to facilitate easier interpretation of the
results.
Overall, when pooling data from the 2012-13 and 2013-14 school years, NC NTSP evaluation
sample teachers were significantly more likely to return to a North Carolina public school and to
their same LEA than were Non-RttT Comparison evaluation sample teachers. However, given
the need to keep teachers in high-need environments, the most important retention results are for
teachers returning to the same low-performing school. As shown in Figure 3 (following page),
the program-wide predicted probability of school-level retention was 73%—significantly higher
than for the Non-RttT Comparison evaluation sample (66%). These school-level predicted
probabilities of retention were above 70% in all four NC NTSP regions and were significantly
higher than those for Non-RttT Comparison evaluation sample teachers in the ECU, UNCC, and
UNCG regions.
Examining school-level retention by cohort, Figure 4 (following page) indicates that NC NTSP
Cohort 1 teachers had significantly higher retention rates than did Non-RttT Comparison
evaluation sample teachers. When school-level retention was calculated for Cohort 2 in the 2014-
15 school year, results demonstrated that they also had significantly higher retention rates than
did Non-RttT Comparison evaluation sample teachers.
Please see Appendix I for: (1) more details on the retention models; (2) a complete set of
retention results, including both comparison groups and models focusing on first-year teachers
only; and (3) retention results with a more comprehensive set of NC NTSP and comparison
sample teachers.
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 21
Figure 3. NC NTSP School-Level Retention (2013-14 and 2014-15)
Note: +, *, and ** indicate statistically significant differences at the 0.10, 0.05, and 0.01 levels, respectively.
Figure 4. NC NTSP School-Level Retention Results by Cohort
Note: +, *, and ** indicate statistically significant differences at the 0.10, 0.05, and 0.01 levels, respectively.
NC NTSP Cohort 2 does not have a bar for returning in 2013-14 because they entered the program in 2013-14.
73.04** 77.26**
70.22 72.66+ 72.35+
65.78
0
20
40
60
80
NC NTSP
Overall
ECU Region UNC-CSLD
Region
UNCC Region UNCG Region Non-RttT
Comparison
Pre
dic
ted
Pro
ba
bil
ity
of
Ret
urn
ing
74.11+ 75.76** 74.18** 71.12* 71.67*
67.14 64.92 65.78
0
20
40
60
80
Returns in 2013-14 Returns in 2014-15 Overall: 2013-14 & 2014-15
Pre
dic
ted
Pro
ba
bil
ity
of
Ret
urn
ing
NC NTSP Cohort 1 NC NTSP Cohort 2 Non-RttT Comparison
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 22
Conclusions
Summary of Findings
Implementation
Over the course of the four-year grant period, the NC NTSP was developed, staffed, and
implemented from the ground up and has grown to scale, serving over 1,100 teachers in 114
schools as of the 2013-14 school year. There were substantial differences in program
participation and implementation across the four NC NTSP regions. Furthermore, as the NC
NTSP doubled in size in 2013-14, there was a drop in attendance at institutes and professional
development sessions, as well as fewer instructional coach visits per teacher.
Teacher Knowledge, Skills, and Attitudes
A large majority of NC NTSP evaluation sample teachers who responded to the PSI-BT survey
felt that the program components helped their confidence, knowledge, and skills in teaching. In
addition, there was a statistically significant difference in the proportion of NC NTSP evaluation
sample respondents who felt the program components had a positive impact on their teaching,
compared to similar services provided by their own school. This significant difference also
existed between NC NTSP perceptions of program utility and Non-RttT Comparison sample
perceptions of analogous school-provided services.
Teacher Effectiveness
Overall, NC NTSP evaluation sample teachers were significantly more effective than Non-RttT
Comparison evaluation sample teachers in fifth and eighth grade science, based on students’
End-of-Grade exam results. When assessing results by cohort, significant EVAAS results were
concentrated within NC NTSP Cohort 1 teachers—Cohort 1 teachers were significantly more
effective in elementary and middle grades mathematics and reading (both in 2012-13 in isolation
and with two years of pooled data) and were more effective in fifth and eighth grade science in
2013-14. There was only one positive and significant result for Cohort 2 teachers, who were
significantly more effective than Non-RttT Comparison evaluation sample teachers in fifth and
eighth grade science in 2013-14.
In terms of teacher evaluation ratings, there were no significant differences between NC NTSP
evaluation sample teachers and Non-RttT Comparison evaluation sample teachers in the overall
models. By cohort, NC NTSP Cohort 1 teachers had significantly higher evaluation ratings on
four North Carolina Professional Teaching Standards (Leadership, Content Knowledge,
Facilitating Student Learning, and Reflecting on Practice) in the 2013-14 school year, while
Cohort 2 teachers had significantly lower evaluation ratings on a fifth Standard (Classroom
Environment).
Teacher Retention
Overall, NC NTSP evaluation sample teachers were significantly more likely to return to
teaching in North Carolina public schools, to the same LEA, and to the same low-performing
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 23
schools in comparison to Non-RttT Comparison evaluation sample teachers. Focusing on school-
level retention, NC NTSP teachers from both cohorts were significantly more likely to return to
the same low-performing school than were Non-RttT Comparison evaluation sample teachers.
Limitations
Two primary limitations should be taken into account when interpreting the findings in this
report:
1. The Evaluation Team’s ability to isolate the impact of the NC NTSP is diminished by two
factors. First, the NC NTSP serves schools that received the highest proportion of RttT
interventions, some of which—District and School Transformation—are providing support
services similar to those provided by the NC NTSP. Thus, choosing comparison sample
schools outside of this eligible sample, like the Non-RttT Comparison group, precludes the
ability to hold these additional RttT interventions constant. Second, the NC NTSP is
universally offered in all of North Carolina’s lowest-achieving schools, meaning
comparisons with schools that are eligible yet non-participating may not take into account
potential confounding factors related to selection into the program. If there are differences
between the NC NTSP sample and the comparison groups that the Evaluation Team does
not control for and that influence the outcomes of interest, then the evaluation results will
be biased.
2. A second challenge of this evaluation lies in the fact that the data used in these analyses
represent three program components (institutes, instructional coaching, and professional
development) implemented across four regional sites with two cohorts of teachers and in a
model that has greatly expanded in scope over the course of the evaluation period. Simply
put, given the time required for program development and scaling up, a longer evaluation
period is needed to assess adequately the effectiveness of a consistent program model
implemented as intended for first-, second-, and third-year teachers over the course of three
years.
Recommendations and Next Steps
There are three overarching recommendations for the NC NTSP after the close of the RttT grant:
1. Given the overwhelming need for effective teachers who remain in low-performing schools,
the impacts of the NC NTSP on teacher value added to student achievement and teacher
retention support a recommendation to sustain the program beyond the end of the RttT
grant.
2. Findings from this evaluation should be used to explore the disparities in program
implementation by region. Going forward, the Evaluation Team recommends that program
implementation fidelity be monitored formally to obtain reasonable consistency between
regions.
3. Further study is required to determine whether the decline in program participation and
effectiveness in the 2013-14 school year is an artifact of the substantial scaling-up of the
program between 2012-13 and 2013-14. If the efficacy of the program remains diluted
when implemented at scale, the program implementers should explore options such as: (1)
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 24
making agreements with participating LEAs and schools to require attendance at institutes
and professional development; (2) concentrating program resources on higher-intensity
instructional coaching; and/or (3) limiting program size to ensure a high-intensity program.
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 25
Appendix A. Evaluation Sample
The Evaluation Team defined the NC NTSP evaluation sample for the 2012-13 academic year as
all first- and second-year teachers who began receiving NC NTSP services by December 2012
and who worked in schools that were eligible for and agreed to participate in the NC NTSP (RttT
schools).12
This excludes teachers who began receiving NC NTSP supports in January 2013 or
later, novice teachers receiving NC NTSP supports in non-RttT schools, the small number of
third-year teachers served by the program in 2012-13, and TFA corps members. For the 2013-14
academic year, the Evaluation Team defined the NC NTSP evaluation sample as all first-,
second-, and third-year teachers who began receiving NC NTSP supports by December 2013 and
who worked in schools that were eligible for (RttT schools) and agreed to participate in the NC
NTSP. While this includes third-year teachers, who were fully served by the program in 2013-
14, this again excludes teachers who began receiving NC NTSP supports in January 2014 or
later, novice teachers receiving NC NTSP supports in non-RttT schools, and TFA corps
members. For this final RttT evaluation report, the Evaluation Team excluded these groups
because (1) estimates of program performance need to be based upon a sample of teachers who
received NC NTSP supports for a majority of the school year; (2) the primary objective of the
Evaluation Team is to evaluate the performance of the program in its intended treatment area—
RttT schools; and (3) TFA corps members are significantly more likely to exit teaching after
their two-year service commitment, are significantly more effective, on average, than other
novice teachers (particularly in mathematics and science courses), and already receive induction
services from TFA, and thus, did not fully participate in the NC NTSP.
As shown in the left panel of Table A1 (following page), in the 2012-13 academic year the NC
NTSP evaluation sample consisted of 344 teachers working in 59 schools and 16 LEAs. A large
majority (72%) of this treatment sample were first-year teachers, and as stated above, all these
teachers worked in RttT schools and entered the program by December 2012. The right panel of
Table A1 shows that in the 2013-14 academic year the NC NTSP evaluation sample consisted of
808 teachers working in 91 schools and 25 LEAs. Nearly 50% of this treatment sample were
first-year teachers, 36% were second-year teachers, and 15% were in their third year of teaching.
All these teachers worked in RttT schools and entered the NC NTSP by December 2013.
12
The directive of the NC NTSP is to provide comprehensive induction services to novice teachers employed in the
state’s lowest-performing schools—schools that in the year before RttT began were either in the lowest 5% of all
schools in terms of student achievement or had graduation rates below 60%. We refer to these schools as “RttT
schools.”
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 26
Table A1. NC NTSP Evaluation Sample
2012-13
NC NTSP
Evaluation Sample
2013-14
NC NTSP
Evaluation Sample
Teacher Count 344 808
Teachers by Region
ECU 66 189
UNC-CSLD 80 237
UNCC 120 182
UNCG 78 200
1st Year Teacher % 71.80 48.51
2nd
Year Teacher % 28.20 36.14
3rd
Year Teacher % n/a 15.35
Teachers in RttT Schools 344 808
Teachers Served by NC NTSP
Before January 344 808
TFA Corps Members n/a n/a
School Count 59 91
Schools by Region
ECU 12 23
UNC-CSLD 15 29
UNCC 18 19
UNCG 14 20
LEA Count 16 25
LEAs by Region
ECU 5 9
UNC-CSLD 4 5
UNCC 4 6
UNCG 3 5
To assess the impact of the NC NTSP, the Evaluation Team implemented a comparison group
design to contrast the outcomes of NC NTSP teachers with those of other novice teachers
working in low-performing schools. Isolating the effects of the NC NTSP on participating
teachers is particularly challenging because several other RttT interventions were also
concentrated in RttT schools over the same time period. The most notable was the District and
School Transformation (DST) initiative, which provided regular professional development and
coaching designed to improve the effectiveness and retention of teachers in RttT schools.
To address these challenges, the Evaluation Team created two different comparison groups to
better isolate the impact of the NC NTSP. For the first group, the Evaluation Team used school
performance composite data from the 2011-12 school year to identify schools in the bottom
decile of performance that were not eligible for (non-RttT schools) and did not receive NC NTSP
services. In the 2012-13 school year this comparison group, labeled Non-RttT Comparison,
consisted of all the first- and second-year teachers who began working in these schools by
December 2012. In the 2013-14 school year this comparison group consisted of all the first-,
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 27
second-, and third-year teachers who began working in these schools by December 2013.13
Like
the NC NTSP evaluation sample, this Non-RttT Comparison evaluation sample excludes TFA
corps members and those beginning work in these schools after December. Overall, the left panel
of Table A2 shows that the evaluation sample for the Non-RttT Comparison group consisted of
1,033 teachers working in 147 schools and 48 LEAs in the 2012-13 school year; in 2013-14, the
Non-RttT Comparison evaluation sample consisted of 1,507 teachers working in 149 schools and
48 LEAs.
For the second comparison group, the Evaluation Team identified RttT schools (eligible for the
NC NTSP) that did not participate in the NC NTSP. In the 2012-13 school year this comparison
group, labeled NC NTSP Eligible Comparison, consisted of all the first- and second-year
teachers who began working in these schools by December 2012. In the 2013-14 school year this
comparison group consisted of all the first-, second-, and third-year teachers who began working
in these schools by December 2013. As with the evaluation samples for the NC NTSP and the
Non-RttT Comparison, the evaluation sample for the NC NTSP Eligible Comparison group
excludes TFA corps members and those beginning work in these schools after December.
Overall, the right panel of Table A2 shows that the evaluation sample for the NC NTSP Eligible
Comparison group consisted of 201 teachers working in 32 schools and 18 LEAs in the 2012-13
school year. In 2013-14 the NC NTSP expanded to serve an additional set of RttT schools.
Therefore, in the 2013-14 school year, the evaluation sample for the NC NTSP Eligible
Comparison group was reduced in size and consisted of 169 teachers working in 16 schools and
11 LEAs. In the main body of the final evaluation report, the Evaluation Team only reports
results for the Non-RttT Comparison sample; throughout the report appendices, the Evaluation
Team provides results for both the Non-RttT Comparison and the NC NTSP Eligible Comparison
groups.
Table A2. Evaluation Sample Comparison Groups
Like the NC NTSP sample, both of these comparison groups consist of novice teachers working
in low-performing schools. The Non-RttT Comparison group provides a larger sample for
13
Because the NC NTSP fully served third year teachers in the 2013-14 school year, the Evaluation Team included
third year teachers in both comparison samples in 2013-14.
Characteristics
Non-RttT Comparison NC NTSP Eligible Comparison
2012-13
Evaluation
Sample
2013-14
Evaluation
Sample
2012-13
Evaluation
Sample
2013-14
Evaluation
Sample
Teacher Count 1033 1507 201 169
1st Year Teacher % 60.47 40.69 60.70 39.05
2nd
Year Teacher % 39.53 34.77 39.30 33.14
3rd
Year Teacher % n/a 24.53 n/a 27.81
Teachers Hired Before
January 1033 1507 201 169
TFA Corps Members n/a n/a n/a n/a
School Count 147 149 32 16
LEA Count 48 48 18 11
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 28
analyses; however, since it is comprised of schools that did not receive RttT services,
comparisons to this group do not allow for isolation of the impact of the NC NTSP from other
RttT programs. The NC NTSP Eligible Comparison group addresses this concern by comparing
NC NTSP teachers to novice teachers who received other RttT supports (primarily through
DST). Because this group is much smaller, it provides less statistical power for detecting
differences in outcomes. In addition, it is unknown why these schools declined to participate in
the NC NTSP (in 2012-13 and 2013-14) and why some schools chose to enter the program in
2013-14. Therefore, analyses may not take into account other factors related to non-participation.
Throughout subsequent appendices, the Evaluation Team presents results for the NC NTSP
evaluation sample and the Non-RttT Comparison and NC NTSP Eligible Comparison evaluation
samples. In appendices focused on teacher value-added to student achievement, teacher
evaluation ratings, and teacher retention, the Evaluation Team presents two additional sets of
results: (1) those for all teachers served by the NC NTSP and all novice teachers in comparison
sample schools, and (2) those for an amended NC NTSP evaluation sample that includes teachers
served in non-RttT schools who entered the NC NTSP by December of the academic year and
were not TFA corps members.
Table A3 (following page) presents school characteristics for the NC NTSP evaluation sample
and for the evaluation samples for each of the comparison groups in 2012-13 and 2013-14. In
2012-13, NC NTSP teachers worked in schools with (1) more students qualifying for subsidized
school meals; (2) more racial and ethnic minority students; (3) higher short-term suspension rates
and higher violent acts rates (in comparison to NC NTSP Eligible Comparison schools); (4)
higher percentages of novice teachers; and (5) lower performance composites (in comparison to
the Non-RttT Comparison schools). In 2013-14, NC NTSP teachers worked in schools with (1)
more students qualifying for subsidized school meals (in comparison to Non-RttT Comparison
schools); (2) more racial and ethnic minority students; (3) higher short-term suspension rates (in
comparison to Non-RttT Comparison schools); and (4) lower performance composites.
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 29
Table A3. School Characteristics for the NC NTSP and Comparison Groups
Characteristic
2012-13 School Year 2013-14 School Year
NC NTSP
Sample
Non-RttT
Comparison
Group
NC NTSP
Eligible
Comparison
Group
NC NTSP
Sample
Non-RttT
Comparison
Group
NC NTSP
Eligible
Comparison
Group
Free and Reduced-Price
Lunch Percentage 93.58 85.93
** 87.16
+ 91.78 87.40** 86.54
Racial/Ethnic
Minority Percentage 90.87 78.37
** 82.31
* 89.63 78.67**
80.55+
Short-Term Suspension
Rate (Per 100 Students) 48.86 33.52
** 27.56
** 38.14 25.50** 26.30
Violent Acts Rate
(Per 1000 Students) 12.24 10.96 4.82
** 8.71 10.46 6.30
Total Per-Pupil
Expenditures $11,001 $10,052
+ $11,136 $10,534 $10,307 $10,214
Performance Composite 20.04 23.25** 22.55 31.07 34.29
** 39.94
*
Novice Teacher
Percentage 32.99 27.19
** 26.59
** 32.93 30.29 32.63
School Level
Elementary/Elementary-
Middle Combination
35
(59.32%)
89
(60.54%)
23
(71.88%)
57
(62.64%)
94
(63.09%)
10
(62.50%)
Middle School 10
(16.95%)
24
(16.33%)
2
(6.25%)
13
(14.29%)
23
(15.44%)
1
(6.25%)
High School 14
(23.73%)
33
(22.45%)
7
(21.88%)
21
(23.08%)
31
(20.81%)
5
(31.25%)
K-12 School 0
(0.00%)
1
(0.68%)
0
(0.00%)
0
(0.00%)
1
(0.67%)
0
(0.00%)
School Count 59 147 32 91 149 16
Note: This table displays school characteristics for schools in the NC NTSP, Non-RttT Comparison, and NC NTSP Eligible Comparison samples. +, *, and
** indicate statistically significant differences between NC NTSP schools and Non-RttT Comparison/NCNTSP Eligible Comparison schools at the 0.10,
0.05, and 0.01 levels, respectively.
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 30
Appendix B. Data Sources
The Evaluation Team collected the data presented in this final evaluation report from the UNC-
GA implementation team, NC NTSP evaluation sample teachers and comparison sample
teachers, and administrative datasets supplied by the North Carolina Department of Public
Instruction (NCDPI). Specifically, this final evaluation report draws upon the following data
sources: (1) participation records from each of the components of the NC NTSP—institutes,
instructional coaching, and professional development; (2) survey responses by NC NTSP
evaluation sample teachers and Non-RttT Comparison sample teachers; and (3) student test
scores, student demographics, classroom rosters, teacher evaluation ratings, certified salaries,
school personnel, and school characteristics files provided by NCDPI. With these data sources
the Evaluation Team examined levels of participation in NC NTSP components, NC NTSP
teachers’ perceptions of program utility and self-efficacy/job satisfaction, and the impact of the
NC NTSP on teacher value-added to student achievement, teacher evaluation ratings, and teacher
retention. Below, we elaborate on the data sources used in this final evaluation report.
NC NTSP Program Documents
To examine the evaluation sample’s participation in the components of the NC NTSP, the UNC-
GA program implementers supplied the Evaluation Team with requested program documents.
For the 2012-13 year this included (1) the curricula, agendas, and rosters of attendees for the
2012 NC NTSP Summer and Winter Institutes; (2) counts of NC NTSP instructional coach visits
with each teacher; and (3) the agendas and rosters of attendees for NC NTSP professional
development sessions. Data for the 2013-14 year included (1) the curricula, agendas, and rosters
of attendees for the 2013 NC NTSP Regional Institutes and Statewide Institute; (2) counts of in-
person and virtual NC NTSP instructional coach visits with each teacher; (3) contact hours—in-
person and virtual—between NC NTSP teachers and program instructional coaches; and (4) the
agendas and rosters of attendees for NC NTSP professional development sessions.
Survey Responses by NC NTSP and Non-RttT Comparison Sample Teachers
To assess (1) the perceptions of NC NTSP teachers regarding the focus and utility of NC NTSP
components, and (2) the perceptions of both NC NTSP and Non-RttT Comparison sample
teachers regarding school-provided novice teacher supports, school context, teacher practices,
self-efficacy, and job satisfaction, the Evaluation Team partnered with North Carolina State
University’s (NCSU) College of Education to administer the Perceptions of Success Inventory—
Beginning Teachers (PSI-BT) survey in the spring of 2013 and 2014 (see the end of Appendix B
for a complete survey).14
For each survey administration the sample included NC NTSP and
Non-RttT Comparison sample teachers in the evaluation sample. Both sets of teachers completed
the regular PSI-BT items. In addition, NC NTSP teachers completed items developed by the
Evaluation Team assessing the components—institutes, instructional coaching, professional
development—of the NC NTSP. Researchers at NCSU used both online and paper-based
mediums to administer the PSI-BT in spring 2013; in spring 2014 NCSU researchers
administered the survey online only.
14
Please see http://ncsu.edu/succeed/beginning-teachers/ for more information on the development and use of the
PSI-BT instrument.
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 31
For the 2012-13 school year, the left panel of Table B1 presents response rates on the PSI-BT for
the NC NTSP and Non-RttT Comparison samples overall, and for each NC NTSP region. Nearly
half of the NC NTSP teachers in the evaluation sample responded to the survey (49.26%), with
higher response rates for those in the ECU region. To encourage responses in the Non-RttT
Comparison sample, the Evaluation Team provided a $5 financial incentive to survey
completers; approximately 41% of Non-RttT Comparison sample teachers responded to the
survey. The right panel of Table B1 displays PSI-BT response rates for the 2013-14 school year.
Slightly more than half of the NC NTSP teachers in the evaluation sample responded to the
survey (51.12%), with higher response rates for the ECU region and lower response rates for the
UNC-CSLD region. As in 2012-13, the Evaluation Team offered a $5 financial incentive to Non-
RttT Comparison sample teachers to complete the survey but only 17% responded in the 2013-14
school year.
Table B1. PSI-BT Response Rates
Survey Group
2012-13 PSI-BT Administration 2013-14 PSI-BT Administration
Survey
Respondents
Administered
Surveys
Response
Rate
Survey
Respondents
Administered
Surveys
Response
Rate
NC NTSP Overall 167 339 49.26% 409 800 51.12%
ECU 47 66 71.21% 147 187 78.61%
UNC-CSLD 33 77 42.86% 78 236 33.05%
UNCC 49 118 41.53% 88 179 49.16%
UNCG 38 78 48.72% 96 198 48.48%
Non-RttT
Comparison Group 308 759 40.58% 179 1053 16.99%
Due to the possibility for bias in the respondent sample, Table B2 (following page) displays data
on participation in NC NTSP components for those NC NTSP teachers who did and did not
respond to the survey (both overall and by region). In the 2012-13 school year, NC NTSP
teachers who responded to the PSI-BT were more likely to attend an institute (66% for
respondents versus 45% for non-respondents), received more instructional coaching visits per
month (4.18 visits for respondents versus 3.33 for non-respondents), and attended more
professional development sessions (2.87 PD sessions for respondents versus 1.60 PD sessions for
non-respondents). In the 2013-14 school year, differences in program participation between PSI-
BT respondents and non-respondents were reduced in magnitude; however, respondents were
still more likely to attend an institute, to receive more instructional coaching visits, and to attend
more professional development sessions. Given these differences, it is important to carefully
interpret all survey results.
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 32
Table B2. Participation in NC NTSP Components, by PSI-BT Response Status
Region
Institute Attendance
Average Instructional
Coaching Visits
Per Month
Number of Professional
Development Sessions
Attended
Respond
Non
Respond Respond
Non
Respond Respond
Non
Respond
2012
-13 P
SI-
BT
Ad
min
istr
ati
on
NC NTSP
Overall 66.47% 45.35% 4.18 3.33 2.87 1.60
ECU 80.85% 57.89% 6.35 5.20 2.89 1.21
UNC-CSLD 66.67% 70.45% 2.26 1.94 4.15 3.32
UNCC 67.35% 33.33% 3.53 3.31 1.86 0.67
UNCG 47.37% 32.50% 4.02 4.01 3.03 1.53
2013-1
4 P
SI-
BT
Ad
min
istr
ati
on
NC NTSP
Overall 23.47% 17.14% 2.86 1.92 0.93 0.58
ECU 27.89% 25.00% 4.11 2.80 1.17 0.45
UNC-CSLD 21.79% 17.72% 1.81 1.59 1.60 1.04
UNCC 19.32% 13.19% 2.81 2.36 0.10 0.15
UNCG 21.88% 16.67% 1.87 1.70 0.79 0.27
Administrative Data from NCDPI
To assess the impact of the NC NTSP on teacher value-added to student achievement, teacher
evaluation ratings on the North Carolina Professional Teaching Standards (NCPTS), and teacher
retention, the Evaluation Team used administrative datasets provided by NCDPI. Specifically, to
estimate teacher value-added, the Evaluation Team used student test scores and demographics,
classroom rosters, school personnel, and school characteristics files. This allows for the
connection of students to their prior scores, demographics, teachers, and classroom peers;
teachers to their characteristics; and students and teachers to the characteristics of their schools.
To examine teacher evaluation ratings, the Evaluation Team used ratings on each of the five
NCPTS that are directly assessed by school administrators. Finally, for teacher retention, the
Evaluation Team used certified salary files to determine whether individuals returned to a
teaching position—overall, within the same LEA, or within the school school—in North
Carolina public schools.
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 33
PSI-BT Survey
Unless otherwise noted, PSI-BT items have a six-point scale: strongly disagree, disagree,
slightly disagree, slightly agree, agree, and strongly agree.
Domain One: School- or LEA-Provided Mentor Support
Directions: Please respond to the survey questions below concerning the mentor provided by
your SCHOOL or SCHOOL DISTRICT.
1) In my current experience in my school, a mentoring relationship is or would be important to me.
2) Has your school or school district assigned you a mentor?
-If yes, continue answering mentoring questions
-If no, skip to colleague support
3) In what month did you first meet with your school or school district assigned mentor?
1) July, 2) August, 3) September . . . 12) June
4) How much time do you spend meeting with your school or school district provided mentor
each month?
1) 0-30 minutes, 2) 30 minutes to 1 hour, 3) 1-2 hours, 4) 2-3 hours, 5) 3-4 hours, 6) 4-5
hours, or 7) more than 5 hours
5) If you have a subject or grade-level specialty, does your school or school district assigned
mentor teach the same subject or grade-level?
1) Yes, 2) No, or 3) Not applicable
6) My school or school district assigned mentor has provided assistance with classroom
management.
7) My school or school district assigned mentor has provided assistance with instructional
concerns.
8) My school or school district assigned mentor has provided assistance related to
communication with parents or caregivers of my students.
9) My school or school district assigned mentor has provided assistance with unit or lesson
planning.
10) My school or school district assigned mentor has provided assistance with analysis of student
assessment data.
11) My school or school district assigned mentor is empathetic.
12) My school or school district assigned mentor encourages me to reflect about my teaching.
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 34
13) Of the success you have had as a beginning teacher, what amount would you attribute to help
from your school or school district assigned mentor?
1) a great deal, 2) quite a bit, 3) some, 4) hardly any, or 5) none
14) Overall, my school or school district assigned mentor has been helpful in developing my
confidence, knowledge, and skills in teaching.
Domain Two: Colleague Support
Directions: Please respond to the survey questions below concerning the support you receive
from colleagues in your school.
15) I have opportunities for meaningful conversations with other novice teachers in a setting free
of evaluation.
16) I have common planning times with other teachers at my same grade level or subject area.
17) I have opportunities to visit and observe exemplary teachers.
18) I have a colleague in my same subject area or grade level who will answer my questions.
Domain Three: Administration Support
Directions: Please respond to the survey questions below concerning the support you receive
from administrators in your school.
19) The administration at my school provides appropriate feedback for my discipline decisions.
20) The administration at my school encourages me to be an effective teacher.
21) The administration has oriented me to the school and staff.
22) I have on-going face-to-face communication with my administration.
23) The administration provides me with effective instructional leadership.
Domain Four: Classroom Management
Directions: Please respond to the survey questions below concerning your classroom
management behaviors and success.
24) I have developed clear routines and procedures for my classroom that are aligned with school
policy.
25) I have implemented consistent routines and procedures in my classroom.
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 35
26) My routines and procedures positively impact the behavior of my students.
27) The discipline in my classroom is supportive of a good learning environment for my
students.
28) I feel in control when I am teaching.
29) My students’ behaviors are consistent with my classroom expectations.
30) I am able to use communication to diffuse disruptive student behavior.
Domain Five: Encouraging Student Success
Directions: Please respond to the survey questions below concerning your instructional
practices.
31) I am able to successfully teach students with a variety of ability levels.
32) I am able to motivate all students.
33) I am able to use a variety of teaching strategies to provide my students with instruction that is
effective for them.
34) I am able to effectively teach my students from diverse backgrounds.
35) I am able to frame my instructional decisions based on my students’ learning.
Domain Six: Curricular and Instructional Resources
Directions: Please respond to the survey questions below concerning the curricular and
instructional resources provided by your SCHOOL or SCHOOL DISTRICT.
36) I have adequate instructional supplies, such as copy paper, a functioning copier, and
pens/pencils that I need for teaching.
37) I have been provided with curriculum that aligns with the state’s objectives for my grade
level or subject area.
38) I have the curriculum materials I need to teach effectively.
39) I have been provided with the instructional technology I need to teach effectively.
40) My students have the curricular resources they need to learn effectively.
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 36
41) How many school or school district provided professional development sessions did you
attend?
1) Zero, 2) 1-2, 3) 3-5, 4) 6-10, or 5) more than 10
42) The professional development provided by my school or school district improved my
classroom management skills.
43) The professional development provided by my school or school district improved my ability
to create rigorous, standards-aligned unit and lesson plans.
44) The professional development provided by my school or school district improved my ability
to create rigorous, standards-driven assessments.
45) The professional development provided by my school or school district improved my ability
to analyze student assessment data and adjust instruction.
46) The professional development provided by my school or school district improved my ability
to reflect on my teaching practice.
47) The professional development provided by my school or school district provided me with
instructional strategies that improved my students’ learning.
48) Overall, the professional development provided by my school or school district was helpful
in developing my confidence, knowledge, and skills in teaching.
Domain Seven: Assignment and Workload
Directions: Please respond to the survey questions below concerning your current teaching
assignments and workload.
49) I think the number of preparations I have for my classes is appropriate for a novice teacher.
50) I have at least one period per day that I can devote without interruption to planning for my
classes.
51) My overall teaching workload is reasonable.
52) Novice teachers are allowed to choose whether to take on extra duties or not.
53) If this is your first year teaching, did you have a reduced teaching schedule or number of
preparations?
1) Yes, 2) No, or 3) Not my first year teaching
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 37
Domain Eight: Assessment and Evaluation
Directions: Please respond to the survey questions below concerning how you assess and
evaluate student learning.
54) I informally assess each student on a daily basis.
55) I formally assess each student on a weekly basis.
56) I use North Carolina’s academic standards to create classroom assessments.
57) I am able to write quality unit/chapter tests for my students.
58) I am able to effectively create classroom assessments other than classroom tests.
59) I use a variety of measures to assess my students.
60) I use formative assessment in my classroom.
61) I use summative assessment in my classroom.
62) I have time to interpret my students’ assessment data.
Domain Nine: Parental Contacts
Directions: Please respond to the survey questions below concerning your contact with parents
and/or caregivers.
63) The parents or caregivers of my students are supportive of their child’s progress in school.
64) I am able to effectively communicate with my students’ parents or caregivers.
65) I have adequate guidance and support in working with parents or caregivers.
Domain Ten: Satisfaction and Commitment
Directions: Please respond to the survey questions below concerning your job satisfaction and
commitment to teaching.
66) In general, I am satisfied with my current job.
67) I consider teaching to be my ideal career.
68) I feel inspired to instruct students to the best of my ability.
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 38
69) If someone could change any of the following items, which ones would be the most
important to improve your satisfaction with your job? Choose the THREE most important
items only.
70) I am most interested in participating in and/or learning more about?
71) Think about your intentions regarding teaching. Which response best fits your current
intentions?
Domain Eleven: Demographics
Please respond to the survey questions below concerning basic demographic information.
72) What level of teaching experience do you currently possess?
1) I am a first year teacher, 2) I am a second year teacher, 3) I am a third year teacher,
or 4) I have taught for more than 3 years
73) What is the grade-level or subject-area your primarily teach?
1) PK, 2) K, 3) 1st grade, 4) 2
nd grade, 5) 3
rd grade, 6) 4
th grade, 7) 5
th grade, 8) Middle
school math, 9) Middle school reading, 10) Middle school science, 11) Middle school
social studies, 12) High school math, 13) High school English, 14) High school
science, 15) High school social studies, 16) Exceptional children, 17) PE/Health, 18)
Art/music, or 19) Other
74) In what month of the 2012-13 school year did you begin teaching?
1) July, 2) August, 3) September . . . 12) June
75) I received my preparation to teach through:
1) Traditional four-year university program 2) Master of Arts in Teaching (MAT) 3)
Licensure or certificate program 4) Lateral or alternative entry program 5) Other
76) I received my teacher training at:
1) North Carolina public university 2) North Carolina private university 3) Out-of-state
university 4) On-line university/program 5) Teach For America 6) Other
NC NTSP-Specific Items Added to the PSI-BT
Unless otherwise noted, these items have a 6 point scale: strongly disagree, disagree, slightly
disagree, slightly agree, agree, and strongly agree.
NC NTSP Institutes
1) After the North Carolina New Teacher Support Program Institute(s) I was better prepared to
set ambitious academic goals for my students.
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 39
2) After the North Carolina New Teacher Support Program Institute(s) I was better understood
the Common Core Standards and North Carolina Essential Standards.
3) After the North Carolina New Teacher Support Program Institute(s) I was better prepared to
plan units and lessons to reach academic goals.
4) After the North Carolina New Teacher Support Program Institute(s) I was better prepared to
track student progress and adjust instruction.
5) After the North Carolina New Teacher Support Program Institute(s) I was better prepared to
implement instructional strategies that improve student learning.
6) After the North Carolina New Teacher Support Program Institute(s) I was better prepared to
create a respectful classroom environment that encourages student achievement.
7) After the North Carolina New Teacher Support Program Institute(s) I was better prepared for
the school year.
8) Overall, the North Carolina New Teacher Support Program Institute(s) was helpful in
developing my confidence, knowledge, and skills in teaching.
NC NTSP Instructional Coaching
9) Has the North Carolina New Teacher Support Program assigned you an instructional coach?
-If yes, continue answering instructional coaching questions
-If no, skip to NC NTSP professional development questions
10) In what month did you first meet with the instructional coach provided by the North Carolina
New Teacher Support Program?
-Please provide a drop-down box of month choices, from July through June
11) How much time do you spend meeting with the instructional coach provided by the North
Carolina New Teacher Support Program each month?
-Please provide a drop-down box of the following time choices: 1) 0-30 minutes 2) 30
minutes to
1 hour 3) 1-2 hours 4) 2-3 hours 5) 3-4 hours 6) 4-5 hours 7) more than 5 hours
12) The virtual coaching provided by the North Carolina New Teacher Support Program
effectively addressed my questions and concerns.
-Responses: Strongly disagree, disagree, slightly disagree, slightly agree, agree, strongly
agree, AND “Did not have virtual coaching”
13) My instructional coach assigned by the North Carolina New Teacher Support Program has
provided assistance with classroom management.
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 40
14) My instructional coach assigned by the North Carolina New Teacher Support Program has
provided assistance with instructional concerns.
15) My instructional coach assigned by the North Carolina New Teacher Support Program has
provided assistance related to communication with parents or caregivers of my students.
16) My instructional coach assigned by the North Carolina New Teacher Support Program has
provided assistance with unit or lesson planning.
17) My instructional coach assigned by the North Carolina New Teacher Support Program has
provided assistance with the Common Core State Standards and/or North Carolina Essential
Standards.
18) My instructional coach assigned by the North Carolina New Teacher Support Program has
provided assistance with analysis of student assessment data.
19) My instructional coach assigned by the North Carolina New Teacher Support Program is
empathetic.
20) My instructional coach assigned by the North Carolina New Teacher Support Program
encourages me to reflect about my teaching.
21) Of the success you have had as a beginning teacher, what amount would you attribute to help
from your instructional coach assigned by the North Carolina New Teacher Support
Program?
-Please provide a drop-down box with the following choices: 1) None 2) Hardly any 3)
Some 4) Quite a bit 5) A great deal
22) Overall, my instructional coach provided by the North Carolina New Teacher Support
Program has been helpful in developing my confidence, knowledge, and skills in teaching.
NC NTSP Professional Development
23) In your opinion, what factor would MOST improve attendance at North Carolina New
Teacher Support Program professional development sessions?
-Responses: 1) Providing a financial stipend 2) Holding professional development
sessions during the school day; 3) Holding professional development sessions after
school 4) Improving professional development curriculum; 5) Encouragement from
schools/principals to attend; 6) Making the NTSP professional development your only
required professional development (no longer have to attend school/district PD sessions)
24) Did you attend any North Carolina New Teacher Support Program professional development
sessions?
-If yes, proceed to NC NTSP professional development questions
-If no, skip to PSI-BT questions
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 41
25) The professional development provided by the North Carolina New Teacher Support
Program improved my classroom management skills.
26) The professional development provided by the North Carolina New Teacher Support
Program improved my ability to create rigorous, standards-aligned unit and lesson plans.
27) The professional development provided by the North Carolina New Teacher Support
Program improved my ability to create rigorous, standards-driven assessments.
28) The professional development provided by the North Carolina New Teacher Support
Program improved my ability to analyze student assessment data and adjust instruction.
29) The professional development provided by the North Carolina New Teacher Support
Program improved my ability to reflect on my teaching practice.
30) The professional development provided by the North Carolina New Teacher Support
Program provided me with instructional strategies that improved my students’ learning.
31) Overall, the professional development provided by the North Carolina New Teacher Support
Program was helpful in developing my confidence, knowledge, and skills in teaching.
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 42
Appendix C. To What Extent was the NC NTSP Implemented as it was Intended?
As stated in North Carolina’s RttT grant proposal, the directive of the NC NTSP is to provide
comprehensive induction services to novice teachers employed in the state’s lowest-performing
schools—schools that in the year before RttT began were either in the lowest 5% of all schools
in terms of student achievement or had graduation rates below 60%. The program started with a
limited implementation in the 2011-12 school year, serving 35 beginning teachers in 13 schools
and five LEAs. Given this small sample of teachers and the delays in program implementation15
the final evaluation report focuses on the 2012-13 and 2013-14 school years.
As shown in the left panel of Table C1 (following page), in the 2012-13 academic year the NC
NTSP scaled up to provide induction supports to 542 novice teachers in 77 schools and 19 LEAs.
Over 70% of these program participants were first-year teachers, 25% were second-year
teachers, and only a small percentage were in their third year of teaching. Ninety-four percent
(510 out of 542) of these teachers worked in RttT schools; the remaining teachers worked in non-
RttT schools who had sought out NC NTSP induction services for their beginning teachers.
Seventy-seven percent (417 out of 542) of these teachers entered the NC NTSP before January
2013. Those who entered in January 2013 or after were either late hires and/or beginning
teachers in schools that agreed to enter the NC NTSP in the spring of 2013. Finally, 7% (38 out
of 542) of these teachers were Teach for America (TFA) corps members receiving additional
induction services—mentoring and professional development—from TFA.
The right panel of Table C1 (following page) indicates that in the 2013-14 academic year the NC
NTSP scaled up again to provide induction supports to 1,108 novice teachers in 114 schools and
27 LEAs. Approximately 50% of these program participants were first-year teachers, 33% were
second-year teachers, and 15% were in their third year of teaching. Eighty-one percent (903 out
of 1,108) of these teachers worked in RttT schools; the remaining teachers worked in non-RttT
schools who had sought out NC NTSP induction services for their beginning teachers. Over 93%
(1,035 out of 1,108) of these teachers entered the NC NTSP before January 2014. Those who
entered in January 2014 or after were predominantly late hires. Finally, 4% (47 out of 1,108) of
these teachers were TFA corps members receiving additional induction services from TFA.
Overall, the NC NTSP provided comprehensive induction supports to novice teachers in 73 RttT
schools in the 2012-13 school year and 91 RttT schools in the 2013-14 school year. By the end
of the RttT funding period, this represents over three-quarters of the lowest achieving schools to
which the program was tasked to serve.
Subsequent appendices display results for the NC NTSP evaluation sample only, for the
complete sample of NC NTSP teachers (as shown in Table C1, following page), and for an
amended NC NTSP evaluation sample for all non-TFA corps members who entered the program
by December of the academic year.
15
The NC NTSP provided a summer institute in 2011 but did not provide instructional coaching and professional
development until the spring of 2012.
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 43
Table C1. To what extent was the NC NTSP implemented as it was intended?
Characteristics 2012-13 School Year 2013-14 School Year
Teacher Count 542 1108
Teachers by Region
ECU 112 237
UNC-CSLD 157 348
UNCC 139 268
UNCG 134 255
1st Year Teacher % 71.99 50.63
2nd
Year Teacher % 25.42 33.84
3rd
Year Teacher % 2.60 15.52
Teachers in RttT Schools 510 903
Teachers Served by NC NTSP
Before January 417 1035
TFA Corps Members 38 47
School Count 77 114
Schools by Region
ECU 19 30
UNC-CSLD 22 37
UNCC 18 24
UNCG 18 23
LEA Count 19 27
LEAs by Region
ECU 6 10
UNC-CSLD 5 5
UNCC 4 7
UNCG 4 5
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 44
Appendix D. To What Extent did the NC NTSP Reach its Target Population?
NC NTSP Institutes
In advance of the 2012-13 school year, the NC NTSP held a week-long summer institute in
Chapel Hill. Given the challenges of securing high attendance rates at a centralized institute in
advance of the school year—late hiring, conflicting LEA/school beginning teacher induction,
and professional development—the UNC-GA program implementers also offered a three-day,
make-up winter institute in December 2012. As shown in the top panel of Table D1 (following
page), 63 teachers in the NC NTSP evaluation sample attended the summer institute and 126
teachers (mutually-exclusive groups) attended the make-up winter institute. Of the 344 teachers
in the 2012-13 NC NTSP evaluation sample, nearly 55% attended an institute. Across regions,
however, there was sizable variation in attendance—more than two-thirds of the evaluation
sample teachers in the ECU and UNC-CSLD regions attended an institute while only 47% and
40% of the evaluation sample teachers from the UNCC and UNCG regions attended.
In an effort to boost institute attendance in the 2013-14 school year, the UNC-GA program
implementers offered two-day regional institutes—in the ECU, UNC-CSLD, UNCC, and UNCG
regions—in advance of the school year (late July and early August) and a three-day, statewide
institute in late September. As shown in the bottom panel of Table D1 (following page), 50
teachers in the NC NTSP evaluation sample attended a regional institute and 143 teachers (non-
mutually exclusive groups) attended the statewide institute. Of the 808 NC NTSP teachers in the
2013-14 evaluation sample, only 21% attended any institute—unlike 2012-13, institute
attendance was fairly comparable across regions.
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 45
Table D1. Evaluation Sample Attendance at NC NTSP Institutes
Institute
Overall
Teacher
Attendance
Teacher Attendance by Region
ECU
Region
UNC-
CSLD
Region
UNCC
Region
UNCG
Region
2012
-13 S
chool
Yea
r Summer Institute
63
(18.31%)
10
(15.15%)
28
(35.00%)
10
(8.33%)
15
(19.23%)
Winter Institute 126
(36.63%)
39
(59.09%)
25
(31.25%)
46
(38.33%)
16
(20.51%)
Any
Institute
189
(54.94%)
49
(74.24%)
53
(66.25%)
56
(46.67%)
31
(39.74%)
No Institute 155
(45.06%)
17
(25.76%)
27
(33.75%)
64
(53.33%)
47
(60.26%)
Total Teachers 344 66 80 120 78
2013-2
014 S
chool
Yea
r
Regional Institute 50
(6.19%)
16
(8.47%)
12
(5.06%)
11
(6.04%)
11
(5.50%)
Statewide
Institute
143
(17.70%)
45
(23.81)
42
(17.72%)
20
(10.99%)
36
(18.00%)
Any
Institute
166
(20.54%)
52
(27.51%)
45
(18.99%)
30
(16.48%)
39
(19.50%)
No Institute 642
(79.46%)
137
(72.49%)
192
(81.01%)
152
(83.52%)
161
(80.50%)
Total Teachers 808 189 237 182 200
NC NTSP Instructional Coaching
To assess the frequency of NC NTSP instructional coaching during the 2012-13 school year, the
UNC-GA program implementers provided the Evaluation Team with counts of in-person
coaching visits. Table D2 (third page following) displays these data by NC NTSP region and
teacher experience (BT1=first-year teacher; BT2=second-year teacher). Overall, Table D2
conveys two main points. First, NC NTSP instructional coaches made frequent visits to the
classrooms of the novice teachers they coached; however, the average number of visits varied
substantially across regions. In the UNCC and UNCG regions, NC NTSP teachers typically
averaged three to four in-person visits from their instructional coach each month—approximately
once a week. In the ECU region, teachers received more frequent instructional coaching,
particularly early in the school year and at the start of the new semester, while in the UNC-CSLD
region, NC NTSP teachers received approximately two visits per month—once every other
week. Second, across regions, first-year teachers typically received more in-person coaching
visits than their second-year peers.
For the 2013-14 school year, the UNC-GA program implementers made two changes to their
instructional coaching and data collection efforts: (1) completing virtual instructional coaching
visits with NC NTSP teachers, and (2) tracking the number of in-person and virtual contact hours
spent with NC NTSP teachers. In the tables below for the 2013-14 school year, Table D3 (third
page following) displays counts of in-person coaching visits with NC NTSP teachers, Table D4
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 46
(third page following) displays counts of virtual instructional coaching visits with NC NTSP
teachers, and Table D5 (third page following) displays total contact hours—in-person and
virtual—between NC NTSP teachers and their instructional coaches. Each of these tables
displays data by region, overall, and by level of teacher experience. 16
In Table D3 (second page following), values for 2013-14 in-person instructional coaching visits
indicate two key results. First, as in 2012-13, there was significant variation across regions in the
average number of instructional coaching visits per teacher. On average, NC NTSP teachers in
the ECU region received approximately three to four in-person visits per month, teachers in the
UNCC region received slightly more than two visits per month, and teachers in the UNC-CSLD
and UNCG regions received less than two visits per month. Second, on average, NC NTSP
teachers received fewer in-person instructional coaching visits in 2013-14 than in 2012-13. For
example, NC NTSP teachers in the ECU region often received four or more (particularly in
August/September, October, and January) coaching visits per month in 2012-13, but in 2013-14
they averaged three to four visits per month. NC NTSP teachers in the UNCG region went from
three to four visits per month in 2012-13 to less than two visits per month in 2013-14.
Turning to virtual instructional coaching, Table D4 (third page following) shows that NC NTSP
regions did not significantly rely upon virtual coaching to address novice teacher needs—at
most, NC NTSP teachers were averaging slightly more than one virtual coaching session per
month. Similar to the in-person coaching numbers, substantial variation existed across regions.
In the ECU region, NC NTSP teachers averaged approximately one virtual coaching session per
month and in the UNCC region, NC NTSP teachers typically averaged between one-half and one
virtual coaching session per month. By contrast, NC NTSP teachers in the UNC-CSLD and
UNCG regions often averaged less than one-third or one-tenth of a virtual instructional coaching
session per month; essentially, the NC NTSP instructional coaching in those two regions was
almost entirely conducted through in-person, classroom visits.
Table D5 (third page following) displays total contact hours, both in-person and virtual, between
NC NTSP teachers and program instructional coaches in the 2013-14 school year. It should be
noted that the UNC-GA program implementers switched to a new data management system early
in the 2013-14 school year and as a result, contact hours for August and September are not
available for (and thus not included in) the contact hours shown in Table D5. Like the substantial
variation in instructional coaching visits shown in Tables D2-D4, Table D5 indicates that, on
average, NC NTSP teachers in the ECU and UNCC regions experienced significantly more
contact hours than their NC NTSP peers in the UNC-CSLD and UNCG regions. NC NTSP
teachers in the ECU region averaged more than 33 in-person hours and nearly eight virtual
contact hours throughout the 2013-14 school year; NC NTSP teachers in the UNCC region
averaged nearly 29 in-person hours and five virtual contact hours throughout the school year.
Over an eight month period from October through May this is approximately five hours of
instructional coaching per month for NC NTSP teachers in the ECU region, and four hours of
instructional coaching per month for NC NTSP teachers in the UNCC region. In contrast, NC
NTSP teachers in the UNC-CSLD and UNCG regions averaged approximately ten in-person
hours and less than one virtual contact hour throughout the school year; over an eight month
16
The UNC-GA program implementers switched to a new data management system early in the 2013-14 school
year and as a result in-person and virtual instructional coaching data is not available for August and September.
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 47
period from October to May, this is less than one and a half hours of instructional coaching per
month. Across regions, these contact hours typically vary by level of teacher experience, with
first-year teachers receiving the most instructional coaching and third-year teachers receiving the
least amount of instructional coaching. For example, by level of experience, NC NTSP teachers
in the UNCC region averaged 32, 28, and 21 in-person instructional coaching hours,
respectively.
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 48
Table D2. Average Number of NC NTSP Instructional Coaching Visits per Teacher (2012-13)
Month
ECU Region Visits UNC-CSLD Region Visits UNCC Region Visits UNCG Region Visits
Overall BT 1 BT 2 Overall BT 1 BT 2 Overall BT 1 BT 2 Overall BT 1 BT 2
August &
September 8.23 8.67 7.25 2.20 2.60 1.43 2.60 2.81 2.23 6.78 7.07 6.00
October 9.39 10.00 8.10 3.31 3.50 2.96 3.05 3.20 2.80 3.97 3.79 5.20
November 4.23 4.00 4.71 2.34 2.46 2.12 3.55 3.54 3.57 4.17 4.15 4.30
December 4.17 4.40 3.67 1.69 1.71 1.65 2.93 3.11 2.58 3.44 3.50 3.00
January 7.21 7.44 6.71 2.21 2.39 1.85 3.43 3.49 3.31 3.87 3.93 3.50
February 4.43 4.73 3.81 1.68 1.47 2.12 3.40 3.49 3.23 3.67 3.65 3.80
March 4.81 5.30 3.75 1.84 1.75 2.04 3.72 3.81 3.54 3.73 3.75 3.60
April 4.97 5.44 3.95 1.44 1.39 1.54 3.58 3.70 3.33 3.37 3.35 3.50
May &
June 6.54 7.30 4.90 2.22 2.14 2.40 4.35 4.32 4.38 4.78 4.96 3.60
Table D3. Average Number of NC NTSP In-Person Instructional Coaching Visits per Teacher (2013-14)
Month
ECU Region Visits UNC-CSLD Region Visits UNCC Region Visits UNCG Region Visits
Overall BT 1 BT 2 BT 3 Overall BT 1 BT 2 BT 3 Overall BT 1 BT 2 BT 3 Overall BT 1 BT 2 BT 3
October 4.61 4.98 4.37 4.13 1.91 1.91 2.05 1.70 2.68 2.95 2.16 2.98 2.87 3.00 2.81 2.15
November 3.70 3.73 4.00 2.97 1.53 1.66 1.55 1.19 2.15 1.95 2.31 2.34 1.90 2.01 1.87 1.23
December 3.13 3.20 3.60 2.00 1.39 1.46 1.40 1.22 1.91 1.74 1.89 2.39 1.34 1.37 1.32 1.23
January 3.23 3.42 3.43 2.28 1.53 1.68 1.46 1.30 2.03 2.05 2.02 2.00 1.81 1.91 1.84 0.77
February 3.88 4.14 4.09 2.69 1.49 1.68 1.46 1.09 2.31 2.59 2.06 2.06 1.93 1.89 2.13 1.00
March 4.62 4.92 4.42 4.19 1.87 1.99 1.90 1.54 3.14 3.45 3.00 2.59 1.47 1.30 1.73 1.23
April 3.26 3.36 3.45 2.57 1.48 1.59 1.63 0.96 2.55 2.71 2.47 2.28 1.46 1.48 1.48 1.23
May &
June 4.43 4.56 4.44 4.06 2.05 2.16 2.15 1.60 3.86 3.99 3.77 3.72 1.44 1.46 1.57 0.46
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 49
Table D4. Average Number of NC NTSP Virtual Instructional Coaching Visits per Teacher (2013-14)
Month
ECU Region Virtual Visits UNC-CSLD Region Virtual Visits UNCC Region Virtual Visits UNCG Region Virtual Visits
Overall BT 1 BT 2 BT 3 Overall BT 1 BT 2 BT 3 Overall BT 1 BT 2 BT 3 Overall BT 1 BT 2 BT 3
October 0.97 1.10 0.63 1.31 0.08 0.09 0.08 0.07 1.11 1.48 0.98 0.32 0.10 0.09 0.14 0.00
November 1.35 1.45 1.18 1.41 0.34 0.37 0.38 0.21 0.65 0.59 0.66 0.81 0.19 0.20 0.17 0.31
December 0.80 0.75 0.94 0.69 0.24 0.20 0.36 0.13 0.53 0.49 0.53 0.61 0.05 0.07 0.04 0.00
January 1.26 1.41 1.09 1.16 0.38 0.38 0.51 0.17 0.54 0.58 0.59 0.34 0.22 0.19 0.26 0.15
February 1.19 1.10 1.23 1.38 0.27 0.29 0.29 0.20 0.67 0.80 0.53 0.63 0.06 0.05 0.09 0.00
March 1.19 0.98 1.37 1.42 0.04 0.07 0.01 0.02 0.51 0.57 0.41 0.53 0.07 0.07 0.07 0.00
April 0.27 0.20 0.39 0.20 0.08 0.09 0.09 0.02 0.47 0.50 0.39 0.56 0.04 0.02 0.07 0.00
May &
June 0.80 0.88 0.72 0.74 0.09 0.15 0.04 0.04 1.20 1.32 1.13 1.03 0.20 0.14 0.30 0.00
Table D5. Average Total Contact Hours with NC NTSP Instructional Coaches (2013-14)
Region and Teacher Experience Average In-Person Hours Average Virtual Hours
ECU Overall 33.49 7.67
ECU 1st Year Teachers 36.34 7.79
ECU 2nd
Year Teachers 32.52 8.32
ECU 3rd
Year Teachers 27.26 6.00
UNC-CSLD Overall 10.38 0.97
UNC-CSLD 1st Year Teachers 11.47 1.25
UNC-CSLD 2nd
Year Teachers 10.92 0.94
UNC-CSLD 3rd
Year Teachers 6.93 0.35
UNCC Overall 28.81 4.80
UNCC 1st Year Teachers 32.28 4.42
UNCC 2nd
Year Teachers 27.92 4.66
UNCC 3rd
Year Teachers 21.27 6.10
UNCG Overall 9.70 0.50
UNCG 1st Year Teachers 9.72 0.37
UNCG 2nd
Year Teachers 10.52 0.70
UNCG 3rd
Year Teachers 4.36 0.23
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 50
NC NTSP Professional Development
For the 2012-13 school year, Table D6 presents the number of NC NTSP evaluation sample
teachers in attendance at each of the NC NTSP professional development sessions (overall and
by region). Table D7 displays the average number of professional development sessions attended
per teacher, and the percentage of teachers attending zero, one, two, three, four, five, and six
professional development sessions. Overall, Tables D6 and D7 indicate that within the NC NTSP
evaluation sample (1) attendance at professional development sessions was low, with zero
sessions attended as the modal value, and (2) significant heterogeneity existed across regions in
attendance, with NC NTSP teachers in the UNC-CSLD region attending an average of 3.59
sessions (out of six)17
and NC NTSP teachers in the UNCC region attending an average of 1.17
sessions (out of six).
Table D6. Evaluation Sample Attendance at NC NTSP by PD Session and Region (2012-13)
Region
PD
Session 1
PD
Session 2
PD
Session 3
PD
Session 4
PD
Session 5
PD
Session 6
ECU 29
(43.94%)
30
(45.45%)
26
(39.39%)
27
(40.91%)
27
(42.86%)
20
(31.75%)
UNC-CSLD 57
(79.17%)
54
(72.97%)
50
(65.79%)
49
(61.25%)
42
(54.55%)
35
(45.45%)
UNCC 10
(8.47%)
27
(22.88%)
34
(28.81%)
27
(23.08%)
19
(16.52%)
23
(20.35%)
UNCG 23
(30.67%)
25
(32.05%)
25
(32.05%)
33
(42.31%)
33
(42.31%)
37
(47.44)
Total 119
(35.95%)
136
(40.48%)
135
(39.94%)
136
(39.88%)
121
(36.34%)
115
(34.74%)
Table D7. Average Number of PD Sessions by Region for Evaluation Sample (2012-13)
NC NTSP
Overall
Region
ECU UNC-CSLD UNCC UNCG
Avg. Number of PD
Sessions Attended 2.22 2.41 3.59 1.17 2.26
Nu
mb
er o
f P
D
Ses
sion
s A
tten
ded
Zero 34.01% 25.76% 12.50% 56.67% 28.21%
One 14.24% 15.15% 8.75% 14.17% 19.23%
Two 11.63% 16.67% 13.75% 6.67% 12.82%
Three 10.76% 12.12% 13.75% 9.17% 8.97%
Four 6.40% 4.55% 6.25% 6.67% 7.69%
Five 11.63% 16.67% 13.75% 5.00% 15.38%
Six 11.34% 9.09% 31.25% 1.67% 7.69%
Teacher Count 344 66 80 120 78
17
The higher PD attendance values for the UNC-CSLD reflect the fact that several LEAs and schools in the region
required their NC NTSP teachers to attend the NC NTSP PD sessions.
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 51
For the 2013-14 school-year, Table D8 presents the number of NC NTSP evaluation sample
teachers in attendance at each of the NC NTSP professional development sessions (by region and
overall). Table D9 displays the average number of professional development sessions attended
per teacher, and the percentage of teachers attending zero, one, two, three, four, five, and six
professional development sessions. Overall, in comparison to the 2012-13 school year,
attendance was much lower at the 2013-14 NC NTSP professional development sessions. During
the 2012-2013 school year, NC NTSP evaluation sample teachers attended (on average) 2.22
professional development sessions, while the average number of sessions attended per teacher in
2013-14 was 0.76. Likewise, in 2012-13, 34% of NC NTSP evaluation sample teachers did not
attend any program professional development sessions; in 2013-14, 71% of teachers did not
attend any program professional development sessions. As in 2012-13, there were variations in
attendance across regions, with NC NTSP teachers in the ECU and UNC-CSLD regions
attending an average of 1.03 and 1.22 sessions (out of six) and NC NTSP teachers in the UNCC
and UNCG regions attending an average of 0.14 and 0.52 sessions (out of six).
Table D8. Evaluation Sample Attendance at NC NTSP by PD Session and Region (2013-14)
Region
PD
Session 1
PD
Session 2
PD
Session 3
PD
Session 4
PD
Session 5
PD
Session 6
ECU 32
(17.78%)
29
(15.59%)
42
(22.34%)
37
(20.00%)
41
(22.16%)
13
(7.07%)
UNC-CSLD 51
(22.77%)
59
(25.54%)
54
(22.98%)
39
(16.67%)
49
(20.94%)
38
(16.31%)
UNCC 9
(4.97%)
3
(1.65%)
5
(2.75%)
6
(3.30%)
2
(1.10%)
0
(0.00%)
UNCG 24
(12.50%)
19
(9.55%)
18
(9.05%)
16
(8.04%)
6
(3.03%)
21
(10.61%)
Total 116
(14.93%)
110
(13.78%)
119
(14.80%)
98
(12.25%)
98
(12.27%)
72
(9.03%)
Table D9. Average Number of PD Sessions by Region for Evaluation Sample (2013-14)
NC NTSP
Overall
Region
ECU UNC-CSLD UNCC UNCG
Avg. Number of PD
Sessions Attended 0.76 1.03 1.22 0.14 0.52
Nu
mb
er o
f P
D
Ses
sion
s A
tten
ded
Zero 70.92% 55.56% 65.40% 87.91% 76.50%
One 10.52% 16.93% 3.80% 10.99% 12.00%
Two 5.57% 11.11% 7.17% 0.55% 3.00%
Three 4.95% 6.35% 7.17% 0.55% 5.00%
Four 3.22% 7.41% 4.64% 0.00% 0.50%
Five 2.48% 1.06% 6.75% 0.00% 1.00%
Six 2.35% 1.59% 5.06% 0.00% 2.00%
Teacher Count 808 189 237 182 200
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 52
Appendix E. How do Teachers Perceive the Impact of the NC NTSP Components on their
Confidence, Knowledge, and Skills in Teaching?
NC NTSP Institutes
To what extent do NC NTSP teachers feel the Institutes positively impacted their teaching?
To measure attendees’ perceptions of the utility of the NC NTSP institutes, the Evaluation Team
developed a set of survey questions, aligned with the institute curriculum, asking attendees to
indicate the extent to which they felt better prepared (after the NC NTSP institute) to carry out
key teaching practices. The Evaluation Team included these items on the PSI-BT survey
administered in 2013 and 2014 (see Appendix B).
Table E1 presents the summative institute survey item, asking institute attendees the extent to
which the NC NTSP institute(s) was helpful in developing their confidence, knowledge, and
skills in teaching. Overall, the top panel of Table E1 shows that of those NC NTSP evaluation
sample teachers who attended a 2012 institute and responded to the PSI-BT, 80% agreed or
strongly agreed with the statement. This value varied from a high of 94% in the UNCC region to
a low of 59% in the UNC-CSLD region. The bottom panel of Table E1 shows that 82% of the
NC NTSP evaluation sample teachers who attended a 2013 institute and responded to the PSI-BT
agreed or strongly agreed that the institute was helpful in developing their confidence,
knowledge, and skills in teaching; this value varied from a high of 100% in the UNCG region to
a low of 65% in the UNC-CSLD region.
Table E1. NC NTSP Institute Summative Survey Item
NTSP
Instructional
Coaching Item
Response
Groups n
Strongly
Disagree Disagree
Slightly
Disagree
Slightly
Agree Agree
Strongly
Agree
2012-13 NC NTSP Institutes
Overall, the NC
NTSP Institute(s)
was helpful in
developing my
confidence,
knowledge, and
skills in teaching.
Overall 110 2.73% 0.00% 2.73% 14.55% 42.73% 37.27%
ECU 38 0.00% 0.00% 2.63% 15.79% 44.74% 36.84%
UNC-CSLD 22 9.09% 0.00% 9.09% 22.73% 36.36% 22.73%
UNCC 33 0.00% 0.00% 0.00% 6.06% 45.45% 48.48%
UNCG 17 5.88% 0.00% 0.00% 17.65% 41.18% 35.29%
2013-14 NC NTSP Institutes
Overall, the NC
NTSP Institute(s)
was helpful in
developing my
confidence,
knowledge, and
skills in teaching.
Overall 97 1.03% 3.09% 1.03% 12.37% 43.30% 39.18%
ECU 43 2.33% 4.65% 0.00% 9.30% 34.88% 48.84%
UNC-CSLD 17 0.00% 5.88% 5.88% 23.53% 35.29% 29.41%
UNCC 16 0.00% 0.00% 0.00% 25.00% 43.75% 31.25%
UNCG 21 0.00% 0.00% 0.00% 0.00% 66.67% 33.33%
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 53
NC NTSP Instructional Coaching
To what extent do NC NTSP teachers feel instructional coaching positively impacted their
teaching?
To summatively assess the perceptions of NC NTSP evaluation sample teachers regarding the
quality of NC NTSP instructional coaching, the Evaluation Team included the following items
on the 2013 and 2014 PSI-BT surveys:
1. Overall, my (A) NC NTSP instructional coach and/or (B) school assigned mentor has been
helpful in developing my confidence, knowledge, and skills in teaching.
2. Of the success you have had as a beginning teacher, what amount would you attribute to help
from your (A) NC NTSP instructional coach and/or (B) school assigned mentor.
NC NTSP teachers answered both A and B items; Non-RttT Comparison sample teachers
answered B items only. Given the results shown in Appendix B, the Evaluation Team notes that
extrapolating survey results to the full sample of NC NTSP and comparison sample teachers
risks ignoring differences associated with non-response and should be done with caution.
Overall, the top panel of Table E2 (following page) shows that during the 2012-13 school year,
78% of NC NTSP evaluation sample respondents agreed or strongly agreed that their program
instructional coach was helpful in developing their confidence, knowledge, and skills in
teaching. By comparison, 60% of NC NTSP evaluation sample teachers and 53% of Non-RttT
Comparison sample teachers responded similarly regarding their school provided mentor. For the
2013-14 school year, the bottom panel of Table E2 indicates that 77% of NC NTSP evaluation
sample respondents agreed or strongly agreed with the summative instructional coaching item.
By comparison, 60% of NC NTSP evaluation sample teachers and 40% of Non-RttT Comparison
sample teachers responded similarly regarding their school provided mentor. Each of these
differences—within the NC NTSP sample and between NC NTSP and Non-RttT Comparison
sample teachers—were statistically significant.
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 54
Table E2. Summative Instructional Coaching/Mentoring Survey Items
Survey Item
Response
Groups n
Strongly
Disagree Disagree
Slightly
Disagree
Slightly
Agree Agree
Strongly
Agree
2012-2013 School Year
Overall, my IC
provided by the
NTSP has been
helpful in
developing my
confidence,
knowledge, and
skills in teaching.
Overall 165 5.45% 3.03% 1.21% 12.73% 33.94% 43.64%
ECU 47 4.26% 4.26% 2.13% 19.15% 38.30% 31.91%
UNC-CSLD 33 9.09% 6.06% 0.00% 12.12% 39.39% 33.33%
UNCC 48 6.25% 2.08% 2.08% 14.58% 25.00% 50.00%
UNCG 37 2.70% 0.00% 0.00% 2.70% 35.14% 59.46%
Overall, my school
or school district
assigned mentor has
been helpful in
developing my
confidence,
knowledge, and
skills in teaching.
Overall 159 7.55% 6.29% 5.66% 20.13% 32.08% 28.30%
ECU 42 11.90% 4.76% 2.38% 7.14% 38.10% 35.71%
UNC-CSLD 33 9.09% 3.03% 15.15% 27.27% 30.30% 15.15%
UNCC 47 8.51% 8.51% 2.13% 23.40% 29.79% 27.66%
UNCG 37 0.00% 8.11% 5.41% 24.32% 29.73% 32.43%
Comparison
Group 297 10.77% 7.07% 5.05% 24.24% 28.28% 24.58%
2013-14 School Year
Overall, my IC
provided by the
NTSP has been
helpful in
developing my
confidence,
knowledge, and
skills in teaching.
Overall 401 1.75% 3.24% 2.74% 15.21% 30.92% 46.13%
ECU 144 0.00% 0.69% 4.17% 13.89% 25.69% 55.56%
UNC-CSLD 74 2.70% 6.76% 2.70% 21.62% 33.78% 32.43%
UNCC 87 4.60% 3.45% 2.30% 11.49% 35.63% 42.53%
UNCG 96 1.04% 4.17% 1.04% 15.62% 32.29% 45.83%
Overall, my school
or school district
assigned mentor has
been helpful in
developing my
confidence,
knowledge, and
skills in teaching.
Overall 386 4.15% 8.29% 5.44% 22.02% 29.79% 30.31%
ECU 134 1.49% 10.45% 5.97% 28.36% 24.63% 29.10%
UNC-CSLD 73 5.48% 4.11% 5.48% 13.70% 39.73% 31.51%
UNCC 85 5.88% 10.59% 5.88% 25.88% 23.53% 28.24%
UNCG 94 5.32% 6.38% 4.26% 15.96% 35.11% 32.98%
Comparison
Group 164 6.71% 13.41% 8.54% 31.10% 24.39% 15.85%
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 55
Table E3. Summative Instructional Coaching/Mentoring Survey Items
Survey Item
Response
Groups n
None at
All
Hardly
Any Some
Quite a
Bit
Great
Deal
2012-13 School Year
Of the success you
have had as a
beginning teacher,
what amount would
you attribute to help
from your IC
assigned by the
NTSP.
Overall 165 0.61% 6.67% 30.30% 34.55% 27.88%
ECU 47 0.00% 10.64% 36.17% 29.79% 23.40%
UNC-CSLD 33 0.00% 6.06% 48.48% 33.33% 12.12%
UNCC 48 2.08% 8.33% 22.92% 43.75% 22.92%
UNCG 37 0.00% 0.00% 16.22% 29.73% 54.05%
Of the success you
have had as a
beginning teacher,
what amount would
you attribute to help
from your school or
school district
assigned mentor.
Overall 159 8.81% 22.01% 23.27% 26.42% 19.50%
ECU 42 11.90% 11.90% 21.43% 28.57% 26.19%
UNC-CSLD 33 18.18% 30.30% 24.24% 21.21% 6.06%
UNCC 47 4.26% 25.53% 21.28% 25.53% 23.40%
UNCG 37 2.70% 21.62% 27.03% 29.73% 18.92%
Comparison
Group 298 10.74% 19.13% 27.52% 28.19% 14.43%
2013-14 School Year
Of the success you
have had as a
beginning teacher,
what amount would
you attribute to help
from your IC
assigned by the
NTSP.
Overall 401 3.99% 8.48% 29.43% 30.67% 27.43%
ECU 144 1.39% 6.94% 25.69% 29.17% 36.81%
UNC-CSLD 74 8.11% 14.86% 24.32% 35.14% 17.57%
UNCC 87 5.75% 8.05% 33.33% 29.89% 22.99%
UNCG 96 3.12% 6.25% 35.42% 30.21% 25.00%
Of the success you
have had as a
beginning teacher,
what amount would
you attribute to help
from your school or
school district
assigned mentor.
Overall 387 5.68% 13.44% 31.78% 25.32% 23.77%
ECU 134 2.99% 16.42% 34.33% 24.63% 21.64%
UNC-CSLD 72 8.33% 6.94% 31.94% 27.78% 25.00%
UNCC 87 8.05% 17.24% 29.89% 20.69% 24.14%
UNCG 94 5.32% 10.64% 29.79% 28.72% 25.53%
Comparison
Group 164 12.20% 23.17% 35.37% 20.12% 9.15%
The top panel of Table E3 indicates that in the 2012-13 school year, 62% of NC NTSP
evaluation sample respondents attributed “quite a bit” or a “great deal” of their teaching success
to help from their NC NTSP instructional coaches. By comparison, 46% of NC NTSP evaluation
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 56
sample teachers and 43% of Non-RttT Comparison sample teachers responded similarly
regarding their school provided mentor. For the 2013-14 school year, the bottom panel of Table
E3 (previous page) shows that 58% of NC NTSP evaluation sample respondents attributed “quite
a bit” or a “great deal” of their teaching success to help from their NC NTSP instructional
coaches. By comparison, 49% of NC NTSP evaluation sample teachers and 29% of Non-RttT
Comparison sample teachers responded similarly regarding their school provided mentor. Each
of these differences—within the NC NTSP sample and between NC NTSP and Non-RttT
Comparison sample teachers—were statistically significant.
NC NTSP Professional Development
To what extent do NC NTSP teachers feel the professional development positively impacted their
teaching?
To summatively assess the perceptions of NC NTSP evaluation sample teachers regarding the
quality of NC NTSP professional development sessions, the Evaluation Team included the
following item on the 2013 and 2014 PSI-BT surveys: Overall, the professional development
provided by (A) the NC NTSP and/or (B) my school or school district has been helpful in
developing my confidence, knowledge, and skills in teaching. NC NTSP teachers answered both
A and B items; Non-RttT Comparison sample teachers answered the B item only. Unfortunately,
a problem with the 2013-14 administration of the PSI-BT survey—responses for “agree” and
“disagree” took on the same response value—prevents the Evaluation Team from presenting the
2013-14 results. Given the results shown in Appendix B, the Evaluation Team notes that
extrapolating survey results to the full sample of NC NTSP and comparison sample teachers
risks ignoring differences associated with non-response and should be done with caution.
Overall, the top panel of Table E4 (following page) shows that in the 2012-13 school year 87%
of NC NTSP evaluation sample teachers who attended NC NTSP professional development
sessions and responded to the PSI-BT agreed or strongly agreed with the survey item. By
comparison, 66% of NC NTSP evaluation sample respondents answered similarly for their
school provided professional development; 60% of Non-RttT Comparison sample respondents
answered similarly for their school provided professional development. Each of these
differences—within the NC NTSP sample and between NC NTSP and Non-RttT Comparison
sample teachers—were statistically significant.
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 57
Table E4. Summative Professional Development Survey Item
Survey Item
Response
Groups n
Strongly
Disagree Disagree
Slightly
Disagree
Slightly
Agree Agree
Strongly
Agree
Overall, the PD
provided by the
NC NTSP was
helpful in
developing my
confidence,
knowledge, and
skills in
teaching.
Overall 132 0.76% 0.00% 1.52% 10.61% 45.45% 41.67%
ECU 39 0.00% 0.00% 0.00% 12.82% 53.85% 33.33%
UNC-CSLD 26 3.85% 0.00% 0.00% 15.38% 50.00% 30.77%
UNCC 32 0.00% 0.00% 3.12% 9.38% 50.00% 37.50%
UNCG 35 0.00% 0.00% 2.86% 5.71% 28.57% 62.86%
Overall, the PD
provided by my
school or
school district
was helpful in
developing my
confidence,
knowledge, and
skills in
teaching.
Overall 164 4.27% 0.00% 6.71% 23.17% 50.61% 15.24%
ECU 47 0.00% 0.00% 8.51% 19.15% 59.57% 12.77%
UNC-CSLD 32 15.62% 0.00% 3.12% 25.00% 34.38% 21.88%
UNCC 48 2.08% 0.00% 8.33% 31.25% 47.92% 10.42%
UNCG 37 2.70% 0.00% 5.41% 16.22% 56.76% 18.92%
Comparison
Group 305 5.25% 0.00% 7.54% 27.21% 46.23% 13.77%
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 58
Appendix F. How do Teachers Perceive the Impact of the NC NTSP on their Self-Efficacy
and Job Satisfaction?
To measure perceptions of self-efficacy and job satisfaction for NC NTSP and Non-RttT
Comparison sample teachers, the Evaluation Team relied on a set of pre-existing items on the
PSI-BT survey. Table F1 (following page) displays these results for the 2012-13 school year and
Table F2 (second page following) presents these results for the 2013-14 school year. Like the
survey responses presented earlier, due to potential non-response bias these results should be
interpreted with caution.
In terms of self-efficacy for the 2012-13 school year, 79% of NC NTSP evaluation sample
respondents (versus 73% of Non-RttT Comparison evaluation sample respondents) agreed or
strongly agreed with the statement that they can successfully instruct students with a variety of
ability levels; 60% of NC NTSP evaluation sample respondents (versus 54% of Non-RttT
Comparison evaluation sample respondents) agreed or strongly agreed with the statement that
they can motivate all students; and 84% of NC NTSP evaluation sample respondents (versus
78% of Non-RttT Comparison evaluation sample respondents) agreed or strongly agreed with the
statement that they feel inspired to instruct students to the best of their ability. For the third self-
efficacy item (“I feel inspired to instruct students to the best of my ability”), the difference
between NC NTSP and Non-RttT Comparison sample teachers was statistically significant.
Regarding job satisfaction, 59% of NC NTSP evaluation sample respondents (versus 55% of
Non-RttT Comparison evaluation sample respondents) agreed or strongly agreed with the
statement that they were satisfied with their current job, and 71% of NC NTSP evaluation sample
respondents (versus 67% of Non-RttT Comparison evaluation sample respondents) agreed or
strongly agreed with the statement that they considered teaching to be their ideal career.
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 59
Table F1. Self-Efficacy and Job Satisfaction Questions (2012-13)
Efficacy and
Job
Satisfaction
Items
Response
Groups n
Strongly
Disagree Disagree
Slightly
Disagree
Slightly
Agree Agree
Strongly
Agree
I am able to
successfully
teach students
with a variety
of ability
levels.
Overall 166 0.00% 1.81% 2.41% 16.87% 62.05% 16.87%
ECU 47 0.00% 2.13% 2.13% 17.02% 63.83% 14.89%
UNC-CSLD 33 0.00% 0.00% 0.00% 27.27% 60.61% 12.12%
UNCC 49 0.00% 2.04% 4.08% 12.24% 65.31% 16.33%
UNCG 37 0.00% 2.70% 2.70% 13.51% 56.76% 24.32%
Comp. Group 307 0.33% 1.63% 2.28% 23.13% 55.05% 17.59%
I am able to
motivate all
students.
Overall 166 0.00% 6.02% 7.83% 25.90% 47.59% 12.65%
ECU 47 0.00% 4.26% 0.00% 29.79% 55.32% 10.64%
UNC-CSLD 33 0.00% 6.06% 15.15% 18.18% 45.45% 15.15%
UNCC 49 0.00% 8.16% 10.20% 28.57% 44.90% 8.16%
UNCG 37 0.00% 5.41% 8.11% 24.32% 43.24% 18.92%
Comp. Group 307 1.95% 4.89% 11.73% 27.36% 41.37% 12.70%
I feel inspired
to instruct
students to the
best of my
ability.
Overall 164 3.05% 1.83% 4.88% 6.10% 36.59% 47.56%
ECU 46 2.17% 4.35% 6.52% 8.70% 36.96% 41.30%
UNC-CSLD 32 3.12% 0.00% 6.25% 3.12% 40.62% 46.88%
UNCC 49 4.08% 2.04% 2.04% 2.04% 36.73% 53.06%
UNCG 37 2.70% 0.00% 5.41% 10.81% 32.43% 48.65%
Comp. Group 308 0.97% 1.62% 6.17% 13.31% 34.74% 43.18%
In general, I
am satisfied
with my
current job.
Overall 164 7.93% 5.49% 7.93% 19.51% 34.15% 25.00%
ECU 46 6.52% 6.52% 2.17% 19.57% 36.96% 28.26%
UNC-CSLD 32 9.38% 9.38% 15.62% 25.00% 21.88% 18.75%
UNCC 49 8.16% 6.12% 0.00% 22.45% 34.69% 28.57%
UNCG 37 8.11% 0.00% 18.92% 10.81% 40.54% 21.62%
Comp. Group 308 6.82% 7.79% 10.71% 20.13% 36.69% 17.86%
I consider
teaching to be
my ideal
career.
Overall 164 4.27% 3.66% 4.27% 17.07% 40.24% 30.49%
ECU 46 2.17% 4.35% 4.35% 26.09% 32.61% 30.43%
UNC-CSLD 32 6.25% 0.00% 6.25% 9.38% 43.75% 34.38%
UNCC 49 6.12% 4.08% 2.04% 14.29% 42.86% 30.61%
UNCG 37 2.70% 5.41% 5.41% 16.22% 43.24% 27.03%
Comp. Group 308 3.57% 5.84% 5.84% 17.53% 32.14% 35.06%
Turning to the 2013-14 school year (Table F2, following page), responses to the same three self-
efficacy survey items indicate that 84%, 69%, and 82% of NC NTSP evaluation sample
respondents (versus 74%, 55%, and 75% of Non-RttT Comparison evaluation sample
respondents) agreed or strongly agreed with the survey items. Regarding job satisfaction,
responses to the same two survey items show that 57% and 68% of NC NTSP evaluation sample
respondents (versus 48% and 61% of Non-RttT Comparison evaluation sample respondents)
agreed or strongly agreed with the survey items. For all five of these self-efficacy and job
satisfaction items, the differences between NC NTSP and Non-RttT Comparison sample teachers
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 60
were statistically significant. These differences must be interpreted carefully, however, due to
non-response bias and the low response rate for the Non-RttT Comparison sample teachers.
Table F2. Self-Efficacy and Job Satisfaction Questions (2013-14)
Survey Items
Response
Groups n
Strongly
Disagree Disagree
Slightly
Disagree
Slightly
Agree Agree
Strongly
Agree
I am able to
successfully
teach students
with a variety
of ability
levels.
Overall 408 0.00% 0.49% 1.72% 13.73% 57.11% 26.96%
ECU 147 0.00% 0.68% 1.36% 14.97% 57.14% 25.85%
UNC-CSLD 77 0.00% 1.30% 1.30% 15.58% 54.55% 27.27%
UNCC 88 0.00% 0.00% 3.41% 12.50% 52.27% 31.82%
UNCG 96 0.00% 0.00% 1.04% 11.46% 63.54% 23.96%
Comp. Group 178 0.56% 1.69% 3.93% 19.66% 52.81% 21.35%
I am able to
motivate all
students.
Overall 408 1.23% 3.68% 6.37% 19.85% 45.59% 23.28%
ECU 147 0.68% 3.40% 6.80% 15.65% 50.34% 23.13%
UNC-CSLD 77 1.30% 6.49% 11.69% 14.29% 40.26% 25.97%
UNCC 88 2.27% 4.55% 6.82% 30.68% 37.50% 18.18%
UNCG 96 1.04% 1.04% 1.04% 20.83% 50.00% 26.04%
Comp. Group 178 2.81% 2.81% 11.80% 28.09% 39.89% 14.61%
I feel inspired
to instruct
students to the
best of my
ability.
Overall 406 1.48% 1.97% 1.97% 13.05% 35.71% 45.81%
ECU 146 2.05% 2.05% 1.37% 11.64% 37.67% 45.21%
UNC-CSLD 78 2.56% 2.56% 1.28% 10.26% 35.90% 47.44%
UNCC 87 1.15% 3.45% 3.45% 16.09% 37.93% 37.93%
UNCG 95 0.00% 0.00% 2.11% 14.74% 30.53% 52.63%
Comp. Group 177 2.26% 4.52% 2.82% 15.25% 31.07% 44.07%
In general, I
am satisfied
with my
current job.
Overall 407 7.13% 8.35% 6.63% 21.13% 35.38% 21.38%
ECU 147 4.76% 6.80% 6.80% 24.49% 34.69% 22.45%
UNC-CSLD 78 11.54% 10.26% 6.41% 19.23% 35.90% 16.67%
UNCC 87 8.05% 12.64% 6.90% 22.99% 32.18% 17.24%
UNCG 95 6.32% 5.26% 6.32% 15.79% 38.95% 27.37%
Comp. Group 178 12.36% 10.11% 11.80% 17.98% 35.39% 12.36%
I consider
teaching to be
my ideal
career.
Overall 404 3.47% 5.69% 4.46% 18.32% 32.67% 35.40%
ECU 146 3.42% 4.79% 4.79% 18.49% 30.82% 37.67%
UNC-CSLD 78 3.85% 5.13% 5.13% 20.51% 23.08% 42.31%
UNCC 85 4.71% 7.06% 1.18% 20.00% 42.35% 24.71%
UNCG 95 2.11% 6.32% 6.32% 14.74% 34.74% 35.79%
Comp. Group 178 4.49% 8.43% 6.74% 19.66% 27.53% 33.15%
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 61
Appendix G. To What Extent does the NC NTSP Impact Teacher Effectiveness as
Measured by Teacher Value-Added to Student Achievement?
EVAAS Results
For analyses of teacher value-added to student achievement, the Evaluation Team presents four
sets of results: (1) overall NC NTSP results for the 2012-13 school year, the 2013-14 school
year, and the two years combined; (2) regional NC NTSP results (separate results for teachers in
the ECU, UNC-CSLD, UNCC, and UNCG regions) for the 2012-13 school year, the 2013-14
school year, and the two years combined; (3) cohort NC NTSP results (separate results for
Cohort 1 teachers—entering the program in the 2012-13 school year—and Cohort 2 teachers—
entering the program in 2013-14) for the 2012-13 school year, the 2013-14 school year, and the
two years combined;18
and (4) first-year teacher NC NTSP results, which focus on first-year
teachers in the 2012-13 school year, the 2013-14 school year, and the two years combined.19
Taken together, this set of analyses presents evidence regarding the overall effectiveness of the
program and how that effectiveness may have differed by NC NTSP region, NC NTSP cohort,
and for first-year teachers. These sub-analyses are particularly important given the differences in
NC NTSP treatment components across regions and cohorts and the hypothesis of stronger
program effects for first-year teachers.
To assess the contributions of NC NTSP teachers to student achievement, the Evaluation Team
analyzed EVAAS teacher effectiveness estimates generated by the SAS Institute. The state uses
these estimates for assessing teacher effectiveness for Standard 6 of the North Carolina Educator
Evaluation System. For these models, the Evaluation Team made teacher EVAAS estimates the
outcome variable and regressed this value-added measure on a set of school characteristics and,
when applicable, teacher experience indicators and year fixed effects. The Evaluation Team
chose to control for these variables due to the significant differences in school characteristics
between NC NTSP and comparison sample schools and because these contextual variables are
not accounted for in EVAAS models. The Evaluation Team clustered standard errors at the
school level to account for dependence in the data. Results from these models express the
adjusted-average differences in student achievement—in normal curve equivalency units—
between students taught by NC NTSP teachers and students taught by Non-RttT Comparison or
NC NTSP Eligible Comparison sample teachers. Like the value-added models in the main body
of the report, the Evaluation Team combined EVAAS data from elementary and middle grades
and ran separate models for mathematics, reading, and science (End-of-Grade (EOG) science
exams in grades five and eight). Additionally, the Evaluation Team combined EVAAS data for
End-of-Course (EOC) exams (Biology, English II, and Math I) and ran a single EOC model.
Tables G1 (third page following) and G2 (fourth page following) display overall and regional
NC NTSP EVAAS results for elementary and middle grades and EOC exams. Overall, the top
panel of Table G1 shows that in elementary and middle grades mathematics, students taught by
18
Cohort 1 NC NTSP teachers (evaluation sample) include first- and second-year teachers. Cohort 2 NC NTSP
teachers (evaluation sample) include first-, second-, and third-year teachers. 19
There are a small number of NC NTSP teachers who have zero years of experience in both the 2012-13 and 2013-
14 school years. When estimating these first-year teacher models, the Evaluation Team removed these teachers from
the NC NTSP first-year teacher sample in 2013-14.
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 62
NC NTSP teachers made significantly larger achievement gains than students taught by Non-RttT
Comparison and NC NTSP Eligible Comparison sample teachers in the 2012-13 school year.
There were no significant mathematics differences in 2013-14 or using two years of pooled data.
Like mathematics, in elementary and middle grades reading, students taught by NC NTSP
teachers made significantly larger achievement gains than students taught by Non-RttT
Comparison and NC NTSP Eligible Comparison sample teachers in 2012-13. NC NTSP teachers
were less effective than NC NTSP Eligible Comparison sample teachers in 2013-14 and there
were no statistically significant differences with two years of pooled data. For fifth and eighth
grade science, students taught by NC NTSP teachers made significantly larger achievement gains
than students taught by Non-RttT Comparison sample teachers in 2013-14 and with two years of
pooled data. Examining these overall results by NC NTSP region, the middle and bottom panels
of Table G1 indicate that students taught by NC NTSP teachers in the ECU region made larger
achievement gains, across subjects (mathematics, reading, and science), than students taught by
teachers in either comparison group. NC NTSP teachers in the UNC-CSLD and UNCC regions
were more effective in mathematics and reading than teachers in either comparison group in the
2012-13 school year; NC NTSP teachers in the UNC-CSLD region were less effective than NC
NTSP Eligible Comparison sample teachers in reading in the 2013-14 school year. NC NTSP
teachers in the UNCG region were less effective than NC NTSP Eligible Comparison sample
teachers in reading in 2013-14 and less effective than both comparison groups in fifth and eighth
grade science.
The top panel of Table G2 shows that for EOC exams in 2013-14, students taught by NC NTSP
teachers made significantly smaller achievement gains than students taught by NC NTSP Eligible
Comparison sample teachers. Examining the EVAAS EOC results by NC NTSP region, the
middle and bottom panels of Table G2 indicate NC NTSP teachers from the ECU region were
less effective than both comparison groups in 2013-14 and with two years of pooled data. NC
NTSP teachers from the UNC-CSLD and UNCC regions were less effective than NC NTSP
Eligible Comparison sample teachers in 2013-14. NC NTSP teachers in the UNCG region were
more effective than both comparison groups in 2012-13 and more effective than the Non-RttT
Comparison sample with two years of pooled data.
To determine how these EVAAS estimates may differ by NC NTSP cohort, Tables G3 (fourth
page following) and G4 (fourth page following) present results for elementary and middle grades
and EOC exams. For the 2012-13 school year, results in the top and bottom panels of Table G3
repeat results from Table G1: Cohort 1 NC NTSP teachers were more effective in mathematics
and reading. Examining the 2013-14 school year scores, where data are available for both Cohort
1 and Cohort 2 NC NTSP teachers, results show that Cohort 2 NC NTSP teachers were less
effective than NC NTSP Eligible Comparison sample teachers in reading; furthermore, NC
NTSP teachers in both cohorts were more effective than Non-RttT Comparison sample teachers
in fifth and eighth grade science. With two years of pooled data, Cohort 1 NC NTSP teachers
were more effective than Non-RttT Comparison sample teachers in mathematics and reading. For
EOC exams (shown in Table G4), NC NTSP cohort results in 2012-13 match the overall results
in Table G2. Results in 2013-14 show that NC NTSP teachers in both cohorts were less effective
than NC NTSP Eligible Comparison sample teachers; there were no significant differences with
two years of pooled data.
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 63
Finally, Tables G5 (fourth page following) and G6 (fourth page following) present EVAAS
results for first-year teachers only. For the 2012-13 school year, Table G5 shows that students
taught by first-year NC NTSP teachers made significantly larger achievement gains in
mathematics and reading than students taught by both sets of comparison sample teachers; Table
G6 indicates that first-year NC NTSP teachers were more effective than first-year Non-RttT
Comparison sample teachers on EOC exams. In 2013-14, there were no significant differences
for first-year teachers in mathematics, reading, or fifth and eighth grade science, but NC NTSP
teachers were less effective than first-year NC NTSP Eligible Comparison sample teachers on
EOC exams. Using two years of pooled data, first-year NC NTSP teachers were more effective
than both comparison groups in elementary and middle grades mathematics, more effective than
NC NTSP Eligible Comparison sample teachers in elementary and middle grades reading, and
more effective than Non-RttT Comparison sample teachers in fifth and eighth grade science.
Overall, these EVAAS results are comparable to the student-level value-added results presented
later in this appendix. Evidence indicates that NC NTSP teachers were more effective than
comparison sample teachers in elementary and middle grades. These results were concentrated in
2012-13 (Cohort 1 NC NTSP teachers) and did not extend (outside of fifth and eighth grade
science) into the 2013-14 school year. Conversely, in the 2013-14 school year, NC NTSP
teachers were less effective in multiple comparisons with NC NTSP Eligible Comparison sample
teachers.
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 64
Table G1. Overall and Regional EVAAS Results (Evaluation Sample)
Group
Elementary & Middle Grades Math Elementary & Middle Grades Reading 5th
& 8th
Grade Science
2012-13 2013-14
2012-13 &
2013-14 2012-13 2013-14
2012-13 &
2013-14 2012-13 2013-14
2012-13 &
2013-14
NC NTSP vs.
Non-RttT
Comparison 3.054
** 0.155 0.828 1.670** 0.015 0.259 0.971 1.239
* 1.011
+
NC NTSP vs.
NC NTSP
Eligible
Comparison
3.923** -1.152 1.294 2.012
** -0.917
** 0.229 0.665 -0.043 0.498
NC NTSP Regions vs Non-RttT Comparison
ECU Region 5.623** 1.401 2.219
** 2.582
** 0.774
* 1.116
** 4.047
* 2.992
+ 2.772
*
UNC-CSLD
Region 4.465
* -1.165 -0.007 2.443** -0.733 -0.161 1.892 0.833 1.052
UNCC Region 3.041* 1.727 1.915+
1.130+ 0.090 0.236 -0.868 1.412 0.234
UNCG Region -0.923 -0.915 -1.033 0.295 -0.266 -0.336 -1.578+ 0.221 -0.051
NC NTSP Regions vs NC NTSP Eligible Comparison
ECU Region 6.140** 0.068 2.623
* 2.872
** -0.160 1.055 3.477+ 1.357 2.053
UNC-CSLD
Region 4.982
* -2.499 0.396 2.733**
-1.668** -0.221 1.322 -0.802 0.334
UNCC Region 3.558* 0.393 2.318
+ 1.420
+ -0.845 0.175 -1.438 -0.222 -0.485
UNCG Region -0.407 -2.248 -0.629 0.585 -1.200* -0.397 -2.148
+ -1.414
+ -0.769
Note: The top panel of this table displays differences in adjusted-average EVAAS estimates for NC NTSP teachers and Non-RttT Comparison and NC NTSP
Eligible Comparison sample teachers. The second panel of this table displays differences in adjusted-average EVAAS estimates for NC NTSP teachers, by
region, and the reference group of Non-RttT Comparison sample teachers. The bottom panel of this table displays differences in adjusted-average EVAAS
estimates for NC NTSP teachers, by region, and the reference group of NC NTSP Eligible Comparison sample teachers. +, *, and ** indicate statistically
significant differences at the 0.10, 0.05, and 0.01 levels, respectively.
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 65
Table G2. Overall and Regional EVAAS Results (Evaluation Sample)
Group EOC Exams 2012-13 EOC Exams 2013-14 EOC Exams 2012-13 & 2013-14
NC NTSP vs. Non-RttT
Comparison 0.693 -0.106 0.083
NC NTSP vs.
NC NTSP Eligible Comparison 0.455 -1.812
** -0.464
NC NTSP Regions vs Non-RttT Comparison
ECU Region 0.300 -1.361**
-0.652*
UNC-CSLD Region --- -0.843 -0.727
UNCC Region -0.469 -0.444 -0.370
UNCG Region 1.919* 1.112 1.214
*
NC NTSP Regions vs NC NTSP Eligible Comparison
ECU Region 0.120 -2.727**
-1.047**
UNC-CSLD Region --- -2.208+ -1.122
UNCC Region -0.649 -1.809* -0.765
UNCG Region 1.739+ -0.253 0.819
Note: The top panel of this table displays differences in adjusted-average EVAAS estimates for NC NTSP teachers and Non-RttT
Comparison and NC NTSP Eligible Comparison sample teachers. The second panel of this table displays differences in adjusted-average
EVAAS estimates for NC NTSP teachers, by region, and the reference group of Non-RttT Comparison sample teachers. The bottom panel
of this table displays differences in adjusted-average EVAAS estimates for NC NTSP teachers, by region, and the reference group of NC
NTSP Eligible Comparison sample teachers. +, *, and ** indicate statistically significant differences at the 0.10, 0.05, and 0.01 levels,
respectively.
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 66
Table G3. Cohort NC NTSP EVAAS Results (Evaluation Sample)
Group
Elementary and Middle Grades Math Elementary and Middle Grades Reading 5th
and 8th
Grade Science
2012-13 2013-14
2012-13 &
2013-14 2012-13 2013-14
2012-13 &
2013-14 2012-13 2013-14
2012-13 &
2013-14
NC NTSP Cohorts vs Non-RttT Comparison
NC NTSP
Cohort 1 3.054
** 0.639 1.483
+ 1.670
** 0.307 0.759
* 0.971 1.208
+ 1.127
NC NTSP
Cohort 2 --- -0.189 -0.011 --- -0.183 -0.273 --- 1.261
+ 0.862
NC NTSP Cohorts vs NC NTSP Eligible Comparison
NC NTSP
Cohort 1 3.923
** -0.675 1.860 2.012
** -0.617 0.679 0.665 -0.071 0.581
NC NTSP
Cohort 2 --- -1.503 0.366 --- -1.107
** -0.353 --- -0.018 0.316
Note: The top panel of this table displays differences in adjusted-average EVAAS estimates between Cohort 1 and Cohort 2 NC NTSP teachers and Non-RttT
Comparison sample teachers. The bottom panel of this table displays differences in adjusted-average EVAAS estimates between Cohort 1 and Cohort 2 NC
NTSP teachers and NC NTSP Eligible Comparison sample teachers. +, *, and ** indicate statistically significant differences at the 0.10, 0.05, and 0.01 levels,
respectively.
Table G4. Cohort NC NTSP EVAAS Results (Evaluation Sample)
Cohort 2012-13 EOC Exams 2013-14 EOC Exams 2012-13 & 2013-14 EOC Exams
NC NTSP Cohorts vs Non-RttT Comparison
NC NTSP Cohort 1 0.693 -0.088 0.029
NC NTSP Cohort 2 --- -0.122 0.202
NC NTSP Cohorts vs NC NTSP Eligible Comparison
NC NTSP Cohort 1 0.455 -1.790*
-0.530
NC NTSP Cohort 2 --- -1.823*
-0.356
Note: The top panel of this table displays differences in adjusted-average EVAAS estimates between Cohort 1 and Cohort 2 NC NTSP teachers and
Non-RttT Comparison sample teachers. The bottom panel of this table displays differences in adjusted-average EVAAS estimates between Cohort 1
and Cohort 2 NC NTSP teachers and NC NTSP Eligible Comparison sample teachers. +, *, and ** indicate statistically significant differences at the
0.10, 0.05, and 0.01 levels, respectively.
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 67
Table G5. First-Year NC NTSP EVAAS Results (Evaluation Sample)
Group
Elementary and Middle Grades Math Elementary and Middle Grades Reading 5th
and 8th
Grade Science
2012-13 2013-14
2012-13 &
2013-14 2012-13 2013-14
2012-13 &
2013-14 2012-13 2013-14
2012-13 &
2013-14
NC NTSP vs.
Non-RttT
Comparison 2.985
* 0.161 1.442+
1.331** -0.341 0.115 1.211 1.183 1.442
+
NC NTSP vs.
NC NTSP
Eligible
Comparison
5.974** -1.172 3.472
* 2.389
** -0.425 0.943+ 1.741 -0.070 1.430
Note: This table displays differences in adjusted-average EVAAS estimates between first-year NC NTSP teachers and first-year Non-RttT Comparison and NC
NTSP Eligible Comparison sample teachers. +, *, and ** indicate statistically significant differences at the 0.10, 0.05, and 0.01 levels, respectively.
Table G6. First-Year NC NTSP EVAAS Results (Evaluation Sample)
Group 2012-13 EOC Exams 2013-14 EOC Exams 2012-13 & 2013-14 EOC Exams
NC NTSP vs. Non-RttT
Comparison 1.304
* -0.047 0.596
NC NTSP vs.
NC NTSP Eligible Comparison 0.775 -2.922
** -0.341
Note: This table displays differences in adjusted-average EVAAS estimates between first-year NC NTSP teachers and first-year Non-RttT
Comparison and NC NTSP Eligible Comparison sample teachers. +, *, and ** indicate statistically significant differences at the 0.10, 0.05, and 0.01
levels, respectively.
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 68
Student-Level Value-Added Results
As an alternative way to assess the contributions of NC NTSP teachers to student achievement,
the Evaluation Team estimated ordinary least squares (OLS) regression models with students’
standardized test scores (standardized within subject, grade, and year for End-of-Grade exams
and subject and year for End-of-Course exams) as the outcome variable, an extensive set of
student, classroom, teacher, and school characteristics to help isolate the impact of the NC NTSP
on adjusted-average student achievement gains, and standard errors clustered at the school level
to account for dependence in the data. Results from these models express the adjusted-average
differences in student achievement—expressed in standard deviations of student achievement—
between students taught by NC NTSP teachers and students taught by Non-RttT Comparison or
NC NTSP Eligible Comparison sample teachers. Given the relatively small sample of teachers
from the NC NTSP and the comparison groups who taught a tested grade/subject, the Evaluation
Team combined data from elementary and middle grades (four to eight) and ran separate models
for mathematics, reading, and science (EOG science exams in grades five and eight).
Additionally, the Evaluation Team combined data for the three high school (grades nine through
twelve) EOC exams (Biology, English II, and Math I) and ran a single high school EOC model.
In this appendix, the Evaluation Team displays results for the evaluation sample and for (1) the
complete sample of teachers served by the NC NTSP and in the comparison groups, and (2) an
amended evaluation sample that includes NC NTSP teachers who worked in non-RttT schools—
still excluding TFA corps members and those entering the NC NTSP in January or later.
Tables G7 (second page following) and G8 (third page following) display overall and regional
NC NTSP value-added results for elementary and middle grades and high school. Overall, the
top panel of Table G7 shows that in elementary and middle grades mathematics, students taught
by NC NTSP teachers made significantly larger achievement gains than students taught by Non-
RttT Comparison and NC NTSP Eligible Comparison sample teachers in the 2012-13 school
year. There were no significant mathematics differences in 2013-14; using two years of pooled
data, NC NTSP teachers were more effective than Non-RttT Comparison sample teachers. Like
mathematics, in elementary and middle grades reading, students taught by NC NTSP teachers
made significantly larger achievement gains than students taught by Non-RttT Comparison and
NC NTSP Eligible Comparison sample teachers during the 2012-13 school year. For fifth and
eighth grade science, students taught by NC NTSP teachers made significantly larger
achievement gains than students taught by NC NTSP Eligible Comparison sample teachers in
2012-13, Non-RttT Comparison sample teachers in 2013-14, and both comparison groups using
two years of pooled data.
Examining these overall results by NC NTSP region, the middle and bottom panels of Table G7
indicate that students taught by NC NTSP teachers in the ECU region made larger achievement
gains across subjects (mathematics, reading, and science), than students taught by teachers in
either comparison group. NC NTSP teachers in the UNC-CSLD region were more effective in
mathematics and reading than teachers in either comparison group in the 2012-13 school year but
less effective in the 2013-14 school year. NC NTSP teachers in the UNCC region were more
effective in mathematics (particularly in 2012-13) and in fifth and eighth grade science; NC
NTSP teachers from the UNCG region were less effective than teachers from both comparison
groups in mathematics in the 2013-14 school year and less effective than Non-RttT Comparison
sample teachers in fifth and eighth grade science in 2012-13.
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 69
The top panel of Table G8 (second page following) shows that for high school grades (nine
through twelve) EOC exams in 2013-14 and with two years of pooled data, students taught by
NC NTSP evaluation sample teachers made significantly smaller achievement gains than
students taught by Non-RttT Comparison and NC NTSP Eligible Comparison sample teachers.
Examining these results by NC NTSP region, the middle and bottom panels of Table G8 indicate
that NC NTSP teachers from the UNC-CSLD and UNCG regions were more effective than
comparison sample teachers in the 2012-13 school year. NC NTSP teachers in the ECU region
were less effective in 2013-14—the coefficients for NC NTSP in the other regions were negative
but not significant—and with two years of pooled data; NC NTSP teachers in the UNC-CSLD
and UNCC regions were less effective than NC NTSP Eligible Comparison sample teachers with
two years of pooled data.
To examine how these value-added estimates may differ by NC NTSP cohort, Tables G9 and
G10 (third page following) present results for elementary and middle grades and high school. For
the 2012-13 school year, results in the top and bottom panels of Table G9 repeat results from
Table G7: Cohort 1 NC NTSP teachers were more effective in mathematics, reading, and fifth
and eighth grade science. Examining the 2013-14 school year, where data is available for both
Cohort 1 and Cohort 2 NC NTSP teachers, results show that (1) there were no significant
differences for Cohort 1 teachers in mathematics or reading; (2) Cohort 2 teachers were less
effective in mathematics than NC NTSP Eligible Comparison sample teachers; and (3) Cohort 1
and Cohort 2 NC NTSP teachers were more effective than Non-RttT Comparison sample
teachers in fifth and eighth grade science. With two years of pooled data, Cohort 1 NC NTSP
teachers were more effective than both comparison groups in mathematics and fifth and eighth
grade science and more effective than Non-RttT Comparison sample teachers in reading. For
high school EOCs, NC NTSP cohort results in 2012-13 match the overall results reported in
Table G8. Results for both NC NTSP cohorts in 2013-14 and with two years of pooled data
indicate that Cohort 1 teachers were less effective than Non-RttT Comparison sample teachers
and that both NC NTSP cohorts were less effective than NC NTSP Eligible Comparison sample
teachers.
Finally, Tables G11 and G12 (fourth page following) present value-added results for first-year
teachers only. For the 2012-13 school year, Table G11 shows that students taught by first-year
NC NTSP teachers made significantly larger achievement gains in mathematics, reading, and
fifth and eighth grade science than students taught by both sets of comparison sample teachers.
First-year NC NTSP teachers were also more effective in 2013-14 in fifth and eighth grade
science and with two years of pooled data in mathematics and fifth and eighth grade science.
High school grades EOC results in Table G12 show that in the 2012-13 school year students
taught by first-year NC NTSP teachers made significantly larger achievement gains than students
taught by first-year Non-RttT Comparison sample teachers; the opposite was true in 2013-14, as
first-year NC NTSP teachers were significantly less effective than first-year Non-RttT
Comparison sample teachers.
Overall, evidence from elementary and middle grades indicates that NC NTSP teachers were
more effective than comparison sample teachers. These results were concentrated in 2012-13
(Cohort 1 NC NTSP teachers) and did not extend (outside of fifth and eighth grade science) into
the 2013-14 school year. Conversely, in the 2013-14 school year, NC NTSP teachers (overall
and by cohort) were less effective than comparison group teachers in high school grades EOCs.
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 70
Table G7. Overall and Regional NC NTSP Value-Added Results (Evaluation Sample)
Group
Elementary and Middle Grades Math Elementary and Middle Grades Reading 5th
and 8th
Grade Science
2012-13 2013-14
2012-13 &
2013-14 2012-13 2013-14
2012-13 &
2013-14 2012-13 2013-14
2012-13 &
2013-14
NC NTSP vs.
Non-RttT
Comparison 0.194
** 0.013 0.056+
0.122** 0.000 0.025 0.108 0.128
* 0.122
*
NC NTSP vs.
NC NTSP
Eligible
Comparison
0.266** -0.101 0.087 0.121
* -0.025 0.035 0.189+ 0.102 0.178
*
NC NTSP Regions vs Non-RttT Comparison
ECU Region 0.289**
0.086+
0.125**
0.176**
0.084**
0.111**
0.256+
0.210+
0.198+
UNC-CSLD
Region 0.197
* -0.051 -0.006 0.166**
-0.068+ -0.000 0.076 0.082 0.073
UNCC Region 0.206** 0.064 0.111
** 0.084
+ -0.026 -0.007 0.113 0.199*
0.189*
UNCG Region -0.058 -0.096+ -0.089 -0.038 -0.030 -0.033 -0.193
* -0.040 -0.072
NC NTSP Regions vs NC NTSP Eligible Comparison
ECU Region 0.353** -0.034 0.151
* 0.172
* 0.064 0.123*
0.333* 0.190 0.256
*
UNC-CSLD
Region 0.261
** -0.170
* 0.020 0.162**
-0.088+ 0.011 0.152 0.063 0.131
UNCC Region 0.269** -0.056 0.138
* 0.080 -0.046 0.004 0.190 0.179 0.247*
UNCG Region 0.006 -0.215* -0.063 -0.042 -0.050 -0.021 -0.116 -0.059 -0.014
Note: The top panel of this table displays differences in adjusted-average student achievement between NC NTSP teachers and Non-RttT Comparison and NC
NTSP Eligible Comparison sample teachers. The second panel of this table displays differences in adjusted-average student achievement between teachers from
each NC NTSP region and the reference group of Non-RttT Comparison sample teachers. The bottom panel of this table displays differences in adjusted-average
student achievement between teachers from each NC NTSP region and the reference group of NC NTSP Eligible Comparison sample teachers. +, *, and **
indicate statistically significant differences at the 0.10, 0.05, and 0.01 levels, respectively.
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 71
Table G8. Overall and Regional NC NTSP Value-Added Results (Evaluation Sample)
2012-13 EOC Exams 2013-14 EOC Exams 2012-13 & 2013-14 EOC Exams
NC NTSP vs. Non-RttT
Comparison 0.068 -0.099
* -0.057
+
NC NTSP vs.
NC NTSP Eligible Comparison 0.019 -0.151
* -0.100
**
NC NTSP Regions vs Non-RttT Comparison
ECU Region 0.045 -0.264**
-0.147**
UNC-CSLD Region 0.218+ -0.064 -0.058
UNCC Region -0.027 -0.077 -0.066
UNCG Region 0.254** -0.067 0.025
NC NTSP Regions vs NC NTSP Eligible Comparison
ECU Region -0.012 -0.292**
-0.180**
UNC-CSLD Region 0.161 -0.092 -0.091*
UNCC Region -0.084 -0.105 -0.099+
UNCG Region 0.197* -0.095 -0.008
Note: The top panel of this table displays differences in adjusted-average student achievement between NC NTSP teachers and Non-RttT Comparison
and NC NTSP Eligible Comparison sample teachers. The second panel of this table displays differences in adjusted-average student achievement
between teachers from each NC NTSP region and the reference group of Non-RttT Comparison sample teachers. The bottom panel of this table
displays differences in adjusted-average student achievement between teachers from each NC NTSP region and the reference group of NC NTSP
Eligible Comparison sample teachers. +, *, and ** indicate statistically significant differences at the 0.10, 0.05, and 0.01 levels, respectively.
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 72
Table G9. Cohort NC NTSP Value-Added Results (Evaluation Sample)
Group
Elementary and Middle Grades Math Elementary and Middle Grades Reading 5th
and 8th
Grade Science
2012-13 2013-14
2012-13 &
2013-14 2012-13 2013-14
2012-13 &
2013-14 2012-13 2013-14
2012-13 &
2013-14
NC NTSP Cohorts vs Non-RttT Comparison
NC NTSP
Cohort 1 0.194
** 0.045 0.102**
0.122** 0.031 0.064
** 0.108 0.123*
0.131*
NC NTSP
Cohort 2 --- -0.014 -0.004 --- -0.018 -0.022 --- 0.132
+ 0.106
NC NTSP Cohorts vs NC NTSP Eligible Comparison
NC NTSP
Cohort 1 0.265
** -0.070 0.122*
0.121* 0.007 0.069 0.189
+ 0.097 0.185*
NC NTSP
Cohort 2 --- -0.130
+ 0.016 --- -0.043 -0.017 --- 0.106 0.160+
Note: The top panel of this table displays differences in adjusted-average student achievement between Cohort 1 and Cohort 2 NC NTSP teachers and Non-RttT
Comparison sample teachers. The bottom panel of this table displays differences in adjusted-average student achievement between Cohort 1 and Cohort 2 NC
NTSP teachers and NC NTSP Eligible Comparison sample teachers. +, *, and ** indicate statistically significant differences at the 0.10, 0.05, and 0.01 levels,
respectively.
Table G10. Cohort NC NTSP Value-Added Results (Evaluation Sample)
Cohort 2012-13 EOC Exams 2013-14 EOC Exams 2012-13 & 2013-14 EOC Exams
NC NTSP Cohorts vs Non-RttT Comparison
NC NTSP Cohort 1 0.068 -0.118*
-0.064+
NC NTSP Cohort 2 --- -0.079 -0.044
NC NTSP Cohorts vs NC NTSP Eligible Comparison
NC NTSP Cohort 1 0.019 -0.172*
-0.109**
NC NTSP Cohort 2 --- -0.133+
-0.089+
Note: The top panel of this table displays differences in adjusted-average student achievement between Cohort 1 and Cohort 2 NC NTSP teachers and
Non-RttT Comparison sample teachers. The bottom panel of this table displays differences in adjusted-average student achievement between Cohort 1
and Cohort 2 NC NTSP teachers and NC NTSP Eligible Comparison sample teachers. +, *, and ** indicate statistically significant differences at the
0.10, 0.05, and 0.01 levels, respectively.
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 73
Table G11. First-Year NC NTSP Value-Added Results (Evaluation Sample)
Group
Elementary and Middle Grades Math Elementary and Middle Grades Reading 5th
and 8th
Grade Science
2012-13 2013-14
2012-13 &
2013-14 2012-13 2013-14
2012-13 &
2013-14 2012-13 2013-14
2012-13 &
2013-14
NC NTSP vs.
Non-RttT
Comparison 0.205
** 0.006 0.099**
0.088* -0.017 0.014 0.271
** 0.202
* 0.192
**
NC NTSP vs.
NC NTSP
Eligible
Comparison
0.330** -0.128 0.176
* 0.118
* 0.038 0.076 0.235*
0.757**
0.286*
Note: This table displays differences in adjusted-average student achievement between first-year NC NTSP teachers and first-year Non-RttT Comparison and NC
NTSP Eligible Comparison sample teachers. +, *, and ** indicate statistically significant differences at the 0.10, 0.05, and 0.01 levels, respectively.
Table G12. First-Year NC NTSP Value-Added Results (Evaluation Sample)
Group 2012-13 EOC Exams 2013-14 EOC Exams 2012-13 & 2013-14 EOC Exams
NC NTSP vs. Non-RttT
Comparison 0.176
** -0.181
+ 0.014
NC NTSP vs.
NC NTSP Eligible
Comparison
0.037 -0.180 -0.074
Note: This table displays differences in adjusted-average student achievement between first-year NC NTSP teachers and first-year Non-RttT Comparison and NC
NTSP Eligible Comparison sample teachers. +, *, and ** indicate statistically significant differences at the 0.10, 0.05, and 0.01 levels, respectively.
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 74
Student-Level Value-Added Results: Complete and Amended Evaluation Samples
Table G13. Overall and Regional NC NTSP Value-Added Results (Complete Sample)
Group
Elementary and Middle Grades Math Elementary and Middle Grades Reading 5th
and 8th
Grade Science
2012-13 2013-14
2012-13 &
2013-14 2012-13 2013-14
2012-13 &
2013-14 2012-13 2013-14
2012-13 &
2013-14
NC NTSP vs.
Non-RttT
Comparison
0.049 0.014 0.026 0.094** -0.004 0.020 -0.029 0.118
* 0.062
NC NTSP vs.
NC NTSP
Eligible
Comparison
0.159** -0.108 0.051 0.085
+ -0.029 0.030 0.082 0.056 0.104
NC NTSP Regions vs Non-RttT Comparison
ECU Region 0.159*
0.073+
0.096*
0.119**
0.078**
0.099** -0.022 0.159 0.096
UNC-CSLD
Region 0.050 -0.016 -0.001 0.132
** -0.072
** -0.006 0.005 0.078 0.028
UNCC Region 0.144* 0.033 0.063
+ 0.073 -0.024 -0.015 0.076 0.202**
0.147+
UNCG Region -0.187** -0.073 -0.113
* -0.049 -0.011 -0.018 -0.314** -0.043 -0.130
*
NC NTSP Regions vs NC NTSP Eligible Comparison
ECU Region 0.252** -0.054 0.115
+ 0.108
+ 0.057 0.109* 0.086 0.095 0.144
UNC-CSLD
Region 0.143
* -0.144
+ 0.017 0.121*
-0.092* 0.004 0.113 0.014 0.076
UNCC Region 0.237** -0.095 0.082 0.062 -0.044 -0.005 0.184 0.138 0.195
+
UNCG Region -0.094 -0.201* -0.094 -0.060 -0.031 -0.008 -0.205
+ -0.098 -0.082
Note: The top panel of this table displays differences in adjusted-average student achievement between NC NTSP teachers and Non-RttT Comparison and NC
NTSP Eligible Comparison sample teachers. The second panel of this table displays differences in adjusted-average student achievement between teachers from
each NC NTSP region and the reference group of Non-RttT Comparison sample teachers. The bottom panel of this table displays differences in adjusted-average
student achievement between teachers from each NC NTSP region and the reference group of NC NTSP Eligible Comparison sample teachers. +, *, and **
indicate statistically significant differences at the 0.10, 0.05, and 0.01 levels, respectively.
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 75
Table G14. Overall and Regional NC NTSP Value-Added Results (Amended Evaluation Sample)
Group
Elementary and Middle Grades Math Elementary and Middle Grades Reading 5th
and 8th
Grade Science
2012-13 2013-14
2012-13 &
2013-14 2012-13 2013-14
2012-13 &
2013-14 2012-13 2013-14
2012-13 &
2013-14
NC NTSP vs.
Non-RttT
Comparison 0.189
** 0.017 0.051+
0.121** -0.003 0.021 0.093 0.117
* 0.107
*
NC NTSP vs.
NC NTSP
Eligible
Comparison
0.260** -0.090 0.084 0.119
* -0.025 0.030 0.174+ 0.079 0.162
*
NC NTSP Regions vs Non-RttT Comparison
ECU Region 0.288** 0.066 0.109
** 0.176
** 0.082
** 0.105
** 0.238
+ 0.178 0.182+
UNC-CSLD
Region 0.182
* -0.006 0.014 0.164**
-0.070** -0.009 0.033 0.048 0.029
UNCC Region 0.204** 0.049 0.091
* 0.083
+ -0.026 -0.007 0.114 0.220**
0.191*
UNCG Region -0.058 -0.075 -0.082 -0.037 -0.015 -0.022 -0.219* -0.037 -0.075
NC NTSP Regions vs NC NTSP Eligible Comparison
ECU Region 0.351** -0.046 0.138
* 0.172
* 0.063 0.115*
0.320* 0.139 0.245
*
UNC-CSLD
Region 0.245
** -0.119 0.042 0.160**
-0.089+ 0.001 0.115 0.009 0.092
UNCC Region 0.267** -0.063 0.119
+ 0.079 -0.044 0.003 0.196 0.181 0.254*
UNCG Region 0.005 -0.187* -0.054 -0.041 -0.034 -0.012 -0.137 -0.076 -0.012
Note: The top panel of this table displays differences in adjusted-average student achievement between NC NTSP teachers and Non-RttT Comparison and NC
NTSP Eligible Comparison sample teachers. The second panel of this table displays differences in adjusted-average student achievement between teachers from
each NC NTSP region and the reference group of Non-RttT Comparison sample teachers. The bottom panel of this table displays differences in adjusted-average
student achievement between teachers from each NC NTSP region and the reference group of NC NTSP Eligible Comparison sample teachers. +, *, and **
indicate statistically significant differences at the 0.10, 0.05, and 0.01 levels, respectively.
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 76
Table G15. Overall and Regional NC NTSP Value-Added Results (Complete Sample)
2012-13 EOC Exams 2013-14 EOC Exams 2012-13 & 2013-14 EOC Exams
NC NTSP vs. Non-RttT
Comparison -0.025 -0.085
* -0.069
+
NC NTSP vs.
NC NTSP Eligible Comparison -0.024 -0.096
+ -0.079
**
NC NTSP Regions vs Non-RttT Comparison
ECU Region -0.053 -0.311**
-0.202**
UNC-CSLD Region 0.081 -0.070 -0.047
UNCC Region -0.122* -0.071 -0.077
+
UNCG Region 0.090 -0.061 -0.008
NC NTSP Regions vs NC NTSP Eligible Comparison
ECU Region -0.048 -0.308**
-0.205**
UNC-CSLD Region 0.085 -0.067 -0.050
UNCC Region -0.118 -0.068 -0.080+
UNCG Region 0.094 -0.059 -0.012
Note: The top panel of this table displays differences in adjusted-average student achievement between NC NTSP teachers and Non-RttT Comparison and
NC NTSP Eligible Comparison sample teachers. The second panel of this table displays differences in adjusted-average student achievement between
teachers from each NC NTSP region and the reference group of Non-RttT Comparison sample teachers. The bottom panel of this table displays differences
in adjusted-average student achievement between teachers from each NC NTSP region and the reference group of NC NTSP Eligible Comparison sample
teachers. +, *, and ** indicate statistically significant differences at the 0.10, 0.05, and 0.01 levels, respectively.
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 77
Table G16. Overall and Regional NC NTSP Value-Added Results (Amended Evaluation Sample)
2012-13 EOC Exams 2013-14 EOC Exams 2012-13 & 2013-14 EOC Exams
NC NTSP vs. Non-RttT
Comparison 0.079
+ -0.060 -0.018
NC NTSP vs.
NC NTSP Eligible Comparison 0.014 -0.083 -0.059
+
NC NTSP Regions vs Non-RttT Comparison
ECU Region 0.055 -0.292**
-0.170**
UNC-CSLD Region 0.174 -0.047 -0.025
UNCC Region -0.030 -0.041 -0.011
UNCG Region 0.232** -0.043 0.053
NC NTSP Regions vs NC NTSP Eligible Comparison
ECU Region -0.005 -0.300**
-0.196**
UNC-CSLD Region 0.114 -0.056 -0.051
UNCC Region -0.090 -0.050 -0.037
UNCG Region 0.172* -0.052 0.027
Note: The top panel of this table displays differences in adjusted-average student achievement between NC NTSP teachers and Non-RttT Comparison and
NC NTSP Eligible Comparison sample teachers. The second panel of this table displays differences in adjusted-average student achievement between
teachers from each NC NTSP region and the reference group of Non-RttT Comparison sample teachers. The bottom panel of this table displays differences
in adjusted-average student achievement between teachers from each NC NTSP region and the reference group of NC NTSP Eligible Comparison sample
teachers. +, *, and ** indicate statistically significant differences at the 0.10, 0.05, and 0.01 levels, respectively.
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 78
Table G17. Cohort NC NTSP Value-Added Results (Complete Sample)
Group
Elementary and Middle Grades Math Elementary and Middle Grades Reading 5th
and 8th
Grade Science
2012-13 2013-14
2012-13 &
2013-14 2012-13 2013-14
2012-13 &
2013-14 2012-13 2013-14
2012-13 &
2013-14
NC NTSP Cohorts vs Non-RttT Comparison
NC NTSP
Cohort 1 0.049 0.030 0.043 0.094
** 0.014 0.050* -0.029 0.168
** 0.075
NC NTSP
Cohort 2 --- 0.004 0.002 --- -0.012 -0.013 --- 0.086 0.042
NC NTSP Cohorts vs NC NTSP Eligible Comparison
NC NTSP
Cohort 1 0.159
** -0.093 0.066 0.085+ -0.011 0.055 0.082 0.105 0.115
NC NTSP
Cohort 2 --- -0.119 0.024 --- -0.038 -0.007 --- 0.023 0.082
Note: The top panel of this table displays differences in adjusted-average student achievement between Cohort 1 and Cohort 2 NC NTSP teachers and Non-RttT
Comparison sample teachers. The bottom panel of this table displays differences in adjusted-average student achievement between Cohort 1 and Cohort 2 NC
NTSP teachers and NC NTSP Eligible Comparison sample teachers. +, *, and ** indicate statistically significant differences at the 0.10, 0.05, and 0.01 levels,
respectively.
Table G18. Cohort NC NTSP Value-Added Results (Amended Evaluation Sample)
Group
Elementary and Middle Grades Math Elementary and Middle Grades Reading 5th
and 8th
Grade Science
2012-13 2013-14
2012-13 &
2013-14 2012-13 2013-14
2012-13 &
2013-14 2012-13 2013-14
2012-13 &
2013-14
NC NTSP Cohorts vs Non-RttT Comparison
NC NTSP
Cohort 1 0.189
** 0.039 0.095**
0.121** 0.035 0.066
** 0.093 0.165**
0.146*
NC NTSP
Cohort 2 --- 0.003 0.006 --- -0.020 -0.021 --- 0.088 0.059
NC NTSP Cohorts vs NC NTSP Eligible Comparison
NC NTSP
Cohort 1 0.260
** -0.067 0.121*
0.119* 0.013 0.071 0.174
+ 0.122 0.190*
NC NTSP
Cohort 2 --- -0.104 0.032 --- -0.042 -0.017 --- 0.044 0.103
Note: The top panel of this table displays differences in adjusted-average student achievement between Cohort 1 and Cohort 2 NC NTSP teachers and Non-RttT
Comparison sample teachers. The bottom panel of this table displays differences in adjusted-average student achievement between Cohort 1 and Cohort 2 NC
NTSP teachers and NC NTSP Eligible Comparison sample teachers. +, *, and ** indicate statistically significant differences at the 0.10, 0.05, and 0.01 levels,
respectively.
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 79
Table G19. Cohort NC NTSP Value-Added Results (Complete Sample)
Cohort 2012-13 EOC Exams 2013-14 EOC Exams 2012-13 & 2013-14 EOC Exams
NC NTSP Cohorts vs Non-RttT Comparison
NC NTSP Cohort 1 -0.025 -0.102*
-0.084*
NC NTSP Cohort 2 --- -0.076+ -0.054
NC NTSP Cohorts vs NC NTSP Eligible Comparison
NC NTSP Cohort 1 -0.024 -0.116 -0.095*
NC NTSP Cohort 2 --- -0.091+
-0.065+
Note: The top panel of this table displays differences in adjusted-average student achievement between Cohort 1 and Cohort 2 NC NTSP teachers and
Non-RttT Comparison sample teachers. The bottom panel of this table displays differences in adjusted-average student achievement between Cohort 1
and Cohort 2 NC NTSP teachers and NC NTSP Eligible Comparison sample teachers. +, *, and ** indicate statistically significant differences at the 0.10,
0.05, and 0.01 levels, respectively.
Table G20. Cohort NC NTSP Value-Added Results (Amended Evaluation Sample)
Cohort 2012-13 EOC Exams 2013-14 EOC Exams 2012-13 & 2013-14 EOC Exams
NC NTSP Cohorts vs Non-RttT Comparison
NC NTSP Cohort 1 0.079+
-0.090+ -0.038
NC NTSP Cohort 2 --- -0.042 0.004
NC NTSP Cohorts vs NC NTSP Eligible Comparison
NC NTSP Cohort 1 0.014 -0.119 -0.083*
NC NTSP Cohort 2 --- -0.071 -0.040
Note: The top panel of this table displays differences in adjusted-average student achievement between Cohort 1 and Cohort 2 NC NTSP teachers and Non-
RttT Comparison sample teachers. The bottom panel of this table displays differences in adjusted-average student achievement between Cohort 1 and
Cohort 2 NC NTSP teachers and NC NTSP Eligible Comparison sample teachers. +, *, and ** indicate statistically significant differences at the 0.10, 0.05,
and 0.01 levels, respectively.
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 80
Table G21. First-Year NC NTSP Value-Added Results (Complete Sample)
Group
Elementary and Middle Grades Math Elementary and Middle Grades Reading 5th
and 8th
Grade Science
2012-13 2013-14
2012-13 &
2013-14 2012-13 2013-14
2012-13 &
2013-14 2012-13 2013-14
2012-13 &
2013-14
NC NTSP vs.
Non-RttT
Comparison 0.094
+ 0.018 0.067 0.079* 0.001 0.024 0.079 0.156
* 0.131
*
NC NTSP vs.
NC NTSP
Eligible
Comparison
0.265** -0.145 0.129
* 0.101
+ 0.026 0.075 0.167 0.115 0.177+
Note: This table displays differences in adjusted-average student achievement between first-year NC NTSP teachers and first-year Non-RttT Comparison and NC
NTSP Eligible Comparison sample teachers. +, *, and ** indicate statistically significant differences at the 0.10, 0.05, and 0.01 levels, respectively.
Table G22. First-Year NC NTSP Value-Added Results (Amended Evaluation Sample)
Group
Elementary and Middle Grades Math Elementary and Middle Grades Reading 5th
and 8th
Grade Science
2012-13 2013-14
2012-13 &
2013-14 2012-13 2013-14
2012-13 &
2013-14 2012-13 2013-14
2012-13 &
2013-14
NC NTSP vs.
Non-RttT
Comparison 0.205
** 0.007 0.094**
0.088* -0.016 0.011 0.271
** 0.203
** 0.198
**
NC NTSP vs.
NC NTSP
Eligible
Comparison
0.330** -0.127 0.170
* 0.118
* 0.033 0.073 0.235*
0.756**
0.288*
Note: This table displays differences in adjusted-average student achievement between first-year NC NTSP teachers and first-year Non-RttT Comparison and NC
NTSP Eligible Comparison sample teachers. +, *, and ** indicate statistically significant differences at the 0.10, 0.05, and 0.01 levels, respectively.
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 81
Table G23. First-Year NC NTSP Value-Added Results (Complete Sample)
2012-13 EOC Exams 2013-14 EOC Exams 2012-13 & 2013-14 EOC Exams
NC NTSP vs. Non-RttT
Comparison -0.009 -0.175
* -0.090
+
NC NTSP vs.
NC NTSP Eligible Comparison -0.049 -0.121
+ -0.118
*
Note: This table displays differences in adjusted-average student achievement between first-year NC NTSP teachers and first-year Non-RttT Comparison
and NC NTSP Eligible Comparison sample teachers. +, *, and ** indicate statistically significant differences at the 0.10, 0.05, and 0.01 levels,
respectively.
Table G24. First-Year NC NTSP Value-Added Results (Amended Evaluation Sample)
2012-13 EOC Exams 2013-14 EOC Exams 2012-13 & 2013-14 EOC Exams
NC NTSP vs. Non-RttT
Comparison 0.146
** -0.165
* -0.004
NC NTSP vs.
NC NTSP Eligible Comparison 0.037 -0.073 -0.056
Note: This table displays differences in adjusted-average student achievement between first-year NC NTSP teachers and first-year Non-RttT Comparison
and NC NTSP Eligible Comparison sample teachers. +, *, and ** indicate statistically significant differences at the 0.10, 0.05, and 0.01 levels,
respectively.
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 82
Appendix H. To What Extent does the NC NTSP Impact Teacher Effectiveness as
Measured by the North Carolina Educator Evaluation System (NCEES) Teacher
Evaluation Ratings?
For analyses of teacher evaluation ratings, the Evaluation Team presents four sets of results: (1)
overall NC NTSP results for the 2012-13 school year, the 2013-14 school year, and the two years
combined; (2) regional NC NTSP results (separate results for teachers in the ECU, UNC-CSLD,
UNCC, and UNCG regions) for the 2012-13 school year, the 2013-14 school year, and the two
years combined; (3) cohort NC NTSP results (separate results for Cohort 1 teachers—entering
the program in the 2012-13 year—and Cohort 2 teachers—entering the program in 2013-14) for
the 2012-13 school year, the 2013-14 school year, and the two years combined;20
and (4) first-
year teacher NC NTSP results, which focus on first-year teachers in the 2012-13 school year, the
2013-14 school year, and the two years combined.21
Taken together, this set of analyses presents
evidence regarding the overall effectiveness of the program and how that effectiveness may have
differed by NC NTSP region, NC NTSP cohort, and for first-year teachers. These sub-analyses
are particularly important given the differences in NC NTSP treatment components across
regions and cohorts and the hypothesis of stronger program effects for first-year teachers.
Since only a minority of educators teach in tested grade/subjects and many important aspects of
teaching will not be fully captured by teachers’ value-added to student achievement, the
Evaluation Team analyzed teachers’ observation-based evaluation ratings on the North Carolina
Professional Teaching Standards (NCPTS). There are five Standards directly assessed by school
principals—Demonstrating Leadership (Standard 1), Establishing a Respectful Classroom
Environment (Standard 2), Content Knowledge (Standard 3), Facilitating Student Learning
(Standard 4), and Reflecting on Practice (Standard 5)—and for each Standard school principals
rate teachers at one of five levels—Not Demonstrated, Developing, Proficient, Accomplished,
and Distinguished. For these analyses the Evaluation Team estimated ordered logistic regression
models where the outcome variable was a teacher’s evaluation rating on the one to five scale
(where one was Not Demonstrated and five was Distinguished). These models controlled for
teacher experience and school characteristics to help isolate the impact of the NC NTSP on
teachers’ ratings, and clustered standard errors at the school level to account for dependence in
the data. Results from these models express the odds of NC NTSP evaluation sample teachers
receiving higher evaluation ratings than their Non-RttT Comparison and NC NTSP Eligible
Comparison sample peers. Statistically significant odds ratios greater than one indicate higher
evaluation ratings; statistically significant odds ratios less than one indicate lower evaluation
ratings. In this appendix, the Evaluation Team presents results for the evaluation sample and for
(1) the complete sample of teachers served by the NC NTSP and in the comparison groups, and
(2) an amended evaluation sample that includes NC NTSP teachers who worked in non-RttT
schools—still excluding TFA corps members and those entering the NC NTSP in January or
later.
20
Cohort 1 NC NTSP teachers (evaluation sample) include first- and second-year teachers. Cohort 2 NC NTSP
teachers (evaluation sample) include first-, second-, and third-year teachers. 21
There are a small number of NC NTSP teachers who have zero years of experience in both the 2012-13 and 2013-
14 school years. When estimating these first-year teacher models, the Evaluation Team removed these teachers from
the NC NTSP first-year teacher sample in 2013-14.
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 83
Table H1 (following page) displays overall and regional NC NTSP evaluation rating results. The
top panel of Table H1 indicates that there are no statistically significant differences in evaluation
ratings between NC NTSP teachers and teachers in either comparison group. Despite the
presence of a common, statewide teacher evaluation rubric, examining these evaluation rating
results by NC NTSP region must be done with caution since evaluation ratings may differ across
distinct regions of the state due to factors unassociated with the NC NTSP. Nonetheless, results
in 2013-14 and with two years of pooled data show that NC NTSP teachers in the UNCC region
have significantly higher evaluation ratings than their Non-RttT Comparison sample peers—for
Standard 1 (Demonstrating Leadership), Standard 4 (Facilitating Student Learning), and
Standard 5 (Reflecting on Practice)—and significantly higher evaluation ratings than their NC
NTSP Eligible Comparison sample peers with two years of pooled data—for Standard 2
(Establishing a Respectful Classroom Environment), Standard 4 (Facilitating Student Learning),
and Standard 5 (Reflecting on Practice). NC NTSP teachers in the remaining regions—ECU,
UNC-CSLD, and UNCG—are generally rated no differently than their comparison sample peers.
To determine whether these evaluation rating results differ by NC NTSP cohort, Table H2
(second page following) presents separate evaluation rating results for Cohort 1 (first served by
the NC NTSP in 2012-13) and Cohort 2 (first served by the NC NTSP in 2013-14) NC NTSP
teachers. In reference to Non-RttT Comparison sample teachers, the top panel of Table H2
indicates that in the 2013-14 school year NC NTSP Cohort 1 teachers had significantly higher
evaluation ratings for Standard 1, Standard 3, Standard 4, and Standard 5; NC NTSP Cohort 2
teachers had significantly lower evaluation ratings for Standard 2 in 2013-14 and with two years
of pooled data. The bottom panel of Table H2 shows that there were no statistically significant
differences in evaluation ratings between NC NTSP Eligible Comparison sample teachers and
NC NTSP teachers in either cohort.
Finally, Table H3 (second page following) presents evaluation rating results for first-year
teachers only. The odds ratios indicate that first-year NC NTSP teachers are generally rated no
differently than their first-year comparison sample peers—only one result, comparing NC NTSP
and NC NTSP Eligible Comparison sample teachers for Standard 5 in 2012-13 is statistically
significant. Across models, results in Tables H1-H3 suggest that, outside of NC NTSP teachers
in the UNCC region and Cohort 1 NC NTSP teachers in 2013-14, there was generally no
difference in evaluation ratings between NC NTSP and comparison sample teachers.
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 84
Table H1. Overall and Regional NC NTSP Evaluation Rating Results (Evaluation Sample)
Group
2012-13 School Year 2013-14 School Year 2012-13 & 2013-14 School Years
S1 S2 S3 S4 S5 S1 S2 S3 S4 S5 S1 S2 S3 S4 S5
NC NTSP vs.
Non-RttT
Comparison
1.077 1.114 0.998 0.814 0.981 1.097 0.902 1.006 1.079 1.036 1.072 0.960 0.993 0.975 0.996
NC NTSP vs
NC NTSP
Eligible
Comparison
1.267 1.381 1.275 1.096 1.384 0.785 0.767 0.842 0.965 0.811 1.023 1.064 1.033 1.001 1.107
NC NTSP Regions vs Non-RttT Comparison
ECU 0.755 1.372 0.413 0.485+ 0.900 1.018 0.828 0.879 0.820 1.022 0.878 0.921 0.666 0.655 0.941
UNC-CSLD 1.188 0.793 1.408 0.995 1.097 0.809 0.776 0.866 1.018 0.934 0.834 0.719 0.906 0.896 0.886
UNCC 1.563 1.388 1.450 1.169 1.386 1.972+ 1.650 1.713 1.849
+ 1.947
+ 1.822
+ 1.601 1.613 1.588 1.735+
UNCG 0.854 0.878 1.068 0.811 0.658 0.950 0.697 0.820 0.933 0.672 0.928 0.771 0.931 0.913 0.675
NC NTSP Regions vs NC NTSP Eligible Comparison ECU 0.989 2.017 0.598 0.662 1.530 0.809 0.783 0.802 0.776 0.881 0.891 1.085 0.736 0.713 1.118
UNC-CSLD 1.557 1.166 2.037+ 1.357 1.864 0.643 0.734 0.789 0.964 0.805 0.846 0.847 1.002 0.975 1.052
UNCC 2.048 2.040 2.099 1.595 2.354+ 1.568 1.561 1.562 1.750 1.678 1.848 1.886
+ 1.783 1.728+
2.061*
UNCG 1.119 1.290 1.546 1.108 1.119 0.755 0.659 0.748 0.883 0.579 0.941 0.908 1.029 0.994 0.803
Note: The top panel of this table displays odds ratios for earning higher evaluation ratings for NC NTSP teachers in reference to Non-RttT Comparison and NC
NTSP Eligible Comparison sample teachers. The second panel of this table displays odds ratios for each NC NTSP region in comparison to the Non-RttT
Comparison group; the bottom panel of this table displays odds ratios for each NC NTSP region in comparison to the NC NTSP Eligible Comparison group.
Values above 1 indicate greater odds of a higher evaluation rating; values below 1 indicate reduced odds of a higher evaluation rating. S1=Leadership;
S2=Classroom Environment; S3=Content Knowledge; S4=Facilitating Student Learning; and S5=Reflecting on Practice. +, *, and ** indicate statistically
significant differences at the 0.10, 0.05, and 0.01 levels, respectively.
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 85
Table H2. Cohort NC NTSP Evaluation Rating Results (Evaluation Sample)
Cohort
2012-13 School Year 2013-14 School Year 2012-13 and 2013-14 School Years
S1 S2 S3 S4 S5 S1 S2 S3 S4 S5 S1 S2 S3 S4 S5
NC NTSP Cohorts vs Non-RttT Comparison
NC NTSP
Cohort 1 1.082 1.128 0.997 0.842 0.980 1.722
* 1.388 1.709+
1.844*
1.572+ 1.376 1.267 1.323 1.203 1.257
NC NTSP
Cohort 2 --- --- --- --- --- 0.845 0.700
+ 0.748 0.811 0.801 0.779 0.658* 0.694 0.752 0.730
NC NTSP Cohorts vs. NC NTSP Eligible Comparison
NC NTSP
Cohort 1 1.364 1.615 1.378 1.106 1.621 1.270 1.228 1.448 1.628 1.265 1.294 1.391 1.359 1.220 1.399
NC NTSP
Cohort 2 --- --- --- --- --- 0.623 0.619 0.634 0.715 0.645 0.732 0.722 0.713 0.763 0.812
Note: The top panel of this table displays odds ratios for earning higher evaluation ratings for Cohort 1 and Cohort 2 NC NTSP teachers in comparison to the
Non-RttT Comparison group; the bottom panel of this table displays odds ratios for earning higher evaluation ratings for Cohort 1 and Cohort 2 NC NTSP
teachers in comparison to the NC NTSP Eligible Comparison group. Values above 1 indicate greater odds of a higher evaluation rating; values below 1 indicate
reduced odds of a higher evaluation rating. S1=Leadership; S2=Classroom Environment; S3=Content Knowledge; S4=Facilitating Student Learning; and
S5=Reflecting on Practice. +, *, and ** indicate statistically significant differences at the 0.10, 0.05, and 0.01 levels, respectively.
Table H3. First-Year NC NTSP Evaluation Rating Results (Evaluation Sample)
Group
2012-13 School Year 2013-14 School Year 2012-13 & 2013-14 School Years
S1 S2 S3 S4 S5 S1 S2 S3 S4 S5 S1 S2 S3 S4 S5
BT1 NC
NTSP vs
Non-RttT
Comparison
1.175 1.149 1.121 0.961 1.108 0.840 0.892 1.006 1.029 0.823 1.044 1.054 1.070 1.023 0.948
BT1 NC
NTSP vs NC
NTSP
Eligible
Comparison
1.338 1.356 1.111 0.830 1.455+ 0.832 0.494 0.576 0.697 0.925 1.166 1.067 0.902 0.753 1.213
Note: This table displays odds ratios for earning higher evaluation ratings for first-year NC NTSP teachers in reference to first-year Non-RttT Comparison and
NC NTSP Eligible Comparison sample teachers. Values above 1 indicate greater odds of a higher evaluation rating; values below 1 indicate reduced odds of a
higher evaluation rating. S1=Leadership; S2=Classroom Environment; S3=Content Knowledge; S4=Facilitating Student Learning; and S5=Reflecting on
Practice. +, *, and ** indicate statistically significant differences at the 0.10, 0.05, and 0.01 levels, respectively.
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 86
Teacher Evaluation Rating Results: Complete and Amended Evaluation Samples
Table H4. Overall and Regional NC NTSP Evaluation Rating Results (Complete Sample)
2012-13 School Year 2013-14 School Year 2012-13 & 2013-14 School Years
S1 S2 S3 S4 S5 S1 S2 S3 S4 S5 S1 S2 S3 S4 S5
NC NTSP vs.
Non-RttT
Comparison
0.835 0.920 0.965 0.849 0.821 1.071 1.067 1.171 1.133 1.031 0.988 1.011 1.091 1.024 0.945
NC NTSP vs.
NC NTSP
Eligible
Comparison
1.054 1.208 1.192 1.088 1.285 0.823 0.922 1.044 0.980 0.875 0.941 1.061 1.096 1.002 1.087
NC NTSP Regions vs Non-RttT Comparison
ECU 0.534+ 0.886 0.416
+ 0.483
* 0.733 0.917 0.763 0.825 0.713 0.823 0.751 0.790 0.623 0.594* 0.766
UNC-CSLD 0.927 0.766 1.672+ 1.157 1.101 0.858 0.931 0.941 0.988 0.967 0.836 0.835 1.094 0.974 0.963
UNCC 1.390 1.444 1.368 1.164 1.263 1.687 2.147*
2.489**
2.238**
1.765* 1.593 1.922
* 2.014
* 1.824
* 1.559
+
UNCG 0.778 0.764 0.973 0.889 0.578* 0.955 0.801 0.967 0.996 0.750 0.884 0.782 0.975 0.943 0.677
+
NC NTSP Regions vs NC NTSP Eligible Comparison
ECU 0.691 1.262 0.544 0.626 1.233 0.766 0.714 0.786 0.660 0.750 0.762 0.893 0.669 0.622+ 0.929
UNC-CSLD 1.200 1.090 2.188* 1.500 1.853
* 0.717 0.870 0.896 0.914 0.882 0.849 0.945 1.174 1.020 1.167
UNCC 1.799 2.056 1.790 1.509 2.125 1.410 2.007+
2.370*
2.070+ 1.609 1.616 2.174
* 2.162
* 1.910
* 1.889
*
UNCG 1.008 1.087 1.274 1.152 0.972 0.798 0.749 0.920 0.921 0.684 0.897 0.884 1.046 0.988 0.820
Note: The top panel of this table displays odds ratios for earning higher evaluation ratings for NC NTSP teachers in reference to Non-RttT Comparison and NC
NTSP Eligible Comparison sample teachers. The second panel of this table displays odds ratios for each NC NTSP region in comparison to the Non-RttT
Comparison group; the bottom panel of this table displays odds ratios for each NC NTSP region in comparison to the NC NTSP Eligible Comparison group.
Values above 1 indicate greater odds of a higher evaluation rating; values below 1 indicate reduced odds of a higher evaluation rating. S1=Leadership;
S2=Classroom Environment; S3=Content Knowledge; S4=Facilitating Student Learning; and S5=Reflecting on Practice. +, *, and ** indicate statistically
significant differences at the 0.10, 0.05, and 0.01 levels, respectively.
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 87
Table H5. Overall and Regional NC NTSP Evaluation Rating Results (Amended Evaluation Sample)
Group
2012-13 School Year 2013-14 School Year 2012-13 & 2013-14 School Years
S1 S2 S3 S4 S5 S1 S2 S3 S4 S5 S1 S2 S3 S4 S5
NC NTSP vs.
Non-RttT
Comparison
1.011 1.031 0.965 0.783 0.927 1.099 1.073 1.218 1.163 1.079 1.047 1.035 1.135 1.031 1.012
NC NTSP vs.
NC NTSP
Eligible
Comparison
1.225 1.336 1.266 1.081 1.356 0.831 0.967 1.096 1.075 0.890 1.002 1.125 1.161 1.048 1.119
NC NTSP Regions vs Non-RttT Comparison
ECU 0.745 1.338 0.402 0.482+ 0.893 0.901 0.738 0.830 0.734 0.862 0.805 0.819 0.643 0.606 0.823
UNC-CSLD 1.066 0.732 1.363 0.929 1.023 0.850 0.891 0.977 0.997 0.949 0.818 0.777 0.973 0.869 0.887
UNCC 1.563 1.396 1.459 1.177 1.372 1.750 2.187*
2.752**
2.332**
1.895* 1.694 1.936
* 2.250
** 1.876
** 1.714
*
UNCG 0.763 0.741 0.965 0.774 0.630 1.033 0.848 1.026 1.041 0.815 0.939 0.818 1.032 0.951 0.751
NC NTSP Regions vs NC NTSP Eligible Comparison
ECU 0.989 2.010 0.594 0.665 1.523 0.740 0.721 0.796 0.727 0.764 0.819 0.974 0.722 0.668 0.980
UNC-CSLD 1.416 1.100 2.011+ 1.281 1.745 0.698 0.871 0.936 0.987 0.840 0.833 0.923 1.092 0.958 1.057
UNCC 2.076 2.096 2.153+ 1.623 2.340
+ 1.437 2.137+
2.639*
2.308* 1.679 1.726 2.301
* 2.525
** 2.067
* 2.042
*
UNCG 1.013 1.113 1.424 1.067 1.074 0.848 0.828 0.984 1.030 0.722 0.957 0.971 1.158 1.048 0.895
Note: The top panel of this table displays odds ratios for earning higher evaluation ratings for NC NTSP teachers in reference to Non-RttT Comparison and NC
NTSP Eligible Comparison sample teachers. The second panel of this table displays odds ratios for each NC NTSP region in comparison to the Non-RttT
Comparison group; the bottom panel of this table displays odds ratios for each NC NTSP region in comparison to the NC NTSP Eligible Comparison group. Values
above 1 indicate greater odds of a higher evaluation rating; values below 1 indicate reduced odds of a higher evaluation rating. S1=Leadership; S2=Classroom
Environment; S3=Content Knowledge; S4=Facilitating Student Learning; and S5=Reflecting on Practice. +, *, and ** indicate statistically significant differences at
the 0.10, 0.05, and 0.01 levels, respectively.
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 88
Table H6. Cohort NC NTSP Evaluation Rating Results (Complete Sample)
Cohort
2012-13 School Year 2013-14 School Year 2012-13 and 2013-14 School Years
S1 S2 S3 S4 S5 S1 S2 S3 S4 S5 S1 S2 S3 S4 S5
NC NTSP Cohorts vs Non-RttT Comparison
NC NTSP
Cohort 1 0.858 0.925 0.965 0.868 0.847 1.464 1.322 1.570
+ 1.652
* 1.413 1.112 1.106 1.224 1.125 1.084
NC NTSP
Cohort 2 --- --- --- --- --- 0.927 0.968 1.071 0.985 0.885 0.850 0.899 0.966 0.913 0.798
NC NTSP Cohorts vs. NC NTSP Eligible Comparison
NC NTSP
Cohort 1 1.056 1.261 1.237 1.096 1.398 1.151 1.142 1.356 1.398 1.213 1.056 1.160 1.223 1.094 1.243
NC NTSP
Cohort 2 --- --- --- --- --- 0.729 0.837 0.925 0.834 0.760 0.807 0.942 0.965 0.889 0.916
Note: The top panel of this table displays odds ratios for earning higher evaluation ratings for Cohort 1 and Cohort 2 NC NTSP teachers in comparison to the
Non-RttT Comparison group; the bottom panel of this table displays odds ratios for earning higher evaluation ratings for Cohort 1 and Cohort 2 NC NTSP
teachers in comparison to the NC NTSP Eligible Comparison group. Values above 1 indicate greater odds of a higher evaluation rating; values below 1 indicate
reduced odds of a higher evaluation rating. S1=Leadership; S2=Classroom Environment; S3=Content Knowledge; S4=Facilitating Student Learning; and
S5=Reflecting on Practice. +, *, and ** indicate statistically significant differences at the 0.10, 0.05, and 0.01 levels, respectively.
Table H7. Cohort NC NTSP Evaluation Rating Results (Amended Evaluation Sample)
Cohort
2012-13 School Year 2013-14 School Year 2012-13 & 2013-14 School Years
S1 S2 S3 S4 S5 S1 S2 S3 S4 S5 S1 S2 S3 S4 S5
NC NTSP Cohorts vs Non-RttT Comparison
NC NTSP
Cohort 1 1.010 1.030 0.965 0.821 0.931 1.493 1.311 1.601
+ 1.713
* 1.477 1.226 1.170 1.286 1.144 1.197
NC NTSP
Cohort 2 --- --- --- --- --- 0.959 0.981 1.154 1.017 0.934 0.885 0.905 1.033 0.925 0.842
NC NTSP Cohorts vs. NC NTSP Eligible Comparison
NC NTSP
Cohort 1 1.289 1.505 1.361 1.087 1.550 1.151 1.176 1.382 1.548 1.223 1.164 1.284 1.320 1.159 1.333
NC NTSP
Cohort 2 --- --- --- --- --- 0.739 0.879 0.996 0.919 0.773 0.839 0.992 1.060 0.937 0.938
Note: The top panel of this table displays odds ratios for earning higher evaluation ratings for Cohort 1 and Cohort 2 NC NTSP teachers in comparison to the
Non-RttT Comparison group; the bottom panel of this table displays odds ratios for earning higher evaluation ratings for Cohort 1 and Cohort 2 NC NTSP
teachers in comparison to the NC NTSP Eligible Comparison group. Values above 1 indicate greater odds of a higher evaluation rating; values below 1 indicate
reduced odds of a higher evaluation rating. S1=Leadership; S2=Classroom Environment; S3=Content Knowledge; S4=Facilitating Student Learning; and
S5=Reflecting on Practice. +, *, and ** indicate statistically significant differences at the 0.10, 0.05, and 0.01 levels, respectively.
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 89
Table H8. First-Year Teacher NC NTSP Evaluation Rating Results (Complete Sample)
Group
2012-13 School Year 2013-14 School Year 2012-13 & 2013-14 School Years
S1 S2 S3 S4 S5 S1 S2 S3 S4 S5 S1 S2 S3 S4 S5
NC NTSP vs.
Non-RttT
Comparison
0.889 0.937 0.882 0.940 0.813 0.813 0.994 1.155 1.133 0.811 0.866 0.962 1.046 1.049 0.798
NC NTSP vs.
NC NTSP
Eligible
Comparison
1.118 1.167 0.886 0.745 1.328 0.901 0.622 0.826 0.739 1.011 1.021 0.938 0.875 0.724 1.168
Note: This table displays odds ratios for earning higher evaluation ratings for first-year NC NTSP teachers in reference to first-year Non-RttT Comparison and NC
NTSP Eligible Comparison sample teachers. Values above 1 indicate greater odds of a higher evaluation rating; values below 1 indicate reduced odds of a higher
evaluation rating. S1=Leadership; S2=Classroom Environment; S3=Content Knowledge; S4=Facilitating Student Learning; and S5=Reflecting on Practice. +, *,
and ** indicate statistically significant differences at the 0.10, 0.05, and 0.01 levels, respectively.
Table H9. First-Year Teacher NC NTSP Evaluation Rating Results (Amended Evaluation Sample)
Group
2012-13 School Year 2013-14 School Year 2012-13 & 2013-14 School Years
S1 S2 S3 S4 S5 S1 S2 S3 S4 S5 S1 S2 S3 S4 S5
NC NTSP vs.
Non-RttT
Comparison
1.105 1.066 1.035 0.948 1.030 0.828 1.031 1.201 1.142 0.924 0.955 1.027 1.138 1.053 0.942
NC NTSP vs.
NC NTSP
Eligible
Comparison
1.303 1.315 1.066 0.830 1.412 0.905 0.717 0.916 0.897 1.052 1.109 1.047 0.995 0.810 1.213
Note: This table displays odds ratios for earning higher evaluation ratings for first-year NC NTSP teachers in reference to first-year Non-RttT Comparison and NC
NTSP Eligible Comparison sample teachers. Values above 1 indicate greater odds of a higher evaluation rating; values below 1 indicate reduced odds of a higher
evaluation rating. S1=Leadership; S2=Classroom Environment; S3=Content Knowledge; S4=Facilitating Student Learning; and S5=Reflecting on Practice. +, *,
and ** indicate statistically significant differences at the 0.10, 0.05, and 0.01 levels, respectively.
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 90
Appendix I. To What Extent does the NC NTSP Impact the Retention of Novice Teachers
to the Same School, LEA, and the State?
For analyses of teacher retention, the Evaluation Team presents four sets of results: (1) overall
NC NTSP results for the 2012-13 school year, the 2013-14 school year, and the two years
combined; (2) regional NC NTSP results (separate results for teachers in the ECU, UNC-CSLD,
UNCC, and UNCG regions) for the 2012-13 school year, the 2013-14 school year, and the two
years combined; (3) cohort NC NTSP results (separate results for Cohort 1 teachers—entering
the program in the 2012-13 school year—and Cohort 2 teachers—entering the program in 2013-
14) for the 2012-13 school year, the 2013-14 school year, and the two years combined;22
and (4)
first-year teacher NC NTSP results, which focus on first-year teachers in the 2012-13 school
year, the 2013-14 school year, and the two years combined.23
Taken together, this set of analyses
presents evidence regarding the overall effectiveness of the program and how that effectiveness
may have differed by NC NTSP region, NC NTSP cohort, and for first-year teachers. These sub-
analyses are particularly important given the differences in NC NTSP treatment components
across regions and cohorts and the hypothesis of stronger program effects for first-year teachers.
To determine whether NC NTSP evaluation sample teachers were more likely to remain in
teaching than their Non-RttT Comparison and NC NTSP Eligible Comparison sample peers, the
Evaluation Team estimated the probability that NC NTSP teachers and teachers from each of the
comparison groups would return to teaching in the state. Specifically, the Evaluation Team
estimated models for three types of retention: (1) returning to any North Carolina public school
in the following school year (2013-14 and 2014-15); (2) returning to the same LEA in the
following school year; and (3) returning to the same school in the following school year. For
these analyses the Evaluation Team estimated logistic regression models, where the outcome
variable was a ‘1’ if the teacher returned in the following school year and a ‘0’ if the teacher did
not. In these models the Evaluation Team controlled for teacher experience and school
characteristics, to help isolate the impact of the NC NTSP on teacher retention, and clustered
standard errors at the school level to account for dependence in the data. Post-estimation, the
Evaluation Team converted the odds ratios to predicted retention probabilities to facilitate easier
interpretation of the results. In this appendix, the Evaluation Team displays results for the
evaluation sample and for (1) the complete sample of teachers served by the NC NTSP and in the
comparison groups, and (2) an amended evaluation sample that includes NC NTSP teachers who
worked in non-RttT schools—still excluding TFA corps members and those entering the NC
NTSP in January or later.
Table I1 (second page following) presents overall and regional NC NTSP retention results.
Pooling data over both evaluation years—2013-14 and 2014-15—the top panel of Table I1
shows that NC NTSP teachers were significantly more likely to return to NCPS than their Non-
RttT Comparison sample peers. Regarding LEA retention, NC NTSP teachers were significantly
more likely to return to the same LEA than NC NTSP Eligible Comparison teachers in 2013-14
22
Cohort 1 NC NTSP teachers (evaluation sample) include first- and second-year teachers. Cohort 2 NC NTSP
teachers (evaluation sample) include first-, second-, and third-year teachers. 23
There are a small number of NC NTSP teachers who have zero years of experience in both the 2012-13 and 2013-
14 school years. When estimating these first-year teacher models, the Evaluation Team removed these teachers from
the NC NTSP first-year teacher sample in 2013-14.
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 91
and Non-RttT Comparison sample teachers with two years of pooled data. The strongest NC
NTSP retention results concerned returning to the same low-performing school—NC NTSP
teachers were significantly more likely to return to the same school than Non-RttT Comparison
sample teachers in 2013-14, 2014-15, and in both years, combined, and more likely to return
than NC NTSP Eligible Comparison sample teachers with two years of pooled data. Examining
these results by NC NTSP region reduces statistical power to detect effects and must be done
with caution since rates of teacher persistence differ across the distinct regions of the state due to
factors unassociated with the NC NTSP. Nonetheless, results show the strongest retention
results, particularly school-level retention, for NC NTSP teachers in the ECU region. NC NTSP
teachers from the UNC-CSLD, UNCC, and UNCG regions also have significantly higher school
retention rates than Non-RttT Comparison sample teachers.
To understand how these retention results may differ according to the year of NC NTSP entry,
Table I2 (second page following) presents separate retention results for Cohort 1 (first served by
the NC NTSP in 2012-13) and Cohort 2 (first served by the NC NTSP in 2013-14) NC NTSP
teachers. The top panel of Table I2 indicates that Cohort 1 teachers were significantly more
likely than their Non-RttT Comparison sample peers to return to NCPS, the same LEA, and the
same school. Significant retention results for Cohort 2 were limited to school-level retention. The
bottom panel of Table I2 displays fewer significant differences between NC NTSP teachers by
cohort and NC NTSP Eligible Comparison sample teachers. Cohort 1 NC NTSP teachers were
more likely to return to the same LEA in the 2013-14 school year and the same low-performing
school in 2014-15 and with two years of pooled data; Cohort 2 NC NTSP teachers were more
likely to return to the same low-performing school with two years of pooled data.
Finally, Table I3 (second page following) displays retention results for first-year teachers only.
These results are significant for first-year NC NTSP teachers (in 2012-13) returning in the 2013-
14 school year—NC NTSP first-year teachers were more likely than Non-RttT Comparison
sample teachers to return to NCPS and more likely than both comparison groups to return to the
same LEA and the same school—but there are no significant differences for first-year NC NTSP
teachers (in 2013-14) returning in 2014-15. Coupled with the values in Tables I1 and I2, results
suggest that the NC NTSP had its strongest retention impacts on teachers returning to the same
low-performing school and for Cohort 1 NC NTSP teachers.
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 92
Table I1. Overall and Regional NC NTSP Retention Results (Evaluation Sample)
Group
Returns to NCPS Returns to the Same LEA Returns to the Same School
Returns in
2013-14
Returns in
2014-15
Overall:
2013-14 &
2014-15
Returns in
2013-14
Returns in
2014-15
Overall:
2013-14 &
2014-15
Returns in
2013-14
Returns in
2014-15
Overall:
2013-14 &
2014-15
NC NTSP 87.66 84.99 85.74
79.61 76.67 77.47
74.11
72.72
73.04
Non-RttT
Comparison 84.36 82.53 83.09
+ 74.37 73.94 73.98+
67.14+
64.95**
65.75**
NC NTSP
Eligible
Comparison
86.11 87.65 86.76 72.11+ 76.81 74.49 65.66 66.44 65.99
*
NC NTSP Region Results vs. Non-RttT Comparison Group
ECU Region 90.14 86.31 87.62+ 83.80 79.26 80.49
+ 81.39
* 75.25
* 77.26
**
UNC-CSLD
Region 86.16 83.70 83.88 72.71 74.49 73.26 66.21 73.15
* 70.22
UNCC Region 86.15 84.02 84.85 77.84 74.66 76.32 72.67 71.71 72.66+
UNCG Region 88.67 86.05 86.99 83.29 78.39 79.91* 74.89 70.80 72.35
+
Non-RttT
Comparison 84.39 82.54 83.11 74.51 73.91 74.07 67.31 64.91 65.78
NC NTSP Region Results vs. NC NTSP Eligible Comparison Group
ECU Region 90.14 86.31 87.62 83.80+ 79.26 80.49 81.39
* 75.25
+ 77.26
**
UNC-CSLD
Region 86.16 83.70 83.88 72.71 74.49 73.26 66.21 73.15 70.22
UNCC Region 86.15 84.02 84.85 77.84 74.66 76.32 72.67 71.71 72.66
UNCG Region 88.67 86.05 86.99 83.29 78.39 79.91+ 74.89 70.80 72.35
NC NTSP
Eligible
Comparison
86.24 87.67 86.77 72.19 77.12 74.51 65.81 66.61 66.02
Note: The top panel of this table displays predicted probabilities of retention for NC NTSP teachers and teachers in each comparison group. The second panel of
this table displays results for each NC NTSP region in comparison to the Non-RttT Comparison group; the bottom panel of this table displays results for each NC
NTSP region in comparison to the NC NTSP Eligible Comparison group. +, *, and ** indicate statistically significant differences at the 0.10, 0.05, and 0.01
levels, respectively.
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 93
Table I2. Cohort NC NTSP Retention Results (Evaluation Sample)
Group
Returns to NCPS Returns to the Same LEA Returns to the Same School
Returns in
2013-14
Returns in
2014-15
Overall:
2013-14 &
2014-15
Returns in
2013-14
Returns in
2014-15
Overall:
2013-14 &
2014-15
Returns in
2013-14
Returns in
2014-15
Overall:
2013-14 &
2014-15
NC NTSP
Cohort 1 87.66 87.87
** 86.87
+ 79.61 79.59+
79.08+
74.11+
75.76**
74.18**
NC NTSP
Cohort 2 --- 83.54 84.45 --- 75.28 75.61 --- 71.12
* 71.67
*
Non-RttT
Comparison 84.36 82.48 83.07 74.37 73.82 73.95 67.14 64.92 65.74
NC NTSP
Cohort 1 87.66 87.87 86.87 79.61
+ 79.59 79.08 74.11 75.76+
74.18*
NC NTSP
Cohort 2 --- 83.54 84.45 --- 75.28 75.61 --- 71.12 71.67
+
NC NTSP
Eligible
Comparison
86.11 87.65 86.80 72.11 76.97 74.68 65.66 66.47 66.05
Note: The top panel of this table displays predicted probabilities of retention for Cohort 1 and Cohort 2 NC NTSP teachers in comparison to the Non-RttT
Comparison group; the bottom panel of this table displays results for Cohort 1 and Cohort 2 NC NTSP teachers in comparison to the NC NTSP Eligible
Comparison group. +, *, and ** indicate statistically significant differences at the 0.10, 0.05, and 0.01 levels, respectively.
Table I3. First-Year Teacher NC NTSP Retention Results (Evaluation Sample)
Group
Returns to NCPS Returns to the Same LEA Returns to the Same School
Returns in
2013-14
Returns in
2014-15
Overall:
2013-14 &
2014-15
Returns in
2013-14
Returns in
2014-15
Overall:
2013-14 &
2014-15
Returns in
2013-14
Returns in
2014-15
Overall:
2013-14 &
2014-15
NC NTSP BT1 87.29 81.79 84.04 80.22 72.82 75.70 77.42 67.99 71.72
Non-RttT
Comparison 81.03
* 81.81 81.18 71.35* 74.21 72.42 65.43
** 66.37 65.58*
NC NTSP
Eligible
Comparison
83.51 86.46 83.87 68.62* 74.21 70.11 61.04
** 67.47 63.11*
Note: This table displays predicted probabilities of retention for first-year NC NTSP teachers and first-year teachers in the Non-RttT Comparison and NC NTSP
Eligible Comparison groups. +, *, and ** indicate statistically significant differences at the 0.10, 0.05, and 0.01 levels, respectively.
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 94
Teacher Retention Results: Complete and Amended Evaluation Samples
Table I4. Overall and Regional NC NTSP Retention Results (Complete Sample)
Group
Returns to NCPS Returns to the Same LEA Returns to the Same School
Returns in
2013-14
Returns in
2014-15
Overall:
2013-14 &
2014-15
Returns in
2013-14
Returns in
2014-15
Overall:
2013-14 &
2014-15
Returns in
2013-14
Returns in
2014-15
Overall:
2013-14 &
2014-15
NC NTSP 85.46 84.21 84.60
77.27 75.52 76.04
69.66 71.88
71.24
Non-RttT
Comparison 82.85 81.23 81.72
+ 72.56 72.81 72.63+ 65.65 62.71
** 63.81
**
NC NTSP
Eligible
Comparison
83.41 88.03 85.72 68.87* 77.23 72.93 62.41 63.61
* 62.99
**
NC NTSP Region Results vs. Non-RttT Comparison Group
ECU Region 85.67 83.33 84.64 76.56 73.37 74.85 67.53 69.49+
69.37+
UNC-CSLD
Region 81.71 82.73 82.01 70.90 74.16 72.72 64.72 72.61
** 69.74
+
UNCC Region 86.02 83.91 84.74 77.56 76.32 76.89 69.71 74.08*
73.14**
UNCG Region 88.17 86.97*
87.26*
83.37*
78.28+
79.91**
75.65*
70.92+
72.59**
Non-RttT
Comparison 82.95 81.31 81.81 72.71 72.86 72.72 65.81 62.71 63.88
NC NTSP Region Results vs. NC NTSP Eligible Comparison Group
ECU Region 85.67 83.33 84.64 76.56 73.37 74.85 67.53 69.49 69.37+
UNC-CSLD
Region 81.71 82.73 82.01 70.90 74.16 72.72 64.72 72.61
+ 69.74
+
UNCC Region 86.02 83.91 84.74 77.56+ 76.32 76.89 69.71 74.08
+ 73.14
*
UNCG Region 88.17 86.97
87.26 83.37** 78.28
79.91
* 75.65
* 70.92 72.59*
NC NTSP
Eligible
Comparison
83.14 87.94 85.53 68.50 77.06 72.66 62.11 63.39 62.72
Note: The top panel of this table displays predicted probabilities of retention for NC NTSP teachers and teachers in each comparison group. The second panel of
this table displays results for each NC NTSP region in comparison to the Non-RttT Comparison group; the bottom panel of this table displays results for each NC
NTSP region in comparison to the NC NTSP Eligible Comparison group. +, *, and ** indicate statistically significant differences at the 0.10, 0.05, and 0.01
levels, respectively.
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 95
Table I5. Overall and Regional NC NTSP Retention Results (Amended Evaluation Sample)
Group
Returns to NCPS Returns to the Same LEA Returns to the Same School
Returns in
2013-14
Returns in
2014-15
Overall:
2013-14 &
2014-15
Returns in
2013-14
Returns in
2014-15
Overall:
2013-14 &
2014-15
Returns in
2013-14
Returns in
2014-15
Overall:
2013-14 &
2014-15
NC NTSP 86.95 84.99 85.48
78.71 76.46 77.08
72.82 72.93
72.97
Non-RttT
Comparison 84.21 82.37 82.91
+ 74.23 73.74 73.75+ 67.12 64.83
** 65.57
**
NC NTSP
Eligible
Comparison
86.02 87.55 86.78 72.09 76.57 74.33 65.62 66.43+
66.02*
NC NTSP Region Results vs. Non-RttT Comparison Group
ECU Region 90.04 85.09 86.80 83.73 76.20 78.29 81.25*
72.64*
75.16*
UNC-CSLD
Region 84.03 84.35 83.70 70.36 74.75 73.05 63.70 73.52
** 70.25
UNCC Region 85.78 84.05 84.75 77.30 76.60 77.39 72.14 73.58+
74.01*
UNCG Region 88.01 86.55 87.05+ 81.95 78.49 79.63
* 72.84 71.89 72.67*
Non-RttT
Comparison 84.28 82.39 82.95 74.41 73.75 73.87 67.31 64.82 65.62
NC NTSP Region Results vs. NC NTSP Eligible Comparison Group
ECU Region 90.04 85.09 86.80 83.73+ 76.20 78.29 81.25
* 72.64 75.16*
UNC-CSLD
Region 84.03 84.35 83.70 70.36 74.75 73.05 63.70 73.52
+ 70.25
UNCC Region 85.78 84.05 84.75 77.30 76.60 77.39 72.14 73.58 74.01+
UNCG Region 88.01 86.55 87.05 81.95 78.49 79.63+ 72.84 71.89 72.67
NC NTSP
Eligible
Comparison
86.07 87.56 86.75 72.06 76.58 74.15 65.70 66.40 65.87
Note: The top panel of this table displays predicted probabilities of retention for NC NTSP teachers and teachers in each comparison group. The second panel of
this table displays results for each NC NTSP region in comparison to the Non-RttT Comparison group; the bottom panel of this table displays results for each NC
NTSP region in comparison to the NC NTSP Eligible Comparison group. +, *, and ** indicate statistically significant differences at the 0.10, 0.05, and 0.01
levels, respectively.
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 96
Table I6. Cohort NC NTSP Retention Results (Complete Sample)
Group
Returns to NCPS Returns to the Same LEA Returns to the Same School
Returns in
2013-14
Returns in
2014-15
Overall:
2013-14 &
2014-15
Returns in
2013-14
Returns in
2014-15
Overall:
2013-14 &
2014-15
Returns in
2013-14
Returns in
2014-15
Overall:
2013-14 &
2014-15
NC NTSP
Cohort 1 85.46 86.51
+ 85.05
+ 77.27 76.45 76.59 69.66 73.63**
70.72**
NC NTSP
Cohort 2 --- 83.29 84.08 --- 75.11 75.42 --- 71.18
** 71.82
**
Non-RttT
Comparison 82.85 81.22 81.72 72.56 72.82 72.62 65.65 62.70 63.81
NC NTSP
Cohort 1 85.46 86.51 85.05 77.27
* 76.45 76.59 69.66 73.63*
70.72*
NC NTSP
Cohort 2 --- 83.29 84.08 --- 75.11 75.42 --- 71.18
+ 71.82
**
NC NTSP
Eligible
Comparison
83.41 88.03 85.74 68.87 77.21 73.01 62.41 63.63 62.95
Note: The top panel of this table displays predicted probabilities of retention for Cohort 1 and Cohort 2 NC NTSP teachers in comparison to the Non-RttT
Comparison group; the bottom panel of this table displays results for Cohort 1 and Cohort 2 NC NTSP teachers in comparison to the NC NTSP Eligible
Comparison group. +, *, and ** indicate statistically significant differences at the 0.10, 0.05, and 0.01 levels, respectively.
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 97
Table I7. Cohort NC NTSP Retention Results (Amended Evaluation Sample)
Group
Returns to NCPS Returns to the Same LEA Returns to the Same School
Returns in
2013-14
Returns in
2014-15
Overall:
2013-14 &
2014-15
Returns in
2013-14
Returns in
2014-15
Overall:
2013-14 &
2014-15
Returns in
2013-14
Returns in
2014-15
Overall:
2013-14 &
2014-15
NC NTSP
Cohort 1 86.95 87.76
* 86.30
+ 78.71 77.91 77.80 72.82 74.78*
73.11**
NC NTSP
Cohort 2 --- 83.86 84.71 --- 75.90 76.41 --- 72.15
** 72.82
**
Non-RttT
Comparison 84.21 82.32 82.90 74.23 73.66 73.75 67.12 64.82 65.57
NC NTSP
Cohort 1 86.95 87.76 86.30 78.71 77.91 77.80 72.82 74.78
+ 73.11
+
NC NTSP
Cohort 2 --- 83.86 84.71 --- 75.90 76.41 --- 72.15 72.82
*
NC NTSP
Eligible
Comparison
86.02 87.51 86.81 72.09 76.63 74.44 65.62 66.43 66.02
Note: The top panel of this table displays predicted probabilities of retention for Cohort 1 and Cohort 2 NC NTSP teachers in comparison to the Non-RttT
Comparison group; the bottom panel of this table displays results for Cohort 1 and Cohort 2 NC NTSP teachers in comparison to the NC NTSP Eligible
Comparison group. +, *, and ** indicate statistically significant differences at the 0.10, 0.05, and 0.01 levels, respectively.
Table I8. First-Year Teacher NC NTSP Retention Results (Complete Sample)
Group
Returns to NCPS Returns to the Same LEA Returns to the Same School
Returns in
2013-14
Returns in
2014-15
Overall:
2013-14 &
2014-15
Returns in
2013-14
Returns in
2014-15
Overall:
2013-14 &
2014-15
Returns in
2013-14
Returns in
2014-15
Overall:
2013-14 &
2014-15
NC NTSP BT1 86.28 81.37 83.55 78.65 71.94 74.84 72.11 67.49 69.54
Non-RttT
Comparison 81.92
+ 82.54 82.00 71.63* 74.41 72.77 65.83
+ 65.26 65.35
NC NTSP
Eligible
Comparison 80.63
+ 86.70 82.24 65.08** 75.56 68.11
+ 57.43
** 63.82 59.15**
Note: This table displays predicted probabilities of retention for first-year NC NTSP teachers and first-year teachers in the Non-RttT Comparison and NC NTSP
Eligible Comparison groups. +, *, and ** indicate statistically significant differences at the 0.10, 0.05, and 0.01 levels, respectively.
NC NTSP: Final RttT Evaluation Report
August 2015
Consortium for Educational Research and Evaluation–North Carolina 98
Table I9. First-Year Teacher NC NTSP Retention Results (Amended Evaluation Sample)
Group
Returns to NCPS Returns to the Same LEA Returns to the Same School
Returns in
2013-14
Returns in
2014-15
Overall:
2013-14 &
2014-15
Returns in
2013-14
Returns in
2014-15
Overall:
2013-14 &
2014-15
Returns in
2013-14
Returns in
2014-15
Overall:
2013-14 &
2014-15
NC NTSP BT1 87.16 82.07 84.00 79.58 72.90 75.36 76.62 68.24 71.35
Non-RttT
Comparison 80.80
* 81.79 81.07 71.11* 74.29 72.34 65.17
** 66.51 65.50*
NC NTSP
Eligible
Comparison
83.41 85.69 83.75 68.64* 73.10 69.88 60.98
** 66.84 63.07*
Note: This table displays predicted probabilities of retention for first-year NC NTSP teachers and first-year teachers in the Non-RttT Comparison and NC NTSP
Eligible Comparison groups. +, *, and ** indicate statistically significant differences at the 0.10, 0.05, and 0.01 levels, respectively.
Contact Information:
Please direct all inquiries to Kevin Bastian
© 2015 Consortium for Educational Research and Evaluation–North Carolina