Connecting Teacher Preparation to Teacher Induction

Outcomes for Beginning Teacher Induction Programs

Twenty-five years ago, the modal category of experience in the U.S. teacher workforce was 15 years. Today, the modal experience category is 1 year, and over one-quarter of the teacher workforce has less than 5 years of experience (Ingersoll & Merrill, 2012). This greening of the teacher workforce raises concerns given the relatively lower performance and retention outcomes for novice teachers (Clotfelter et al., 2007, 2010; Henry, Bastian, & Fortner, 2011; Ingersoll & Smith, 2004; Papay & Kraft, 2015) and the concentration of novice teachers in high-need schools (Bastian, Henry, & Thompson, 2013; Clotfelter et al., 2005; Lankford et al., 2002). Fortunately, research indicates that novice teachers have a tremendous capacity for on-the-job development, which policymakers may capitalize on to increase novice teacher effectiveness and retention (Henry et al., 2011; Papay & Kraft, 2015).

Comprehensive teacher induction programs are one approach to take advantage of this ability for teacher improvement. Previous studies have found positive relationships between induction programs and teacher retention (DeAngelis, Wall, & Che, 2013; Kelly, 2004; Smith & Ingersoll, 2004), the quality of teaching practices (Evertson & Smithey, 2000; Stanulis & Floden, 2009), and student achievement gains (Fletcher, Strong, & Villar, 2008; Rockoff, 2008). Furthermore, evidence suggests that strong induction programs and/or more intensive induction program participation can benefit teacher outcomes (DeAngelis et al., 2013; Fletcher et al., 2008; Fletcher & Strong, 2009; Kapadia, Coca, & Easton, 2007; Rockoff, 2008; Smith & Ingersoll, 2004). For example, Smith and Ingersoll (2004) show that having a mentor in the same subject area, combined with collaborative activities with other novice teachers, significantly lowers first-year teacher attrition rates. Rockoff (2008) finds that novice teachers spending more time with their mentors have higher student achievement gains in mathematics and reading than peers with fewer hours of mentoring. While induction programs do not always have measurable impacts (Glazerman et al., 2010), investing resources into novice teachers appears to be a promising way to increase their performance and retention (Ingersoll & Strong, 2011).

Despite these findings, there are two gaps in the teacher induction literature that we aim to address. First, it is unknown whether certain induction providers (e.g., school districts, state agencies, outside nonprofits) are more effective than others. Most relevant to the present study, there are several potential advantages for university-based induction programs, yet only a few studies have investigated the relationships between university-based induction programs and teacher outcomes. These studies returned promising findings but were small in scale, did not always include a comparison group, and did not examine impacts on student achievement (Colbert & Wolff, 1992; Davis & Higdon, 2008; Kelly, 2004; Schaffer, Stringfield, & Wolfe, 1992; Stanulis & Floden, 2009). Second, despite the critical need for high-quality beginning teacher supports in low-performing schools, we do not know outcomes for programs targeted to such environments or whether there are specific aspects of induction that may work best in these environments. The present study addresses many of these concerns by focusing on a university-based induction program for lowest-performing schools, including a large study sample, and analyzing three policy-relevant teacher outcomes (value-added, evaluation ratings, and retention).

University-Based Induction Programs: Potential Advantages and Disadvantages

Examining the performance and retention outcomes for teachers participating in university-based induction programs is particularly important because such programs may entail several advantages over other induction providers. First, given the natural connections between universities and beginning teachers, university-based induction programs may have a greater understanding of the strengths and weaknesses of beginning teachers and be able to provide more targeted supports. While this is particularly true for university-based induction programs that support their program graduates, this can also be true of programs, like the NTSP, that support teachers with all forms of preparation. Related to this, induction programs from university settings may be better able to direct research-based strategies and resources to beginning teachers (Stanulis & Floden, 2009). This may be particularly true for the NTSP given that it is supporting lowest-performing schools that may lack the fiscal and human resources for high-quality induction. ¹ Second, university-based teacher education programs are often very familiar and have partnerships with the districts and schools in their surrounding area. These partnerships—frequently borne out of student teaching and field placements—mean that university-based induction programs know the context of these schools and can offer supports to help teachers succeed and persist in their school environments (Colbert & Wolff, 1992; Smith, 2007). Furthermore, these connections between universities and districts/schools allow for a reciprocal relationship—that best practices from the field can come back to improve teacher education and university-based induction (Zeichner, 2010). Finally, university-based induction programs can provide mentors who are independent from K–12 schools and districts. This independence may help beginning teachers and mentors develop stronger relationships since the role of mentors is not to evaluate or assess but rather to support and improve the practices of beginning teachers (Desimone et al., 2014; Feiman-Nemser et al., 2000).

Despite these potential advantages, there are also two potential concerns with university-based induction programs. First, recent research questions the quality and rigor of teacher education programs (Brouwer & Korthagen, 2005; Labaree, 2004; NCTQ, 2014a, 2014b) and the performance of program graduates (Boyd et al., 2007). Here, the concern is straightforward: If teacher education programs are not doing a good job of preparing teachers, will they do a good job in providing early-career supports? Second, because university-based induction programs are not embedded in districts/schools, there are limitations to their scope of influence. Specifically, university-based induction programs cannot directly impact school culture or offer certain induction components (e.g., reduced teaching loads for first-year teachers or time to observe master teachers), and it may be challenging for them to interact with novice teachers on a daily/frequent basis (Smith, 2007). Additionally, university-based induction programs cannot mandate teacher participation in induction components—this is important since research suggests that more intensive program participation benefits teacher outcomes (Fletcher et al., 2008; Ingersoll & Strong, 2011; Rockoff, 2008).

Description of the New Teacher Support Program

As part of North Carolina’s $400 million RttT grant, faculty from UNC system Colleges of Education designed and implemented the NTSP with the following components: (a) in-person and virtual instructional coaching throughout the school year, (b) ongoing professional development sessions (six in each academic year), and (c) multiday, off-site institutes held prior to and early in the school year. These induction supports were designed to increase teacher knowledge of and competency in the Common Core State Standards, academic goal setting, backwards planning, assessment, classroom management, successful instructional strategies, reflection/data-driven decision making, and integrating into the school community. Importantly, the NTSP did not supplant existing school-based induction services or RttT programs; rather, it layered on the status quo of teacher supports in participating schools.

Unlike some other university-based induction programs that only serve graduates of their respective institutions, the NTSP provided induction supports to first-, second-, and third-year teachers, regardless of their preparation, working in the state’s lowest-performing schools—as identified by the North Carolina Department of Public Instruction (NCDPI) for the state’s RttT grant (RttT-eligible schools). ² Participation in the NTSP was not mandatory, and not all of these lowest-performing schools opted to participate. While there are concerns about this selection bias, this offers a suitable comparison group that we discuss in the analytical sample section.

Given both the geographic concentrations of these lowest-performing schools and the capacity of UNC system institutions to run the NTSP, administrators at the UNCGA selected four UNC system institutions as regional anchor sites to implement the program—East Carolina University (ECU), the UNC Center for School Leadership Development (UNC-CSLD), UNC Charlotte (UNCC), and UNC Greensboro (UNCG). These institutions include the largest teacher preparation program in North Carolina (ECU), two large institutions located in urban centers (UNCC and UNCG), and an extension of the UNCGA that administers other statewide K–12 education programs for teachers and school leaders (UNC-CSLD). Program administrators and College of Education faculty planned the NTSP during the 2010–2011 and 2011–2012 school years, piloted elements of the program in the latter half of the 2011–2012 school year (for 35 teachers), and then fully implemented the program in 2012–2013 and 2013–2014. With this rollout, we focus on outcomes for participating teachers in the 2012–2013 and 2013–2014 school years.

For the K–12 schools/teachers in their respective geographic regions, these four anchor sites provided face-to-face and virtual instructional coaching and organized six professional development sessions in each academic year. To carry out these responsibilities, the institutions recruited, hired, and trained practicing and retired master teachers to serve as full-time NTSP instructional coaches. This full-time status may be important given positive outcomes for beginning teachers supported by full-time rather than part-time mentors (Fletcher & Strong, 2009). These coaches were assigned to novice teachers in participating NTSP schools and tasked with modeling effective teaching strategies, providing resources, assisting with student behavioral issues, engaging in lesson and unit planning, analyzing student achievement data, promoting teachers’ self-efficacy, and encouraging teacher reflection. In addition, these instructional coaches, in collaboration with faculty at each regional institution, organized and implemented the NTSP professional development sessions. Lastly, program administrators at a central NTSP office, in conjunction with the instructional coaches and faculty at the regional anchor sites, planned and implemented the NTSP institutes. ³ Together, this organizational structure was intended to provide a common framework for teacher supports while allowing regions to differentiate services based on the needs of their teachers.

The empirical warrant for these program components provides a foundation for a thoughtfully planned and executed induction model. The evidence base for the inclusion of instructional coaching is clear: A range of studies show positive relationships between mentoring and beginning teacher outcomes (Fletcher & Strong, 2009; Rockoff, 2008; Smith & Ingersoll, 2004; Stanulis & Floden, 2009). Regarding professional development, research indicates that sporadic and disconnected workshops often fail to lead to changes in teachers’ beliefs, practices, and effectiveness (Wei, Darling-Hammond, Andree, Richardson, & Orphanos, 2009; Yoon, Duncan, Lee, Scarloss, & Shapley, 2007). However, high-quality professional development, characterized by its duration, coherence, active-learning opportunities, collective participation, and expert facilitation, is associated with positive teacher outcomes (Garet, Porter, Desimone, Birman, & Yoon, 2001; Penuel, Fishman, Yamaguchi, & Gallagher, 2007). The structure of the NTSP institutes and professional development—ongoing professional learning that is led by instructional coaches and faculty, connected to program coaching, and differentiated according to group needs—meets many of these guidelines.

Overall, the components of the NTSP are not new—many induction programs offer mentoring and professional development. Rather, the NTSP stands out for four reasons. First, it was designed and implemented by faculty and staff at university-based teacher education programs and full-time instructional coaches hired by those programs. Thus, the potential advantages of university-based induction may be infused into these common induction components. Second, the NTSP serves all novice teachers regardless of their preparation. This may allow the NTSP to have a broader impact than university-based models targeted at program graduates. Third, the NTSP targets induction supports to low-performing schools. This provides fiscal and human resources to schools in need and tests whether university-based induction can succeed with teachers working in more challenging environments. Finally, the NTSP is part of a statewide university system—rather than a single university or group of universities working collectively. Although the initial implementation of the NTSP was orchestrated through four universities, the structure of the UNC system—with 15 institutions across North Carolina—enhances the capacity of the NTSP to provide a cohesive yet differentiated program and to scale up and provide induction services to novice teachers across all regions of the state.

Analytical Sample

Under the requirements of North Carolina’s RttT grant, the UNC system made the NTSP available to the 108 lowest-performing schools in the state—schools identified as the lowest 5% of all schools in terms of student achievement in the 2008–2009 school year and/or high schools with graduation rates below 60% in the 2008–2009 school year. ⁴ This focal cohort of schools remained static throughout the RttT grant—schools above the lowest 5% or graduation rate threshold were not eligible for RttT resources in later years. In 2012–2013, the first full year of program implementation, the NTSP analytical sample includes all of the first- and second-year teachers who worked in schools that were eligible for RttT services and agreed to participate in the program. For the 2013–2014 school year, the NTSP expanded in size by providing induction supports to (a) a second cohort of teachers, (b) third-year teachers in both cohorts, and (c) an additional set of lowest-performing schools that had chosen not to participate in the NTSP during the 2012–2013 school year. The NTSP also continued to provide services to the novice teachers from the first year of the program. Therefore, our 2013–2014 NTSP analytical sample includes all of the first-, second-, and third-year teachers who worked in schools that were eligible for and participated in the NTSP. This NTSP analytical sample excludes schools and/or teachers who began receiving NTSP supports in the second half (after December) of each school year since we contend that NTSP results should be based on a sample of teachers who received program supports for a majority of the school year. Of the 108 eligible lowest-performing schools, 59 participated in the NTSP for the full 2012–2013 school year, ⁵ and 91 participated for the full 2013–2014 school year. Overall, our 2012–2013 NTSP analytical sample includes 377 teachers working in 59 lowest-performing schools and 16 school districts; in 2013–2014, the NTSP sample includes 846 teachers working in 91 lowest-performing schools and 25 school districts.

Isolating the relationships between NTSP participation and teacher outcomes is challenging since North Carolina implemented the program in the state’s lowest-performing schools that were also receiving other RttT supports. Most notable among these was the District and School Transformation (DST) initiative, which provided regular professional development and coaching designed to turn around the achievement of the state’s lowest-performing schools. To address this confounding, we created two different comparison groups.

For our primary comparison group, we identified lowest-performing schools that were eligible for but did not participate in the NTSP. In the 2012–2013 school year, this group, labeled NTSP-eligible comparison, consists of all the first- and second-year teachers working in these schools; in the 2013–2014 school year, this group consists of all the first-, second-, and third-year teachers working in these schools. To examine the comparability of samples just prior to NTSP rollout, Appendix Table A1 (available online) shows that in the 2011–2012 year, schools that would go on to participate in the NTSP had higher percentages of minority students, higher short-term suspension rates, and lower performance composites (percentage of state assessments passed) than schools that would be in the NTSP-eligible comparison sample. Overall, the 2012–2013 NTSP-eligible comparison sample consists of 205 teachers working in 32 lowest-performing schools and 18 school districts; in 2013–2014, the NTSP-eligible comparison sample includes 176 teachers working in 17 lowest-performing schools and 12 school districts.

For a secondary comparison group, we used data from 2011–2012—the school year immediately preceding the NTSP rollout—to identify schools in the bottom decile of performance that were not eligible for RttT services. ⁶ We focused on this more recent data (2011–2012) rather than the 2008–2009 data the state used to identify RttT-eligible schools because (a) schools participating in the NTSP were still very low-performing in 2011–2012 and (b) we wanted to identify a set of comparison schools that were low-performing in a period more proximal to the time the NTSP was being implemented. Appendix Table A1 shows that in 2011–2012, schools that would go on to participate in the NTSP had higher percentages of economically disadvantaged and minority students, lower performance composites, and greater concentrations of novice teachers than schools in this second comparison sample. In the 2012–2013 school year, this comparison group, labeled non-RttT comparison, consists of all the first- and second-year teachers working in these schools; in the 2013–2014 school year, this comparison group consists of all the first-, second-, and third-year teachers working in these schools. Overall, the 2012–2013 non-RttT comparison sample includes 1,144 teachers working in 148 low-performing schools and 48 school districts; in 2013–2014, the non-RttT comparison sample includes 1,635 teachers working in 149 low-performing schools and 48 schools districts. As with the NTSP analytical sample, we exclude comparison sample teachers—from both the NTSP-eligible and non-RttT samples—who began working in the second-half of the school year (after December).

Like the NTSP analytical sample, both of these comparison groups consist of novice teachers working in low/lowest-performing schools. ⁷ We prefer the NTSP-eligible sample since it allows for direct comparisons between NTSP teachers and other novice teachers receiving RttT supports. Essentially, we can better isolate the associations between NTSP participation and teacher outcomes with the NTSP-eligible sample. However, there are two concerns with the NTSP-eligible sample. First, it is unknown why these schools declined to participate in the NTSP and why some schools chose to enter the program in 2013–2014 (after not participating in 2012–2013). As such, these comparisons may not take in to account factors related to nonparticipation; differences in outcomes may be attributable to these unobserved school characteristics. ⁸ Second, the NTSP-eligible sample is small and provides less statistical power for detecting differences in outcomes. The non-RttT comparison group helps address this concern by providing a much larger sample for analyses. However, comparisons to this group do not allow us to isolate the impact of NTSP from other RttT programs—we compare NTSP teachers to their peers in other low-performing schools receiving business-as-usual supports—and therefore, positive results for NTSP versus non-RttT comparison sample teachers may be due to other RttT services that NTSP teachers receive. In the results section, we primarily focus on the differences in performance and persistence between NTSP and NTSP-eligible teachers. To the extent that the nonparticipation of the NTSP-eligible sample is due to factors positively (negatively) associated with novice teacher outcomes, these estimates may be a lower (upper) boundary of NTSP influence.

Table 1 displays select individual and school characteristics for the NTSP treatment sample and the NTSP-eligible and non-RttT comparison samples during our study years. Across both 2012–2013 and 2013–2014, the teacher characteristics panel shows that a significantly higher percentage of NTSP teachers are in their first year of teaching and entered the profession alternatively. ⁹ Additionally, a significantly higher percentage of NTSP teachers are a racial or ethnic minority. The school characteristics panel indicates that schools served by NTSP have significantly higher concentrations of economically disadvantaged and racial/ethnic minority students, higher short-term suspension rates, lower performance composites, and higher concentrations of novice teachers (in 2012–2013). Essentially, despite efforts to identify similar schools—schools designated as lowest-performing (NTSP-eligible) and schools in the bottom decile of performance in the year prior to NTSP rollout (non-RttT)—Table 1 shows the challenge in constructing comparison groups that are fully comparable to the lowest-performing schools served by the NTSP. For the purpose of assessing outcomes for NTSP teachers, however, we contend that the direction of these teacher- and school-level differences may mask rather than inflate program results. That is, the NTSP sample has a greater prevalence of characteristics associated with negative teacher outcomes. As detailed in the following, we include teacher and school covariates in our teacher outcome analyses to address some of these concerns.

OPEN IN VIEWER

Table 1

Characteristics of NTSP and Comparison Sample Teachers and Schools

	NTSP Treatment 2012–2013	NTSP-Eligible Comparison 2012–2013	Non-RttT Comparison 2012–2013	NTSP Treatment 2013–2014	NTSP-Eligible Comparison 2013–2014	Non-RttT Comparison 2013–2014
Teacher count	377	205	1,144	846	176	1,635
School count	59	32	148	91	17	149
District count	16	18	48	25	12	48
Teacher characteristics
First-year teacher	68.70	60.98 +	61.01**	49.05	40.34*	40.78**
Second-year teacher	31.30	39.02 +	38.99**	35.58	32.95	35.66
Third-year teacher	n/a	n/a	n/a	15.37	26.70**	23.55**
Traditionally prepared	63.46	86.83**	75.26**	66.71	85.22**	75.60**
Alternative entry	32.97	11.71**	22.56**	29.53	13.07**	22.53**
Minority	40.37	22.39**	27.73**	47.53	18.13**	28.17**
Female	77.72	79.51	78.72	79.47	78.74	79.63
School characteristics
Percentage free and reduced-price lunch	93.58	87.16 +	86.02**	91.78	87.33	87.40**
Percentage racial/ethnic minority	90.87	82.31*	78.39**	89.63	81.60 +	78.67**
Short-term suspension rate (per 100 students)	48.86	27.56**	33.43**	38.14	28.16	25.50**
Violent acts rate (per 1,000 students)	12.24	4.82**	10.91	8.71	6.14	10.46
Performance composite	20.04	22.55	23.21**	31.07	38.65 +	34.29**
Percentage novice teacher	32.99	26.59**	27.13**	32.93	33.23	30.29
Per-pupil expenditures	11,001.01	11,136.66	10,057.51 +	10,534.76	10,068.34	10,307.40
Average district salary supplement	2,831.86	2,791.56	3,216.38	2,941.27	2,680.16	3,227.33

Note. The top panel of this table displays counts of unique teachers, schools, and school districts in the NTSP, NTSP-eligible, and non-RttT comparison samples in the 2012–2013 and 2013–2014 school years. The middle panel displays select teacher characteristics for these groups; the bottom panel displays select school characteristics for these groups. NTSP = New Teacher Support Program; RttT = Race to the Top.

+

p < .10, *p < .05, **p < .01, between a comparison group (NTSP-eligible or non-RttT) and NTSP.

Outcome Measures

Covariates

To account for differences in school characteristics (as shown in Table 1) and better isolate the associations between NTSP participation and teacher outcomes, we include a set of current-year (2012–2013 or 2013–2014) school characteristics in our value-added, evaluation rating, and retention models. These include school size and school size squared, ¹² total per-pupil expenditures, average teacher salary supplements, short-term suspension rates (per 100 students), violent acts rates (per 1,000 students), the percentage of free and reduced-price lunch students, and the percentage of racial and ethnic minority students. While most of these school covariates are outside the control of the NTSP, we acknowledge that program supports could influence the short-term suspension and violent acts rate variables. Therefore, as a specification check, we estimated value-added, evaluation rating, and retention models excluding these two measures of school orderliness. Results are qualitatively similar with and without these covariates. ¹³ Models also include teacher experience indicators and year fixed effects; we insert subject area indicators (e.g., biology, U.S. history, Algebra II) into value-added analyses in secondary grades.

Comparisons

In the present study, we use two years of administrative data across two NTSP cohorts and four NTSP regional sites. While the NTSP administrators used a structured program model, there was variation in program implementation, across cohorts and regions, due to different needs/context in regions and the substantial scaling up of the program in 2013–2014 (see Appendix B, available online, for program participation details). To account for these differences, we estimate four sets of models.

First, in our overall models, we pool data from the 2012–2013 and 2013–2014 school years and compare outcomes for NTSP teachers versus those of NTSP-eligible and non-RttT comparison sample teachers. Specifically, in these overall models, we make NTSP teachers the reference category and test for significant performance and retention differences between NTSP and comparison sample teachers (NTSP-eligible and non-RttT). As a supplement to these analyses, we estimate two additional overall models to assess whether (a) alternative entry teachers in the NTSP have outcomes that differ from those in the comparison samples and (b) differential teacher attrition biases our overall results. We include these alternative entry analyses given research showing that alternative entry teachers are initially less effective but narrow effectiveness gaps over time (Boyd, Grossman, Lankford, Loeb, & Wyckoff, 2006). This suggests that alternative entry teachers may have more to gain from an induction program. Likewise, we estimate a set of value-added and evaluation rating analyses controlling for a teacher’s retention outcome (will or will not return to the school in the following year) since the attrition of more or less effective teachers may over- or understate the performance differences between NTSP and comparison sample teachers. Appendix Table A3 (available online) also displays descriptive data on exiting teachers in our NTSP, NTSP-eligible, and non-RttT samples.

Second, in our regional models, we pool data from 2012–2013 and 2013–2014 and compare outcomes for the subset of NTSP teachers within each regional site (ECU, UNC-CSLD, UNCC, and UNCG) to the full sample of NTSP-eligible and non-RttT comparison teachers. Specifically, in these regional models, we iteratively make teachers from the NTSP-eligible and non-RttT comparison samples the reference category and include indicators for NTSP teachers in each region (with an additional indicator for the comparison group that is not the reference category). In these models, we also include a set of regional indicator variables for both NTSP and comparison sample teachers (e.g., a “Charlotte” variable for NTSP and comparison sample teachers working in Charlotte-Mecklenburg schools and the surrounding area). These indicators help us better adjust for regional differences common to NTSP and comparison sample teachers—for example, in teacher labor markets, student population—that may influence teacher outcomes. As a specification check, we also estimate regional models that limit the analytical sample to NTSP and comparison sample teachers working in the same region (e.g., a model limited to NTSP and comparison sample teachers working in the ECU region only).

Third, in our cohort models, we compare outcomes for teachers who entered NTSP in 2012–2013 (Cohort 1) and 2013–2014 (Cohort 2) versus those of NTSP-eligible and non-RttT comparison sample teachers. Specifically, for the 2012–2013 school year, we make Cohort 1 NTSP teachers the reference category and test for significant performance and retention differences between NTSP and comparison sample teachers (NTSP-eligible and non-RttT). For the 2013–2014 school year, we iteratively make teachers from the NTSP-eligible and non-RttT comparison samples the reference category and include indicators for Cohort 1 and Cohort 2 NTSP teachers (with an additional indicator for the comparison group that is not the reference category).

Finally, in our dosage models, we limit the sample to NTSP teachers only and assess the associations between program participation and teacher performance and retention. Specifically, we enter each of our three dosage measures—attending a NTSP institute, the total number of NTSP professional development sessions attended (out of six), and the average number of NTSP instructional coach visits per month—into separate teacher performance and retention models.

Given the need to understand the conditions in which programs work and not just whether programs work, these regional, cohort, and dosage analyses may be particularly important (Bryk, 2015).

Value-Added Models

To assess whether NTSP teachers have higher value-added scores than their comparison sample peers, we specified an ordinary least squares regression model with teachers’ standardized EVAAS estimates as the outcome variable. These value-added models cluster standard errors at the school level and control for teacher experience, year fixed effects, and a set of school covariates. Controlling for model covariates, focal coefficients (from overall, regional, cohort, and dosage models) indicate whether participation in the NTSP is associated with teachers’ value-added estimates. The basic estimation equation is as follows:

Valuead d ijst = β Suppor t ist + δ Ex p it + θ Schoo l ist + ε ist , ,

where Valuead d ijst is a teacher’s value-added estimate (standardized) in subject-area j at school s and time t; Suppor t ist is a set of indicators for the NTSP or NTSP-eligible and non-RttT samples in models to assess overall, regional, cohort, and dosage results; β estimates the average relationship between the Support indicators and teacher value-added; Ex p it and Schoo l ist represent a set of teacher experience indicators and school covariates; and ϵ _ist is a disturbance term representing unexplained variation in teacher value-added.

Evaluation Rating Models

To examine teacher performance for a larger sample and for a broader set of teaching competencies, we estimate relationships between NTSP participation and teachers’ evaluation ratings on the five North Carolina Professional Teaching Standards directly assessed by school administrators. With a 2–5 outcome variable (developing to distinguished), we started with an ordered logit model to test the proportional odds assumption—that the relationship between each pair of outcome groups and model covariates was the same (Brant, 1990; O’Connell, 2006). For each of the five teaching standards, the overall likelihood ratio test strongly rejected the null hypothesis; however, a closer examination of the results showed that the proportional odds assumption was met by the focal NTSP variables and violated by the school covariates. With these results, we changed our analytic approach to an ordered logistic regression with partial proportional odds (Fullerton, 2009), where the focal NTSP coefficients are fixed across the evaluation rating outcomes and regression coefficients for the other model covariates are not constrained to be the same. Controlling for model covariates and clustering standard errors at the school level, focal coefficients (from overall, regional, cohort, and dosage models) indicate whether participation in the NTSP is associated with teachers’ evaluation ratings. The equation for the partial proportional odds evaluation rating models is as follows:

Logit ( C istj ) = α j − β Suppor t ist − δ Ex p it − θ Schoo l ist ,

where for each teaching standard, C istj is the cumulative probability that teacher i at school s and time t is in the jth category or higher; α _j are the cutpoints from the partial proportion ordered logit model; Suppor t ist is a set of indicators for the NTSP or NTSP-eligible and non-RttT samples in models to assess overall, regional, cohort, and dosage results; β estimates the average (fixed) relationship between the Support indicators and teacher evaluation ratings; and Ex p it and Schoo l ist represent a set of teacher experience and school covariates whose relationships with evaluation ratings (δ and θ) can vary across evaluation rating levels.

Teacher Retention Models

Given that novice teachers are more likely to exit the profession and that low-performing schools struggle to retain their teaching workforce, we assess the relationships between NTSP participation and teacher retention (Boyd, Lankford, Loeb, & Wyckoff, 2005; Hanushek, Kain, & Rivkin, 2004). For these analyses, we specify logistic regression models with controls for teacher experience and school covariates and standard errors clustered at the school level. Our preferred approach examines returning to the same school (low-performing) in the following year; additional analyses examine returning to the same school district. Controlling for model covariates, focal coefficients indicate whether participation in the NTSP is associated with the odds of returning to teach. The equation for teacher retention is as follows:

Pr ( Retur n ist = 1 ) = exp ( Suppor t ist + δ Ex p it + Schoo l ist ) 1 + exp ( Suppor t ist + δ Ex p it + Schoo l ist ) ,

where Retur n ist is a binary outcome equal to 1 for teacher i at school s and time t if he or she returns to teach in the same school (district) in the following school year; Suppor t ist is a set of indicators for the NTSP or NTSP-eligible and non-RttT samples in models to assess overall, regional, cohort, and dosage results; and Ex p it and Schoo l ist represent a set of teacher experience and school covariates.

Is NTSP Participation Associated With Teacher Performance and Retention?

Overall, the left and middle panels of Table 2 indicate that performance outcomes—value-added and evaluation ratings—for NTSP teachers are no different than those for teachers in the NTSP-eligible (our preferred reference category receiving other RttT treatments) and non-RttT (not receiving other RttT treatments) comparison samples. While the teacher value-added estimates are positive in elementary and middle grades, the coefficients are negative for secondary grades exams, particularly in comparison to the NTSP-eligible sample. In the top panel of Appendix Table A4 (available online), we show that NTSP teachers are significantly less effective than NTSP-eligible comparison sample teachers in non-STEM (English and social studies/history) secondary grades exams; there are no differences in teacher effectiveness in STEM subject areas. ¹⁴ The right panel of Table 2 shows that there are significant differences in retention between NTSP teachers and teachers in both comparison samples. Controlling for model covariates, NTSP teachers have a 72.43% predicted probability of returning to the same low-performing school in the following year. By comparison, the predicted probabilities for NTSP-eligible and non-RttT comparison sample teachers are 64.24% and 65.08%. To contextualize the magnitude of these retention differences, we note that (a) the unadjusted school retention rate in our sample was 66.89 and (b) if the comparison groups were the same size as the NTSP sample (1,206 teacher-year observations), NTSP schools would need to replace 332 novice teachers, NTSP-eligible schools would need to replace 431 novice teachers, and non-RttT schools would need to replace 421 novice teachers during our study period.

OPEN IN VIEWER

Table 2

Overall Results for NTSP Teachers

	Outcome: Teacher Value-Added				Outcome: Teacher Evaluation Ratings					Outcome: Retention
	Elementary and Middle Grades Math	Elementary and Middle Grades Reading	Fifth- and Eighth- Grade Science	Secondary Grades EOC and Final Exams	Leadership	Classroom Environment	Content Knowledge	Facilitating Student Learning	Reflecting on Practice	Returning to the Same School
NTSP vs. NTSP-eligible	0.258(0.240)	0.136(0.265)	0.197(0.307)	−0.215(0.148)	0.976(0.920)	1.020(0.933)	0.997(0.988)	0.955(0.826)	1.084(0.677)	1.462*(0.016)
NTSP vs. non-RttT	0.161(0.134)	0.133(0.123)	0.333(0.208)	−0.054(0.124)	1.063(0.721)	0.976(0.885)	0.985(0.941)	0.988(0.944)	0.952(0.790)	1.409**(0.001)
Observations	652	893	258	820	3,862	3,860	3,863	3,862	3,862	4,389

Note. The left panel displays differences in adjusted-average Education Value-Added Assessment System (EVAAS) estimates (standardized) for NTSP teachers versus NTSP-eligible and non-RttT comparison sample teachers. Cells report coefficients with standard errors in parentheses. The middle panel displays odds ratios for earning higher evaluation ratings for NTSP teachers versus NTSP-eligible and non-RttT comparison sample teachers. Cells report p values in parentheses below odds ratios. The right panel displays odds ratios for returning to the same school in the following year for NTSP teachers versus NTSP-eligible and non-RttT comparison sample teachers. Cells report p values in parentheses below odds ratios. NTSP = New Teacher Support Program; EOC = end-of-course; RttT = Race to the Top.

*

p < .05. **p < .01.

As a supplement to these overall models, we performed two different specification checks. First, given the sizable proportion of alternative entry teachers in the NTSP sample (Table 1) and prior research indicating that alternative entry teachers may make more rapid on-the-job effectiveness gains (Boyd et al., 2006)—suggesting that they may benefit more from comprehensive induction—we assessed whether alternative entry teachers supported by the NTSP have outcomes that differ from those in the comparison samples. Here, interaction coefficients in Appendix Table A5 (available online) reveal no significant differences between alternative entry teachers in the NTSP and comparison samples. Second, to assess whether the attrition of more or less effective teachers may over- or understate the performance differences between NTSP and comparison sample teachers, we re-ran our value-added and evaluation rating models including an indicator for whether a teacher will return to the same school in the following year. These results, shown in Appendix Table A6 (available online), indicate that teachers who will return in the following school year have significantly higher value-added estimates and evaluation ratings than their peers who will leave. However, NTSP results in Table 2 are comparable to those in Appendix Table A6, suggesting that differential teacher attrition does not meaningfully influence our overall estimates. To complement these analyses, we also display teacher and school characteristics (in Appendix Table A3) for exiting teachers in the NTSP, NTSP-eligible, and non-RttT samples. In comparison to NTSP-eligible teachers, this table indicates that a lower percentage of first-year, third-year, alternative entry, and minority NTSP teachers exit their schools. Given the descriptive characteristics in Table 1, it is not surprising that NTSP teachers exit schools with slightly higher percentages of economically disadvantaged and minority students and slightly lower performance composites.

Do Results for NTSP Teachers Vary by Region and Cohort?

While the NTSP had a common framework for supporting teachers, differences in the four university anchor sites (ECU, UNC-CSLD, UNCC, and UNCG) and the schools/school districts they served led to variation in program participation across regions. Likewise, the dramatic scale-up of the program—more than doubling in size from 377 to 846 teachers—was associated with a decrease in program participation in the 2013–2014 school year. Appendix B depicts these regional and cohort participation differences. For example, Appendix Figures B1–B4 (available online) show that: (a) a higher percentage of NTSP teachers attended a program institute in 2012–2013 versus 2013–2014 and that across both years NTSP teachers in the ECU region had the highest institute attendance rates; (b) the average number of instructional coach visits per month was highest in the ECU region and that across regions there were fewer instructional coach visits in 2013–2014; ¹⁵ and (c) there was a steep decline in NTSP professional development attendance in 2013–2014, with few teachers in the UNCC and UNCG regions attending any program professional development. These participation differences across regions and cohorts motivate further analyses to determine whether NTSP teachers in the regions and cohort with more intensive program participation have better performance and retention outcomes. This is particularly important since the knowledge gained in understanding under what circumstances a program works may benefit the future performance of the program or similar programs.

Our primary regional analyses compare outcomes for the NTSP teachers within each regional site versus the full sample of NTSP-eligible and non-RttT comparison teachers. These models also include a set of regional indicator variables (for both NTSP and comparison sample teachers) to adjust for differences across regions—for example, in teacher labor markets, student populations—that may influence teacher performance and retention. For these analyses, Table 3 indicates that NTSP teachers in the ECU region have higher value-added estimates than NTSP-eligible comparison sample teachers in elementary and middle grades mathematics and reading; results versus non-RttT teachers are generally comparable to those versus NTSP-eligible teachers. To put the magnitude of these value-added results into perspective, we note that statewide, the average difference in standardized EVAAS estimates between first- and second-year teachers in elementary and middle grades mathematics is 33% of a standard deviation; in elementary and middle grades reading, the average difference is 17% of a standard deviation. Contrary to the positive results in elementary and middle grades, NTSP teachers in the ECU region, along with those in the UNC-CSLD region, have significantly lower value-added estimates than NTSP-eligible teachers for the state’s secondary grades EOC and final exams. Appendix Table A4 indicates that the negative ECU results are concentrated in STEM subject areas; negative results for UNC-CSLD teachers are concentrated in non-STEM subject areas. While our regional analyses reveal no significant differences in evaluation ratings (middle panel of Table 3), the right panel of Table 3 indicates that NTSP teachers in the ECU region have significantly higher within-school retention rates than NTSP-eligible and non-RttT comparison sample teachers. Controlling for model covariates, the predicted school retention probabilities are 77.96, 67.28, 72.04, and 71.70 for NTSP teachers in the ECU, UNC-CSLD, UNCC, and UNCG regions, respectively. Finally, when comparing within region only (e.g., comparing NTSP teachers in the ECU region to comparison sample teachers in the same geographical area), Appendix Table A7 (available online) shows that NTSP teachers in the ECU region maintain significantly higher value-added estimates in elementary and middle grades mathematics and reading and significantly higher within-school retention rates.

OPEN IN VIEWER

Table 3

Regional Results for NTSP Teachers

	Outcome: Teacher Value-Added				Outcome: Teacher Evaluation Ratings					Outcome: Retention
	Elementary and Middle Grades Math	Elementary and Middle Grades Reading	Fifth- and Eighth-Grade Science	Secondary Grades EOC and Final Exams	Leadership	Classroom Environment	Content Knowledge	Facilitating Student Learning	Reflecting on Practice	Returning to the Same School
Regional results: NTSP vs. NTSP-eligible
ECU Region	0.680**(0.229)	0.635*(0.293)	0.684(0.421)	−0.506**(0.143)	0.980(0.954)	1.152(0.723)	0.725(0.428)	0.670(0.202)	1.244(0.561)	1.887**(0.003)
UNC-CSLD region	−0.130(0.257)	−0.001(0.295)	0.130(0.411)	−0.522 + (0.298)	0.710(0.354)	0.744(0.390)	1.193(0.711)	0.962(0.917)	0.876(0.730)	1.097(0.674)
UNCC region	0.162(0.348)	−0.184(0.340)	−0.205(0.534)	−0.252(0.227)	1.364(0.412)	1.333(0.448)	1.349(0.407)	1.452(0.301)	1.430(0.329)	1.375(0.159)
UNCG region	−0.187(0.291)	−0.311(0.270)	−0.535(0.382)	−0.140(0.252)	0.785(0.545)	0.781(0.549)	0.633(0.341)	0.920(0.840)	0.747(0.423)	1.352(0.191)
Regional results: NTSP vs. non-RttT
ECU region	0.683**(0.183)	0.748**(0.169)	0.980*(0.410)	−0.252 + (0.148)	1.074(0.828)	1.118(0.777)	0.722(0.429)	0.662(0.211)	1.117(0.787)	1.887**(0.002)
UNC-CSLD region	−0.127(0.204)	0.113(0.231)	0.426(0.335)	−0.268(0.249)	0.778(0.464)	0.722(0.315)	1.187(0.715)	0.950(0.881)	0.787(0.540)	1.097(0.637)
UNCC region	0.166(0.288)	−0.070(0.274)	0.090(0.464)	0.002(0.163)	1.494(0.200)	1.294(0.374)	1.342(0.326)	1.434(0.191)	1.285(0.392)	1.375 + (0.074)
UNCG region	−0.184(0.231)	−0.197(0.208)	−0.239(0.282)	0.114(0.217)	0.860(0.675)	0.758(0.441)	0.630(0.309)	0.909(0.796)	0.671(0.252)	1.352(0.114)
Observations	652	893	258	820	3,862	3,860	3,863	3,862	3,862	4,389

Note. The left panel displays differences in adjusted-average Education Value-Added Assessment System (EVAAS) estimates (standardized) for NTSP teachers, by region, versus NTSP-eligible (top) and non-RttT (bottom) comparison sample teachers. The middle panel displays odds ratios for earning higher evaluation ratings for NTSP teachers, by region, versus NTSP-eligible and non-RttT comparison sample teachers. p values are in parentheses below odds ratios. The right panel displays odds ratios for returning to the same school in the following year for NTSP teachers, by region, versus NTSP-eligible and non-RttT comparison sample teachers. p values are in parentheses below odds ratios. NTSP = New Teacher Support Program; EOC = end-of-course; RttT = Race to the Top; ECU = East Carolina University; UNC-CSLD = University of North Carolina Center for School Leadership Development; UNCC = University of North Carolina Charlotte; UNCG = University of North Carolina Greensboro.

+

p < .10. *p < .05. **p < .01.

Turning to cohort analyses, Table 4 indicates that NTSP teachers in the first program cohort—entering in the 2012–2013 school year—have significantly higher value-added estimates than NTSP-eligible and non-RttT comparison sample teachers in elementary and middle grades mathematics and reading in the 2012–2013 school year. Conversely, Cohort 1 NTSP teachers have significantly lower value-added estimates than non-RttT comparison sample teachers on secondary grades EOC and final exams. In 2013–2014, both cohorts of NTSP teachers are significantly less effective than NTSP-eligible teachers in elementary and middle grades reading; Cohort 2 NTSP teachers are also significantly less effective for secondary grades exams. Both NTSP cohorts have higher value-added estimates than non-RttT comparison sample teachers in fifth- and eighth-grade science.

OPEN IN VIEWER

Table 4

Cohort Results for NTSP Teachers (Value-Added)

	Elementary and Middle Grades Mathematics		Elementary and Middle Grades Reading		Fifth- and Eighth- Grade Science		Secondary Grades EOC and Final Exams
	2012–2013	2013–2014	2012–2013	2013–2014	2012–2013	2013–2014	2012–2013	2013–2014
Cohort results: NTSP vs NTSP-eligible
NTSP Cohort 1	0.851**(0.280)	−0.220(0.328)	0.914**(0.303)	−0.343 + (0.200)	0.150(0.468)	0.035(0.328)	−0.235(0.161)	−0.137(0.278)
NTSP Cohort 2	—	−0.405(0.321)	—	−0.486**(0.159)	—	0.062(0.294)	—	−0.485 + (0.276)
Cohort Results: NTSP vs. non-RttT
NTSP Cohort 1	0.565**(0.214)	0.150(0.188)	0.770**(0.187)	0.075(0.196)	0.155(0.361)	0.440 + (0.262)	−0.237+(0.127)	0.224(0.178)
NTSP Cohort 2	—	−0.034(0.153)	—	−0.068(0.154)	—	0.468 + (0.248)	—	−0.123(0.160)
Observations	269	383	283	610	114	144	302	518

Note. The top panel displays differences in adjusted-average Education Value-Added Assessment System (EVAAS) estimates (standardized) between Cohort 1 and Cohort 2 NTSP teachers versus NTSP-eligible comparison sample teachers. The bottom panel displays differences in adjusted-average EVAAS estimates (standardized) between Cohort 1 and Cohort 2 NTSP teachers versus non-RttT comparison sample teachers. NTSP = New Teacher Support Program; EOC = end-of-course; RttT = Race to the Top.

+

p < .10. **p < .01.

For teacher evaluation ratings, cohort results in Table 5 do not reveal any significant differences for Cohort 1 NTSP teachers in 2012–2013—when there were significant value-added results. In 2013–2014, evaluation ratings for Cohort 1 NTSP teachers are comparable to the NTSP-eligible sample. Conversely, evaluation ratings for Cohort 2 NTSP teachers are significantly lower for four standards versus NTSP-eligible teachers and two standards versus non-RttT teachers. During their second year in the program (2013–2014), we find that Cohort 1 NTSP teachers have significantly higher evaluation ratings than non-RttT comparison sample teachers on four evaluation standards.

OPEN IN VIEWER

Table 5

Cohort Results for NTSP Teachers (Evaluation Ratings)

	Leadership		Classroom Environment		Content Knowledge		Facilitating Student Learning		Reflecting on Practice
	2012–2013	2013–2014	2012–2013	2013–2014	2012–2013	2013–2014	2012–2013	2013–2014	2012–2013	2013–2014
Cohort results: NTSP vs. NTSP-eligible
NTSP Cohort 1	1.258(0.472)	1.225(0.549)	1.519(0.159)	1.197(0.624)	1.458(0.269)	1.477(0.265)	1.104(0.729)	1.450(0.267)	1.521(0.160)	1.338(0.335)
NTSP Cohort 2	—	0.588 + (0.093)	—	0.565 + (0.069)	—	0.563 + (0.069)	—	0.646(0.149)	—	0.596 + (0.052)
Cohort results: NTSP vs. non-RttT
NTSP Cohort 1	1.074(0.753)	1.697*(0.029)	1.233(0.332)	1.376(0.244)	1.209(0.508)	1.719 + (0.051)	0.914(0.712)	1.708*(0.026)	0.973(0.908)	1.606 + (0.067)
NTSP Cohort 2	—	0.814(0.360)	—	0.649*(0.027)	—	0.655 + (0.069)	—	0.761(0.172)	—	0.715(0.124)
Observations	1,559	2,303	1,559	2,301	1,559	2,304	1,559	2,303	1,558	2,304

Note. The top panel displays odds ratios for earning higher evaluation ratings for Cohort 1 and Cohort 2 NTSP teachers versus NTSP-eligible comparison sample teachers. The bottom panel displays odds ratios for earning higher evaluation ratings for Cohort 1 and Cohort 2 NTSP teachers versus non-RttT comparison sample teachers. p values are in parentheses below odds ratios. NTSP = New Teacher Support Program; RttT = Race to the Top.

+

p < .10. *p < .05.

Finally, in Table 6, we present cohort retention results. Here, in comparison to NTSP-eligible and non-RttT comparison sample teachers, Cohort 1 NTSP teachers have significantly higher within-school retention rates—returning in 2013–2014 and 2014–2015. Within-school retention is no different between Cohort 2 NTSP teachers and NTSP-eligible teachers (although the result approaches statistical significance); Cohort 2 NTSP teachers have significantly higher within-school retention rates than non-RttT comparison sample teachers. The predicted retention probabilities for returning to the same low-performing school in 2014–2015 are as follows: 75.06 for Cohort 1, 70.76 for Cohort 2, 63.55 for NTSP-eligible comparison sample teachers, and 64.03 for non-RttT comparison sample teachers.

OPEN IN VIEWER

Table 6

Cohort Results for NTSP Teachers (Retention)

	School Retention
	Returns in 2013–2014	Returns in 2014–2015
Cohort results: NTSP vs. NTSP-eligible
NTSP Cohort 1	1.491 + (0.096)	1.726*(0.028)
NTSP Cohort 2	—	1.388(0.113)
Cohort results: NTSP vs. non-RttT
NTSP Cohort 1	1.377 + (0.072)	1.691**(0.007)
NTSP Cohort 2	—	1.359*(0.014)
Observations	1,717	2,671

Note. This table displays odds ratios for returning to the same school in the 2013–2014 and 2014–2015 years for Cohort 1 and Cohort 2 NTSP teachers versus NTSP-eligible and non-RttT comparison sample teachers. p values are in parentheses below odds ratios. NTSP = New Teacher Support Program; RttT = Race to the Top.

+

p < .10. *p < .05. **p < .01.

Is Intensity of NTSP Participation Associated With Outcomes?

Results from the previous section indicate that NTSP teachers in the region and the year/cohort with the most intensive program participation have positive outcomes. This suggests that dosage may matter. To more formally test this and assess whether certain NTSP components are significantly associated with teacher outcomes, we limited our sample to NTSP teachers and estimated dosage models controlling for levels of participation in the NTSP institutes, professional development, and instructional coaching—entered into separate regression models. While we acknowledge selection concerns—for example, instructional coaches may frequently visit teachers who are struggling, teachers decide whether to attend program institutes and professional development—these models may identify components that benefit teachers and help NTSP implementers target resources toward more effective programmatic practices.

Dosage results in Table 7 show that an average of one more instructional coaching visit per month is associated with significantly higher teacher value-added estimates in mathematics (Grades 4–8) and secondary grades EOC and final exams. ¹⁶ The frequency of instructional coach visits is not significantly associated with value-added in reading or science or teacher evaluation ratings. An additional coaching visit per month is also associated with significantly higher levels of school retention. Specifically, one more instructional coaching session per month—approximately nine more visits throughout the school year—is associated with a 2 percentage point increase in retention for NTSP teachers.

OPEN IN VIEWER

Table 7

Dosage Results for NTSP Teachers

	Outcome: Teacher Value-Added				Outcome: Teacher Evaluation Ratings					Outcome: Retention
	Elementary and Middle Grades Math	Elementary and Middle Grades Reading	Fifth- and Eighth-Grade Science	Secondary Grades EOC and Final Exams	Leadership	Classroom Environment	Content Knowledge	Facilitating Student Learning	Reflecting on Practice	Returning to the Same School
Attended NTSP institute	−0.023(0.205)	0.160(0.234)	−0.240(0.360)	0.153(0.212)	0.977(0.907)	0.764(0.161)	1.488 + (0.078)	1.095(0.652)	1.205(0.383)	1.148(0.443)
Total n of NTSP professional development sessions	−0.028(0.056)	0.073(0.055)	−0.170 + (0.092)	−0.055 + (0.028)	0.973(0.645)	0.880*(0.012)	0.989(0.849)	0.983(0.734)	0.955(0.415)	1.014(0.754)
Average n of instructional coach visits per month	0.131 + (0.067)	−0.044(0.054)	0.022(0.090)	0.146**(0.048)	0.965(0.619)	0.965(0.640)	0.864(0.109)	0.925(0.282)	1.016(0.843)	1.092 + (0.058)
Observations	162	232	66	197	1,041	1,041	1,044	1,042	1,044	1,193

Note. The left panel displays differences in adjusted-average Education Value-Added Assessment System (EVAAS) estimates (standardized) for NTSP teachers with varying levels of participation in the NTSP components. Cells report coefficients with standard errors in parentheses. The middle panel displays odds ratios for earning higher evaluation ratings for NTSP teachers with varying levels of participation in the NTSP components. Cells report p values in parentheses. The right panel displays odds ratios for returning to the same school in the following year for NTSP teachers with varying levels of participation in the NTSP components. Cells report p values in parentheses. NTSP = New Teacher Support Program; EOC = end-of-course.

+

p < .10. *p < .05. **p < .01.

Beyond the frequency of instructional coaching, there is one positive association between institute attendance and higher evaluation ratings on the content knowledge standard. Conversely, the total number of professional development sessions attended is negatively associated with two value-added estimates—fifth- and eighth-grade science and secondary grades exams—and one evaluation rating—classroom environment. Due to selection concerns, these results should not be overinterpreted; nonetheless, they suggest that the frequency of instructional coaching may matter, while other program components may be less important to teacher success.