The Effects of School Turnaround in Tennessee’s Achievement School District and Innovation Zones

Background on Turnaround

The federal government’s efforts to turn around the nation’s lowest performing schools were preceded by frustration stemming from comprehensive school reform (CSR). From 1988 through 2005, the federal government distributed nearly US$2 billion in grants for CSR, which required schools to choose and adopt a whole school reform model that prescribed the teaching and learning process to be followed throughout school. Although some rigorous evaluations of whole school reform models have shown positive effects on student achievement, for example, Success for All, many have found no effect or negative effects (Bifulco, Duncombe, & Yinger, 2005; Borman et al., 2007; Gross, Booker, & Goldhaber, 2009). The inconsistent and mainly ineffective CSR reforms that focused on building educational infrastructure in low-performing schools (Peurach & Neumerski, 2015) opened the door for bolder and more intrusive interventions designed to disrupt the status quo and make more fundamental changes in the education system as well as school personnel and operations.

The theory of change for federally subsidized turnaround policies has two lines of action, one based on disrupting the status quo and the second to establish the “educational infrastructure for large scale school turnaround” (Peurach & Neumerski, 2015). To disrupt the status quo, Mass Insight (Calkins, Guenther, Belfiore, & Lash, 2007), which generated the first report detailing the turnaround approach, focused on changes to school governance and/or management. In their report, a change in governance, that is, removing low-performing schools from their local districts and placing them under the auspices of the state, could be implemented as a reform strategy and also as a means of encouraging districts and schools to volunteer for local turnaround. After a state takeover, another term for change in governance, the state education agency, special commissioner, or a separate statewide district would establish policies, financial arrangements, and accountability procedures. This change in governance is exemplified by the ASD in Tennessee. Changing management requires appointing a new school management organization to oversee the day-to-day operations of the school including making personnel decisions and establishing hours of operation. Although these two types of changes are often done in tandem, changes in management can take place with or without state takeover. In Tennessee, the ASD enacted a simultaneous change in school governance and management with the new statewide district governing these schools while managing some of them directly and placing others under the management of CMOs. The iZone schools in Tennessee are an example of a management change that was decoupled from a change in governance. The iZone schools remained under the auspices of their district but were managed separately by a special, semiautonomous unit within the district. In all federally funded turnaround models, managers were expected to replace the principal and some or all of the school staff, and grant autonomy to the new principal to change school operations. Changes in school personnel and operations alone were considered “marginal” and similar to the CSR reforms, which had proven insufficient to rapidly improve the performance of chronically low-performing schools (Calkins et al., 2007). A fourth and final type of change was the infusion of federal funds, which increased the financial and human resources available in these schools.

In addition to disrupting the status quo for the low-performing schools, federal turnaround policies provided additional resources to support collaboration with a lead turnaround partner, which increased the capacity for building educational infrastructure (Peurach & Neumerski, 2015). Most states lacked sufficient capacity to provide technical assistance needed by the low-performing schools for functions such as recruiting and retaining effective staff or developing and implementing teacher evaluation models, so at least 47 states turned to external lead turnaround partners or other intermediaries to provide or supplement the assistance needed (Tanenbaum et al., 2015).

The federal “theory of change” underlying the need to disrupt the status quo was codified into four models of reform authorized for receipt of federal RttT and SIG funds: (a) “transformation,” (b) “turnaround” (with a narrower definition than the term used in the broader context of reform), (c) “restart,” and (d) “closure” (Perlman & Redding, 2010). The most frequently implemented model is transformation, which mandates principal replacement, more rigorous teacher evaluation, increased learning time, and grants autonomy for school operations. The turnaround model goes further by requiring replacement of the principal and at least half of the teaching staff and greater autonomy for the new principal. Neither of these models required changes in governance or management. Restart requires fundamental change to schools by transferring school management responsibilities to an independent entity such as a CMO. Some states have chosen to couple the change in management with governance changes by placing the school under the auspices of an entity other than the local school district, such as the ASD in Tennessee. The restart approach assumes that districts, because of the influence of teachers and their unions over elected school boards and entrenched bureaucracies, will not have the capacity, the will, or both to turn around low-performing schools (Chubb & Moe, 1990). From the perspective of Chubb and Moe, changing the management and/or governance is hypothesized to be necessary to meaningfully reform schools. Therefore, a fundamental question asks whether districts can institute reforms that are significant enough to improve chronically low-performing schools, or whether the resistance of elected school boards and the entrenched bureaucracies of school districts are too powerful to turn around low-performing schools. If reform cannot be effectively overseen and managed by local districts, it would suggest that outside management is necessary to observe significant improvement. In this article, we have the opportunity to examine these issues as we examine reforms both under and outside the governance of a local school district, which will provide strong insights into the theoretical debate of the ability of districts to significantly improve low-performing schools.

Literature Review

Although there have been a number of published studies examining the 1990s federally funded initiative of CSR models for turning around low-performing schools (Berends, Bodilly, & Kirby, 2002; Bifulco et al., 2005; Borman, Hewes, Overman, & Brown, 2003; Desimone, Payne, Federovicious, Henrich, & Finn-Stevenson, 2004; Gross et al., 2009) as well as recent studies examining school closures (Brummet, 2014; Carlson & Lavertu, 2016; de la Torre et al., 2013; Engberg, Gill, Zamarro, & Zimmer, 2012; Ruble, 2015), research on the use of federal turnaround models including those subsidized by RttT and SIGs have recently begun to emerge. The earliest of this work examined the use of state takeover as a means of improving chronically low-performing schools—Pennsylvania’s takeover of the Philadelphia school district resulting in the turnover of the management of 45 low-performing schools to Education Management Organizations (EMOs) in the early 2000s. Philadelphia’s example could be best characterized as a restart approach focusing on state takeover of schools with governance and management turned over to external operators. Researchers found that these schools did not outperform the gains compared with other schools within Philadelphia (Gill, Zimmer, Christman, & Blanc, 2007; MacIver & MacIver, 2006) despite the additional resources these schools were provided. ¹ It is worth noting certain restrictions were placed on schools concerning what reforms they could implement, including a restriction to maintain the schools as neighborhood schools.

More recently, with the incentives associated with RttT and SIGs as well as the NCLB waivers, the number of locations adopting state takeover and turnaround strategies has grown. As the number has grown, so too has the research (although much is unpublished at this point) with various research designs examining different approaches with mixed results (Dee, 2012; Dougherty & Weiner, 2015; Heissel & Ladd, 2016; Henry & Guthrie, 2016; Papay, 2015; Ruble, 2015; Schueler et al., 2015; Strunk, Marsh, Hashim, & Bush-Mecenas, 2016). A summary of these turnaround studies highlighting the changes in governance (state takeover), management (local, state, or CMO), staffing and operations, and teaching and learning along with the effect estimates are presented in Table 1. Of the four studies in which state takeover was involved, two directly run by the state and one by CMOs found positive effects whereas the fourth, run by the state, did not. Of the three studies with reforms primarily replacing personnel and changing school operations, two found positive effects and one found negative effects in reading, and one found positive effects in math, whereas the other two found no effects in math.

OPEN IN VIEWER

Table 1

Summary of Turnaround Studies

Functional requirements of public schools	Philadelphia (Gill, Zimmer, Christman, & Blanc, 2007)	New Orleans (Ruble, 2015)	Los Angeles (Strunk et al., in press)	Massachusetts (Papay, 2015)	Lawrence, Massachusetts (Schueler et al., 2015)	Rhode Island (Dougherty & Weiner, 2015)	North Carolina (Henry & Guthrie, 2016)
Governance Authority, oversight, and accountability	State established an SRC as a replacement for the local school district board	State establishes RSD to takeover and oversee low-performing schools			State appointed receiver to position of district-wide authority, who selected a new central office team		State created the District and School Transformation Department and identified low-performing schools for one of the four turnaround interventions
System management Leadership and resource allocation	SRC selects CMOs to manage schools	RSD selects CMO to manage schools	District maintains management, but replaces some principals	District maintains management, but is required to implement several state initiatives	Receiver and central office team selects outside operators to manage lowest performing schools, shares responsibilities with operators	Districts maintain management, but some replace principals	Districts maintained management, but most principals were replaced
Site management and operations School personnel and resources	CMO oversees school operations with some constraints from the SRC	CMO oversees school operations	District and school leaders maintain authority over operations, but replace some teachers	Districts and schools maintain authority over operations, but are required to implement several state initiatives	Receiver replaces teachers, receivers and operators manage schools	Districts and schools maintain authority over operations, but are required to implement several state initiatives	Districts and schools maintain authority over operations, but receive leadership (school and district) coaches and professional development
Teaching and learning Curriculum and instruction; Classroom-level management	CMO establishes curriculum and sets expectations for teaching and learning	RSD and CMO establish curriculum and set expectations for teaching and learning		Schools maintain authority over teaching and learning, but are required to implement several state initiatives	Receivers and operators set expectations for teaching and learning	Schools maintain authority over teaching and learning, but are required to implement several state initiatives	Schools maintain authority over teaching and learning, but receive instructional coaching and targeted professional development
Impact	No overall effects	0.1–0.2 SD increase in ELA, 0.1–0.4 SD increase in math	0.14 SD increase in ELA, no effects in math	0.14–0.38 SD increase in ELA and math	0.02–0.03 SD increase in ELA, 0.17–0.19 SD increase in math	Up to 0.35 SD decrease in ELA, no effects in Math	0.02–0.03 positive effects on student achievement in all turnaround schools (Henry & Guthrie 2015), no local average treatment effects in middle and elementary schools (Heissel & Ladd, 2016; Henry & Guthrie, 2016)

Note. SRC = School Reform Commission; RSD = Recovery School District; CMO = charter management organization; ELA = English/Language Arts.

Overall, although these studies have been insightful for whether particular turnaround models can change school performance, they generally have not simultaneously compared state takeover and district-managed turnaround approaches or public versus private management of restarted schools. In this article, we examine the effectiveness of state takeover by including turnaround reforms both within and outside of the district. We also examine the effectiveness of different management approaches, including a district-managed transformation approach that changed leadership, other personnel, school operations, and instructional methods. We compare this approach with a turnaround approach that involved state takeover and school management by the state or CMOs, which represents a restart approach. Ultimately, the analysis provides insights into the question of whether it is necessary to remove schools from district governance to effectively turnaround low-performing schools.

Background on Turnaround in Tennessee

Inspired by the potential of RttT funding, Tennessee passed legislation called First to the Top in January 2010, which created the ASD. With this legislation in hand, the state applied for RttT funding, and, in March of that year, Tennessee was awarded US$500 million that called for the State Commissioner of Education to identify the state’s lowest achieving 5% of Title I schools, known as priority schools. More than 80% of these schools, were located in Memphis, the rest in Nashville and Chattanooga with only two priority schools residing outside of these three cities. These schools faced a number of possible interventions. Among these possible interventions, none has been bolder and, consequently, more controversial than the ASD—a restart strategy in which a new state school district removed schools from their home districts and either directly managed these schools (ASD run) or contracted management responsibilities to a CMO partner (CMO run). The stated goal of the ASD was to move the academic performance of schools taken over from the bottom 5% of schools to the top quartile of schools in Tennessee within 5 years.

As initially conceived by the original First to the Top legislation, once a school was selected for the ASD, the school would remain in the ASD for at least 5 years. The school would return to the home district conditional on the performance of both the school and the home district (Elementary and Secondary Education Act [ESEA] Flexibility Request, 2012). Although the application did not dismiss the possibility of the state solely operating ASD schools acting as a pseudo CMO, the emphasis was on a hybrid model where the state takes over and partners with CMOs to manage the ASD schools. To achieve the goal of raising student achievement to the top quartile of schools in the state, both the ASD and the CMOs were given autonomy to hire talented education professionals with emphasis on teachers (Race to the Top Application for Initial Funding, 2010). It is important to point out that although CMOs were tapped to run schools, the schools remained neighborhood schools rather than schools of choice in which parents and students must opt-in to attend. The ASD used several criteria to decide which of the state’s priority schools would be taken over, including the schools’ feeder patterns and ability to match the school and its community with an operator (ESEA Flexibility Request, 2012). For example, students in 9 of the 12 schools with elementary grades, which were placed under the auspices of the ASD during the study period, matriculated into five of the eight schools with middle grades taken over by ASD. One of the ASD middle schools that did not receive students from an ASD elementary grade school was a standalone takeover in Nashville, whereas one elementary grade ASD school and one middle-grade ASD school fed into charter schools managed by the same CMO though not under the auspices of the ASD.

In 2012–2013, the ASD took over the first cohort of 6 schools with 3 schools run by CMOs and 3 run directly by the ASD. In 2013–2014, the ASD added a second cohort of 11 schools, 8 run by CMOs and 3 run directly by ASD. Finally, in the 2014–2015 school year, a third cohort of 8 schools were added, all CMO run, whereas 2 schools opened in the 2013–2014 school year were merged in with other ASD schools. By the 2014–2015 school year, 23 schools were operating under the auspices of ASD with 5 managed directly by the ASD and 18 managed by CMOs.

As an alternative to the state-takeover approach, some priority schools remained under the governance of the district but were managed by semiautonomous districts-within-a-district, known as iZones in which schools were clustered as originally envisioned in the Mass Insight report on turnaround (Calkins et al., 2007). Several districts throughout Tennessee have adopted iZones including Memphis (Shelby County Schools), Metropolitan Nashville Public Schools, and Hamilton County (Chattanooga) Schools. In the 2012–2013 school year, 13 schools opened under iZone with seven opening in Memphis that year, the remainder in Nashville. In the following 2013–2014 school year, 11 total iZone schools opened—5 in Memphis and 6 in Chattanooga. In the final year of analysis, 4 schools opened in 2014–2015 school year, all in Memphis. In Nashville and Chattanooga, all schools identified as priority schools except for 3 schools, one taken over by ASD, another closed, and another already managed by a CMO, were placed into their respective iZones. Memphis used school feeder patterns to assign schools to their iZone. Eight of the iZone schools serving elementary grades fed students into 8 of the iZone schools serving middle grade (1 school was K–8), which in turn fed their students into six iZone high schools. These schools remained under the auspices of their respective districts, but their management changed, to a district-within-a-district and these schools experienced new leadership because principals were replaced, which is consistent with the transformation approach. These schools were also given greater autonomy, and although in many cases, the schools retained more than half of their teachers, they experienced high teacher exit rates compared with other priority schools. On average, 45% of teachers exited iZone schools in their first year of operation, a lower exit rate than schools managed by CMOs, which replaced almost all teachers.

To attract and retain high-quality teachers in iZone schools, the districts offered substantial raises to teachers who remained in or transferred to these schools (Kebede, 2016). For instance, using statewide teacher-level data provided by the state Tennessee Department of Education, including salary data, we found that although the rest of the district teachers in Memphis had a 5% increase in pay in the first-year schools operated as iZone schools, teachers who stayed in or transferred into an iZone school received a 11% and 19% increase in pay, respectively.

In addition, given the emphasis on attracting or retaining highly effective teachers and, under the ASD model giving them substantial autonomy, we delve into the issue more deeply by describing the relative effectiveness of the teachers who stayed, left, and entered the ASD, iZone, and other priority schools. Table 2 compares the two measures of teachers’ effectiveness, average value-added rating (TVAAS) of teachers and the ratio of teachers with value-added scores above expectations to those with value-added scores below expectations for teachers who stayed, moved or left, or entered ASD schools (both managed directly the ASD and by CMOs), iZone schools, and non-ASD, non-iZone priority schools. ² Note that TVAAS scores are not available for teachers in their first year, teachers who have not previously taught in Tennessee public schools, or those who did not teach in tested subjects in tested grades.

OPEN IN VIEWER

Table 2

Teacher TVAAS Scores for Stayers, Movers and Leavers, and Incoming Teachers in ASD, iZone, and Other Priority Schools in Tennessee (Averaged Over All Years of Operation Within Each Group): 2012–2013 to 2014–2015

	Average TVAAS scores			Ratio of teachers with above expectations scores (5 + 4) to below expectation scores (1 + 2)
	Stayers	Movers and leavers	All incoming	Stayers	Movers and leavers	All incoming
ASD	2.97	2.95	3.34	0.81	0.94	1.50
Cohort 1	3.13	3.23	3.67	1.00	1.29	2.31
Achievement	3.04	3.32	3.85	0.90	1.44	3.67
Charter	3.60	2.82	3.33	2.00	0.80	1.14
Cohort 2	3.20	2.90	3.21	1.00	0.92	1.22
Achievement	3.20	3.05	3.31	1.00	1.00	1.50
Charter	—	2.67	3.08	—	0.80	1.00
Cohort 3 a
Charter	1.00	2.45	2.47	0.00	0.53	0.50
iZone	3.43	2.78	3.37	1.87	0.75	1.49
Cohort 1	3.59	2.97	3.26	2.33	1.00	1.40
Cohort 2	3.35	2.80	3.65	1.02	0.69	1.91
Cohort 3	2.62	2.00	2.22	0.44	0.28	0.42
Other priority b	2.95	2.76	2.80	0.95	0.72	0.77
Other Tennessee schools b	3.43	3.18	3.18	1.87	1.29	1.29

Note. TVAAS = teachers’ effectiveness, average value-added rating; ASD = Achievement School District; iZone = Innovation Zone.

a

ASD cohort 3 only contains charter-managed schools.

b

Averages for other Tennessee schools come from the same 3 years, 2012–2013 to 2014–2015.

Overall, the table shows that the ASD recruited teachers with slightly higher average scores than the teachers who left those schools, though those retained (stayers) scored below expected growth on average. However, the ASD incoming teachers had a slightly higher ratio of those exceeding expectations to those who did not meet expectations than the iZone schools. However, iZones retained and recruited teachers with higher scores than the teachers who left and retained larger ratios of teachers with above expected growth to those with below expected growth than the ASD schools. Across the ASD cohorts, it appears that the average scores of retained and incoming teachers declined. Although the initial cohort of ASD-run schools seemed to have attracted very high-performing teachers, it also appeared that the teachers who exited ASD schools have scored higher on both measures than the teachers who were retained. It may also be worth pointing out that iZone cohort 3 schools did not attract incoming teachers that were as high performing as the earlier cohorts. Finally, other priority schools in Tennessee retained slightly higher performing teachers than ASD and lower performing than iZone, whereas those teachers exiting priority schools were lower performing than those who exited ASD and iZone schools. Priority schools did not attract as high-performing teachers when compared with ASD and iZone schools. Although management of mobility of teachers who exhibit differential effectiveness is not the only means by which the management of schools can affect outcomes, it is obviously an important means, and these patterns may help interpret any effects on achievement that are found.

Data

We use a statewide student-level longitudinal administrative data set provided to us by the Tennessee Department of Education and compiled by the Tennessee Consortium on Research, Evaluation, and Development. The study time period spans from the 2010–2011 to the 2014–2015 school years and includes a unique student identifier with the school(s) students attend, the respective grades, student demographic characteristics, and test scores in English, mathematics, and science. ³

Test scores are included for two different types of exams. Students in third to eighth grades complete the Tennessee Comprehensive Assessment Program (TCAP) test each year in these subjects. TCAP test scores are standardized statewide by subject, grade, and year. At the secondary level, students complete End of Course (EOC) exams for English I, English II, English III, Algebra I, Algebra II, Biology, and Chemistry upon completion of the course, regardless of grade level. EOC test scores are, therefore, standardized statewide by subject, year, and semester. EOC scores are only included if the student is taking the exam for the first time. English III exam scores were only available for years 2011–2012 and after. Chemistry exam scores were only available for 2014–2015. For each year and subject, only one test score is included for each student. In cases in which students take TCAP and EOC in the same year, TCAP scores take precedence.

Identification of Causal Effects

In the analysis, we make two direct comparisons. The first is to compare the performance of iZone schools and schools under the auspices of ASD with priority schools, which did not undergo turnaround. iZone schools are governed and managed by their home districts, and although these schools receive significant reforms, these schools do not undergo changes in governance. In contrast, ASD schools are removed from the governance of the local districts and put under new management either by the ASD directly or a CMO. Therefore, this primary analysis examines the performance of a set of schools that remain under the governance of the district, but with a more autonomous management structure (iZone schools) as well as schools that includes both new governance and new management (ASD schools). In the second analysis, we explore the management of schools further by comparing the performance of schools under three different types of management—district-managed iZone schools, the state-managed ASD-run schools, and CMO-run schools.

An ideal approach to the analysis would assign schools randomly to the various treatments and a control group. However, such an approach is not practical given the constraints of the policy. Therefore, we use what we believe to be the next best approach—a quasi-experimental design using a difference-in-differences (DD) approach. This approach examines pre- and postachievement gains of the treatment groups relative to the pre- and postachievement gains of similarly low-performing comparison groups, which controls for secular trends in low-performing schools. The basic assumption of this approach, called the “parallel assumption,” is that, conditional on covariates, the average change in outcomes among the treatment group would have been the same as the comparison group absent the policy change. Although this assumption is not directly testable, a number of validity checks can be conducted that examine whether the assumption is plausible and the DD effect estimates provide credible causal effect estimates (Angrist & Pischke, 2009).

Following our findings, we checked the validity of the parallel trends assumption in three ways. First, the preintervention trends in the outcomes of interest for the schools in the treated and comparison schools are compared to see whether they are parallel prior to the intervention, which would strengthen the credibility of the assumption of parallel changes after treatment absent effects of reforms. Next, the possibility of an “anticipatory” or announcement effect is examined. Because the schools to be taken over by ASD or an iZone are named in the prior academic year, the school’s performance may drop due to withdrawal of effort by school personnel, who will have to compete to retain their positions within the school and may reallocate time and effort to seeking other positions. If effort rebounds the following year, the first difference could be exaggerated and appear to be greater than the change in the comparison schools due to anticipatory reactions to turnaround. Finally, we implement a validity test in the spirit of a Granger causality check (Angrist & Pischke, 2009). This check examines differences in outcomes in each of the 5 years preceding the initiation of turnaround for the ASD and iZone schools. If no differences are observed between either the ASD or iZone schools in years prior to the implementation of turnaround, but differences are found after the intervention, the attribution of the postintervention differences to the specific turnaround model is strengthened.

Below, we explain the models in greater detail, but before we do, we describe the analytic sample for our study. As noted above, the DD approach requires us to observe the performance of schools both before and after treatment. Therefore, we exclude any school where we did not have pretreatment test scores for students. In addition, we also exclude any school that did not have tested grades in the relevant school years (e.g., Grades K–2). In Table 3, we highlight the total number of schools for each treatment (i.e., Priority, iZone, and ASD, also disaggregated by ASD run and CMO run) and the number of schools included in our analysis by year. As the table suggests, the number of ASD and iZone schools have grown over time. In addition, there are still a significant number of priority schools, 28, that have not come under the auspices of an iZone or the ASD. ⁴

OPEN IN VIEWER

Table 3

Number of Schools by Reform Approach

Year	Total priority	Non-iZone, non-ASD priority	iZone	ASD schools in operation			ASD schools included in analysis
				ASD run	CMO run	Total	ASD run	CMO run	Total
2012–2013	82	65	11	3	3	6	3	3	6
2013–2014	84 a	45	22	6	11	17	5	6	11
2014–2015	77 b	28	26	5	18	23	5	11	16

Note. iZone = Innovation Zone; ASD = Achievement School District; CMO = charter management organization.

a

The increase in the total number of priority schools from 2012–2013 to 2013–2014 comes from the addition of four new ASD schools, the splitting of one school into two separate schools by the ASD, and the closure of three priority schools.

b

The decrease in the total number of priority schools from 2013–2014 to 2014–2015 comes from the addition of two new ASD schools, the creation of a second school at a former school the ASD took over in 2012–2013, the merging of two ASD schools into other ASD schools, and the closure of eight other priority schools.

Estimating the Effects of iZone and ASD Turnaround Models

As previously mentioned, some priority schools were taken over by the ASD, some joined district iZones, and the remaining underwent no systematic reforms other than the requirement to prepare school improvement plans overseen by their local district. To estimate the impact of iZone and ASD reforms, we compare iZone and ASD schools with the last group—priority schools that were not subject to interventions through the ASD or iZones. To estimate the overall impact of the iZone and ASD schools, we use Equation 1:

y i g s t = B 0 + B 1 T s + B 2 A t + B 3 T s A t + B 4 y i g s t − 1 + X i g t B j + S s t B k + d s + p g + e i g s t ,

where the dependent variable y represents the test score for student i in grade g in school s in year t. We run separate models for each of three dependent variables—reading, math, and science test scores. T is a vector of two binary variables indicating whether school s was ever in one of the two treatments (i.e., ASD, iZone) between the 2012–2013 and 2014–2015 school years. A_t is a vector of binary variables indicating whether treatment occurred in year t. Priority school status began in 2012–2013. Therefore, for our comparison schools, priority schools that have not come under the auspices of the ASD or iZones, this vector takes a value of 0 in all years. For the iZone and ASD, because they took over schools through a phase-in process, assigning the values of 1 or 0 for these schools is complicated. For both ASD and iZone schools, the first possible academic year of treatment was 2012–2013. Therefore, the binary variable indicating iZone status has a value of 1 for the first cohort of iZone schools in years 2012–2013 through 2014–2015 and 0 in years prior to 2012–2013. Similarly, the binary variable indicating ASD status has a value of 1 for the first cohort of ASD schools in years 2012–2013 through 2014–2015 and 0 in years prior to 2012–2013. However, because new ASD and iZone schools were established in 2013–2014 and 2014–2015, the binary dummy variables are modified for these schools such that it distinguishes between actual reform years and years prior to reform. For both the iZone and ASD, cohort 2 treatment started in 2013–2014. Therefore, for cohort 2 iZone and ASD schools, the binary variable indicating iZone or ASD status has a value of 0 in years prior to 2013–2014 and 1 in 2013–2014 and 2014–2015. Finally, for both the iZone and ASD, cohort 3 treatment started in 2014–2015. Therefore, the binary variable indicating iZone or ASD status has a value of 1 in 2014–2015 and 0 in prior years. We should note that although iZone phase-ins were complete school phase-ins, many ASD schools only phased in particular grades at a time. Therefore, again, the binary variable for ASD status is modified to reflect the respective grade phase-ins as well as school phase-ins.

In Equation 1, we also include a lagged test score control variable, y_igst–1, to allow for a value-added interpretation of the dependent variable. ⁵ X_igt is a vector of student characteristics for student i in grade g in year t, which includes gender, race, free and reduced price lunch status, special education status, and English language learner status. All the priority schools, including the CMO-run ASD schools, remained neighborhood schools, serving the students in their assigned catchment areas. However, as previously noted, there were some mergers of schools and closures of other priority schools. Therefore, to control for both individual student mobility and the inflow of new students into schools, we include a mobile student indicator for students who made a nonstructural move into the school that year. The inclusion of these student-level characteristics improves precision in the analysis. S_st is a vector of school-level characteristics for school s in year t, which includes the school’s percentage of minority students and the school’s percentage of free and reduced price lunch students.^6,7 Inclusion of school-level characteristics improves precision and controls for differences in school-level characteristics, including compositional changes that could be associated with student outcomes. d_s allows for a school fixed effect, and p_g allows for a grade-level fixed effect. e_igst is an error term. The school fixed effect controls for any lingering time-invariant school-level characteristics not completely controlled for through the comparison of pre- and postachievement gains of treatment and comparison groups through the DD approach. Finally, standard errors are clustered at the school level to account for lack of independence of students within schools.

In Equation 1, the coefficients of greatest interest are the coefficients of the vector of interactions between T_s and A_t, which represents an overall iZone and ASD effect for the three postturnaround years. Although the interaction variables give us the overall effect for iZone and ASD schools compared with schools that did not undergo a turnaround intervention, we also compare the performance of the two treatments. To do this, we test to see whether the coefficient estimates of the overall effects for iZone and ASD are statistically different from one another using an F test.

To examine the issue of management further, we conduct a second comparison by examining the management of schools in more discrete categories. In the second comparison, we further refine the comparisons to include an examination of each type of school management under the auspices of ASD—direct run and CMO run to the priority schools that did not undergo turnaround. For comparability, we retained the iZone schools in the sample identified by the indicator variables as in the first analysis. Therefore, in a modified analysis using Equation 1, we have three dummy variables included in the vector T_s—a dummy variable indicating whether a school is ever in the iZone, a dummy variable indicating whether a school is ever an ASD-run school, and a dummy variable indicating whether a school is ever a CMO-run school, Similarly, we modify A_t, to indicate years in which a particular school is part of the iZone, ASD-run, or CMO-run treatment. To compare the performance of three types of turnaround, we again use F tests to examine whether the coefficients of interest are statistically different from one another.

Finally, part of the challenge of turning around low-performing schools is the disadvantaged nature of the student population these schools serve. Therefore, it is informative to examine whether there are differential effects by student characteristics. To carry out this analysis, we run several new models that include interactions between student characteristics (i.e., race, free and reduced price lunch, special education, English language learner status) and by overarching treatments (i.e., ASD and iZone schools) as well by management structure (i.e., ASD-run, CMO-run schools). This can inform us whether the results vary by student characteristics, especially for disadvantaged populations.

Comparison Group Balance

Although much of the effect in the DD analysis is driven by the first difference in the treatment group, the second difference plays an essential role in generating plausible causal estimates. As a result, it is important to examine whether the comparison schools represent a strong counterfactual for each treatment. Therefore, we do pairwise comparisons of the observable student characteristics between each treatment group and the comparison group. These comparisons are analogous to randomized design studies that do “balance checks” of observable characteristics of treatment and control groups to provide insight into whether the researchers have evidence that the treatment and control subjects have been randomly assigned (Abdulkadiroglu et al., 2009; Bifulco, 2012; Cullen, Jacob, & Levitt, 2006; Engberg, Epple, Imbrogno, Sieg, & Zimmer, 2014; Hoxby, Kang, & Murarka, 2009; Zimmer & Engberg, 2016). The results, shown in Table 4, suggest that the treatment and control schools are similar on student observable characteristics for the analysis of the iZone and ASD schools with only one significant difference (in bold) in the percent minority between priority nonturnaround schools. Even in this case, the magnitude of the difference is negligible (a difference of two percentage points). Although finding no substantive differences among observable characteristics cannot exclude the possibility of unobservable differences in populations, the lack of finding significant differences provides some confidence in the appropriateness of the comparison groups.

OPEN IN VIEWER

Table 4

Comparisons of the Two Treatment Groups, ASD and iZone, and the Comparison Group, All Other Priority Schools

School characteristic	Priority—non-ASD, non-iZone	Ever ASD	Priority—non-ASD, non-iZone	Ever iZone
Proportion male	0.51	0.51	0.51	0.51
Proportion minority	0.99	0.98	0.99	0.97
Proportion FRPL	0.88	0.89	0.88	0.92
Proportion SpEd	0.15	0.15	0.15	0.17
Proportion ELL	0.02	0.02	0.02	0.02
Proportion mobile	0.31	0.36	0.31	0.30
Average reading score	−0.96	−0.98	−0.96	−1.00
Average math score	−0.95	−0.90	−0.95	−0.94
Average science score	−1.10	−1.14	−1.10	−1.16

Note. Numbers in bold indicate statistically significant differences in the respective school characteristics. ASD = Achievement School District; iZone = Innovation Zone; FRPL = free and reduced price lunch; SpEd = special education; ELL = English language learners.

Results

In Table 5, we displayed the estimated effects of schools under the auspices of ASD and iZone schools (columns 1, 3, and 5) as well as the iZone, ASD-run, and CMO-run schools (columns 2, 4, and 6). In the table (as well as subsequent tables), we presented the results as effect sizes (i.e., estimates are shown as proportions of a standard deviations) as test scores are standardized as previously discussed. To give context to these effect sizes, the average difference between the cutoffs for basic and proficient achievement levels on the TCAP in the 2013–2014 school year was 1.28 standardized units in reading, 1.19 in math, and 1.24 in science. ⁸

OPEN IN VIEWER

Table 5

Results—Overall Effects and Effects by Management

	Reading		Math		Science
	(1)	(2)	(3)	(4)	(5)	(6)
ASD overall	0.01 (0.03)		−0.01 (0.07)		−0.03 (0.07)
iZone	0.10*** (0.03)		0.20*** (0.04)		0.18** (0.06)
CMO run		−0.01 (0.03)		0.05 (0.04)		0.10 (0.06)
ASD run		0.03 (0.04)		−0.03 (0.10)		−0.11 (0.10)
iZone		0.10*** (0.03)		0.20*** (0.04)		0.18** (0.06)
Student-level demographics and lagged test scores	Yes	Yes	Yes	Yes	Yes	Yes
School-level demographics	Yes	Yes	Yes	Yes	Yes	Yes
School fixed effects	Yes	Yes	Yes	Yes	Yes	Yes
Grade fixed effects	Yes	Yes	Yes	Yes	Yes	Yes
p value of F tests
ASD overall vs. iZone	0.02*		0.00**		0.01*
CMO run vs. iZone		0.11		0.02*		0.01**
ASD run vs. iZone		0.00**		0.00***		0.22
ASD run vs. CMO run		0.26		0.42		0.06
R 2	.50	.50	.39	.39	.40	.40
Number of schools	79	79	79	79	79	79
Observations	87,141	87,141	79,615	79,615	75,201	75,201

Note. Standard errors are clustered at the school level. For a full set of coefficients, please see Table A1 of the appendix (available in the online version of the journal). ASD = Achievement School District; iZone = Innovation Zone; CMO = charter management organization.

*

Significant at 5%. **Significant at 1%. ***Significant at 0.1%.

With these magnitudes in context, the overall effects, reported in columns 1, 3, and 5, for the iZone schools across all subjects were positive, statistically significant, and substantively meaningful. In contrast, we did not observe any statistically significant overall effect for ASD schools. The effects on iZone schools were substantively and significantly larger than the effects on ASD schools across all subjects. In columns 2, 4, and 6, when we examined the results for ASD-run and CMO-run schools, we again observed no statistically significant effects for either ASD-run or CMO-run schools. In four of six cases, we observed a larger and statistically significant gain in achievement test scores for iZone schools than ASD-run (reading and math) or CMO-run (math and science) schools. In examining differences among CMO- and ASD-run schools, we did not find any statistically significant differences (although the difference in the estimates for science effect was very close with a p = .06).

Overall, these results were promising for the iZone schools, which were under district governance with modified and more autonomous district management structure. Again, this is in contrast with the ASD schools, which were no longer under the governance or the management of the district. However, it is well known that reforms can take time as research suggests that it takes 3 to 5 years for reforms to take hold (Berends et al., 2002). Therefore, we considered it worth examining the effects by cohort and academic year because we had 3 years of postintervention data for only the first cohorts of ASD and iZone schools. To carry out this analysis, in Equation 1, we modified A_t to include each year of treatment (YR2013, YR2014, and YR2015) and T_s each cohort (ASDC₁, iZoneC₁, ASDC₂, iZoneC₂, and ASDC₃, iZoneC₃). These two sets of dummy variables were then interacted together (e.g., YR2013 × iZoneC₁). In this modified Equation 1, rather than an overall or cumulative effect, the interpretation of these coefficients would be different. Because, in essence, we control for the effect for each prior year by cohort (i.e., we control for the effect of cohort 1 in the first year), the effect in the second year for cohort 1 would be considered an effect over and above the cohort 1 effect in 2013. A similar logic would apply to the third year effect for cohort 1. Also, similar logic would be employed to interpret the effect for cohort 2 schools, although these schools have only been in place 2 years, so it would only have a first and second year effect. For cohort 3, we will only estimate a first-year effect.

To further explore the effects by cohort and by year, we conducted a third variant of Equation 1, in which we further broke down these distinctions into the two different management structures offered by the ASD—ASD direct run and CMO run. This provided effects by cohort, year, and management structure, which allowed us to examine whether there were differential effects by state-management versus CMOs.

The results for the two sets of analyses are shown in Table 6. It is important to note that as we broke down the effects by cohort, by academic years, and by ASD management structures, we lost some of the power to detect statistically significant results relative to the prior analyses due to limited sample sizes. Focusing first on the iZone and ASD schools by cohort by year in columns 1, 3, and 5, we found consistent positive results for iZone schools across years—many of the effects can be deemed as substantively meaningful ranging from effect sizes of 0.12 to 0.35 of a standard deviation. For the ASD schools, the story was more complex with cohorts in most years having no effect, whereas other effect estimates suggested a positive and statistically significant effect for particular cohorts, in particular years and subjects, and still other results suggested a significant negative effect in particular cohorts in particular years and subjects. It was also notable that the positive and negative estimates were generally large in magnitude, including a negative math effect of 0.50 standard deviations for year 1 of cohort 2 ASD schools whereas schools in year 3 in cohort 1 experienced a gain of 0.24 standard deviations in science. Similarly, when the analysis was broken down for ASD schools by CMO-run and ASD-run schools in columns 2, 4, and 6, we again saw inconsistent results as we observed mainly statistically insignificant effects as well as four positive and five negative estimates that were mostly large and statistically significant with more positive and significant effects for the ASD-managed schools.

OPEN IN VIEWER

Table 6

Results—Overall Effects by Cohort by Year and Effects by Management by Cohort by Year

	Reading		Math		Science
	(1)	(2)	(3)	(4)	(5)	(6)
ASD cohort 1, Year 1	−0.01 (0.04)		0.05 (0.07)		0.20* (0.10)
ASD cohort 1, Year 2	0.04 (0.08)		0.05 (0.10)		−0.01 (0.13)
ASD cohort 1, Year 3	0.04 (0.04)		0.02 (0.14)		0.24* (0.12)
ASD cohort 2, Year 1	−0.06 (0.13)		−0.50*** (0.08)		−0.22 (0.16)
ASD cohort 2, Year 2	0.04 (0.06)		0.17* (0.07)		0.15 (0.12)
ASD cohort 3, Year 1	0.01 (0.04)		−0.03 (0.24)		−0.27* (0.11)
iZone cohort 1, Year 1	0.12*** (0.03)		0.22*** (0.04)		0.20** (0.07)
iZone cohort 1, Year 2	0.14** (0.05)		0.31*** (0.05)		0.24** (0.07)
iZone cohort 1, Year 3	0.13** (0.04)		0.16* (0.07)		0.26** (0.08)
iZone cohort 2, Year 1	0.18*** (0.05)		0.35*** (0.08)		0.22 (0.12)
iZone cohort 2, Year 2	0.13*** (0.04)		0.18* (0.08)		0.30** (0.09)
iZone cohort 3, Year 1	−0.03 (0.04)		0.09 (0.06)		0.16*** (0.04)
ASD CMO-run cohort 1, Year 1		−0.06 (0.04)		0.06 (0.06)		0.33*** (0.10)
ASD CMO-run cohort 1, Year 2		0.13 (0.11)		0.21 (0.11)		0.10 (0.12)
ASD CMO-run cohort 1, Year 3		0.06 (0.06)		−0.17 (0.13)		0.13 (0.14)
ASD CMO-run cohort 2, Year 1		−0.19 (0.13)		−0.46* (0.20)		−0.45** (0.14)
ASD CMO-run cohort 2, Year 2		0.05 (0.09)		0.14 (0.09)		0.04 (0.10)
ASD CMO-run cohort 3, Year 1		0.01 (0.04)		−0.03 (0.24)		−0.27* (0.12)
ASD ASD-run cohort 1, Year 1		−0.01 (0.04)		0.06 (0.08)		0.17 (0.11)
ASD ASD-run cohort 1, Year 2		−0.06 (0.07)		−0.10 (0.05)		−0.11 (0.19)
ASD ASD-run cohort 1, Year 3		0.01 (0.03)		0.27* (0.11)		0.41*** (0.09)
ASD ASD-run cohort 2, Year 1		0.09 (0.11)		−0.36*** (0.04)		−0.09* (0.04)
ASD ASD-run cohort 2, Year 2		0.01 (0.06)		0.38*** (0.05)		0.22 (0.19)
iZone cohort 1, Year 1		0.12*** (0.03)		0.22*** (0.04)		0.21** (0.07)
iZone cohort 1, Year 2		0.14** (0.05)		0.31*** (0.05)		0.23** (0.07)
iZone cohort 1, Year 3		0.13** (0.04)		0.16* (0.07)		0.26** (0.08)
iZone cohort 2, Year 1		0.18*** (0.05)		0.35*** (0.08)		0.22 (0.12)
iZone cohort 2, Year 2		0.13*** (0.04)		0.18* (0.08)		0.30** (0.09)
iZone cohort 3, Year 1		−0.03 (0.04)		0.09 (0.06)		0.16*** (0.04)
Student-level demographics and lagged test scores	Yes	Yes	Yes	Yes	Yes	Yes
School-level demographics	Yes	Yes	Yes	Yes	Yes	Yes
School fixed effects	Yes	Yes	Yes	Yes	Yes	Yes
Grade fixed effects	Yes	Yes	Yes	Yes	Yes	Yes
R 2	.50	.50	.39	.39	.41	.41
Number of schools	79	79	79	79	79	79
Observations	87,141	87,141	79,615	79,615	75,201	75,201

Note. Standard errors are clustered at the school level. For a full set of coefficients, please see Table A2 of the appendix (available in the online version of the journal). ASD = Achievement School District; iZone = Innovation Zone; CMO = charter management organization.

*

Significant at 5%. **Significant at 1%. ***Significant at 0.1%.

Therefore, we generally concluded that the results for schools under the auspices of the ASD, as a whole and disaggregated by management structure, have been somewhat inconsistent but mainly are not sufficiently precise to conclude that they are different than zero, that is, not different from the comparison priority schools, which did not undergo turnaround during the period. We also did not observe a consistent pattern of these schools improving over time. This is in contrast with iZone schools, which exhibited a consistent pattern of meaningful positive effects overall and across time.^9,10

Overall, the analyses suggest that iZone schools have shown promising results whereas ASD schools have not, at least yet. As noted previously, a major motivation for school turnaround policies is to improve student achievement of disadvantaged populations. To explore whether there were differential effects by student characteristics, we interacted student characteristics of race, free and reduced price lunch, special education, and English language learner status with the treatment variables of ASD and iZone status as well as the secondary analyses in which we interacted student characteristics with the management structure of ASD-run, CMO-run, iZone schools. Across these various analyses, no consistent patterns emerged. In fact, in general, the coefficients for the interaction terms were small and rarely statistically significant. Because of the lack of substantively meaningful results, and to conserve space, we do not show the results here, but the overall conclusion is that there does not seem to be differential effects by student characteristics across the various treatments.

For evaluations of educational interventions that do not rely on random assignment to treatment, it is customary to examine the robustness of the preferred causal estimates to alternative estimation procedures. We implemented three alternative causal effects identification strategies: (a) adding student fixed effects in place of the lagged student test scores and demographics, (b) substituting an indicator of student proficiency for the student test score value as the outcome of interest, and (c) comparative interrupted time series (CITS) with school-level trends. Student fixed effects (see Appendix A3, available in the online version of the journal) controls for any student characteristics that did not vary during the study period and estimated statistically significant effects for iZone schools of 0.14 for reading, 0.21 for math, and 0.15 for science, which are qualitatively consistent with the effect estimates in the preferred specification. Proficiency of individual students aggregated to school proficiency rates are often used as the measures of school performance provided to the public and, although they only take the measure at one point in the distribution of scores, they provide a salient additional outcome measure. Linear probability model estimates (see Appendix Table A4) substituting an indicator of student proficiency for the test score measure and removing the lagged test score value from the preferred specification produced statistically significant effects of a three percentage point average proficiency gain for reading, a nine percentage point average gain for math, and a seven percentage point average gain for science for the iZone schools, which are again consistent with the effect estimates in the preferred specification. These proficiency gains are over and above the three to 12 percentage point gains among all schools in the posttreatment period.

As a final robustness check, we were able to add four additional prereform years of data to our analysis and estimated effects of the reform using a CITS approach. We do not feature this as the preferred specification because the achievement tests in Tennessee changed 2 years prior to the reform and to obtain the longer prereform series, different assessments had to be standardized by grade and year for the three content domains. In most cases, this produces a “sawtooth effect” in the trends with a drop the year the change occurs, which may bias the effect estimates. The CITS with a school-specific trend that allowed the slope of the trend line to vary after the reform began (see Appendix A5) returned statistically significant effects for iZone schools of 0.13 for reading and 0.20 for math but a statistically insignificant estimate of 0.12 for science, the first two of which are qualitatively consistent with the effect estimates in the preferred specification. The standard error on the CITS effect estimate was 1, one half times as large as the standard error on the effect estimate for the preferred specification, which may be attributed to the inflated variance from the interactions and the lack of significance due to imprecision. Overall, these results suggest that our estimates for iZone schools are robust across the various models. Moreover, our findings for the schools under the auspices of the ASD were largely unchanged with the lone exception of the student fixed effects model returning a small negative overall effect in science (−0.08), which appears to be due to the CMO-run schools (−0.13).

Therefore, we reached a similar overall substantive conclusion—it was not necessary for low-performing schools to be removed from the auspices of a district for low-performing schools to experience meaningful improvement as we observed meaningful achievement effects for turnaround reforms within the district’s controls using our preferred DD approach as well as adding a student fixed effect, using a CITS with school-specific trends, and substituting proficiency for the test score as the outcome of interest.

Validity Checks

As we described above, we tested the validity of the parallel trends assumption in three ways: (a) examine preintervention trends, (b) test for an anticipatory reaction to turnaround, and (c) “Granger” test for differences in the 5 years prior to turnaround. In addition to these checks, we also tested for any effects of the turnaround interventions on the makeup of these schools (school-level covariates used as adjustments in the DD models) that may signal a change in the desirability of enrolling in these schools. Finally, we provide a check on efforts to cream-skim or push out certain groups of students such that student mobility may bias the effect estimates. Critics of charter schools have often claimed that charter schools try to improve the academic profile of their schools as well as reduce costs by recruiting high-ability students and pushing out low-ability students (Ravitch, 2010). ¹¹

First, we examine the trends in the outcomes of interest with particular interest in the trends prior to implementation of turnaround. If outcome trends were parallel prior to turnaround, it supports that the trends were likely to be similar after treatment implementation, except for the response to turnaround. Unfortunately, the current state achievement tests were only employed 3 years prior to the year treatment started as the state adopted a new state test in the 2009–2010 school year. However, we did have student-level data dating back to the 2006–2007 school year using the prior state accountability test. Because these tests employed different standards, it may not be appropriate to use these tests as outcome measures in our primary analyses. However, as with their previous use in a robustness check, we argue that these tests can be useful in examining pretreatment trends as a validity check. To implement this check, we again standardized both the previous and current standardized tests as previously described so that the tests are on a common metric. Because of the phase-in process of the ASD and iZone schools and to maintain a sufficiently large sample size, we centered the years such that Year 0 is the year prior to treatment and Year 1 is the first year of treatment (i.e., Year 0 and Year 1 will be different calendar years for different cohorts of ASD and iZone schools). The results for the trend analyses are presented in Figure 1. The pretreatment trends of the treatment groups and control schools were relatively similar, with slight deviations in Year 0 for math (all treatment groups crossed the comparison group between Year −1 and Year 0 and some crossovers occurred among the treatment groups in Year −1 in science but no crossovers with the comparison group and any treatment group). Overall, the pretreatment trends provide strong support for the parallel assumption for the reading results but may raise slight concerns for the math and science results. However, the magnitude of the change in trends for iZone schools pale in comparison with the effects we observe during treatment.

OPEN IN VIEWER

Figure 1.

Pretreatment trends.

Second, we tested for an anticipatory or announcement effect by artificially assuming that a treatment begins a year before treatment actually began and using the same model implemented for estimating the effects (Equation 1). If the anticipation of turnaround causes test scores to drop in the year before turnaround begins, a rebound in scores could masquerade as a treatment effect in the first year of treatment. It appears that the possibility of an “announcement” effect has been largely ignored by most of the contemporaneous papers examining similar school reform policies, which could potentially have caused an upward bias of their estimates. In examining the results in Table 7, we do not observe any statistically significant effect—either positive or negative. These results should minimize concerns about an announcement effect and provide some support for the DD approach.

OPEN IN VIEWER

Table 7

Anticipatory or “Announcement” Validity Check: Estimates of Effects in the Year Prior to the Implementation of Turnaround

	Reading		Math		Science
	(1)	(2)	(3)	(4)	(5)	(6)
ASD overall	0.06 (0.04)		0.02 (0.08)		0.04 (0.08)
iZone	0.03 (0.03)		0.08 (0.09)		−0.05 (0.04)
CMO run		0.08 (0.05)		0.02 (0.10)		0.05 (0.09)
ASD run		0.02 (0.06)		0.05 (0.11)		0.01 (0.11)
iZone		0.03 (0.03)		0.08 (0.09)		−0.05 (0.04)
R 2	.49	.49	.37	.37	.36	.36
Number of schools	79	79	79	79	79	79
Observations	37,512	37,512	34,468	34,468	32,195	32,195

Note. Standard errors are clustered at the school level. ASD = Achievement School District; iZone = Innovation Zone; CMO = charter management organization.

*

Significant at 5%. **Significant at 1%. ***Significant at 0.1%.

Although the check for an anticipatory reaction using 1 year of prior achievement gains undermines the possibility of an announcement effect, which could bias the effect estimates upward, a Granger test was conducted to assess the possibility that differences could have existed in prior years, which may reappear after turnaround occurred. We added data for ASD, iZone, and comparison schools for the period 5 years prior to the initiation of treatment and included a maximum of 3 years in which any of these schools received treatment. Because we added data described above from a different state testing regime, we conducted this test separately from the test of anticipatory effects, which used the same measures pre- and postturnaround. The results, which appear in Table 8, show no statistical differences between ASD and the comparison schools or iZone and the comparison schools for the 5 years before they entered treatment for the reading or math tests, and in the case of ASD for the 3 years after the schools had been placed in the ASD, which strengthens the credibility of the DD estimates as causal effects. For iZone, the science test score gains are approximately 10% of a standard deviation below those for the comparison schools 2 years and 1 year prior to the implementation of turnaround although too imprecisely estimated to be statistically significant.

OPEN IN VIEWER

Table 8

Differences in Achievement Gains in Either ASD or iZone Schools and the Other Priority Schools From 5 Years Prior to Turnaround Through 3 Years After

	Reading	Math	Science
ASD−5	0.010 (0.028)	0.026 (0.062)	−0.060 (0.051)
ASD−4	0.016 (0.043)	0.089 (0.059)	0.019 (0.057)
ASD−3	−0.014 (0.051)	−0.052 (0.083)	−0.013 (0.074)
ASD−2	−0.008 (0.034)	−0.014 (0.069)	−0.120 (0.068)
ASD−1	0.025 (0.056)	0.053 (0.073)	−0.013 (0.064)
ASD1	0.001 (0.048)	−0.035 (0.099)	−0.076 (0.116)
ASD2	0.043 (0.054)	0.176 (0.069)	0.004 (0.087)
ASD3	0.065 (0.041)	0.095 (0.157)	0.181 (0.126)
iZone–5	−0.012 (0.039)	−0.037 (0.053)	−0.065 (0.046)
iZone–4	0.017 (0.047)	0.035 (0.062)	−0.025 (0.048)
iZone–3	0.002 (0.040)	0.005 (0.067)	−0.024 (0.052)
iZone–2	−0.066 (0.041)	−0.096 (0.072)	−0.098 (0.050)
iZone–1	−0.075 (0.045)	0.019 (0.072)	−0.118 (0.064)
iZone1	0.038 (0.049)	0.157 (0.083)	0.076 (0.067)
iZone2	0.088 (0.045)	0.246*** (0.070)	0.177* (0.079)
iZone3	−0.016 (0.052)	0.105 (0.085)	0.104 (0.069)
R 2	0.44	0.39	0.36
Number of schools	79	79	79
Observations	144,902	131,803	127,073

Note. Standard errors are clustered at the school level. ASD = Achievement School District; iZone = Innovation Zone.

*

Significant at 5%. **Significant at 1%. ***Significant at 0.1%.

We next examine whether the “treatment” had any effect on the makeup of the schools by replacing the outcome of interest (i.e., test scores) in the DD approach with observed student characteristics aggregated to the school level, and examine whether we observe any change in these observable characteristics. Here, we examined the possibility that the reforms or the act of these schools being identified as part of these treatments may affect the desirability for students to enroll in these schools. In Table 9, we did not observe any statistically significant effects, which again, provided support for the DD approach.

OPEN IN VIEWER

Table 9

Testing for the Effect of Treatment on School-Level Characteristics

	ASD	iZone
Male	0.001 (0.010)	0.001 (0.007)
Minority	0.004 (0.004)	0.002 (0.002)
FRPL	−0.001 (0.035)	0.011 (0.017)
ELL	−0.003 (0.004)	0.002 (0.005)
Special Ed	−0.003 (0.007)	−0.003 (0.005)

Note. Standard errors are clustered at the school level. ASD = Achievement School District; iZone = Innovation Zone; FRPL = free and reduced price lunch.

*

Significant at 5%. **Significant at 1%. ***Significant at 0.1%.

Of greatest interest was whether these schools potentially changed the quality of the students by recruiting high-achieving or pushing out low-achieving students. In Figure 2, we examined the relative performance of students either moving in or moving out of priority, ASD, and iZone schools between years. Generally, the average standardized reading score of students transferring into each school type between the 2013–2014 and 2014–2015 school years was about the same as those transferring out with slight deviations in cohorts 1 and 2 ASD schools (and the patterns were different across these two cohorts). We conducted t tests between those transferring in and those transferring out within each school type. Only priority schools yielded a statistically significant difference, which serves as control set of schools in the analyses. However, this was heavily due in part to the large sample size of this particular school type. The results from this table, therefore, suggest that the results from the DD analysis should not be biased.

OPEN IN VIEWER

Figure 2.

Standardized reading scores for between year movers: 2013–2014 to 2014–2015.

In terms of gaining understanding of whether any of the above gaps in prior year test scores would make any difference in the academic profile of individual schools, we conduct what we believe is a unique analysis in Figure 3. In the figure, we focus on proficiency of students (because that is what is publicly reported for schools and the measure in which schools are held accountable) and take the number of incoming students that are proficient minus the number of outgoing proficient students and divide this net sum by the estimated total number of tested students enrolled at the school, consisting of those who transferred out, those who transferred in, and those who did not move. The final value is the net gain or loss of proficient students. Unlike the analysis presented in Figure 2, this analysis takes into account prior achievement scores and the number of the students entering and exiting a school, which provided a measure of the net impact these moves could have had on the school proficiency levels. In other words, the analysis provided insights into whether the school is improving (or diminishing) their academic profile through student transfers. A large positive value would be consistent with acting on the distorted incentive of improving school performance through encouraging high-performing students to transfer into the school and/or encouraging low-performing students to transfer out of the school. The figure suggested no major effects from students transferring on the rate of proficiency as there were no cases in which the proficiency-level changes (either positive or negative) by more than 2%. Overall, the analysis provided no evidence that the schools were strategically recruiting or pushing out students in hopes of improving their academic profile.

OPEN IN VIEWER

Figure 3.

Average effect of mobile students on reading proficiency rates across all students.

Conclusion

In a speech in 2009, then Secretary of Education Arne Duncan suggested that there were approximately 5,000 chronically underperforming schools in the United States and to transform these schools, we needed to institute major interventions, not “tinkering” (Dee, 2012). As a result, the U.S. Department of Education invested in more fundamental and potentially disruptive reforms through RttT and SIG to encourage states and districts to adopt turnaround polices to improve the performance of low-performing schools. These reforms focused less on specific instructional practices and more on governance and management of schools, personnel replacement, and operational reform along with providing additional resources to aid the development of a well-functioning educational infrastructure. Part of the theory for these reforms is that low-performing schools and the districts that manage them do not have the capacity or will to enact fundamental changes to schools, including major changes in staffing (Chubb & Moe, 1990). These reforms often led to changes in both governance and management of low-performing schools in which states took over low-performing schools and either partnered with CMOs or mandated that districts provide schools with significant autonomy, and put in place significant management, staffing, and operational changes. However, in some states, the reforms allowed districts to maintain governance of low-performing schools with a modified management structure. Although a number of studies are beginning to examine these reforms, these studies have not compared the effectiveness of the reforms under the governance of the state and management by nondistrict operators (such as the state or CMOs) with reforms that allowed districts to retain governance, but alter management and make changes in personnel, operations, and resources—for example, iZones.

In this article, we examined both types of reforms. We found little evidence for the improvement of schools when removed from district governance and managed by CMOs and evidence of only slight, occasional improvement for schools under state auspices and managed by the state. However, we do observe significant improvement for schools that remained under the auspices of the district, but were placed in a special district-within-a-district and granted greater autonomy and additional resources. We should note, however, that prior research suggests that reforms take time to take hold and that many of the schools managed either by the ASD or CMOs have been under new management for less than 3 years. Therefore, it may be premature to draw definitive conclusions about the effectiveness of management of schools either through the state or CMOs. Nevertheless, the results provide promise for turnaround practices in which districts retain governance over schools, but schools are given greater managerial autonomy and use additional resources for recruiting and retaining effective teachers and implement other reforms required under the federal transformation model. Although many who advocated for these reforms would argue that the positive results experienced by iZone schools would not have been possible without the pressure created by state takeover and the use of CMOs (Glazer & Egan, 2016), the analysis, at the very least, suggests that it is not necessary for these schools to be managed outside of the districts to experience significant improvement. Moreover, the analysis of the recruitment and retention of effective teachers that we presented suggested that districts, when pressured to manage chronically low-performing schools for improvement, have the capacity to do so through selective teacher retention and recruitment policies. These policies appeared to have been based on incentive pay for effective teachers who were recruited or retained in these schools and performance pay for teachers, principals, assistant principals, and other certified and noncertified staff.

Given these results, a number of states currently considering an “ASD-like” approach (including Nevada, North Carolina, and South Carolina) should consider whether it is necessary for schools to be managed by outside providers to experience significant improvement. If these states consider an “iZone alternative” (for instance, given these results, policymakers in North Carolina have authorized an “iZone-like approach”), then it would be helpful to know the mechanisms these schools employed to gain these results. Although we lack the data to definitively answer this question, one strategy these schools utilized was an increase in teachers’ pay to retain and attract high-quality teachers. This does raise the question of whether the iZone approach of recruiting and retaining highly effective teachers is scalable if there is a fixed pool of such teachers who are willing to work in the lowest achieving schools. Evidence consistent with this concern is found in Table 2, which shows that the third cohort of iZone and ASD schools appears to be less able to recruit and keep effective teachers. In addition, it raises the question of what effect the approach could have on schools losing the high-quality teachers. Although our findings indicate that turnover among the ASD schools was higher than the iZone and other priority schools and that more effective teachers were more likely to leave the ASD schools, the reasons for the turnover could not be established with the data available for this study. Prior research indicates that particularly in charter schools, turnover appears to be associated with a decline in teachers’ trust in the principal, salary and benefits, and difficult working conditions such as the heavy workload and management of student discipline (Gross & DeArmond, 2010; Malloy & Wohlstetter, 2003; Miron & Applegate, 2007; Stuit & Smith, 2010; Torres, 2016, 2016; Torres & Oluwole, 2015). However, the prevalence of Teach For America teachers in schools managed by CMOs, their 2-year service commitment, and their associations with departing from teaching more quickly than most other novice teachers (Henry, Bastian, & Smith, 2012) may be another credible hypothesis. Given that these concerns lie mainly with management, it is possible that any positive effect of the changes in governance of the ASD, particularly CMO-run schools, may have been undermined by management practices. Therefore, we do not argue that change in governance is more or less important than change in management, but that change in management without change in governance is sufficient for significant improvement. Finally, the effectiveness of the iZone should focus attention on the questions of whether teachers’ effectiveness can be enhanced and developed through teacher evaluation and incentive policies (Taylor Eric & Tyler John, 2012) and selective retention (Dee & Wyckoff, 2013). These issues are beyond the scope of the current article but the effects of teacher recruitment, retention, development, and leaving on both turnaround schools and the schools from which these teachers are recruited should be a focus of future research.