Abstract
The federal School Improvement Grant (SIG) program allocated US$7 billion over nearly a decade in an effort to produce rapid and lasting improvements in schools identified as low performing. In this article, we use a regression discontinuity design to estimate the effect of Ohio’s SIG turnaround efforts on student achievement and school administration. The results indicate that Ohio’s SIG program significantly increased reading and math achievement, with effects in both subjects of up to 0.20 standard deviations in the second year after SIG eligibility identification. Estimates for the third year are somewhat larger, in the range of one quarter of a standard deviation. We provide evidence that these effects were primarily attributable to schools that implemented the SIG Turnaround model. We also show that SIG eligibility had a positive effect on per-pupil spending, but no average effect on administrative outcomes, including staff turnover, the number of staff members in the school, and school closure. These null overall effects mask heterogeneity across SIG models, however. Most notably, Turnaround schools experienced more turnover than they otherwise would have, whereas Transformation schools experienced less.
Keywords
Introduction
We estimate the effect of Ohio’s SIG turnaround efforts on student achievement and school administration. In particular, we estimate the effect of SIG eligibility on student achievement levels, per-pupil funding levels, staff turnover, and school closure. The focus on the effect of SIG eligibility—the intention-to-treat (ITT) parameter—provides insight into the effects of the SIG program. However, because nearly all SIG-eligible schools in Ohio received SIG awards, the estimated effects of SIG receipt—the treatment-on-the-treated (TOT) parameter—are very similar. We estimate these effects using a regression discontinuity (RD) design, exploiting the SIG eligibility criterion stating that schools qualified for these interventions if they ranked in the bottom 5% of eligible schools in terms of a weighted student proficiency rate calculated by the Ohio Department of Education (ODE). Our analyses are based on data contained in multiple sets of records provided by ODE via the Ohio Education Research Center (OERC), including individual-level records for all students, teachers, and principals in Ohio public schools between the 2008–2009 and 2014–2015 school years. We supplement these records with a wide range of annual school- and district-level data spanning the same time period.
Our analyses show that Ohio’s SIG program had large positive effects on student achievement. These effects first emerge in the second year after a school is identified as SIG eligible, with reading effects in excess of 0.11 SDs and math effects of more than 0.14 SDs. In the third year, SIG eligibility is estimated to increase reading achievement by at least 0.20 SDs and math achievement by at least 0.18 SDs, and some specifications suggest achievement gains of up to 0.26 SDs in math and 0.27 SDs in reading. These achievement results are robust to a variety of alternative analytic decisions and model specifications.
The positive achievement effects coincide with a significant infusion of cash into SIG-eligible schools. We estimate that SIG eligibility increases annual per-pupil funding by US$1,500 to US$3,000 across the 3-year grant period, which suggests that SIG funds did not supplant district funds. Furthermore, we provide evidence that these positive achievement effects were driven by schools that implemented SIG’s Turnaround model, as opposed to the Transformation model. Interestingly, our analysis also finds SIG eligibility to have little effect on several other school administrative outcomes, including staff turnover, the number of staff members in the school, and school closure. However, for several of these outcomes, we provide evidence of heterogeneous effects across schools that implemented the SIG Transformation model and those that implemented the Turnaround model.
We proceed by first providing context for our analysis, reviewing existing research relevant to school turnaround initiatives and describing Ohio’s SIG program in detail. We then describe the data and the RD approach we use to estimate the effects of the turnaround interventions. Next, we present the results, showing the effects of the turnaround intervention on student achievement and school administration. Finally, we conclude the article by discussing the implications of our results for school turnaround policy, as well as for research on these initiatives.
Background on School Turnaround Initiatives
There is a long history of efforts to improve the quality of low-performing schools in the United States. Slavin (1989) characterized many of the early efforts as piecemeal—targeting particular aspects of educational delivery or specific student populations—and generally unsuccessful, due at least in part to the short-lived and fragmented nature of many of these initiatives (see also Gross, Booker, & Goldhaber, 2009; Hess, 1998). Partially in response to the perceived failure of this scattershot approach, comprehensive school reform (CSR) programs steadily gained popularity beginning in the late 1980s and extending through the 2000s. These initiatives were based on the notion that coordinated efforts to improve schools as a whole were more likely to have a meaningful impact than the targeted and disjointed efforts that preceded them. 1 These federal grant programs—administered primarily under Title I of the Elementary and Secondary Education Act (ESEA)—were quite specific about which interventions qualified as CSR, stipulating that they must involve evidence-based strategies for improving everything from school management and instruction to fostering parental and community involvement. Research examining the impact of various CSR programs on student achievement found mixed results. Although many studies were of limited quality (Herman et al., 2008), there is good evidence that some CSR models had a positive impact on achievement (e.g., see Bifulco, Duncombe, & Yinger, 2005; Borman, Hewes, Overman, & Brown, 2003; Gross et al., 2009).
The encouraging evidence emerging from CSR evaluations contributed to the federal government’s decision in the late 2000s to incentivize implementation of more aggressive school turnaround models. 2 The U.S. Department of Education (USED) encouraged the adoption of such policies through the promise of additional financial resources or waivers from certain accountability provisions of No Child Left Behind (NCLB). This set of more aggressive turnaround strategies includes closing a school completely, restarting a school as a charter school or one managed by an education management organization, or, to various degrees, reconstituting a school’s leadership and instructional staff through mandatory and data-driven hiring and firing processes.
A body of evidence suggests that these turnaround models could lead to improvements in student achievement. Research on school closure and charter conversions, for example, indicates that closure can benefit students in failing schools, provided that affected students switch to schools of sufficiently higher quality to compensate for disruptions induced by closure or conversion (Abdulkadiroğlu, Angrist, Hull, & Pathak, 2016; Bross, Harris, & Liu, 2016; Brummet, 2014; Carlson & Lavertu, 2015; Carlson & Lavertu, 2016a). The turnaround models requiring reconstitution of leadership and staff are supported by work demonstrating the importance of principals (Branch, Hanushek, & Rivkin, 2012; Grissom, Kalogrides, & Loeb, 2015) and teachers (Chetty, Friedman, & Rockoff, 2014; Hanushek, 2011; Rivkin, Hanushek, & Kain, 2005). Perhaps most encouraging for supporters of these models, though, is work using data from the District of Columbia’s teacher evaluation system, which shows that the replacement of low-performing teachers with higher quality ones increased student achievement by a statistically significant and substantively meaningful magnitude (Adnot, Dee, Katz, & Wyckoff, 2017). Finally, the general focus on data-driven decision making in these turnaround models also has support in the literature (Carlson, Borman, & Robinson, 2011; Strunk & McEachin, 2014; Strunk, McEachin, & Westover, 2014).
Although research suggests that turnaround strategies could lead to improved student outcomes, it is not a foregone conclusion. Several studies have found school closure to have no meaningful effect on student achievement, apparently because students do not end up in schools of sufficiently greater quality (Brummet, 2014; de la Torre & Gwynne, 2009; Engberg, Gill, Zamarro, & Zimmer, 2012). Similarly, the effectiveness of turnaround models requiring principal or teacher replacement depends on the supply of human capital. If the supply of educational personnel—teachers, principals, or school staff—is low or recruitment is difficult, as tends to be the case in low-achieving, high-poverty urban and rural districts (e.g., see Boyd, Lankford, & Wyckoff, 2008; Clotfelter, Ladd, & Vigdor, 2007, 2010; Cowen, Butler, Fowles, Streams, & Toma, 2012; Jackson, 2009), personnel could very well be of comparable or lower quality than those they replace. Even if existing staff is replaced with personnel of comparable quality, such “churn” has itself been shown to have a negative impact on student achievement (Atteberry, Loeb, & Wyckoff, 2017).
Overall, research does not provide an unambiguous prediction concerning the effects of school turnaround efforts on student outcomes. Instead, the evidence suggests that the utility of these interventions is likely to be context specific.
SIGs in Ohio
Authorized under Title I of the ESEA, the SIG program allocates formula-based grants to state education agencies (SEAs). The SEAs then make competitive awards to local school districts that demonstrate a compelling plan for using the funds to improve student achievement in SIG-eligible schools. Districts’ plans must adhere to federal SIG requirements, which were significantly revamped in 2009. This redesign coincided with a one-time influx of US$3 billion to the program through the American Recovery and Reinvestment Act (ARRA) in addition to the regular annual appropriation of approximately US$500 million. Of this $3.5 billion, Ohio received more than US$130 million to award to districts with eligible schools. The state distributed this money to three cohorts of districts and their eligible schools, with each cohort receiving grant dollars over a 3-year period. According to data from ODE, the median annual SIG award for each school was about US$800,000, or about US$2,200 per pupil, and more than half of that was dedicated to personnel salaries and fringe benefits.
Prior to receiving funds from the USED, ODE was required to provide USED with a list of SIG-eligible schools and outline the criteria they would use to award grants to districts. The SIG eligibility criteria were complex, consisting of the three eligibility tiers summarized in Table 1. In general, a school’s eligibility was a function of its Title I receipt status under NCLB’s school improvement process, grades served (i.e., elementary or middle school vs. high school), achievement levels, and graduation rates. Although schools in all three tiers were eligible to receive SIG funds, program requirements specified that districts prioritize Tier 1 and Tier 2 schools over those in Tier 3. 3
School Improvement Grant Eligibility Criteria, by Tier
Note. NCLB = No Child Left Behind.
The achievement measures determining Tier 1 and Tier 2 SIG eligibility are instrumental to our analyses below and, thus, warrant further explanation. Consistent with federal guidelines, ODE determined the lowest achieving 5% of schools using an average of two proficiency calculations: (a) a weighted proficiency rate in math and reading for each building in the most recent school year and (b) an average of this weighted proficiency rate over the five most recent school years. ODE then rank ordered schools based on a “combined proficiency rate” that weighted these two proficiency calculations equally. 4 These ranking rules were applied separately for Tier 1 and Tier 2 schools and for each cohort.
After identifying all schools eligible to receive SIG funds, ODE invited districts to submit applications that demonstrated commitment and capacity to improve student achievement in SIG-eligible schools. These applications included detailed narratives and budgets for each school that would receive funds. SIG regulations required Tier 1 and Tier 2 schools awarded SIG funds to implement one of four federally approved turnaround models: Closure, Restart, Transformation, or Turnaround. The Closure model required a school to shut down permanently, whereas the Restart model mandated that a school convert to a charter school or one operated by an education management organization. The Transformation and Turnaround models both required replacing the principal and implementing a number of instructional and operational reforms. The Turnaround model further entailed replacing at least 50% of the school staff, whereas the Transformation model required adopting a new governance model and an educator evaluation system that included student achievement growth as a significant component. 5 Table 2 summarizes the three cohorts of SIG grants in Ohio. In particular, it lays out the timing of SIG eligibility identification and grant administration, depicts the number of districts and schools receiving SIG awards, and presents the number of Tier 1 and Tier 2 schools that implemented each of the four SIG turnaround models. The table shows that the first cohort of schools was identified as eligible in the 2009–2010 school year—we consider this the first treatment year for the cohort—and grant funds were first administered the following year. The second cohort of SIG awards was made in 2010–2011, but there was then a 3-year gap between the second and third SIG cohorts. Table 2 further shows that ODE made awards to 39 schools in 13 districts in Cohort 1, 44 schools in 29 districts in Cohort 2, and 24 schools in 11 districts in Cohort 3—we cannot include Cohort 3 in our analysis for reasons we describe in greater detail below. It also demonstrates that Tier 1 and Tier 2 schools overwhelmingly elected to implement the Transformation and, to a lesser extent, Turnaround models.
Ohio SIG Award Process, by Cohort
Note. The “Other” category includes Tier 3 schools that were not required to implement a turnaround model, as well as a small number of schools that closed shortly after identification and one school that implemented both Turnaround and Transformation models. SIG = School Improvement Grant.
Data
The analyses are based on data contained in multiple sets of records. First, for each of the first two SIG cohorts, ODE provided the combined proficiency rate calculations used to rank schools to determine eligibility. These calculations are critical to the research design we employ. Unfortunately, the 3-year gap between the second and third SIG cohorts, coupled with the terms of the evaluation with ODE that specified a focus on the first two cohorts, prevents us from including the final cohort in our analysis. Second, ODE provided a wide variety of annual school-level information from the 2006–2007 through 2014–2015 school years, including enrollment, demographic and socioeconomic composition, expenditures, attendance and mobility rates, and characteristics of the teaching staff. We also obtained several annual school-level achievement-related measures, including average standardized achievement levels in reading and math, scores on Ohio’s performance index, and estimated value added in reading and math. The performance index has a 0 to 120 scale and captures the achievement level of a school’s students across multiple subjects (math, reading, writing, science, and social studies) on state assessments. School value-added estimates are calculated annually for use in Ohio’s accountability system and are available from ODE in normal curve equivalent (NCE) units, which we converted to SD units for use in our analyses. 6
Third, ODE provided student-level data via the OERC. These records include identifiers for the schools and districts that students attended, as well as information on student demographic characteristics, such as sex, race/ethnicity, economic disadvantage, disability status, and English language learner (ELL) status. They also include students’ scale scores on the Ohio Achievement Assessment (OAA)—administered in Grades 3 to 8—which we standardized by subject, grade, and year.
Finally, we obtained annual staff-level records from ODE via the OERC for the 2007–2008 through 2013–2014 school years. These records contain information on staff members’ position categories—education professional, administrator, support staff, and so forth—and their specific positions within those categories, such as teacher, principal, and custodian, respectively. The records also include the individual’s date of hire and, if applicable, separation, as well as the reason for the separation (e.g., resigned, retired, or terminated). We use these records to create the staff turnover measures that we employ in our analyses below.
Empirical Strategy: RD Design
We leverage the eligibility criteria determining SIG eligibility—particularly the proficiency rate thresholds demarcating the bottom 5% of schools—to estimate the effects of Ohio’s school turnaround efforts on student outcomes and school administration. As described above, schools were generally eligible for the turnaround intervention if they were in the bottom 5% of schools in terms of the combined proficiency rate. 7 Thus, an arbitrary threshold separates schools eligible to receive turnaround interventions from those that were not. For example, in the first SIG cohort, a school with a combined proficiency rate of 29.15 was a Tier 1–eligible school, but one with a combined proficiency rate of 29.2 was not. We exploit this policy rule to estimate the effects of turnaround efforts using an RD design, which bases identification of the effect of turnaround on the assumption that schools just below the eligibility threshold are essentially indistinguishable from those just above the threshold.
Sample and Construction of Running Variable
We estimate the effect of SIG eligibility—the ITT parameter—for Tier 1 schools in the first two cohorts of the program. 8 For each of the first two cohorts, we use the combined proficiency rate data that ODE provided to identify all potential Tier 1 schools, as well as the cutoff corresponding to the lowest 5% of those schools.
ODE selected the first cohort of SIG-eligible schools from 724 Title I–served schools in NCLB’s school improvement process (i.e., potential Tier 1 schools). The bottom 5% thus consisted of 36 schools and a corresponding cutoff of 29.175 on the combined proficiency rate measure. Schools below this cutoff were Tier 1 eligible on the basis of their achievement levels, whereas schools above the cutoff were not. 9 In the second SIG cohort, there were 765 potential Tier 1 schools. The lowest 5% thus consisted of 38 schools and 33.66 was the corresponding cutoff on the combined proficiency rate measure. Ohio administered their SIG program such that, in theory, a school’s award status in the first round of SIG awards did not affect its eligibility for the second round. That is, if a school was a SIG winner in the first round, but fell in the bottom 5% of schools in the second round, then it was formally eligible to receive a second-round SIG award as well. 10 In practice, though, no school that received a SIG award in the first round also received an award in round 2.
For each cohort we subtracted the combined proficiency rate cutoff identifying the lowest 5% of schools from each school’s score on that measure. The resulting value captures a school’s distance from the cutoff determining Tier 1 SIG eligibility—at least on the basis of achievement levels—and serves as the running variable in our analyses below. We then extracted annual student, school, and district records for all potential Tier 1 schools, starting with the year of eligibility identification and extending through the subsequent 4 years. Finally, to increase our ability to detect effects of SIG eligibility, we pooled the two cohorts to create a single data set. RD designs have less statistical power than other designs, particularly when the treatment is assigned at a group—rather than individual—level, as is the case with SIG. In such cases, compared with an experimental design, RD designs require a sample 4 times larger to generate estimates with the same degree of precision (Schochet, 2008). Consequently, evaluations of school-level interventions using an RD design commonly pool across time or space (or both) to increase statistical power. For example, the federal SIG evaluation pools across states (Dragoset et al., 2017), even though the program was administered separately by each state. Similarly, LiCalsi et al. (2015) and Zimmer, Henry, and Kho (2017) pool across cohorts in their respective evaluations of Massachusetts’s and Tennessee’s turnaround efforts (although only the former study employs an RD design).
Our final, pooled data set contains observations over a 5-year period for all students attending a school with the potential for Tier 1 SIG eligibility on the basis of its achievement level. Specifically, the data set contains more than 2.8 million observations from approximately 741,000 unique students attending potential Tier 1 schools, 62 of which were eligible in either the first or second cohort (or both).
For the year of SIG eligibility identification, Table 3 presents summary statistics for all students in the data set, as well as separately for those in schools below and above the eligibility threshold. The table demonstrates that students attending SIG-eligible schools are much more likely to be Black, Hispanic, economically disadvantaged, ELLs, and disabled than their peers in schools above the Tier 1 eligibility cutoff. They are also less likely to be classified as a native English speaker or gifted, and they exhibit much lower achievement scores than students in SIG-ineligible schools. Finally, Table 3 illustrates that the average student below the Tier 1 eligibility threshold attends a school with a lower average enrollment, attendance rate, performance index score, and estimated value added, compared with the school of the average student above the SIG eligibility cutoff.
Sample Means, by Position Relative to SIG Eligibility Threshold
Note. SIG = School Improvement Grant; ELL = English language learner.
Validity of the Design
The first step in establishing the validity of our RD design involves demonstrating a difference in the likelihood of SIG receipt for schools on either side of the eligibility threshold. Figure 1 provides such a demonstration. The markers in the figure represent the mean proportion of students attending schools that received a SIG award in defined bins below and above the eligibility threshold. The line is a smoothed, kernel-weighted mean fit separately on each side of the cutoff. 11 The figure shows that more than 75% of students just below the eligibility threshold attended a school that received a SIG award, whereas only about 5% of students above the threshold were enrolled in a school that received SIG funds. 12
Mean proportion of students in schools receiving SIG awards, by distance from SIG eligibility threshold.
Figure 1 illustrates a substantial discontinuity at the eligibility threshold in the probability of attending a school that received a SIG award, but the validity of the RD design could be threatened if schools were able to manipulate their performance on the criterion determining SIG eligibility—the combined proficiency rate. We believe manipulation of this metric to be unlikely for two main reasons. First, the combined proficiency rate is based on data that extend 5 years into the past, meaning that much of the data informing SIG eligibility were finalized long before schools were aware they would be used for that purpose. Second, the threshold determining SIG eligibility is a relative one, rather than an absolute one. Consequently, in the unlikely event that a school wanted to manipulate its performance, it would not have a clear threshold to target. Rather, the school’s eligibility would depend on the simultaneous performance of other schools, which is unobservable to the potential manipulating school. Taken together, these conditions make manipulation by schools highly unlikely.
Nonetheless, we conducted tests designed to detect potential manipulation. First, we simply plotted the density of cases around the eligibility threshold to assess whether there is a disproportionate stacking of students on either side of the cutoff, which would be evidence of potential manipulation. Figure 2 presents these plots. The left-hand panel of the figure provides a density plot on the basis of student-level data, whereas the right-hand panel is based on school-level data. Neither panel provides evidence of disproportionate stacking around the cutoff. This visual evidence is corroborated by the results of the statistical test proposed by McCrary (2008), which fails to reject the null hypothesis of no change in the density of cases at the threshold. 13
Distribution of students (left panel) and schools (right panel).
Although these tests provide strong evidence that there is no disproportionate stacking of cases at the threshold, it is still possible that the characteristics of students—and their corresponding schools—just below the SIG eligibility threshold are systematically different from those just above the eligibility cutoff. Any such differences would threaten the validity of the RD design. To assess the likelihood of such differences, we systematically compare the observable, pretreatment characteristics of students on either side of the eligibility threshold. We perform this comparison by estimating
where O represents an observable characteristic of student i in school–cohort combination s during the year of SIG eligibility identification. The observable characteristics are modeled as a flexible function of a school’s distance from the SIG eligibility threshold
Table 4 presents the coefficient estimates and standard errors for the indicator of SIG eligibility for each observable characteristic we analyze. The results reveal no significant differences at the eligibility cutoff for any of the 23 observable characteristics when the running variable is specified as a linear term interacted with the SIG eligibility indicator. The second specification returns one estimate that is significant at p < .10, but none that is significant at p < .05. Considered together, the results in this section provide confidence in the validity of our RD design and, thus, the causal nature of the estimates in our analyses below.
Coefficients and Standard Errors on SIG Eligibility Indicator From OLS Model Predicting Observable Characteristics
Note. Standard errors clustered by school in parentheses below coefficient estimates. All estimates based on analytic sample containing all students in schools within 20 points of the SIG eligibility threshold. Model controls for distance from the reclassification threshold using the specification denoted in the table. SIG = School Improvement Grant; OLS = ordinary least square.
p < .10. **p < .05. ***p < .01.
Student Achievement Analysis
Leveraging our RD design, we first estimate the effect of SIG eligibility on student achievement by estimating a series of ordinary least squares (OLS) models, one for each of the 4 years following SIG identification. The general model specification for each year is
where Y is standardized achievement in reading or math for student i in school–cohort combination s. In this model,
The parameter of interest in this model is
We supplement the estimated effects of SIG eligibility from the series of cross-sectional models with an analysis based on a dynamic RD technique proposed by Cellini, Ferreira, and Rothstein (2010). This technique provides the potential for greater efficiency through simultaneous estimation of the dynamic treatment effects of SIG eligibility. We implement this approach through the following regression model:
where Y is standardized achievement in reading or math for student i in school–cohort combination s in year-relative-to-SIG-identification k during calendar year t. Students’ achievement outcomes are modeled as a flexible function of their school’s distance from the SIG identification threshold—with the coefficients allowed to vary by k—as well as an indicator for attending a school scoring below the eligibility threshold at k = 0. The coefficients on this indicator, represented by
Table 5 presents results from estimating Equations 2 and 3. The top panel of the table presents results from the series of cross-sectional models, whereas the bottom panel presents results from the dynamic models. The left-hand side of the table presents estimates of the effect of SIG eligibility on reading achievement, whereas the right-hand side presents estimated effects on math scores. The results from the top panel of Table 5 are presented visually in Figures 3 (reading) and 4 (math). Separately for each year, these figures plot mean achievement by the distance from the eligibility threshold. Each plot also contains a separate line of best fit on each side of the eligibility threshold.
Coefficients and Standard Errors for Indicators of SIG Eligibility From Models Predicting Student Achievement Levels, by Subject, Year, and Modeling Approach
Note. Dependent variable is scale score standardized by year, subject, and grade. Standard errors clustered by school in parentheses below coefficient estimates. Estimates based on analytic samples containing all students in schools within 20 points of the SIG eligibility threshold in the year of SIG eligibility identification. Models control for distance from the eligibility threshold using a linear term interacted with the SIG eligibility indicator. Cross-sectional models contain a school-level measure of baseline achievement. Dynamic models based on the approach proposed by Cellini, Ferreira, and Rothstein (2010). SIG = School Improvement Grant.
p < .10. **p < .05. ***p < .01.
Mean reading achievement, by distance from the SIG eligibility threshold.
Mean math achievement, by distance from the SIG eligibility threshold.
The results in the top panel of Table 5 show that SIG eligibility has no effect on achievement in the first year after eligibility identification. The point estimates are approximately 0.05 SDs in magnitude and insignificant across both subjects. Estimates for the second year after identification, however, are much larger in magnitude. In reading, SIG eligibility is estimated to have an effect of 0.15 to 0.20 SDs, depending on the modeling approach. The corresponding estimates for math are 0.17 to 0.21 SDs. The estimates are statistically significant at p < .10 or lower across both subjects and modeling approaches.
The third-year estimates are even larger than those from the second year. In reading, the estimated effects of SIG eligibility across the two modeling approaches are 0.22 and 0.27, respectively, and each is statistically significant at p < .10. The analogous math effects are 0.22 and 0.26, with the former estimate significant at p < .10 and the latter at p < .05. The fourth-year estimates are somewhat smaller than those from the third year, particularly in the cross-sectional models. In these models, SIG eligibility is estimated to increase fourth-year reading and math scores by 0.11 and 0.09 SDs, respectively, which represent meaningful declines from the third-year estimates of 0.22 SDs. The fourth-year estimates from the dynamic model are larger in magnitude than their cross-sectional analogs. Specifically, the estimated fourth-year effects of SIG eligibility on reading and math achievement are 0.22 and 0.17 SDs, respectively. These estimates are only slightly smaller than the third-year estimates. Below, we show that a nonparametric approach returns fourth-year estimates consistent with those from the dynamic model, which indicates that the cross-sectional estimates are the outliers. There are differences between the cross-sectional and dynamic modeling strategies that likely account for the differences between the two sets of estimates. Most notably, the dynamic modeling strategy estimates all yearly treatment effects simultaneously using a single regression, whereas the cross-sectional approach estimates these yearly effects with a series of regressions using subsets of the data. Thus, although both approaches entail estimating gains using the same pretreatment achievement baseline, the gain estimates in the dynamic model are adjusted for differences in average achievement gains across years.
Considered together, the results in Table 5 provide strong evidence that SIG eligibility had a statistically significant—and substantively large—effect on student achievement in the second and third years after SIG eligibility identification, and likely the fourth as well. Analyses in the online appendix show that our results are insensitive to alternative specifications of the running variable (see Table A1 and Figures A1 and A2 in the online version of the journal) and to different bandwidths of the running variable (see Table A2 in the online version of the journal). We also show that estimates of the TOT parameter are only slightly larger than the ITT estimates presented above (see Table A3 in the online version of the journal), which is unsurprising given the large proportion of eligible schools receiving SIG awards. Finally, in the online appendix, we show that the estimated effects of SIG eligibility on annual student achievement gains are consistent with the estimated effects on student achievement levels shown in Table 5 (see Table A4).
In addition to estimating the effects of SIG eligibility using an RD design and the parametric models presented in Equations 2 and 3, we also estimate effects using the nonparametric estimator—along with the associated optimal bandwidth selection and bias-corrected inference procedures—proposed by Calonico, Cattaneo, and Titiunik (2014a; 2014b) and Calonico, Cattaneo, Farrell, and Titiunik (2017). Specifically, we estimate Equation 2 with kernel-based local linear regressions, estimated separately with a triangular kernel on each side of the SIG eligibility threshold. We address the fact that standard bandwidth selectors may lead to downward bias in confidence intervals for the estimated treatment effects—and, thus, overrejection of the null hypothesis of no effect—by using the bias-corrected confidence intervals proposed in Calonico, Cattaneo, and Titiunik (2014b). We account for the clustering in our running variable during the bandwidth selection process (see Bartalotti & Brummet, 2017) by using the technique detailed in Calonico et al. (2017).
Table 6 presents the results of this analysis, and they are quite similar to the parametric estimates presented in Table 5. In particular, they show no significant effects of SIG eligibility in the first year after eligibility identification. However, meaningful positive estimates begin to emerge in the second year after eligibility identification, with reading estimates of 0.11 to 0.12 SDs and math estimates of 0.14 to 0.15 SDs, although the estimates are relatively imprecise and do not reach a conventional level of statistical significance. The third-year estimates are even larger than those from the second year, with estimates of about 0.20 SDs in both reading and math. The fourth-year estimates are comparable with, if slightly smaller than, the third-year estimates—SIG eligibility increases scores by 0.17 to 0.18 SDs in each subject. As we note above, the fourth-year estimates from this approach are closer in magnitude to those from the dynamic models in Table 5. Taken together, the results provide evidence that the positive effects of SIG eligibility persist throughout all 4 years following SIG eligibility identification. In the online appendix, we show that school-level estimates of the effects of SIG eligibility are remarkably similar in magnitude to the student-level estimates presented in Table 6, and oftentimes more precise (see Table A5 in the online version of the journal). 16
Coefficients and Standard Errors for Indicators of SIG Eligibility From Nonparametric Estimator Predicting Student Achievement Levels, by Subject and Year
Note. Standard errors clustered by school in parentheses below coefficient estimates. Estimates produced by the approach proposed by Calonico, Cattaneo, and Titiunik (2014b). SIG = School Improvement Grant.
p < .10. **p < .05. ***p < .01.
Turnaround Versus Transformation Model
Earlier we noted that schools receiving a SIG award were required to implement one of four federally approved turnaround models: Closure, Restart, Transformation, or Turnaround. Table 2 demonstrates that nearly every school implemented either the Turnaround or Transformation model. To gain insight into whether the positive achievement effects seen in Tables 5 and 6 are primarily attributable to one particular model, or whether they both produce positive effects, we estimate variants of Equations 2 and 3, where we interact indicators for implementing the SIG Turnaround and Transformation models with the indicator for SIG eligibility. We stress that the results of this analysis cannot be interpreted as estimates of the causal effect of implementing the Turnaround or Transformation model. Ohio schools and districts worked collaboratively to determine which SIG model a school would implement. This collaboration was designed to identify the optimal SIG model for a given school, but it also renders adoption of these models endogenous. Consequently, the results should be interpreted as mean achievement differences between students attending SIG-ineligible schools and those attending schools that, respectively, implemented the Turnaround or Transformation model, conditional on schools’ distance from the SIG eligibility threshold.
Table 7 presents the results of this analysis. The results make clear that—compared with SIG-ineligible schools—schools implementing the Turnaround model exhibited substantial achievement gains beginning in the second year after SIG eligibility identification. In both reading and math and across both the cross-sectional and dynamic models, the gains associated with implementation of the Turnaround model are about 0.16 to 0.22 SDs in the second year and range from 0.21 to 0.30 SDs in the third and fourth years after eligibility identification. Schools that implemented the Transformation model also exhibited achievement gains, but they are smaller in magnitude and uniformly insignificant. In the online appendix, we demonstrate that a difference-in-differences approach produces similar conclusions to those presented in Table 7, specifically that implementation of the Turnaround model is associated with larger achievement gains than implementation of the Transformation model (see Table A8 in the online version of the journal). Together, the results provide evidence that the SIG Turnaround model generated larger achievement increases than the Transformation model, although we cannot rule out the possibility that these differences are attributable to selection.
Coefficients and Standard Errors for SIG Turnaround and Transformation Models From Models Predicting Student Achievement Levels, by Subject, Year, and Modeling Approach
Note. Dependent variable is scale score standardized by year, subject, and grade. Standard errors clustered by school in parentheses below coefficient estimates. Estimates represent mean achievement differences between SIG-ineligible schools and schools that implemented the Turnaround and Transformation models, respectively. Estimates based on analytic samples containing all students in schools within 20 points of the SIG eligibility threshold in the year of SIG eligibility identification. Models control for distance from the eligibility threshold using a linear term interacted with the SIG eligibility indicator. Cross-sectional models contain a school-level measure of baseline achievement. Dynamic models based on the approach proposed by Cellini, Ferreira, and Rothstein (2010). SIG = School Improvement Grant.
p < .10. **p < .05. ***p < .01.
Changes in Student Composition
The SIG program was intended to improve student outcomes by increasing school effectiveness. It is possible, though, that the positive effects of SIG presented in Tables 5 and 6 were driven by changes in student composition at SIG-eligible schools. Relatively high-achieving students may have moved into SIG-eligible schools and produced the positive achievement effects presented above. To test for this possibility, we first constructed an annual student-level measure indicating whether the student attended the same school as she did the previous year. Then, using the RD design described above, we specify this measure as the outcome in models identical in structure to the one presented in Equation 2. Negative and significant coefficients on the measure of SIG eligibility would indicate that SIG-eligible schools enroll a greater proportion of new students than schools just above the eligibility threshold. Such a result would be consistent with an influx of higher achieving students into SIG-eligible schools that could drive the achievement effects shown in Tables 5 and 6.
Table A9 in the online appendix shows, however, that the coefficient on the measure of SIG eligibility is actually positive and significant in the year of eligibility identification. In other words, SIG eligibility increases the likelihood that a student remains enrolled in the same school during the first year of SIG by about 7 percentage points. The estimates for the second through fourth years following SIG eligibility identification are also positive and range from 3 to 6 percentage points, depending on the particular year, but not all are significant. These results hold when we estimate a variant of the model that excludes students attending the lowest grade of a school, as these students are highly unlikely to have attended the same school the prior year.
The significant positive effect of SIG eligibility on the probability of remaining enrolled in the same school leaves open the possibility that these schools disproportionately retained their high-achieving students. To provide evidence on this possibility, we estimated a variant of the model described above where we interact the SIG eligibility indicator with a student’s reading achievement score. A positive and significant coefficient on this interaction would indicate disproportionate retention of high-achieving students in SIG-eligible schools. The results in Table A10 in the online appendix, however, show the coefficient on this interaction to be negative and significant in the first 2 years following SIG eligibility identification and insignificant thereafter. To address the possibility that the retention of higher performing students is an artifact of the positive achievement effects of SIG, the third column of the table presents results from a specification where achievement is fixed at a student’s achievement level in the final pretreatment year. The coefficients on the interaction remain negative and significant in this specification, and even somewhat larger in magnitude. This provides strong evidence that low-achieving students were disproportionately retained in SIG-eligible schools in the years following eligibility identification. These results, together with the estimated effects on student test score gains and school value added (see Online Appendix Tables A6 and A7 in the online version of the journal), suggest that the positive achievement effects of SIG eligibility were driven by true school improvement, as opposed to a changing student population.
School Administration and Staffing Analysis
School Finance
Schools identified as SIG recipients were awarded grants that ranged from about US$750,000 to US$2 million to be spent over a 3-year period. These grants were intended to increase expenditure levels at the schools receiving SIG awards. However, because grant funds generally flowed through district offices, it is possible that districts managed finances in a manner such that the grants supplanted, rather than supplemented, existing school spending. Dragoset et al. (2017) present evidence that such a scenario may have been occurring in the early years of SIG.
We take advantage of the fact that Ohio maintains school-level expenditure data to perform analyses intended to provide insight into whether SIG grants to Ohio schools supplemented or supplanted existing expenditures. Specifically, we employ the RD design described above, coupled with the data on school expenditures, to estimate the effect of SIG eligibility on per-pupil expenditures. 17 We complement the RD estimates with results from a difference-in-differences model. This model contains fixed effects for school-by-cohort, calendar year, and year relative to SIG identification. Interactions between a SIG eligibility indicator and the indicators for year relative to SIG identification represent the difference-in-differences estimates of the effect of SIG eligibility—see the online appendix for more detail on the difference-in-differences approach. Table 8 presents the results of these analyses. They illustrate that SIG eligibility has no significant effect on per-pupil expenditure levels in the year following SIG eligibility identification. That is to be expected, as the 3-year grants did not begin until the following year. The results also indicate that SIG eligibility increased per-pupil expenditures by about US$1,500 to US$2,000 in the second year after eligibility identification, by US$2,600 to US$3,300 in the third year, and by US$1,200 to US$1,600 in the fourth year, which corresponds to the final year of the grant period.
Coefficients and Standard Errors From Models Predicting Per-Pupil Expenditures, by Year Relative to SIG Eligibility Identification
Note. Standard errors clustered by school in parentheses below coefficient estimates. All estimates based on analytic sample containing all schools within 20 points of the SIG eligibility threshold in the year of SIG eligibility identification. Models control for distance from the eligibility threshold using a linear term interacted with the SIG eligibility indicator. Cross-sectional models contain a school-level measure of baseline achievement. Dynamic models based on the approach proposed by Cellini, Ferreira, and Rothstein (2010). SIG = School Improvement Grant; RD = regression discontinuity.
p < .10. **p < .05. ***p < .01.
These estimates are generally consistent with the size of the typical SIG award in Ohio. Indeed, award data provided by ODE indicate that the average award over the 3-year grant period was about US$2.25 million, which averages out to about US$750,000 annually. When spread across the enrollment in SIG schools, the average award comes out to about US$2,300 per student annually. However, the ODE data show SIG schools’ award budgets propose spending a greater portion of the award in the first and second years, compared with the third year of the grant. Specifically, the data indicate that the average school proposed spending approximately US$2,600 per pupil in the first year of the award, US$2,200 in the second, and US$2,000 in the final year of the award. These annual averages do not perfectly correspond to the estimates in Table 8, but there are multiple reasons we might expect imperfect alignment. First, the averages described here are based on schools’ budgeted award expenditures rather than actual award expenditures, and these two figures do not always align. Unfortunately, we do not have data on actual school-level SIG expenditures. Second, our RD approach estimates the effect of SIG eligibility at the threshold, whereas the proposed expenditures represent averages across all awarded schools, not just those at the threshold. Third, some districts may account for money disbursed near the end of a fiscal year on the budget for the following year.
Although we cannot completely rule out the possibility that some funds were supplanted, the results in Table 8 provide evidence that the SIG grants awarded to Ohio schools supplemented existing expenditures. In addition, the results illustrate that the magnitude of these supplements was substantial. Our data indicate that the average SIG-eligible school spent approximately US$14,100 per pupil in the year of SIG eligibility identification. Compared against this base, the estimated effects correspond to spending increases of 11% to 14%, 18% to 23%, and 9% to 11% in the second, third, and fourth years after SIG eligibility identification, respectively. 18
Table 9 presents results from specifications of the RD and difference-in-differences models where we interact indicators for implementing the SIG Turnaround and Transformation models with the indicator for SIG eligibility. Across all three modeling approaches, results for schools that implemented the SIG Transformation model indicate positive and statistically significant per-pupil spending increases in the second, third, and fourth years following SIG eligibility. In contrast, only one of the 12 estimates for schools that implemented the Turnaround model are statistically significant, although several of the nonsignificant estimates are of a meaningful magnitude—in the range of US$1,500 to US$3,000. The lack of significance of the Turnaround estimates is at least partially attributable to the fact that only 15 schools implemented this SIG model, which results in large standard errors. Together, the results in Table 9 provide suggestive evidence that both the Turnaround and Transformation models are associated with per-pupil spending increases, although only the estimates for the Transformation model are significant.
Coefficients and Standard Errors for SIG Turnaround and Transformation Models From Models Predicting Per-Pupil Expenditures, by Year and Modeling Approach
Note. Standard errors clustered by school in parentheses below coefficient estimates. All estimates based on analytic sample containing all schools within 20 points of the SIG eligibility threshold in the year of SIG eligibility identification. Models control for distance from the eligibility threshold using a linear term interacted with the SIG eligibility indicator. Cross-sectional models contain a school-level measure of baseline achievement. Dynamic models based on the approach proposed by Cellini, Ferreira, and Rothstein (2010). SIG = School Improvement Grant; RD = regression discontinuity.
p < .10. **p < .05. ***p < .01.
A natural question that emerges from the results in Table 9 is why the Transformation model entails significant spending increases but the Turnaround model does not. The requirements of each model may provide at least a partial answer to this question. Both models required a change in school leadership and a number of instructional and operational reforms. However, the Turnaround model further entailed replacing at least 50% of the school staff—a requirement absent from the Transformation model—whereas the Transformation model required schools to adopt a new governance model and an educator evaluation system that included student achievement growth as a significant component. These two components of the Transformation model were not mandated by the Turnaround model. It seems likely that the requirements of the Transformation model, particularly development of a new governance model and evaluation system, were costlier endeavors than the Turnaround mandate of staff turnover. This is particularly likely to be true if schools implementing the Turnaround model replaced terminated staff with less experienced individuals earning lower salaries.
School Staffing
In the prior section, we described how the SIG turnaround models were designed to change leadership and staffing in schools receiving SIG awards. To empirically examine how Ohio’s SIG initiative affected school staffing, we perform a series of analyses based on the same RD design described above. The first analysis in this vein estimates the effect of SIG eligibility on staff turnover. Specifically, using staff-level data, we estimate the effect of SIG eligibility on the probability of staff members remaining employed at the same school 3 and 4 years, respectively, after SIG eligibility identification. We estimate this effect using a variant of Equation 2, where we specify an indicator of staff turnover as the outcome. As with the RD analyses in prior sections, we present results from a model where the running variable is specified as a linear term that is interacted with the SIG eligibility indicator. In the online appendix, we show that other specifications return substantively similar results (see Table A11 and Figure A3 in the online version of the journal). We estimate the model separately for all school staff and only for teachers. We cluster standard errors by school in all models.
The results of this analysis are presented in the top two panels of Table 10, with corresponding visual representation in Figure 5. Table 10 shows that SIG eligibility had no significant effect on the stability of school staffing, with the “all staff” estimates very close to zero. For teachers only, the results indicate that SIG eligibility is estimated to increase the probability of being employed at the same school 3 and 4 years after potential SIG identification by .05, but these estimates are quite imprecise and do not approach statistical significance. Inspection of Figure 5 suggests that the null effect of SIG eligibility on staffing stability is due to high levels of staff turnover on both sides of the SIG eligibility cutoff, rather than noncompliance with the requirements of the SIG models. On average, only about 30% of staff at a school just above the SIG eligibility cutoff are still employed at that school 4 years later.
Coefficients and Standard Errors for Indicators of SIG Eligibility From Models Predicting Staff Turnover
Note. Standard errors clustered by school in parentheses below coefficient estimates. All estimates based on analytic sample containing staff in schools within 20 points of the SIG eligibility threshold in the year of SIG eligibility identification. Model controls for distance from the reclassification threshold using linear term interacted with the SIG eligibility indicator. SIG = School Improvement Grant.
p < .10. **p < .05. ***p < .01.
Mean proportion of staff employed at school 3 and 4 years after SIG eligibility identification, by distance from the SIG eligibility threshold.
The bottom panel of Table 10 presents results from an analysis of the effect of SIG eligibility on annual staff turnover for each of the first 4 years following potential SIG eligibility identification. The results show that SIG eligibility had no significant effect on annual turnover of either teachers or school staff more generally. The results suggest that SIG eligibility may actually decrease teacher turnover in the years following SIG eligibility identification—three of the four point estimates are negative—but none of the estimates is sufficiently precise to reach statistical significance. Taken together, the results in Table 10 indicate that SIG had no meaningful effect on staff turnover rates, with additional analysis suggesting that the null effect on attributable to similarly high turnover rates at schools on either side of the SIG eligibility threshold.
It is possible that the null effects presented in Table 10 mask heterogeneity across the two main SIG models, Turnaround and Transformation. Table 11 presents estimates from a specification that interacts indicators for implementing the SIG Turnaround or Transformation models with the indicator for SIG eligibility. The results provide evidence of significant heterogeneity, particularly in the case of teachers. Teachers in schools that implemented the Transformation model were 14 percentage points more likely to remain employed at the same school 3 years later, compared with teachers at SIG-ineligible schools. In contrast, teachers in schools that implemented the Turnaround model were about 15 percentage points less likely to remain employed at the same school 3 years down the line. Although these estimates cannot be interpreted causally—the decision to implement a particular SIG model is likely endogenous—these results provide important evidence that the effects of SIG on school staffing played out quite differently across the two primary SIG models.
Coefficients and Standard Errors for SIG Turnaround and Transformation Models From Models Predicting Staff Turnover, by Year
Note. Standard errors clustered by school in parentheses below coefficient estimates. Estimates represent mean differences in the probability of remaining employed between SIG-ineligible schools and schools that implemented the Turnaround and Transformation models, respectively. Estimates based on analytic samples containing all staff or teachers in schools within 20 points of the SIG eligibility threshold in the year of SIG eligibility identification. Models control for distance from the eligibility threshold using a linear term interacted with the SIG eligibility indicator. SIG = School Improvement Grant.
p < .10. **p < .05. ***p < .01.
SIG eligibility had no overall effect on staff turnover, but it is possible that it could have affected staffing levels. For example, schools and districts could use SIG funds to hire additional educational staff, which would not be reflected in turnover rates. To examine this possibility, we used the staff-level data described above to generate annual school-level counts of four staffing categories—all staff, teachers, education professionals, and administrators. We then use the RD design to estimate the effect of SIG eligibility on staffing levels across these four categories. We estimate these effects using both the cross-sectional and dynamic modeling approaches we have employed throughout the analysis. We specify the models as we did for the analysis of staff turnover, although the results are robust to alternative analytic decisions (see Table A12 in the online appendix). 19
The results presented in Table 12 show that SIG eligibility generates a significant reduction in the number of school administrators in the first year of SIG eligibility. The number of administrators is estimated to decline by 1 to 1.5 full-time equivalent (FTE) in that year, depending on the modeling approach. This is perhaps unsurprising given that both the SIG Turnaround and Transformation models require a change in school leadership, and it may take some time to fully reconstitute the leadership team. There is some evidence of SIG eligibility reducing the total number of staff in a school by about 4 to 6 FTEs in the year following SIG identification, but only the estimate from the dynamic RD model approaches significance (p < .10). Table 12 shows that the point estimates for teachers and educational professionals are positive and of a potentially meaningful magnitude, particularly in the second and third years, where SIG eligibility is estimated to increase the number of FTEs in each category by 1.5 to 3. However, these estimated effects are quite imprecise and do not approach statistical significance. In addition, they tail off in the fourth year after SIG eligibility identification.
Coefficients and Standard Errors for Indicators of SIG Eligibility From Models Predicting Staffing Levels, by Staff Position, Year, and Modeling Approach
Note. Standard errors clustered by school in parentheses below coefficient estimates. Estimates based on analytic samples containing all staff or teachers in schools within 20 points of the SIG eligibility threshold in the year of SIG eligibility identification. Models control for distance from the SIG eligibility threshold with a linear term interacted with the SIG eligibility indicator. Cross-sectional models contain a baseline measure of staffing levels. Dynamic models based on the approach proposed by Cellini, Ferreira, and Rothstein (2010). SIG = School Improvement Grant.
p < .10. **p < .05. ***p < .01.
Table A13 in the online appendix provides evidence that any increases in the number of teachers or education professionals come from schools that implemented the Transformation model. The point estimates for schools implementing the Transformation model are positive and significant for the second and third years following SIG eligibility identification, whereas the point estimates for the Turnaround model in those years are negative, albeit insignificant. These results are consistent with the Transformation versus Turnaround turnover results presented in Table 11 above.
School Closure
Our final analysis estimates the effect of SIG eligibility on the probability of school closure. There are at least two ways in which SIG eligibility could result in school closure. First, and most obvious, SIG recipients could implement the Closure or Restart turnaround models, which are two of the four federally approved strategies that SIG awardees could implement. However, Table 2 demonstrated that almost no school elected to implement either of these models. Second, schools that are SIG eligible but not awarded SIG funds may be closed due to lack of assistance or support for improving the quality of a school. We note that a similar phenomenon could occur with schools just above the SIG eligibility threshold—districts may decide to close these low-performing schools as they are ineligible for SIG awards that could facilitate improvement. In addition to being an interesting outcome in its own right, this closure analysis has potential implications for the interpretation of the estimated effects of SIG on student achievement. In particular, if SIG eligibility is found to have an effect—either positive or negative—on the probability of closure, then it is possible that the large positive effects of SIG eligibility on student achievement are at least partially attributable to differential sample attrition.
We estimate the effect of SIG eligibility on school closure separately for each of the 4 years following SIG eligibility identification. As with the analysis of school staffing levels, we use school-level data and employ both the cross-sectional and dynamic modeling approaches, specifying the sample and models in the same way we did above. Table 13 presents the estimated effects of SIG eligibility on closure. The results illustrate that SIG eligibility has no significant effects on the probability of closure—across all years and both modeling approaches the point estimates are insignificant. In addition to demonstrating that SIG eligibility has no effect on the probability of closure, the results in Table 13 provide evidence that the large positive effects of SIG eligibility on student achievement are not attributable to differential attrition between schools that are SIG eligible and those that are not. Further analysis demonstrated that these null effects hold across both the Turnaround and Transformation models. 20
Coefficients and Standard Errors for Indicators of SIG Eligibility From Models Predicting School Closure, by Subject, Year, and Specification
Note. Standard errors clustered by school in parentheses below coefficient estimates. All estimates based on analytic sample containing all schools within 20 points of the SIG eligibility threshold in the year of SIG eligibility identification. Models control for distance from the SIG eligibility threshold with a linear term interacted with the SIG eligibility indicator. Dynamic models based on the approach proposed by Cellini, Ferreira, and Rothstein (2010). SIG = School Improvement Grant; RD = regression discontinuity.
p < .10. **p < .05. ***p < .01.
Considered together, the results in this section indicate that SIG eligibility led to a significant increase in per-pupil expenditures, with these increased expenditures largest in schools that implemented the Transformation model. Interestingly, there was no overall effect of SIG eligibility on school staffing—either staff turnover or staffing levels—but we provide evidence that this null mean effect masked significant heterogeneity in staffing outcomes across the two primary SIG models, Turnaround and Transformation.
Discussion and Conclusion
The federal SIG program is the latest in a long line of initiatives intended to improve the quality of low-performing schools. SIG was designed to achieve such improvements by providing schools with additional financial resources and by requiring them to make significant changes to several aspects of their educational delivery, most notably their leadership and staffing. In this article, we begin to evaluate whether Ohio’s school turnaround efforts under SIG produced the intended quality improvements. In particular, we use data from two SIG cohorts to estimate the effect of SIG eligibility on student achievement levels, as well as on school spending, staff turnover, and school staffing levels. We estimate these effects in an RD framework, leveraging the criteria stating that schools were eligible to receive a SIG award if they ranked in the bottom 5% of eligible schools in terms of a weighted student proficiency rate calculated by ODE.
The results of our analysis indicate that SIG eligibility had a large positive effect on the achievement of students attending those schools. These effects first emerge in the second year after a school is identified as SIG eligible, with reading and math effects up to 0.20 SDs. In the third year, SIG eligibility is estimated to increase reading achievement by at least 0.20 SDs and math achievement by at least 0.18 SDs, and some specifications suggest achievement gains of up to 0.26 SDs in math and 0.27 SDs in reading. There is some evidence from the cross-sectional models that the achievement gains diminish in the fourth year following SIG eligibility identification, but our other modeling strategies—the dynamic model, the nonparametric approach, and even the cross-sectional model with different bandwidths—suggest that the gains persist through the fourth year.
Although the preponderance of the evidence points toward the persistence of these positive achievement effects through the full award period, it is worth considering the possibility that the effects diminish or disappear in the final award year, or even in postaward years. The SIG program was designed to generate lasting improvements in the quality of awarded schools. Our results indicate that Ohio’s SIG program got off to a good start on this score, but they provide little indication as to whether these initial quality improvements will be maintained over the long haul. However, if prior school improvement efforts are any guide, there are reasons to think that these achievement increases may ultimately prove transitory. The history of education reform is replete with examples of promising initiatives—some of which even generate positive short-term impacts—that ultimately fizzle out (Hess, 1998; Tyack & Cuban, 1995). Several aspects of the education policy arena, including the impatience of policymakers and funders, the desire of superintendents to leave their mark via new policy reforms, and the general pull of the status quo, often serve to limit the staying power of even promising reform efforts. Our results do not provide evidence on the long-term impacts of Ohio’s SIG initiative, but they do suggest that such inquiry would be worthwhile.
Along with providing estimates of the causal effects of SIG eligibility, our results also provide evidence of potential heterogeneity across schools that implemented the Turnaround and Transformation models. Specifically, our results indicate that the positive achievement effects are primarily attributable to schools that implemented SIG’s Turnaround model. With respect to spending, we estimate that SIG eligibility increases annual per-pupil funding by US$1,500 to US$3,000 across the 3-year grant period but provide evidence that the spending increases were somewhat larger in schools that implemented the Transformation model, although results suggest that both Turnaround and Transformation schools realized spending increases. Finally, the null effect of SIG eligibility on staff turnover masks the fact that teachers in Turnaround schools were significantly less likely to still be employed at that school 3 and 4 years later, whereas teachers at Transformation schools were substantially more likely to remain employed at their school. Considered together, our results are consistent with a scenario where the staff replacement requirements of the Turnaround model, coupled with the increased financial resources, generated the observed achievement increases. However, the evidence is not definitive on this score.
Our findings are at odds with those of the federally sponsored SIG evaluation, which found SIG implementation to have no effect on student achievement or attainment (Dragoset et al., 2017). The federal evaluation was based on an RD design and drew upon data from a nonrepresentative sample of 60 districts spread across 22 states. Such an approach provides valuable evidence about the effects of SIG across a geographically diverse set of districts, but there are also potential benefits to focusing on specific contexts. Indeed, in a federal system such as ours, policy is implemented and administered by states, districts, and schools. With SIG in particular, SEAs were given meaningful autonomy within the federally defined parameters to administer the program. For example, Ohio was given some latitude in creating the metric on which the bottom 5% of schools were deemed SIG eligible. Similarly, the state had autonomy in designing both the call for districts to submit proposals for SIG funds and the process for selecting those schools and districts that would receive SIG awards. More generally, the state contexts in which SIG was implemented and administered undoubtedly vary significantly, with some of those contexts more conducive than others to school turnaround policies achieving the intended school improvement.
Given this variation in context—and the potential role that context may play in the ultimate success of turnaround policies—it is important for evaluations to take the issue seriously. There are several ways this can be done. One approach involves conducting separate evaluations across distinct contexts (i.e., states) and then attempting to identify the contextual factors associated with policy success or failure after a sufficient number of evaluations have been conducted. The charter school literature has evolved in this manner, with a number of earlier studies devoted to evaluating the charter sectors in various states or cities (e.g., Abdulkadiroğlu, Angrist, Dynarski, Kane, & Pathak, 2011; Bifulco & Ladd, 2006; Witte, Weimer, Shober, & Schlomer, 2007; Zimmer et al., 2009) and later work placing more emphasis on identifying the factors or contexts that facilitate successful charter schools (e.g., Angrist, Pathak, & Walters, 2013; Dobbie & Fryer, 2013; Fryer, 2014). There are signs that the recent school turnaround literature may take a similar path. The literature now contains studies evaluating school turnaround efforts in a number of jurisdictions, including California (Dee, 2012), North Carolina (Heissel & Ladd, 2016; Henry & Guthrie, 2015), Massachusetts (LiCalsi et al., 2015; Papay, 2015), Ohio, Tennessee (Zimmer et al., 2017), and Los Angeles (Strunk et al., 2014). Our positive results mirror those from Massachusetts (LiCalsi et al., 2015; Papay, 2015) and California (Dee, 2012) but are somewhat at odds with the mixed results from North Carolina (Heissel & Ladd, 2016) and Tennessee (Zimmer et al., 2017). As this set of studies grows, it seems likely that future work will begin to systematically consider the contexts and conditions under which school turnaround strategies are and are not likely to prove effective.
This study provides evidence that school turnaround efforts can be successful in the context of Ohio, but does less to pin down the particular aspects of the context that facilitate success. Future work would do well to further analyze potential mechanisms that could contribute to the success of Ohio’s SIG initiative, particularly given the effects of SIG on per-pupil spending levels and the differences between schools that implemented the Transformation and Turnaround models. When making any such assessments, however, it is important to recognize the localized nature of our estimated effects. In particular, our estimates apply to schools just below the SIG eligibility threshold—this analysis does not generalize to schools further away from the cutoff. We highlight two factors to consider in making any such judgment. On one hand, schools far away from the cutoff may lack the capacity for quality improvements that schools more proximate to the cutoff might possess. Alternatively, schools far from the cutoff may have greater room for improvement and, thus, may benefit from turnaround efforts to an even greater degree than schools just below the eligibility threshold. Regardless of whether our results generalize to SIG-eligible schools further down the distribution, our findings suggest that schools further up the distribution of ODE’s “combined proficiency rate” might also benefit from turnaround interventions.
Finally, our results highlight the SIG program’s blunt requirement that states identify the lowest 5% of schools in the state as SIG eligible. The quality of a fifth percentile school has the potential to meaningfully vary across states—the quality of a school at this point of Massachusetts’ distribution is likely to substantively differ from the quality of a school at the same point of Mississippi’s—and, perhaps, contribute to heterogeneity in the effects of turnaround evaluations, particularly those that rely on an RD design. 21 Such considerations suggest the potential value of working to identify an empirically driven approach applicable across contexts to identifying schools that could benefit from turnaround interventions. More generally, efforts to improve the quality of low-achieving schools are unlikely to abate in the future, and research on school turnarounds could help guide and improve such efforts.
Supplemental Material
EEPA760218_Online_Appendices – Supplemental material for School Improvement Grants in Ohio: Effects on Student Achievement and School Administration
Supplemental material, EEPA760218_Online_Appendices for School Improvement Grants in Ohio: Effects on Student Achievement and School Administration by Deven Carlson and Stéphane Lavertu in Educational Evaluation and Policy Analysis
Footnotes
Acknowledgements
We thank the Ohio Department of Education—particularly Sherry Panizo, Eben Dowell, Jo Hannah Ward, Matt Cohen, and Paul Conaway—for their help in tracking down the information we needed to complete the analysis. We also thank the staff at the Ohio Education Research Center, particularly Lisa Neilson and Josh Hawley, for providing data. Finally, we thank Andrew McEachin, Randy Olsen, and two anonymous reviewers for providing comments and suggestions that improved the quality of this article.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The Ohio Education Research Center provided financial support for the analysis on which this article is based.
Notes
Authors
DEVEN CARLSON is Presidential Research Professor, associate professor of political science, and associate director for education at the National Institute for Risk and Resilience at the University of Oklahoma. His research analyzes the operations of school accountability and school choice policies, and explores their effects on social, economic, and political outcomes.
STÉPHANE LAVERTU is an associate professor at The Ohio State University’s John Glenn College of Public Affairs. His research focuses on K–12 education governance, including school boards, charter schools, school finance, accountability systems, and interventions such as school closure and turnaround.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
