Examining the Timing of Educational Changes Among Elementary Schools After the Implementation of NCLB

Summarizing Some Effects of NCLB

Now, 15 years after the initial implementation of NCLB, it is widely accepted that key results were the expansion of federal oversight of public education through annual testing, public results of progress, state report cards, upgraded teacher qualifications, and funding changes (Dee & Jacob, 2011; Dee et al., 2013; Education Week Research Center, 2011). It is less clear whether NCLB resulted in noticeable improvement in the educational processes and academic performance of schools that demonstrated the greatest need.

School Selection

We drew the sample of schools from the population of elementary schools in the State of Hawaii, beginning after the initial implementation of NCLB during the 2002-2003 school year through the 2011-2012 school year. We used 2011-2012 as the final year, since Hawaii changed its accountability system in 2013, after receiving a waiver from NCLB AYP reporting to implement an alternative assessment approach as part of Race to the Top (American Recovery and Reinvestment Act, 2009). Forty-six elementary schools, 40 of which were Title I schools, met the criterion of entering NCLB restructuring, which represented 26% of the state’s elementary schools. We also included a comparison group of 28 randomly selected elementary schools that did not enter restructuring, approximately 37% of which were Title I schools, with the intent of having three groups with relatively similar sample sizes. This enhances the homogeneity of variance assumption in comparing growth means between groups (Keppel, 1991).

Our final sample consisted of 74 elementary schools. Within each school, there were 10 years of repeated measures data on each process indicator. The overall sample therefore consisted of 740 teacher observations of each process indicator at Level 1 nested within 74 schools at Level 2, which were in three known groups of schools defined by NCLB compliance status. The groups included a comparison group of 28 schools that met AYP (Group 0), 19 schools that entered restructuring but did not exit before 2013 (Group 1), and 27 schools that entered and exited restructuring (Group 2). Our initial power estimates required to detect a difference of 1.0 in the means of schools’ process indicators among the three groups of schools indicated our sample design had sufficient power (0.90) to detect our hypothesized effects. ¹ All data were public and collected from the state’s education website.

Actions Leading to Restructuring

Under NCLB, schools that failed to meet AYP targets passed through a series of incremental help stages over a 6-year period culminating with restructuring. First, when a total student population or subgroup of students failed to meet state AYP targets for two consecutive years (e.g., reading, mathematics, retention rate, attendance rate), the school was identified as “needing improvement,” which opened student transfer and service options. Second, if the identified school still failed to meet AYP over time, districts were required to take “corrective action” (e.g., decrease authority of school administration, implement new curriculum) and then to plan for possible restructuring. Finally, if the school did not meet AYP targets for six consecutive years, it entered restructuring status. To exit from any sanction level and return to “good standing,” a school had to meet all applicable AYP targets for two consecutive years.

In Figure 1, we summarize the AYP status of the three groups of schools. In this type of longitudinal analysis, it is important to note that not all restructured schools managed to exit restructuring status before the study ended. It is likely others may have exited with more time, had the state not been granted an NCLB waiver. Elementary schools that received Title I funding prior to NCLB’s implementation were required to carry over previous AYP status calculated under a prior accountability system, which placed them further along in the progressive sanction process after NCLB implementation. As a result, 2005 was the first year a Title I school entered restructuring and, consequently, from 2005 to 2008, all of the elementary schools that entered restructuring status were Title I schools. Figure 1 indicates schools that entered and exited restructuring (Group 2) on average began in “corrective action” regarding AYP status (coded as 4). Schools that entered restructuring but did not exit (Group 1) on average began in “Improvement Year 2” (coded as 3). For schools that were not Title I schools, states could also apply NCLB improvement sanctions, which were first determined by AYP results beginning in the 2003 school year. As a result, 2009 was the first year a non-Title I school entered restructuring status. Five non-Title I schools entered in restructuring in 2009 or later (with one entering 2012). The sixth non-Title I school entered restructuring in 2009, but it managed to exit in 2012. Table 1 summarizes the number of elementary schools that entered restructuring each year, the number that exited each year, and the resulting total number of schools that were in NCLB restructuring status during each year of the study.

OPEN IN VIEWER

Figure 1.

Average AYP status of three groups of schools over time.

OPEN IN VIEWER

Table 1.

Schools Entering and Exiting Restructuring by Year During NCLB.

	2003	2004	2005	2006	2007	2008	2009	2010	2011	2012
Schools entered	0	0	16	11	8	0	5	3	1	2
Schools exited	0	0	0	2	4	8	1	2	8	2
Total schools in restructuring	0	0	16	25	29	21	25	26	19	19

Note. NCLB = No Child Left Behind.

Restructuring Options

Hawaii schools primarily followed the “other” restructuring prescription of NCLB (Manwaring, 2010), specifically hiring outside providers to manage some aspects of school-level reforms (Hawaii Department of Education [HIDOE], 2009). Typical services provided for restructured schools included leadership training, instructional and technical support, evaluation of student achievement results, and staff professional development. External service providers served primarily as consultants in charge of implementing changes that would improve student performance and building necessary professional capacity to sustain academic growth (HIDOE, 2009). In 2008-2009, for example, three external service providers (i.e., America’s Choice, Edison Learning, and Educational Testing Service) were contracted to work with 43 restructured elementary and secondary schools, with service contracts ranging from $820 to $456,000 per individual school (HIDOE, 2009).

Variables in the Model

Descriptive Statistics

In Table 2, we summarize the descriptive statistics for the contextual variables in our analysis by NCLB status group. In this type of time series analysis, it is important to note that not all of the schools that entered restructuring managed to exit before the study ended. In examining the number of years it took for schools to enter restructuring status, we found schools that exited restructuring typically were Title I schools that entered restructuring status earlier on average than schools that entered but did not exit. As we indicated earlier, this longer timeline was in part because of a few non-Title I schools that did not enter restructuring until 2009 or later. We also found schools that exited restructuring spent on average four years in restructuring status—suggesting nearly a decade to fall in and subsequently to climb out of restructuring status. We also found school enrollment sizes and the composition of students receiving English services on average were lower in Group 2 than Group 1 schools. Regarding staffing, on average, there were higher percentages of fully qualified teachers in Group 2 schools than in Group 1 schools.

OPEN IN VIEWER

Table 2.

Descriptive Statistics by School Restructuring Status.

Group	Years Entry	Years Exit	Experience	Highly Qualified	Enroll	ELL	Low SES
0, M (SD)	0.00 (0.00)	0.00 (0.00)	13.14 (2.78)	89.57 (7.45)	513.75 (235.70)	11.18 (10.88)	.43 (.26)
1, M (SD)	5.84 (2.54)	0.00 (0.00)	10.63 (2.85)	82.26 (11.71)	751.58 (242.90)	13.58 (10.58)	.63 (.22)
2, M (SD)	4.02 (1.13)	4.06 (1.87)	12.39 (3.91)	83.59 (11.66)	409.07 (235.60)	8.55 (9.23)	.71 (.13)
Total, M (SD)	NA (NA)	NA (NA)	12.22 (3.36)	85.51 (10.64)	536.62 (270.39)	10.84 (10.28)	.58 (.24)

Note. ELL = English language learner; SES = socioeconomic status; AYP = adequate yearly progress; Group 0 = met AYP; Group 1 = entered but did not exit; Group 2 = entered and exited.

Model 1: Specifying the Trends in Examining Changes in Process Indicators

In situations where we wish to examine developmental change over two or more different temporal periods, we can specify a piecewise growth model (Raudenbush & Bryk, 2002). Within schools, we specify a baseline (Time 1) linear trend describing perceived changes in each process indicator as schools attempted to meet AYP. ³ For schools that entered NCLB restructuring status, we specify a second linear trend (Time 2) beginning with the year they entered restructuring status. For schools that exited NCLB restructuring status, we specify a third linear trend (Time 3) corresponding with the first of two years they made AYP before exiting. For school i, observed at time t, we define the complete within-school model as follows:

y t i = β 0 i + β 1 i t i m e 1 t i - β 2 i t i m e 2 t i + β 3 i t i m e 3 t i + ε t i .

where β_0i is the initial status intercept for a given process indicator, β _1i describes the expected positive baseline rate of change in a given process indicator for all schools after NCLB implementation, β _2i describes the expected decrement in the baseline trajectory for those schools that entered restructuring, β _3i describes the expected positive increment in the baseline trajectory corresponding with meeting AYP initially in schools that exited restructuring, and ε _ti represents residual variation. We note that schools that were not restructured have β_2i and β_3i coefficients fixed to 0, and schools that did not exit restructuring have β_3i coefficients fixed to 0.

Because there are multivariate y outcomes within schools (i.e., one for each process indicator), we utilized the Mplus modeling framework to define a two-level multivariate growth model (Muthén & Muthén, 1998-2012). More specifically, for school i at time t, we specify the following within-school model:

y i t = μ + B i x i t + ε i t ,

where µ is a vector of measurement intercepts, B i is a matrix of regression coefficients explaining time-related changes ( x i t ) in each indicator (see Equation 1), which can vary between schools, and ε i t is a vector of time-specific errors contained in Ψ W . Between schools, we can collect the random effects regarding the indicator trends as a η i vector as follows:

η i = α + B w i + ζ i ,

where α and B are structural parameters, w i are school-level predictors (if needed), and ζ i is a vector of residuals with covariance matrix Ψ B . In this first model, our main interest is to estimate the piecewise growth models for the four process indicators across the 10-year period of the study and collect the set of random effects at the school level for subsequent comparison.

Model 2: Specifying the Between-School Mixture Model

Second, we specify a two-level growth mixture model. As Muthén (2001) notes, growth mixture models (GMM) are useful in identifying subsets (or mixtures) within a population that share similar growth trajectories. ⁴ Within schools, we specify the same piecewise growth models, as in Equation 2. Between schools, the mixture model describes differences in the growth means among the three known classes, or groups, of schools. At the school level (i), the latent classes are defined with the subscript c to indicate they comprise a categorical latent variable representing the three known groups of schools:

η i = α c + B c w i + ς c i ,

where α c is a vector of class-specific of means summarizing the random time-related slopes for the set of process indicators, B c is a class-specific matrix of regression coefficients for any between-school predictors, and ς c i is a vector of residuals in equations contained in Ψ _B . In mixture modeling analyses, a multinomial logistic regression equation is also specified to explain the probability of membership in the latent classes comprising the categorical latent variable:

P ( C i = c ) = e x p ( α c + β c w i ) ∑ c e x p ( α c + β c w i ) .

This implies the school demographic and staffing predictors (w_i) may explain the probability of membership of the individual schools in the known groups. We summarize the complete model as tested in Figure 3. In the figure, the arrow from the categorical latent variable (C) to the random slope coefficients indicates their means vary across the known latent classes. The arrows from the context and staffing variables to the latent categorical variable indicate they are used to explain membership in the groups.

OPEN IN VIEWER

Figure 3.

Proposed two-level random coefficients growth mixture model with known classes.

Coding the Growth Trajectories

In Table 3, we illustrate how we coded the timing of restructuring status for individual schools in our database. We accomplished this by using a separate coding scheme for the three different trends specified in our piecewise growth model summarized in Equation 1. For each school in the study, yearly changes in each educational process indicator during the first period (Time 1) were coded as 0, 1, . . . , n with initial status (2002) defined as 0. Schools that did not enter restructuring were coded 0-9 for the 10-year period of the study. For the second period (Time 2), each of the restructured schools entered restructuring status during various years beginning in 2005. A second trajectory was coded as 0, 1, . . . , n for these schools, which represents an expected decrement to the baseline change trajectory during the period of restructuring. As shown in Table 3, a school that entered restructuring status during the third year after implementation (T₂ = 2005) would be coded 0 for 2005, indicating its entry into restructuring. For Time 3, the subset of restructured schools that subsequently exited were hypothesized to follow another modified trajectory (defined as T₃), which was also coded 0, 1, . . . , n with 0 specified as the first year of two consecutive years the school met AYP.

OPEN IN VIEWER

Table 3.

Example of Coding of Schools That Were Not Restructured, Schools That Entered Restructuring (Time 2 = 0), and Schools That Exited Restructuring (Time 3 = 0) During NCLB.

	2003	2004	2005	2006	2007	2008	2009	2010	2011	2012
Not restructured	0	1	2	3	4	5	6	7	8	9
Restructured	0	1	T2	1	2	3	4	5	6	7
Exited restructuring	0	1	T2	1	2	3	4	T3	1	2

Note. NCLB = No Child Left Behind.

Examining the Mean Distributions of Changes in Process Indicators

We first examine the distributions of perceived changes in the process indicators, which were estimated from our piecewise growth model specifying the three hypothesized trends after NCLB implementation (i.e., see Model 1 in Table 4). Our application of PE theory to NCLB restructuring events suggests we should observe contrasting distributions of perceived organizational changes during more stable periods versus punctuated periods after schools entered restructuring. As True et al. (1999) noted, if changes were truly random and not subject to a punctuated process, the frequency of change magnitudes should follow a normal distribution. One simple test is to examine the kurtosis of the distribution. High values of kurtosis represent an empirical distribution that has more observations at the peak and the tails of the distribution than a normal distribution; therefore, it can serve as a test of whether or not a frequency distribution is shaped as one would expect from a PE process (Robinson, 2007). We centered the estimates on 0, with 0 defining “average” perceived change for each period.

OPEN IN VIEWER

Table 4.

Simultaneous Estimation of Piecewise Growth Models for Entire Sample (Model 1) and Known Groups (Model 2).

Variable	Model 1				Model 2
	Total (N = 74)				Group 0 (N = 28)		Group 1 (N = 19)		Group 2 (N = 27)
	Estimate	SE	Kurtosis	SE	Estimate	SE	Estimate	SE	Estimate	SE
Time 1
Leadership	4.74	0.22	0.84	0.55	4.86	0.10	4.75	0.08	4.61	0.08
Improvement	5.69	0.17	0.81	0.55	5.82	0.05	5.48	0.08	5.71	0.06
Classroom	6.37	0.11	0.37	0.55	6.26	0.11	6.46	0.10	6.41	0.07
Support	5.53	0.19	−0.75	0.55	5.58	0.07	5.52	0.09	5.48	0.09
Time 2 a
Leadership	−0.96	0.44	3.13	0.68			−0.86	0.08	−1.03	0.09
Improvement	−0.67	0.35	3.24	0.68			−0.56	0.08	−0.76	0.09
Classroom	−0.53	0.33	−0.10	0.68			−0.76	0.06	−0.37	0.09
Support	−1.35	0.35	1.82	0.68			−1.32	0.11	−1.37	0.07
Time 3 b
Leadership	1.83	0.47	−0.72	0.87					1.83	0.07
Improvement	1.32	0.33	0.30	0.87					1.32	0.05
Classroom	0.57	0.22	−1.07	0.87					0.55	0.04
Support	1.21	0.32	−1.21	0.87					1.21	0.06

a

Model 1_{T1 + T2} versus Model 1_T1 Δχ²_{(4 df)} = 66.448, p < .001. ^bModel 1_{T1 + T2 + T3} versus Model 1_{T1 + T2} Δχ²_{(4 df)} = 24.892, p < .001.

First, during the initial trend before restructuring (Time 1), Model 1 in Table 4 indicates that teacher perceptions of changes in each process indicator were normally distributed. Kurtosis coefficients ranged from −0.75 to 0.84, standard error (SE) = 0.55. We can use the ratio of the kurtosis coefficient to its SE as a simple statistical test of whether or not the distribution is normally distributed, with ratios higher than 1.96 indicating a significant departure from normality at p < .05 (Rice, 1996). Applying this test to the Time 1 estimates, all perceptions of change were normally distributed. Second, once schools entered restructuring (Time 2), the trends describing teacher perceptions of changes for three of the four indicators were not normally distributed (with ratios of kurtosis to the standard error exceeding 1.96, p < .05). Only perceived change in classroom instructional practices was normally distributed. These results suggested that, within restructured schools, teacher perceptions of change followed a punctuated sequence for the majority of process indicators—that is, with more perceived small-magnitude and large-magnitude changes than expected in a normal distribution. Third, for schools that exited, initially meeting AYP targets (Time 3) corresponded with small positive perceived changes in all indicators, with all change distributions again consistent with normal distributions.

In Figure 4, for Time 2 we illustrate the results visually for changes in leadership, sustained improvement, and student support. The figure indicates the density of observations at various magnitudes of change. For comparison purposes, we overlaid a normal curve with the same mean and standard deviation as the sample on the empirical distribution for each trend. During Time 2, for example, we can see among schools that entered restructuring the perceived changes for leadership, sustained focus on improvement, and student support do not appear to follow normal distributions. This set of results in Figure 4 provides empirical support for our hypotheses regarding expected variation in the nature of teacher perceptions about changes in key process indicators during different periods of NCLB implementation.

OPEN IN VIEWER

Figure 4.

Distributions of teacher-perceived changes in focus on sustained improvement, leadership, and student support after entering restructuring.

Model 1: Examining the Piecewise Growth Trajectories

Table 4 presents the piecewise growth trajectories for the process indicators. The metric in the table is the estimated percentage of teachers in the school who strongly agreed or agreed with the quality of each indicator as implemented in their school. Model 1 provides the average slope estimates for the relevant sample of schools at each hypothesized time point in our proposed model (i.e., 74 schools at Time 1, 46 schools at Time 2, 27 schools at Time 3). Model 2 provides the results of our growth mixture model, with growth parameters separated by the three known groups of schools. We centered the initial status intercepts for each indicator at 50.00, so we do not include them in Table 4.

For Model 1, the intraclass correlations (ICC) for the initial status intercepts (not tabled) suggested considerable variation in teacher perceptions of each indicator between schools (i.e., ICC_Lead = 0.34; ICC_Improve = 0.40; ICC_Class =0.35; ICC_Support = 0.44). The coefficients indicate, for example, about 34% of the variation in leadership perceptions and about 44% of the variation in student support perceptions were between schools. ⁵ Larger ICC coefficients indicate stronger agreement among teachers within schools regarding the changes and, hence, greater differences between schools in perceptions about the quality of their educational processes. We would expect considerable variation in perceptions of the quality of school processes, given schools’ varied success in meeting AYP during the initial period. We also found ICC coefficients were larger in the group of schools that consistently met AYP than in the group that eventually entered restructuring status. This indicated greater agreement (or cohesion) among teachers regarding changes in the former group than among teachers in the latter group.

The Model 1 results suggest that the specified model captured schools’ process changes in expected ways, with all process slopes statistically significant at p < .05. ⁶ More specifically, during the baseline period (Time 1), teacher perceptions of the average improvement in each indicator was positive. This implies perceived improvement as schools attempted to meet rising AYP targets. Of course, the actual trajectories of individual schools deviated considerably from the average trajectories (e.g., see Figures 5 and 6). Among schools that entered restructuring, as expected, perceived change in process indicators slowed on average while the schools were in restructuring status (Time 2). In schools that exited restructuring, perceptions regarding processes changed direction again once they initially met AYP, indicating positive increments (Time 3). The pattern of results in Model 1 therefore also provided evidence of the model’s validity in depicting the hypothesized direction of changes in teacher perceptions regarding internal educational process indicators corresponding with schools’ NCLB compliance status.

OPEN IN VIEWER

Figure 5.

Estimated mean collaborative leadership trajectory and observed trajectories of schools entering and exiting restructuring (Group 2).

OPEN IN VIEWER

Figure 6.

Estimated mean collaborative leadership trajectory and observed trajectories of schools entering but not exiting restructuring (Group 1).

We also constructed chi-square tests to examine the fit of alternative time-related models to the data. First, we found that a piecewise model with Time 1 and Time 2 trend parameters fit the data better than an initial model, which assumed only a single (Time 1) trend over time (Δχ ² , 4 df = 66.448, p < .001). This provided empirical evidence confirming that entering restructuring was associated with a statistically significant, decrement in teacher perceptions regarding the process indicators. Moreover, because schools entered restructuring in different years, this result strengthens the evidence from the time-series design that teachers’ perceptions of process trends before and after entering restructuring changed noticeably after the event. ⁷ Second, we found that the complete piecewise model (i.e., Model 1 in Table 4), which included Time 1, Time 2, and Time 3 slope parameters, fit significantly better than the previous model with only the Time 1 and Time 2 parameters (Δχ ² , 4 df = 24.892, p < .001). Once again, because schools exited at multiple times after entering restructuring, this result provides further empirical evidence of a second significant change in teacher perceptions regarding improvement in educational processes in schools that exited restructuring. This new trend corresponded with the timing of initially meeting AYP targets.

Because individual trajectories can vary considerably from average estimated trajectories (Singer & Willett, 2003), we also examined plots of the average estimated Model 1 process trajectories of each group against the actual teacher-perceived changes in that group. We noted the actual perceptions for each indicator followed a similar trajectory to the predicted trajectory. As an example, we provide illustrations of predicted and actual trajectories regarding the leadership indicator among a subset of schools that exited restructuring (Group 2) and a subset that did not exit (Group 1) in Figures 5 and 6, respectively. The figures illustrate the different trends in the two groups of restructured schools regarding the perceptions of leadership processes. Overall, the match between our estimated trajectories and the actual trajectories of teacher perceived changes in the four process indicators led us to conclude our piecewise growth model in Table 4 captured the actual changes in teacher perceptions of their schools’ practices accurately during the proposed periods associated with NCLB implementation.

Model 2: Examining Differences in Processes and School Covariates Among Groups

The last part of our analysis examines which observed changes in school process indicators and school contextual variables were most associated with membership in the three groups of schools comprising the sample. We summarize the results in two tables for ease of presentation. Model 2 in Table 4 provides the observed means of the process indicators for the “known” groups of schools. Table 5 provides the corresponding between-group multinomial logistic regression coefficients that explain individual membership in the three known groups.

OPEN IN VIEWER

Table 5.

Multinomial Logistic Regression Results of School Context Variables Explaining Group Membership.

			95% Confidence Interval		Odds Ratio
	Log Odds	SE	Lower	Upper
Reference group = Group 0
Group 1 (Restructured)
Staff teaching experience	−0.205**	0.102	−0.440	0.045	1.068
Fully qualified teachers	−0.067	0.056	−0.180	0.044	0.908
Enrollment size	0.006***	0.002	0.001	0.010	1.006
English language services	−0.005	0.041	−0.087	0.076	0.995
Low-SES composition	1.175**	0.559	0.081	2.266	3.238
Group 2 (Exited)
Staff teaching experience	0.062	0.102	−0.193	0.327	1.064
Fully-qualified teachers	0.005	0.043	−0.080	0.090	1.006
Enrollment size	−0.002	0.002	−0.005	0.002	0.998
English language services	−0.147***	0.042	−0.228	−0.064	0.882
Low-SES composition	2.351***	0.457	1.489	3.191	10.496
Reference group = Group 1
Group 2 (Exited)
Staff teaching experience	0.267*	0.158	−0.053	0.567	1.306
Fully qualified teachers	0.072*	0.042	−0.016	0.161	1.075
Enrollment size	−0.007***	0.002	−0.011	−0.004	0.993
English language services	−0.142**	0.047	−0.236	−0.046	0.882
Low-SES composition	1.176*	0.636	−0.085	2.418	3.241

Note. SES = socioeconomic status.

*

p < .10. **p < .05. ***p < .01.

During the first trend (Time 1), the Model 2 results suggest teachers in schools that consistently met AYP (Group 0) perceived greater positive changes in collaborative leadership, focused and sustained action on improvement, and the quality of student support programs than teachers in schools that eventually entered restructuring. We utilized discriminant analysis as a follow-up means to determine which process indicators primarily separated the three groups during each trend, since this type of analysis is not part of the mixture model output. The analyses indicated that the means of the process indicators differed significantly across the groups of schools during Time 1 (p < .001), with perceptions regarding focus on sustained improvement dominating in separating schools that were not restructured (Group 0) from schools that were subsequently restructured (Groups 1 and 2). Additionally, differences in perceived improvement and collaborative leadership dominated in separating schools that entered restructuring and exited (Group 2) from schools that entered but did not exit (Group 1). ⁸ To interpret these results more specifically, Model 2 in Table 4 shows that during Time 1 mean perceptions regarding their school’s focus on sustained improvement were higher in Group 0 (5.82) than in Group 1 (5.48) and Group 2 (5.71). Means for sustained improvement were higher in Group 2 than Group 1, but perceived changes in collaborative leadership were higher in Group 1 (4.75) than Group 2 (4.61). This evidence is consistent with the view that teachers in schools that met AYP consistently perceived greater attention focused on sustained academic improvement than teachers in schools that eventually entered restructuring. These latter schools were dealing with NCLB progressive improvement steps (i.e., improvement, corrective action, planning) along the way, as we earlier summarized in Figure 1.

With respect to Time 2, the results indicated that during restructuring, teachers in schools that eventually exited perceived larger declines in all process indicators except classroom instructional practices compared with teachers in schools that did not exit restructuring. Follow-up discriminant analysis indicated that during restructuring (Time 2), the means of the four process indicators differed significantly across Groups 1 and 2 (p < .01). Group differences were primarily due to perceptions regarding changes in classroom practices and, to a lesser extent, changes in focus on sustained school improvement and collaborative leadership. ⁹ In Table 4, during restructuring, the average negative disruption in classroom practices was considerably larger in Group 1 (−0.76) than Group 2 (−0.37) schools. Mean declines in focus on sustained school improvement were larger in Group 2 schools (−0.76) than in Group 1 schools (−0.56), as were perceived declines in collaborative leadership (i.e., −1.03 vs. −0.86, respectively). Finally, with respect to schools that exited, the means describing changes in all indicators were substantial and positive (Time 3), with all effects significant at p < .001.

In Table 5, we present the results of the multinomial regression model to predict school membership in the known groups. The logit coefficients show the change in the predicted logged odds of membership in the group specified versus the reference group for a one-unit change in the independent variables. We provide tests at p < .10 and p < .05, since the sample size for schools that did not exit was a bit smaller than ideal.

First, compared with the reference group which consistently met AYP targets (Group 0), schools which entered but did not exit restructuring (Group 1) were characterized by lower average years of staff teaching experience (log odds = −0.205, p < .10), larger student enrollments (log odds = 0.006, p < .01), and larger concentrations of low-SES students (log odds = 1.175, p < .10). Compared to schools that consistently met AYP, schools that entered and exited restructuring (Group 2), were characterized by lower proportions of students receiving English language services (log odds = −0.147, p < .01), and larger concentrations of low socioeconomic students (log odds = 2.351, p < .01). Since both Group 1 and Group 2 schools had significantly higher concentrations of low-SES students than the comparison group, we can isolate SES composition as a common variable leading to the probability of entering restructuring. More specifically, the performance of low-SES students was one subgroup criterion in meeting AYP. Lower average teaching experience and larger student enrollments also appeared to play some role in differentiating schools that consistently met AYP from schools that entered but did not exit restructuring, and lower percentages of students requiring English language services (another subgroup criterion) appeared to define schools that entered and managed to exit restructuring compared to the reference group of schools.

Second, Table 5 also provides information regarding how schools that exited (Group 2) differed from schools that did not exit (Group 1). Using Group 1 schools as the reference group, the results suggest schools that exited restructuring had greater staff teaching experience (log odds = 0.267, p < .10), greater percentages of fully qualified teachers under NCLB guidelines (log odds = 0.072, p < .10; smaller student enrollments (log odds = −0.007, p < .01), smaller percentages of students receiving English language services (log odds = −0.142, p < .05) and smaller proportions of low-SES students (log odds = 1.176, p < .10). These results imply schools exiting restructuring had somewhat greater teaching capacity and lower concentrations of some targeted AYP subgroups than schools that did not exit restructuring.

Did NCLB Restructuring Disrupt Normal Educational Processes in Restructured Schools?

Dynamic analyses of organizational processes center on changes in social interactions, leadership, and decision-making structures in explaining how organizations evolve through a process of convergence and reorientation (Kelly & McGrath, 1988; Langlois & Robertson, 1993). We constructed a dynamic model focusing on the timing of mandatory restructuring and a key internal response—initially meeting AYP targets—and their effects on teacher perceptions of changes of school process indicators. Our goal was to enhance the theoretical and practical understanding of how schools may respond to external mandates for academic improvement.

We found NCLB restructuring, as an external policy action, was largely successful in disrupting process indicators among prolonged ineffective schools in ways that led to some reconfiguration of their leadership and organizational processes. More specifically, the results confirmed the timing of entering restructuring status produced a significant discontinuity in teacher perceptions regarding the quality of leadership, instructional practices, student support resources, and sustained focus on improvement compared with their perceptions before they entered restructuring and their peers’ perceptions in schools that consistently met AYP. This finding was consistent with Hochbein and Duke (2011), who noted the quality of internal processes was largely responsible for prolonged failure in chronically ineffective schools. Follow-up analyses pointed to larger differences in school-wide focus on sustained improvement and collaborative leadership primarily separated schools that consistently met rising AYP targets from schools that were failing to meet AYP and progressing deeper into NCLB sanctions. More specifically, teachers in schools meeting AYP perceived a stronger school focus directed toward student academic improvement and greater implementation of collaborative leadership than teachers in schools that entered restructuring. These dimensions included actions such as assessing student academic needs, building staff commitment and participation for improving the school’s instructional capacity, making changes in curriculum and instructional practices, and sharing responsibility for student success.

For those schools that entered restructuring, teachers’ perceived a considerable loss of cohesion regarding three key internal indicators, as evidenced by increased variation in responses and a non-normal distribution of the perceived changes, consistent with punctuated change processes (Baumgartner & Jones, 1993; Haveman et al., 2001; Tydeman, 2009). Follow-up results indicated statistically significant differences in the set of indicators between the two groups of schools that entered restructuring. Group differences were largely the result of teacher perceptions regarding the status of classroom practices and, to a lesser extent, their school’s long-term focus on improvement. In particular, teachers in schools that eventually exited restructuring perceived less disruption in the quality of their classroom practices during restructuring than teachers in schools that did not exit restructuring.

Did Restructuring Produce Any Noticeable Internal Changes in Schools That Exited?

Our other concern was examining whether disruptions in normal practices of failing schools translated into changes in practices leading to school improvement. We found a considerably greater proportion of elementary schools in Hawaii (59%) were able to exit restructuring status than in several previous studies on NCLB restructuring, where success in exiting over the periods studied was below 25% (CEP, 2009) and 16% (CEP, 2008). Among schools that exited restructuring, we isolated a second significant change in teacher perceptions regarding the quality of school processes. This change corresponded with meeting AYP targets in the first of two consecutive years before exiting. The largest perceived changes were in the effectiveness of the school’s collaborative leadership and its focus on sustained action for improvement. This finding was consistent with research identifying shared leadership’s strategic role in school improvement (e.g., Day et al., 2010; Duke, 2008; Hallinger & Heck, 2011). This second discontinuity in teacher responses implies changes in these processes were associated with observed improvement in meeting AYP targets. Although the timing of the process changes was concomitant with AYP improvement, their impact on student learning was not as clear. This is because meeting AYP targets was not simply a matter of instructional improvement—given the potential for different subgroup components to define AYP across school settings.

Finally, our examination of school context and staffing indicators also contributed information regarding differences between schools that exited restructuring and schools that did not. We found exiting schools had higher average staff experience, higher percentages of fully qualified teachers under NCLB guidelines, lower student enrollments, and lower proportions of low-SES students and students needing English language services than schools that did not exit. This provided some evidence that student composition often provides additional challenges in meeting performance targets (Duke, 2008; Hochbein & Duke, 2011; Pruitt & Bowers, 2014), while teacher staffing variables (e.g., experience, credentialing) are often positively associated with school improvement (e.g., Duke, 2008; Heck, 2007, Hallinger & Heck, 2011). We caution, however, that school improvement defined as meeting AYP at best is only “broadly consistent” with previous studies that assessed improvement with stronger, more clearly-defined outcome measures such as growth in student achievement scores.

Can We Isolate Relevant Internal Conditions Leading to AYP Improvement?

If we examine improvement broadly over time, we find support for the view that schools as organizations do evolve through a process of convergence and reorientation (Kelly & McGrath, 1988; Van de Ven & Poole, 1990). More specifically, the process of evolutionary development took a decade or longer for schools that experienced prolonged decline in AYP productivity. These schools often had significant contextual challenges to begin with, compared with schools that consistently met AYP, including greater concentrations of low socioeconomic status students, higher student enrollments, and lower percentages of fully qualified teachers under NCLB than schools that met AYP targets consistently. This prolonged failure to meet AYP led to restructuring, followed by a period in restructuring status often extending four or more years, before they met AYP targets consistently enough to exit and return to normal accountability status. This implies that school improvement often takes a considerably longer time to unfold than policymakers may assume (Day et al., 2010; Hallinger & Heck, 2011; Louis & Miles, 1991).

We were less successful in determining how changes among relevant process variables were associated with improvement in meeting AYP after the varied time schools spent in restructuring status. For example, it is unclear from our data whether the schools facing the greatest AYP challenges due to their diverse student demographics were likely to fail anyway—because of the number of AYP subgroup targets they had to meet—regardless of the quality of their instructional programs or improvement efforts (e.g., CEP, 2008; Manwaring, 2010). Although NCLB sanctions disrupted the inner workings of schools, in some cases, they may have resulted in needless time and financial resources spent on meeting specific subgroup targets that could have been placed elsewhere to impact overall educational improvement (Forte, 2010). For example, previous research has noted the increased resources and scope of reform effort necessary to improve schools with challenges due to student composition (Duke, 2008; Hochbein & Duke, 2011; Murphy & Meyers, 2007; Rothstein, 2004; Rowan et al., 2004). Another challenge in determining the possible impact of restructuring on academic improvement is that, although NCLB sanctions provided a well-defined series of progressive steps leading up to restructuring, “restructuring” itself could have resulted from very different shortcomings related to meeting AYP (Forte, 2010)—even among a set of elementary schools in the same state utilizing the same tests to evaluate AYP. This was because the number of subgroups included in AYP criteria could differ widely across schools due to student background diversity, resulting in varied strategies schools used to meet AYP proficiency standards (Neal & Schanzenbach, 2010).

Our anecdotal evidence in talking with personnel at schools that exited suggested they utilized two primary “other” NCLB options in addressing improvement. One strategy was to use student achievement data to focus improvement on the “low hanging fruit” students who were close to being proficient (e.g., Neal & Schanzenbach, 2010). This was an expedient strategy to improve the likelihood of meeting specific AYP shortfalls. A second strategy used aggregate data to focus instructional improvement in areas where students were struggling. Most schools used a combination of both strategies. Neither strategy, however, suggested schools made considerable changes in teachers’ actual classroom practices (e.g., Mintrop & Trijillo, 2007). The fact that teachers did not perceive changes in classroom practices were “punctuated” during restructuring is consistent with the view that teachers’ classroom routines are generally protected organizational space, even during disruptions of significant external reform (e.g., Cuban, 1990; Tyack & Cuban, 1995). As Cuban argued, even with some formal accountability, daily delivery of instruction is virtually decoupled from administration and policymaking.

We note that readers should consider our results along with several limitations. First, we found the time-time series design, coupled with piecewise growth modeling, was a useful approach in testing and confirming the existence of separate trends regarding changes in school processes by facilitating the consideration of various alternative models. We note, however, in this type of longitudinal modeling of events, if time intervals are poorly chosen, it is possible to miss the hypothesized effect because the interval chosen was either too short to detect the effect or too long, and so the effect faded. To address this potential problem, in our model investigations, we applied suggestions for examining varied lengths of possible causal lags (Selig & Preacher, 2009). We also tested alternative time-series models with fewer trends and found them less accurate in describing the data. We took care to center the timing of our last hypothesized process trend on the year each restructured school initially met AYP targets. We caution, however, that changes in teacher perceptions, rather than driving internal changes, likely result from simultaneous internal school changes taking place—such that observed changes in leadership, instructional capacity, and student support and improvements in outcomes likely represent a mutually reinforcing system (Hallinger & Heck, 2011).

Second, we caution that longitudinal models examining the timing of changes in organizational processes and subsequent improvement in productivity do not provide complete protection against rival explanations, for example, possible selection-bias arguments (Cook & Campbell, 1979). For example, teachers may perceive changes in internal processes more positively in schools with fewer demographic challenges or schools that are generally higher achieving. They may also seek those types of schools in which to work. Related to this point, given that NCLB had numerous ways that schools with greater student diversity could fail to meet AYP, compared with more homogeneous schools, it is more difficult to conclude that failure to meet AYP targets was solely due to poor instructional practices and, similarly, that addressing this problem would guarantee AYP success (Kim & Sunderman, 2004). We also reiterate in a longer period of study, other schools likely would also have exited restructuring—a limitation which is consistent with most longitudinal studies examining event histories.

Third, despite ample evidence that the annual school survey measured the process indicators well, there may be other indicators, such as the types of external support provided to the schools, which contributed to AYP improvement. It would be a mistake to think the richness and complexity of organizational change can be fully measured by a single instrument administered on a yearly basis. The study did not capture the day-to-day actions that influenced improvement; so while it may be preferable to make multiple observations regarding changes over the course of a year, it is currently not typically done. Although there is likely some bias in these types of broad measures of schools’ instructional practices, our results suggest they were useful in monitoring schools’ internal processes within the NCLB policy environment.

Despite these limitations, we note the time-series approach was promising in isolating the timing of external policy events, such as NCLB restructuring, and their subsequent impact on changes in school processes indicators. We are less optimistic that NCLB’s progressive sanctions led to substantial changes in the quality of instructional practices, as school personnel tended to “tinker” with more expedient, and less disruptive instructional strategies to meet accountability demands (Cuban, 1990; Tyack & Cuban, 1995). This was a weakness within the policy itself, which aimed at externally mandated improvement, yet without a concrete definition of its meaning or any empirically tested means of attaining it. Regarding regulatory punctuations, we noted that for schools that had been declining over time, the timing of progressive sanctions such as corrective action appeared to provide some advantage in terms of the extent to which actually landing in mandatory sanctions subsequently may have stimulated organizational responses making it possible for intentional (and unintentional) changes to take hold (Haveman et al., 2001). Further research examining the timing of accountability policies and internal school responses may build on our initial effort by including more detailed school measures regarding the implementation and effects of alternative internal strategies that school personnel utilized in successfully improving performance. While acknowledging the interaction of local context with any improvement strategy, we are still in need of more-detailed information regarding how personnel choose strategic actions and the extent to which such actions penetrate well-engrained classroom routines. Such information would help construct a more complete picture of how accountability mandates and local school responses can facilitate academic improvement in chronically underperforming schools.