Changes in School Engagement as a Function of No Child Left Behind: A Comparative Interrupted Time Series Analysis

School Engagement and Student Outcomes

School engagement refers broadly to a student’s investment in or commitment to school. It is conceptualized and measured as an inherently multidimensional construct encompassing students’ behaviors, thoughts, and feelings about school (Fredricks et al., 2004; Reschly & Christenson, 2012; Wang & Degol, 2014; Wang, Fredricks, Ye, Hofkens, & Linn, 2017). Behavioral engagement measures school participation and may include measures of attendance and participation in extracurricular, school-based activities. Cognitive engagement reflects mental investment in school. It can be measured with indicators of school relevance and endorsements of the long-run benefits of school as well as measures of academic effort and thoughtfulness. Emotional engagement includes students’ affective responses to school, peers, and teachers, and is defined as students’ sense of belongingness in or connectedness to school as well as their enjoyment of school. These dimensions are meaningfully linked within individuals and together represent a holistic sense of students’ investment in school.

Developmental theories such as stage-environment fit theory (Eccles et al., 1993; Wigfield, Eccles, Schiefele, Roseser, & Davis-Kean, 2006) and self-determination theory (Deci et al., 1991; Deci & Ryan, 1994; Ryan & Deci, 2009) assert that the social and emotional climate of a school engenders student motivation, which is acted out through engagement. Engaged students internalize the goals, values, and skills promoted by schools, leading to student effort and use of school-based supports in the pursuit of healthy developmental goals, resulting in enhanced academic and socio-emotional outcomes. Conversely, disengaged students become disillusioned with the tasks and requirements of school, do not take advantage of school supports for academic and socio-emotional growth, and may leave school altogether.

Evidence for the importance of engagement comes from an extensive body of literature linking school engagement with a variety of student outcomes, both academic and socio-emotional (Fredricks et al., 2004; Wang & Degol, 2014). Engagement has been linked to both effort in school and academic achievement (L. H. Anderman, 2003; Appleton et al., 2008; Goodenow, 1993; Thompson, Iachan, Overpeck, Ross, & Gross, 2006; Wang & Holcombe, 2010). For example, Wang and Holcombe (2010) found that students’ school environments were linked to their academic achievement in adolescence through their school engagement. Engagement is also related to fewer suspensions (McNeely, 2005) and school completion (e.g., Appleton et al., 2008; Archambault, Janosz, Morizot, & Pagani, 2009; Wang & Peck, 2013). For example, Wang and Peck (2013) demonstrated that students with low levels of school engagement were 19 more percentage points more likely to drop out of school than their peers with higher levels of engagement. Socio-emotionally, decreases in engagement have been linked with increases in substance use, delinquent behavior (Hirschfield & Gasper, 2011; Li et al., 2011; Li & Lerner, 2013; Wang & Fredricks, 2014; Wang & Peck, 2013), and depressive symptoms (Li & Lerner, 2013; Wang & Peck, 2013) among students aged 10 to 17 across diverse racial and ethnic groups. For example, Hirschfield and Gasper (2011) found that engagement was associated with lower level of delinquent behavior both in and out of school among early adolescents, a finding replicated by Li and colleagues (2011). Similarly, Henry, Knight, and Thornberry (2012) demonstrated that the absence of engagement was a robust predictor of high school dropout, substance use, and delinquent behavior.

School Features and School Engagement

Research indicates that although engagement is driven in part by student characteristics, school characteristics also influence students’ engagement, suggesting that engagement can be altered by educational policy. Indeed, previous experimental and observational research has demonstrated school-level variability in student engagement and identified features of schools linked to students’ engagement (e.g., E. M. Anderman, 2002; Battistich et al., 1995; Fredricks et al., 2004; Koth, Bradshaw, & Leaf, 2008; McNeely, Nonnemaker, & Blum, 2002; Payne, 2008). For example, Solomon, Battistich, and colleagues report that an intervention designed to improve the socio-emotional climate of a school by promoting cooperative learning, developmentally informed discipline policies, interpersonal helping, and prosocial behavior increased students’ engagement with school (Battistich et al., 1997; Solomon et al., 1996). Similarly, McNeely and colleagues (2002) report that students have higher levels of engagement in schools with fewer total students, responsive discipline policies, and better classroom management using a nationally representative sample. Using the same data, McNeely and Falci (2004) demonstrated that the beneficial influence of emotional engagement with school was largely driven by students’ reported relationships with their teachers. These findings align with conceptual work highlighting the importance of students feeling autonomous and responded to (e.g., Deci & Ryan, 1994; Ryan & Deci, 2009). Taken together, this literature suggests that schools that respond to students’ needs and interests, discipline students in developmentally appropriate ways, build positive relationships between students and school personnel, and promote a positive social and emotional milieu can effectively foster students’ engagement.

School Engagement and No Child Left Behind

Despite evidence suggesting that school engagement is both important for student development and influenced by the school context, school engagement is not accounted for in many state and federal policy initiatives (Cohen et al., 2009; Fredricks et al., 2004; Osterman, 2000; Wang & Fredricks, 2014), including the controversial federal education policy, No Child Left Behind. Signed into law in January of 2002 and effective through 2015, NCLB required that all public school children in states that accepted federal dollars become proficient in math and reading by 2014. It held schools accountable to this goal by testing whether schools made Adequate Yearly Progress (AYP), as determined by the state, toward 100% proficiency each year. Schools administered a state-created, standards-based assessment each spring, and schools that failed to meet AYP were required by law to take specific corrective actions, including informing parents of schools’ AYP status, offering students transfer to schools meeting AYP, and offering supplemental services. Sanctions became increasingly stringent if schools continued to fail, culminating in a complete restructuring and restaffing of the school.

NCLB emerged from a growing accountability movement in education that had several critics in the 1990s and early 2000s. Scholars asserted that strong accountability provisions, particularly those tied to sanctions as in NCLB, were likely to undermine student engagement through several mechanisms. First, consequential accountability policies could incentivize teachers to focus exclusively on tested material rather than content that may be of interest and motivating to students (Dee, Jacob, & Schwartz, 2013; Griffith & Scharmann, 2008; Hannaway & Hamilton, 2008; McMurrer, 2007; Pederson, 2007). Students may disengage from content that seems relevant only in the context of a test. Second, teachers may teach this content in ways that reduce student interaction and stifle critical thinking, such as lecture-based and test preparation pedagogies specifically tied to how questions are asked on the high-stakes assessments (Au, 2007; Diamond, 2007). This shift in pedagogy may interfere not only with students’ interest in academic content but also their emotional engagement with school. Classroom strategies that reduce opportunities for students’ autonomy and peer interactions have been previously shown to adversely affect students’ bonding with their teachers and each other as well as their investment in school more generally (e.g., Connell & Wellborn, 1991; Downer, Rimm-Kaufman, & Pianta, 2007; McNeely et al., 2002). Third, a focus on test material may incentivize teachers to spend less time getting to know students in order to maximize instructional time. Teacher relationships are a core component of students’ emotional connection to school; the absence of these relationships is likely to decrease engagement (e.g., Murray, 2009; Roorda, Koomen, Spilot, & Oort, 2011; Wang & Holcombe, 2010; Woolley & Bowen, 2007). Finally, the stress engendered by strict accountability may reduce teacher responsivity and increase negative mood—stress that is likely to intensify as the probability of sanctions increases. Teachers may be more likely to view and react to disruptions as threatening to their job security rather than normative youth behavior. This stress and frustration may alter disciplinary strategies and reduce teachers’ ability to bond with students, decreasing students’ emotional bond with school, their sense of safety and belonging at school, and ultimately their engagement (Deci et al., 1991; Fredricks et al., 2004; McNeely et al., 2002; Osterman, 2000).

It is also possible that NCLB had a positive impact on students’ engagement. Students may have responded favorably to the heightened focus on academics and higher academic expectations engendered by accountability policies. High academic expectations may communicate to students that they are important and valued by their teachers, potentially enhancing emotional engagement with school. For example, literature exploring the impact of academic press, the term used to describe school climates that emphasize academic excellence (Goddard, Sweetland, & Hoy, 2000), suggests that an academic focus facilitates bonding with school, a key component of emotional engagement with school (Ma, 2003). Similarly, Dee and Jacob (2011, described in more detail in the following) report a positive NCLB effect on teachers’ report that absenteeism, tardiness, and apathy are not a problem in their school, providing some evidence that teachers perceived enhanced indicators of behavioral engagement post-NCLB. Additionally, academic achievement is one of the strongest predictors of student engagement. Previous work examining the relationship between accountability policies and academic achievement found a positive relationship between accountability and academic achievement (Carnoy & Loeb, 2002; Hanushek & Raymond, 2005; Jacob, 2005; Neal & Schanzenbach, 2010). Insofar as NCLB was able to enhance student achievement, as has been demonstrated in several evaluations (Dee & Jacob, 2011; Wong et al., 2015), it may have also boosted engagement.

Prior work examining NCLB impacts has not been able to adjudicate among these hypotheses. Because of the national scope of the law, it is difficult to identify an appropriate comparison group, and evaluations of the law have been few. Only one national analysis has examined whether NCLB impacted students’ enjoyment of learning. Reback, Rockoff, and Schwartz (2014) used a sample of fifth graders in the 2003–2004 school year to demonstrate that accountability pressure from NCLB, operationalized as whether or not students’ schools were close to the AYP margin, does not adversely affect students’ enjoyment of math and reading as reported by students. Using the same empirical strategy, Whitney and Candelaria (2017) find no influence of NCLB on students’ social and emotional skills, including their perceived interest and competence in academic subjects. However, these studies stand in contrast to a larger body of work that asserts that NCLB, and in particular the testing and accountability components, have increased student anxiety and reduced students’ enjoyment of and engagement in school (e.g., B. D. Jones, 2007; M. G. Jones, Jones, & Hargrove, 2003; Meier & Wood, 2004; Nichols & Berliner, 2007). Moreover, some studies have presented mixed findings. For example, using a rigorous quantitative strategy in a large sample of students in North Carolina, Holbein and Ladd (2017) find that while accountability pressure decreases student absences and tardiness, it also increases student misbehavior.

Research on proposed mechanisms is also inconclusive. The most rigorous designs used to assess NCLB have exclusively examined student achievement. These studies provide strong evidence that NCLB enhanced student mathematics achievement but little evidence that NCLB improved reading achievement (Dee & Jacob, 2011; Wong et al., 2009, 2015). However, scholars have also used teacher and administrator surveys, teacher interviews, and classroom observations to provide evidence that NCLB reduced teacher autonomy, motivation, and morale (Cochran-Smith & Lytle, 2006; Finnegan & Gross, 2007; M. G. Jones et al., 2003); reduced time spent on nontested content and subjects (Dee et al., 2013; Griffith & Scharmann, 2008; Hannaway & Hamilton, 2008; M. G. Jones et al., 2003; McMurrer, 2007; Pederson, 2007); promoted the use of lecture-based and test preparation pedagogies (Au, 2007; Diamond, 2007); and reduced the overall level of instructional support in classrooms (Plank & Condliffe, 2013). Again, most of these studies were local in scope, and several do not include a comparison group.

In sum, the current literature cannot identify whether NCLB has impacted students’ engagement with school, a substantial gap given the potential for this construct to impact important short- and long-term outcomes for youth. Moreover, the literature on the broader impact of NCLB remains contentious and does not permit a strong hypothesis regarding the impact of the law on students’ school engagement. It may be that NCLB’s strict accountability and singular focus on high-stakes test performance reduced student engagement school, as widely hypothesized; however, it may also be that the academic focus created by high-stakes testing and consequential accountability encouraged schools to attend more closely to students’ needs as learners, ultimately facilitating both academic achievement and student engagement.

Estimating NCLB Impacts

Much of the debate over NCLB has been fueled in part by the difficulty of evaluating a complex federal program that simultaneously impacted every public school in the nation; therefore, relatively few studies have evaluated the overall impact of NCLB. Moreover, evidence suggests that NCLB’s deference to states in terms of curriculum standards, test development, and accountability schedules has resulted in uneven implementation across states, with some states only loosely implementing NCLB (Davidson, Reback, Rockoff, & Schwartz, 2013; Steifel, Schwartz, & Chellman, 2007) or postponing their implementation altogether (e.g., “backloading” AYP targets; Chudowsky & Chudowsky, 2008). Indeed, Davidson et al. (2013) report that AYP failure rates varied from 1% to 80% across states. Taken together, the lack of a clear comparison group and the wide variety in treatment implementation has limited research examining NCLB impacts.

The most rigorous evaluations of NCLB to date have addressed these challenges by estimating comparative interrupted time series (CITS) models (e.g., Dee & Jacob, 2011; Lee, 2006; Wong et al., 2009, 2015). CITS models build on a traditional interrupted time series (ITS) model, in which the dependent variable of interest (engagement) is plotted over time. If there is an interruption, or change in the time trend, after the introduction of the law (NCLB in 2002), this change could be due to the influence of the policy. This interruption could occur immediately and would appear as a sudden change in the height of the line, or it could occur cumulatively, appearing as a change in the line’s slope after the introduction of the law (after 2002). In an ITS model, however, unmeasured third variables are likely to influence the time trend and may be correlated with the introduction of the law, resulting in a biased estimate. In a CITS model, a comparison group is introduced that should be influenced by any potential confounds but is not influenced by the treatment of interest. If there is a change in the relationship between the treatment and comparison groups after the introduction of NCLB, this is interpreted as the result of the policy. If the change in the comparison group after the introduction of the policy accurately reflects what would have happened to the treatment group, this differential change can be interpreted as a causal effect; if there are systematic differences between the treatment and comparison group that vary over time, then the estimate may still be biased.

Three previous studies have used a CITS design to explore the national impact of NCLB on academic achievement. These studies took advantage of between-state differences in NCLB’s implementation by creating synthetic comparison groups, or groups of states that were subject to the broader American policy and educational context in the years before and after NCLB’s enactment in 2002 but were arguably not impacted by NCLB itself. For example, Lee (2006) and Dee and Jacob (2011) contrasted states with and without consequential accountability (CQA) policies prior to the implementation of NCLB. These authors took advantage of the 30 states that had implemented some form of CQA in the 1990s, prior to NCLB (see Table 1). These authors reasoned that the thrust of NCLB was the introduction of federally mandated consequences for poorly performing schools; if states already applied consequences to failing schools, then NCLB should have a much smaller impact on these states. Put another way, if CQA does impact student engagement, this impact should be larger after 2002 in states experiencing CQA for the first time. This impact could occur immediately, shown in a change in the intercept for treated versus comparison group states, or over time, shown in a change in the post-NCLB slope for the treated versus comparison group.

OPEN IN VIEWER

Table 1

States With Consequential Accountability Prior to No Child Left Behind

Early Implementers, Baseline Specification
States	Year
IL	1992
WI	1993
TX	1994
IN, KS, KY	1995
NC, NV, OK	1996
AL, RI, WV	1997
DE, MA, MI, NM, NY, VA	1998
AR, CA, CT, FL, LA, MD, SC	1999
VT, GA, OR, TN	2000
AK	2001

Source. Dee and Jacob (2011).

Similarly, Wong and colleagues (2009) contrasted states that implemented NCLB with varying levels of rigor. They identified these states by contrasting proficiency rates on state tests with the National Assessment of Educational Progress (NAEP) assessments to demonstrate substantial variation in the difficulty of states’ assessments. Specifically, states with high standards (HS) were defined as those with more similar pass rates on the state-created achievement tests as compared to NAEP, whereas states that have high pass rates on state-created tests relative to NAEP (and thus much higher pass rates overall than the HS states) were classified as low standards (LS) states (the 25 remaining states were classified as “medium standards,” see Table 2). These differences in proficiency standards resulted in differential AYP failure rates, sanctions and corrective actions, and ultimately school restructuring (Wong et al., 2009). Indeed, Wong et al. demonstrated that states in the HS group were more likely to institute new curriculum, solicit outside expert advice, replace school staff, and extend the school day than the LS states. Based on these observed differences, Wong et al. conducted a CITS analysis comparing high standards states (treatment group) and low standards states (comparison group). This contrast estimates the impact of the full suite of NCLB reforms rather than consequential accountability only, though it likely underestimates the true impact of NCLB given that all states did implement NCLB to some extent. As with the timing of CQA contrast, the HS-LS contrast could impact student engagement immediately, as a shift in engagement in the first post-NCLB timepoint (2004), or could reflect cumulative changes, as a differential change in the post-NCLB engagement trajectories for high versus low standards states.

OPEN IN VIEWER

Table 2

States by NCLB Implementation Standards and Timing of Consequential Accountability

State	HS	LS	NCLB CQA	Prior CQA
AK				x
AL				x
AR	x			x
AZ	x		x
CA	x			x
CO		x	x
CT		x		x
DC	x		x
DE				x
FL				x
GA		x		x
HI	x		x
IA			x
ID			x
IL				x
IN				x
KS				x
KY	x			x
LA				x
MA	x			x
MD				x
ME	x		x
MI				x
MN		x	x
MO	x		x
MS			x
MT			x
NC		x		x
ND			x
NE		x	x
NH		x	x
NJ			x
NM				x
NV				x
NY	—	—		x
OH			x
OK				x
OR				x
PA			x
RI	x			x
SC	x			x
SD			x
TN		x		x
TX		x		x
UT			x
VA		x		x
VT	—	—		x
WA	x		x
WI		x		x
WV				x
WY	x		x

Note. Data from Wong, Cook, and Steiner’s (2009) classification of states into low and high standards implementers of NCLB and Dee and Jacob’s (2011) classification of early and late adopters of consequential accountability. VT and NY not included in high and low standards analyses. HI and ND included in neither the HS-LS nor timing of CQA contrasts due to insufficient observations. NCLB = No Child Left Behind; CQA = consequential accountability; HS = high standards; LS = low standards.

Finally, Wong et al. (2009, 2015) also estimated a public versus private school contrast in which public school students were compared to private school students across all states. Because private schools were largely not subject to NCLB, this contrast is a straightforward comparison of students who did and did not experience NCLB-based changes to their schools. Like the HS-LS contrast, this comparison estimates the impact of the full scope of the NCLB legislation, and impacts could occur immediately or over time; however, estimated NCLB effects based on this contrast should be interpreted in light of the substantial heterogeneity in NCLB’s implementation (e.g., Davidson et al., 2013; Steifel et al., 2007).

Present Study

The present study is an exploratory analysis that replicates the two states-based contrasts, along with a public versus private school contrast, to estimate the association between the implementation of NCLB and students’ engagement with school. By using three contrasts—the public versus private contrast, the high versus low standards contrast (Wong et al., 2009, 2015), and the timing of consequential accountability contrast (Dee & Jacob, 2011)—the present study is able to explore the mechanisms by which NCLB impacted student outcomes. Specifically, although the public versus private and HS-LS contrasts from Wong and colleagues (2009, 2015) examine the full scope of NCLB, the Dee and Jacob (2011) CQA contrast is only able to examine the NCLB impact derived from the addition of accountability policy. By comparing these contrasts, the present study aims to identify whether NCLB as a whole impacted engagement and what role CQA specifically had in changing students’ engagement in the post-NCLB era. The present study will thus inform states as they implement ESSA and may alter their current accountability systems. Given the diverse student outcomes associated with school engagement, estimating the relationship between NCLB and engagement remains an important step in making ESSA an effective reauthorization.

Method

Sample

The sample for this study is drawn from the children of the female NLSY79 respondents and consists of youth aged 10 to 14 who completed the engagement scale between 1988 and 2010. The total unique student level N is 11,512 across states and years; each year, the student N ranges from 539 to 1,733. The sample is restricted to students who have a state identifier and valid information on public versus private school attendance and responded to at least 6 items from the engagement survey. Because some students provided more than one observation between ages 10 and 14, the total number of observations is 14,794.

Though this sample is national in scope, it is not nationally representative. The data are representative only of the children of American females aged 14 to 21 in 1979, not of American students from 1988 to 2010. There are three main implications of this sampling feature that are important for this analysis. First, while all states are represented, samples are nonrandom within states and families sort into states for unknown reasons, thus within-state samples are likely not representative of all students in a given state. Second, sample sizes are variable across states, with some states represented by very few observations and some not contributing to every timepoint. Of the 600 state-year cells, 82 are empty, and 33 have one observation and thus do not contribute to the analysis. Finally, sample sizes are time sensitive. Specifically, because all mothers of students in the sample span six years in age, there are fewer observations in the later years of the time series as mothers become older (mothers are roughly 52 years old in 2010), and sample demographics change as well (see Tables 3 and 4).

OPEN IN VIEWER

Table 3

Number of Observations by Contrast and Year, Student Engagement

	Public Versus Private		Wong, Cook, and Steiner (2015)			Dee and Jacob (2011)
Year	Treatment	Comparison	HS	MS	LS	Treatment	Comparison
1988	747	27	185	295	227	179	568
1990	1,078	45	254	445	331	250	828
1992	1,678	55	419	645	526	368	1,310
1994	1,593	98	405	608	515	360	1,233
1996	1,558	130	392	593	512	377	1,181
1998	1,426	152	365	547	445	347	1,079
2000	1,122	133	272	443	352	297	825
2002	1,263	158	295	508	391	329	934
2004	1,091	159	252	453	314	263	828
2006	873	125	203	354	268	203	670
2008	646	98	153	260	192	162	484
2010	468	71	120	174	146	111	357

Note. Data are drawn from the Maternal and Child Supplement to the National Longitudinal Survey of Youth. The Wong et al. (2015) column represents the HS-LS contrast, where HS indicates a high standards state, MS indicates medium, and LS indicates low. Two states are omitted from the HS-LS contrast because they cannot be categorized as high, medium, or low standards. The Dee and Jacob (2011) column represents the consequential accountability (CQA) contrast, where treatment states are states in which CQA begins with adoption of No Child Left Behind, and comparison states are states that had prior CQA.

OPEN IN VIEWER

Table 4

Sample Characteristics by Year

	1988	1990	1992	1994	1996	1998	2000	2002	2004	2006	2008	2010
Female	50.26	48.35	49.62	51.03	51.84	49.43	47.89	49.68	50.00	48.90	49.46	53.25
White	43.02	32.77	35.08	40.27	44.61	47.08	55.06	51.37	53.84	54.41	56.05	58.07
Black	40.18	46.84	41.55	36.78	32.41	30.74	28.29	28.64	26.16	25.35	24.46	22.82
Hispanic	16.80	20.39	23.37	22.95	22.99	22.18	16.65	19.99	20.00	20.24	19.49	19.11
Firstborn	75.45	71.06	64.92	52.99	44.79	37.45	32.35	29.98	26.64	24.35	22.98	20.03
INR	2.77	1.89	1.96	2.17	2.83	2.46	3.03	3.34	3.67	4.13	4.25	3.69
N	774	1,123	1,733	1,691	1,688	1,578	1,255	1,421	1,250	998	744	539

Note. Data are drawn from the Maternal and Child Supplement to the National Longitudinal Survey of Youth. Table shows the percent of the sample in each year that falls into female, white, black, Hispanic, and firstborn categories. INR = income-to-needs ratio.

To address these sampling issues, the present study includes a first stage in the analysis. In this first stage, I regress all observations on demographic characteristics—gender, age, race, income-to-needs ratio, and firstborn status, characteristics that have been linked to students’ engagement and may have changed as mothers aged—and use the residuals from this regression and students’ engagement scores. In this way, I remove the influence of these demographic characteristics from student scores and thus equate the states and years in these background attributes. Moreover, all regressions are weighted by the precision of the state-year or type-year observation (see Analytic Strategy) to down-weight the influence of states with a particularly small number of observations with divergent engagement scores, that is, the states in which I have less confidence about the accuracy of the estimate. Additionally, several sensitivity tests are performed. These tests vary the requirements for inclusion in the sample and assess how robust the conclusions are to weighting decisions. Specifically, the minimum number of individuals contributing to each state year observation is varied to increase the reliability of the individual state observations, raw engagement scores (as opposed to residualized) are used as the dependent variable, unweighted analyses are presented such that each state contributes equally to the estimates, students contributing duplicate observations are removed from the sample, and the time series is shortened so that it ends in 2008 (prior to the year with the smallest sample).

Analytic Strategy

As noted previously, the present study uses variability in NCLB timing and implementation to assess NCLB impacts on engagement using a comparative interrupted time series (CITS) design. In a CITS specification, regression lines are fit separately for each group (treated and untreated) by time period (before and after NCLB, i.e., 1988–2002 and 2004–2010). If the treatment and comparison time trends change in different ways after the introduction of NCLB (e.g., a differential change in either the height of the line or the slope compared to the pre-NCLB trend), this change is consistent with an NCLB-based impact on school engagement. It is important to look at the combined influence of immediate (intercept) and cumulative (slope) changes particularly because the changes of NCLB were cumulative by design: schools worked toward a long-term goal and were increasingly exposed to the potential for sanctions; this increasing probability of sanctions is both a key mechanism of the law and potentially an important driver of any NCLB-based impacts on students’ engagement. As such, NCLB could have both immediate (intercept) and cumulative (slope) effects.

Three contrasts were used to explore associations between NCLB and school engagement. The first contrast compares public schools to private schools, which largely do not receive Title 1 dollars and therefore were not subject to NCLB. In this contrast, an NCLB effect would be observed if there was a change in the post-NCLB (e.g., 2004–2010) engagement trajectory relative to that of private schools. By comparing public and private schools, the overall impact of NCLB is tested, but this contrast does not allow for state variability in NCLB’s implementation.

The second contrast replicates the high versus low standards contrast introduced by Wong et al. (2009). As discussed previously, Wong et al. capitalize on state variability in the implementation of NCLB to create high, medium, and low standards groups (see Table 2) based on differences between NAEP pass rates and state-created test pass rates. HS states had more difficult state exams and thus had on average more schools failing AYP and took more NCLB-mandated corrective actions. Conversely, LS states had easier state-created achievement tests and thus had a low proportion of schools failing AYP and took fewer corrective actions (Wong et al., 2009). In this way, schools in LS states faced lower accountability pressure than those in HS states and because of the lack of corrective actions, essentially implemented NCLB less strongly. In this contrast, an NCLB effect would be observed if the post-NCLB change in the HS states was more extreme than the change in the LS states. Because the HS states are distinguishable from the LS states in terms of both difficulty of tests and AYP schedules, both post-NCLB intercept (differential changes in the level of engagement from 2002 to 2004) and slope (differential changes in the 2004 through 2010 slopes) changes could be observed. As noted previously, this HS-LS contrast tests the full suite of NCLB-based reforms but likely understates the true impact of NCLB because the comparison group is made of states that at least partially implemented NCLB.

The third contrast is based on the timing of consequential accountability. This specification replicates the dichotomous Dee and Jacob (2011) specification, comparing states that had some form of CQA prior to NCLB with those that did not implement CQA until the passing of NCLB. In this contrast, the treated group experienced CQA for the first time with NCLB, and all others are treated as comparison observations. This contrast explores the impact of CQA as mandated by NCLB rather than the full suite of NCLB-based reforms. As noted previously, this contrast is predicated on the idea that changes after NCLB’s implementation will be larger in states that had not previously experienced CQA. Put another way, NCLB-based changes in the school context will be larger in states with no prior experiences with CQA. Thus, like the HS-LS contrast, it is likely to somewhat underestimate the impact of NCLB because states that had prior CQA should still be experiencing NCLB-based changes in their accountability systems (e.g., Fuhrman, 1999; Goertz & Duffy, 2001). These changes may be immediate (intercept, 2004) changes or cumulative (slope, 2004–2010) changes as sanctions based on accountability accumulate. Because this analysis is predicated on the idea that prior CQA policies may have already impacted students’ engagement; specifically, modeling pre-NCLB trends in engagement with respect to the timing of the introduction of CQA policy may be of interest. To explore this question, a sensitivity test is conducted in which the heterogeneity in timing of the introduction of CQA policy within the comparison group is explicitly modeled by coding the treatment indicator as the number of years a state did not have CQA (Dee & Jacob, 2011). These models more flexibly estimate the impact of each additional year of CQA as well as the NCLB effect. Findings are similar (see Sensitivity Analyses section in the following); however, because of the small number of observations available in the present data, the dichotomous indicator (treatment states are those without prior CQA, comparison are those with prior CQA) remains the preferred specification.

Models

To model the public versus private school contrast, I estimate the following:

Y jt = β 0 + β 1 ( year ) t + β 2 ( public ) j + β 3 ( post _ NCLB ) jt + β 4 ( year × public ) jt + β 5 ( post _ NCLB × year ) jt + β 6 ( post _ NCLB × public ) jt + β 7 ( year × post _ NCLB × public ) jt + ε jt

In this model Y_jt represents type-year school engagement means. β₆ represents the change in intercept after the introduction of NCLB for public as compared to private schools, and β₇ represents the differential slope change. Thus, the immediate NCLB effect is estimated by β₆+β₇, and the effect of NCLB in 2010 (the last year of data in this analysis) is estimated by β₆+ (β₇× 8).

To model the high versus low standards contrast, I estimate the following:

Y st = β 0 + β 1 ( post _ NCLB ) t + β 2 ( year × post _ NCLB ) t + β 3 ( year × high _ standards ) st + β 4 ( year × medium _ standards ) st + β 5 ( post _ NCLB × high _ standards ) st + β 6 ( post _ NCLB × medium _ standards ) st + β 7 ( year × post _ NCLB × high _ standards ) st + β 8 ( year × post _ NCLB × medium _ standards ) st + β 9 ( student _ teacher _ ratio ) st + β 10 ( state _ poverty ) st + γ t + δ s + ε st .

In this model, Y_st represents state-year school engagement means. β₅ represents the change in intercept after the introduction of NCLB for HS as opposed to LS states, and β₇ represents the differential slope change. Thus, the immediate NCLB effect is estimated by β₅+β₇, and the effect of NCLB in 2010 is estimated by β₅+ (β₇× 8). Note that this model estimates separate slopes and intercepts for the low, medium, and high standards states; high versus low standards contrasts are presented in the text, following Wong et al. (2015). This specification also includes state and year fixed effects, δ _s and γ _t , respectively.

Finally, to model the timing of consequential accountability contrast, I estimate the following:

Y st = β 0 + β 1 ( year ) t + β 2 ( post _ NCLB ) t + β 3 ( years _ since _ NCLB ) t + β 4 ( year × no _ prior _ CA ) st + β 5 ( post _ NCLB × no _ prior _ CA ) st + β 6 ( no _ prior _ CA × years _ since _ NCLB ) st + β 7 ( student _ teacher _ ratio ) st + β 8 ( state _ poverty ) st + δ s + ε st .

In this model, Y_st represents state-year school engagement means. β₅ represents the change in intercept after the introduction of NCLB for states without prior CQA as opposed to those who already had CQA, and β₆ represents the differential slope change. Thus, the immediate NCLB effect is estimated by β₅+β₆, and the effect of NCLB in 2010 is estimated by β₅+ (β₆× 8). This specification also includes state fixed effects, δ _s .

Preprocessing, Weighting, and Standard Errors

As noted previously, these analyses used data drawn from the NLSY, which is not a nationally representative sample, unlike other CITS-based research on NCLB. Preprocessing and weighting strategies were used to address this data limitation. First, to create the state-year (or type-year in the public vs. private contrast) means that served as the dependent variable, all (N = 14,794) school engagement scores were regressed on the demographic variables listed in the following, and each observation’s residual from this regression was used as their engagement score. Essentially, this process was used to take each student’s engagement score and create a new score that reflected whether the student had high, low, or average school engagement net of demographic factors. For example, if on average females scored a half-point higher on the engagement scales, then by capturing the residual from a regression of engagement on gender, the preprocessing technique would remove the half-point advantage from female scores, equating them, on average, to their male peers. The model was estimated as shown in the following:

y = α + β 1 gender + β 2 age + β 3 black + β 4 Hispanic + β 5 firstborn + β 6 INR + ε i

Standard errors were clustered by maternal ID. The mean of each group of individual student state-year (or type-year) residuals was then calculated to be used as the state-year (and type-year) observations in the final models. Results were not sensitive to what covariates are included in the first stage, and a sensitivity test in which models are run without this residualizing process was conducted.

Second, all analyses were weighted using the precision of the estimated state or type mean, defined as the inverse of the squared standard error of the standard error of the state (or type) mean. That is, the standard errors of the state-year observations are averaged by state—the estimated precision is the inverse of the squared standard error of this calculated mean. Thus, states with more precisely estimated state-year means contributed more to the estimated NCLB effects than states with more variable means (typically those states with fewer observations). In this way, the analysis down-weights states that may have artificially high or low engagement scores based on having a small number of observations or states with students with highly disparate engagement scores, in which we can have less confidence. However, as noted previously, sensitivity checks were conducted to explore whether results were sensitive to the use of these weights or a strategy that instead dropped state-year observations made up of a small number of observations. Finally, standard errors were adjusted using STATA’s CLUSTER option.

Results

Table 5 reports findings from the public versus private, high versus low standards, and timing of CQA contrasts for both the full engagement scale and the academic engagement scale. Each panel presents the differential intercept and differential slope for the contrast as well as the estimated impact of NCLB immediately (in 2004), in 2007, and in 2010. The first row, the intercept shift, presents the estimated change in school engagement for the treated group at the first timepoint after the introduction of NCLB (in 2004) relative to the comparison group (e.g., for public vs. private school students, HS vs. LS states, and states with no prior CQA vs. states that had prior CQA). Thus, it does not represent the absolute change in engagement from 2002 to 2004 but rather how different this change was for the treatment versus comparison group. The second row in each panel provides an estimate of the differential slope after the introduction of NCLB. This term represents the difference in the engagement trajectory of the treatment versus comparison group from 2004 to 2010; thus, a negative slope can be interpreted as a decreasing trajectory relative to the comparison group, and a positive slope can be interpreted as an increasing trajectory relative to the comparison group. The last three rows represent the estimated NCLB effect in 2004, 2007, and 2010 using the estimates from the CITS model, calculated as described previously.

OPEN IN VIEWER

Table 5

Associations Between NCLB Adoption and Engagement From 1988 to 2010, Full Engagement and Academic Engagement Scales

	Full Engagement Scale				Academic Engagement Scores
	b	SE	p	SD	b	SE	p	SD
Public versus private schools
Intercept shift for public versus private schools	.051	.046	.276	.121	.058	.064	.376	.105
Slope change for public versus private schools	−.002	.007	.821	−.004	−.009	.010	.405	−.016
2004 NCLB impact for public versus private schools	.048	.047	.320	.112	.041	.072	.582	.075
2007 NCLB impact for public versus private schools	.043	.057	.460	.102	.014	.092	.880	.025
2010 NCLB impact for public versus private schools	.038	.072	.608	.090	−.012	.117	.918	−.022
High versus low standards states
Intercept shift for HS versus LS states	.070	.019	.000	.167	.066	.030	.035	.120
Slope change for HS versus LS states	−.019	.006	.003	−.045	−.014	.007	.050	−.025
2004 NCLB impact for HS versus LS states	.033	.012	.009	.079	.039	.022	.079	.071
2007 NCLB impact for HS versus LS states	−.022	.020	.262	−.052	−.003	.022	.898	−.005
2010 NCLB impact for HS versus LS states	−.078	.035	.033	−.186	−.044	.036	.229	−.080
Consequential accountability in 2002 versus earlier
Intercept shift for on-time versus early CQA states	.047	.032	.147	.112	.121	.052	.024	.220
Slope change for on-time versus early CQA states	−.019	.007	.011	−.045	−.042	.010	.000	−.076
2004 NCLB impact for on-time versus early CQA adopters	.010	.024	.665	.024	.037	.047	.432	.067
2007 NCLB impact for on-time versus early CQA adopters	−.045	.024	.067	−.107	−.088	.056	.121	−.160
2010 NCLB impact for on-time versus early CQA adopters	−.101	.038	.012	−.240	−.214	.077	.008	−.389

Note. Data are drawn from the Maternal and Child Supplement to the National Longitudinal Survey of Youth. Models using high and low standards and timing of accountability include state-level student-teacher ratio and proportion of families in poverty. All models are weighted by the precision of the type-year or state-year observations used to generate each state-year (or type-year) mean. The standard deviation for the residualized student level for the full engagement measure is .42; for academic engagement scores, the standard deviation is equal to .55 across all contrasts. NCLB = No Child Left Behind; CQA = consequential accountability; HS = high standards; LS = low standards.

Panel 1 displays the estimates for the public versus private school contrasts. This model revealed a positive intercept shift for public school students relative to private school students (b = .05, p = .28) but a negative, near zero slope term (b = –.00, p = .82) for the full engagement scale. Neither of these estimates reached statistical significance, however, suggesting that across all states, NCLB was not associated with changes in engagement for public school students relative to private school students (see Figure 1). Estimates were very similar in the academic engagement model, though a slightly larger negative slope (though still nonsignificant) led to a negative estimated NCLB effect in 2010. This estimate was very small in magnitude however (d = –.02) and nonsignificant.

OPEN IN VIEWER

Figure 1.

School engagement in public and private schools from 1988 through 2010.

Panel 2 displays CITS results for the HS-LS contrast. Across both the full engagement and academic engagement scales, this panel reveals statistically significant intercept and slope terms. Specifically, for the full engagement scale, immediately following the introduction of NCLB, students in HS states reported an increase in engagement that was on average .07 points higher than students in LS states (.17 of a standard deviation, p < .01). However, the slope term was also statistically significant and negative (b = –.02, p < .01), suggesting that this initial boost decreased over time by roughly .05 of a standard deviation each year. Thus, by 2010, the NCLB effect was negative (see Figure 2) and roughly a fifth of a standard deviation in size (p < .05). For the academic engagement scale, estimates were similar in magnitude but smaller, thus the 2010 NCLB effect was about half the magnitude (d = –.080) and not conventionally significant (p = .229).

OPEN IN VIEWER

Figure 2.

School engagement in high and low standards states from 1988 through 2010.

Panel 3 displays the differential intercept and slope for states that adopted CQA for the first time with the implementation of NCLB as compared to states that adopted CQA prior to NCLB. A slightly different pattern emerged for this contrast. For the full engagement scale, this panel reveals a similar in magnitude but not statistically significant difference in post-NCLB intercepts (b = .05, p = .15) and a statistically significant negative slope term (b = –.02, p < .05) for the treatment relative to control states, equivalent to –.05 of a standard deviation each year. Thus, these models again suggest that the average level of student engagement in states that implemented CQA for the first time with the passage of NCLB decreased relative to students in states that had previously adopted CQA. By 2010, the NCLB impact on students’ engagement in states that adopted CQA as mandated by NCLB for the first time in 2002 was .24 of a standard deviation lower than the impact for students in states that had previously adopted CQA (p < .05; see Figure 3). Findings were similar for the academic engagement scale; however, in these analyses, the intercept term was substantially larger and statistically significant (d = .22, p < .05) such that the CQA findings mimicked those of the HS-LS contrast, that is, an initial boost in engagement followed by a decline that led to a statistically significant negative NCLB effect by 2010.

OPEN IN VIEWER

Figure 3.

School engagement in states with and without consequential accountability (CQA) policies prior to No Child Left Behind from 1988 through 2010.

In sum, analyses suggest that the NCLB impact on student engagement depends on which NCLB mechanism formed the basis of the contrast. While the public versus private school contrast revealed no NCLB effect, both the HS-LS and CQA contrasts suggest an immediate positive but increasingly negative NCLB effect on engagement over time, a pattern that was consistent across the full engagement and academic engagement scales.

Discussion

Prior to the introduction of No Child Left Behind, a subset of educational scholars voiced concern that consequential accountability policies would erode students’ engagement with school and thereby undermine students’ academic achievement and social skill development (e.g., Deci et al., 1991; Osterman, 2000), a concern that continued to be voiced throughout NCLB’s tenure (e.g., M. G. Jones, 2007; Kirp, 2015). However, this concern clashed with literatures associating academic press with engagement (Ma, 2003), demonstrating a positive association between NCLB and mathematics achievement (Dee & Jacob, 2011; Wong et al., 2015), and suggesting that NCLB did not detrimentally impact students’ enjoyment of school (Reback et al., 2014) or teachers’ perceptions of behavioral engagement (Dee & Jacob, 2011). The present study examined directly whether NCLB impacted students’ self-reported school engagement using a comparative interrupted time series design, the most rigorous design that has been used to assess NCLB to date, using the only national data set able to support such an analysis.

This study provides some suggestive evidence that federal education policy may be able to impact students’ engagement with school. First, across two measures of engagement, the public versus private comparison, which contrasted public school students to private school students, found no statistically significant NCLB effect. This is notable as the public versus private contrast is arguably the purest contrast; public school students were treated by NCLB, and private school students were not. However, both previous CITS studies (Dee & Jacob, 2011; Wong et al., 2009, 2015) and other theoretical papers (e.g., Davidson et al., 2013; Steifel et al., 2007) highlight the variability in NCLB’s implementation, suggesting that between-state analyses are warranted.

In both of the between-state contrasts—the high versus low standards and timing of consequential accountability models—the present study found a small, immediate increase in school engagement relative to the comparison group (though this was not conventionally significant for the CQA contrast) followed by diminished engagement in treatment relative to comparison states over time. This pattern was consistent across the full engagement and academic engagement scales. In both contrasts, by 2007, the estimated NCLB impact was negative (though nonsignificant at conventional levels for the HS-LS contrast), and by 2010, the estimated effect was negative and significant, equal to between a fifth and a quarter of a standard deviation (p < .05). The HS-LS and CQA models were robust to analyses that dropped small states, used raw as opposed to residualized dependent variables, and had similar patterns (though not significant for CQA) for the unweighted analyses. However, the HS-LS models were sensitive to both removing students who were observed more than once and dropping the 2010 wave. This last sensitivity is particularly notable given that the slope coefficient drives the negative NCLB impacts. As such, the HS-LS findings in particular should be interpreted with caution.

One way to interpret this pattern is that some of the early changes made by the states—such as the development of streamlined standards, curricula, and tests; provision support to struggling schools; and increased instructional time (U.S. Department of Education, 2007; Wong et al., 2009)—may have boosted engagement but that over time, accountability pressure—specifically the increased likelihood of falling into sanctions—may have eroded school engagement, consistent with previously conducted local studies demonstrating decreases in student engagement in response to high-stakes testing and accountability systems (e.g., M. G. Jones et al., 2003; Nichols & Berliner, 2007). Indeed, the presence of negative slope effects rather than intercept changes drives the findings of this study, and the increasingly negative association between NCLB and engagement parallels the increase in NCLB-based sanctions over time. In 2006, 29% of schools were classified as failing under NCLB, a percentage that rose to nearly 40% by 2010 (Usher, 2012). Though the present study cannot identify what NCLB mechanisms likely impacted engagement, previous research suggests that narrowed curricula, reduced instructional support and autonomy in the classroom, and increased teacher anxiety (e.g., Au, 2007; Diamond, 2007; Finnegan & Gross, 2007; Griffith & Scharmann, 2008; Hannaway & Hamilton, 2008; McMurrer, 2007; Pederson, 2007; Plank & Condliffe, 2013) may have played a role.

These findings are important to understanding the potential impact of educational policy for several reasons. First, despite the negative association between NCLB and students’ engagement, it is meaningful that these analyses demonstrated that distal educational policy—for example, state and federal— may have the power to shape students’ engagement. It is also notable that across both between-state contrasts, there was an immediate boost in student engagement. NCLB was a complex, multifaceted law that impacted students’ experiences in diverse ways. This evidence suggests that some of NCLB’s provisions did promote student engagement; however, as noted previously, it cannot identify which levers were most beneficial. For practitioners and policymakers to make the most of the flexibility ushered in with ESSA, it is critical that research identify the key policy levers for building student engagement and that changes to NCLB-era accountability systems do not discard these valuable provisions.

It is also relevant to educational policy that the introduction of consequential accountability under NCLB seemed to be more detrimental for students’ engagement than prior instantiations of consequential accountability (e.g., Table B6 in the online version of the journal). Prior to NCLB, consequential accountability systems were created at the state level and were more flexible than NCLB, designed with state budgetary constraints in mind, emphasized school improvement plans, and used incentives as well as sanctions (e.g., Fuhrman, 1999; Goertz & Duffy, 2001). These more tailored plans may have been differently associated with student engagement because they were less stressful for teachers and administrators, involved the provision of assistance from state education agencies, and may have led to less school failure than NCLB’s consequential accountability. Moreover, it is plausible that the widespread concern over the law among teachers’ unions and in the public discourse (e.g., Gerson, 2007; Kirp, 2015; Ravitch, 2010; Saad, 2012) may have heightened teacher and administrator sensitivity to accountability pressure under NCLB, particularly in states that had no previous experience with consequential accountability systems to reassure them. With the introduction of the Every Student Succeeds Act and the excitement among educators at the end of NCLB (e.g., Walker, 2015; Weingarten, 2015), states may be able to use NCLB’s systems as a base to transition to tailored, supportive consequential accountability systems and reduce some of the anxiety in schools and among educators, potentially increasing engagement while maintaining the academic emphasis introduced by NCLB.

Finally, it is important to interpret these findings in light of previous work highlighting the positive impact of NCLB on academic achievement (Dee & Jacob, 2011; Wong et al., 2009, 2015) and evidence indicating that NCLB did not have negative impacts on students’ interest in or enjoyment of school (Reback et al., 2014; Whitney & Candelaria, 2017) or teachers’ perceptions of students’ behavioral engagement (Dee & Jacob, 2011). First, these previous studies did not assess students’own report of engagement, nor did they assess the full, multidimensional engagement construct, which may account for the diverging findings.

Second, these studies used samples of varying student ages. While previous research on academic outcomes has found consistent positive impacts of NCLB on fourth-grade mathematics achievement, findings for eighth graders are much more mixed, suggesting that NCLB impacts on engagement may also vary by age. Students in both Whitney and Candelaria’s (2017) and Reback et al.’s (2014) study were in fifth grade; teachers reporting on engagement in the Dee and Jacob study were representative across all student ages. The sample in the present study included youth aged 10 to 14 (roughly fourth through eighth grade) and is thus not directly comparable to the samples used in previous research and should be compared with caution.

Third, it is possible that NCLB had varying impacts on the dimensions of engagement, which may have in turn led to differential associations with engagement-linked outcomes. For example, NCLB may have enhanced behavioral engagement but decreased emotional engagement. Indeed, many of the early NCLB-based changes—extended instructional time, tailored curricula, and specific skill-based assessments—would be consistent with climates of academic press that put a strong focus on students’ behavioral engagement and academic outcomes (Goddard et al., 2000) but not necessarily their emotional engagement. Moreover, in a recent analysis, Holbein and Ladd (2017) find that although the accountability pressure engendered by NCLB improved students’ attendance, it also increased students’ misbehavior, highlighting the potential for nuance in how students reacted to NCLB. NCLB may have made students more likely to do what they were supposed to do academically—which could improve achievement—but may have also negatively influenced how students felt or thought about school. It is possible that the narrowing of curricula and focus on test preparation brought on by NCLB could have improved achievement by creating classrooms that were focused on the skills necessary to do well on high-stakes tests but simultaneously decreased students’ enjoyment of and sense of belonging in school. Notably, the full engagement scale used in the present study drew heavily from items typically used to assess emotional engagement, and these items seem to drive the present findings (see Table B9 in the online version of the journal), suggesting that students’ liking of school and perceptions of school’s emotional support may have been more dramatically influenced by NCLB than other components of engagement. Future research should probe NCLB’s effects on outcomes most commonly linked to emotional engagement in particular, including substance use, delinquent behavior, depressive symptoms, and high school completion. Devising accountability systems that maintain a motivation for positive academic change and a focus on students’ skills while minimizing teacher anxiety and disengaging pedagogical shifts—school features that are more strongly linked to emotional engagement—should be a key goal for researchers, policymakers, and practitioners moving forward.