Concurrent and Predictive Validity of the Student Engagement Instrument

Abstract

The Student Engagement Instrument (SEI) is a self-report measure of cognitive and affective engagement with school. Prior SEI validation studies have focused primarily on construct validity through analyses of internal consistency, factor analysis, and measurement invariance. Results are presented here from a two-pronged study of the criterion validity of SEI scores. Using a middle school sample (N = 35,900), concurrent validity was assessed through analyses of group differences in SEI scores across student subgroups expected to differ in cognitive and affective engagement levels: behaviorally disengaged versus non-disengaged, high-risk versus low-risk disability status, and high versus low academic achievement. Next, through multiple logistic regression analyses, the 4-year predictive validity of SEI scores for on-time graduation and dropout was assessed in a cohort of first-time ninth graders (N = 11,588). Nearly all SEI factors demonstrated directionally consistent associations with each criterion, including considerable long-term predictive associations with both dropout and on-time graduation.

Keywords

Student Engagement Instrument student engagement graduation dropout middle school high school

Twenty-first century American youth belong to an age in which graduating from high school with college-ready skills carries a substantial social and economic significance (Kirsch, Braun, Yamamoto, & Sum, 2007; Rumberger, 2011). Although high school graduation rates overall appear to be on the rise in the United States, they remain low for some demographic groups, such as children from low-income families, students with learning disabilities, and foreign-born Latin American students (U.S. Department of Education, 2009). Dropout theory and empirical studies indicate a critical role for student engagement and disengagement, respectively, in school completion and dropout (Reschly & Christenson, 2012). In recent years, dropout prevention research has included engagement indicators in the development of early warning systems (EWS) that identify middle and high school students who are at-risk for dropout (Allensworth & Easton, 2007; Balfanz, Herzog, & Mac Iver, 2007; Henry, Knight, & Thornberry, 2012), with findings showing that students’ attendance, discipline records, and course grades are powerful indicators of whether a student is falling off the graduation path.

In the last decade, student engagement has increasingly been viewed as a construct relevant for all students (Christenson, Reschly, & Wylie, 2012) and as an organizing heuristic for school reform efforts and the Response-to-Intervention framework (Christenson et al., 2008). Although there are some conceptual variations across models of student engagement, there is consensus that it is multidimensional—involving aspects of behavior, cognition, and emotion (Fredricks, Blumenfeld, & Paris, 2004). Behavioral engagement includes indicators such as rule-following and low levels of disruptive behaviors (e.g., infrequent absences, office referrals), as well as involvement in academic tasks (e.g., persistence, class participation) and participation in extracurricular activities (e.g., school athletics, school government). Affective engagement, representing an internal aspect of student engagement, refers to students’ emotional experiences with school (e.g., interest, happiness, belonging). The third category, cognitive engagement, also internal, refers to psychological investment in learning (e.g., preference to be challenged and exceed basic academic requirements) and self-regulation (e.g., use of metacognitive strategies, managing distractions).

Student Engagement Instrument (SEI)

Many studies have investigated student engagement using behavioral indicators; few have distinctly included indicators of the cognitive and affective subtypes (Appleton, Christenson, Kim, & Reschly, 2006; Fredricks et al., 2004). The SEI, developed by Appleton et al. (2006), is a survey designed to measure self-perceptions of cognitive and affective engagement. SEI data are intended to complement behavioral and academic engagement data readily available in school records, and the SEI is available to schools free of charge. (See Appleton et al., 2006, for the complete set of SEI items.) The SEI was developed in alignment with school completion research (e.g., Finn, 1989) and the Check & Connect intervention model (see Christenson et al., 2008). In the SEI’s conceptual framework (Appleton, Christenson, & Furlong, 2008; Appleton et al., 2006), the behavioral dimension described above is further divided into behavioral and academic engagement subtypes, the former indicated by variables such as attendance rate and extracurricular activity, the latter indicated by variables such as time-on-task and homework completion. The SEI’s framework stresses the dynamic relationships between context, engagement, and outcomes (Appleton et al., 2006, 2008) and holds that engagement is both an outcome and a process that relates to other outcomes (Reschly & Christenson, 2012).

Although indicators of behavioral and academic engagement are commonly available in student records, data on cognitive and affective engagement typically are not. The SEI measures the other two broad subtypes of engagement through a five-factor measurement model (Betts, Appleton, Reschly, Christenson, & Huebner, 2010). Three of these factors tap into affective engagement: Teacher–Student Relationships (TSR), Family Support for Learning (FSL), and Peer Support for Learning (PSL). Illustrative example items for each of these include At my school, teachers care about students (TSR); When I have problems at school, my family/guardian(s) are willing to help me (FSL); and Other students at school care about me (PSL). The other two factors on the SEI—Future Goals and Aspirations (FGA) and Control and Relevance of School Work (CRSW)—relate to cognitive engagement. Representative example items for these include School is important for achieving my future goals (FGA) and The tests in my classes do a good job of measuring what I’m able to do (CRSW).

Promising evidence of the validity of SEI scores has been demonstrated in prior studies. In the pilot study of the SEI, Appleton et al. (2006) reported findings on the latent factor structure, internal consistency, and concurrent validity of the SEI, using the responses of 1,931 ninth graders. Internal consistency estimates for the SEI ranged from α = .72 to .88, and small correlations (range r = |.00| to |.25|, mean r = |.12|) were found, generally in expected directions, with several concurrent indicators associated with school completion (i.e., grade point average [GPA], standardized achievement scores, and suspensions). Following the pilot study, the SEI has been included in studies of school functioning involving students in Grades 7 through 10 (Lewis, Huebner, Reschly, & Valois, 2009; Reschly, Huebner, Appleton, & Antaramian, 2008), and in a follow-up validation study conducted by Betts et al. (2010), who found good internal consistency (α = .70-.80) and evidence of factorial invariance of the SEI’s factor structure among students in Grades 6 to 12. Although Appleton et al. (2006) found a six-factor structure to yield the best fit and interpretability in their pilot study, Betts et al. indicated that a five-factor model was more parsimonious.

Purpose and Scope of Present Study

In the pilot study, as noted above, correlations between SEI scores and concurrent indicators of student performance (e.g., GPA, suspension) were generally in expected directions but small in magnitude, raising questions regarding the practical value of the instrument. No additional studies have examined the criterion validity of SEI scores. Furthermore, no studies have examined their long-term predictive validity. With this in mind, we developed two research questions regarding the criterion validity of SEI scores:

Research Question 1: Concurrent validity. Do SEI scores differ meaningfully and consistently between groups of middle school students that are likely to differ considerably in terms of student engagement?

Research Question 2: Predictive validity. When following a cohort of high school students over time, do SEI scores in ninth grade predict dropout and on-time graduation 4 years later?

To address these research questions, we designed a two-pronged study that extends knowledge of the SEI in several ways. Addressing the issues raised by the original validation study, our study includes a larger and more diverse sample, allowing us to analyze differences among students belonging to groups at opposing extremes of engagement. Results from these analyses provide additional evidence regarding the degree, direction, and consistency of the SEI’s relationship with educational indicators (Study 1). In addition, our 4-year study of a cohort of first-time ninth graders provides the first look at the long-term predictive validity of SEI scores (Study 2). We view this study as one piece in a body of research on the validity of SEI scores and as an indication of what can be expected when the SEI is used in an applied setting. The practical nature of the questions addressed here should be valuable to researchers and stakeholders interested in evidence relevant to screening for risk of school failure.

Criterion Variables and Hypotheses

In engagement research, dropout is not viewed as an isolated event, but as the culmination of a long-term process of disengagement from school, which typically begins much earlier than the high school years (Alexander, Entwisle, & Horsey, 1997; Appleton et al., 2008; Balfanz et al., 2007). With this in mind, we were interested in the concurrent associations of SEI scores with early indicators of dropout risk, as well as the predictive association of SEI scores with distant high school outcomes.

Concurrent Criterion Variables: Early Indicators of Risk

In choosing concurrent criteria, our focus was on variables available in school records for nearly all students that would indicate early risk for school failure. This led us to include data on attendance, office discipline referrals (ODRs), special education status, and achievement. Poor attendance and disciplinary problems are considered indicators of behavioral disengagement from school (Fredricks et al., 2004) and are highly predictive of high school dropout, even as early as sixth grade (Balfanz et al., 2007). Because the SEI was developed to complement behavioral engagement data, we hypothesized that SEI scores would be considerably different between typical students and those demonstrating extreme levels of behavioral disengagement, as measured by attendance and ODR data.

Students in special education are a heterogeneous group; correspondingly, graduation and dropout rates among disability categories vary considerably (Schifter, 2011). We were interested in comparing the SEI scores of a disability category associated with high risk of dropout with a category with lower risk. According to national data provided by the U.S. Department of Education (2010), among students in special education, students identified with an emotional/behavioral disorder (EBD) had the lowest graduation rate (50.0%) and highest dropout rate (38.8%), whereas students with a speech/language impairment (SLI) had the highest graduation and lowest dropout rates (70.5% and 6.7%, respectively). We expected to find meaningful differences in SEI scores between these groups, with higher scores expected for students in the lower risk group.

Finally, evidence from a variety of studies shows that higher levels of engagement—including indicators of cognitive and affective engagement—associate with higher achievement (Fredricks et al., 2004). Based on findings by Appleton et al. (2006), we suspected that mid-range levels of engagement may not differentiate between students much in terms of achievement, but that students at opposing ends of academic achievement should differ considerably in engagement levels, with well above average achievement levels having considerably higher SEI scores than students with well below average achievement.

Predictive Criterion Variables: Distant High School Outcomes

Evidence from longitudinal studies involving behavioral, cognitive, and affective engagement indicators have shown that higher engagement is associated with a reduced likelihood of dropping out (Balfanz et al., 2007; Janosz, Archambault, Morizot, & Pagani, 2008; Wylie & Hodgen, 2012). Accordingly, we hypothesized that higher SEI factor scores in ninth grade would associate with a greater probability of graduating on-time 4 years later and with a lower probability of dropout.

General Methods

In the two studies that follow, we used the five-factor SEI model (Betts et al., 2010), measuring three facets of affective engagement—TSR, PSL, and FSL—and two facets of cognitive engagement—CRSW and FGA. Students responded to SEI items using a 4-point Likert-style response format (1 = strongly disagree, 2 = disagree, 3 = agree, 4 = strongly agree), where higher scores signify higher levels of engagement. Factor scores were computed as the mean response value across factor items. Administration of the SEI involved standardized procedures, including oral administration to control for reading level. These studies were conducted with the approval of the district and university institutional review boards.

Missing Data

Across studies, SEI items and achievement scores (in Study 1 only) were the only variables with missing data. In the SEI data, each item had at least one missing value across cases. However, no items were missing more than 1% of values, and less than 1% of all possible SEI values were missing overall. Furthermore, less than 10% of students were missing one response or more, and 96% of these students were missing only one to three items. Missingness of data was low, sample size was large, and assumptions of data being missing at random were reasonable, given the nature of the measure and the extensiveness of the data available. In each sample, SEI factor scores were generated using the mean across items if 75% or more of items loading onto the factor were complete. When SEI factor scores from the spring administration were missing, a seasonally adjusted fall score was utilized if available (equal to the fall score minus the mean difference between fall and spring, by factor and grade level). This resulted in complete SEI data for more than 93% of the sample. The remaining 6.2% to 6.3% of missing SEI factor scores across all cases and the 4.6% of missing math and reading z scores (in Study 1) were handled using multiple imputation (Little & Rubin, 2002). Five imputations were generated using all available data and through chained equations with predictive mean matching (Little, 1988). All results involving SEI or achievement data represent pooled estimates across imputations. Imputations and all analyses were performed using Stata Version 13 software (StataCorp, 2013).

Study 1: Concurrent Validity

Sample

Cross-section of middle school students

The middle school sample (N = 35,900) consisted of students in Grades 6 (33.6%), 7 (34.6%), and 8 (31.8%), who were enrolled ≥ 50% of the 2007-2008 school year in a southeastern U.S. district. All data were from existing school records, including SEI data, which were collected in fall 2007 and spring 2008 as part of a district-wide student advisement program. Sample size was comparable with state estimates for the district’s fall 2007 enrollment in Grades 6 to 8 (36,166). The sample was 48.5% female. Ethnicity was 35.1% White, 27.4% Black, 22.8% Hispanic, 10.3% Asian, 4.1% Multiracial, and less than 1% Native American. English was the primary language for 68.5% of the sample, with 18.8% speaking Spanish as a primary language, 2.5% Korean, and 1.7% Vietnamese. The remaining 8.5% comprised students speaking one of a variety of other languages (e.g., Hindi, Mandarin). Students receiving special education services constituted 13.6% of the sample, and 43.6% of the sample was eligible for free or reduced-price lunch.

Operational Definitions

Behavioral disengagement

Students were classified as behaviorally disengaged if, during the 2007-2008 school year, they were absent 10% or more of enrolled days and cited for more than one disciplinary infraction that resulted in detention or suspension. These markers were chosen based on disengagement and dropout prediction research with middle school student data conducted by Balfanz et al. (2007) and on the types of student data available for this study.

Disability risk status

Our operational definitions of disability risk status were informed by national dropout data for students with disabilities (U.S. Department of Education, 2010) and by the constraints of our data (i.e., we excluded low-incidence categories due to small sample sizes, as well as high-incidence categories with less than a 50% SEI response rate). Students identified with EBD as their primary area of exceptionality were categorized as having a higher risk disability. Students with SLI listed as their primary area of exceptionality were assigned to the lower risk disability category.

Academic achievement

Achievement was measured using student scores on state-wide standardized end-of-year math and reading tests for Grades 6 through 8. Well above average achievement was defined as scoring 1 SD or more above the district mean on each of these assessments. Well below average achievement was defined as scoring 1 SD or more below the state mean on each assessment.

Analyses

Associations with early indicators of risk

Comparing SEI factor score means across groups, Cohen’s d and effect size (ES) interpretative guidelines for educational research were used to evaluate whether a mean difference was educationally significant (d ≥ .25) or clinically significant (d ≥ .50; Wolf, 1986). For parsimony, rather than conducting t tests or other similar traditional significance tests, we assessed both statistical and practical significance based on Cohen’s d estimates and their 95% confidence intervals (CIs; Nakagawa & Cuthill, 2007).

Study 2: Predictive Validity

Sample

Cohort of first-time ninth graders

The sample for predictive validity analyses (N = 11,588) consisted of all students in the district who were enrolled ≥ 50% of the 2007-2008 school year and were entering ninth grade for the first time. This sample was 49.8% female. Ethnicity was 37.4% White, 26.5% Black, 20.4% Hispanic, 10.5% Asian, 4.6% Multiracial, and <1% Native American. English was the primary language for 72.3% of students, with 15.5% speaking Spanish and 12.2% speaking one of a variety of other languages. Students with disabilities constituted 10.9% of the sample, and 43.9% of the sample were eligible for free or reduced-price lunch.

Operational Definitions

No Child Left Behind (2002) signified a policy shift toward high school completion by requiring states to track graduation rates, defining graduation as “the percentage of students who graduate . . . with a regular diploma in the standard number of years” [see 20 U.S.C. § 6311(b)(2)(C)(vi)]. We operationalized on-time graduation and dropout in accordance with this definition. Specifically, students who dropped out were identified in school records as leaving within the time frame of the study (2007-2011) for any of the following reasons: enrollment in adult education, expelled/suspended, financial hardship, incarcerated, low grades, marriage/parenting, removed for non-attendance, or transferred to juvenile justice. Although research suggests that a large portion of students who leave a district later fail to graduate (Rumberger, 2004), in an effort to keep our outcome variables as reliable as possible, our definition of dropout does not include students who transferred to a private school or another district, left to attend home school, or left for unknown reasons. These relatively stringent criteria resulted in a dropout definition that does not represent many true but unknown non-graduates, thus likely underestimating the true cohort dropout rate. Likewise, the true cohort on-time graduation rates are likely somewhat higher than reported as non-graduates include any students in the cohort who were not identified as graduating.

Analyses

Predictive associations with graduation and dropout

Using the ninth-grade cohort, we first evaluated whether SEI scores in ninth grade affected the likelihood of on-time graduation, defined as graduating with a full diploma by the end of the summer following the cohort’s expected senior year. Next, we assessed whether scores affected the likelihood of dropout within that same time frame. A series of multiple logistic regressions were used to estimate the statistical significance and ES of the association between each SEI factor and each of these binary outcomes. Each regression included one continuous SEI factor as the independent variable and gender, ethnicity, and economic disadvantage as covariates. Coefficients of the effect of each SEI factor score on the outcome are expressed in odds ratios (ORs), where an OR = 1.0 indicates no association, an OR > 1.0 indicates an increase in the likelihood of the outcome as the factor score increases, and an OR < 1.0 indicates a decrease in the likelihood of the outcome as the factor score increases (Agresti, 2007).

Results

Study 1: SEI Associations With Early Risk Indicators

Group means and 95% CIs for SEI factor scores are presented in Table 1. Cohen’s d ESs for group differences are reported and represented graphically with 95% CIs in Figure 1. Each group mean comparison of SEI factor scores was statistically significant (p < .05 level), and all effects but one (CRSW by achievement level) were in the expected direction.

Table 1.

SEI Factor Score Means and 95% CIs by Group.

	SEI factor
	TSR	PSL	FSL	FGA	CRSW
Behaviorally disengaged
Yes (n = 1,086)	2.39	3.05	3.31	3.43	2.83
	[2.34, 2.45]	[3.01, 3.08]	[3.26, 3.35]	[3.39, 3.47]	[2.79, 2.86]
No (n = 34,814)	2.74	3.21	3.54	3.69	3.03
	[2.73, 2.75]	[3.20, 3.21]	[3.53, 3.54]	[3.68, 3.69]	[3.02, 3.03]
Disability status
EBD (n = 616)	2.59	2.93	3.39	3.53	2.96
	[2.52, 2.65]	[2.86, 2.99]	[3.33, 3.45]	[3.49, 3.58]	[2.91, 3.01]
SLI (n = 587)	2.83	3.11	3.52	3.63	3.09
	[2.78, 2.89]	[3.06, 3.16]	[3.47, 3.56]	[3.59, 3.67]	[3.05, 3.13]
No disability (n = 34,697)	2.74	3.21	3.53	3.69	3.02
No disability (n = 34,697)	[2.73, 2.74]	[3.20, 3.22]	[3.52, 3.54]	[3.68, 3.70]	[3.01, 3.02]
Math and reading achievement
Well below average (n = 1,888)	2.62	3.03	3.40	3.47	3.04
Well below average (n = 1,888)	[2.59, 2.65]	[3.00, 3.05]	[3.38, 3.43]	[3.44, 3.50]	[3.01, 3.06]
Other (n = 31,603)	2.73	3.21	3.53	3.69	3.02
	[2.72, 2.74]	[3.20, 3.22]	[3.52, 3.53]	[3.68, 3.70]	[3.02, 3.03]
Well above average (n = 2,409)	2.92	3.30	3.63	3.79	2.97
Well above average (n = 2,409)	[2.90, 2.94]	[3.28, 3.32]	[3.61, 3.65]	[3.77, 3.80]	[2.95, 2.99]

Note. 95% CIs in brackets. SEI = Student Engagement Instrument; CI = confidence interval; TSR = Teacher–Student Relationships; PSL = Peer Support for Learning; FSL = Family Support for Learning; FGA = Future Goals and Aspirations; CRSW = Control and Relevance of School Work; EBD = emotional/behavioral disorder; SLI = speech/language impairment.

Figure 1.

Effect sizes (ESs; Cohen’s d) for group mean differences on SEI factor scores.

Behavioral disengagement

Compared with students without the combination of frequent absences and multiple detentions/suspensions, d values indicated that behaviorally disengaged students reported lower TSR and FGA at a level of clinical significance. Furthermore, all three other factor comparison differences were educationally significant. All associations were in the expected direction.

Disability status

Compared with peers identified with a lower risk disability (SLI), students with a higher risk disability (EBD) reported lower TSR, PSL, and CRSW to a degree that was educationally significant. Although differences were in the expected direction and statistically significant for the FGA and FSL factors, ESs for these mean differences did not meet criteria for educational significance.

Math and reading achievement

Overall, the strongest student engagement association was found with comparisons between high versus low achievement. At a level of clinical significance, students who scored well above the district mean in math and reading reported higher PSL and FGA than students who scored well below the state mean. Differences in SEI factor scores on TSR and FSL approached clinical significance. Contrary to our hypothesis, CRSW was rated higher in the low achievement group. This relationship was statistically significant, but did not reach a threshold of educational significance.

Study 2: Predictive Associations With Graduation and Dropout

As of August 2011, 65.5% of the 2007-2008 ninth-grade cohort were on-time graduates (n = 7,594), 2.5% had earned a certificate of completion (n = 291), 5.8% were re-enrolled for year 5 (n = 666), 10.8% had transferred out of the district (n = 1,253), 10.4% had left for unknown reasons (n = 1,204), and 4.9% were identified officially as dropouts (n = 566). Results of multiple logistic regression analyses are presented in Table 2. In regressions with on-time graduation as the dependent variable (1 = yes, 0 = no), the OR for the effect of every SEI factor was statistically significant at the p < .001 level. The same was found for regressions with dropout as the dependent variable. FGA demonstrated the strongest association for each outcome. Specifically, holding gender, ethnicity, and economic disadvantage constant, a one-unit increase in FGA score was associated with at least a twofold increase in the odds of earning a diploma in 4 years, as well as a twofold decrease in the odds of dropping out. Similar to findings from concurrent validity analyses with the middle school cohort, CRSW exhibited the weakest associations overall with educational outcomes.

Table 2.

Multiple Logistic Regressions of Each SEI Factor on Graduation and Dropout.

		SEI factor
Outcome		TSR	PSL	FSL	FGA	CRSW
On-time graduation	OR	1.54* (.07)	1.45* (.06)	1.62* (.07)	2.38* (.11)	1.24* (.05)
n = 7,594 of 11,588	95% CI	[1.41, 1.67]	[1.33, 1.57]	[1.48, 1.78]	[2.17, 2.61]	[1.15, 1.36]
Dropout	OR	0.55* (.05)	0.67* (.07)	0.54* (.04)	0.45* (.04)	0.59* (.06)
n = 566 of 11,588	95% CI	[0.45, 0.67]	[0.54, 0.82]	[0.46, 0.63]	[0.37, 0.55]	[0.47, 0.73]

Note. Standard error for the OR in parentheses. Regressions controlled for gender, ethnicity, and free/reduced-price lunch eligibility. SEI = Student Engagement Instrument; OR = odds ratio; CI = confidence interval; TSR = Teacher–Student Relationships; PSL = Peer Support for Learning; FSL = Family Support for Learning; FGA = Future Goals and Aspirations; CRSW = Control and Relevance of School Work.

OR is statistically different from 1.0, p < .001.

Discussion

Our analyses of group differences in SEI factor scores showed statistically significant associations in all investigated group comparisons, with all but one association in the hypothesized direction. Although the ESs for most of these differences reached basic thresholds for practical significance, most did not reach levels of clinical significance. This pattern is consistent with the small correlations reported by Appleton et al. (2006) for SEI factor associations with disciplinary problems and academic achievement. Unlike Appleton et al.’s findings, however, some group differences were moderate in ES. Namely, self-ratings on TSR and FGA were substantially lower among students exhibiting high absence rates and signs of behavior problems. And scores on PSL and FGA were substantially higher for high-achieving than for low-achieving students. It is important to note that, in some cases, these stronger ESs are an artifact of asking the association question differently. For instance, investigating whether high achievers’ SEI scores are higher than those of low achievers ignores students who fall somewhere in between. In other words, although group differences were moderate in ES between high and low achievers, comparisons between the high or low group and the average would be smaller.

Another result consistent with previous research was the counterintuitive negative association between CRSW and academic achievement. Although the ES for this relationship was not of practical significance, it was statistically significant and is consistent with findings by Appleton et al. (2006) of small negative correlations with standardized reading and math scores (r = −.29 and −.26, respectively). To the best of our knowledge, this is the only other study of the SEI examining this association, and it is presently unclear why students who excel academically on standardized tests would be more likely to rate themselves lower on CRSW. One possibility is the influence of context. For example, in an environment that is not challenging enough, high-achieving students may be less likely to perceive that tests do a good job of measuring their competence. Another possible explanation may be psychometric. Specifically, Betts et al. (2010) found generally lower item factor loadings on CRSW than on other factors, as well as one cross-loading of a CRSW item, which may suggest construct validity issues that need to be addressed. Taken together, these results highlight a need for further examination of CRSW items and their relationship with achievement.

Our study of predictive validity demonstrated that—even when controlling for important demographic characteristics—SEI self-ratings in ninth grade predicted dropout and on-time graduation 4 years later with considerable consistency. Coefficients for the effect of SEI factor scores on each outcome were statistically significant in all comparisons, and many ORs indicated small to moderate ESs on the change in the likelihood of the outcome as SEI factor scores increased. These findings suggested clear, meaningful, and predictable associations between varying engagement levels and highly relevant educational outcomes—further support of the criterion validity of SEI scores. FGA stood out as the factor with the strongest associations to both dropout and graduation. This may not be surprising, as items on FGA relate specifically to plans to continue education following high school. Nonetheless, it is promising that student self-ratings in ninth grade on items about future educational goals provide predictive information that may help educators identify and intervene earlier with students whose expectations of post-secondary enrollment are fading.

Limitations, Implications, and Future Directions

The retrospective nature of the methods employed in this study precludes statements of causal relationships. Furthermore, the base rate in the population for some indicators (e.g., dropout) was somewhat low. To address these limitations, we encourage replication in other districts, particularly in settings with substantially higher risk for school failure.

Overall, the findings of this study suggest that greater confidence may be placed in the interpretation of SEI scores for students in the process of disengaging from school. Although many previously reported correlations between SEI scores and indicators of behavioral engagement/disengagement and achievement were small to non-significant, the results of this study show that—when following students over several years and when investigating groups representing extremes of the engagement continuum—there are consistent and educationally meaningful associations between SEI scores and relevant outcomes.

Most researchers and educational stakeholders agree that early screening for risk is a necessary component of systematic, evidence-based prevention programs (Christenson & Thurlow, 2004). Student engagement has been consistently identified as an important variable in school completion research (Reschly & Christenson, 2012), and although engagement is regarded as a multifaceted construct (Fredricks et al., 2004), researchers and practitioners tend to rely primarily on behavioral indicators. Cognitive and affective engagement data require additional effort to collect, but our findings suggest that it may be quite valuable. This research adds to a growing body of evidence that the SEI yields useful data regarding cognitive and affective engagement. Future research should explore how the SEI can be used in conjunction with other indicators in multifactor risk identification efforts.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

References

Agresti

(2007). An introduction to categorical data analysis (2nd ed.). Hoboken, NJ: John Wiley. doi:10.1002/0470114754

Alexander

K. L.

Entwisle

D. R.

Horsey

C. S.

(1997). From first grade forward: Early foundations of high school dropout. Sociology of Education, 70, 87-107. doi:10.2307/2673158

Allensworth

E. M.

Easton

J. Q.

(2007). What matters for staying on-track and graduating in Chicago public schools: A close look at course grades, failures, and attendance in the freshman year. Chicago, IL: Consortium on Chicago School Research at the University of Chicago.

Appleton

J. J.

Christenson

S. L.

Furlong

M. J.

(2008). Student engagement with school: Critical conceptual and methodological issues of the construct. Psychology in the Schools, 45, 369-386. doi:10.1002/pits.20303

Appleton

J. J.

Christenson

S. L.

Kim

Reschly

A. L.

(2006). Measuring cognitive and psychological engagement: Validation of the Student Engagement Instrument. Journal of School Psychology, 44, 427-445. doi:10.1016/j.jsp.2006.04.002

Balfanz

Herzog

Mac Iver

D. J.

(2007). Preventing student disengagement and keeping students on the graduation path in urban middle-grades schools: Early identification and effective interventions. Educational Psychologist, 42, 223-235. doi:10.1080/00461520701621079

Betts

J. E.

Appleton

J. J.

Reschly

A. L.

Christenson

S. L.

Huebner

E. S.

(2010). A study of the factorial invariance of the Student Engagement Instrument (SEI): Results from middle and high school students. School Psychology Quarterly, 25, 84-93. doi:10.1037/a0020259

Christenson

S. L.

Reschly

A. L.

Appleton

J. J.

Berman

Spanjers

Varro

(2008). Best practices in fostering student engagement. In Thomas

Grimes

(Eds.), Best practices in school psychology (5th ed., pp. 1099-1199). Bethesda, MD: National Association of School Psychologists.

Christenson

S. L.

Reschly

A. L.

Wylie

(2012). Preface. In Christenson

S. L.

Reschly

A. L.

Wylie

(Eds.), Handbook of research on student engagement (pp. v-ix). New York, NY: Springer. doi:10.1007/978-1-4614-2018-7

10.

Christenson

S. L.

Thurlow

M. L.

(2004). School dropouts: Prevention, considerations, interventions, and challenges. Current Directions in Psychological Science, 13, 36-39. doi:10.1111/j.0963-7214.2004.01301010.x

11.

Finn

J. D.

(1989). Withdrawing from school. Review of Educational Research, 59, 117-142. doi:10.3102/00346543059002117

12.

Fredricks

J. A.

Blumenfeld

P. C.

Paris

A. H.

(2004). School engagement: Potential of the concept, state of the evidence. Review of Educational Research, 74, 59-109. doi:10.3102/00346543074001059

13.

Henry

K. L.

Knight

K. E.

Thornberry

T. P.

(2012). School disengagement as a predictor of dropout, delinquency, and problem substance use during adolescence and early adulthood. Journal of Youth and Adolescence, 41, 156-166. doi:10.1007/s10964-011-9665-3

14.

Janosz

Archambault

Morizot

Pagani

L. S.

(2008). School engagement trajectories and their differential predictive relations to dropout. Journal of Social Issues, 64, 21-40.doi:10.1111/j.1540-4560.2008.00546.x

15.

Kirsch

Braun

Yamamoto

Sum

(2007). America’s perfect storm: Three forces changing our nation’s future. Princeton, NJ: Educational Testing Service.

16.

Lewis

A. D.

Huebner

E. S.

Reschly

A. L.

Valois

R. F.

(2009). The incremental validity of positive emotions in predicting school functioning. Journal of Psychoeducational Assessment, 27, 397-408. doi:10.1177/0734282908330571

17.

Little

R. J. A.

(1988). Missing-data adjustments in large surveys. Journal of Business and Economic Statistics, 6, 287-296. doi:10.1080/07350015.1988.10509663

18.

Little

R. J. A.

Rubin

D. B.

(2002). Statistical analysis with missing data. Hoboken, NJ: Wiley.

19.

Nakagawa

Cuthill

I. C.

(2007). Effect size, confidence interval and statistical significance: A practical guide for biologists. Biological Reviews, 82, 591-605. doi:10.1111/j.1469-185X.2007.00027.x

20.

No Child Left Behind Act of 2001, Pub. L. No. 107-110, § 115 Stat. 1425 (2002).

21.

Reschly

A. L.

Christenson

S. L.

(2012). Jingle, jangle, and conceptual haziness: Evolution and future directions of the engagement construct. In Christenson

S. L.

Reschly

A. L.

Wylie

(Eds.), Handbook of research on student engagement (pp. 3-19.) New York, NY: Springer. doi:10.1007/978-1-4614-2018-7_1

22.

Reschly

A. L.

Huebner

E. S.

Appleton

J. J.

Antaramian

(2008). Engagement as flourishing: The contribution of positive emotions and coping to adolescents’ engagement at school and with learning. Psychology in the Schools, 45, 419-431. doi:10.1002/pits.20306

23.

Rumberger

R. W.

(2004). Why students drop out of school. In Orfield

(Ed.), Dropouts in America: Confronting the graduation rate crisis (pp. 131-155). Cambridge, MA: Harvard Education Press.

24.

Rumberger

R. W.

(2011). Dropping out: Why students drop out of high school and what can be done about it. Cambridge, MA: Harvard University Press. doi:10.4159/harvard.9780674063167

25.

Schifter

(2011). High school graduation of students with disabilities: How long does it take? Exceptional Children, 77, 409-422.

26.

StataCorp. (2013). Stata statistical software: Release 13. College Station, TX: Author.

27.

U.S. Department of Education. (2009). The condition of education 2009 (NCES Publication No. 2009-081). Washington, DC: Author.

28.

U.S. Department of Education. (2010). Data tables for OSEP state reported data: Part B exiting (2010-11). Retrieved from http://tadnet.public.tadnet.org/pages/712

29.

Wolf

F. M.

(1986). Meta-analysis: Quantitative methods for research synthesis. Newbury Park, CA: Sage.

30.

Wylie

Hodgen

(2012). Trajectories and patterns of student engagement: Evidence from a longitudinal study. In Christenson

S. L.

Reschly

A. L.

Wylie

(Eds.), Handbook of research on student engagement (pp. 585-599). New York, NY: Springer. doi:10.1007/978-1-4614-2018-7_28