A Bifactor Model of School Engagement

Abstract

School engagement involves cognitive, emotional, and behavioral components that overlap conceptually. This conceptual ambiguity has led to measures that have either consisted of one general factor or separate correlated factors. However, neither approach can sufficiently account for both the uniqueness and the overlap of the subcomponents. The bifactor model has been recommended to determine the degree to which a measure is unidimensional versus multidimensional. In this study, we examined the validity of a multidimensional measure of school engagement in adolescence, the Behavioral-Emotional-Cognitive School Engagement Scale (BEC-SES; Li & Lerner, 2013), by comparing the model fit and predictive power of the widely-used one- and three-factor models with a bifactor model. Using data from 561 youth in Iceland (46% girls, M_age at Wave 1 = 14.3 years, SD = 0.3), only the multidimensional models (i.e., the three-factor and bifactor models) gave a good fit to the data. We then assessed the predictive power of the multidimensional models for academic achievement. The addition of academic achievement as an outcome variable to the bifactor model revealed that general school engagement, as well as specific behavioral engagement, predicted achievement. These findings are distinct from previous results using three-factor models, which indicated that behavioral engagement alone predicted later achievement. The results of the current study support the use of a bifactor model when using measures of school engagement.

Keywords

academic achievement adolescence bifactor models school engagement

School engagement has been identified by researchers, teachers, and policy makers as an important asset for promoting school success and for addressing academic problems, such as school dropout and poor academic achievement (Appleton, Christenson, & Furlong, 2008; Fredricks, Blumenfeld, & Paris, 2004). School engagement¹ has been defined as ‘[the] student’s active participation in academic and co-curricular or school related activities, and commitment to educational goals and learning.… It is a multidimensional construct that consists of behavioral (including academic), cognitive, and affective subtypes’ (Christenson, Reschly, & Wylie, 2012, pp. 816–817).

The three aspects of school engagement, that is, students’ cognitions, feelings, and behaviors, have been shown to be important for school success. Behavioral engagement reflects a student’s willingness to participate in school related activities, such as attending classes. Emotional engagement includes a student’s feelings about his or her school, such as his or her sense of belonging to school. Cognitive engagement describes a student’s willingness to invest in cognitive abilities that relate to learning, such as self-regulated learning (Christenson et al., 2012; Li & Lerner, 2013).

The high correlations frequently observed among behavioral, emotional, and cognitive engagement (see e.g. Li & Lerner, 2013) has raised questions about the multidimensionality of school engagement. Acknowledging the extent to which a measure is multidimensional is important, as secondary dimensions can be lost when inappropriate models, such as one-factor models, or models that do not acknowledge the common variance of the factors, are fit to multidimensional data (Ackerman, 1992; Reise, 2012). The current study examined the validity of the school engagement construct by comparing the model fit of three measurement models of school engagement (a one-factor, a three-factor and a bifactor model of school engagement) and by comparing the criterion validity of the good-fitting models by testing how strongly each predicts academic achievement.

The role of school engagement in academic success

In one of the earliest reviews of the concept of school engagement, Mosher and McGowan (1985) explained that physical presence in schools can be legislated (i.e., by making school attendance mandatory), whereas school engagement cannot. A student’s presence at school is necessary for, but does not guarantee, school engagement and the successful completion of compulsory school (Reschly & Christenson, 2012). For instance, Finn’s (1989) Participation-Identification Model highlighted that successful school completion results from a gradual process of school engagement that involves a student identifying school-related goals and actively participating in school-related activities. As such, school engagement develops and entails bidirectional relations between students and their school contexts (Finn, 1989).

A relational developmental systems (RDS) perspective underlines this bidirectional nature of school engagement (Overton, 2015). In one of the most influential RDS models of Positive Youth Development (PYD), Lerner and colleagues (see e.g. Lerner, Phelps, Forman, & Bowers, 2009; Lerner, Lerner, & Benson, 2011) have identified school engagement as one of the key strengths of adolescents that, when aligned with ecological assets, promotes positive youth development. Similarly, influential researchers of motivation and school success, such as Eccles (2004), emphasize that the context, that is, schools, need to align better with the developmental needs of their students to support person-context fit and ensure that all students are motivated and engaged in their education.

There is considerable empirical support for the importance of school engagement for later school success (Reschly & Christenson, 2012). For example, a recent study found that 60% of high school dropouts could be identified based on their sixth-grade school engagement alone (i.e., attendance, misbehavior, and course failures; Balfanz, Herzog, & Mac Iver, 2007). Accordingly, one of the more important impacts of research on school engagement has been a shift away from focusing on individual characteristics, such as IQ, for promoting school success, and focus on understanding person-context fit in educational practices (Appleton et al., 2008; Fredricks et al., 2004).

The three-dimensional nature of school engagement

As already explained, the strong correlations among the subdimensions of school engagement have raised questions about the uniqueness of each subdimension (Fredricks et al., 2004). An example of the strong correlation among the subdimensions is a recent study on the factorial invariance of the Student Engagement Instrument (SEI) where subfactors of cognitive and emotional engagement were very strongly correlated among adolescences in Grades 6 through 12 (r = .79; Reschly, Betts, & Appleton, 2014).

The conceptual definition of school engagement has raised questions about the homogeneity of each subdimension (Betts, 2012). The heterogeneity of the subdimensions is reflected in their overlap with several other processes that have been identified by psychological and educational research, such as goal setting, self-regulated learning, social development, internal motivation, and reward contingencies (Betts, 2012). The overlap between individual subdimensions of school engagement and other processes has, for example, been observed in the strong correlation (r = .51 –.72) between measures of cognitive engagement and measures of motivational engagement among adolescences in Grades 9 through 12 (Betts, Appleton, Reschly, Christenson, & Huebner, 2010).

The wide range of concepts that relate to each subdimension have made it difficult to define three separate measures of engagement that include all the relevant aspects of each subdimension. Researchers have therefore frequently identified specific aspects of each subdimension and then used these attributes as indicators of general school engagement. This practice assumes that school engagement represents a unified, yet multidimensional, construct (Betts, 2012). As such, the bifactor model has been recommended as a potentially useful approach to partitioning the item variance into separate general and specific factors that can then be evaluated to better understand the structure of school engagement (Betts, 2012; Reise, 2012).

The bifactor measurement model

The bifactor measurement model, first described by Holzinger and Swineford (1937), has recently been rediscovered as an important approach to representing multidimensional measures in factor analysis and structural equation modeling (Reise, 2012; von Eye, Martel, Lerner, Lerner, & Bowers, 2011). The use of bifactor models allows researchers to examine a single common factor that represents a multidimensional construct, while also acknowledging the uniqueness of the individual dimensions that comprise it. More specifically, the bifactor model specifies that the covariance among a set of items can be accounted for by two processes; a single general factor that reflects the common variance among all the items, and a set of specific factors that reflect additional covariation among subsets of items. All factors are typically assumed to be orthogonal (i.e., uncorrelated), meaning that items representing different dimensions are hypothesized to correlate only because of their shared variance with the general factor (Betts, 2012; Reise, 2012; Reise, Morizot, & Hays, 2007).

To our knowledge, no study has yet measured school engagement as a single common construct while also recognizing its tripartite nature by using a bifactor model. The scarcity of such research was confirmed by searching for the words ‘bifactor/bi-factor’ and ‘school/student engagement’ in titles, keywords, and abstracts on the Web of Science™. This search revealed only three journal articles and one book chapter containing both words, none of which assessed all three dimensions of school engagement as defined by Christenson et al. (2012).

The current study

The current study is a four-wave longitudinal study that took place at the beginning and end of Grade 9 (Waves 1 and 2, respectively), and the beginning and end of Grade 10 (Waves 3 and 4, respectively) in Iceland. In the study, we examined the validity of a school engagement measure at Wave 1 in two ways, first, by comparing the fit of three models of school engagement and, second, by comparing the predictive validity of good-fitting models at Wave 1 for predicting scores on a standardized achievement test (the Icelandic National Examinations; INE) at Wave 3. Finally, we confirmed the reliability of the best fitting model by examining the longitudinal factorial invariance of the best fitting measure across all four waves.

More specifically, using the Behavioral-Emotional-Cognitive School Engagement Scale (BEC-SES; Li & Lerner, 2013), we tested three rival measurement models: a single-factor model of general school engagement; a three-factor model of behavioral, emotional, and cognitive school engagement; and bifactor model with school engagement as a general factor and three specific behavioral, emotional, and cognitive engagement factors. Because previous research has found that the subdimensions of school engagement are highly correlated (r > .50; Li & Lerner, 2013), we followed the recommendation of Reise et al. (2007) and hypothesized that the measure of school engagement reflected two distinct sets of processes. First, we argued that students would exhibit systematic differences in how they would respond to all items, indicating a global school engagement factor. In addition, we hypothesized that each facet of school engagement (i.e., behavioral, emotional, and cognitive) would display systematic between-person variation that would be independent of the global school engagement factor. We therefore hypothesized that a bifactor model would fit our data better than either competing model.

After testing the rival measurement models, INE scores were added as an outcome variable to models that displayed good model fit. Based on previous research on school engagement and academic achievement (Chase, Hilliard, Geldhof, Warren, & Lerner, 2014; Christenson et al., 2012), we hypothesized that a three-factor model of school engagement would indicate a strong positive relationship between behavioral engagement and academic achievement, and a weak relationship between academic achievement and emotional and cognitive engagement. Based on this same research, we hypothesized that a general factor of a bifactor model of school engagement would positively predict later academic achievement. Due to the scarcity of research, the analyses on the relation of specific school engagement factors and academic achievement in the bifactor model were purely exploratory. Furthermore, based on previous research on the distortion that may occur when fitting inappropriate models to multidimensional data (Ackerman, 1992; Reise, 2012), we hypothesized that a bifactor model of school engagement would fit the data significantly better than a three-factor model, which would in turn fit the data significantly better than a unidimensional model. After selecting the best-fitting model, we tested factorial invariance of the best-fitting model across the four waves. We hypothesized that a configural, weak, and strong factorial invariance could be established across the four waves of measurement.

Method

The current study is part of a four-wave longitudinal investigation of adolescent development in Iceland conducted at the beginning of Grade 9 and lasting through the end of Grade 10 in the Icelandic compulsory school system. Academic achievement data was collected concurrently with the third wave of measurement and merged with the overall dataset.

Participants

We randomly selected 20 out of the 54 medium- to large-sized schools (>20 Grade 9 students) located in the Reykjavík capital area and the adjacent Reykjanes peninsula. Out of the 20 selected schools, 15 agreed to participate. Each participating school received a book as a gift for their school library for their participation. In order to increase the number of schools that participated, and thereby ensuring more diverse responses at the school level, two classrooms in each school were selected at random in schools that had more than two classrooms. These 30 classrooms had a total of 625 students. A total of 561 parents (90%) gave written consent for their child’s participation, and 539 (96%) of youth with parental consent participated at Wave 1 (mean age 14.3, SD = 0.3, 46% girls). The population in Reykjavík and the Reykjanes area includes 66.4% of all Icelandic children and is socially heterogeneous (Table 1).

Procedure

Participants completed a paper-and-pencil survey during a 40-minute school visit by trained research staff. Standardized instructions were used to ensure that data collection was uniformly administered. Students who were absent during the school visit were contacted by e-mail, mail, or phone, and asked to complete and return the survey by mail.

Measures

We describe our measures below. For each measure, the model-based reliability estimate coefficient ω (McDonald, 1999) was calculated to indicate the proportion of the scale variance that was due to all common factors (Zinbarg, Revelle, Yovel, & Li, 2005). Coefficient ω is analogous to coefficient α (Reise, 2012); therefore reliability estimates above the .70 level were interpreted as indicators of adequate reliability (Kline, 2011).

Behavioral-Emotional-Cognitive School Engagement Scale (BEC-SES)

To measure school engagement, we used the Behavioral-Emotional-Cognitive School Engagement Scale (BEC-SES) developed by Li and Lerner (2013). The BEC-SES consists of the three subscales of school engagement: behavioral, emotional, and cognitive. Each subscale was measured using five items administered using a four-point Likert scale (answer options differed across scales, see below). Abbreviated item content can be seen in Table 2.

Table 1.

Sample descriptive statistics at Wave 1.

Females n (%)	249 (46.20)
Age mean (SD)	14.28 (0.28)
Mothers with only compulsory education n (%)	77 (14.81)
Foreign language spoken at home* n (%)	31 (5.82)
Fewer than 25 books at home n (%)	65 (12.22)
Behavioral-Emotional-Cognitive School Engagement Scale (1–4) mean (SD)	3.32 (0.45)
Behavioral engagement (1–4) mean (SD)	3.49 (0.45)
Emotional engagement (1–4) mean (SD)	3.04 (0.56)
Cognitive engagement (1–4) mean (SD)	3.41 (0.56)

Note. Total number of participants = 561. *In Iceland, a foreign language spoken at home frequently serves as an indicator of a household minority status.

Table 2.

The Behavioral-Emotional-Cognitive School Engagement Scale. Abbreviated item content and frequencies at Wave 1.

No.	S	Abbreviated item content	C1%	C2%	C3%	C4%\
1	Behavioral engagement	Come to class unprepared*	3.2%	22.4%	47.4%	26.9%
2	Behavioral engagement	Complete homework on time	0.9%	7.4%	16.2%	75.6%
3	Behavioral engagement	Skip classes without permission*	0.2%	2.9%	10.1%	86.9%
4	Behavioral engagement	Take part in group (class) discussions	2.7%	12.6%	33.0%	51.7%
5	Behavioral engagement	Work hard to do well in school	0.9%	5.0%	26.2%	67.9%
6	Emotional engagement	Feel part of my school	6.6%	10.4%	51.3%	31.6%
7	Emotional engagement	Care about the school	7.0%	14.2%	46.3%	32.5%
8	Emotional engagement	Happy to be at my school	3.8%	10.1%	37.7%	48.5%
9	Emotional engagement	Don’t find school fun and exciting*	9.7%	21.0%	41.7%	27.6%
10	Emotional engagement	Enjoy the classes I am taking	5.2%	15.8%	58.7%	20.3%
11	Cognitive engagement	Learn as much as I can at school	2.7%	7.5%	41.1%	48.7%
12	Cognitive engagement	Is important to make good grades	2.0%	3.6%	20.5%	74.0%
13	Cognitive engagement	The things I learn at school are useful	2.5%	11.3%	46.1%	40.0%
14	Cognitive engagement	Think a lot about how to do well in school	3.8%	17.1%	42.0%	37.2%
15	Cognitive engagement	School is very important for later success	1.4%	2.7%	22.3%	73.6%

Note. These are average results over 20 data sets. *Reverse-worded item.

A three-factor model of BEC-SES has shown evidence of strong measurement equivalence between boys and girls, between youth of different socioeconomic status, and across youth in US Grades 9 through 11 (Li & Lerner, 2012). The measure was translated into Icelandic by two independent translators. The translations were reconciled by researchers fluent in both languages and pretested with 77 Grade-9 students from a single school. Coefficient ω for the whole BEC-SES in the current sample was .95.

Behavioral engagement

Behavioral engagement included five items whose content ranged from shallow engagement (e.g., class attendance) to deep engagement (e.g., effort). The subscale focuses on students’ voluntary behaviors within the school context to minimize possible confounding effects of non-student related variables (an example of academic behaviors outside the school context that can be confounded by non-student variables, is participation in private tutoring, which may be related to social economic status). For each item, respondents were asked to rate how often they engaged in specific behaviors using a scale from 1 (never) to 4 (always). Coefficient ω for behavioral engagement in the current sample was .82.

Emotional engagement

The emotional engagement subscale included five items that assessed students’ sense of belonging and their affect toward school. Happiness, excitement, and enjoyment were used to measure three related, yet distinct, types of positive affect. Items used to tap school connectedness assessed different aspects of the emotional relationships students had with their school and classes. The respondents were asked indicate their agreement to five emotional statements on a scale from 1 (strongly disagree) to 4 (strongly agree). Coefficient ω for emotional engagement in the current sample was .87.

Cognitive engagement

Cognitive engagement was measured by five items designed to assess the extent to which students valued education and things learned at school, as well as their thoughts about learning. More specifically, goal orientation, identification with school, and perceptions of the link between students’ lives and school were included as core indicators of cognitive engagement. The respondents were asked indicate their rate of agreement to five cognitive statements on a scale from 1 (strongly disagree) to 4 (strongly agree). Coefficient ω for cognitive engagement in the current sample was .90.

Academic achievement

Icelandic National Examinations (INE) scores were retrieved from the Icelandic Educational Testing Institute (IETI; 2014). The IETI administers an annual INE in language skills (Icelandic), mathematics and English at the beginning of Grade 10. The exam is multidimensional and includes subcomponents that measure, for example, algebra, geometry, grammar, and spelling. The standardized scores range from 0 to 60 with a mean of 30 and a standard deviation of 10. The single academic achievement factor was fit to the three observed test scores using the direct maximum likelihood estimator. A unidimensional model was saturated, χ2 (0) = 0, p < .001; CFI = 1.00; RMSEA = .00, 90% CI (.00, .00). The academic achievement factor was clearly manifested in the total test scores of Icelandic, mathematics, and English with standardized factor loadings of 0.93, 0.83, and 0.72, respectively. Coefficient ω for academic achievement in the current sample was .88.

Data analysis

A series of factor analyses and structural equation models was estimated using version 7.3 of the Mplus software package (Muthén & Muthén, 1998–2012). The estimates of latent factors were scaled using the fixed factor method (see Little, 2013), setting the variance of each latent factor to unity. For the BEC-SES, a bifactor model was defined where each specific factor was indicated by the items suggested by the previously established three-factor model (see Li & Lerner, 2012). In addition, we defined a global school engagement factor that was indicated by all the items across the three specific factors. No cross-loadings or item-correlations were allowed. In addition, for identification purposes of the bifactor model, the correlations between all latent factors (general and specific) were set to zero within and across measurements.

Model fit was estimated by evaluating several fit indices: the chi-square statistic for the WLSMV estimation method, the root mean square error of approximation (RMSEA), and the comparative fit index (CFI). Smaller chi-square and RMSEA values (RMSEA ≤ .06), and higher CFI values (≥ .95) indicate a good model fit (West, Taylor, & Wu, 2013). Differences in model fit were confirmed with a chi-square difference tests using the DIFFTEST option in Mplus. The chi-square difference tests were further supplemented by comparing Bayesian information criterion (BIC) values, where a smaller value indicates a better fit.

Due to the low number of response options for our Likert-type data, we treated all indicators as categorical and estimated all models using robust weighted least squares (WLSMV). During the four waves of measurement, 91%, 86%, 90%, and 87% of the participants had complete data on all the school engagement items, respectively. After the last wave, 68% of the participants had complete data across the four waves of measurement. We considered the missing data to be missing at random (MAR). Correlational analysis revealed significant correlations between several variables, such as self-reported grades, mother’s education, father’s education, and mother’s occupation and missing cases at later waves. These background variables were used to inform the creation of 20 imputed datasets without missing values using the multiple imputation feature of Mplus.

The results in the manuscript are, as noted, the pooled results from 20 imputations with one exception, this exception is the difference testing of nested models at Wave 1 using the DIFFTEST feature of Mplus. The DIFFTEST feature is currently not available for the analysis of imputed data. Given the small amount of missingness during Wave 1 (9%), and given that comparative analysis using imputed and non-imputed data showed very similar results, we based the nested model comparisons on non-imputed data. Because this approach uses a pairwise present approach to address missingness, we conditioned all items on the same covariates as used to inform the imputed datasets.

We calculated the intraclass correlation (ICC) for all the items used in the analysis for both class and school level in a series of two-level unconditional models. All ICC values were lower than .10, which has been considered a minimum to produce appreciable bias in standard errors if multilevel statistical techniques are not used (Kline, 2011). To minimize the risk of making a Type 1 error, we ran all the CFA and SEM models twice producing correct standard errors using a sandwich estimator, first based on the class level variation, and again based on the school level variation. The CFA models showed no appreciable bias in standard errors under either condition. The SEM models, however, showed an appreciable bias in standard errors for the regression coefficients when clustering on school level was not taken into account. We therefore decided, as we are only interested in the individual level, to use the COMPLEX feature of Mplus and produce correct standard errors using a sandwich estimator based on the school level clustering.

We established configural, weak, and strong longitudinal factorial invariance for the bifactor model using a method for models with ordered-categorical data described by Millsap and Yun-Tein (2004). Residual variances of same indicators at all waves were allowed to correlate. Furthermore, we allowed cross-time stability correlations among same factors, but no correlations were allowed with other factors, within or across measurement occasions (see Little, 2013). We evaluated the invariance constraints using a guideline made by Chung and Rensvold (2002), where a change of more than .01 in the comparative fit index (CFI) indicates that the assumption of invariance does not hold.

Results

Abbreviated item content and frequencies for the 15-item Behavioral-Emotional-Cognitive School Engagement Scale are summarized in Table 2. Items were not markedly skewed with the exception of Item 3, in response to which 87% of the participants said they had never skipped class without permission. The item was retained, as the WLSMV method has generally performed well with skewed ordered categorical variables when sample sizes are not small (about N = 200; Kline, 2011).

Confirmatory factor analyses of the BEC-SES

The WLSMV estimation method was used to fit three measurement models to the data: a one-factor model, a three-factor model, and a bifactor model. Model identification was established by fixing the variance of each latent variable to unity. Model fits are summarized in Table 3. The one-factor model exhibited inadequate fit, χ²(90) = 552.37; CFI = .90; RMSEA = .10, because of large chi-square and RMSEA values and a low CFI value. The three-factor model showed an acceptable fit, χ²(87) = 227.68; CFI = .97; RMSEA = .05, with a significant reduction in the chi-square value compared to the nested one-factor model. In addition, the model showed an acceptable RMSEA value and a good CFI value. The bifactor model, however, provided the best fit of the three models, with the lowest chi-square value and good RMSEA and CFI values, χ²(75) = 149.89; CFI = .98; RMSEA = .04. A chi-square difference test using the DIFFTEST option in Mplus confirmed that the three-factor model fit the data better than the nested one-factor model, Δχ²(3) = 211.65, p < .001, and that the bifactor model fit the data better than the three-factor model, Δχ²(12) = 87.84, p < .001. The DIFFTEST results were further supplemented by estimating the Bayesian information criterion (BIC). The BIC for the bifactor model was substantially lower than the BIC values for the two other models (see Table 3), indicating a better fit for the bifactor model (see Raferty, 1995).

Table 3.

CFA fit statistics for the Behavioral-Emotional-Cognitive School Engagement Scale measurement models at Wave 1.

	χ²	s		Δχ²	Δdf	CFI	RMSEA	BIC
One-factor model	552.373	15.52	90			.90	.10	16324.44
Three-factor model	227.675	8.17	87	324.70	3	.97	.05	15808.16
Bifactor model	149.885	5.38	75	77.79	12	.98	.04	15772.19

Note. These are average results over 20 data sets. The BIC was retrieved with a separate CFA using maximum likelihood estimation; Chi-square difference tests between nested models at Wave 1 were conducted with non-imputed data.

The standardized factor loadings of the different models can be seen in Table 4. The one-factor model was well defined and highly reliable (ω = .93), with factor loadings ranging from .42 (participation in classroom discussions) to .79 (caring about the school). The three-factor model was well defined and reliable (behavioral engagement, ω = .82; emotional engagement, ω = .87; and cognitive engagement, ω = .90), with factor loadings ranging from 0.49 to 0.88 for the same items as the minimum and maximum factor loadings in the one-factor solution. The general school engagement scale in the bifactor model was also well defined and highly reliable (ω = .93), with factor loadings ranging from 0.40 (come to class unprepared) to 0.76 (learn as much as I can at school). The difference in maximum and minimum factor loadings between models indicates that general school engagement has a qualitatively different meaning when each specific factor (i.e., behavioral, emotional, and cognitive) has been separated from the general school engagement factor. Although also highly reliable (behavioral engagement, ω = .84; emotional engagement, ω = .87, and cognitive engagement, ω = .91), the three specific factors were less well defined than the general factor. All the specific factor loadings were significant at the p < .01 level, although one behavior engagement factor item, which refers to participation in class discussions (Item 4), showed a particularly low loading (0.11).²

Table 4.

Standardized factor loadings of the three measurement models for the Behavioral-Emotional-Cognitive School Engagement Scale at Wave 1.

Item	One-factor	Three-factor			Bifactor
Item	School engagement scale	Behavioral engagement	Emotional engagement	Cognitive engagement	School engagement scale (general)	Behavioral engagement (specific)	Emotional engagement (specific)	Cognitive engagement (specific)
1	0.48**	0.56**			0.40**	0.82**
2	0.70**	0.80**			0.67**	0.46**
3	0.67**	0.74**			0.64**	0.30**
4	0.42**	0.49**			0.44**	0.11**
5	0.70**	0.85**			0.74**	0.20**
6	0.74**		0.81**		0.64**		0.45**
7	0.79**		0.88**		0.64**		0.67**
8	0.75**		0.83**		0.62**		0.56**
9	0.55**		0.62**		0.47**		0.41**
10	0.58**		0.66**		0.56**		0.29**
11	0.75**			0.81**	0.76**			0.21**
12	0.75**			0.85**	0.74**			0.39**
13	0.73**			0.76**	0.60**			0.51**
14	0.72**			0.77**	0.64**			0.47**
15	0.74**			0.80**	0.60**			0.67**

Note. These are average results over 20 data sets.

**p < 0.01.

All of the subfactors in the three-factor model correlated strongly with each other, with latent correlation coefficients ranging from r = .65 between emotional and cognitive engagement to r = .72 between cognitive and behavioral engagement. The remaining correlation between emotional and behavioral engagement was r = .66.

Criterion validity: Latent regression analyses of school engagement and academic achievement

INE scores at Wave 3 were added—as a continuous outcome variable—to the three-factor and the bifactor measurement models from Wave 1 to assess the relative performance of the different measurement models in predicting academic achievement (see Figure 1). The one-factor model was not included due to the poor model fit established in the CFA.

Figure 1.

Empirical results of a structural equation model where a bifactor model of school engagement at the beginning of Grade 9 (Wave 1) predicts Icelandic national examination scores at the beginning of Grade 10 (Wave 3). Total number of participants = 561. The variances of the latent factors were set to unity to allow for identification. For clarity, only significant (p < .01) factor loadings and regression coefficients are shown in the diagram. Non-significant regression coefficients and fit indices can be found in Table 5.

The WLSMV estimation method was again used to fit these structural equation models to the data, and model identification was again enabled by setting the variance of each latent variable to unity. The fit indices of the models and latent regressions are shown in Table 5. The three-factor and the bifactor models showed a good fit; the bifactor model at Wave 1 fit the data significantly better than the three-factor model, according to a chi-square difference test using the DIFFTEST option in Mplus, Δχ²(13) = 61.25, p < .001.

Table 5.

SEM fit statistics and standardized regression coefficients of Behavioral-Emotional-Cognitive School Engagement Scale models at Wave 1 predicting Icelandic National Examinations scores (Icelandic, mathematics, and English combined) at Wave 3.

	χ²	s	df	CFI	RMSEA	R²	β
Three-factor model	263.462	7.09	129	.97	.04	.36
	Behavioral engagement						0.73**
	Emotional engagement						−0.09
	Cognitive engagement						−0.12
Bifactor model	204.369	5.21	116	.98	.04	.36
	General school engagement						0.51**
	Specific behavioral engagement						0.25**
	Specific emotional engagement						−.12
	Specific cognitive engagement						−.13

Note. These are average results over 20 data sets; Chi-square difference tests between nested models at Wave 1 were conducted with non-imputed data.

**p < 0.01; *p < .05.

In the three-factor model, only the behavioral engagement factor strongly predicted subsequent INE scores, β = 0.73, 95% CI (0.49, 0.98). In contrast, the bifactor model at Wave 1 produced two separate direct effects. The general school engagement factor produced a strong direct effect, β = 0.51, 95% CI (0.37, 0.65) and, in addition, the specific behavioral engagement factor produced a moderate direct effect, β = 0.25, 95% CI (0.06, 0.44) on the INE scores. The specific emotional and specific cognitive factors had weak and non-significant effects. The bifactor and three-factor school engagement models at Wave 1 both explained 36% of the variance of the INE scores.

Factorial invariance of the bifactor model of BEC-SES

Finally, in order to ensure that the structure of school engagement does not substantially vary over time, our last analytic step was to test factorial invariance of the bifactor solution, we examined the consistency of measurement of the bifactor model by establishing configural, weak, and strong factorial invariance across the four waves of available data. Scale identification was obtained by using guidelines described by Millsap and Yun-Tein (2004), the results can be seen in Table 6. The configural invariance model showed excellent fit with an average CFI of .978 and a standard deviation of only .001 across the 20 datasets. The weak invariance model was specified by fixing the individual factor loadings to be equal across the four waves. This specification caused a very small improvement in model fit, increasing the CFI by .001 while the standard deviation of the CFI remained small (.001). The strong invariance model was further specified by fixing individual thresholds to be equal across the four waves. The strong invariance model gave the same CFI and standard deviation as the weak invariance model. Differences in CFI between invariance models were well below the .01 criterion chosen for the comparison, which supported configural, weak, and strong factorial invariance across the four waves.

Table 6.

Model fit statistics for the tests of measurement invariance of general and specific aspects of behavioral, emotional and cognitive engagement across four waves.

	χ²	s	df	Δχ²	Δdf	CFI	s	RMSEA	s
Configural model	1843.028	9.43	1536			.978	.001	.019	.000
Weak invariance	1909.856	11.25	1614	66.83	78	.979	.001	.018	.000
Strong invariance	1987.091	11.49	1692	77.24	78	.979	.001	.018	.000

Note. These are average results over 20 data sets.

Discussion

This study contributes to the growing school engagement literature in three ways as will be explained in the following sections. To summarize, first, the study confirms the tripartite nature of school engagement. Second, the significantly better fit of the bifactor model suggests that, rather than being strictly unidimensional or adhering strictly to a tripartite structure, the construct of school engagement may be conceptualized as having multiple dimensions that share substantial overlap. The bifactor model confirmed that a reliable general school engagement factor underlies all of the school engagement items of the BEC-SES, regardless of their behavioral, emotional, and cognitive origin. Third, the bifactor model showed that the school engagement items gave rise to three specific factors.

Furthermore, the diverse factor loadings of the specific factors indicated that behavioral, emotional, and cognitive engagement, above and beyond general school engagement, was poorly defined in the BEC-SES. The poor definition of the specific factors can be used to inform further development in the measurement of school engagement. The specific emotional factor can, for example, be refined by separating the factor into two specific emotional engagement factors, one that it sensitive to the emotional engagement in the school in general, and another that is sensitive to emotional engagement in classes specifically.

Comparing rival models of school engagement

The three-factor and bifactor models gave adequate and good fit to the data, respectively. The one-factor model, however, fit the data poorly. The poor fit of the one-factor model provides a reason not to encourage the use of a one-factor model with the BEC-SES items. The three-factor and bifactor models both accounted for the multidimensional nature of school engagement, giving both models a much better fit than the one-factor model. However, the high correlation among the subscales in the three-factor model became problematic when the three-factor model was used to predict INE scores, as the majority of the effect was due to a general school engagement factor, which was not modeled in the three-factor model. Because of the lack of the general school engagement dimension in the three-factor model, we do not recommend it as a means to represent school engagement. The bifactor model, on the other hand, took the multidimensional nature of school engagement into account by modeling a general factor and specific subfactors, in essence combining the strengths of the one- and three-factor models.

In addition, the bifactor model provided valuable information about the meaning of the school engagement construct. The general factor had adequate loadings from all items, suggesting that all of the items were adequate manifestations of general school engagement. When looking at each of the subcomponents, the specific behavioral factor loadings were very heterogeneous. The specific behavior engagement factor was defined by strong factor loadings for Item 1 (coming to class unprepared) and Item 2 (completing homework on time). These strong factor loadings indicated that the specific behavioral factor modeled in this study was mainly a measure of academic behavior. In contrast, for example, Item 5 (work hard to do well in school) had a very weak factor loading for the specific behavioral factor but a very strong factor loading for the general school engagement factor. The low factor loading indicates that working hard to do well at school is a poor manifestation of the current definition of specific behavioral engagement but, instead, represents general school engagement very well.

The specific emotional engagement factor also had heterogeneous factor loadings, with the strongest factor loading for an item that assessed how much students cared about their schools. The item with the weakest factor loading for the specific emotional factor (Item 10) differed conceptually from the other emotional items. Item 10 referred to the classes the student was taking and did not refer to the school in general as the other items. This finding suggests that emotional engagement in classes may be a different specific dimension than more general emotional engagement in the school and, as such, general school engagement may be better represented by separating the specific emotional engagement dimension by different contexts of the school environment.

The specific cognitive engagement factor also had heterogeneous factor loadings, with the strongest factor loadings for items that index the importance of school for later success (Item 15) and the usefulness of things learned in school (Item 13). This finding indicates that the specific cognitive engagement modeled in this study was mainly a measure of the practical importance of school. The weakest factor loading for the specific cognitive engagement factor was associated with Item 11 (learn as much as I can). Item 11 differs from the other cognitive items as it closely relates to internal motivation. The low factor loading indicates that internal motivation is a poor manifestation of the current definition of specific cognitive engagement, but that it is a strong indicator of global engagement.

Taken together, the factor loadings of the specific factors of the bifactor model demonstrate that if the specific factors are to be used as valid measures to inform practice, each of the three specific factors should be further separated into more fine-grained and better defined specific factors. In contrast, the factor loadings of the general factor show that the general factor is strongly manifested in all the proposed items and fits the data very well when the specific factors have been parsed out.

Predicting academic achievement with different models of school engagement

As hypothesized, the three-factor and the bifactor models of school engagement at Wave 1 both positively predicted academic achievement at Wave 3. The three-factor model strongly predicted academic achievement, but only through the behavioral factor. The bifactor model strongly predicted academic achievement through the general school engagement factor. In addition, the bifactor model predicted academic achievement through a specific behavioral factor, but only to a moderate degree over and above the general school engagement factor. As the specific behavioral factor was, by definition, not related to general school engagement, the measure represents students who chose to engage, or not to engage, academically in school-related activities regardless of the student’s level of general school engagement. A student with high specific behavioral engagement could therefore attend school and do his or her homework without necessarily caring about school or finding school important. The results indicated that such behaviors positively predicted academic achievement to a moderate degree. However, the results also indicated that general school engagement, when behavioral, emotional, and cognitive aspects are included, is a separate factor that strongly predicted academic achievement.

The findings from the latent regression analyses suggest that interventions aimed at improving academic achievement may have a considerable effect if they focus on all aspects (i.e., behavioral, emotional, and cognitive) of school engagement. This is in sharp contrast to previous findings that indicate that, of the three school engagement dimensions, only behavioral engagement predicts academic achievement (see Chase et al., 2014). In addition, interventions that include an additional emphasis on aspects of specific behavioral engagement (i.e., feeling prepared for class, finishing homework, attending class) are likely to be associated with additional improvement in academic achievement. Further research is needed to develop and confirm the existence of the specific behavioral engagement factor and its implications for academic achievement.

Limitations, strengths and implications for future research

Some issues should be considered when interpreting the results of this study. A key limitation is that the major source of information was self-report. This form of data collection may bias the results, as the observed correlations between the different items may be due to common method variance rather than representing actual relations among underlying latent constructs. An exception to the self-report data-collection, and a strength of the study, was the inclusion of INE scores, which provided a more valid and normally distributed measure of academic achievement than the frequently used self-reported grades. In general, research would benefit from a cross-validation of the BEC-SES obtained from additional sources, such as from parent and teacher reports or through classroom observations.

We wish to emphasize, when discussing the findings of this study, that the predictive effects do not imply causation. However, the predictive effects in this study highlighted the consequences of mis-specifying multidimensional models with highly correlated subfactors. Future research into the factors associated with, and development of school engagement should consider multiple covariates and methods to better understand individual and group changes during adolescence. Another limitation to this study is that it was conducted with a limited age range in a homogeneous cultural area. This sampling restricts the generalizability of the research results to students of different ages and the results may not represent the findings based on youth from other cultures or subgroups within the Icelandic population. The research results would benefit from a cross-cultural and cross-group validation of future studies.

In general however, the results confirm that school engagement ‘consists of behavioral (including academic), cognitive, and affective subtypes’ as defined by Christenson and colleagues (2012, pp. 816–817), without any one subtype outweighing the other two. The significantly better fit of the bifactor model suggests that, rather than being unidimensional or multidimensional, school engagement is characterized by both a single and multiple dimensions. Furthermore, the results showed that important secondary dimensions can be lost when using nested models, such as the three-factor model. Our results suggest that a bifactor model is the best way to represent a comprehensive measure of school engagement.

Footnotes

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: University of Iceland Research Fund and University of Iceland Doctoral Fund.

Notes

References

Ackerman

T. A.

(1992). A didactic explanation of item bias, item impact, and item validity from a multidimensional perspective. Journal of Educational Measurement, 29(1), 67–91. doi:10.1111/j.1745-3984.1992.tb00368.x

Appleton

J. J.

Christenson

S. L.

Furlong

M. J.

(2008). Student engagement with school: Critical conceptual and methodological issues of the construct. Psychology in the Schools, 45(5), 369–386. doi:10.1002/Pits.20303

Balfanz

Herzog

Mac Iver

D. J.

(2007). Preventing student disengagement and keeping students on the graduation path in urban middle-grades schools: Early identification and effective interventions. Educational Psychologist, 42(4), 223–235.

Betts

(2012). Issues and methods in the measurement of student engagement: Advancing the construct through statistical modeling. In Christenson

S. L.

Reschly

L. A.

Wylie

(Eds.), Handbook of research on student engagement (pp. 783–805). New York, NY: Springer.

Betts

J. E.

Appleton

J. J.

Reschly

A. L.

Christenson

S. L.

Huebner

E. S.

(2010). A study of the factorial invariance of the Student Engagement Instrument (SEI): Results from middle and high school students. School Psychology Quarterly, 25(2), 84–93. doi:10.1037/A0020259

Chase

P. A.

Hilliard

L. J.

Geldhof

G. J.

Warren

D. J. A.

Lerner

R. M.

(2014). Academic achievement in the high school years: The changing role of school engagement. Journal of Youth and Adolescence, 43(6), 884–896. doi:10.1007/s10964-013-0085-4

Cheung

G. W.

Rensvold

R. B.

(2002). Evaluating goodness-of-fit indexes for testing measurement invariance. Structural Equation Modeling, 9(2), 233–255. doi:10.1207/S15328007sem0902_5

Christenson

S. L.

Reschly

L. A.

Wylie

(Eds.). (2012). Handbook of research on student engagement. New York, NY: Springer.

Eccles

J. S.

(2004). Schools, academic motivation, and stage-environment fit. In Lerner

R. M.

Steinberg

(Eds.), Handbook of Adolescent Psychology (2nd ed.). Hoboken, NJ: Wiley.

10.

Finn

J. D.

(1989). Withdrawing from school. Review of Educational Research, 59(2), 117–142. doi:10.3102/00346543059002117

11.

Fredricks

J. A.

Blumenfeld

P. C.

Paris

A. H.

(2004). School engagement: Potential of the concept, state of the evidence. Review of Educational Research, 74(1), 59–109.

12.

Holzinger

K. J.

Swineford

(1937). The bi-factor method. Psychmetrika, 2(1), 41–54.

13.

Icelandic Educational Testing Institute. (2014). The homepage of the Icelandic Educational Testing Institute. Retrieved from http://www.namsmat.is

14.

Kline

R. B.

(2011). Principles and practice of structural equation modeling (3rd ed.). New York, NY: Guilford Press.

15.

Lerner

J. V.

Phelps

Forman

Bowers

E. P.

(2009). Positive youth development. In Lerner

R. M.

Steinberg

L. D.

(Eds.), Handbook of adolescent psychology (3rd ed., Vol. 1, pp. 524–558). Hoboken, NJ: Wiley.

16.

Lerner

R. M.

Lerner

J. V.

Benson

J. B.

(2011). Positive youth development: Research and applications for promoting thriving in adolescence. Advances in Child Development and Behavior, 41, 1–17.

17.

Y. B.

Lerner

R. M.

(2012). Testing across-group and longitudinal measurement equivalence of a tripartite measure of adolescent school engagement. Washington, DC: American Institutes for Research.

18.

Y. B.

Lerner

R. M.

(2013). Interrelations of behavioral, emotional, and cognitive school engagement in high school students. Journal of Youth and Adolescence, 42(1), 20–32. doi:10.1007/s10964-012-9857-5

19.

Libbey

H. P.

(2004). Measuring student relationships to school: Attachment, bonding, connectedness, and engagement. Journal of School Health, 74(7), 274–283. doi:10.1111/j.1746-1561.2004.tb08284.x

20.

Little

T. D.

(2013). Longitudinal structural equation modeling. New York: Guilford Press.

21.

Mcdonald

R. P.

(1999). Test theory: A unified treatment. NJ: Lawrence Erlbaum Associates.

22.

Millsap

R. E.

Yun-Tein

(2004). Assessing factorial invariance in ordered-categorical measures. Multivariate Behavioral Research, 39(3), 479–515. doi:10.1207/S15327906mbr3903_4

23.

Mosher

McGowan . (1985). Assessing student engagement in secondary schools: Alternative conceptions, strategies of assessing, and instruments. University of Wisconsin: Research and Development Center.

24.

Muthén

L. K.

Muthén

B. O.

(1998–2012). Mplus User's Guide (7th ed.). Los Angeles, CA: Muthén & Muthén.

25.

Overton

W. F.

(2015). Processes, relations, and relational-developmental-systems. In Lerner

R. M.

Overton

W. F.

Molenaar

P. C. M.

(Eds.), Handbook of child psychology and developmental science (7th ed., Vol. 1, pp. 9–63). Hoboken, NJ: Wiley.

26.

Raftery

A. E.

(1995). Bayesian model selection in social research. Sociological Methodology, 25, 111–163. doi:10.2307/271063

27.

Reise

S. P.

(2012). The rediscovery of bifactor measurement models. Multivariate Behavioral Research, 47(5), 667–696. doi:10.1080/00273171.2012.715555

28.

Reise

S. P.

Morizot

Hays

R. D.

(2007). The role of the bifactor model in resolving dimensionality issues in health outcomes measures. Quality of Life Research, 16, 19–31. doi:10.1007/s11136-007-9183-7

29.

Reschly

A. L.

Betts

Appleton

J. J.

(2014). An examination of the validity of two measures of student engagement. International Journal of School & Educational Psychology, 2(2), 106–114. doi:10.1080/21683603.2013.876950

30.

Reschly

A. L.

Christenson

S. L.

(2012). Jingle, jangle, and conceptual haziness: Evolution and future directions of the engagement construct. Handbook of Research on Student Engagement, 3–19. New York, NY: Springer.

31.

von Eye

Martel

M. M.

Lerner

R. M.

Lerner

J. V.

Bowers

E. P.

(2011). Interpreting theory and method in the study of positive youth development: The sample case of gender-specificity and longitudinal stability of the dimensions of intentional self-regulation (selection, optimization, and compensation). In Lerner

R. M.

Lerner

J. V.

Benson

J. B.

(Eds.), Positive youth development. Advances in child development and behavior (Vol. 41, pp. 347–372). London, UK: Elsevier.

32.

West

Taylor

(2013). Model fit and model selection in structural equation modeling. In Hoyle

R. H.

(Ed.), Handbook of structural equation modeling (pp. 209–231). NY: Guilford.

33.

Zinbarg

R. E.

Revelle

Yovel

(2005). Cronbach’s alpha, Revelle’s beta, and McDonald’s (omega H): Their relations with each other and two alternative conceptualizations of reliability. Psychometrika, 70(1), 123–133. doi:10.1007/s11336-003-0974-7