Cross-Cultural Validation of the Student Engagement Instrument for Chilean Students

Abstract

Student engagement is associated with various aspects of students’ school experiences, including student achievement, high school completion, and post-secondary success. As measurement of student engagement has grown in countries around the world, few studies have been conducted in South America. This study examined a translated version of the Student Engagement Instrument, widely used in the U.S. and other countries, in a study of 2337 adolescents in Chile. Consistent with prior research, confirmatory factor analyses revealed a six-factor solution as the best fit for the data. However, fewer items were retained than on the studies of the SEI with students in the U.S. The Future Goals and Aspirations and Extrinsic motivation subscales were associated, in expected directions, with achievement 1 year later.

Keywords

adolescents student engagement Chile cross-cultural

Student engagement is a multifaceted construct comprised of students’ emotion, cognition, and behavior (Fredricks et al., 2004). These dimensions are commonly operationalized in terms of indicators such as self-regulation (cognitive), belonging and relationships with teachers or peers (emotion), and school-related misbehavior or participation (behavior). Although the origins of student engagement are drawn from the literature on high school dropout, engagement is considered relevant for all students from kindergarten through college. Indicators of students’ engagement are associated with both short- (e.g., achievement, Lei et al., 2018) and long-term outcomes (e.g., high school completion, Lovelace et al., 2014; college attendance, Fraysier et al., 2020). Further, much of the interest in student engagement may be attributed to its amenability to intervention (Reschly & Christenson, 2022).

Although international interest in student engagement continues to grow, there are questions regarding the universality of the construct. Such questions include the similarities and differences in youth development, measurement, and intervention design relative to students’ engagement across cultures (Reschly & Christenson, 2022). One large study of students from 12 countries (Canada, China, Cyprus, Estonia, Greece, Malta, Portugal, Romania, South Korea, United Kingdom, and United States) with students in grades 7–9 found evidence of universality in that girls were more engaged than boys, there were declines in engagement across grades, and that contextual influences (i.e., instructional practices, teacher, and parent support) were associated with students’ engagement. Conversely, means were significantly different across countries and parent influence was found to have a stronger association with students’ engagement in countries that were higher on collectivism (Lam et al., 2016). Notably, few, if any studies of students’ engagement have been conducted in South America.

The purpose of this study was to examine the Student Engagement Instrument (SEI; Appleton et al., 2006) with adolescents in Chile. The SEI is well-suited to cross-cultural examination. It was developed based on a conceptual model of student engagement that grew out of work with the school completion program, Check & Connect, around the time in which interest in student engagement was growing rapidly (e.g., National Research Council, 2004). In addition, the paper-and-pencil SEI was made freely available. Permissions to use the SEI have been granted to educators or scholars from 70 countries, all 50 U.S. States, and several U.S. territories (E. Klem, personal communication, June 24, 2019). Based on research with the SEI conducted in other countries (e.g., Moriera et al., 2009; Virtanen et al., 2016, 2018), we hypothesized that we would find six dimensions (3 subscales each for cognitive and affective engagement). Moreover, we expected that each SEI dimension would predict higher levels of achievement and attendance 1 year later, consistent with previous studies, after controlling for age and gender.

Method

Participants

Data were collected in the school population in the Valparaíso, Bío-Bío, and Metropolitana regions of Chile. A total of 2337 student between 10 and 19 years old participated (M = 15.6, SD = 1.0), of which 55.0% were female, and 55.1% attended 9th grade, and 44.9% attend 10th grade from public schools.

Procedure

Bilingual mental health professionals from the Ministry of Education translated the SEI into Spanish. The translation was validated with four Spanish native speakers and professional translators (see supplemental materials). The SEI was administered during the regular school day in the second semester (October and November) of 2016 by professionals who implemented a government-sponsored dropout prevention program (Life Skills). All students present that day complete the whole survey.

Our research team obtained formal authorization of the schools and the passive consent of parents or guardians. Additionally, the students received a letter of assent, where they were given the option to participate and assured that their abstention would not have negative consequences. No abstentions were reported during this process.

Measures

Student Engagement Instrument

The SEI was originally published with 35 self-report items on a Likert scale (1 = Strongly disagree; 4 = Strongly Agree; Subsequent studies have used a 5-point Likert type scale with a neutral mid-point). The SEI was based on a framework of student engagement that grew out of work with the Check & Connect school completion program (Christenson & Pohl, 2020). Check & Connect is one of the few school completion programs recognized by IES as having a promising effect on students staying in school (Reschly, 2020). The program draws broadly from literature in resilience, motivation, systems theory, mentoring, and dropout. The developers noted that some indicators of students’ engagement at school and with learning were readily available to educators and intervention staff (e.g., attendance, homework completion rates, participation in class and school) whereas others, particularly those that represented students’ thoughts and feelings about school and their relationships, required students’ self-report. The SEI was designed specifically to measure these higher inference forms of students’ engagement (i.e., cognitive and affective engagement).

The SEI was piloted with a large, diverse group of 9th grade students in the U.S. Exploratory and confirmatory factor analyses (CFA) supported the use of either a 5- or 6- factor model (Appleton et al., 2006). When used as a six-factor model, there are 3 subscales on each type. The affective engagement dimensions are: The affective dimensions are: (1) Family Support for Learning (FSL), (2) Peer Support for Learning (PSL), and (3) Teacher Student Relationships (TSR). The cognitive dimensions are: (1) Future Goals and Aspirations (FGA), (2) Control and Relevance of Schoolwork (CRSW), and (3) Extrinsic Motivation (EM). The sixth factor, EM, is frequently dropped from research studies conducted with the SEI due to a low number of items (n = 2) and that both were reverse coded whereas many schools that use the SEI elect to keep the 6th factor due to its perceived importance in their setting (O’Donnell & Reschly, 2020). Most studies examine scores for each subscale or dimension (e.g., Fraysier et al., 2020; Lovelace et al., 2014); however, a total score may also be used.

Several studies in the U.S. have confirmed the factor structure of the SEI and provide evidence of measurement invariance across gender and grades 6–12 and longitudinal predictive validity with high school graduation and college attendance (see O’Donnell & Reschly, 2020 for a review). Internal consistency reliability of each of the dimensions or subscales is adequate, with estimates ranging from .72 (Extrinsic Motivation to .88 (Teacher–Student Relationships; Appleton et al., 2006).

There are three published studies of the SEI in languages and countries outside of the U.S. In Portugal, Moreira et al., (2009) found support for the six-factor solution; however, measures of internal consistency were adequate but lower than those typically found in the U.S. and some items did not load as expected. Finnish researchers found evidence confirming the five-factor structure of the SEI and associations, as expected, with measures of achievement, self-esteem, and burnout (Virtanen et al., 2016). Another study examined a brief version of the SEI with students from Finland, Denmark, and Portugal, finding evidence of measurement invariance across country, gender, and different levels of academic performance (Virtanen et al., 2018).

Achievement

Student academic records were used as an indicator of achievement over time, consistent with other studies (e.g., Reschly, 2020). In Chile, students’ academic records have a range of 1–7 points, with higher scores indicating higher achievement. For the purpose of the study, we used the final average score in 2017.

Demographic Variables

Gender was coded as a dummy variable (1 = female), included as a control variable because females are more likely to show positive signs of engagement and less likely to be disengaged (e.g., Lawson & Masyn, 2015). We include age because student engagement declines as students’ move from middle to high school (e.g., NRC, 2004), which was entered as a continuous variable.

Data Analysis

The factor structure was examined via CFA with Mplus 8.4. We analyzed the goodness of fit of two alternative factor structures: a) six correlated factors and b) six first-order factors and one second-order factor. Student Engagement Instrumentation used a four-point ordinal scale and therefore the multivariate normal distribution assumption was not met. Thus, we obtained the matrix of polychoric correlations and estimated the parameters of the model using the Mean and Variance Adjusted Weighted Least Squares (WLSMV) method. The above procedure generated robust information regarding the statistics and adjustment of the analyzed model, as well as adequately estimating the parameters and their standard errors (Finney & DiStefano, 2013; Forero et al., 2009; Lei & Wu, 2012). The quality of the models was examined using: a) χ², b) CFI, c) TLI, and d) RMSEA and its 90% confidence interval. The criteria for good model fit included a non-significant χ² statistic, values of 0.95 or higher for CFI and TLI, and a value less than 0.08 for RMSEA (Hu & Bentler, 1999; West et al., 2012).

Finally, we proceeded to analyze the predictive ability of the SEI on a measure of academic achievement 1 year later. This analysis was carried out using an SEM model. In this model, academic achievement was the dependent variable, factor scores were specified as predictors, while age and sex were included as control variables.

Results

The quality of the adjustment of the both CFA models were satisfactory (Table 1). CFI, TLI, and RMSEA met or exceeded criteria for fit. However, based on modification indices, the second order factor model was allowed to include a correlation among two first-order factors: F1 and F5 (r = −.78). Standardized factor loadings for this final model may be found in Tables 2 and 3; these values vary from 0.53 to 0.85 for the first-order factors and 0.28 to 0.91 for the second-order factor. Internal consistency reliability based on parcels of items was alpha = .89 for the whole scale.

Table 1.

Goodness-of-fit Indices for the CFA-WLSMV Factor Models of the SEI Scale.

Models	χ² (df)	p	CFI	TLI	RMSEA (90% CI)
CFA^a	2232.32 (545)	<.001	0.93	0.92	0.037 (0.035–0.038)
CFA second order	2568.20 (554)	<.001	0.92	0.91	0.040 (0.038–0.041)
CFA second order + correlated factors	2363.16 (553)	<.001	0.92	0.92	0.038 (0.036–0.039)
Achievement 2017^b	2525.99 (658)	<.001	0.92	0.92	0.035 (0.034–0.037)

^aN = 2321.

^bN = 2301.

Table 2.

First-order Factor Loadings for Student Engagement Instrument.

Factors	Item Number	Standardized Loading
F1 Teacher–Student Relationships (TSR)	U3	0.71
	U5	0.68
	U10	0.60
	U13	0.72
	U16	0.75
	U21	0.70
	U22	0.69
	U27	0.69
	U31	0.76
F2 Peer Support for Learning (PSL)	U4	0.63
	U6	0.73
	U7	0.74
	U14	0.73
	U23	0.80
	U24	0.76
F3 Family Support for Learning (FSL)	U1	0.67
	U12	0.73
	U20	0.78
	U29	0.77
F4 Control and Relevance of Schoolwork (CRSW)	U2	0.55
	U9	0.67
	U15	0.62
	U25	0.70
	U26	0.67
	U28	0.58
	U33	0.68
	U34	0.74
	U35	0.53
F5 Future Goals and Aspirations (FGA)	U8	0.79
	U11	0.69
	U17	0.72
	U19	0.84
	U30	0.85
F6 Extrinsic Motivation (EM)	U18	0.64
	U32	0.85

Table 3.

Second-order Factor Loadings for Student Engagement Instrument.

	Standardized Loading
F1 Teacher–Student Relationships (TSR)	0.84
F2 Peer Support for Learning (PSL)	0.65
F3 Family Support for Learning (FSL)	0.78
F4 Control and Relevance of Schoolwork (CRSW)	0.91
F5 Future Goals and Aspirations (FGA)	0.81
F6 Extrinsic Motivation (EM)	0.28

To examine the predictive ability of SEI on achievement the following year, we ran a SEM model, controlling for age and sex. Due to missing values in age and sex, 20 (0.9%) participants were excluded. The longitudinal model has adequate adjustment results (Table 1). Following Figure 1, general factor has (β = 0.11) a positive relationship with achievement 1 year later. In addition, younger students (β = −0.08) and female participants (β = 0.23) had better grades. Overall, the model a variance 7% for achievement (R² = 0.07).

Figure 1.

Second-order SEI Factor Predicting Achievement One Year Later.

Discussion

Our study provides further international evidence for the factorial structure for SEI and is among the first studies of students’ engagement conducted in South America. However, some factors did not predict student achievement 1 year later. In particular, FGA and EM were the only factors associated with future academic achievement, consistent with the cognitive features of student engagement but missing the emotional dimension. These differences may be due to features of the Chilean school system in which students with lower academic achievement tend to be concentrated in the public schools whereas those with higher achievement are more likely to be in private schools. Students may already have dropped out by the time of our data collection and/or there may be negative peer effects on students’ engagement and achievement in lower achieving schools. Future studies should explore the way Chilean adolescents self-report their information about emotional engagement while further examining its effect on achievement and other indicators of well-being. Students’ engagement at school and with learning consists of alterable variables that are directly linked to important outcomes, such as achievement, high school graduation, and college attendance. Thus, the interest in measuring students’ engagement is driven by the desire to intervene and positively impact students’ educational outcomes. The psychometric examination of instruments such as the SEI is one step toward this goal; however, work is still emerging in how student engagement data may be most effectively linked to intervention (e.g., Fredricks et al., 2019; Reschly, 2020). Engagement research is an important source of information for school personnel to provide better support for student learning and achievement, especially for reforms in Chile promoting high school completion (Espínola & Claro, 2010). Thus, validated national and international instruments represent important tools for this development.

Supplemental Material

Supplemental Material - Cross-Cultural Validation of the Student Engagement Instrument for Chilean Students

Supplemental Material for Cross-Cultural Validation of the Student Engagement Instrument for Chilean Students by Jorge J. Varela, Roberto Melipillán, Amy L. Reschly, Ana Maria Squicciarini Navarro, Felipe Peña Quintanilla, and Paola Sánchez Campos in Journal of Psychoeducational Assessment

Footnotes

Declaration of Conflicting Interests

The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: The SEI has been made available for free to researchers and educators around the world. Amy Reschly may receive royalties from a web-based version of the Student Engagement Instrument.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iDs

Jorge J. Varela

Amy L. Reschly

Supplemental Material

Supplemental material for this article is available online.

References

Appleton

J. J.

Christenson

S. L.

Kim

Reschly

A. L.

(2006). Measuring cognitive and psychological engagement: Validation of the student engagement instrument. Journal of School Psychology, 44(5), 427–445. http://doi.org/10.1016/j.jsp.2006.04.002

Christenson

S. L.

Pohl

(2020). The relevance of student engagement: The impact of and lessons learned implementing Check & Connect. In Reschly

A. L.

Pohl

Christenson

S. L.

(Eds.), Student engagement: Effective academic, behavioral, cognitive, and affective interventions at school (pp. 3–30). Springer.

Espínola

Claro

J. P.

(2010). Estrategias de prevención de la deserción en la educación secundaria: Perspectiva latinoamericana. Revista de Educación, 257–280.

Finney

S. J.

DiStefano

(2013). Nonnormal and categorical data in structural equation modeling. In Hancock

G. R.

Mueller

R. O

(Eds.), Structural equation modeling. A second course (2nd ed., pp. 439–492). Information Age Publishing.

Forero

C. G.

Maydeu-Olivares

Gallardo-Pujol

(2009). Factor analysis with ordinal indicators: A Monte Carlo study comparing DWLS and ULS estimation. Structural Equation Modeling, 16(4), 625–641. https://doi.org/10.1080/10705510903203573

Fraysier

K. W.

Reschly

Appleton

J. J.

(2020). Predicting postsecondary enrollment with secondary student engagement data. Journal of Psychoeducational Assessment, 38(7), 882–899. http://doi.org/10.1177/0734282920903168

Fredricks

J. A

Blumenfeld

P. C.

Paris

A. H.

(2004). School engagement: Potential of the concept, state of the evidence. Review of Educational Research, 74(1), 59–109. http://doi.org/10.3102/00346543074001059

Fredricks

Reschly

A. L.

Christenson

S. L.

(2019 ). Engagement interventions: Overview and state of the field. In Fredricks

Reschly

A. L.

Christenson

S. L.

(Eds.), Handbook of Student Engagement Interventions: Working with Disengaged Youth. Elsevier.

Bentler

(1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6(1), 1–55. https://doi.org/10.1080/10705519909540118

10.

Lam

S.-F.

Jimerson

Shin

Cefai

Veiga

F. H.

Hatzichristou

Polychroni

Kikas

Wong

B. P. H.

Stanculescu

Basnett

Duck

Farrell

Liu

Negovan

Nelson

Yang

Zollneritsch

(2016). Cultural universality and specificity of student engagement in school: The results of an international study from 12 countries. British Journal of Educational Psychology, 86(1), 137–153. https://doi.org/10.1111/bjep.12079

11.

Lawson

M. A.

Masyn

K. E.

(2015). Analyzing profiles, predictors, and consequences of student engagement dispositions. Journal of School Psychology, 53, 63–86. https://dx-doi-org.web.bisu.edu.cn/10.1016/j.jsp.2014.11.004

12.

Lei

Cui

Zhou

, (2018). Relationships between student engagement and academic achievement: A meta-analysis. Social Behavior and Personality, 46(3), 517–528. http://doi.org/10.2224/sbp.7054

13.

Lei

P.-W.

(2012). Estimation in structural equation modeling. In Hoyle

R.H.

(Ed.), Handbook of structural equation modeling (pp. 164–179). The Guilford Press.

14.

Lovelace

M. D.

Reschly

A. L.

Appleton

J. J.

Lutz

M. E.

(2014). Concurrent and predictive validity of the student engagement instrument. Journal of Psychoeducational Assessment, 32(6), 509–520. http://doi.org/10.1177/0734282914527548

15.

Moreira

P. A. S.

Vaz

F. M.

Dias

P. C.

Petracchi

(2009). Psychometric properties of the Portuguese version of the Student Engagement Instrument. Canadian Journal of School Psychology, 24, 303–317. https://doi.org/10.1177/0829573509346680

16.

National Research Council & the Institute of Medicine (2004). Engaging schools: Fostering high school students’ motivation to learn. Washington, DC: National Academies Press.

17.

O’Donnell

K. C.

Reschly

A. L.

(2020). Assessment of student engagement. In Reschly

A. L.

Pohl

A. J.

Christenson

S. L.

(Eds), Student engagement: Effective academic, behavioral, cognitive, and affective interventions at school (pp. 55–76). Springer.

18.

Reschly

A. L.

(2020). Dropout prevention and student engagement. In Reschly

A. L.

Pohl

Christenson

S. L.

(Eds), Student engagement: Effective academic, behavioral, cognitive, and affective interventions at school. (pp. 31–54). Springer.

19.

Reschly

A. L.

Christenson

S. L.

(Eds.). (2022). Handbook of research on student engagement. (2nd ed.). Springer.

20.

Virtanen

Kiuru

Lerkkanen

M.-K.

Poikkeus

A.-M.

Kuorelahti

(2016). Assessment of student engagement among junior high school students and associations with self-esteem, burnout, and academic achievement. Journal for Educational Research Online, 8(2), 136–157. https://doi.org/10.25656/01:12430

21.

Virtanen

T. E.

Moreira

Ulvseth

Andersson

Tetler

Kuorelahti

(2018). Analyzing measurement invariance of the students’ engagement instrument brief version: The cases of Denmark, Finland, and Portugal. Canadian Journal of School Psychology, 33(4), 297–313. http://doi.org/10.1177/0829573517699333

22.

West

S. G.

Taylor

A. B.

(2012). Model fit and model selection in structural equation modeling. In Hoyle

R. H.

(Ed.), Handbook of structural equation modeling (pp. 209–231). Guilford.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.47 MB