Does School Climate Mean the Same Thing in the United States as in Mexico? A Focus on Measurement Invariance

Abstract

School climate is an important construct for guiding violence prevention efforts in U.S. schools, but there has been less consideration of this concept in its neighboring country Mexico, which has a higher prevalence of violence. The U.S. Department of Education outlined a three-domain conceptualization of school climate (i.e., safe and supportive schools model) that includes engagement, safety, and the school environment. To examine the applicability of this school climate model in Mexico, the present study tested its measurement invariance across middle school students in the United States (n = 15,099) and Mexico (n = 2,211). Findings supported full invariance for engagement and modified-safety scales indicating that factor loadings and intercepts contributed almost equally to factor means, and scale scores were comparable across groups. Partial invariance was found for the environment scales. Results of a multigroup confirmatory factor analysis (MGCFA) consisting of all 13 school climate scales indicated significantly positive associations among all scales in the U.S. sample and among most scales in the Mexico sample. Implications of these findings are discussed.

Keywords

school climate measurement invariance Mexico international comparison violence prevention

The National School Climate Council (2007) defined school climate as the “norms, values and expectations that support people feeling socially, emotionally and physically safe. People are engaged and respected” (p. 5). Positive school climate has been linked with several positive outcomes for students. For example, students experiencing a positive school climate tend to experience less bullying and peer victimization (Cornell, Shukla, & Konold, 2015; Shukla, Konold, & Cornell, 2016; Steffgen, Recchia, & Viechtbauer, 2013; Waasdorp, Bradshaw, & Leaf, 2012), are less likely to be suspended (Bradshaw, Mitchell, & Leaf, 2010; Gregory, Cornell, & Fan, 2011), and have more desirable psychological and behavioral outcomes (Loukas & Murphy, 2007; Way, Reddy, & Rhodes, 2007) than those who do not. Moreover, a positive school climate is consistently associated with higher engagement with the school (Cornell, Shukla, & Konold, 2016; Gill, Ashton, & Algina, 2004) and better academic achievement (Wang & Eccles, 2013). Not surprisingly, school climate is often a target of school improvement and safety efforts, such as the Every Student Succeeds Act.

A challenge to research on school climate is its measurement, as it is a broad, multidimensional construct. Researchers have operationally defined it differently depending upon their interest, often focusing on disciplinary structure, relationship between students and adults in the school, students’ engagement with the school, student safety, and physical environment (Shukla et al., 2016; Thapa, Cohen, Guffey, & Higgins-D’Alessandro, 2013). The American Institutes for Research (AIR; 2016) in collaboration with the U.S. Department of Education (USDOE) outlined a three-domain model of school climate (i.e., safe and supportive schools model) which has been validated by Bradshaw, Waasdorp, Debnam, and Johnson (2014). Specifically, their findings supported this conceptualization of school climate and identified constructs that composed the following three broad domains: engagement (i.e., connection to teachers, student connectedness, academic achievement, school connectedness, equity, and parent engagement), environment (i.e., rules and consequences, physical comfort, and support, disorder), and safety (i.e., perceived safety, bullying and aggression, and drug use).

However, this comprehensive conceptualization of school climate has not been empirically studied in other countries. A better understanding of the conceptualization and measurement of school climate is needed, particularly in countries which have high rates of school and community violence and fewer resources for school-based prevention. Such research in Mexico is important and timely, as violence has sharply increased since 2007, especially among young people. The youth homicide rate increased from 7.8 to 25.5 persons per 100,000 inhabitants between 2007 and 2010 (World Bank, 2012). In addition, young people committed almost half of the crimes in 2010 (World Bank, 2012). As a result of these troubling times, there is a need for a greater focus on school climate and related prevention efforts to address youth violence in Mexico. It is important to investigate the factors within the school context that may explain why school-aged youth engage in risk behaviors, particularly as the school climate may be targeted as a modifiable factor in prevention initiatives. However, school systems’ ability to develop and implement such initiatives may be hampered by a vague description that leaves school leaders lacking an understanding of how to develop positive school climates, highlighting the need for clearer models shown to fit the culture and context. Toward that end, the current study aimed to examine the measurement of school climate in Mexico in relation to the USDOE’s three-domain model.

School Climate Research in Mexico

Compared with the United States, there has been relatively limited research on school climate in Mexico. However, the available research in Mexico has demonstrated a similarly positive association between student academic achievement and school climate factors, such as student sense of belonging, availability of classroom resources, teacher attendance and punctuality, positive classroom interactions, and parent engagement and home inputs (United Nations Educational, Scientific and Cultural Organization [UNESCO], 2015). Conversely, violence in the school environment or neighborhood has been linked with poorer academic outcomes (Caudillo & Torche, 2014; UNESCO, 2015). Other research has examined the association between school climate and student behaviors and found that relationships with parents and peers as well as strong connections with local institutions and schools are consistently predictive of a variety of positive youth behaviors (Cunningham & Bagby, 2010); moreover, having more experienced teachers and greater participation among school stakeholders were associated with fewer behavioral problems in the classroom and subsequently improved classroom time management (Moriconi & Bélanger, 2015). In addition, both classroom (teachers’ attitudes, teaching methodology, efficient use of time, and classroom climate) and school-level factors (teachers’ commitment to the school, school management style, existence of school objectives that are known and shared by the school community, and the school’s concern for educational themes) have been linked with better academic behavior, social interactions, and school satisfaction among students (Murillo Torrecilla & Hernández Castilla, 2011).

The existing literature also highlights the urgent need to improve the current school climate in Mexican schools. For example, of 23 countries participating in the 2013 Organization for Economic Co-Operation and Development (OECD; 2009) Teaching and Learning International Survey (TALIS), Mexico consistently ranked as one of the most problematic countries surveyed in terms of school climate. Approximately one in five Mexican students reported a low sense of belonging with the school and have low school participation, with children from disadvantaged backgrounds much more likely to report these issues than children from higher social backgrounds (OECD, 2009).

Importantly, although studies have drawn on the elements of USDOE conceptualization of school climate, the fit of the entire three-domain model (student engagement, school environment, and safety) has not been examined in Mexico. Whereas the bulk of the findings guided by the USDOE conceptualization do support its utility in Mexico, other research suggests that factors may contribute differently to school climate in this context. For example, whereas students’ sense of belonging has been shown to positively correlate with academic achievement, students’ feelings of connectedness with their teachers were surprisingly associated with poorer academic achievement (Weiss & García, 2015). The authors suggested that a different type of teacher relationship is valued in Mexico schools—Perhaps rather than a warm personal connection, the most successful students typically require minimal attention from teachers and have a more formal relationship with their teachers (Weiss & García, 2015). Together, these findings suggest the potential differences in school climate between the United States and Mexico; however, for any meaningful comparison, an essential first step is the establishment of measurement invariance (MI) across groups.

MI

To examine and identify potential group differences on constructs, like school climate, MI needs to be established across respondent groups (Byrne & van De Vijver, 2010). Specifically, MI determines whether differences in observed construct means across groups are attributed only to the construct-level variances (Meredith, 1993). It can be tested by imposing more restrictive constraints on the measurement model of latent variables across groups in three stages (Sass, 2011): (a) configural invariance, (b) metric invariance (or weak/partial invariance), and (c) scalar invariance (or strong/full invariance).

Configural invariance tests if the hypothesized factor structure of a study measure is valid across different groups of participants. Once configural invariance is established, equality of unstandardized factor loadings is imposed across groups to test for metric invariance. Specifically, metric invariance indicates that association between items and their latent factor is identical across groups, and one-unit increase in score on an item has a comparable unit increase on its factor score across groups. Finally, scalar invariance is tested by constraining the unstandardized loadings as well as intercepts to be equal across groups. Scalar invariance suggests that groups can be compared on scale-score means. Lack of scalar invariance may occur if one group of participants responds more strongly to an item or a set of items than other group for the same level of latent trait (i.e., same factor-score mean). Scalar noninvariance could also be due to differential reference-frame for responding and/or social desirability in groups (Chen, 2007). It is important that researchers first establish MI before attributing the scale-score differences to groups.

Present Study

Researchers typically assume that items are understood and interpreted in the same way across respondents, and answered with the same frame of reference regardless of group membership (Segeritz & Pant, 2013). However, this assumption has not been tested in the context of school climate for U.S. versus Mexican youth. The current study examined MI of the USDOE’s model of school climate among the U.S. and Mexican middle school students, with particular focus on the three broad domains of safety, engagement, and environment, and the relevant subscales. More specifically, this study explored the following research questions:

Research Question 1: Is there evidence of configural invariance, metric invariance, and scalar invariance for the engagement, environment, and safety domains of school climate across U.S. and Mexican students?

Research Question 2: Is there evidence of the convergent validity of three-factor school climate model for U.S. and Mexican students?

Method

Participants and Procedure

The U.S. sample included 15,099 students (50.5% Male; age: M = 12.82, SD = 0.87) from 27 public middle schools from the state of Maryland. The racial breakdown for the U.S. sample was 26.9% White, 27.9% African American, 22.2% Hispanic, 9.4% Asian, 2.8% Native American, .4% Native Hawaiian or other Pacific Islander, and 10.5% Others. The Mexican sample included 2,211 (49.1% Male; age: M = 13.67, SD = 1.04) students from eight public middle schools in a medium to low socioeconomic metropolitan area of Guadalajara, Mexico. In both Mexico and the United States, the survey was administered online during school hours with each school having a uniquely identifying password.

Measure

The Maryland Safe and Supportive Schools (MDS3) School Climate Survey was developed by the Johns Hopkins Center for Youth Violence Prevention in collaboration with project partners. The survey was designed based on the three domains of the USDOE model of school climate (AIR, 2016) and focus groups with different stakeholders (i.e., students, district personnel, and school administrators). The MDS3 School Climate Survey is comprised of 56 core items and prior studies have validated the three main scales: safety, engagement, and environment (see Bradshaw et al., 2014, for details). For the Spanish version of the MDS3 measure, all items were translated into Spanish and then back translated by native Spanish speakers from Mexico and Colombia.

Student engagement was measured with six subscales: Teacher Connectedness (six items), Student Connectedness (five items), Student Achievement (four items), Whole-School Connectedness (four items), Culture of Equity (four items), and Parent Involvement (five items). School environment consisted of four subscales: School Rules and Consequences (five items), Physical Comfort (four items), Support (three items), Disorder (five items); whereas safety consisted of three subscales (three items each): Bullying, Physical Safety, and General Drug Use. All answer choices were on a 4-point Likert-type scale from strongly agree to strongly disagree, and all items were coded with high scores representing a more favorable school climate.

Analyses

To address the first research question, MI tests were conducted separately for the domains of engagement, environment, and safety across groups using the Mplus 7.1 software (Muthén & Muthén, 1998-2016). To examine whether the model fit adequately for both groups, configural invariance was tested by allowing all model parameters to vary freely across groups as illustrated elsewhere (Sass, 2011). This was essentially a multigroup confirmatory factor analytic model (MGCFA) which served as a baseline model. Goodness of model fit criteria included root mean square error of approximation (RMSEA; ≤.10; Fan, Thompson, & Wang, 1999), standardized root mean square residual (SRMR; ≤.08), and comparative fit index (CFI) ≥ .95 (Hu & Bentler, 1999).

If configural invariance was present, the next step was to impose more restrictive conditions by holding the unstandardized factor loadings equal across groups for investigating the metric invariance. Contemporary research suggests that the likelihood ratio test that is typically employed to evaluate competing nested models is overly sensitive to large samples (Meade, Johnson, & Braddy, 2008; Sass, 2011). Therefore, some methodological literature recommends evaluating MI through acceptable changes in alternative fit indices (i.e., CFI, RMSEA, SRMR). For example, Cheung and Rensvold (2002) suggested ΔCFI ≤ .01 for testing MI. Recently, Chen (2007) recommended ΔCFI ≤ .005, supplemented by ΔRMSEA ≤ .01 or ΔSRMR ≤ .025 for testing metric invariance; and ΔCFI ≤ .005, supplemented by ΔRMSEA ≤ .01 or ΔSRMR ≤ .005 for scalar invariance when the groups are unequal as in the case of this study. However, Meade et al. (2008) recommended a more conservative criterion of ΔCFI ≤ .002 for testing invariance. In conclusion, there is no consensus in literature on one criterion for testing invariance, and a researcher may need to take a subjective call in inferring if there exists MI (Sass, 2011). Thus, we considered changes in alternative fit indices, as well as the magnitude of difference in parameter estimates between successive models across the groups.

Next, unstandardized factor loadings and intercepts were constrained to be equal across groups for evaluating scalar invariance (or strong factorial invariance). All analyses were conducted using the maximum likelihood estimation procedure with robust standard errors (MLR estimator). The choice of appropriate estimator is important for obtaining unbiased parameter estimates. Although weighted least square means and variances adjusted (WLSMV) estimator could be more suitable for variables measured on 4-point Likert-type scale (Rhemtulla, Brosseau-Liard, & Savalei, 2012), MLR is more suitable for testing MI based on changes in alternative fit indices like ΔCFI and ΔRMSEA than WLSMV (Sass, Schmitt, & Marsh, 2014). Moreover, the methodological studies which recommended MI testing criteria based on model fit statistics primarily employed the maximum likelihood estimation procedure (e.g., Chen, 2007; Meade et al., 2008). Therefore, the choice of MLR estimator for MI testing was most appropriate. The standard errors were adjusted to accommodate the nested data structure (students nested within their schools) using the sandwich estimator in Mplus by specifying type = COMPLEX in analysis command.

Finally, the convergent validity of school climate measures was examined to answer the second research question. For this purpose, an MGCFA was conducted where all 13 subfactors (six engagement factors, four environment factors, and three safety factors) were allowed to be correlated with one another.

Results

MI for Engagement

Findings of MGCFA revealed a six-factor model: teacher connectedness, student connectedness, student achievement, whole-school connectedness, culture of equity, and parent involvement. The model fit the data well; CFI = .945, RMSEA = .047, and SRMR = .049 (Table 1; Configural Model). Comparison between configural invariance and metric invariance models indicated that the fit statistics did not significantly change; ΔCFI = .002, ΔRMSEA < .001, ΔSRMR = .003. In other words, these results supported metric invariance across U.S. and Mexico students for engagement. Finally, the scalar invariance model was compared with the metric invariance model. The change in model fit statistic was marginally higher (.008) than the recommended cutoff (.005) for CFI, but within the acceptable limits for RMSEA and SRMR; ΔRMSEA = .002 < .01, and ΔSRMR = .003 < .005. In total, there was evidence for a strong invariance (configural, metric, and scalar) across the two groups for the engagement factors. The freely estimated intercept and factor loading values are presented in Table 2 for both the groups. Overall, the intercept and loading values across different items of the engagement scales exhibited an identical pattern across both the groups (Table 2) providing a supporting evidence for a full MI. The scale reliability estimates ranged between .51 and .90 and are presented in the form of alpha coefficient values.

Table 1.

Model Fit Statistics.

Model	CFI	ΔCFI	RMSEA	ΔRMSEA	SRMR	ΔSRMR
Engagement
Configural	0.945		0.047		0.049
Metric	0.943	0.002	0.047	<0.001	0.052	0.003
Scalar	0.935	0.008^a	0.049	0.002	0.055	0.003
Environment
Configural	0.955		0.044		0.035
Metric	0.954	0.001	0.043	0.001	0.037	0.002
Scalar	0.94	0.014^a	0.048	0.005	0.045	0.008^a
Safety
Configural	0.983		0.053		0.042
Metric	0.978	0.005^a	0.057	0.004	0.053	0.011^a
Scalar	0.973	0.005^a	0.059	0.002	0.063	0.010^a
Safety–Modified
Configural	0.995		0.03		0.018
Metric	0.994	0.001	0.03	<0.001	0.022	0.004
Scalar	0.99	0.004	0.037	0.007	0.035	0.013

Note. CFI = comparative fit index; RMSEA = root mean square error of approximation; SRMR = standardized root mean square residual.

Indicates change in fit indices that is greater than the recommended cutoff value.

Table 2.

Completely Standardized Parameter Estimates for Multigroup Confirmatory Factor Analysis.

Scales and Items	Intercepts		Factor loading
Scales and Items	The United States	Mexico	The United States	Mexico
Engagement
Teacher connectedness (α = .88 for the United States and α = .82 for Mexico)
My teachers listen to me when I have something to say.	3.47	4.10	0.78	0.73
My teachers care about me.	3.65	4.06	0.83	0.71
Teachers respect the students.	3.48	4.33	0.80	0.72
My teachers tell me when I do a good job.	3.68	4.34	0.67	0.50
At this school, my teachers notice when I am not there.	3.52	4.11	0.66	0.65
At this school, students trust the teachers.	2.93	3.80	0.74	0.67
Student connectedness (α = .87 for the United States and α = .81 for Mexico)
I feel like I belong here.	3.28	4.34	0.67	0.53
Students help one another.	3.22	3.67	0.81	0.77
Students respect one another.	2.78	3.05	0.79	0.71
At this school, students like one another.	3.25	3.72	0.74	0.65
At this school, students trust one another.	3.02	3.46	0.76	0.79
Student achievement (α = .77 for the United States and α = .76 for Mexico)
My teachers believe that I can do well in school.	4.32	4.42	0.82	0.69
I believe I can do well in school.	4.60	4.69	0.58	0.72
My teachers always want me to do my best.	4.50	4.88	0.82	0.71
It is important to finish high school.	7.33	5.96	0.44	0.52
Whole-school connectedness (α = .86 for the United States and α = .77 for Mexico)
Students and staff feel pride in this school.	3.31	4.21	0.71	0.55
I enjoy learning at this school.	3.02	4.38	0.82	0.78
I like this school.	2.84	3.57	0.82	0.57
I like coming to school.	2.68	4.11	0.76	0.75
Culture of Equity (α = .82 for the United States and α = .79 for Mexico)
At this school, students of all races are treated the same.	2.99	3.89	0.80	0.71
At this school, all students are treated the same, regardless of whether their parents are rich or poor.	3.06	4.07	0.81	0.74
At this school, boys and girls are treated equally well.	3.11	4.10	0.76	0.75
The school provides instructional materials that reflect my culture, ethnicity, and identity.	2.97	3.65	0.59	0.60
Parent involvement (α = .73 for the United States and α = .67 for Mexico)
The school tries to involve parents or guardians.	3.38	3.94	0.70	0.65
When I do something good at school, my parent(s) or guardian(s) usually hears about it.	2.46	3.99	0.48	0.58
Parents or guardians often come to my school to help out.	2.36	3.28	0.61	0.66
My parent(s) or guardian(s) feels welcome at this school.	4.05	5.11	0.66	0.70
If I do something bad at school, my parent(s) or guardian(s) hears about it.	4.42	4.92	0.52	0.56
Environment
Rules and consequences (α = .76 for the United States and α = .67 for Mexico)
Students listen to the teachers.	3.01	3.47	0.63	0.51
At this school, teachers can handle students who disrupt class.	2.99	3.43	0.67	0.50
There are clear rules about student behavior.	3.67	4.00	0.63	0.59
Students are rewarded for positive behavior.	2.64	2.99	0.59	0.52
Everyone knows what the school rules are.	3.34	3.68	0.61	0.56
Physical comfort/cleanliness (α = .78 for the United States and α = .75 for Mexico)
The bathrooms in this school are clean.	2.34	2.93	0.71	0.59
The school is usually clean and well-maintained.	2.98	3.57	0.78	0.79
The temperature in this school is comfortable all year.	2.62	3.22	0.56	0.65
This school has a bright and pleasant appearance.	3.01	3.62	0.72	0.61
Support (α = .76 for the United States and α = .70 for Mexico)
Teachers at my school help students with their problems.	3.38	3.68	0.83	0.76
Students who need help for their problems are able to get it through school.	3.39	3.80	0.78	0.75
There is someone at school who I can talk to about personal problems.	2.96	3.35	0.57	0.51
Disorder (α = .51 for the United States and α = .41 for Mexico)
Students disobey the rules.	2.51	2.57	0.35	0.29
Misbehaving students get away with it.	2.80	2.74	0.47	0.35
There are often broken windows, doors, or desks in this school.	3.47	2.41	0.50	0.35
Vandalism of school property is a problem at this school.	2.98	2.63	0.47	0.53
Safety
Bullying and aggression (α = .63 and .65 for the United States; α = .41 and .66 for Mexico for three-item and two-item scale)
Physical fighting between students.	2.43	2.41	0.63	0.63
Harassment or bullying of students.	2.25	2.01	0.69	0.81
Students intervene with bullying.	2.63	2.68	0.52	0.05
Physical safety (α = .59 for the United States and α = .51 for Mexico)
I feel safe at this school.	4.22	5.10	0.78	0.49
I feel safe going to and from school.	4.47	4.71	0.69	0.41
Programs for violence.	2.77	3.56	0.43	0.59
General drug use (α = .90 for the United States and α = .90 for Mexico)
Students’ drug use (such as marijuana, Lysergic acid diethylamide [LSD], cocaine, ecstasy)	3.00	1.87	0.85	0.87
Students’ tobacco use (cigarettes, chew, cigars)	3.37	2.03	0.89	0.90
The students at my school use alcohol (such as beer, wine, liquor).	3.70	2.13	0.86	0.84

MI for Environment

Configural invariance was supported by the model fit statistics (CFI = .945, RMSEA = .047, and SRMR = .049; Table 1 Metric Model) of the baseline MGCFA model which consisted of four factors: school rules and consequences, physical comfort, support, and disorder. Change in fit indices was within acceptable limits after imposing equal loading constraints indicating metric invariance. However, scalar invariance was not found; ΔCFI = .014. Inspection of freely estimated intercepts (Table 2) suggested that Mexican students were likely to have higher intercepts than U.S. students for the same factor-score means for school rules and consequences, physical comfort, and support scales.

MI for Safety

MGCFA demonstrated a good model fit indicating a configural invariance; CFI = .983, RMSEA = .053, and SRMR = .042. This model consisted of three factors: bullying and aggression, physical safety, and general drug use. However, metric invariance was not found, ΔCFI = .005, ΔSRMR = .011. Furthermore examination revealed that this metric noninvariance was primarily due to significant difference in factor loading for one item. For the bullying and aggression factor, the item “students intervene with bullying” loaded well (.52) for the U.S. group, but loaded poorly (.05) for the Mexico group (Table 2). Once this item was removed, full invariance was achieved for the safety factors. The modified-safety configural invariance model fitted the data well; CFI = .995, RMSEA = .03, SRMR = .018. On imposing more restrictive conditions for testing metric and scalar invariance, the fit indices did not change significantly (see Table 1).

Convergent Validity Examination

A multigroup CFA was conducted which included all 13 subfactors of school climate (i.e., six factors for engagement, four for environment, and three for safety). These subfactors were allowed to covary freely across groups to examine the convergent validity of school climate scales. Table 3 presents the correlation coefficient values for relations between the latent factors for both groups. Overall, the model fit was acceptable fit (CFI = .92, RMSEA = .037, SRMR = .048).

Table 3.

Correlations Between Latent Factors of School Climate for the United States (Lower Diagonal) and Mexican (Upper Diagonal) Samples.

Scales	1	2	3	4	5	6	7	8	9	10	11	12	13
1 Student–teacher	—	.67***	.72***	.67***	.55***	.81***	.68***	.44***	.70***	.01	−.02	.56***	.01
2 Student–student	.71***	—	.41***	.53***	.45***	.65***	.55***	.43***	.57***	−.05	.04	.54***	−.04
3 Academic engagement	.84***	.55***	—	.79***	.42***	.59***	.51***	.34***	.45***	.03	−.03	.48***	.02
4 School Connectedness	.80***	.71***	.72***	—	.38***	.62***	.52***	.38***	.48***	.08*	−.03	.65***	.01
5 Culture of equity	.65***	.60***	.55***	.59***	—	.56***	.68***	.43***	.63***	<.001	.10***	.51***	.06**
6 Parent involvement	.81***	.67***	.69***	.75***	.63***	—	.68***	.50***	.72***	−.03	−.04	.57***	−.02
7 Rules	.79***	.69***	.65***	.72***	.72***	.80***	—	.48***	.73***	−.14**	−.04	.56***	−.01
8 Comfort	.61***	.56***	.47***	.65***	.58***	.61***	.70***	—	.53***	−.17**	.02	.47***	−.01
9 Support	.80***	.63***	.67***	.69***	.70***	.76***	.82***	.62***	—	−.10*	−.04	.53***	−.04
10 Disorder	.46***	.46***	.37***	.49***	.51***	.44***	.54***	.58***	.46***	—	.36***	.08*	.33***
11 Bullying	.32***	.47***	.22***	.38***	.44***	.31***	.38***	.39***	.32***	.60***	—	.06*	.81***
12 Physical safety	.64***	.64***	.57***	.69***	.63***	.64***	.66***	.60***	.62***	.54***	.53***	—	.03
13 Drug use	.27***	.23***	.27***	.27***	.28***	.23***	.27***	.25***	.23***	.40***	.56***	.34***	—

Note. Lower off-diagonal represent correlations for U.S. students and upper off-diagonal values represent Mexican students. All variables were coded so that higher values reflect more positive perceptions of school climate.

p < .05. **p < .01. ***p < .001.

For the U.S. sample, all factor correlations were significantly positive with values ranging from .23 to .84 (Table 3). More specifically, the correlation values for engagement factors ranged between .55 and .84. For factors associated with environment, correlation values were between .46 and .82; and for safety factors the values ranged between .34 and .56. In addition, the correlations were significantly positive for factors across the broad dimensions of engagement, environment, and safety. Students reporting higher levels of student–teacher connectedness were likely to report higher levels of student–student connectedness, academic achievement, school connectedness, culture of equity in school, and parental involvement with school. Fairer school rules and consequences were linked with more physical comfort and support in school, physical safety, and less disorder, bullying, and drug usage in school than those reporting lower levels.

A similar pattern was observed for the Mexican sample for all of the subfactors, except for disorder, bullying, and drug use in school (Table 3, upper off-diagonal values). For the engagement subfactors, the correlation coefficient values were significantly positive and ranged between .38 and .81. Except for school disorder, all the subfactors of environment were significantly associated with one another with coefficient values ranging from .48 to .73. Unlike our expectation, Mexican students who reported higher levels of fair school rules, physical comfort, and support in school were likely to report higher levels of disorder (note that the variables were recoded so that the higher values indicated a more positive perception of school climate). Likewise, students who reported a positive perception of the subfactors of engagement and environment did not necessarily report less bullying and/or drug usage in their schools.

Discussion

Taken together, the findings suggested that the USDOE’s safe and supportive school climate model is valid for Mexico. Specifically, full MI was achieved for engagement and safety (after modification) measures, meaning that factor loadings and intercepts contribute almost equally to the factor means, and scale scores are comparable across groups. Moreover, partial invariance was achieved for the environment scales, indicating that the loadings contribute equally to the factor. In other words, a one-unit increase in an observed item-score is linked with an equal unit increase in its scale score across groups. Therefore, researchers could compare the relations among environment scales across groups, but should avoid means comparison of observed scale scores.

In addition, evidence for convergent validity of these school climate scales was obtained, as the associations between most scales were significantly positive and consistent across groups. However, the relations among the disorder, bullying, and drug use scales were rather perplexing, especially in the Mexican group. It is possible that these unexpected variations were due to differences in student understanding, due to discrepancies in translation or a different level of familiarity with the response options provided. Specifically, it is notable that these three scales were the only ones phrased in the reverse, such that a higher score indicated a poorer rather than more positive school climate. Alternatively, it may be that these variables—indicators of vandalism and disrepair, peer-on-peer aggression, and substance use—are more commonly experienced in these urban, low-to-middle socio-economical status (SES) Mexican communities and therefore have a weaker association with the more positive school climate variables. Certainly, more research is needed to examine and better understand these differential associations.

As noted above, Weiss and García (2015) found that greater student–teacher connectedness was associated with poorer academic achievement in Mexico, leading them to speculate that perhaps a different type of teacher relationship is valued in this context. To the contrary, the current study showed that student–teacher connectedness positively correlated with all other engagement variables in the Mexico sample. This current finding aligned with other research indicating strong connections with schools and institutions were associated with positive youth behaviors (Cunningham & Bagby, 2010). Again, this discrepancy between current and previous findings suggests a need for more research from multiple perspectives to clarify these relationships.

The global economic impacts and costs resulting from the consequences of physical, psychological, and sexual violence against children can be as high as US$7 trillion, much higher than its estimated prevention cost (Pereznieto, Montes, Routier, & Langston, 2014). As 95% of Mexican children complete primary school and nearly three fourths attend secondary (United Nations Children’s Fund [UNICEF], 2013), school-based initiatives focused on preventing or mitigating the effects of such violence are a feasible, cost-effective approach for providing safe and protective environments for a large number of Mexican children. Interestingly, many school-wide interventions for improving school climate have been developed in the United States. (e.g., Bradshaw et al., 2010). If the conceptualization of school climate is valid for the Mexico schools, researchers could test for the efficacy of such interventions in that context. Literature highlights a clear need for strategies and initiatives to improve school climate in Mexico, where teachers and principals report numerous hindrances to academic instruction (OECD, 2009), schools struggle to foster an inclusive environment for disadvantaged students and those with special education needs (Forlin, Cedillo, Romero-Contreras, Fletcher, & Rodriguez Hernández, 2010), and well-intentioned government initiatives may lack sufficient support, clarity, and resources for effective implementation (Forlin et al., 2010).

Limitations and Future Directions

This analysis tested a model using data collected through an assessment tool developed based on the USDOE conceptualization of school climate. Although these results suggest the factors assessed do hold similar relationships to underlying school climate constructs in both the United States and Mexico, we cannot conclude that these are all the possible factors that contribute to school climate in Mexico. Other factors may be locally relevant in Mexico which are not typically included in the U.S. conceptualization. For example, the Parent Involvement subscale reflects the school-centric approach to parent–school partnerships identified in the United States but does not include items assessing parent-led initiatives that research suggests may better reflect the nature of these partnerships in Mexico (Dotson-Blake, 2010). The instrument also does not assess the role of community members in fostering positive school climates, although research again suggests that community members, even those with no children in the school, may be much more involved in Mexican schools and that schools themselves may play a role in promoting youth civic engagement (Dotson-Blake, 2010). Although this analysis is an informative initial effort, further qualitative research to understand local perceptions of additional factors contributing to school climate could be used to adapt and improve on existing measures and develop more comprehensive models of school climate in Mexico.

Although this analysis adds to the literature by demonstrating the fit of the three-domain school climate model in both the United States and Mexico, for substantive interpretation of the score results it is important to recognize that the Mexico sample was much smaller than the U.S. sample and consisted of students from only eight schools, whereas the U.S. sample included 27 schools throughout the state of Maryland. Future research focusing on comparisons between the United States and Mexico would benefit from including a bigger more diverse sample of Mexican young people (e.g., geographically, economically).

Conclusion

Research comparing school climate in the United States and Mexico would be particularly helpful in identifying areas in which either the United States or Mexico struggles or excels in promoting positive school climates. This would enable researchers and educators to evaluate the cross-cultural applicability of existing prevention initiatives aimed at improving school climates. The current study contributes to the growing body of school climate literature by providing a basis for which to begin this comparative research.

Footnotes

Acknowledgements

The authors would like to thank the Maryland State Department of Education and Sheppard Pratt Health System for their support of this research through the Maryland Safe and Supportive Schools Project.

Authors’ Note

The opinions expressed are those of the authors and do not represent views of the sponsors.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was funded in part by grants from the U.S. Department of Education, the William T. Grant Foundation, and the National Institute of Justice.

References

American Institutes for Research. (2016). ED school climate survey. Retrieved from https://safesupportivelearning.ed.gov/edscls

Bradshaw

C. P.

Mitchell

M. M.

Leaf

P. J.

(2010). Examining the effects of schoolwide positive behavioral interventions and supports on student outcomes results from a randomized controlled effectiveness trial in elementary schools. Journal of Positive Behavior Interventions, 12, 133-148.

Bradshaw

C. P.

Waasdorp

T. E.

Debnam

K. J.

Johnson

S. L.

(2014). Measuring school climate in high schools: A focus on safety, engagement, and the environment. Journal of School Health, 84, 593-604.

Byrne

B. M.

van De Vijver

F. J.

(2010). Testing for measurement and structural equivalence in large-scale cross-cultural studies: Addressing the issue of nonequivalence. International Journal of Testing, 10, 107-132.

Caudillo

M. L.

Torche

(2014). Exposure to local homicides and early educational achievement in Mexico. Sociology of Education, 87, 89-105.

Chen

F. F.

(2007). Sensitivity of goodness of fit indexes to lack of measurement invariance. Structural Equation Modeling, 14, 464-504.

Cheung

G. W.

Rensvold

R. B.

(2002). Evaluating goodness-of-fit indexes for testing measurement invariance. Structural Equation Modeling, 9, 233-255.

Cornell

Shukla

Konold

(2015). Peer victimization and authoritative school climate: A multilevel approach. Journal of Educational Psychology, 107, 1186-1201.

Cornell

Shukla

Konold

T. R.

(2016). Authoritative school climate and student academic engagement, grades, and aspirations in middle and high schools. AERA Open, 2. doi: https://doi.org/10.1177/2332858416633184.

10.

Cunningham

Bagby

(2010). Factors that predispose youth to risk in Mexico and Chile. The World Bank Human Development Network Children & Youth Unit. Retrieved from http://elibrary.worldbank.org/doi/abs/10.1596/1813-9450-5333

11.

Dotson-Blake

(2010). Learning from each other: A portrait of family-school-community partnerships in the United States and Mexico. Professional School Counseling, 14, 101-114.

12.

Fan

Thompson

Wang

(1999). Effects of sample size, estimation methods, and model specification on structural equation modeling fit indexes. Structural Equation Modeling, 6, 56-83. doi:10.1080/10705519909540119

13.

Forlin

Cedillo

I. G.

Romero-Contreras

Fletcher

Rodriguez Hernández

H. J.

(2010). Inclusion in Mexico: Ensuring supportive attitudes by newly graduated teachers. International Journal of Inclusive Education, 14, 723-739.

14.

Gill

M. G.

Ashton

Algina

(2004). Authoritative schools: A test of a model to resolve the school effectiveness debate. Contemporary Educational Psychology, 29, 389-409.

15.

Gregory

Cornell

Fan

(2011). The relationship of school structure and support to suspension rates for Black and White high school students. American Educational Research Journal, 48, 904-934.

16.

Bentler

P. M.

(1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1-55. doi:10.1080/10705519909540118

17.

Loukas

Murphy

J. L.

(2007). Middle school student perceptions of school climate: Examining protective functions on subsequent adjustment problems. Journal of School Psychology, 45, 293-309.

18.

Meade

A. W.

Johnson

E. C.

Braddy

P. W.

(2008). Power and sensitivity of alternative fit indices in tests of measurement invariance. Journal of Applied Psychology, 93, 568-592.

19.

Meredith

(1993). Measurement invariance, factor analysis and factorial invariance. Psychometrika, 58, 525-543.

20.

Moriconi

Bélanger

(2015). Student behaviour and use of class time in Brazil, Chile and Mexico: Evidence from TALIS 2013. OECD Education Working Papers, 112. doi:10.1787/5js6bhlchwmt-en

21.

Murillo Torrecilla

F. J.

Hernández Castilla

. (2011). School factors associated with socio-emotional development in Latin American countries. RELIEVE, 17, 1-23. Retrieved from http://www.uv.es/RELIEVE/v17n2/RELIEVEv17n2_2.htm

22.

Muthén

L. K.

Muthén

(1998-2016). Mplus [computer software]. Los Angeles, CA: Author.

23.

National School Climate Council. (2007). The school climate challenge: Narrowing the gap between school climate research and school climate policy, practice guidelines and teacher education policy. Available from http://nscc.csee.net/

24.

Organization for Economic Co-Operation and Development. (2009). Creating effective teaching and learning environments: First results from TALIS. Retrieved from http://www.oecd.org/edu/school/creatingeffectiveteachingandlearningenvironmentsfirstresultsfromtalis.htm

25.

Pereznieto

Montes

Routier

Langston

(2014). The costs and economic impact of violence against children. Overseas Development Institute and the Child Fund Alliance. Retrieved from https://www.childfund.org/uploadedFiles/public_site/media/Articles/current/2014/ODI%20Report%20%20The%20cost%20and%20economic%20impact%20of%20violence%20against%20children.pdf

26.

Rhemtulla

Brosseau-Liard

P. É.

Savalei

(2012). When can categorical variables be treated as continuous? A comparison of robust continuous and categorical SEM estimation methods under suboptimal conditions. Psychological Methods, 17, 354-373. doi:10.1037/a0029315

27.

Sass

D. A.

(2011). Testing measurement invariance and comparing latent factor means within a confirmatory factor analysis framework. Journal of Psychoeducational Assessment, 29, 347-363.

28.

Sass

D. A.

Schmitt

T. A.

Marsh

H. W.

(2014). Evaluating model fit with ordered categorical data within a measurement invariance framework: A comparison of estimators. Structural Equation Modeling: A Multidisciplinary Journal, 21, 167-180.

29.

Segeritz

Pant

H. A.

(2013). Do they feel the same way about math? Testing measurement invariance of the PISA “students’ approaches to learning” instrument across immigrant groups within Germany. Educational and Psychological Measurement, 73, 601-630.

30.

Shukla

Konold

Cornell

(2016). Profiles of student perceptions of school climate: Relations with risk behaviors and academic outcomes. American Journal of Community Psychology, 57, 291-307.

31.

Steffgen

Recchia

Viechtbauer

(2013). The link between school climate and violence in school: A meta-analytic review. Aggression and Violent Behavior, 18, 300-309.

32.

Thapa

Cohen

Guffey

Higgins-D’Alessandro

(2013). A review of school climate research. Review of Educational Research, 83, 357-385.

33.

United Nations Children’s Fund. (2013). At a glance: Mexico—Statistics. Retrieved from http://www.unicef.org/infobycountry/mexico_statistics.html

34.

United Nations Educational, Scientific and Cultural Organization. (2015). TERCE executive summary—Associated factors 2015. Retrieved from http://www.unesco.org/new/en/santiago/terce/

35.

Waasdorp

T. E.

Bradshaw

C. P.

Leaf

P. J.

(2012). The impact of schoolwide positive behavioral interventions and supports on bullying and peer rejection: A randomized controlled effectiveness trial. Archives of Pediatrics & Adolescent Medicine, 166, 149-156.

36.

Wang

M. T.

Eccles

J. S.

(2013). School context, achievement motivation, and academic engagement: A longitudinal study of school engagement using a multidimensional perspective. Learning and Instruction, 28, 12-23.

37.

Way

Reddy

Rhodes

(2007). Students’ perceptions of school climate during the middle school years: Associations with trajectories of psychological and behavioral adjustment. American Journal of Community Psychology, 40, 194-213.

38.

Weiss

C. C.

García

(2015). Student engagement and academic performance in Mexico: Evidence and puzzles from PISA. Comparative Education Review, 59, 305-331.

39.

World Bank. (2012). Youth violence in Mexico: Report of the situation, the legal framework and government programs. Washington DC: World Bank. Retrieved from http://documents.worldbank.org/curated/en/277681468155375869/La-violencia-juvenil-en-Mexico-reporte-de-la-situacion-el-marco-legal-y-los-programas-gubernamentales