Abstract
The present study is a systematic review of school climate measures developed for a Spanish-speaking K-12 population. Four instruments were included in the analyses representing data of 9,236 students with a mean of sample size of 2,309 students. Evidences for test content and internal structure were the most frequent representation of psychometric validity. Internal consistency, represented by Cronbach’s alpha, ranged between 0.62 and 0.94. In addition, three instruments were translated from scales developed in the United States as evidence for test content validity. The findings of validity evidence, internal consistency, and context will guide school counselors in deciding the appropriate school climate measure to use among United States Latina/o students in Spanish. Consequently, the school climate assessment will compel school stakeholders to use results to advocate for the growth and inclusivity of Latina/o students.
The percentage of school-aged children who are Hispanic and heritage Spanish speakers has steadily increased during the last decade (National Center for Education Statistics, 2016; U.S. Census Bureau, 2019). As these population groups continue to increase, so will their representation in K-12 public schools along with a related need to understand the conditions that promote their success (Hussar & Bailey, 2016). Taggart (2018) identified five factors associated with Latina/o academic achievement: (1) demographic variables, (2) sociocultural variables, (3) academic experiences, (4) psychological variables, and (5) school/institutional variables. One interpretation of Taggart’s findings is that those variables which were identified as contributing to student development are synonymous with the construct of school climate. This proposition is of note when considering that schools are the principle site of socialization and peer relationships for all students (Cemalcilar, 2010), but particularly for Latina/o youth whose foundational experiences of cultural climate are developed through interactions in on K-12 campuses. Therefore, as research related to the relationships between school climate and to academic success progresses, so should the representation of the Latina/o population experiences (Reynolds, Lee, Turner, Bromhead, & Subasic, 2017).
Researchers have described school climate as related to physical and social-emotional safety (Cohen, McCabe, Michelli, & Pickeral, 2009; Lopez & Ascorra, 2012), supportive systems (Lopez & Ascorra, 2012; Way, Reddy, & Rhodes, 2007), and discipline (Kasen, Berenson, Cohen, & Johnson, 2004). Furthermore, the U.S. Department of Education (2014) referenced school climate in reference to physical structures regarding the effect it will have on attending students. The combination culminates an awareness of social and relational climate among students, teachers, administrators, and stakeholders. Zahid (2014) described the direct and indirect process school climate has on school belongingness, intrinsic motivation, peer and teacher relationships, and the outcomes that ensue in academic performance. An unwelcoming environment, or perceptions of discriminations, is directly related to a student’s sense of belonging on the campus (Arbona, Fan, & Olvera, 2018). Therefore, the consideration of holistic school climate will guide the achievement outcomes and social growth of Latinas/os in schools across the country (Thapa, Cohen, Guffey, & Higgins-D’Alessandro, 2013).
School climate became the topic of discussion by identifying positive perceptions of rules, safety, and expectations leading to accomplished academic outcomes (Zullig, Koopman, Patton, & Ubbes, 2010). Consequently, scales measuring school climate are invaluable tools that allow stakeholders insight into student’s perceptions, academic outcomes, and the rationale of interventions. The recent translation to fit Spanish-speaking populations allow further insight into cultural interpretation of school climate. Authors such as Garcia Raga and Lopez Martin (2011) identified cross-cultural applications of school climate through the term Convivencia (coexistence), stating it should be part of the core teachings in educational institutions. In the face of cross-cultural applications of constructs such as school climate, validity and reliability evidence provide important sources of information to consider the suitability of use with a particular group.
Validity and Reliability Evidence as Indicators of Applicability to Spanish Speakers
Five types of validity evidence can support evaluating the applicability of a measure for any population of students. First, content-related evidence is established when assessment developers align items with operational definitions of a construct such as school climate (Lambie, Blount, & Mullen, 2017). Next, evidence of response processes is demonstrated as participants affirm understanding item purpose, intent, and meaning in ways consistent with the intended response patterns (Peterson, Peterson, & Powell, 2017). When measurement developers identify factors among a set of items, patterns of relatedness are found to create labels for the factor (Watson, 2017). The newly labeled factors are now defined subsections of the measure. Relations with other variables account for convergent and divergent relationships contribute to the assessment criteria (Balkin, 2017). Convergent validity will demonstrate a construct is well-defined in a measure when comparing scoring trends to similar or related constructs. The final evidence of validity is reporting the positive or negative impacts by comparing the measurement report with other variables, demonstrating instrument utility (Spurgeon, 2017).
Consistent precision of a measurement is essential to score interpretations and provides measurement fidelity (Bardhoshi & Erford, 2017). Furthermore, the reliability of an instrument is representative for the population in the original development of the instrument. A measure of reliability evidence is provided by the degree items of an assessment are consistently related to each other. Available reports include the split-half reliability, Kuder-Richardson Formula 20, and coefficient alpha (Bardhoshi & Erford, 2017). Cronbach’s coefficient alpha estimates the split-half reliability by providing a score between zero and one (Bardhoshi & Erford, 2017). Cronbach’s alpha interpretation ranges from poor reliability (0.5) to excellent reliability (0.90). Psychometric properties including reliability techniques increase assurance and context for measurement use, therefore an essential inclusion in measurement evaluation (Bardhoshi & Erford, 2017).
Together, the types of validity and reliability evidence build the prominence and distinctive characteristics of a measure. However, when measures are applied to different populations, researchers are advised to consider relevance of construct definition to the new population (Lenz, Gómez Soler, Dell’Aquilla, & Uribe, 2017). Although an instrument is valid among an English-speaking population, it does not guarantee the same validity findings would be relatable to a Spanish-speaking population. The process of determining psychometric properties is performed by population.
Measuring School Climate on Spanish-Speaking Campuses
The safety perceptions and social-emotional development emphasis of students make school climate a national priority across campuses. Lopez Martin and Garcia Raga (2006) credited positive coexistence in schools to developing responsible, capable individuals providing to an active society. School climate has become so essential into school performances, that Spain and Latin American communities struggled to find a translation to fit their personal belief of how to integrate the construct. The resulting word of coexistence helped explain how students must learn to work together with peers and professionals through interpersonal skills, contributing to a supportive network, and creating a safe environment among each other. While English-driven measures and Spanish-driven measures compare their perceptions of school climate, it is clear that school climate is a topic of interest among many cultures and countries. Therefore, school climate measures become crucial proactive and reactive resource tools to ensure positive academic, social, and emotional well-being student growth. Specifically, measures that offer a Spanish option appropriate for Latinx/Hispanic students will provide clear perceptions (Lopez, Bilbao, Ascorra, Diez, & Morales, 2014). The systematic review of available Spanish school climate measures breaks down the information to determine relevance and credibility for individual schools.
Purpose of the Study and Research Questions
Since students spend the majority of their growth in an education setting, stakeholders can speculate schools have a significant influence on student growth development. When researchers provide school stakeholders with accurate assessments of the school climate perceptions among the student population, active support can then be implemented to foster healthy personal, social, and academic development. There are scarce school climate measures whose target population includes Spanish-speaking students. Moreover, the lack of a systematic review of psychometric information regarding the Spanish-adapted measures leaves school administrators without resources to make an informed test selection for their student population. The purpose of this study was to create a systematic review that highlights the school climate measures developed or adapted for a Spanish-speaking population. The study was guided by the following research questions
Research Question 1: What contemporary Spanish-language school climate measures are available?
Research Question 2: What are the characteristics of the Spanish-language school climate measures?
Research Question 3: What reliability and validity evidence was reported for the initial Spanish-language school climate measure adaptation?
Method
We used a systematic approach to identify, code, and evaluate school climate measures used in a Spanish-language population. I chose an objective and parsimonious strategy to display the information of measure characteristics and reliability and validity evidence.
Inclusion and Exclusion Criteria
The inclusion criteria consisted of the following requirements: (a) the target population was Spanish-speaking students in a K-12 setting; (b) the article included psychometric properties of the adapted or initially developed school climate measure; (c) researchers published the article in Spanish or English; and (d) published in a peer-reviewed journal. We excluded studies if the measure did not focus on the assessment of students and, specifically, school climate.
Search Strategies
We applied two search strategies: (1) electronic database searches and (2) reviewed reference lists of eligible articles. We searched the databases Psych INFO, ERIC, Academic Search Complete, Campbell Collaboration, and Google Scholar with a 25-year range period, 1993-2017. The keywords used in the search process involved “Medida de Clima Escolar” and “Escala de Clima Escolar.” Documents were first identified and separated according to the information presented in their title and abstract review. We saved the articles in Portable Document Format (PDF) files with first author and year labels. Finally, we reviewed eligible articles to evaluate the development and psychometric information further and review referenced lists for additional articles.
Data Extraction and Coding
The first author was trained by the third author in the methods of search and coding articles for systematic reviews. The purpose of the article was reviewed to check if it was the first time the psychometric properties of the instrument were being identified through a Latina/o population. If the first criterion was met, information of the instrument was gathered such as number of items, school climate subsections, sample characteristics, method of item development, and response format.
Next a table was created with columns that accounted for Cronbach’s alpha and the five types of validity evidence. The table allowed for authors to quickly view a summary of the psychometric properties presented in each article. The first author reviewed the item development for evidence of test content and the response process. Evidence for test content was met when the instrument developers interviewed students asking their definition of school climate, or if the instrument was translated. Evidence of response process was marked if developers interviewed students while they were completing the measure. Evidence for internal structure was sought throughout the method section of the article. It was marked if developers identified structure through the use of exploratory or confirmatory factor analysis. Evidence of relations with other variables and consequences of testing was found in the method sections when developers used convergent or divergent validity and compared instrument results to academic outcomes. Cronbach’s alpha was found in the results section of the article and recorded in the table. If the first other was confused as to which evidence of validity was presented, a discussion and negotiation between all authors was conducted to determine the inclusion of the evidence.
Results
The search results yielded seven potential articles that held significance regarding the inclusion criteria and warranted further review. After applying the inclusion/exclusion criteria, four documents met the requirements (See Table 1). The four documents included (a) Cuestionario Para Evaluar el Clima Escolar (CECSCE; School Climate Evaluation Questionnaire) (Trianes, Blanca, de la Morena, Infante, & Raya, 2006); (b) Escala de Clima Social Escolar (ECLIS; Social School Climate Scale) (Aron, Milicic, & Armijo, 2012); (c) Escala de Clima Escolar (School Climate Scale) (Lopez et al., 2014); and (d) Escala de Convivencia Escolar (ECE; School Coexistence Scale) (Del Rey, Casas, & Ortega-Ruiz, 2017). There were 9,236 students across the four studies with a mean sample size of 2,309 students. The gender statistics reported across the studies were boys (n = 4,944; 53.5%) and girls (n = 4,556; 49.3%).
Description of Studies and Measures Included in Review of School Climate Literature.
Psychometric Validity and Reliability Evidence
After organizing the psychometric information of the studies, a preference for validity evidence became apparent when adapting school climate measures to a Spanish-speaking population (see Table 2). We organized the types of validity evidence into categories of evidence reported in all studies (Tier 1), evidence reported in one of the studies (Tier 2), and evidence that was not reported (Tier 3). From Tier 1, all included studies (j = 4) reported evidence for Test Content and Internal Structure. Two of the studies (j = 2; 50%) justified the content of the measure by adopting an English version, one study (25%) relied on the findings from literature and work experience of the authors, and the final study (25%) developed the measure by incorporating different structures from previous instruments. Moreover, since the authors relied on either adapting an English version or using material from separate instruments, the second-most favored evidence for validity involved demonstrating the interrelation between items and confirm structural dimensions.
Psychometric Characteristics of Measures Included in Review of School Climate Literature.
In Tier 2, only one (25%) of the four studies identified evidence for relations with other variables. Trianes et al. (2006) accomplished the third evidence for validity by conducting convergent validity between the school climate measure and the Cuestionario de Violencia Escolar (School Violence Questionnaire). In Tier 3, none of the studies accounted for evidence related to response processes and evidence based on consequences of testing. Across the studies, reliability coefficients were reported and ranged between 0.62 and 0.94, signifying that one measure held excellent reliability while others held poor reliability.
Compositional Characteristics of the Identified Measures
Cuestionario Para Evaluar el Clima Escolar (CECSCE)
The CECSCE (Trianes et al., 2006) is translated version of the California School Climate and Safety Survey (Furlong, Morrison, & Boles, 1991). The self-reported questionnaire is intended for adolescents (M = 15.98) which measures two general themes in school climate: el clima de escolar del centro (center school climate), and el clima escolar referente al Profesorado (the school climate regarding the teaching staff). The CECSCE 14-item assessment uses a Likert-type scale that ranges from 1 (nunca; never) to 5 (siempre; always). The higher the score, the more likely the school conveys a positive climate to the students. Within the central school climate, students are asked such as “El colegio esta muy ordenado y limpio” (The school is very organized and clean) and “Mi clase tiene un aspecto muy agradable” (my class has a friendly aspect) (Trianes et al., 2006). In the school climate regarding teaching staff, students are presented with questions that ask “Los profesores de este colegio son agradables con los estudiantes” (the professors of this school are kind to students) and “Los profesores me dicen cuando hago un buen trabajo” (the professors tell me when I did a good job) (Trianes et al., 2006). Among Chilean adolescents, the CECSCE held a Cronbach’s alpha between 0.72 and 0.77 to show good internal consistency and convergent validity with the School Social Behavior Scale (SSBS).
Escala de Clima Social Escolar (ECLIS)
The ECLIS (Aron et al., 2012) is another self-reported instrument with 82 items studied among students with a mean age of 13. Aron et al. (2012) use literature finding and personal experiences to develop the ECLIS, identifying four dimensions within the school climate structure: student social climate perceptions of professors (30 items), friends (15 items), general school satisfaction (10 items), and infrastructure satisfaction (27 items). The ECLIS also uses a Likert-type scale in which the participant rates how relatable the statement is to their experience. An example of the Perception of Professors dimension would be “Tratan a todos los igual” (They treat all students equally), and the participant will rate the statement with “Todos” (All), “La Mayoria” (Most), “Pocos” (Few), and “Ninguno” (None). In the Infrastructure and Friends category, a student would read “Mi sala de clases esta bien ilumindad” (My classroom is well lit) and “Tengo confianza en mis companeros” (I am confident in my friends), respectively, with a rating of “Siempre” (Always), “Casi Siempre” (Almost always), “Pocas veces” (Rarely), and “Nunca” (Never). Finally, the general perception of the school will have a statement such as “Tiene actividades extra-programaticas” (It has extra-curricular activities) with a rating of “Muy de acuerdo” (Strongly agree), “De Acuerdo” (Agree), “No tan de Acuerdo” (Do not agree), and “En desacuerdo” (Never). The reliability coefficient ranges from 0.62 to 0.89 which describes weak and strong reliability.
Escala de Clima Escolar
The Escala de Clima Escolar (Lopez et al., 2014) is translated from the California School Climate Inventory (Khoury-Kassabri, Benbenishty, Astor, & Zeira, 2004). The Escala de Clima Escolar had participants in sixth and eighth grade and divided school climate items into four categories in the measure: normas claras (clear rules), participacion y apoyo de profesores (professor support and participation), apoyo social (social support), and normas contra la violencia (rules against violence). The 19-item measure is an adaptation of the California School Climate Inventory Likert-type scale with a high score indicating a strong, favorable school climate. Statements from the inventory can be characterized in all four categories, showing a strong affiliation toward one category. For example, “Las reglas en mi escuela tiene benefiecious” (The rules in my school have benefits) fits all categories but best fits with “Normas” (Rules) the best and “Apoyo Social” (Social support) the second best. The Escala de Clima Escolar demonstrates strong reliability with a coefficient of 0.89.
Escala de Convivencia Escolar (ECE)
The ECE (Del Rey et al., 2017) is a self-reported instrument geared toward adolescents (M = 14.21) with 50 items that are divided into eight categories: gestion de interpersonal positive (positive interpersonal management), vicitmizacion (victimization), disruptividad (disruption), red social de iguales (peer social network), aggression (aggression), ajuste normative (normative adjustment), indiscipline (indiscipline), desidia docente (teacher neglect). The measure is in a Likert-type format requiring participants to rate the statement with the frequency of the occurrence on a scale of 0 (nunca; never) to 5 (siempre; always). In positive interpersonal management, a student would read “Hay buenas relaciones entre los profesores/as y alumnus/as” (There are good relationships between the teachers and students); victimization would include “He tenido miedo de venir a la escuela” (I have been afraid to come to school); disruption would include “Algun companero/a me ha golpeado” (Some friend had hit me); peer social network example statement would include “los alumnus nos llevamos bien” (the students get along); aggression statements would include “He golpeado a algun companero/a” (I have hit a friend); normative adjustment includes “Aprendo” (I learn); indiscipline statements include “Cuantas veces te han castigado” (How many times have you been punished); and teacher neglect example statements would be “Los profesores/as solo explican para los listos/as de la clase” (The teachers only explain to the ready students). The ECE accounted for the highest reliability coefficient of 0.94.
Discussion
The goal of this review was to identify school climate measures available to Spanish-speaking populations supported by psychometric properties that were developed between 1993 and 2017. Using the 25-year time-period and criteria, four school climate measures were identified with varying reliability coefficients and construct conceptualizations, yet similar validity evidence quotients.
Three of the four measures accounted for test content evidence by translating the original measure in Spanish. One study used the experiences of the authors and items from convergent measures as the sources of test content evidence (Aron et al., 2012). However, none of the studies demonstrated evidence of response process, creating a question as to whether the measures read in a sequentially comprehensible format (Peterson et al., 2017) in Spanish similar to the English counterparts. The gaps in evidence are a motive for the holistic consideration of the provided measures. The school measures identified subcategories including the most common; central climate (j = 3), school safety and violence (j = 4), teacher support (j = 3), and social support (j = 2). The reason for all measures including school safety and violence as a category can be attributed to the translation of “Convivencia” implying the question asking if the student can coexist in their school environment.
The measure which was not a translation from English did not separate school climate as a straightforward category, but instead created distinct construct sections, including clear rules, teacher support, peer support, and rules against violence (Lopez et al., 2014). The subcategories assist in providing a categorical list that offers an individual with a basic, yet effective, conceptualization idea of what to expect from a school climate measure. One of the four measures provided evidence of relationship with other variables (Trianes et al., 2006), leaving the other measures to provide only two validity quotients. The development and translation of the measures were completed in two different Spanish-speaking regions, Spain and Chile. Readers would consider the regional dialect and social institutional concerns among the population. Interestedly, the subcategory that both regions included all four measures, translated and originally developed, was school safety and violence, followed by overall central climate and teacher support. Lenz and Wester’s (2017) described of validation as a process of including the personal experiences as context for the practicality of the measure. The decision between measures would reflect the compatibility of construct and population characteristic representation.
The same holistic consideration is taken in measure choice when viewing the reliability evidence. The measures demonstrated reliability scores that ranged from questionable to excellent internal consistency, 0.62 to 0.94, respectively. Dunn, Baguley, and Brunsden (2014) simplified the limitation of reliability to being that the researcher will never know the precise variability of the true score and error score when calculating the reliability coefficient. In addition, the limitations when using alpha for reliability has the possibility of overstated estimations (Dunn et al., 2014). Therefore, reliability coefficients should not be the determining factor in measure choice. The false assurance counterintuitively provided by the internal consistency estimate is disputed by comparing the participant characteristics and academic setting of the measure.
Altogether, the findings included four measures directed toward a Spanish-speaking population. A collective attention to contextual characteristics, supporting validity evidence, and internal consistency estimates will support the decision-making process for local establishments. For example, United States communities that border Mexico communities and accommodate a high Latinx population may wish to have inclusive evaluations with psychometric evidence among English and Spanish-speaking populations, and will be more inclined to use the assessments that were translated from English (Del Rey et al., 2017; Trianes et al., 2006; Lopez et al., 2014). Along with regional location, the decision process will include the school climate deconstruction through subcategories. The ECE by Del Rey et al. (2017) breaks down school climate into the most categories (j = 8) in 50 items, possibly indicating a more comprehensive interpretation of school climate. However, CECSCE by Trianes et al. (2006) provides a succinct measure that includes three subcategories and 14 items. Thus, school stakeholders would consider the measure with most relevance regarding conceptualization expectation, length, psychometric evidence, and contextual properties.
Implications for School Personnel Working With Spanish-Speaking Students
The impact of our findings will affect school counselors on a local, state, and national level. At a local level, school districts are data-driven for many categories including academic success, attendance, and school incidences. Although school incidences are a grading component for school evaluations, school climate measures will add depth to campus atmosphere, interpersonal relationships, and supportive expectations. The American School Counselor Association (ASCA) encourages school counselors to aggregate school data and determine the campus needs. A school climate assessment will provide quantifiable evidence for implementing supportive, preventive, and reactive programs that will benefit overall campus progress.
The distribution of information will make educators cognizant of disparities in the classroom. Therefore, results of school climate measures will encourage educators to adjust classroom dynamics. The new classroom dynamics will create a paradigm of relationship building skills among students, focusing on inclusivity of the Latina/o students. The shift can influence displays, such as anchor charts and manipulatives, throughout the campus, creating a welcoming environment and physical structure for Latina/o students.
Administrators’ goals will be broadened to include social-emotional growth among student rather than purely considering academic outcome. They will be required to invest in more interactions with students individually and in groups. Conversations hosting topics of diversity and inclusion will integrate into school curriculums. As a result, administrators will inherit more accountability of holistic student growth across diverse campus populations.
Furthermore, schools have become accustomed to advertising to communities to increase registration and attendance for funding purposes. Advertisements and presentations supported by school climate measures will provide a depiction of the positive school atmosphere and support provided in the district. The categorical breakdown of school climate will advise parents of how well current students feel in peer support, instructor support, school safety, and central school climate. Since the measures are intended for Spanish-speaking populations, the presentations would assure Latinx parents their child will be in an environment that is supportive of a diverse student demographic. Community leaders will be inclined to contribute and support a platform in which students are learning acceptance, rapport, and positive interactions.
As school districts utilize the school climate measures to gauge campus atmosphere, the data will lend opportunity to identify commonalities that may be addressed for Spanish-speaking students. A Spanish measure will help Spanish-speaking students become more engaged and confident in their evaluation and responses for school climate. School counselors will use the data to advocate for students on state and national level. The quantitative data provided with personal experiences will enforce a well-supported voice in legislation for inclusive, cooperative environments.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
