Development and Initial Validation of the Cultural Match Scale for Higher Education Settings

Abstract

Cultural mismatch theory of inequality and associated research indicates that a mismatch between independent cultural norms of institutions of higher education and students’ interdependent backgrounds account for disparities in academic achievement for students from historically underrepresented groups. Despite growing interest in interventions to reduce cultural mismatch, there was not yet a comprehensive measure of cultural match. We developed and validated the Cultural Match Scale (CMS) composed of 24 Likert-scale items. The CMS measures Navigation of Institutional Culture, Connection to Community, and Connection to Institutional Values. Data from 858 participants were utilized to obtain validity and reliability evidence. Three factors with a general method factor for all negatively worded items showed the best fit. Consistent with the prior findings, relationships with four criterion measures showed medium to large magnitudes of correlations. Reliability was acceptable to excellent levels. The use of the CSM was discussed along with the needs for future research.

Keywords

cultural mismatch sense of fit higher education validation

Introduction

Cultural mismatch theory of inequality describes how, when the cultural norms of an institution fail to match the norms prevalent among those underrepresented in those institutions, individuals can experience a sense of cultural mismatch that can negatively impact their sense of self, social identity, relations to others, access to opportunity within institutions, and performance (Stephens et al., 2019). A cultural mismatch with one’s institutions can result in feelings of distress, uncertainty, and isolation, while cultural match is defined by easy navigation of the institutional culture, feeling confident and competent, and ultimately results in greater success (Stephens et al., 2012). Research applying cultural mismatch theory to higher education settings indicates that mismatch between students’ culture and the institutional culture may be responsible for disparities in academic achievement and persistence, especially for first-generation students and those from historically underrepresented groups (HUGs; Harackiewicz et al., 2013; Hecht et al., 2021, Stephens et al., 2012; 2014; Tibbets et al., 2016).

Independent cultural norms tend to dominate U.S. higher education institutions (Markus & Kitayama, 1991; Stephens et al., 2012), which tend to match those of white, middle- and upper-class Americans (Markus, 2017). When these institutions promote mainstream, independent cultural norms, they often exclude interdependent norms more characteristic of underrepresented groups, like first-generation college students, individuals from working class backgrounds, and racial and ethnic minorities (Markus, 2017; Markus & Kitayama, 1991; Phinney et al., 2006; Stephens & Townsend, 2015). This mismatch creates barriers for HUGs that can perpetuate inequality within organizations (Stephens & Townsend, 2015).

Research on cultural mismatch in first generation and underrepresented students found significantly greater endorsement of interdependent motives for attending college and fewer independent motives than continuing generation students, creating a sense of mismatch (Phinney et al., 2006; Stephens et al., 2012). These interdependent motives for attending college negatively correlated with grades, academic belonging, and perceived confidence (Harackiewicz et al., 2013; Hecht et al., 2021; Stephens et al., 2012; Tibbetts et al., 2016).

Working within the theoretical framework of cultural mismatch, successful interventions have been developed to increase students’ sense of match. Results from these studies showed that these interventions reduced disparities between continuing and first-generation students on academic persistence and performance (Harackiewicz et al., 2013; Stephens et al., 2012, 2014; Tibbets et al., 2016; Townsend et al., 2019), reduced stress and anxiety (Stephens et al., 2015), and increased adjustment to college life, and academic and social engagement (Stephens et al., 2014).

With increasing interest in interventions addressing cultural mismatch in college students, it is surprising that a comprehensive measure designed to directly assess this has not yet been developed. That is, previous studies indirectly measured cultural mismatch through auxiliary covariates such as motives for attending college, sense of belonging, social support, self-efficacy, and self-construal (e.g., Harackiewicz et al., 2013; Stephens et al., 2014; Stephens et al., 2015; Tibbets et al., 2016; Townsend et al., 2019; Walton & Cohen, 2007), instead of using a measure representing theoretical constructs of cultural mismatch.

Prior Studies Using Indirect Measurement of Cultural Mismatch

Walton and Cohen (2007) demonstrated that interventions targeting students’ belonging at their university and in their academic field increased sense of fit, defined as social belonging, among Black students. Predicting belonging uncertainty due to historical underrepresentation and stereotype threat, they found that Black participants questioning their social connectedness to their field had a lower sense of fit in computer science. Conversely, students encouraged to interpret concerns about belonging as unrelated to their social identity evaluated their fit more positively.

Stephens et al. (2012, 2014) extended the work on social identity, addressing cultural mismatch experienced as a result of social class defined by first-generation status. They developed a difference education intervention comprised of a student panel highlighting success from diverse social-class backgrounds. Townsend et al. (2019) adapted this difference education intervention for online education. They found that it enhanced first-generation students’ comfort in interpersonal interactions on campus and marginally improved their sense of fit. Again, sense of fit in this study focused only on students’ sense of social belonging.

Building on prior research into cultural mismatch, Phillips et al. (2020) developed a 12-item measure of Subjective Sense of Fit; however, it captured only the aspects of students’ comfort and shared values with their institution. They found that students’ endorsement of interdependent motives for attending college corresponded to lower subjective sense of fit.

Similarly, Tibbets et al. (2016) developed a measure of fit. This measure has only two items focused on students’ values and goals aligning with those of their university (value affirmation). They found that an interdependent value affirmation intervention resulted in first-generation students perceiving greater fit than continuing generation counterparts. Using this measure, Hecht et al. (2021) found that first-generation students reported lower levels of fit than continuing generation students.

The available measures explained above only partially capture the theoretical definition of cultural mismatch with a relatively small number of items. Thus, the goal of this study was to develop a measure which directly evaluates cultural mismatch based on the theoretical domains of cultural mismatch in the literature.

This study was part of the National Institutes of Health (NIH) Diversity Program Consortium (DPC) Dissemination and Translation Awards (DaTA) initiative designed to advance understanding of individual and institutional factors affecting training experiences and career development for biomedical researchers from a wide variety of backgrounds.

In this article, we explained the theoretical constructs employed to develop items in the Cultural Match Scale (CMS) and provided reliability and validity evidence obtained through two phases. A total of four criterion measures from prior studies investigating cultural match/mismatch were utilized for their similarity in constructs and partial overlap with aspects of cultural match/mismatch in a higher education setting. As part of the validation process, hypotheses consistent with past research were also tested; it was predicted that because universities are more independent in nature, students who are first-generation and/or from HUGs who are more interdependent in their self-construal would have lower correlations with cultural match scores, indicating lower feeling of cultural match (i.e., cultural mismatch).

Method

Participants

Data were collected in two phases: pilot and validation. Participants were recruited from upper division biomedical courses offered in a Hispanic Serving Institution (HSI) located in Southern California. Participants volunteered to complete either all measures or participate in a qualitative interview about a targeted selection of the items for extra credit. Cognitive interviews were conducted during the pilot phase and interview data were used with the quantitative data for item revision.

During the pilot phase (from March to April 2021) a total of 687 potential participants were invited to participate. Of those, 334 students participated in the survey (48.6% response rate), and 17 participated in cognitive interviews. During the validation phase (from September to October 2021) 1054 potential participants were invited to participate. Of those, 525 students participated in the survey (49.8% response rate). The demographics of the participants, as presented in Table 1, closely mirrored the composition of the student population at the campus where the data were collected, specifically concerning three specifications that the current NIH grant project focused on; approximately 56% of students were Hispanic, 61% were first-generation students, and 50% were Pell-Grant eligible.

Table 1.

Demographic Information.

	Pilot Phase (n = 334)		Validation Phase (n = 525)
	Scale response data (n = 317)	Interview data (n = 17)	Scale response data (n = 525)
Gender	Man (23.0%)	Man (35.3%)	Man (26.7%)
	Woman (74.8%)	Woman (64.7%)	Woman (71.2%)
	Other (1.3%)		Other (0.4%)
	Not to disclose (0.6%)		Not to disclose (1.3%)
	Missing (0.3%)		Missing (0.4%)
Race/Ethnicity	American Indian/Alaska Native (2.3%)	American Indian/Alaska Native (5.9%)	American Indian/Alaska native (1.3%)
	Asian (11.7%)	Asian (11.8%)	Asian (12.6%)
	Black/African American (3.8%)	Hispanic/Latino[a] (47.1%)	Black/African American (3.4%)
	Hispanic/Latino[a] (62.8%)	White (47.1%)	Hispanic/Latino[a] (57.3%)
	Native Hawaiian/Other Pacific Islander (1.3%)	Other (2.0%)	Native Hawaiian/Other pacific Islander (0.4%)
	White (29.3%)		White (37.3%)
	Other (1.6%)		Other (1.7%)
	Not to disclose (1.6%)		Not to disclose (1.9%)
Major	Applied physics (0.3%)	Biology (29.4%)	Applied physics (1.7%)
	Biology (28.7%)	Chemistry (5.9%)	Biology (20.4%)
	Chemistry (2.2%)	Health science (35.3%)	Chemistry (2.5%)
	Health science (23.3%)	Psychology (29.4%)	Health science (20.8%)
	Nursing (10.7%)		Nursing (1.5%)
	Psychology (24.3%)		Psychology (35.2%)
	Others (10.2%)		Others (17.5%)
	Missing (0.3%)		Missing (0.4%)
Number of semesters	2^nd (28.7%)	2^nd (17.6%)	3^rd (29.3%)
	4^th (29.7%)	4^th (35.3%)	5^th (20.8%)
		8^th (17.6%)
Age	M = 22.2, SD = 3.6	M = 24.7, SD = 3.8	M = 23.5, SD = 5.9
GPA (cumulative)	M = 3.3, SD = 0.4	M = 3.1, SD = 0.4	M = 3.3, SD = 0.5
Transfer student	48.6%	82.4%	56.0%
First generation	61.5%	70.6%	63.8%
Pell grant	51.4%	52.9%	43.0%

The sample size for the pilot phase was determined based on the minimum sample size for confirmatory factor analyses recommended in Wolf et al. (2013).¹ For the validation phase, the minimum total sample size of 500 was set based on the recommendations in the literature (Comrey & Lee, 1992). A power analysis was also run to determine the minimum sample size for underrepresented groups (e.g., Hispanic and first-generation students). Using a correlation of r = .2, the minimum sample size of 260 for underrepresented groups was established. A total of 51 participants who did not respond to any CMS questions were excluded from analyses. Participants with incomplete CMS responses were utilized for the estimation of latent factor scores.

Cultural Match Scale

The development of the CMS began with a thorough review of the current theoretical and empirical literature to create a list of the important aspects of cultural match. Cultural mismatch (Stephens et al., 2012) and related constructs, including cultural discontinuity (Tyler et al., 2008), defined as occurring when the behavior a student exhibits in their home environment varies widely from the behavior they exhibit to fit in at school. We initially identified six key areas: (1) understanding expectations (Stephens et al., 2012); (2) understanding a path to achievement (Stephens et al., 2014); (3) ease of navigation of institutional culture (Stephens et al., 2012); (4) sense of shared experience (Walton & Cohen, 2007); (5) connection to university community (Jury et al., 2017; Walton & Cohen, 2007); (6) and sense of shared values (Stephens et al., 2012) (refer to the theoretical definitions of each area under Figure A1 in the Supplemental Materials).

Subsequently, four experts in education, psychology, and psychometrics reviewed existing items and developed new or modified items pertinent to these six areas. The initial review of these items revealed that (1) the first two areas (understanding expectations and understanding a path to achievement) could be subsumed under the third area (ease of navigation of institutional culture), resulting in the consolidation of these aspects into one overarching domain of Navigation of Institutional Culture and (2) the fourth and fifth areas (sense of shared experience and connection to university community) could be thematically integrated into the domain of Connection to Community. The sixth area (sense of shared values) underwent a renaming process and was identified as Connection to Institutional Values, with theoretical emphasis placed on students’ perceptions of alignment with institutional values. The first domain (F1), Navigation of Institutional Culture, includes academic preparedness and is defined by ease with understanding academic expectations, paths to achievement, and available institutional resources and support. High scores in this domain reflect students’ ease in navigating their institutional culture, and the domain is demonstrated through students utilizing campus resources, fostering mentoring relationships with professors, and displaying confidence in expressing their knowledge and opinions in classes. The second domain (F2), Connection to Community, includes a sense of belonging to the university community, common identity with other students, and representation (i.e., other students share similar cultural, racial, ethnic, and socio-economic backgrounds). Students’ feelings of connection to their community would be demonstrated in their engagement with peers, in social and academic activities on campus, and in encouraging others with similar backgrounds to attend their institution. The third domain (F3), Connection to Institutional Values, captures the sense of match experienced when one’s own values correspond to those expressed by the university through campus materials, administrators, and professors. Students with a high connection to institutional values would demonstrate this through espousing and taking pride in the attitudes and values of their university. Figure A1 in the Supplemental Materials illustrates the conceptual connections between the initial six areas and the three domains. The CMS is composed of 24 Likert-scale items with a five-option rating scale ranging from 1 (strongly disagree) to 5 (strongly agree). The three domains have 12, 7, and 5 items, respectively.

Criterion Measures

Four scales were employed as criterion measures for the validation of CMS: Comfort in Interactions (Townsend et al., 2019), Social fit (Stephens et al., 2014; Townsend et al., 2019), Sense of Social Fit (Walton & Cohen, 2007), and Empowerment (Townsend et al., 2019).

Comfort in Interactions (Townsend et al., 2019) measures whether students feel comfortable engaging in interpersonal behaviors in college settings (7 items; α = .82) based on a seven-point Likert scale from 1 (strongly disagree) to 7 (strongly agree). It was developed to capture the aspect of students’ sense of cultural match that corresponds to feelings of comfort and ease with common social interactions in college, like comfort sharing opinions with other students. Pilot testing revealed strong correlations between first-generation status and comfort in interactions, and interventions designed to increase cultural match were shown to increase scores on this measure among first-generation students (Townsend et al., 2019).

Walton and Cohen’s (2007) Sense of Social Fit measure assesses the degree to which students feel they belong or fit in socially at their university (17 items, α = .89). This scale uses a five-point Likert scale from 1 (strongly agree) to 5 (strongly disagree). The original scale measured fit within a computer science major but was adapted to refer to school broadly. Walton and Cohen (2007) used prior research to establish five motivational constructs, one of which was social fit. A series of factor analyses yielded a final measure of 17 items assessing the social fit subscale. The authors cited further validation of the measure in an unpublished manuscript (Walton & Cohen, 2005).

Social fit (Stephens et al., 2014; Townsend et al., 2019) was adapted from Walton and Cohen’s (2007) 17-item measure but is comprised of six unique items (α = .86) pilot tested and revised to capture the experience of feeling part of the larger university and academic community. Items are scored on a seven-point Likert scale from 1 (strongly disagree) to 7 (strongly agree). Factor analysis confirmed that all items loaded onto a single factor (Stephens et al., 2014). Social fit was shown to increase among first-generation students following an intervention aimed at increasing cultural match (Townsend et al., 2019).

Empowerment (Townsend et al., 2019) includes eight items from three measures that assess the entire construct of students’ academic empowerment (perceived preparation, academic efficacy, and learner empowerment). Items are scored on a seven-point Likert scale from 1 (strongly disagree) to 7 (strongly agree) and the alpha coefficients ranged from .84 to .89 (Townsend et al., 2019). Two items assessing perceived preparation were adapted from the measure used by Stephens et al. (2014). Three items were taken from the academic efficacy subscale of the Patterns of Adaptive Learning Scales (PALS; Midgley et al., 2000), which measures students’ perceptions of their competence in their schoolwork. The PALS is widely used in assessment of elementary and secondary education students, has been used in samples of university students, has strong psychometric properties, and has demonstrated strong predictive and concurrent validity (Anderman et al., 2005; Midgley et al., 1993, 2000). The last three items were taken from the Learner Empowerment Measure (LEM; Frymier et al., 1996), which measures students’ feelings of autonomy and agency over their learning behaviors. The 30-item LEM was assessed for construct validity with college students and was found to be a reliable and valid measure (Frymier et al., 1996).

In addition to the use of four criterion measures, for additional validity evidence, the relationships between CMS scores and three background variables (gender, ethnicity, first-generation status) were examined to test whether CMS scores show the consistent findings from past research; because universities are more independent in nature, we hypothesized that HUGs/and first-generation status would be negatively correlated with CMS scores, indicating lower feeling of cultural match.

Procedure

After providing consent, participants in the survey portion of the pilot and validation studies completed a series of measures on Qualtrics. These included demographic questions and psychological measures including the CMS and the criterion measures.

Data Analysis

Confirmatory factor analyses (CFAs) were conducted to examine the factor structure of the CMS. The comparative fit index (CFI), Tucker-Lewis index (TLI), root mean square error of approximation (RMSEA), and the standardized root mean square residual (SRMR)² were used to evaluate global model fit. Relative model fit was evaluated through a deviance test for nested models using the DIFFTEST function in Mplus (Muthén & Muthén, 1998-2017). For nonnested model comparisons, Bayesian information criterion (BIC) values were employed along with the criteria from Raftery (1995). In a relative model fit comparison for two nonnested models, a model with a BIC value smaller by 10 or more indicates a better fit compared to the other model with a larger BIC value. To obtain chi-square-based model fit indices, the weighted least squares means and variance adjusted (WLSMV) estimator was employed, and the maximum likelihood (ML) estimator was employed for loglikelihood-based indices including BIC values.

Omega and alpha³ coefficients were employed for reliability coefficients. An omega coefficient is an indicator of reliability especially when item loadings are varying and there is a possibility of correlated item errors. For the estimation of omega coefficients, robust maximum likelihood (MLR) estimators were utilized.

To examine group equivalence, correlations between three CMS domain scores (F1, F2, and F3) and four criterion measure scores were compared among subgroups. Three group memberships considered were gender (male vs. female), Hispanic (Hispanic vs. non-Hispanic), and first-generation status (first-generation vs. non-first generation). Bartram (2013) recommended this correlational approach over item-level approaches (e.g., differential item functioning analysis and measurement invariance test) for two reasons; (1) item-level approaches cannot reveal any group-related systematic bias (e.g., acquiescence effects) and (2) group equivalence can still be obtained when item-level invariance was random. Thus, this study investigated whether the correlations between CMS scores and four criterion scores were equivalent among the three group memberships. To compare correlations and obtain mean correlations, Fisher’s z transformation was employed. To examine relationships with criterion measures, structural equation models (SEMs) were utilized. Two different scoring methods, factor analysis (FA) and classical test theory (CTT; summing item scores) approaches, were used. R (R Core Team, 2022) and Mplus (Muthén & Muthén, 1998-2017) were used for all data analyses. All evidence from the validation phase were presented based on the frameworks in Skaggs (2022) which emphasize approaches from Messick (1989, 1995) and the Standards for Educational and Psychological Testing (American Educational Research Association [AERA] et al., 2014).

Results

Pilot Phase

Factor Structure

Using the response data from 316 respondents, three models (1-, 2-, and 3-factor structures) were compared to determine the most appropriate factor structure for the CMS based on the theoretical framework; a 1-factor model was tested to check whether unidimensional CMS scores can be utilized; a 2-factor model was tested to examine whether the second (Connection to Community) and third (Connection to Institutional Values) domains can be combined into the broader domain of Connection; and a 3-factor model was tested to check whether the theoretical structure composed of three domains fit the data well. All factors were correlated in the 2- and 3-factor models. Among the three models, the 3-factor model showed an acceptable fit with the CFI value of .90 (Bentler, 1990) and the TLI value of .89 was close to the Bentler and Bonnett’s (1980) recommendation of .90. The SRMS value was .07, which met the criterion value of less than .08 (Hu & Bentler, 1999). However, the RMSEA value (.11, 90% CI: .10–.11) was larger than the acceptable cutoff of .08 (Browne & Cudeck, 1992). The deviance test⁴ results using the DIFFTEST function in Mplus showed that the 1- and 2-factor models fit significantly worse than the 3-factor model, respectively ( $Χ^{2} (3) = 164.71, p < . 001; Χ^{2} (2) = 44.20, p < . 001)$ .To check unidimensionality in each factor, a separate model fit for each factor in the 3-factor model was also assessed. CFI values showed that the first factor (F1: Navigation of Institutional Culture) and the third factor (F3: Connection to Institutional Values) were unidimensional (CFI_F1 = .95, TLI_F1 = .94, SRMR _F1 = .05, CFI_F3 = .99, TLI_F3 = .98, SRMR _F3 = .02). The second factor was close to acceptable fit (F2: Connection to Community, CFI_F2 = .87, TLI_F2 = .81, SRMR _F2 = .08). All RMSEA values were above the cutoff of .08.

Reliability

Omega coefficients, instead of Cronbach’s alpha, were computed for each factor because the 3-factor model fit the data best and the assumption of tau equivalence was not met. The MLR estimator was employed to obtain error variance of each item to compute omega coefficients. The omega coefficients were .88, .81, and .60 for F1, F2, and F3, respectively.

Cognitive Interviews

A total of 17 cognitive interviews were conducted. Probing questions were generated to investigate participants’ item comprehension, interpretation, and the information retrieved to answer CMS items. The pilot version of the CMS was composed of 23 Likert-scale items with five options from 1 (strongly disagree) to 5 (strongly agree). Table 2 summarized the changes made based on item loading values and cognitive interview results; five items were revised, three items were removed, and four new items were added into the final version of the CMS.

Table 2.

Item Revision.

	Pilot version	Final version
Number of items
F1	11	12
F2	8	7
F3	4	5
Item revision
F1_item 4	I feel like my career goals do not fit with the goals of (name of institution).	I feel like my career goals match what (name of institution) expects from its students.
F1_item 5	I feel that the students at (name of institution) receive information about the goals they should attain.	Removed
F1_item 9	I feel like succeeding at (name of institution) overall comes naturally to me.	I feel like my past experiences have prepared me to succeed at (name of institution).
F1_item 11		[New] I regularly think about leaving (name of institution) or transferring to another university.
F1_item 12		[New] I feel like I have to change myself in order to be successful at (name of institution).
F2_item 1	I feel like I experience challenges at (name of institution) that other students do not experience.	I experience academic challenges that other (name of institution) students do not experience because of my background.
F2_item 4	I Feel very isolated at (name of institution).	Removed
F2_item 6	I feel very different from most other students at (name of institution).	I feel like I am treated differently from other students at (name of institution).
F3_item 2	I feel that I value the same things that (name of institution) values based on my experience of (name of institution) events and affiliated centers (e.g., center names).	I feel like I value the same things that (name of institution) values based on my experiences of (name of institution) events and affiliated centers.
F3_item 3	I Feel that I have to act very differently from the way I act at home in order to be successful at (name of institution).	Removed
F3_item 5		[New] I feel that my values are represented at (name of institution).
F3_item 6		[New] I feel like the things that are important to me are also important to (name of institution).

Items Removed

Two items with relatively low loadings were removed (F1_item 5 = 0.41, F3_item 3 = 0.30). An additional two items, F2_item 4 and F3_item 3, from the pilot version were removed based on responses from the cognitive interviews. Because data collection occurred during the COVID-19 pandemic, the feeling of isolation that F2_item 4 measured was interpreted in the context of social isolation due to the pandemic, not the target construct related to feeling isolated in relation to one’s university culture. F3_item 3 was also removed because most participants (11 out of 17 interview participants) interpreted “act very differentially from the way I act at home in order to be successful at (name of institution)” as being more professional on campus, rather than understanding it as the intended construct of cultural match (e.g., change their personality, identity, or culture). The intended construct is, for example, if one is experiencing mismatch, they may need to adopt a different set of norms for their university setting versus their home environment.

Items Revised

There was one item showing a low loading value (F2_item 1 = 0.32). After a content expert’s review, this item was decided to be revised (Table 2) and included for the validation phase. Differential interpretations were also found during the cognitive interviews for five additional items, and so these were also revised. Interpretation of the phrase “goals of (name of institution)” from F1_item 4 varied (e.g., educational goals and cultural inclusiveness), and so was revised to “what (name of institution) expects from its students” to make the content more student-centered. The interpretation of “challenges” in F2_item 1 was broader than the intended domain of academic challenges. Only three interview participants interpreted this as academic or learning difficulty, whereas 14 participants connected this to personal issues such as financial difficulties, discrimination, or COVID-19 related challenges, so this question was revised with more detail. The interpretation of the phrase “different from most other students” from F2_item 6 was also confusing to participants. A total of 10 participants stated that every individual is different, or they are quite similar because they are all college students. So, the item was revised as “I am treated differently from other students at (name of institution)” to ask about whether students perceive any differences in treatment. In F1_item 9, “comes naturally to me” meant different things to different students, so to better relate this item to the area of comfort with institution, the additional phrase, “past experience” was included in the revised item. Finally, the example provided for affiliated centers in F3_item 2 was removed because it led to a limited scope in retrieving information because of the specific example included.

New Items

After evaluating the items based on quantitative and qualitative analyses, to ensure that all target aspects of cultural match were represented, four additional items, F1_item 11, F1_item 12, F3_item 5, and F3_item 6, were included (Table 2). All 24 items from the final version were included in Table A1 (Item Map in the Supplemental Materials).

Validation Phase

Factor Structure

Separate model fit for each factor was evaluated to check unidimensionality before assessing global model fit for all models. The unidimensional model fit was good for F1; however, it was poor for F2 due to negatively worded items. The modification indices showed that the five negatively worded items were highly correlated; for example, the expected change in chi-square value was 384.93 if a residual correlation between F2_item 4 and F2_item 6 was added. Therefore, a general method factor was considered for further model comparisons, introduced in models M2-m and M3-m. This method factor, which all five negatively worded items are loaded on, is not correlated with the other three substantive factors. It is not uncommon to consider a general method factor for negatively worded items as described in the methodology literature (e.g., DiStefano & Motl, 2006; Maydeu-Olivares & Coffman, 2006). Table 3 displays all six models considered to determine the factor structure of CMS. As explained in the pilot results, the 2-factor model (M2) where F2 and F3 were combined into one domain of Connection was also considered in the validation phase. All three factors (F1, F2, and F3) were correlated both in M2, M3, M2-m, and M3-m. However, the general method factor was not correlated with the three factors in M2-m, and M3-m (see Figure 1 for the specification of M3-m).

Table 3.

Model Fit.

Model	Chi-square	df	CFI	TLI	RMSEA (90% CI)	SRMR	BIC	ΔBIC
3-factor model: Unidimensionality
- F1	1022.67	54	.94	.92	.18 (.17–.19)	.06
- F2	749.50	14	.76	.65	.31 (.29–.33)	.15
- F3	216.61	5	.98	.95	.27 (.25–.31)	.04
M1: 1-factor model	4510.78	252	.81	.79	.18 (.17–.18)	.09	27,617.06	1374.40
M2: 2-factor model	3397.08	251	.86	.84	.15 (.15–.16)	.08	26,897.60	654.94
M3: 3-factor model	3069.16	249	.87	.86	.14 (.13–.15)	.08	26,782.38	539.72
M1-m: 1-factor model with a method factor	2775.31	247	.89	.87	.14 (.13–.14)	.07	27,058.53	815.87
M2-m: 2-factor model with a method factor	1574.21	246	.94	.93	.10 (.09–.10)	.06	26,335.74	93.08
M3-m: 3-factor model with a method factor	1458.03	244	.95	.94	.10 (.09–.10)	.05	26,242.66

Note. ΔBIC values are the differences between M3-m and the corresponding models.

Figure 1.

M3-m Model with Factor Loadings. Note. F1 = Navigation of Institutional Culture, F2 = Connection to Community, F3 = Connection to Institutional Values, F_m = General Method Factor (for negatively worded items). Unstandardized factor loadings along with three factor correlations are provided.

Based on the CFI, SRMR (Hu & Bentler, 1999), and TLI (Bentler & Bonett, 1980) criteria, M2-m and M3-m showed good fit. Relative model fit was examined between M2-m and M3-m because the correlation between F2 and F3 was relatively high (r = .85) in M3-m results. The deviance test revealed that M2-m fit significantly worse than M3-m ( $Χ^{2} (2) = 63.23, p < . 001$ , see footnote 2 about creating a nested structure for a deviance test). Additionally, relative model fit comparisons among all nested and non-nested models were conducted using the BIC values. As displayed in Table 3, the BIC value from M3-m was the smallest, and the BIC value differences between M3-m and the other five models were larger than the criterion value of 10. Thus, M3-m was chosen as the final structure of the CMS (Table 3).

Factor loadings from the M3-m model were provided in Figure 1. All loadings were significant at p < 001. The values were obtained with the use of WLSMV estimator. Loadings from the negatively worded items (F1_item 11, F1_item 12, F2_item 1, F2_item 5, F2_item 7) were relatively low because they loaded on both their substantive factor and the method factor.

Scoring and Reliability

Considering that the CTT scoring approach (summing item scores to obtain (sub)scale scores) are prevalent in educational and psychological tests, the assumptions of tau equivalence (all factor loadings being equal) and equal residual variance (all item error variances being equal) were tested with the MLR estimator. Based on the RMSEA criterion <.08 for a reasonable fit (Browne & Cudeck, 1992, 1993), all three factors met both assumptions (RMSEA_F1 = .076 [.071 - .080], RMSEA_F2 = .071 [.066 - .076], RMSEA_F3 = .073 [.069 - .078], RMSEA_method = .080 [.076 - .085]). Thus, the CTT scoring approach can be used for F1, F2, and F3 scores. The correlations between scores from the FA-based approach and those from the CTT approach were $r_{F 1}$ = .90, $r_{F 2}$ .82, and $r_{F 3}$ .97, respectively. Regarding the relatively low correlation found in F2, we postulated the proportion of negatively worded items (43%, 3 out of 7 items) in F2 may adversely affect the relationship.

Since the assumption of tau equivalence was met, reliability analyses for both the factor-based and CTT-based approaches were conducted. The omega coefficients based on the M3-m factor structure were .88, .75, and .87. Alpha coefficients were .88, .74, and .86 for F1, F2, and F3, respectively, which met the adequate (α ≥ .70) to excellent (α > .80) levels (DeVellis & Thorpe, 2022; Nunnally 1994).

Relationships with Criterion Variables

Before examining the relationships between CMS scores and the four criterion measures, model fit of each criterion measure and reliability (omega coefficients) were examined. A one-factor model showed good fit for all four criterion measures. All model fit was acceptable based on the RMSEA (Browne & Cudeck, 1992, 1993; MacCallum et al., 1996) and SRMR criteria (Hu & Bentler, 1999), however, CFI and TLI did not show an acceptable fit (see Table A2 in the Supplemental Materials for model fit indices). CMS scores positively correlated with students’ comfort in interaction (.51–.60) and empowerment (.54–.73). The two social fit scales were highly positively correlated with CMS scores, and most highly correlated with F2: Connection to Community (.87 with Social Fit and .85 Sense of Social Fit, see Table 4).

To ensure group equivalence in relations to the four criterion measures, correlations among CMS scores (F1, F2, and F3) and the four criterion measure scores were examined across three group memberships (gender, Hispanic, and first-generation status). The mean correlations between CMS and the four criterion measure scores were .63 for male and .73 for female groups. The mean values were computed using 12 z-transformed correlation coefficients (three CMS scores $\times$ four criterion measure scores). The relationship between 12 pairs of z-transformed coefficients between the two gender groups was .95, which demonstrates that the relationships between CMS and four criterion measure scores are equivalent between the two gender groups (see Table A3 in the Supplemental Materials for all coefficient values). The same inferences can be made from the Hispanic group comparison results. The mean correlations were .71 for Hispanic and .67 for non-Hispanic groups. The relationship between 12 pairs of coefficients between the two ethnic groups was .90. However, for first- and non-first-generation groups this relationship was found to be slightly smaller, compared to the previous two groups. The correlation among the 12 pairs of coefficients was .78, with the mean correlation of .72 for first-generation and .65 for non-first-generation groups. To further examine if the 12 pairs of coefficients were significantly different between the first-and non-first-generation groups, a paired-samples t test was conducted. The mean difference of .07 was not significantly different, t (11) = 1.43, p = .18, d = 0.31. Therefore, the group equivalence between the CSM and criterion measure scores were concluded.

Regarding the relationships with three background variables, the gender variable was not correlated with CMS scores, as hypothesized. However, CMS scores showed positive correlations with the ethnicity variable (Hispanic vs. non-Hispanic). It is possible that the unique nature of the institution where this research was conducted, an HSI with a majority of students (57 %) identifying as Hispanic/Latino[a], impacted these results. Perhaps the match between students’ ethnicity and the university is more important in contributing to overall sense of match than self-construal. In addition, although first-generation status was negatively correlated with CMS scores, it was not significant (Table 4).

Table 4.

Relationships With Criterion Variables.

Measure	Reliability: Omega	Correlation
Comfort in interactions	.89	F1 = .60***
		F2 = .53***
		F3 = .51***
Social fit	.91	F1 = .70***
		F2 = .87***
		F3 = .72***
Empowerment	.89	F1 = .73***
		F2 = .55***
		F3 = .54***
Sense of social fit	.85	F1 = .63***
		F2 = .85***
		F3 = .72***
Gender		F1 = −.01 (p > .05)
		F2 = .01 (p > .05)
		F3 = .20 (p > .05)
Hispanic		F1 = .15 **
		F2 = .21 ***
		F3 = .14 **
First generation status		F1 = −.06 (p > .05)
		F2 = −.13 (p > .05)
		F3 = −.03 (p > .05)

Note. ** significant at p < .01, *** significant at p < .001.

Discussion

The current study provided reliability and validity evidence for the CMS through pilot and validation phases using data collected from 859 participants. The CMS is composed of three domains: Navigation of Institutional Culture, Connection to Community, and Connection to Institutional Values. Each domain includes 12, 7, and 5 Likert-scale items with five options from 1 (strongly disagree) to 5 (strongly agree).

Based on the guidelines from the Standards for Educational and Psychological Testing (AERA et al., 2014), evidence based on both internal structure and relations to other variables were examined. Response processes obtained through cognitive interviews were also incorporated into item revisions. The 3-factor model with a general method factor, which all five negatively worded items are loaded on, showed the best fit. Since the assumption of tau equivalence was met, summing item scores (CTT-based scoring) is acceptable for each domain. Omega and alpha reliability coefficients were from adequate to excellent levels. Relationships with four criterion measures (comfort in interaction, social fit, sense of social fit, and empowerment) were analyzed to examine whether the findings were consistent with findings in the literature. CMS scores showed medium to large magnitudes of correlations with the four criterion measures which were aligned with findings from prior research (e.g., Harackiewicz et al., 2013; Hecht et al., 2021; Stephens et al., 2012; Tibbets et al., 2016). The equivalent correlational relationships among CMS scores (F1, F2, and F3) and the four criterion measure scores across three group memberships (gender, Hispanic, and first-generation status) demonstrated the group equivalence. Overall, the validation results supported that interpretations using CMS scores are reliable and valid to represent the degree to which college students feel that the institutional culture of their university matches the culture they identify and feel comfortable with.

As mentioned earlier, the CMS was developed as part of NIH’s DaTA initiative project. The use of CSM scores was intended to assess students’ level of cultural match in a higher education setting to aid in the evaluation of interventions designed to enhance students’ success and persistence in biomedical fields. Aligned with the intended use, we believe the CMS can be a useful tool for institutions of higher education to assess students’ cultural match and develop programs and interventions to increase sense of fit or belonging, which ultimately facilitate academic success.

The cultural mismatch theory of inequality predicts that students belonging to more interdependent groups will have lower cultural match. It is surprising then that CMS scores were higher for students who identify as Hispanic than for non-Hispanic students in the current study. Past research demonstrating the independent nature of university culture (Stephens et al., 2012) sampled primarily majority white institutions and found higher rates of independence among more elite universities. Further research on the institutional culture of minority serving institutions like HSIs and Historically Black Colleges may show differences in culture when compared to these prior samples. More research on match in relation to students’ race and ethnicity may help to clarify the ways in which this aspect of identity is unique from first-generation status and social class.

Although these results provide evidence for the reliability and validity of CMS scores, this study is not without limitations. As mentioned, data was collected during the COVID-19 pandemic, when classes were almost entirely online, making it challenging to compare these results to prior studies conducted pre-COVID. Future directions for this research will include further validation of the CMS across a wider range of college students, both online and in-person. With larger samples, item response theory (IRT) models can be examined to enhance accuracy in score estimation. In light of the grant study’s specific focus on Hispanic students, group equivalence was examined across three categories of subgroups, including Hispanic versus non-Hispanic groups. However, as emphasized in the Standards for Educational and Psychological Testing (AERA et al., 2014), validation is an ongoing process. Therefore, we recommend that future studies examine measurement invariance among additional subgroups, including a more diverse range of racial/ethnic groups. We hope that future research will use the CMS to assess the effectiveness of interventions designed to enhance students’ sense of match directly and compare these changes in match to outcome variables related to academic persistence and success. This will aid in building a robust understanding of who experiences mismatch, how it impacts academic outcomes, and how it can be improved.

Supplemental Material

Supplemental Material - Development and Initial Validation of the Cultural Match Scale for Higher Education Settings

Supplemental Material for Development and Initial Validation of the Cultural Match Scale for Higher Education Settings by HyeSun Lee and Melissa Soenke in Journal of Psychoeducational Assessment

Footnotes

Acknowledgments

We would like to thank Drs. Sonsoles de Lacalle and Hugo Tapia for their insights and support in developing the Cultural Match Scale.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Research reported in this publication was supported by the National Institute of General Medical Sciences of the National Institutes of Health under Award Number U01GM138431. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

ORCID iD

HyeSun Lee

Supplemental Material

Supplemental material for this article is available online.

Notes

References

American Educational Research Association, American Psychological Association, National Council on Measurement in Education, and Joint Committee on Standards for Educational and Psychological Testing . (2014). Standards for educational and psychological testing. American Educational Research Association.

Anderman

E. M.

Urdan

Roeser

(2005). The patterns of adaptive learning survey: History, development, and psychometric properties. In Indicators of positive development conference (pp. 223–235). Springer. https://doi.org/10.1007/0-387-23823-9_14

Bartram

(2013). Scalar equivalence of OPQ32: Big Five profiles of 31 countries. Journal of Cross-Cultural Psychology, 44(1), 61–83. https://doi.org/10.1177/0022022111430258

Bentler

P. M.

(1990). Comparative fit indexes in structural models. Psychological Bulletin, 107(2), 238–246. https://doi.org/10.1037/0033-2909.107.2.238

Bentler

P. M.

Bonett

D. G.

(1980). Significance tests and goodness of fit in the analysis of covariance structures. Psychological Bulletin, 88(3), 588–606. https://doi.org/10.1037/0033-2909.88.3.588

Browne

M. W.

Cudeck

(1992). Alternative ways of assessing model fit. Sociological Methods and Research, 21(2), 230–258. https://doi.org/10.1177/0049124192021002005

Browne

M. W.

Cudeck

(1993). Alternative ways of assessing model fit. In: Bollen

Long

(Eds.), Testing structural equation models (pp. 136-162). Sage.

Comrey

A. L.

Lee

H. A.

(1992). First course in factor analysis. Lawrence Erlbaum Associates, Inc.

DeVellis

R. F.

Thorpe

C. T.

(2022). Scale development: Theory and applications (5th ed.). Sage Publications.

10.

DiStefano

Motl

R. W.

(2006). Further investigating method effects associated with negatively worded items on self-report surveys. Structural Equation Modeling: A Multidisciplinary Journal, 13(3), 440–464. https://doi.org/10.1207/s15328007sem1303_6

11.

Frymier

A. B.

Shulman

G. M.

Houser

(1996). The development of a learner empowerment measure. Communication Education, 45(3), 181–199. https://doi.org/10.1080/03634529609379048

12.

Harackiewicz

J. M.

Canning

E. A.

Tibbetts

Giffen

C. J.

Blair

S. S.

Rouse

D. I.

Hyde

J. S.

(2013). Closing the social class achievement gap for first-generation students in undergraduate biology. Journal of Educational Psychology, 106(2), 375–389. https://doi.org/10.1037/a0034679

13.

Hecht

C. A.

Priniski

S. J.

Tibbetts

Harackiewicz

J. M.

(2021). Affirming both independent and interdependent values improves achievement for all students and mitigates cultural mismatch for first-generation college students. Journal of Social Issues, 77(3), 851–887. https://doi.org/10.1111/josi.12416

14.

L. T.

Bentler

P. M.

(1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6(1), 1–55. https://doi.org/10.1080/10705519909540118

15.

Jury

Smeding

Stephens

N. M.

Nelson

J. E.

Aelenei

Darnon

(2017). The experience of low-SES students in higher education: Psychological barriers to success and interventions to reduce social-class inequality. Journal of Social Issues, 73(1), 23–41. https://doi.org/10.1111/josi.12202

16.

Kline

R. B.

(2011). Principles and practice of structural equation modeling. Guilford Press.

17.

MacCallum

R. C.

Browne

M. W.

Sugawara

H. M.

(1996). Power analysis and determination of sample size for covariance structure modeling. Psychological Methods, 1(2), 130–149. https://doi.org/10.1037/1082-989X.1.2.130

18.

Markus

H. R.

(2017). American = independent? Perspectives on Psychological Science, 12(5), 855–866. https://doi.org/10.1177/1745691617718799

19.

Markus

Kitayama

(1991). Culture and the self: Implications for cognition, emotion, and motivation. Psychological Review, 98(2), 224–253. https://doi.org/10.1037/0033–295X.98.2.224

20.

Maydeu-Olivares

Coffman

D. L.

(2006). Random intercept item factor analysis. Psychological Methods, 11(4), 344–362. https://doi.org/10.1037/1082-989X.11.4.344

21.

Messick

(1989). Validity. In Linn

R. L.

(Ed.), Educational measurement (3rd ed., pp. 13–103). American Council on Education and MacMillan.

22.

Messick

(1995). Validity of psychological assessment: Validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. American Psychologist, 50(9), 741–749. https://doi.org/10.1037/0003-066X.50.9.741

23.

Midgley

Maehr

M. L.

Hruda

Anderman

E. M.

Anderman

Freeman

K. E.

Gheen

Kaplan

Kumar

Middleton

M. J.

Nelson

Roeser

Urdan

(2000). Manual for the Patterns of adaptive learning scales (PALS). University of Michigan.

24.

Midgley

Maehr

M. L.

Urdan

(1993). Manual for the Patterns of adaptive learning survey (PALS). University of Michigan.

25.

Muthén

L. K.

Muthén

B. O.

(1998-2017). Mplus user’s guide (8th ed.). Muthén and Muthén.

26.

Nunnally

Bernstein

(1994). Psychometric theory. McGraw-Hill Higher, Inc.

27.

Phillips

L. T.

Stephens

N. M.

Townsend

S. S. M.

Goudeau

(2020). Access is not enough: Cultural mismatch persists to limit first-generation students' opportunities for achievement throughout college. Journal of Personality and Social Psychology, 119(5), 1112–1131. https://doi.org/10.1037/pspi0000234

28.

Phinney

J. S.

Dennis

Osorio

(2006). Reasons to attend college among ethnically diverse college students. Cultural Diversity and Ethnic Minority Psychology, 12(2), 347–366. https://doi.org/10.1037/1099-9809.12.2.347

29.

Raftery

A. E.

(1995). Bayesian model selection in social research. Sociological Methodology, 25, 111–163. https://doi.org/10.2307/271063

30.

R Core Team . (2022). R: A language and environment for statistical computing. R Foundation for Statistical Computing. https://www.R-project.org/

31.

Skaggs

(2022). Test development and validation. Sage Publications, Inc.

32.

Stephens

N. M.

Fryberg

S. A.

Markus

H. R.

Johnson

C. S.

Covarrubias

(2012). Unseen disadvantage: How American universities’ focus on independence undermines the academic performance of first-generation college students. Journal of Personality and Social Psychology, 102(6), 1178–1197. https://doi.org/10.1037/a0027143

33.

Stephens

N. M.

Hamedani

M. G.

Destin

(2014). Closing the social-class achievement gap: A difference-education intervention improves first-generation students' academic performance and all students' college transition. Psychological Science, 25(4), 943–953. https://doi.org/10.1177/0956797613518349

34.

Stephens

N. M.

Townsend

S. S. M.

(2015). The norms that drive behavior: Implications for cultural mismatch theory. Journal of Cross-Cultural Psychology, 46(10), 1304–1306. https://doi.org/10.1177/0022022115600264

35.

Stephens

N. M.

Townsend

S. S. M.

Dittmann

A. G.

(2019). Social-class disparities in higher education and professional workplaces: The role of cultural mismatch. Current Directions in Psychological Science, 28(1), 67–73. https://doi.org/10.1177/0963721418806506

36.

Stephens

N. M.

Townsend

S. S. M.

Hamedani

M. G.

Destin

Manzo

(2015). A difference-education intervention equips first generation college students to thrive in the face of stressful college situations. Psychological Science, 26(10), 1556–1566. https://doi.org/10.1177/0956797615593501

37.

Tibbets

Harackiewicz

J. M.

Canning

E. A.

Boston

J. S.

Priniski

S. J.

Hyde

J. S.

(2016). Affirming independence: Exploring mechanisms underlying a values affirmation intervention for first generation students. Journal of Personality and Social Psychology, 110(5), 635–659. https://doi.org/10.1037/pspa0000049

38.

Townsend

S. S. M.

Stephens

N. M.

Smallets

Hamedani

M. G.

(2019). Empowerment through difference: An online difference-education intervention closes the social class achievement gap. Personality and Social Psychology Bulletin, 45(7), 1068–1083. https://doi.org/10.1177/0146167218804548

39.

Tyler

K. M.

Uqdah

A. L.

Dillihunt

M. L.

Beatty-Hazelbaker

Conner

Gadson

Hughes

Mulder

Owens

Roan-Belle

Smith

Stevens

Henchy

(2008). Cultural discontinuity: Toward a quantitative investigation of a major hypothesis in education. Educational Researcher, 37(5), 280–297. https://doi.org/10.3102/0013189X08321459

40.

Walton

G. M.

Cohen

G. L.

(2005). Excluded: Social identity incompatibility undermines motivation. Manuscript in preparation.

41.

Walton

G. M.

Cohen

G. L.

(2007). A question of belonging: Race, social fit, and achievement. Journal of Personality and Social Psychology, 92(1), 82–96. https://doi.org/10.1037/0022-3514.92.1.82

42.

Wolf

E. J.

Harrington

K. M.

Clark

S. L.

Miller

M. W.

(2013). Sample size requirements for structural equation models: An evaluation of power, bias, and solution propriety. Educational and Psychological Measurement, 76(6), 913–934. https://doi.org/10.1177/0013164413495237

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.15 MB