MIsgivings about measurement invariance

Complicated yet arbitrary benchmarks

Our first point concerns the widespread use of various MI criteria in cross-cultural research. Assessments of MI use complex methods that few researchers appear to fully understand, and typically yield dichotomous evaluative decisions based on seemingly arbitrary benchmarks. The typical answer concerning the origin of whatever specific benchmark is employed is that someone (or an institution such as the Educational Testing Service) published an article recommending it. These well-entrenched metrics, which may include (expected) inter-item correlations, slopes, and intercepts, are taken as compelling in and of themselves, so any further empirical or theoretical justification of the recommended standard generally remains obscure, if it exists at all. The (most likely few) researchers who actually go and study said authoritative article will therefore not necessarily be enlightened. But they will still find that they must follow its “recommendations,” as grant reviewers, journal reviewers, and editors enforce the presence of statistical tests for MI.

It is noteworthy that in almost all cases, the recommended metrics ultimately stemmed from statistical simulations of generated data rather than originating from real (and messy) cross-cultural data (e.g., Bauer et al., 2020; Cole et al., 2019; Wang et al., 2022). The lack of empirical evidence for established metrics is telling. In 2007, Herbert Marsh challenged his fellow colleagues of an SEM association to refute his claim that it is impossible to find good “fit” based on these established metrics using any existing psychological instruments with multiple factors and multiple items per factor (Marsh et al., 2014). No one reported being able to do so. More recent efforts to move away from strict cutoffs and instead to treat MI as a continuous outcome rather than a dichotomous decision are notable must still confront the same issue of establishing a standard metric for defining “partial” or “approximate” invariance (Fischer et al., 2022; Robitzsch & Lüdtke, 2022).

The vantage point of empirical cross-cultural researchers, by which we mean the ones who actually gather data in more than one country, is typically different than that of editors, reviewers, and other armchair critics. After having made the considerable effort to obtain data in more than one cultural context and, often, language, empirical researchers then face a black box where they dump their data in one side (such as into a newly publicized R package), and wait for the output on the other side, with fingers crossed. And then, almost always, they get bad news. Cross-cultural differences are “significantly” non-zero. The RMSEA’s, CFI’s and other metrics fall short from ideal prescriptions. This outcome is typical when data from two cultural contexts are compared. The outcome is inevitable when more than two cultures and languages are included. There is simply no way to establish MI across, say, 63 different cultural contexts and 40 languages. ³

When the by-now unsurprising lack of measurement invariance is found, researchers may be encouraged to remove items from their measure or adjust scores based on the MI criteria to more closely align with the “true” latent scores for a group. Papers that report differences between traditional and “corrected” mean scores and/or construct correlates cite these differences as evidence that MI corrections are necessary for cross-group comparisons (e.g., Lacko et al., 2022). But there is typically little if any evidence that these new scores are any more valid than the original ones, and the adjustments make it difficult to compare results across past and future studies employing the same scale (Robitzsch & Lüdtke, 2022). While some problematic items might be uncovered, such as discovering that a word has been mistranslated (Mueleman et al., 2022), weak item-total or inter-item correlations are not necessarily harmful to a scale’s external validity, and may in fact be helpful ⁴ (Revelle, in press). Additionally, as Welzel and colleagues (2021) illustrated with data from the World Values Survey, larger group differences decrease the variance of scale items, and yet such variance is statistically necessary for finding higher inter-item correlates. Thus, subsetting measures to include only items with strong inter-item connections can reduce group differences by removing extreme items with very little variance despite having meaningful external correlates across countries.

Against prohibition

Discussions of MI often have a prohibitionist tone. A “failure” (that’s the actual word used most often) to achieve MI by conventional criteria is not typically treated as a scientific finding of interest in its own right (but see Fischer et al., 2022 for a recent exception). Rather, it is more often given as a reason, even a “violation” (another often-used word) that implies one should not take the cross-cultural data seriously, or sometimes, even look at them. In a recent paper in which MI was not achieved, the authors primly stated that, therefore, they did not examine their data any further. No wonder they didn’t dare, in the face of a recently published warning that “widespread hidden invalidity in the measures we use... pose[s] a threat to many research findings” (Hussey & Hughes, 2020, p. 166).

Such a prohibitionist tone goes too far. First, the amount of non-invariance required to throw substantive results into question is far from clear and, as noted above, often is evaluated on the basis of seemingly arbitrary benchmarks. Second, the implications of a “failure” of MI depends on the kind of MI one decides to insist on. Do you want to interpret correlations among measures within countries? Then configural MI is enough, and it is indeed often found (e.g., Aluja et al., 2019). But what about if you want to interpret mean differences between countries? Well, then you are said to need “strict” or scalar invariance, which is a very high bar seldom attained.

More balanced treatments of failures of scalar invariance that do not use the overly prohibitionist tone may still instead say something like “all questionnaires showed some noninvariance across countries, indicating that caution needs to be exercised when investigating and interpreting mean differences” (Wetzel et al., 2021, p. 40). The careful, moderate tone of the quote is appreciated but still: what is this finely worded advice supposed to mean? That otherwise you can throw caution to the winds when making mean level comparisons across groups? Of course not. Caution is warranted regardless of the evidence for or against MI, as it is for any scientific study, rendering the conventional pleas for careful interpretation meaningless, and the very motivation for assessing MI unclear.

Meaningful cross-cultural differences should not be the default assumption

An emerging, and perhaps surprising, theme emerging in cross-cultural research is that differences in psychological attributes and processes are turning out to be much smaller than many of us initially expected (Allik & Realo, 2017). The touted fundamental differences between East and West were not only almost absurdly simplistic (Asia is a big and diverse place, as is Europe) but also have turned out to be smaller, less profound, and less consistent than first assumed. East Asia contains many individualists, and Europe and North America have a fair number of collectivists, and while there still might be overall differences, the distributions of individuals in these categories overlap considerably (Takano, in press).

The results of our own international project, which gathered data from more than 60 countries around the world, surprised us as well. We began by expecting that large and meaningful cultural differences would be ubiquitous across the many measures we employed. As we have analyzed and explored our large and complex data set over the past several years, this expectation has not been borne out. For example, in one set of analyses, we examined two measures of happiness, one developed in the US and the other developed in Japan, that were purported to be profoundly different because of the contrasting views of happiness in these two countries. But the data revealed correlates and other results that were much more similar than different around the world. Two countries in which the two measures had particular similar psychometric properties were, ironically, the US and Japan (Gardiner et al., 2020).

We also analyzed situational experience around the world in two samples, one with participants in 20 countries and one with participants from 64 countries. In both samples, we found that individual experiences within countries were more similar to each other than experiences compared across countries, as we expected, but the difference was surprisingly small and indeed, just barely reached statistical significance even with sample sizes in the thousands (Guillaume et al., 2016; Lee et al., 2020). More generally, the distinguished and pioneering cross-cultural researcher Allik (2005) has written about how personality variation across countries has turned out to be (unexpectedly) small compared with variation within countries (see also: Hanel et al., 2018). In the face of all this, how long can we maintain the default assumption that members of different cultures are importantly different from each other in their interpretation of scale items unless “proven” to be the same, and a conventional wisdom that cultural variation in the basic properties of well-established measurement instruments is typically large, consequential, and maybe even fatal?

Consider, again, the nature of cross-cultural versus within-culture variation. Perhaps, indeed, the items on the BFI-2 extraversion scale have a different meaning for someone living in Japan than they do for you. ⁵ But might they not also, to some degree, have a different meaning for your next-door neighbor than they do for you? And can we assume that the former difference in meaning is really all that different, from the latter difference? Juri Allik’s conclusions give reason to be uncertain. Measurement instruments surely have at least somewhat different properties and implications for different individuals. But the data we have seen do not offer a strong basis to presuppose that these properties and implications necessarily vary to any importantly consequential degree according to whether the individuals in question reside in the same or different countries. Perhaps, sometimes, they do. But the burden of proof seems misplaced, given what we are learning about cultural variation elsewhere.

A shift towards external validity

Common applications of MI focus on internal validity and pay little if any attention to external validity, which is a much more appropriate metric of evaluation. Consider two illustrative examples.

First, it is possible to measure a construct with a high degree of internal consistency across groups but still draw incorrect theoretical conclusions when external validity is not the focus. Consider the construct of individualism versus collectivism, one of the first major theories to emerge from cross-cultural psychology and arguably still the most widely referenced today (Smith & Bond, 2022). Variations in questionnaires for measuring collectivism have been developed and refined over the years, often with the goal of improving measurement equivalence across cultural groups such as by adding reversed items to lessen the effects of response style differences (Vignoles et al., 2016). But the measurement of collectivism across cultures has still run into issues, notably that even after establishing measurement equivalence between groups the theoretically expected mean level differences are not found (Wong et al., 2018).

However, as illustrated most recently by Talhelm (2022), the issue with measuring collectivism may not stem from response style biases, detectable and correctable using MI techniques, but from the conceptualization of the measure itself. Specifically, the content of items in most measures of individualism/collectivism is developed by what Talhelm has called a biased view of collectivism as a “socialist utopia” that is inferred rather than observed by cultural psychologists. Only after looking at mean level comparisons and country correlates does this issue with measuring collectivism become apparent. In this case, researchers can establish technical measurement equivalence—that their questionnaire of collectivism is measuring the same thing across groups—but still not actually measure the originally targeted construct (see Takano in press, for a related critique).

A second example of the importance of external validity is offered by the famous World Happiness Report rankings from the Gallup World Poll. These rankings are widely referenced not just in scientific research but in pop culture as well. The rankings stem from a single item that asks people to assess their current relative standing compared to the best possible life they could imagine, and then comparing the average of those item ratings across countries. A plausible theoretical case could be made that someone’s “ideal life” may be culturally specific and thus not be comparable across cultures, or indeed even across people within a culture. However, the question of MI for the World Happiness Report has been generally ignored, largely because its use of a single measurement item protects it from the application of any traditional MI techniques that rely on inter-item relationships. This lack of demonstrated MI, though, has not stopped the yearly country rankings from being widely reported in the media, cited in government policy debates, or even, in the case of Denmark, proclaimed on a welcome banner to international arrivals at the Copenhagen airport. ⁶

Instead, evidence for the validity of single item measures of happiness comes from external validity, both through convergent correlations with other measures of well-being (e.g., Cheung & Lucas, 2014) and from meaningful country-level correlations. For example, average well-being scores are higher in more developed countries where incomes are higher and people are healthier, two well-established predictors of greater well-being among individuals. In countries that face sudden conflict or other negative events, longitudinal studies have observed expected declines in average country level well-being (e.g., before and during the Syrian civil war—Cheung et al., 2020). This external evidence offers convincing grounds for concluding that the World Happiness Report enjoys considerable validity for international comparisons, notwithstanding its lack of conventional establishment of MI.

These two examples highlight a key critique of MI: Conventional assessments of MI are completely internal to the measurement instruments. They focus on the structure of the latent factors of the instruments, and the degree to which this structure is maintained across contexts, and—even more stringently—the intercepts of the items on latent traits or factors. ⁷ This may be all well and good, but internal validity is not the same as external validity and the former is actually not even always necessary for the latter (Revelle, in press). As was noted in a footnote earlier in this paper, this fact has long been understood if not always remembered. The classic examples in personality psychology are the MMPI and the CPI (California Psychological Inventory), the scales of which have well-established validity-in-use for predicting important outcomes, but which “fail” many conventional psychometric tests of internal reliability and factorial homogeneity.

To be clear, we do not advocate for the complete abandonment of reporting internal reliabilities. Examining the properties of an instrument’s performance is still a crucial component to any scientific report, and studies in cross-cultural psychology are no exception. We support efforts to compare the internal reliability of measures across cultural groups, such as comparing patterns of inter-item relationships (Gardiner et al., 2019) or patterns of factor loadings (e.g., Hogan Data Science, 2023), as useful steps in checking translation accuracy. But at present there is an overreliance on strict metrics that are entirely based on internal properties of the measurement for determining the quality of the data.

A useful future direction would be to move away from the almost exclusive focus on the internal properties of our measurement instruments in favor of increased emphasis on external validity (Welzel & Inglehart, 2016). This can and should be done at both the cultural and individual level. At the cultural level, research could assess average levels of measurements with other country-level variables (e.g., Mottus et al., 2010; Allik & Realo, 2017), for example, as illustrated with the previous research on country level correlates of happiness. But lest we fall prey to the ecological fallacy, this kind of research must be complemented by investigations at the individual level, assessing to what degree and when the measure’s correlations with other psychological variables are maintained across cultural contexts, for example, if a measure of happiness correlates with other indicators of well- being within and across some, many, or all countries (e.g., Diener et al., 2010; Gardiner et al., in press). This kind of convergence would be more persuasive evidence for the cross-cultural validity of a measure than even the finest-grained demonstration of strict MI.

Even better—and even more difficult—a measure used in more than one country could be compared in its associations with actual behavior, something that despite psychology’s self- definition as the “science of behavior” continues to get less attention than it should (Baumeister et al., 2007). One small step in the right direction is the use of “anchoring vignettes,” in which respondents read hypothetical descriptions of an individual and are asked to rate their traits (Mottus et al., 2012; Talhelm, 2022). Another promising approach was demonstrated by a study that assessed differences in sociability between residents of Mexico and the United States using naturalistic audio recordings as well as self-reports (Ramírez-Esparza et al., 2009). Research like these examples may lead to an eventual gold standard for cross-cultural psychology, in which behavioral data, and not just self-reports, are compared across cultures. To do this will be difficult and expensive. But we must, sooner or later.

Summary and recommendations