Not All Effects Are Indispensable: Psychological Science Requires Verifiable Lines of Reasoning for Whether an Effect Matters

To encourage psychology researchers to think more thoroughly about effect sizes, recent articles have revisited the argument that effect sizes that might seem small may actually matter (Funder & Ozer, 2019; Götz et al., 2022). Although similar arguments have been made in the past (e.g., Abelson, 1985), the more recent of these articles have received substantial attention. Funder and Ozer (2019) has already gained more than 863 citations, and Götz et al. (2022) gained 48 citations (according to Google Scholar as of February 10, 2022). Many of these citations are used to argue for the importance of research findings. For example, 31 of the 48 citations for Götz et al. are to justify the importance of some effect size. However, none of the articles citing Götz et al. considered any arguments against the importance of the effect size that was observed. Hence, although high-profile publications such as Götz et al. have helped move researchers away from heuristically dismissing effects that are arbitrarily classified as small, the field is now in danger of heuristically accepting all effects as important.

Götz et al. (2022) and Funder and Ozer (2019) provided a sound basis for why the field should avoid classifying effects as small on the basis of arbitrary criteria and, to provide a foundation for a cumulative psychological science, instead accept such “small” effects as the norm. Our article focuses on some of the specific arguments by Götz et al. (and by others) that seem to invoke singular mechanisms that may apply if the observed effect size were generalized to some broader real-world context outside of the study. For example, Götz et al. argued that when considered within the dynamic contexts of everyday life and society, some effect sizes may become larger through accumulation over time or when applied to large populations (see also, Abelson, 1985; Cortina & Landis, 2009; Funder & Ozer, 2019; Ozer & Benet-Martínez, 2006; Silan, 2019). Yet it is highly unlikely that only a single mechanism would influence the effect size when the finding is generalized to new contexts. Therefore, the focus on a single mechanism that amplifies the importance of observed effect sizes is an oversimplification that can lead to excessively optimistic views about whether psychological effects matter.

Indeed, there are going to be boundary conditions for when effects will be important (Sauer & Drummond, 2020) and counteracting mechanisms that can make effects less important. For example, psychological reactance (Steindl et al., 2015), a type of counteracting response, may lead people to resist interventions aimed at changing their views or behaviors, which can prevent even “large” observed effect sizes from being important in the context of interest.

To be precise in language when describing different concepts, we use “observed effect size” in reference to the effect sizes estimated in some studies, which we distinguish from the “generalized effect” (i.e., the expected effect if the study were done in the exact context of application with all the relevant mechanisms applying). For example, take the claim that an observed effect size is important because it would accumulate with repetition in a more dynamic context. This is essentially stating that the generalized effect will be larger than the observed effect size because of the mechanism of accumulation through repetition.

For claims about the potential importance of findings to be specific enough to be logically or empirically verifiable, researchers need to explicitly state the mechanisms that can amplify the importance of an observed effect size and the mechanisms that counteract it in addition to any other relevant considerations that might influence how the effect generalizes. Thus, multiple mechanisms that might operate simultaneously should be considered, and all of the relevant factors that would affect an effect’s importance (i.e., auxiliary assumptions) should be stipulated so that other researchers can evaluate the claims and even test them when appropriate (Uygun Tunç & Tunç, 2020). Researchers would thus be providing verifiable lines of reasoning to justify their claims about an effect’s importance or unimportance.

To facilitate this process, in Table 1, we present examples of (a) mechanisms that amplify importance, (b) mechanisms that counteract importance, and (c) other considerations, such as the target population’s baseline levels on the variables of interest. Each of these mechanisms will vary in relevance across contexts and for different phenomena and will depend on the differences and similarities between the context of the study and the context to which the finding is being generalized. In the Supplemental Material available online, we provide more detailed descriptions of each mechanism and their relevant assumptions.

OPEN IN VIEWER

Table 1.

Examples of Mechanisms That Amplify and Counter an Effect’s Importance and Other Considerations

Amplifying mechanisms
Accumulation through repetition	An observed effect size can be important because of the generalized effect it has via the process of accumulation through repetition. For example, negative stereotypical views held about a person can induce that person to behave in ways that confirm those views. However, empirical studies underestimate the real impact of such self-fulfilling prophecies because the studies typically involve the effects of a single perceiver on a target person, whereas in the context of everyday life, there are likely to be multiple perceivers creating a cumulative and larger effect (Claire & Fiske, 1998). Thus, the claim is that the observed effect sizes of self-fulfilling prophecies in the lab represent larger generalized effects in the world because of the great number of times a person or group is exposed to the phenomenon revealed by the studies.
Amplification through interaction	Sisk et al.’s (2018) meta-analysis showed that the average effect size from growth mindset interventions is “small” and unlikely to be practically important. However, a large randomized controlled trial showed that the effect of a growth-mindset intervention was moderated by whether the school provided high-quality learning opportunities. The effect was therefore smaller when averaged across the whole sample and larger when examining the subsample of at-risk students from lower-performing schools (Yeager et al., 2019). The effect of the intervention on at-risk students was thus amplified by whether the school was high or low achieving.
Cascades	Even if an observed effect size is small according to some arbitrary criteria, the generalized effect in a more dynamic context may involve a causal chain, or cascade, that later results in a larger effect size (i.e., downstream consequences). For example, Bond et al. (2012) ran a randomized controlled trial of political mobilization with 61 million Facebook users in the United States. Messages directly influenced people’s voting behavior, but also their political self-expression on Facebook, which cascaded into differences in the voting behavior of other people who were close Facebook friends with the participants. Given that each participant has multiple Facebook friends, the effect size cascaded into a larger overall effect.
Scaling up	An observed effect size may be important because of the number of people that the generalized effect is scaled to. For example, natural experiments examining the impact of mass-media campaigns on smoking have found effects that may be classified as small according to Cohen’s criteria, for example, but are considered important because of the large number of people who subsequently stopped smoking (Durkin et al., 2012).
Counteracting mechanisms
Habituation	If people habituate to a stimulus, then an observed effect size from a study will be unlikely to have a generalized effect through the process of accumulating additively, and once a population is fully habituated, there will be no generalized effect at all. For example, people become habituated to a reinforcing stimulus the more that stimulus is presented, reducing the effectiveness of the stimulus upon repetition (Lloyd et al., 2014).
Counteracting responses	People may alter their behavior in ways to counteract the observed effect size (see also psychological reactance; Steindl et al., 2015). For example, because people want to be perceived in a way that is consistent with their own self-views, they are often motivated to influence how others perceive them so that they will sometimes respond in ways to convince perceivers holding negative stereotypical views that those stereotypes are wrong (Jussim, 2012; Swann & Ely, 1984). Therefore, observed effect sizes from studies on self-fulfilling prophecies will not always have implications for more dynamic contexts because the generalized effect involves counteracting responses.
Homeostasis	Certain aspects of a system may have a stable set-point around which there can be fluctuations but to which the system reverts over time. Hence, for an observed effect size to matter, either the system must be one that is not governed by a stable set point or the observed effect size must be large enough to create a new set point. For example, hedonic adaptation is when people revert to a set point following changes in life circumstances that cause fluctuations in subjective well-being or positive and negative affect (Diener et al., 2006; Lucas, 2007). Therefore, in systems governed by homeostasis, unless the observed effect size is large enough to create a new set point, the generalized effect may be too small and/or transient to matter.
Counteraction through interaction	Similar to amplification through interaction, there are likely to be factors in the context of application that will interact with the variables of interest to counteract the observed effect size so that the generalized effect is smaller (or nonexistent). For example, Johnson (2012) examined the implementation of several evidence-based marriage interventions in the United States. The results of the interventions comparing treatment groups with a control group ranged from positive effects at one site where the intervention was deployed to null effects at six other sites to negative effects at an eighth site. The problem was that the interventions had been designed on the basis of findings from research that used samples of middle-class couples but was then applied to poorer populations for whom marriage quality, at least as conceptualized by the interventions, was not always highly prioritized. The extent to which people prioritized marriage quality in their lives may have interacted with the interventions to produce an overall negligible effect.
Other considerations
Baseline levels	The target population’s baseline levels on the variables of interest will determine whether an observed effect size will have a generalized effect. Milgram et al. (1969) found that the number of confederates who looked up into an empty sky had a curvilinear effect on the number of passersby who conformed and also looked up: There was little additional impact of having more than five confederates, up to a total of 15.
Stage of development	If there is a developmental process involved in the phenomenon, the state of the target population needs to be accounted for to understand the generalized effect. For example, the relationship between a supervisor and employee develops over a long period. Therefore, if a study investigating supervisor behavior and employee well-being is conducted when the supervisor and employee samples are already in their work roles, then it would be missing the processes important for the development of the relationship (cf. Butler, 2011; Hofmans et al., 2019).

Researchers who wish to argue that an observed effect size is important because of a generalized effect will need to explicitly state the relevant amplifying mechanisms that will operate in the context to which they are generalizing. Moreover, they will need to explicitly state why the counteracting mechanisms are likely to not apply or why the amplifying mechanisms would be stronger. Likewise, researchers who claim that an observed effect size is not important will need to communicate which counteracting mechanisms are assumed to operate and the relevant factors involved. They will further need to justify why the amplifying mechanisms will not apply or why counteracting mechanisms will be stronger. In an ideal case, claims about the importance or unimportance of observed effect sizes should be based on empirical data (Primbs et al., 2021) or, at the very least, stated in a way so that the claims can be corroborated or falsified in future empirical work.

If researchers explicitly state all of the relevant mechanisms that they think would apply, then the assumptions and hypotheses behind claims about the importance or unimportance of effects will be made clear. Such transparent claims about effect-size generalization will be more amenable to formal theoretical work, such as analytical models or simulation studies that test internal consistency and logical coherence (e.g., Smaldino, 2017). Furthermore, communicating hypotheses and assumptions transparently will help highlight where disagreements lie and where future empirical investigations should focus. For example, a justification that explicitly rests on the assumption that an observed effect size will accumulate through repetition can be examined by studies that are specifically designed for this purpose and for ruling out any counteracting mechanisms, such as habituation (Groves & Thompson, 1970). Thus, by making transparent claims about effect-size generalization, evaluating the importance of research findings can become empirically or logically verifiable, perhaps even falsifiable. In the Supplemental Material, we provide a hypothetical example for how our proposal can help to move researchers away from speculation and toward empirical verification.

With increasing scrutiny placed on how researchers report and communicate the implications of their findings (e.g., Premachandra & Lewis, 2022), explicitly stating the assumptions and testable predictions needed to substantiate claims about effect-size generalization could become recommended or required at different stages on the path from psychology to practice. For example, reviewers and editors can ask authors to justify their claims for whether an effect matters in the context to which application is being recommended. Consumers of research findings, such as policy decision makers and other stakeholders, would thus have a guide to help evaluate the quality of claims about the implications of research findings for dynamic real-world contexts.

Researchers should also keep in mind that assessing how effect sizes would generalize is not a one-time event: Research findings may have implications for many different contexts that vary on relevant factors. Different contexts may thus involve different amplifying and counteracting mechanisms. Moreover, the same contexts may change with time. For example, people’s attitudes and behaviors can change once they learn about psychological phenomena or because of changing cultural norms (Gergen, 1973). Therefore, assessing how an observed effect size generalizes should be considered an ongoing process.