Developing,Testing,or Extending Theory with Meta-analyses

Organizational Psychology Review's (OPR) aim “is to publish original conceptual work and meta-analyses in the field of organizational psychology.” Importantly, OPR is by and large a theory journal, meaning that all publications in OPR need to make a substantial theoretical contribution to the literature in organizational psychology and related fields (Reiter-Palmon & Buengeler, 2023). Although OPR primarily publishes theory manuscripts and conceptual reviews, 75% of all meta-analyses that appeared since the journal's inception in 2011 were published in or after 2018, and all published meta-analyses have already amassed more than 1,750 citations on GoogleScholar. These numbers indicate that meta-analyses are becoming increasingly popular, likely because they create impact on both research and practice.

Recently, the editors of OPR stated that meta-analyses which “provide a basis for new theory and extension or adaptation of theory would provide meaningful contributions and therefore would be appropriate to submit to OPR” (Reiter-Palmon & Buengeler, 2023, p. 207). This follows a general trend in our field wherein meta-analyses are increasingly used for theory building and theory testing (Aguinis et al., 2011). In the current editorial, we aim to elaborate on the necessary theoretical contributions of meta-analyses by describing what we expect from meta-analyses and what kind of meta-analyses are (not) likely to be published in OPR (see Table 1 for an overview). Our intention is that outlining the quality standards for meta-analyses, specifically focusing on how meta-analyses can help to move theory forward, will serve as a useful guide for authors who are planning to conduct a theoretically meaningful meta-analysis that they want to submit to OPR.

OPEN IN VIEWER

Table 1.

Conceptual meta-analysis checklist when preparing a submission to Organizational Psychology Review (OPR).

	Check
Theoretical contributions (at least one of the below)
Theory development: The meta-analytic findings are used to develop a new framework or theory.	□
Theory testing: The meta-analytic findings are used to test or challenge a theory (in a new context), to contrast different theories against one another, or to answer methodological or practical questions that have implications for theory.	□
Theory extension: The meta-analytic findings are used to extend or adapt an existing theory.	□
Agenda for future research
The meta-analytic findings are used to identify gaps in or additions to theories, and things we do not yet know about a theory.	□
Methodological deficiencies that hinder the development, test, or extension of a theory are identified and solutions are proposed (if applicable).	□
Methodological considerations
Appropriate meta-analytic methods are chosen to test the hypotheses (e.g., meta-analytic structural equation modeling, relative weights analyses, etc.).	□
PRISMA or MARS guidelines, as well as the methodological OPR meta-analysis checklist, are followed.	□
The generalizability of findings with regard to the theoretical implications is discussed.	□
Potential pitfalls that might limit the theoretical contribution, such as publication bias or low quality of primary studies, are considered and addressed.	□

Theoretical contributions of meta-analyses

The ultimate goal of science is the production of knowledge, and in the last century, organizational psychologists and management scholars have accumulated substantial knowledge that can be summarized using meta-analytic methods. Meta-analyses are mostly used to estimate the true effect for a given relation or to explain variation in the effect size distribution which is valuable for both research and practice. For example, Schmidt and Hunter's (1998) classic meta-analysis about the predictive validity of different selection measures improved selection practices in organizations and moved the entire field of personnel selection forward. However, meta-analyses can and should do more than that, and this holds especially for meta-analyses published in OPR. As stated above, OPR is by and large a theory journal, and meta-analyses published in OPR should therefore make a substantial theoretical contribution. In other words, researchers should not just do a meta-analysis for the sake of doing a meta-analysis. The impetus for a meta-analysis should not be that one has not been done before or that an existing meta-analysis is too old and needs updating. Simply providing meta-analytic effect sizes is not enough (for OPR). Instead, meta-analyses are, at the core, still conceptual review papers, and should answer “big questions and test big ideas” (Humphrey, 2011), meaning that authors should provide a clear rationale for why a meta-analysis is needed and how this meta-analysis helps to move theory forward.

But what does that actually mean? To answer this question, it is important to briefly outline what a theory actually is, and why theories are important for science. An abundance of definitions exists, but an operational one defines theory as a “systematic explanatory statement about the relationships among a set of constructs, with accompanying logic and assumptions” (Ferris et al., 2012, p. 96). As such, theories provide guiding frameworks and make causal claims about the relations between variables from which testable hypotheses can be deduced. Essentially, a theory is just a fancy way of describing that we claim to understand what is going on (Aguinis & Cronin, 2022). We need theories to make accurate predictions about different phenomena given the knowledge of our time, explaining what, why, and how something is happening.

Theories are important for a wide variety of stakeholders: Researchers use them to understand phenomena of interest and to design future studies, managers rely on theories to develop interventions targeted at specific organizational problems, policymakers use them to design policies in ways that lead to desired results, and students use theories to understand organizational problems (see Aguinis & Cronin, 2022, for their perspective on what a good theory is and watch out for a special issue in OPR compiling other perspectives on what a good theory is that will come out soon). Understanding this centrality of theories for both research and practice makes it clear why meta-analyses should make a substantial theoretical contribution (see also Kilduff and O'Sullivan, 2024). By combining data from all available studies examining a certain relation (or relations), meta-analyses have the unique opportunity to develop, test, or extend theories.

Looking backward and forward

Above, we have outlined how meta-analyses can make a theoretical contribution by summarizing findings from past empirical research (i.e., looking backward): Given what we know, here is what works (or does not work), especially regarding theories. By the nature of their tasks, meta-analysts become experts in the literatures that they are summarizing, and this knowledge should be used to develop an agenda for future research (this holds equally for review articles). By this, we do not mean generic statements such as “conduct more longitudinal studies” or “examine additional moderators” because these tend to lack specificity and usually do not provide any meaningful insights. Instead, meta-analyses should systematically examine gaps and deficiencies in the summarized literature that prevent the comprehensive development, tests, or extensions of theories. A good example of this is van den Broeck et al.'s (2021) observation that self-determination theory does not seem to predict destructive behaviors well and that the question “of whether externally regulated people just ‘don’t contribute’ in organizations, as [their] results suggest, or whether they actively cause trouble” (p. 265) still remains largely unanswered. Meta-analyses can also be used to identify methodological deficiencies which limit the ability to conclusively test theories. For example, in their meta-analysis about the moderating role of cultural differences for the relation of leadership styles with followers’ work engagement, Li et al. (2021) develop a detailed agenda for future research that can help overcome the methodological limitations that plague the vast majority of leadership studies (e.g., low statistical power, publication bias, endogeneity, etc.).

Useful meta-analytic methods

Several advances in meta-analytic methods can help researchers to comprehensively develop and test theories, among which meta-analytic structural equation modeling (Cheung, 2015) is probably the most noteworthy. In other words, meta-analytic structural equation modeling has “the potential to reshape a literature's development” (Bergh et al., 2016, p. 494), especially with regard to theories. This method allows researchers to test theoretical models based on meta-analytic data even if none of the included primary studies measured all variables of interest. Most meta-analyses, including those published in OPR (e.g., He et al., 2019; Kong et al., 2019; Pletzer et al., 2024), test theories by relying on path analyses conducted on a meta-analytic correlation matrix, which suffers from the limitation that these analyses do not take the variance of the tested relations into account. To overcome this limitation, researchers should conduct two-stage meta-analytic structural equation modeling based on primary correlations (Cheung, 2015) or rely on full information meta-analytic structural equation modeling which can be used to quantify heterogeneity by calculating credibility intervals around the average path estimates (Cheung, 2018; Yu et al., 2016). Meta-analytic relative weights analyses can supplement these analyses (Tonidandel & LeBreton, 2011) to compare the importance of different predictors. Other meta-analytic advances, such as meta-analyses of individual participant data (Riley et al., 2010), second-order meta-analyses (Schmidt & Oh, 2013), or multilevel meta-analyses (Van den Noortgate et al., 2013) can also be useful to comprehensively test theories. However, the primary use of meta-analytic structural equation modelling is to test theories and it is therefore more suitable for theory testing than other methods.

All meta-analyses (submitted to OPR) should closely follow the PRISMA (Page et al., 2021) or MARS guidelines (Cooper, 2010), and report their methodology in a transparent and reproducible manner. Authors are encouraged to pre-register their meta-analyses and expected to publish all data and analytic scripts (e.g., on a public repository like the Open Science Framework). It is also important to interpret findings critically, especially with regard to generalizability and the influence of different biases on the overall effect size distribution. First, in most meta-analyses, it makes sense to focus on validity generalization instead of on statistical significance, or to define theoretically and practically meaningful effect sizes a priori (Anvari et al., 2023). Second, meta-analyses are dependent on available primary studies, and fundamental deficiencies in the summarized literature will be reproduced in a meta-analysis (i.e., “garbage in, garbage out”) where these deficiencies are much harder to detect. This also holds for publication bias, which can result in incorrectly estimated effect sizes, and researchers should critically analyze and interpret the influence of publication bias on their findings. Third, it can be misleading to combine effect sizes from different kinds of studies. For example, a certain intervention can be effective in some industries, but not in others. By combining studies across industries, one would detect a moderate effect of the intervention, but this effect would be inaccurate for all industries in which the intervention was implemented. Conducting meaningful, theory-driven moderator analyses can help to overcome this limitation. Last, meta-analyses should not be conducted too early because that could “end lines of research prematurely” (Hale & Dillard, 1991, p. 466) by discouraging researchers to conduct studies about a research question that has already been meta-analyzed although it has not yet been sufficiently answered in primary studies.

Authors planning to submit a meta-analysis to OPR should be aware of these pitfalls and address them in their study design. Upon submission, authors will have to declare that they followed the methodological meta-analysis checklist published on OPR's manuscript submission guidelines website (https://journals-sagepub-com-s.web.bisu.edu.cn/author-instructions/OPR). The overarching goal of these measures is to guarantee the quality, transparency, replicability, and reproducibility of all steps involved in a meta-analysis.