Abstract
In this study, we first develop a framework that presents systematicity as an encompassing orientation toward the application of explicit methods in the practice of literature reviews, informed by the principles of transparency, coverage, saturation, connectedness, universalism, and coherence. We then supplement that conceptual development with empirical insights into the reported practices of systematicity in a sample of 165 published reviews across three journals in organizational research. We finally trace implications for the future conduct of literature reviews, including the potential perils of systematicity without mindfulness.
The growing volume and diversity of studies, proliferation of searchable databases, and rapid diffusion of research have contributed to the increased prevalence of literature reviews (Briner et al., 2009; Rousseau et al., 2008). Concurrently, the role and purpose of literature reviews in organizational research have evolved considerably over the past quarter of a century. Traditionally employed to summarize the state of the science, literature reviews are increasingly utilized to integrate, refine, and elaborate theory (Baumeister & Leary, 1997; Grant & Booth, 2009; Leidner, 2018; Post et al., 2020; Rowe, 2014; Sandelowski & Barroso, 2003; Siddaway et al., 2019; Tranfield et al., 2003; Xiao & Watson, 2019). As scientific disciplines become increasingly diversified, literature reviews also provide an antidote to proliferation and balkanization by bridging diverse theories and approaches that address shared questions.
Perhaps not surprisingly, researchers, practitioners, and policymakers have discussed the need for and importance of systematicity to strengthen the methodological rigor of literature reviews. Quite possibly, the most promising development has been the systematic (literature) review methodology (SRM). The SRM is “a specific methodology that locates existing studies, selects and evaluates contributions, analyses and synthesizes data, and reports the evidence in such a way that allows reasonably clear conclusions to be reached about what is and is not known” (Denyer & Tranfield, 2009, p. 671). Importantly, the SRM is not a literature review in the traditional sense but, rather, is “a self-contained research project in itself that explores a clearly specified question, usually derived from a policy or practice problem, using existing studies” (Briner & Denyer, 2009, p. 671). Thus, employing SRM is not always desirable or feasible when conducting a literature review—such as when the purpose is to develop an integrative research framework or to explore encompassing questions spanning multiple research streams (Briner & Denyer, 2012; Denyer & Tranfield, 2009; Tranfield et al. 2003). In short, even though the emergence of SRM has had a positive influence on the methodological rigor of literature reviews, it is not well suited to tackle the full spectrum of questions that literature reviews in organizational research seek to answer.
Acknowledging this, many researchers have selectively borrowed practices from the SRM when conducting literature reviews. Indeed, many published reviews display the features of what can be best described as methodological bricolage—the practice of “combining analytic moves for the purpose of solving a problem or problems tailored to one’s own research project” (Pratt et al., 2020). Consequently, reviewing practices in organizational research continue to be bespoke, haphazard, and inconsistent across the universe of studies. So, although literature reviews in organizational research appear to be becoming more systematic, they are doing so in an unsystematic way, devoid of shared methodological principles and practices (Kunisch et al., 2018). This leads to a natural question: Can review article authors achieve the underlying principles of systematicity by carefully selecting a set of practices appropriate to their research questions and purpose?
To that end, we develop an integrative framework that conceptualizes systematicity as an encompassing orientation toward the application of explicit methods in the practice of literature reviews, informed by the principles of transparency, coverage, saturation, connectedness, universalism, and coherence. By envisioning systematicity as an encompassing orientation, we join others in suggesting that “while not all reviews need to be equally systematic, they all require a minimum level of systematicity” (Paré et al., 2016, p. 496). Beyond specifying the relevant principles, our discussion integrates the current knowledge on systematicity with the core practices of literature reviews. In so doing, our conceptual development addresses the current lack of integration between the principles and practices of systematicity in literature reviews.
We then supplement that conceptual development with empirical insights into the reported practices of systematicity in a sample of 165 published reviews across three journals in organizational research (Journal of Management [JOM], Academy of Management Annals [AMA], and International Journal of Management Reviews [IJMR]). We find that reviews, on average, report less than half of the practice items we track. That said, we also observe a trend toward employing more practices over the past 10 years—and this trend is not solely due to the increased uptake of the SRM. We find significant variability in systematicity practices across reviews, partially explained by the type of question the review seeks to answer and publication outlet. Importantly, we also find that articles reporting more systematicity practices, and in particular those relevant to the principle of transparency, are correlated with elevated levels of research impact.
We finally conclude with the study’s implications for bridging the current and future systematicity practice in literature reviews. We underscore the need for alignment between a study’s systematicity and its purpose, contending that a “one size fits all” approach is problematic. We also discuss promising directions for future research and call for additional empirical research on the practices and consequences of systematicity in published reviews. We suggest that such examinations can be instrumental in organizational research, where the reported practices of systematicity display a great deal of variability across reviews.
Developing a Framework for Systematicity as an Encompassing Orientation
Systematicity is an inherently ill-defined and nebulous concept. In our evaluation, one of the more comprehensive reviews of the idea is found not in literature reviews but in a contribution to the general philosophy of science—Hoyningen-Huene’s (2013) discussion on systematicity and nature of science. To ground the thesis that the systematicity of science distinguishes it from other forms of knowledge, Hoyningen-Huene noted that systematicity broadly refers to the quality of not being “purely random, arbitrary, unmethodical, unplanned, or unordered.” Taking a historical and evolutionary view of the concept in various disciplines, Hoyningen-Huene concluded that the idea is inherently vague and “therefore in need of more precision and concretization.” This is because one could include a range of other qualities in the list of descriptions. Each such quality would, in turn, imbue the concept with additional meaning. We concur with Hoyningen-Huene that rather than an abstract-level understanding, systematicity is better understood in a specific context.
Reviewing prior research, we identified three conceptions (explicit or implicit) of systematicity in the context of literature reviews. First, systematicity can refer to a methodological procedure. The SRM is such an example, characterized as “methodical, comprehensive, transparent, and replicable” (Cochrane Collaboration, as cited in Siddaway et al., 2019, p. 751). Second, systematicity can be conceptualized as a discrete set of review practices, such as: a protocol to guide review execution (Pullin & Stewart, 2006), the employment of a systematic sampling logic and procedures (Hiebl, 2021; Xiao & Watson, 2019), the application of an objective set of criteria and process for screening for inclusion (Madden et al., 2018), data extraction procedures (Lunny et al. 2017), methods of research synthesis (Grant & Booth, 2009), and interpretation of findings (Samnani et al. 2017). And third, systematicity can also be considered an objective standard against which the methodological rigor of literature reviews may be evaluated—such as by reviewers, editors, practitioners, and policymakers (Paré et al., 2016). But there is no definitive list of criteria for evaluating the level of systematicity in literature reviews. As Pare et al. (2016) noted, “when used as a qualifying adjective it refers to the fact that authors are being systematic in their approach to literature reviews, in a general sense as a perceived virtue” (p. 496).
Synthesizing the essential ideas from these conceptions, we propose that systematicity is an encompassing orientation toward the application of explicit methods in the practice of literature reviews, informed by the principles of transparency, coverage, saturation, connectedness, universalism, and coherence (see Figure 1). The practices executed in conformance with these principles are, in turn, central to delivering on the promises of a literature review in terms of its richness, reproducibility, trustworthiness, and utility.
A framework for systematicity as an encompassing orientation in literature reviews.
An encompassing view of systematicity is auspicious in organizational research, wherein literature reviews come in all shapes and sizes due to the diversity of interests and fragmented intellectual terrains. As depicted in Figure 1, our conception is thus most closely aligned with Paré and colleagues (2016), who defined it as a “disposition towards organized, methodic, and orderly inquiry that uses various methods and processes to search, screen, assess, analyze, and interpret relevant information with a view to achieving a set of specific research goals” (p. 496). It is also consistent with Rosseau et al.’s (2008) definition of systematic as meaning “comprehensive accumulation, transparent analysis, and reflective interpretation of all empirical studies” (p. 479). Thus, systematicity is not simply present or absent. Instead, it is a continuum—although not all reviews need to employ a systematic review methodology, virtually all reviews do (and should) exhibit some level of systematicity (Paré et al., 2016). We next discuss each dimension of Figure 1 in turn.
Principles
We suggest six principles are relevant to systematicity as an encompassing orientation—transparency, completeness, saturation, connectedness, universalism, and coherence.
Transparency reflects the extent to which the methodological procedures at each stage of the review process are explicit. To Denyer and Tranfield (2009), there are three core requirements for transparency: (a) reviewers must be open and explicit about the processes and methods employed in the review, (b) reviewers should demarcate the linkages between the observations collected from the literature and the findings or conclusions of the review, and (c) reviewers should make clear the assumptions underpinning the review. Transparency also calls for reviewers to be explicit in the trade-offs made and how those trade-offs were informed by the questions that guide the review.
Completeness refers to the principle of ensuring comprehensive coverage of the literature within the scope of the review. As Briner and colleagues (2009) implored, the aim is “to find all studies relating to the question” (p. 25). This extends to coverage of the gray literature, the “diverse and heterogenous body of material that is made public outside, and not subject to, traditional academic peer-review processes” (Adams et al., 2017, p. 432). However, completeness is a necessary but not sufficient condition for systematicity. Even very complete reviews may rely on only a fraction of the literature to make inferences or justify conclusions.
With origins in grounded theory, saturation means that “no additional data are being found whereby the researcher can develop properties of the category” (Glaser & Strauss, 1967, p. 61). In essence, saturation implies that reviewers have exploited the literature’s insights such that no additional framework elements, themes, propositions, or conclusions are likely to emerge.
Connectedness refers to the principle that reviewers identify the linkages and relationships between the various facets of the literature. The principle follows Hoyningen-Huene’s (2008) concept of epistemic connectedness, whereby “scientific knowledge has more articulate connections to other pieces of knowledge than, especially, everyday knowledge that is more loosely structured” (p. 175). Thus, connectedness captures the extent to which reviewers recognize the full set of relationships within the literature, including logical equivalence, implication, dependence, consistency, and contradiction, among others (Hoyningen-Huene, 2008). It also applies when assimilating empirical findings that address the same concept and relationship or arranging diverse empirical findings into different strands or schools of thought (Sandelowski et al., 2006).
The fifth principle is universalism, the requirement that reviewers set aside their own biases and maintain a stance of impartiality in reviewing the literature. To Merton (1942/1973, p. 270) objectivity precludes particularism. The aim is to draw conclusions by producing an unbiased assessment of the cumulative evidence on a particular topic (Siddaway et al., 2019, p. 751). Citing Baumeister, Siddaway and colleagues (2019) reminded us that reviewers should adopt the mindset of a judge and jury rather than a lawyer: “A judge and jury skeptically evaluate the evidence to render the fairest judgment possible, whereas a lawyer’s approach is to make the best case for one side of the argument” (p. 751). Universalism can be advanced by being reflexive and not taking conventions for granted (Alvesson & Sandberg, 2020, p. 1297).
Finally, systematicity should prevail across the entirety of the review rather than being confined to a discrete subset of practices. This principle of coherence refers to the extent to which reviews apply explicit methods across the entire spectrum of review practices. Coherence calls for a minimum level of systematicity both within and across practices (Edmondson & Mcmanus, 2007). It requires careful attention to the sequential and reciprocal interdependencies among systematicity practices in a review (Xiao & Watson, 2019). Without coherence, reviews may be selectively systematic—systematic in certain areas but not others.
Practices
Having clarified the principles of systematicity, we turn our attention to the specific ways that systematicity can manifest in literature reviews. In the interests of facilitating the relevance of the framework (Figure 1) to different review types, we focus our attention on seven high-level practices: envisioning, explicating, executing, evaluating, encoding, elaborating, and expositing. 1
Envisioning
Systematicity as an encompassing orientation begins with a precise and well-defined research question to orient a review. Although it is impossible to catalog all the types of questions that might motivate reviews, five fundamental orienting questions exist: exploratory, descriptive, evaluative, integrative, and explanatory. Exploratory questions are focused on defining the boundaries, parameters, and characteristics of the literature to rapidly assess what is known on a topic. They are intended to quickly amass an understanding of the key concepts underpinning a research area; examine the extent, range, and nature of research activity; or assess the primary sources and types of evidence available (Arksey & O’Malley, 2005).
Whereas exploratory questions seek to address what literature exists on a topic, reviews motivated by descriptive questions seek to ascertain both what is known and unknown about a given topic. They seek to understand whether and how a body of literature reveals any interpretable patterns or trends concerning propositions, theories, methodologies, or findings (Paré et al., 2015). They are conducive to developing comprehensive theories that offer a qualified understanding of organizational phenomena (Sandberg & Alvesson, 2021).
Evaluative questions focus on critically assessing the state of the science in terms of critical assumptions, paradigms, theory, and methodological conventions. This may involve benchmarking literature against a set of criteria (Xiao & Watson, 2019) such as adequacy, sensitivity, relevance, and rigor (Madden et al., 2018). In their most eloquent form, reviews motivated by evaluative questions seek to develop what Sandberg and Alvesson (2021) referred to as “provoking theory” by showing “often eye-opening and disruptive ways of seeing phenomena” and in so doing, challenging mindsets and stimulating new modes of thinking.
Integrative questions 2 aim to synthesize findings, observations, concepts, and relationships into higher-order themes and categories to advance theory (Sandberg & Alvesson, 2021). Reviews motivated by integrative questions bring order by integrating related concepts and differentiating unrelated concepts. They help eliminate redundancy, conceptual overlaps, and semantic proliferation by condensing and synthesizing lower-level observations and insights into higher-order themes—or identify situations where distinct concepts have been lumped together.
Finally, reviews guided by explanatory questions seek to marshal insights, observations, and findings from the literature into a framework that outlines causal relationships among concepts. They subsume what Rosseau and colleagues (2008) referred to as synthesis by explanation, which focuses on “identifying causal mechanisms and how they operate” (p. 497). By ordering and synthesizing evidence in the literature and marshaling it into a framework that specifies the causal ordering of the themes, categories, and constructs, reviews guided by explanatory questions can help to generate knowledge about the inner workings of an organizational phenomenon (Sandberg & Alvesson, 2020).
Explicating
We refer to explicating as the practice of defining the boundary conditions of the review. Systematicity in explicating is concerned with defining both substantive and procedural boundary conditions. Substantive boundary conditions specify the parameters of the review in terms of theories (e.g., resource dependence, Hillman et al. 2009), construct and contextual conditions (e.g., anger in organizations, Gibson & Callister, 2010), level of analysis (e.g., within-person variability, Dalal et al. 2014), or methodologies (e.g., simulation studies, Baumann et al. 2019). Procedural boundary conditions are concerned with curtailing the review’s scope to specific periods, fields/subfields, journals, data sources, or databases. Sometimes procedural boundaries are inadvertent—for example, using Web of Science rather than Scopus as a reference database creates quality restrictions.
The vital issue for establishing transparency during explication is specifying boundary conditions clearly and plainly. Concerning the completeness principle, a critical decision point is whether the review should include coverage of the gray literature, a heterogeneous body of research outside the traditional peer review process (Adams et al., 2017). In the interests of completeness and transparency, reviewers need to make explicit judgment calls regarding whether to include gray literature and, if so, what types of gray literature should be incorporated. Regarding the first issue, if the review topic is nascent, straddles academic and practitioner domains, or has experienced a significant uptick in attention, then expanding the review’s procedural boundary conditions to include gray literature may well be justified. In considering what types of gray literature to include, reviewers might well be informed by Adams and colleagues’ (2017, p. 435) typology of the gray literature, which is based on source expertise (known or unknown) and outlet control (known or unknown).
Executing
Systematicity in execution involves a disciplined, flexible, and detail-oriented process to ensure completeness within the substantive and procedural boundary conditions of the review. Central here is the development of a strategy that guides the keyword searches that constitute the bulk of the search process. Keywords should be derived directly from the research question. The choice of keywords should ideally strike a balance between exhaustiveness and precision; they should be sensitive to changes across time and regions (Xiao & Watson, 2019). Researchers can also rely on snowballing methods such as backward, ancestry methods—searching the reference sections and bibliographies of identified articles—and forward, citation-tracking methods. With listservs (e.g., RMnet), social media (e.g., LinkedIn), and scholarly platforms (e.g., ResearchGate), it may also be possible to crowdsource articles. Researchers can also contact subject matter experts to identify additional works such as authors of seminal studies or previous reviews. To test the efficacy of the procedures, researchers should see whether known primary studies are found; if not, revisions are warranted.
Systematicity can also manifest itself in clearly defined and explicated stopping rules or heuristics to economize on time and bound opportunity costs when the search results are voluminous. One approach is to predefine an acceptable yield from the search—such as five relevant articles per 100 references scanned—and terminate the search when the yield falls below this level (Booth, 2010). Another approach developed in ecology, capture-mark-recapture, involves capturing an initial sample from a population of interest, marking the elements in the sample with some type of tag, and then releasing the sample back into the population so that the marked elements are available to be recaptured in subsequent sampling exercises. (Kastner et al., 2009, p. 150)
Evaluating
Following the identification of the relevant population of articles, systematicity as an encompassing orientation also manifests itself in the careful screening of identified articles, a practice we refer to as evaluating. Some methodologists recommend a two-stage evaluating procedure—a coarse review of abstracts, followed by refined quality assessment based on a full-text review (Xiao & Watson, 2019). To achieve transparency, one practical approach from work on systematic reviews is to employ PRISMA reporting procedures, such as a diagram of the screening process with the number of articles retrieved and excluded at each stage (e.g., Aguinis et al., 2018; Alla et al., 2017; Simsek et al., 2015). In the interests of universalism, systematicity also calls for an objective quality appraisal of the selected articles. For example, Madden et al. (2018) discussed four criteria for assessing studies that could prove helpful: adequacy (sufficiency of approach to meet research aims), sensitivity (uncovering findings that fit or do not fit with the hypotheses or research questions), relevance (appropriateness of method), and robustness/rigor (systematic nature of the research conducted).
Encoding
The encoding process is concerned with extracting manifest data from the selected articles, sorting and organizing the data, and developing codebooks and other data entry protocols. The first issue in this respect is whether the encoding process is guided by an ex ante (deductive) or ex post (inductive) framework. Those reviews beginning with a well-defined a priori framework will typically follow a deductive logic, operationalizing and converting the framework into a coding scheme and imposing the framework on the literature. In keeping with the principle of universalism, researchers need to be attentive to anomalies and unexpected patterns that necessitate updates to the initial framework (Carroll et al., 2013). An inductive approach to encoding, by contrast, begins without an a priori framework or uses a minimally developed skeleton framework. Such an approach inductively develops a framework by aggregating and synthesizing key themes from the literature. Laplume et al.’s (2008) review of stakeholder theory is an example of the inductive approach. Another inductive approach is “synthesis by explanation,” or “explanations for phenomena by discerning patterns in published articles” (Elsbach & Stigliani, 2018; cf. Rousseau et al., 2008). Systematicity at this stage calls for special attention to coding schemes and protocols. The coding scheme and protocol should be validated with a subsample of articles before encoding begins in earnest.
The principle of saturation is of paramount importance during encoding. A risk here is that reviewers focus on completeness to the detriment of saturation, resulting in reviews that are a “mile wide and an inch deep.” This occurs when reviewers focus on the manifest attributes of studies to the neglect of underlying, latent content. Saturation requires reviewers to gather as much data as is warranted by the research question. At a minimum, essential information and metadata such as the article title, date of publication, researchers, journal, and abstract are captured. Other helpful coding fields include relationships, theories, constructs, variables, hypotheses, propositions, descriptions, and limitations. Beyond these, researchers often identify other study characteristics such as context, sample characteristics, sample size, method, research design, levels of analysis, and phenomena examined (Card, 2012; Gaur & Kumar, 2018). In our experience, researchers would be well served to extract features of the articles at the lowest level of granularity feasible given their research questions.
Elaborating
Elaborating is the process by which extracted codes and data are aggregated, organized, synthesized, and evaluated. In so doing, the principles of transparency, connectedness, and saturation should be at the forefront of reviewers’ considerations. Elaboration culminates in the development of stylized facts, propositions, frameworks, and models of the literature, especially when research questions are integrative and explanatory in nature. Immediately following encoding, researchers begin to aggregate observations, variables, and relationships into higher-order categories and themes. In this step, coding starts to take place at two levels—the individual article and the collective body of articles. For novices to the literature, it is essential that techniques are adopted to identify the connections between different strands of literature and trends and progressions over time. For experienced and expert reviewers, the adoption of methodical procedures to minimize bias and presupposed understandings is essential. Systematicity in elaboration should be guided by established methodological protocols, keeping in mind the principles of connectedness and saturation. We would draw special attention to the role of content analysis. Although content analysis is primarily used as a method to examine difficult to study issues such as executive personality and cognitions (Duriau et al., 2007; Short & Palmer, 2008), it has also been applied to the analysis of extracted data from literature reviews (Gaur & Kumar, 2018). Because it assumes that “groups of words reveal underlying themes, and that, for instance, co-occurrences of keywords can be interpreted as reflecting association between the underlying concepts” (Duriau et al., 2007, p. 6), content analysis can be applied to synthesize both manifest and latent content (Duriau et al., 2007; Gaur & Kumar, 2018). When the aim is to address integrative and explanatory orienting questions, the application of content analysis can entail the discovery of underlying meaning and interpretation of the literature (Post et al., 2020).
Other allied approaches to achieve systematicity in elaboration include thematic synthesis, metasynthesis, or framework synthesis. Thematic synthesis extracts themes from the literature, clusters them, and eventually synthesizes them into analytical themes (Xiao & Watson, 2019). Metasynthesis synthesizes qualitative information to locate key themes, concepts, or theories that provide more novel or powerful explanations of the phenomenon under review (Siddaway et al., 2019). Framework synthesis (Carroll et al. 2011, 2013) involves using the research question to shape an understanding of the topic of the review into an initial conceptual framework which is “developed iteratively as new data are incorporated and themes derived from the data” (Brunton et al., 2020, p. 316). For narrative approaches to literature reviews, techniques such as metanarrative mapping provide a way to develop coherent storylines from the literature (Greenhalgh et al., 2005). 3 These can prove useful in making sense out of contradicting data “by systematically exposing and exploring tensions between research paradigms as set out in their overarching storylines” (Greenhalgh et al., 2005, p. 427).
Systematicity in elaboration also calls for researchers to be cognizant of and appropriately represent the evolutionary trajectory of the literature and the intellectual lineage of ideas. The principle of connectedness requires appreciating how strands of literature converge and diverge over time. From a quantitative perspective, systematicity in elaborating the evolutionary structure of literature can be facilitated using bibliometric techniques such as co-citation analysis and network analysis to identify different strands of theorizing based on citation patterns across different authors, theories, and subfields (see Zupic & Čater, 2015). A novel technique, citation context analysis, enables reviewers to “explore and describe – at a specific and detailed level – how important ideas are used and spread from a source text to subsequent citing works” (Anderson & Lemken, 2020, p. 1). Qualitatively, tracing the life cycles of theories or theoretical perspectives also provides a basis for systematicity. For example, Turner et al. (2018, pp. 41-42) recommended identifying what theories are considered formal theories and informal theories, what informal theories are considered alternative theories to specific formal theories, what empirical tests have been performed, and what refinements have been made to theory as it has evolved.
Expositing
In summarizing and presenting the findings, reviewers will benefit from systematicity in the earlier stages of the review. Rigorous coding facilitates the development of a nested and hierarchical database that allows for a discussion of the findings at multiple levels of granularity. Such an approach also harmonizes the written review with any summary tables extracted from the coding database (Siddaway et al., 2019). By front-loading the coding effort, less rework is necessary if it becomes clear that additional details are required to make or support a claim. For example, suppose one wishes to claim a specific relationship holds across conditions in a literature review. In that case, it is more straightforward to code this information for each article in the review in advance rather than selectively backtracking to a subset of articles. Moreover, consistent coding facilitates the creation of high-dimensional, information-dense graphical representations that illustrate salient features of the literature (Tufte, 2001). For example, word clouds, history flows, multidimensional scaling plots, pictographs, tree maps, heat maps, and other visualizations are easier to build when well-structured data are available (Ertug et al., 2018). At the same time, bibliometrics researchers have honed techniques that visualize citation patterns and demonstrate article centrality (e.g., van Eck & Waltman, 2014).
Promises
Having clarified both the salient principles and practices of systematicity, we complete the development of our framework with the outcomes, or what we label the promises of systematicity in Figure 1. We suggest that systematicity as an encompassing orientation can facilitate the richness, reproducibility, trustworthiness, and utility of literature reviews.
Richness
Although authors strive to publish reviews that advance the extant conversation, reviews differ in their contribution’s richness or strength. Systematicity facilitates higher levels of synthesis and generativity—two core dimensions that underpin a review’s theoretical contribution (see Post et al., 2020). The synthesis dimension entails analyzing and integrating extant theory, whereas the generative dimension focuses on creating new theory. Systematicity may beget synthesis by allowing for more in-depth consideration of manifest and latent ideas, agglutinating diverse views, and combining them to focus understanding and future research rather than merely summarizing and reporting (Post et al., 2020). By contrast, generativity refers to how a review challenges existing theory or alters the trajectory of a research stream. Both within and across these development routes, generativity can vary on a continuum characterized by “incremental” versus “revelatory” (i.e., interesting, surprising, frame-breaking; Corley & Gioia, 2011). Without systematicity, reviews are harder pressed to achieve generativity—by, for example, failing to juxtapose emerging perspectives, analyze assumptions, clarify constructs, establish boundary conditions, and identify salient governing mechanisms (Haddaway et al., 2015; Post et al., 2020; Tranfield et al., 2003).
Reproducibility
Systematicity can also facilitate reproducibility by reducing the odds of propagating errors of omission, commission, or both (Katz, 2006; Ketcham & Crawford, 2007). A core concern that often prevails over literature reviews is whether the conclusions are “grounded in the data retrieved from the review process and not an argument fabricated to support a prior conclusion” (Paré et al., 2016, p. 496). Systematicity allows readers to decide for themselves whether a focal review has employed a comprehensive search process to identify the population of relevant articles (Rowe, 2014) and whether the insights gleaned from the literature warrant a particular conclusion (Denyer & Tranfield, 2009).
Trustworthiness
Systematicity can also contribute to higher trustworthiness (Paré et al., 2016). Indeed, to the principle of universalism, a vital part of systematicity is ensuring that the review appropriately incorporates and weighs both confirming and disconfirming evidence, whether quantitative or qualitative. A common concern is that a review overlooks disconfirming evidence by excluding, discounting, or dismissing studies contrary to the author’s beliefs or values (Cronin & George, 2020). Without sufficient safeguards, insights may be distorted by retrieval bias (due to inadequate or incomplete search), publication bias (reviews that exclude unpublished or gray literature), selection bias (reviews with inadequate selection criteria), selector bias (reviews based on subjective inclusion of studies based on author’s perceptions), within-study bias (variability in the coding of primary studies), and expectancy bias (author’s expectations about the results influence synthesis; Durach et al. 2017). All these biases can, in turn, mitigate a review’s trustworthiness. Systematicity facilitates trustworthiness because it increases the reliability of a review’s conclusions.
Utility
The utility is the fourth salient consideration of systematicity, which broadly involves the issues of “who cares” and “to what end” (Bacharach, 1989; Corley & Gioia, 2011). By enabling richer contributions, trustworthiness, and reproducibility, systematicity can potentially enhance utility—the potential uses (informational or otherwise) of a review by researchers (and even practitioners). A critical development in this regard has been the emergence of the evidence-based management approach (Briner et al., 2009). It has been increasingly suggested that integrative insights and best evidence may serve both organizational researchers and practitioners. In this vein, we agree with the sentiments of Denyer and Tranfield (2009, p. 687) that it is “patronizing in the extreme” to believe that a lack of skill among practicing managers means that they are unable to use the best available academic evidence produced by a systematic process. Absent systematicity, insights may be distorted, and potential biases may reduce a review’s utility among researchers and practitioners. For example, researchers may be less inclined to use nonsystematic reviews to ground hypotheses or develop the theoretical background sections in research articles. As a corollary, systematicity might also influence whether a review accrues citations, a salient dimension of scholarly impact (e.g., Aguinis et al., 2014; Bartunek & Rynes, 2014; Judge et al., 2007).
An Assessment of Published Reviews
As noted from the outset, published reviews in organizational research increasingly employ discrete practices that facilitate systematicity by drawing on techniques such as the SRM. But they have primarily done so in a nonregularized, tactical fashion following the principle of methodological bricolage. Thus, there may be significant variation in the extent to which extant reviews employ each of the seven types of systematicity practices we discussed earlier. Simultaneously, it is unclear to what extent organizational researchers conduct reviews in line with the principles identified. Finally, we are unaware of existing empirical evidence that assesses whether systematicity is associated with improved review outcomes. To make progress toward addressing these issues, we explore five questions to develop some empirical insight into the systematicity of published reviews in organizational research.
First, we sought to determine the prevalence of the set of systematicity practices. Thus, we ask: To what extent are systematicity practices employed, and do they co-occur? Second, because the body of knowledge regarding systematic reviews has blossomed in the past decade, we ask: Are there temporal trends for employing systematicity practices? Our third question asks: Are the increases in systematicity exclusively driven by increased use of the SRM? Fourth, because a review’s orienting question may affect trade-offs between depth and breadth and, consequently, the prioritization of different systematicity practices, a reasonable question follows: Are certain systematicity practices more or less likely to be employed based on reviews’ orienting question?
Our framework (Figure 1) also suggests that scholars who seek to employ these practices may enhance the richness, reproducibility, trustworthiness, and utility of their work. We submit that review articles are, partly, cited due to the reader’s independent judgment that the conclusions drawn therein are reasoned and credible (Bergh et al. 2006). To that end, our fifth question asks: Are systematicity practices associated with reviews’ scholarly impact (citation counts)? In answering this last question, we also consider the alternative explanation that increased impact is derived from the employment of the SRM as a composite whole rather than the employment of discrete practices.
Sample and Data
We selected a sample of 165 review articles published in three leading outlets for organizational review articles: Journal of Management, Academy of Management Annals, and International Journal of Management Reviews. We limited our scope to the period from 2008 to 2018 because several pieces aimed at enhancing review rigor and relevance were published around 2008 (e.g., Cooper et al. 2009; Cropanzano, 2009; Denyer & Tranfield, 2009; Liberati et al., 2009; Rousseau et al., 2008). Journal of Management was a natural choice given its publication of annual literature review issues and reputation. We started by searching for review articles in the Journal of Management using a set of different keywords, including “literature,” “synthesis,” “summary,” “narrative,” and “integrative” for the identified period. After removing duplicates across keyword searches and articles outside of our scope (e.g., meta-analyses), 119 articles were identified.
For a fuller picture, we sought to complement this article set with a 10% sampling of all articles in Academy of Management Annals (18) and International Journal of Management Reviews (28) in the same period. These journals were natural candidates because they are outlets that exclusively publish reviews of organizational research. In total, 165 articles were included in the population, which is summarized in Appendix A in the Supplemental Material, available in the online version of the journal.
To analyze these studies, we created a coding scheme (provided in Appendix B in the Supplemental Material, available in the online version of the journal) to tabulate the presence or absence of practice items, consistent with the seven practices identified in our framework. As an example, one of the evaluating practice items coded pertains to whether an article reported the use of quality appraisal techniques or interrater reliability measures. In total, we coded 52 items (five envisioning practice items, seven explicating practices, six executing practices, five evaluating practices, 17 encoding practices, six elaborating practices, and six expositing practices).
We acknowledge that the absence of a coded item in the published review does not necessarily imply the practice was not considered or performed. Details of protocols may be revealed to the editor and reviewers in earlier drafts or in response correspondence but later eliminated due to page limitations or to highlight an article’s more distinct and novel elements. 4 That said, our research question is with respect to the reported presence of relevant systematicity practices rather than the performance of specific practices—particularly because the consumer of literature reviews is not privy to undisclosed information. It is also the case that we cannot directly observe and code the extent to which extant reviews follow (or fail to follow) the principles identified in our conceptual development because they represent a broader, unobservable mindset. However, we attempt to infer the presence of these principles by analyzing patterns of specific practices employed.
We collected basic information such as publication year and the number of authors. We also coded whether the article was claimed to be systematic and whether it partially or fully employed a systematic literature review methodology, such as the one provided by Tranfield et al. (2003). Finally, we paired the coded and analyzed articles with download and citation data from journal websites and the Web of Science. For reference, key details such as downloads, citation counts, and practices reported are provided in Appendix A in the Supplemental Material, available in the online version of the journal.
Results
Question 1
To what extent are systematicity practices employed, and do they co-occur? We find that the average article reports information related to approximately 40% of the 52 possible coded items (M = 21.24, SD = 8.18). Significant differences exist across journals, F(2, 162) = 15.32, p < .01, with AMA having lower reported practices compared to JOM (d = –10.54, p t = 5.53, < .01), whereas the difference between JOM and IJMR is not significant (d = –1.89, t = 1.19, p > .10). We find significant correlations between many of the practices. For example, articles that report how the population of articles is generated (explicating practices) also tend to report findings (expositing practices) beyond narrative summaries (r = .34, p < .01). Likewise, there is a robust correlation between the use of explicating and evaluating practices (r = .72, p < .01). At the same time, it is clear that they do not all vary in tandem, with certain practices only loosely related (e.g., explicating and elaborating, r = .13). As a percentage of the number of items available to be coded, envisioning practices are relatively common (on average, 70% of such practices are employed). In contrast, elaborating practices are comparatively rare (only 19% are employed, on average). To illustrate the relationship among the various practices and the overall number of practices employed, we provide descriptive statistics in Table 1.
Correlations Among Systematicity Practices.
Note: Coefficient alphas provided on the diagonal in italics. N = 165.
Because the coemployment of different practices is important for assessing coherence, we investigated whether there were underlying patterns of practice items present in the data that cut across the seven practices (in other words, we sought to determine which discrete practice items seemed to be performed in tandem). To do so, we performed an exploratory factor analysis on the 52 items using principal axis factoring and a direct oblimin rotation (because the practice groups are correlated). As shown in Table 2, several interesting observations emerged.
Exploratory Factor Analysis of Systematicity Practices.
Note: Checklist items with an asterisk are associated with the set of practices positively associated with research impact in our regression analyses.
First, we do not find an underlying common factor that explains a considerable proportion of the variance, which would be expected if there is a common, latent factor related to overall systematicity. Instead, seven factors emerged with eigenvalues greater than one, and five factors had a least two variables with loading above .40. The structure of these five extracted factors is provided in Table 2 (as are the proportion of studies that employ each practice). We constructed scale scores and assigned labels based on the specific practices loading onto each factor above .40.
The first factor to emerge relates to search specification—or the systematic reporting of the review scope and boundary conditions. This set spans the explicating, executing, evaluating, and encoding practices. Interestingly, each of these practice items appears to be consistent with the principle of transparency. The second factor contains practices relates to model articulation—specifying, populating, and communicating a theoretical framework drawn from extant findings. In so doing, these practices provide a means to achieve the principle of connectedness. The third factor pertains to encoding data to perform a critical evaluation of prior study details such as data sources, research design, sampling, and specific hypotheses. These practices appear consistent with the principle of universalism—ensuring that findings are properly contextualized and appropriately evaluated. The fourth factor captures specific practices that we would characterize as search thoroughness—or the rigor of the search process. Practices loading on this factor go beyond using keywords alone but, rather, includes steps such as backward and forward searches and covering the gray literature. Each of these practice items would facilitate achieving the principle of completeness. The fifth factor contains practices related to determining the current review’s positioning in the literature. Although this factor does not appear to represent one of the underlying principles of systematicity, appropriate positioning is an essential element for any article, literature review or otherwise. We found that 19 of the 52 practices (36%) do not load substantively onto any of these five factors (with an additional two practices present in every review).
Because these groupings span across multiple practices, we also have initial evidence that previous reviews have established some level of coherence. At the same time, we note that the factors associated with these underlying principles vary in their association with one another. For example, although the principles of transparency and completeness appear to co-occur (r = .43, p < .01), the same cannot be said for connectedness and completeness (r = .02 p > .10). The relationships among these sets of covarying practices and the practice groups are also located in Table 1. We employ these exploratory factors to investigate research outcomes later herein.
Question 2
Are there temporal trends for employing systematicity practices? Returning our focus to the overall number of practices employed, we find a yearly trend present in the data: an increase of approximately one additional practice reported for a typical review article every 2 years (b = 0.70, t = 3.50, p < .01). As shown in Figure 2, the number of observable practices increases over time across all journals. This trend does not significantly vary across journals, even though there are differences in the overall average number of practices reported (as discussed previously). Interestingly, this trend is present and statistically significant for four of the seven practice groupings (with a fifth approaching conventional levels of significance)—indicating the result is not solely driven by isolated changes in one practice.
Systematicity practices reported by journal and year.
Question 3
Are increases in systematicity exclusively driven by increased use of the SRM? One reasonable explanation for the increased presence of systematicity practices over time could be the increased use of SRMs. Concerning the use of SRMs, we find that 29 of the 165 articles are self-described as systematic reviews somewhere in the text. Of these, 13 substantively employ a systematic review methodology, another 10 employ components of that methodology, and the remaining six do not report evidence to suggest that an SRM was employed. Interestingly, we did not discern a strong trend for employing the SRM (b = 0.01, t = 1.40, p > .10) even when employing a conservative definition (i.e., including all reviews that claim to be systematic, b = 0.003, t = 0.33, p > .10). But there were significant differences between the journals in their tendency to publish SRMs, F(2, 162) = 22.71, p < .01, with IJMR having higher odds of publishing such reviews compared to JOM (odds ratio = 3.88, z = 2.90, p < .01) or AMA (odds ratio = 10.99, z = 2.18, p < .05).
As shown in Figure 3, although articles employing a systematic literature review methodology tend to report more systematicity practices, it is clear that they are not the same. Several articles not employing this methodology still employ a high number of systematicity practices. Not all systematic literature reviews possess high levels of systematicity, as measured by the number of practices reported. That said, the employment of an SRM methodology (either partially or substantively) explains a significant portion of the variance in practices employed, F(2, 162) = 10.51, p < .01, R 2 = .12. SRM studies tend to report about 10 more practices on average compared to nonsystematic reviews (b = 9.65, t = 4.30, p < .01). As shown in Figure 4, this increase is distributed across several practice areas and is not concentrated in only one aspect of the process. In summary, although the employment of SRM appears to be a key factor, it is only a partial explanation.
Systematicity practices reported across nonsystematic, partially systematic, and systematic literature reviews.
Number of systematicity practices reported across nonsystematic, partially systematic, and systematic literature reviews.
Question 4
Are certain systematicity practices more or less likely to be employed based on reviews’ orienting question? Our data suggest that review articles seeking to make more ambitious contributions to the literature (e.g., synthesizing findings across disciplines or advancing a novel theoretical framework) tend to utilize more practices than articles that seek to summarize existing findings. As shown in Figure 5, there is a significant mean difference between reviews examining descriptive and integrative questions (d = 4.22, t = 2.15, p < .05) as well as descriptive and explanatory questions (d = 3.10, t = 1.97, p < .05). We also see evidence to suggest that integrative reviews exhibit the steepest increase in the number of practices employed over the past 10 years.
Number of systematicity practices reported by orienting question.
Question 5
Are systematicity practices associated with reviews’ scholarly impact (citation counts)? Examining whether systematicity practices are associated with greater scholarly impact is challenging because although the number of reported practices increases over time, citations trend in the opposite direction (because it takes time for citations to accrue). Moreover, review articles differ in the size and breadth of their audiences, depending on the topic and intended contribution. Citation behavior may also be driven by the overall maturity of the literature and the status of the authors involved. Therefore, we explore the incremental impact of the reported practices on review outcomes by examining the percentage of downloads that result in a citation (i.e., Citations / Downloads × 100). This measure accounts for the relative size of the audience reached. We also control for several other variables that may correlate with both systematicity practices and citation behavior, including the year of publication, the relative novelty of the topic (as measured by the number of Google Scholar hits for the review’s topic), the journal of publication, nature of the orienting question (e.g., exploratory, integrative), the number of downloads, altmetric scores, the nature of the organizing framework (to account for the complexity of insights provided), topic breadth (e.g., specific constructs vs. multiple theories), and the number of authors.
As reported in Table 3 (Model 1), we find a relationship between the number of practices reported and scholarly impact (b = 0.12, t = 2.11, p < .05). Delving deeper, we break down this omnibus average effect in two ways. First, in Model 2, we break all the effects down into each of the seven practices and find that only explicating practices have a significant effect after controlling for the others (b = 0.60, t = 2.02, p < .05). Second, in Model 3, we break down the effect into the five factors previously identified along with a residual factor of all other practices. We find that the first factor, which we argue is related to the principle of transparency, is positively and significantly associated with increased research impact (b = 0.34, t = 2.97, p < .01). We highlight the specific practices associated with this transparency factor throughout the tables and appendices (where appropriate) using asterisks for easy identification because this cluster of practice items is predictive of higher impact in our sample.
Impact of Systematicity Practices on Scholarly Impact.
Note: Robust standard errors in parentheses. N = 161. AMA = Academy of Management Annals; IJMR = International Journal of Management Reviews; SRM = systematic review methodology.
† p < .10.*p < .05. **p < .01. ***p < .001.
Finally, we test the alternative explanation that it is the employment of the SRM, rather than systematicity practices, that is associated with higher levels of research impact. As reported in Model 4, we find that the SRM is not significantly associated with greater research impact (b = 2.11, t = 1.43, p > .10). And in Model 5, we see that this effect size for systematicity practices remains largely unchanged, even with the inclusion of the SRM control (b = 0.10, t = 1.82, p < .10).
Before proceeding to implications, we hasten to add that our assessment is limited because authors make conscious trade-offs between reporting substantive content versus the procedural details of the review—especially across different journals and associated review processes. Moreover, to the extent greater emphasis is placed on the review process in the light of reporting standards such as PRISMA, more detailed information may be available in the future. We caution against overinterpreting these exploratory findings.
Implications
Our results indicate that systematicity practices increasingly appear in published reviews and that this result is not solely driven by a single practice or the increased use of the SRM. At the same time, the results also shed light on improvement areas and opportunities. They suggest that systematicity is not a unitary, reflective construct wherein all practices are employed in concert. Indeed, as Table 1 illustrates, there is significant variation in the correlations across the seven high-level practices we identify (ranging from .13 to .72). It remains an open question as to whether this is a failure to achieve a necessary level of coherence or whether systematicity is nearly decomposable, with certain aspects tightly coupled while others operate more independently.
Moreover, we observe that, on average, less than half of the 52 tracked practice items appear in the final review article manuscripts and appendices. Although some level of variation is likely functional given the differing aims of reviews, our results suggest that future articles may better deliver on the promises of systematicity by employing and reporting additional practices. In particular, to the extent that articles achieve a higher level of transparency, they may enjoy higher levels of research impact. Such an improvement is readily possible because articles on average report fewer than half of the practices that appear to be associated with transparency.
It is important to emphasize that our findings regarding variations in the level of systematicity are a predictable consequence of researchers seeking to increase their work’s systematicity without the aid of a framework or organizing device. Unable, unaware, or unwilling to fully employ the SRM, researchers have selectively deployed various discrete practices. Indeed, researchers have been presented with a bewildering array of methodological advice bespoke to specific review types and phases. Naturally, when the purpose or scope of their review does not nearly fit within these parameters, reviewers engage in bricolage. And to that end, we see in Table 1 that some articles published within the last decade (even with the benefit of plentiful guidance) still employ as few as five discrete practice items—whereas others use as many as 41 of the 52 items tracked. An inspection of the discrete items in Table 2 provides further evidence that barring elements essential to a review article (e.g., statement of the research question, future theoretical directions), there is little consistency in the employment of discrete systematic practices despite the benefit they appear to confer in the market for ideas.
This brings us to the question raised in the introduction: How can literature reviews in organizational research embrace systematicity as an encompassing orientation? To help researchers make more thoughtful choices about the systematicity of their reviews, we outline in Table 4 a protocol of 52 questions about each of the seven review practices. This protocol is intended as a heuristic guide that calls for context-specific judgment in its implementation of specific practices by appealing to the principles shown in Figure 1. We encourage future review article writers to make reasoned judgments about what steps are (or are not) necessary for their specific review. To be clear, we are not advocating for the mindless application of all steps and higher-level practices—each imposes a trade-off of time, resources, and attention. Instead, we provide this menu of alternatives for researchers to consider and select the most appropriate elements for their particular review rather than limiting their systematicity repertoire to practices found through serendipitous means.
A Protocol for Improving Systematicity of Literature Reviews.
Note: Checklist items with an asterisk are associated with the set of practices positively associated with research impact in our regression analyses.
Furthermore, because literature reviews come in many different shapes and sizes (Grant & Booth, 2009; Leidner, 2018; Paré et al. 2015; Rowe, 2014; Sandelowski & Barroso, 2003; Xiao & Watson, 2019), the framework and protocol developed here can help guide, unify, and communicate the fundamental decisions and trade-offs to be made concerning the various review practices. Indeed, our analysis revealed that the number of systematicity practices employed appears to vary according to the type of orienting review question asked. Review articles seeking to make more ambitious contributions to the literature tend to report more systematicity practices—reviews guided by integrative and explanatory research questions report significantly more practices than reviews motivated by descriptive questions. This finding reinforces our core intuition that a one-size-fits-all approach is unlikely to be useful. Thus, researchers need to be clear about the purpose of their review from the outset. In particular, reviewers need to determine whether they intend to report or summarize findings; develop, elaborate, or refine theory; and/or call attention to unanswered questions. Although we lack sufficient statistical power to detect such effects directly in our empirical explorations, it is likely the case that different systematicity thresholds exist across orienting questions—whereby there is a minimum (and perhaps also a maximum) level of systematicity required to impart confidence in the conclusions that varies across question type.
Finally, trade-offs—an inevitable feature of the review process—should be explicit, ex ante, and informed by a review’s orienting question. If the intention is simply to aggregate quantitative evidence on a specific, well-defined question, alternative synthesis approaches such as meta-analyses may be more appropriate. Authors can also consider aligning systematicity practices to their purpose across all issues outlined in the proposed framework.
Future Research Directions
We suggest three promising directions for future research that could further enhance the understanding of systematicity. First, we call on organizational researchers to conduct additional empirical studies to verify and support our findings on the relationship between systematicity and scholarly impact. We would also encourage researchers to develop a richer understanding of the relationship by using a multifaceted, “pluralistic” conception of impact (Aguinis et al., 2014; Haley et al., 2017). We also believe opportunities exist to examine how systematic practices and reporting may shape perceived review richness, reproducibility, and trustworthiness.
Second, we call on researchers to develop a richer understanding of the role of new technological advances such as text mining and machine-learning tools in facilitating systematicity (Antons et al., 2021). For example, visual data mining may provide certain advantages by enabling the categorization of themes and clustering of studies and by determining study importance (Felizardo et al., 2012). Similarly, these techniques may be deployed to aid specific review practices such as the study selection effort (Felizardo et al., 2011). For example, Feng et al. (2017, p. 42) discussed six relevant means by which text mining can facilitate review tasks such as information extraction, retrieval, and visualization and document classification, clustering, and summarization. These techniques and tools could help extract and visualize information embedded in articles and author-supplied keywords, article titles, and abstracts (Kobayashi et al., 2018; Polonioli, 2020; Tang et al., 2019).
Finally, we note that because research methods are ultimately means to ends, recognizing systematicity’s potential complications may help organizational researchers to realize better ends. Systematicity in literature reviews does not come free. Following the principles outlined in Figure 1 may have a displacement effect on the review team’s effort and time. When mindlessly implemented, they can divert limited scholarly attention, effort, and resources. For example, reporting more systematic practices also consumes more manuscript pages (although that constraint is beginning to relax with online appendices). 5 Chiasson (2015) pointed to the danger of a “methodological checklist restricting the authors’ ability to pursue alternative means and ends” (p. 175). More broadly, systematicity without mindfulness may have corrosive effects by overly regulating the interpretation and presentation of studies in a review (MacLure, 2005). A focus on systematicity alone may assume away or downplay the critical contingencies and differences to a review’s quality (MacLure, 2005, p. 394). When researchers strive for systematicity mindlessly, it may, over time, displace other important considerations—or as Chiasson put it, “the methodological drug would kill through an overdose” (p. 174). Thus, the guidance we provide in Figure 1 and Table 4 is intended to provoke and guide thought, not replace it.
Conclusions
Challenging the view that the SRM is universally superior to literature reviews, Greenhalgh et al. (2018) argued that “the under-acknowledged limitations of systematic reviews, along with missed opportunities for undertaking and using literature reviews to extend understanding within a field, risks legitimizing and perpetuating a narrow and unexciting research agenda and contributing to research waste” (p. 4). To provide an alternative that captures the spirit of SRMs without the burden of a rigid methodology, we developed and examined a framework of systematicity as an encompassing orientation toward the application of explicit methods in the practice of literature reviews. Together with the empirical results and the attendant research implications, we hope that the framework will help spark new ideas and discussions on the next generation of rigorous and impactful literature reviews in organizational research.
Supplemental Material
Supplemental Material, sj-pdf-1-orm-10.1177_10944281211008652 - Systematicity in Organizational Research Literature Reviews: A Framework and Assessment
Supplemental Material, sj-pdf-1-orm-10.1177_10944281211008652 for Systematicity in Organizational Research Literature Reviews: A Framework and Assessment by Zeki Simsek, Brian Fox and Ciaran Heavey in Organizational Research Methods
Supplemental Material
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
Supplemental Material
Supplemental material for this article is available online.
Notes
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
