Abstract
Psychologists are spending a considerable amount of time researching and developing interventions in hopes that their efforts can help to tackle some of society’s pressing problems. Unfortunately, those hopes are often not realized—many interventions are developed and reported in journals but do not make their way into the broader world they were designed to change. One potential reason for this is that there may be a gap between the information reported in articles and the information others, such as practitioners, need to implement the findings. We explored this possibility in the current article. We conducted a scoping review to assess the extent to which the information needed for implementation is reported in psychological intervention articles. Results suggest psychological intervention articles report, at most, 64% of the information needed to implement interventions. We discuss the implications of this for both psychological theories and applying them in the world.
I get tired of research that never gets used . . . the question is when will we take all of that energy and all the wonderful things that you do and make it work for people.
One of psychology’s longest reoccurring calls is to do whatever we can to “give psychology away” (P. S. Forscher et al., 2020; Miller, 1969). In the 1940s, Kurt Lewin encouraged researchers to engage in “action research,” particularly that which could improve intergroup relations (Lewin, 1946). In the 1990s, there were discussions about the role psychological research could and/or should have in shaping laws (Ellsworth, 1991). And most recently, in the 2010s, there were calls for governments to develop councils of psychological advisers, similar to the Council of Economic Advisors, an agency in the executive office of the president of the United States, who could help to ensure that the policies set are psychologically informed (B. Schwartz, 2012; Sunstein, 2016; Teachman et al., 2015). The rationale for this most recent push is that we as psychologists have done so much intervention research that our expertise could ostensibly be leveraged to cultivate a healthier, happier, and more sustainable world (Gruber et al., 2019). On their face, these calls seem like a great idea. Psychologists have long conducted intervention research that can speak to and potentially help to address a variety of pressing social problems (Suarez-Balcazar et al., 1992).
In practice, however, it is presently unclear whether the decades of intervention research psychologists and other social scientists have conducted is, in fact, ready for broader implementation and scaling (Goroff et al., 2018). For psychological interventions to have the societal impacts we aspire to have as a field (Teachman et al., 2015), psychologists must—at the very least—report the information that practitioners wanting to implement our research would need to implement it successfully. Practitioners would need to know how differences in social contexts influence intervention efficacy; precisely what types of independent and dependent variables have been examined and what that means for implementation and scaling efficacy; how attitudes, goals, or identity factors influence people’s responses to interventions; how dosage and the timing of the intervention matter for uptake and maintenance; how the source or other intervention agent of change (e.g., doctors, teachers, government officials) influence the efficacy of the intervention; and much more (Earl & Lewis, 2019; Goroff et al., 2018). Without knowing and reporting these things, well-intentioned interventions designed to improve outcomes may have many unintended, life-altering consequences for the people being intervened on. Moreover, policymakers and practitioners whose job it is to decide how to intervene need to know what to expect and how to plan and forecast returns on intervention investments. These are necessary steps in making decisions regarding the adoption, implementation, and scaling of interventions.
Unfortunately, psychologists have traditionally been ill-equipped to take on the challenge of translating insights from studies into effective interventions that can be brought to scale (Yeager et al., 2016). We often conduct our research in laboratory settings (Baumeister et al., 2007; Sears, 1986) with samples that are not representative of the broader world to which we wish to generalize (Henrich et al., 2010; Lewis, 2019; Simons et al., 2017). We are also not well versed in considering and reporting the costs and benefits of implementing interventions in a resource-constrained world (P. S. Forscher et al., 2020; Sunstein, 2016). These may not be big problems for research aiming to make incremental advances to abstract theoretical propositions, but it is a problem in the realm of intervention research: It is extremely rare for interventions that change outcomes in the laboratory to translate to the world outside of the lab with high enough fidelity to be practically meaningful (DeAngelis, 2010). As Yeager and colleagues (2016) noted, “although promising, self-administered psychological interventions have not often been tested in ways that are sufficiently relevant for policy and practice” (p. 375). This contributes to the substantial gap that exists between research and practice (Committee on Health and Behavior: Research, Practice and Policy, Board on Neuroscience and Behavioral Health, 2001, Chap. 7).
To address this gap, researchers and practitioners have developed a series of frameworks and guidelines for conducting and reporting research in ways that would make it possible to use the research for practice. One such framework is the reach, efficacy, adoption, implementation, and maintenance (RE-AIM) framework that was originally developed by Glasgow and colleagues (1999) to provide guidance on how to conduct research in this way in the field of public health. Since the first version in 1999, RE-AIM has evolved considerably in its usage over the past 20 years (planning, reporting, reviews) and been applied in various fields (e.g., environmental change, health policy, aging, child care, quality improvement) and in clinical, community, and corporate settings (Gaglio et al., 2013; Glasgow et al., 2019). RE-AIM outlines multilevel (individual and setting) and multistage indicators (planning, evaluation, reporting) that are essential to consider to achieve sustained population-level effectiveness of interventions (Gaglio et al., 2013; Stenhouse, 2017). Reach assesses whether the sample being studied is representative of the target population of interest by examining the number, proportion, and representativeness of the participants in a study. Efficacy assesses the impacts of the intervention on participants by considering factors such as effects on quality of life, economic impact, as well as negative effects. Adoption assesses the feasibility of translating between the setting of the intervention study and the setting or settings in which the intervention would actually be disseminated and considers factors such as the number, proportion, and representativeness of stakeholders who would be willing to implement the intervention. Implementation assesses factors related to implementation fidelity: how closely must implementers adhere to the original protocol to achieve the same outcomes as the original researchers and how much time and money would that take. Finally, maintenance assesses factors associated with long-term follow-up (to the extent that it is desirable); it focuses on how long researchers followed up with participants and organizations to determine long-term impact.
Current Study
In the current study, we used the guidelines described above from the implementation science literature to assess the extent to which (a sample of) the psychological intervention literature provides the information required for successful implementation, herein referred to as implementation information. To do this, we conducted a scoping review of and used an implementation-science-inspired checklist to code articles published in the past two decades about five categories of psychological interventions that were designed to drive social change in a variety of domains and have become well known in public discourse: belonging (Walton & Cohen, 2011), growth mindset (Aronson et al., 2002), utility value (Hulleman & Harackiewicz, 2009), self-affirmation (Cohen et al., 2006), identity-based motivation (Oyserman, 2015), as well as interventions that combined elements of these. We chose these interventions because they have been studied in a multitude of settings by researchers across various subdisciplines in psychology (e.g., educational, social, organizational, clinical) as well as in neighboring disciplines (e.g., business, communication, education, and public health). Thus, we are biasing the sample of articles in our scoping review toward articles about interventions that have been thoroughly studied and tested and therefore should be the most “shovel ready” for dissemination and implementation. To maximize the chances of implementation readiness of the interventions in the articles we coded, we opted to include articles from the most recent years of research, the years 2000 to 2018, allowing some time after the initial stages of intervention creation to get them closer to dissemination readiness. Although these interventions were conducted in various settings to improve specific outcomes for different target populations, to give readers some familiarity with the general concepts behind each intervention, Table 1 provides a descriptive overview of how the interventions were applied in educational settings to improve student outcomes.
Descriptive Overview of the Five Interventions
We had four primary descriptive goals in this endeavor: first, to get initial estimates of the state of implementation information available in the literature on psychological interventions; second, to determine the types of implementation information that are typically published in the psychological intervention literature; third, to identify gaps between implementation information that is published in the psychological intervention literature and the information required for practitioners to implement them; and fourth, to discuss the implications of any gaps that exist for both theory and practice. To be clear, the goal of the current study was to begin to describe the state of implementation information in published psychology research on social interventions; descriptive research is, in its own right, essential for the advancement of science (Rozin, 2009). The goal was not to compare or rank subfields or researchers. Because our goal was description rather than comparison, we employed the method of a scoping review rather than a systematic review or other meta-analytic techniques.
Method
Brief primer on scoping reviews
Scoping reviews provide an overview of topics or fields by systematically mapping large bodies of literature. Unlike traditional systematic reviews that draw on literature pertaining to a specific question, scoping reviews synthesize and analyze topic- and/or field-related research to provide coherent insight into the conceptual state of research. See Table 2 for a brief overview of the similarities and differences between scoping reviews and traditional systematic reviews.
Differences Between Scoping Reviews and Systematic Reviews (Arksey & O’Malley, 2005)
Scoping reviews map existing literature in terms of volume, nature, and characteristics of research. Scoping reviews are suited for topics that have not been extensively reviewed and are an appropriate method for initial explorations regarding the extent, range, and nature of research activity and allow scholars to identify gaps in the existing literature. Because our goal is to provide a descriptive overview of the presence/absence of implementation information in psychological intervention research, we required a systematic method that enabled a conceptual mapping of topic-related research material that is not bound by specific research questions, designs, or methods; scoping review is an ideal method for this kind of inquiry (Davis et al., 2009).
Scoping review protocol and preregistration
The protocol for conducting this scoping review was preregistered as an open-ended registration at OSF on December 22, 2018 (https://osf.io/dr9qw/). This article was drafted following the reporting guidelines presented in the Preferred Reporting Items for Systematic Reviews and Meta-Analysis Protocols Extension for Scoping Reviews (PRISMA-ScR; Tricco et al., 2018).
Literature search strategy
The literature search was limited to the databases PsychInfo and ERIC. Gray literature and unpublished work were not included because this was not intended to be a comprehensive review. Instead, the goal was to take a small sample of recent intervention-related publications in psychology, particularly five that are discussed widely in public discourse, and assess the level of implementation information they provide. We identified 105 relevant publications: 80 articles on PsychInfo and 25 on ERIC, of which only 88 1 were unique items. See preregistration stored in the OSF repository for the literature search strategy (https://osf.io/dr9qw/).
Inclusion criteria
Of the 88 unique publications, only the 56 that met the inclusion criteria made up the final data set used for coding. We included only peer-reviewed English journal articles published between 2000 and 2018 that contained implementation information for any of the five intervention categories of interest. Articles that did not meet the above inclusion criteria were excluded from further review. See Figure 1 for an overview of the literature search and the screening process.
Overview of literature search and screening.
Adapting the RE-AIM framework for psychology
The creators of RE-AIM posit that all five evaluative components are very important while acknowledging that some may be more important than others for different interventions and its contexts of application. Glasgow and colleagues (2019) encouraged a more pragmatic use of key dimensions that are most relevant for the “particular question, setting, stakeholders, and stage of research” (p. 5) when the comprehensive application of all elements is not practical. Given that this framework was developed for another field (public health interventions), we adapted a RE-AIM checklist to code for variables that appear in psychology research that are equivalent to the RE-AIM components traditionally examined in public health.
Variables coded
The coding variables chart in the OSF repository contains the detailed list and operational definitions of each variable and, to the extent that we can, “exemplars” for the variables. In general, we coded variables associated with intervention reach (three variables), efficacy (nine variables), adoption (11 variables), implementation (12 variables), and maintenance (one variable).
Coding process
All articles that met the inclusion criteria were coded by multiple student coders to assess whether information provided in published articles and/or the supplemental material associated with those articles contained information that is relevant to implementing the intervention. Because our goal was to provide an initial description of the state of the literature, coding was limited to assessing the mere presence or absence of implementation information for each of the variables on the checklist. We did not code the quality of the implementation information (a point we revisit in the discussion). We employed a binary coding strategy in which articles that provided any information relevant to implementation, irrespective of its quality, was coded as 1 to indicate presence of information. No implementation information related to the coding category was coded as 0 to indicate absence of information. Discrepancies were discussed and resolved in weekly coding meetings that took place over two semesters (spring and summer 2019). To generate a summary score of how much implementation information was contained in each article (of all of the implementation categories we coded for), we simply summed up the counts to arrive at percentages that reflect the amount of implementation information that was presented. Percentages were calculated to capture both the presence of implementation information in each article as well as across all articles. When reading and interpreting the results that we report in percentages, readers should remember that these percentages reflect a proportion of the coded counts (see the detailed code sheet stored in the OSF repository for this project at https://osf.io/3wgku/).
Results
The final data set consisted of 56 peer-reviewed journal articles that collectively addressed five categories of psychological interventions: belonging, growth mindset, utility value, self-affirmation, and identity-based motivation (IBM). Most of the articles were about self-affirmation (n = 17) and IBM (n = 16) interventions. Growth mindset (n = 8) and utility value (n = 11) interventions were also fairly represented. However, there were only two articles on belonging interventions. Furthermore, two articles addressed more than one intervention and were categorized as “combined.” Across all of the articles, the different types of interventions provided, on average, 36% of implementation information recommended by implementation scientists.
As described in the Method section, we assessed each evaluation component of the RE-AIM framework using a checklist of questions adapted from the repository of measures and checklists found on re-aim.org (RE-AIM.org, 2020). This checklist consisted of specific indicators that mapped onto the different components. For example, the evaluation component reach was assessed using three questions: Does the study (a) identify the sample, (b) describe the sample, (c) provide explicit exclusion criteria? These three questions collectively represented the evaluation component reach. Articles that provided any information about reach, irrespective of information quality, were included. Overall, most articles had information about reach (71%; who benefits from the intervention) and efficacy (51%; the impact of the intervention), closely followed by adoption (28%; information about target settings and parties who would/should be interested in adopting the intervention) and implementation (27%; information about implementation protocol, which includes time and financial investments). However, the articles had very little information about maintenance (2%).
Figure 2 illustrates in greater detail which pieces of implementation information were present in articles for each RE-AIM component.
Presence of implementation information by reach, efficacy, adoption, implementation, and maintenance (RE-AIM) component. HR = human resources.
As displayed in Figure 2, with respect to reach information, all studies identified and described their sample, but only 14% explicitly provided exclusion criteria. This lack of information about exclusion criteria is concerning given the implications for establishing boundary conditions for theory (Simons et al., 2017) as well as the implications for knowing whether it is appropriate or inappropriate to apply the intervention to some groups but not others (Goroff et al., 2018).
Regarding intervention efficacy, all articles provided information about the problem being addressed, how the intervention addresses the problem, and the expected intervention outcome. Although most (84%) provided information about the effect size, a very few articles reported information about intervention reliability (20%). When assessing for moderating variables, which we termed contextual factors, that fostered or hindered intervention effectiveness, only 25% provided information about factors that fostered, and 11% specified factors that hindered effective outcomes. Furthermore, only 5% provided information about economic factors required to assess the importance of the intervention; that is unfortunate because decision-makers often need that information to decide whether to implement one form of intervention over another. Note that although our codebook included many economic factors such as opportunity costs, consumable costs, sensitivity analyses, and costs incurred by those receiving the intervention (e.g., loss of working hours, travel time), we had to collapse these categories to “economic factors” because of little to no reporting in the articles we coded.
In terms of intervention adoption, although 77% of articles identified stakeholders, only 45% provided information relevant to these stakeholders. Ideally, stakeholder statements should identify key stakeholders from those with direct involvement (e.g., teachers) to those with a higher-level involvement such as policymakers (e.g., superintendent of schools, funding agencies). In addition to identifying these stakeholders, the quality of implementation information can be improved by providing comprehensive information about and for potential key stakeholders such as who these stakeholders are, what roles they play in implementing the intervention and when, and what they need to know to ensure that the intervention can be implemented and maintained. Fifty-four percent of the articles provided information about the human-resource requirements required, and 77% provided information about material and structural resources needed. However, very few articles provided a breakdown of this information by the different phases of the intervention.
For implementation information, all articles provided dosage information, and 79% gave information about the time course for dosage (e.g., Bratter et al., 2016, administered a 15–20 min self-affirmation intervention four times over a school year). Very few other pieces of information about implementation were provided.
Finally, only one study provided information about maintaining the intervention, more specifically, identifying ways to document and share learning regarding the implementation process. See example quotes of the RE-AIM evaluation components and some of its specific indicators in Table 3.
Example Quotes
Interpreting the results
Although there is considerable variance in the numbers between and within the five evaluation components of the RE-AIM framework, readers should keep in mind that the mere presence/mention of the indicator was counted as implementation information regardless of the quality. For example, when coding stakeholder information, we considered studies that mentioned at least one stakeholder as providing implementation information. Ideally, articles should provide a stakeholder statement with comprehensive information about and for potential key stakeholders. As another example, when coding for effect sizes, we looked for at least the presence of effect size units such as Cohen’s d. Often it appears that the authors reported the effect size numbers and left it at that. This approach may be suitable for other academics but not for other stakeholders. In addition to statistical reporting, authors need to translate the effect size number in the context of the intervention and its settings. For example, if an intervention is to improve grades for students, the magnitude of impact can be presented in a way that is relevant to the context at hand (e.g., improvement will be at least a half-point increase in grade point average on a 4.0 scale).
Discussion
One of psychology’s latent values that has manifested frequently over the decades is the value of sharing our research with the broader world—to ensure that our work can contribute to addressing pressing issues in society (Ellsworth, 1991; Gruber et al., 2019; Lewin, 1946; Teachman et al., 2015). In the field, this value is particularly important among intervention scientists who spend much of their working lives conducting the very studies that ought to be able to speak directly to problems of the day (Suarez-Balcazar et al., 1992). Given this value and the number of people conducting research to translate that value into action, we wondered how ready intervention science research in psychology was for implementation. Thus, we reviewed some of psychology’s most well-known interventions to get a preliminary answer.
Of the psychological intervention articles we reviewed, the article that provided the most implementation information contained information for 64% of the implementation categories coded. We suspect, given the generous (presence/absence) coding strategy we adopted, that the estimates reported in this article may be overestimates of the implementation information reported in psychological intervention articles (and supplemental material of those articles); a more stringent analytic approach may have yielded different results (e.g., floor effects).
Implications for theory
On the surface, the questions about implementation readiness asked in this article might seem like purely “applied” questions that would be of little interest to psychologists trying to advance “basic” theories. However, these seemingly applied questions are, in fact, deeply theoretical (Lewis, 2019, 2020). One of social psychologist Kurt Lewin’s most famous quotes is “if you want truly to understand something, try to change it.” At its essence, that is what intervention research in psychology tries to do. By studying questions about how to successfully develop, disseminate, implement, and scale interventions, the field gains more precise theories about the complex interactions between people and their situations (Earl & Lewis, 2019; Goroff et al., 2018; Rothman, 2004).
Examining the implementation information reported in our field’s intervention articles has the potential to teach us about how differences in contexts influence our ability to detect (and replicate) psychological phenomenon (Klein et al., 2018), whether there is heterogeneity in psychological processes between and within populations (Simons et al., 2017; Whitsett & Shoda, 2014), whether and how features of the time periods in which a study was conducted matter for the mental representations that come to people’s minds (Lewis & Michalak, 2019), whether the sources we use in our studies and the ways they communicate matter for people’s metacognitive experiences (Schwarz, 1994), and the downstream consequences of those experiences for their motivation (Oyserman, 2015). Studying these processes brings us closer to understanding how complex dynamics guide human behavior (Goroff et al., 2018; Hovland et al., 1953).
Implications for practice
From a practical perspective, intervention research is driven by the assumption that once tested for effectiveness, interventions will be disseminated and implemented in a variety of relevant settings (Committee on Health and Behavior: Research, Practice and Policy, Board on Neuroscience and Behavioral Health, 2001, Chap. 7). That assumption is misguided. For intervention research to be useful, implementors need information about critical factors that allow them to determine whether and how to implement the intervention. So although it is wonderful for psychologists to develop and test interventions and publish them in peer-reviewed journals, if we do not share the information that is required by implementors, the immense potential our work has for fostering change will not be realized (Lewis & Wai, 2021).
Incentivizing implementation reporting
There seems to be misalignment between the work that psychologists put in to develop and test interventions and the potential for those interventions to have broader impacts in the world. Thus, it might be worthwhile to consider ways of systematically aligning the way we do intervention work with the goal of practical impact right from the very beginning (Stenhouse, 2017). If psychologists were to incorporate designing for implementation/dissemination much earlier in the research pipeline, the possibility for our ideas to have practical relevance would be much greater (Horowitz et al., 2018; Yeager et al., 2016). Although we do not yet have a comprehensive set of solutions for how to achieve this goal, one strategy that is worthy of consideration as a starting point is to potentially modify reporting guidelines to incentivize researchers to consider implementation at the beginning of their intervention research process. Given that peer-reviewed articles are mainly targeted for other researchers, the articles we reviewed succeed in addressing the needs of its primary audience (other academics) but have not addressed the needs of other intended audiences of intervention research: intervention practitioners. Although peer-reviewed articles are not targeted to practitioners and policymakers, these articles are often the primary source of information regarding interventions of interest. To bridge the communication gap between researchers, practitioners, and other decision-making stakeholders, editorial boards of academic outlets, especially those publishing research related to social change, can help by recommending implementation-related reporting guidelines.
Limitations and constraints on generality
The current research has some important limitations that need to be kept in mind when interpreting the results of this article. We have made many of these points earlier in the article, but we want to reiterate them because of their theoretical and practical importance. The data presented in this article are from a scoping review, not a large-scale systematic review or meta-analysis. When interpreting results, we ask readers to remember that this is not a comparative exercise to assess which subdisciplines or interventions within psychology are doing better or worse but is rather a conceptual mapping of the state of implementation information in the field at the current moment in time. We acknowledge that our analysis is based on a framework that was initially developed in another field (public health); reasonable readers may question whether it is fair to hold psychologists to the standards of another field. That is a discussion we should have as a field: What standards should we hold ourselves to, and what frameworks are the most relevant frameworks for psychologists wanting to intervene in the world (see also IJzerman et al., 2020)? At present, psychologists are not expected to report on RE-AIM indicators or other indicators for implementation despite the amount of intervention research that we do in our field. However, in the interest of starting somewhere, we chose to use an already existing evidence-based framework as a starting point to shed light on areas that need attention. The creators of RE-AIM posit that all five evaluative components are very important while acknowledging that some may be more important than others for different interventions and its contexts of application. Glasgow et al. (2019) recommended that the relative importance of the dimensions should be established a priori with all the stakeholders in the planning stages and encouraged a more pragmatic use of key dimensions that are most relevant for the “particular question, setting, stakeholders, and stage of research” (p. 5) when the comprehensive application of all elements is not practical. It might be worthwhile for psychologists to consider collectively coming up with an implementation checklist that is tailored to psychology.
We assessed only peer-reviewed journal articles, which in itself may not be the ideal way to communicate with practitioners and policymakers (Posavak, 1992). However, given that most academic research is published in peer-reviewed journals, starting off with peer-reviewed articles seemed reasonable.
It is also important to remember the role of researcher decisions when interpreting the results. We chose to adopt a binary coding strategy (presence or absence of information) rather than a deeper-diving assessment of the quality of implementation information, and then we aggregated those binary counts to create overall indicators. Alternative operationalizations may lead to different conclusions; one of our motivations for sharing our data set is to encourage others to consider alternative approaches and what they might mean for broader conclusions (B. K. Forscher, 1963; LeBel et al., 2018). Most indicators we used for coding are better suited for interventions that have been extensively tested and are ready for dissemination and implementation. Although our chosen time frame (i.e., 2000–2018) was meant to increase the chances of obtaining implementation-ready intervention reporting, the literature search strategy employed for this scoping review mainly identified articles with intervention research in the very initial stages, for example, laboratory-based experiments testing for effects. Readers should keep this in mind when interpreting the results. On a related note, the lack of later-stage intervention research may have at least two possible explanations: (a) Intervention research rarely moves beyond experiments in laboratory or other heavily controlled settings, and/or (b) implementation information is being communicated in other (i.e., nonacademic) outlets.
Furthermore, we reviewed only peer-reviewed journal articles from a limited time window (2000–2018) for five categories of psychological interventions, interventions that disproportionately draw from the developmental, educational, social, and personality psychology literature. Our claims should be interpreted with those caveats in mind—it is plausible, and even probable, that they do not generalize to all areas of psychology (but see Sakaluk et al., 2019, for related concerns in clinical psychology). Moreover, we evaluated implementation readiness through the lens of one implementation science framework—RE-AIM; conclusions may differ if the articles were to be coded with a different implementation science lens—we encourage others to use our open data set and test this possibility.
Final note
The American Psychological Association, Association for Psychological Science, and the Society for Personality and Social Psychology—and probably other professional psychological organizations—have all recently encouraged psychologists to expand our impact by studying complex issues facing the field and broader society. We personally endorse these calls and believe the best psychological research is that which simultaneously advances theory and is beneficial for practice (see also Berkman, 2017; Lewis & Wai, 2021). To quote Stenhouse (2017), The problem of achieving widespread real-world effectiveness is too important, and too interesting, to leave it to ad-hoc, non-scientific study by practitioners alone. There should be debate about exactly how to address this problem, whether RE-AIM is the ideal framework to address it, and how much resources should be devoted to it. There should not be debate about whether this research needs to be done at all. (p. 767)
Harnessing that collective desire for broader impact, however, and readying the field to truly “give psychology away” requires paying more attention to and reporting on the factors that our intended recipients need to take what we have to give.
Footnotes
Acknowledgements
We thank our team of research assistants, Laura Bagwell, Anjuli Fink, Lucus Tse, and Caiwei Zhu, for their time and efforts in coding the articles for this study. We also thank Cornell University librarian Amelia Kallaher for the guidance provided in developing the literature search strategy for this review. We also thank Patrick Forscher, Bobbie Spellman, and two anonymous reviewers for their helpful comments on an earlier version of the manuscript.
Transparency
Action Editor: Laura A. King
Editor: Laura A. King
