Psychological Science Stepping Up a Level

Transparency and Openness Promotion (TOP) Levels and Where We Are Now

Efforts to increase the rigor and reproducibility of research practices in psychological science, and to make research processes more open and transparent, have been informed by the Guidelines for Transparency and Openness Promotion (TOP) in Journal Policies and Practices (aka “The TOP Guidelines”; Nosek et al., 2015). The TOP Guidelines specify eight standards of open and transparent reporting and publishing, each of which has three levels (higher levels impose more stringent requirements).

Since the introduction of the TOP standards, Psychological Science has operated at the equivalent of Level 1 for the majority of them, as described in Table 1. The one exception is the standard for Replication, at which we operate at Level 3. The journal also recognizes authors’ efforts to increase open science by awarding badges to articles that feature one or more preregistered studies (Preregistered badge), have openly accessible data (Open Data badge), and have openly available materials (Open Materials badge). Since the badge system was introduced in 2014, participation in these practices has grown substantially, as reflected in Table 2. Moreover, the number of articles published with at least one badge reached 83% in 2021; the number of articles published with all three badges hit 30%. Note that Psychological Science does not have a separate badge for analysis code and has not maintained data on sharing of this element of research materials. Nor have we differentiated preregistration of studies from preregistration of analysis plans.

OPEN IN VIEWER

Table 2.

Growth in Number of Badges Awarded in Psychological Science for Open Data, Open Materials, and Preregistration From 2014 to 2021

Badge type	2014	2015	2016	2017	2018	2019	2020	2021
Data (%)	16	33	38	61	66	63	73	78
Materials (%)	15	24	30	48	48	48	54	57
Preregistered (%)	0	2	3	13	25	32	40	42

TOP Levels and Where We Are Going

We have taken important steps in the direction of increasing rigor and reproducibility and opening science. Yet there is more ground to be covered. In December 2019, the Association for Psychological Science (APS) Board of Directors passed a motion to appoint an ad hoc committee chaired by then Board member Maryanne Garry on scientific transparency and open-science practices to identify and advance APS priorities in these areas. The committee came to be known as the Open and Transparent Practices Committee (OTPC). In June 2021, the APS Board approved a motion from the OTPC endorsing the move to TOP Level 2 for APS’s empirical journals. As Editor in Chief of Psychological Science, it has been my pleasure to work with OTPC members Katie Corker, Maryanne Garry (Chair), Morton Ann Gernsbacher, and Marcel van Assen, as well as Publications Committee Chair Elaine Walker, to increase the standards to which we hold articles published in Psychological Science. Keeping in mind the broader conceptualization of open science, even as we make the necessary moves, we are exercising care to ensure that the journal remains open regardless of authors’ abilities to meet what in some cases may be challenging requirements.

In total, there are eight TOP standards (listed in Table 1). For the standard of Replication, Psychological Science already operates at Level 3, so no action is required to meet the journal’s aspirations. For the other seven standards, actions are required to move Psychological Science from Level 1 to Level 2.

For the Citation and Design and Analysis Transparency standards, only minor changes are necessary to adopt Level 2 requirements; most of the action necessary to meet them will be taken at the time of manuscript submission. Submitting authors will notice a new suite of questions about their research practices, each of which is designed to ensure that citation and other standards have been met. Responses to these questions will become part of the PDF of a manuscript created at the time of submission, thus making them readily accessible to editors and reviewers.

For the Preregistration standards (preregistration of studies and analysis plans), Psychological Science will adopt more rigorous standards for examination of preregistration plans to ensure that the plans were actually carried out. At present, review of the plans by the editorial team is uneven. Going forward, we will ensure more uniform examination before certifying an article as eligible for a Preregistered badge. As well, going forward, for a study to earn a Preregistered badge, the preregistration must include not only the study design but also the analysis plan. In recognition that authors may have existing preregistrations that do not feature this element, this requirement will take effect as of January 1, 2023. For all manuscripts submitted as of February 1, 2022, the availability of a preregistration plan will be featured in the main body of the text in an explicit Open Practices Statement. The Statement will indicate whether each of the studies reported in that article was preregistered and if so, how the preregistration can be accessed. More information on the Open Practices Statement is provided below.

For the Data, Analytic Methods, and Research Materials Transparency standards, Level 2 of the TOP Guidelines requires that all data used in reported analyses, all code and scripts used to analyze the data, and all study materials (e.g., questionnaires, stimuli) be made publicly available in a trusted third-party, open-access repository (e.g., OSF) or be included in the manuscript itself (e.g., as Supplemental Material). The formal descriptions of these three standards indicate that exceptions to these requirements should be “rare” and reserved for “legal or ethical” reasons. In recognition of the possibility that authors may have other valid reasons that data, analysis code/scripts, and/or materials cannot be made publicly accessible, Psychological Science is not at this time adopting these strict criteria for third-party open access as a criterion for publication in the journal. Yet we are taking several steps toward full satisfaction of Level 2 criteria, all of which will be effective for manuscripts submitted as of February 1, 2022:

Benchmarks and an Empirical Approach to Further Adoption

In our ongoing efforts to increase the rigor and reproducibility of the work published in Psychological Science, and to further open science, we have set benchmarks for voluntary adoption of open-science practices in the domains in which we are not fully adopting Level 2 standards for publication (data, analysis code/scripts, materials) and will continue to award badges to articles that meet them.

As reflected in Table 2, in 2021, contributing authors to Psychological Science voluntarily made their data publicly accessible 78% of the time. We thank our authors for making these important contributions to open science and encourage them to continue this practice. For authors who are not yet among the growing population, we strongly encourage you to join.

We have set a benchmark of 90% voluntary sharing of data for articles published in 2022. Further, for manuscripts submitted as of February 1, 2022, to earn a badge for Open Data, authors will also be required to deposit their analysis code/scripts in addition to their raw data (many authors do this already—thank you). The rationale for this is that valid replication attempts require not only the “ingredients” of the analysis (i.e., the raw data) but also the procedure for how they were treated.

For authors who do not make their data and analysis code/scripts publicly accessible in a third-party, open-access repository (or include them as Supplemental Material), we would like to know why. This will help us to establish benchmarks for further progress on this front and also to create a more comprehensive set of valid exceptions to the practice. In pursuit of this information, submitting authors will notice a new checklist asking them to indicate whether their data and analysis code/scripts are publicly available and if they are not, which of a number of options best explains why. Authors will have the option of “choose not to answer” as well as “other,” with a text box for explanation. Explanations are entirely voluntary, yet we hope authors will provide them so that we can better understand the determinants of this practice. Eventually, assuming Psychological Science fully adopts Level 2 for data and analysis code/script sharing, this information will help us to craft an expanded set of valid exceptions to the requirement (i.e., beyond legal or ethical constraints).

Voluntary sharing of materials is not as common as sharing of data. As reflected in Table 2, in 2021, contributing authors to Psychological Science made their materials publicly accessible 57% of the time. We thank our authors for making these important contributions to open science and encourage them to continue this practice. As with data sharing, for authors who are not yet among the growing population who share their materials, we strongly encourage adoption of the practice.

We have set a benchmark of 72% voluntary participation for articles published in 2022. The benchmark is more modest than that for data sharing, in recognition that the growth curve for materials sharing is shallower than that for data sharing. Also as for data sharing, for authors who do not make their study materials publicly accessible in a third-party, open-access repository (or include them as Supplemental Material), we would like to know why. Submitting authors will be asked whether their materials are publicly available and if they are not, which of a number of options best explains why. Authors will have the option of “choose not to answer” as well as “other,” with a text box for explanation. Again, explanations are entirely voluntary yet will help us to better understand the determinants of this practice. Eventually, assuming Psychological Science fully adopts Level 2 for materials sharing, this information will help us to craft a set of valid exceptions to the requirement.

New: Open Practices Statement in Submitted Manuscripts

At present, authors who preregister their studies and/or who make their data and/or materials publicly available indicate so in their manuscripts. Empirical articles published in the journal include an Open Practices section describing whether and where such materials are available. However, this section is added during the production process; how authors convey this information in their submitted manuscripts during the peer-review process is highly varied. It is not unusual that information on each of these badge-eligible practices is sprinkled throughout the Method section or even relegated to footnotes. This makes it difficult for editors and reviewers to know what is—and is not—available and where and how they can access it.

To facilitate access to information on open-science practices during peer review, as of February 1, 2022, we will require each submitted manuscript reporting new empirical work to include an Open Practices Statement indicating whether each of the studies reported was preregistered and whether the data, analysis code/scripts, and/or materials are available on a permanent third-party archive (or are included in Supplemental Material). This statement will appear at the end of the Introduction, before the first Method section; this statement will not count against the word limits. If data, analysis code/scripts, and/or materials are not available on a permanent third-party archive or included in Supplemental Material, authors will be required to inform readers how they may access study-related materials and data or of restrictions thereon. These statements should be direct and to the point. Authors are not required to offer explanations for the restrictions, but they may do so if they choose. Example statements are provided in the Submission Guidelines (https://www.psychologicalscience.org/publications/psychological_science/ps-submissions). Commentaries that do not report new data do not need to include an Open Practices Statement.

Conclusions

Psychological Science is among the leaders in the open-science movement. Authors publishing in the journal have voluntarily adopted many open-science practices, some of which are recognized with badges. In efforts to continue the progress toward open science and to further increase the rigor and reproducibility of the work published in the journal, Psychological Science is taking steps to adopt TOP Level 2 standards. In some cases, we are embracing the guidelines whole cloth, virtually immediately. For other guidelines, we are adopting a more conservative approach. The intention of this process is to ensure that we do not eliminate Psychological Science as the publication outlet of choice for authors who have not only legal or ethical constraints on open access but also other considerations. By taking an empirically based approach to understanding the determinants of open-access sharing behavior, we stand to further strengthen the foundation of psychological science, to further increase access to the practices of psychological science as well as its products, and to leave no author (or potential author) behind.