What do Incoming University Students Believe About Open Science Practices in Psychology?

Abstract

Background

Understanding students’ naive conceptions about the norms that guide scientific best practice is important so that teachers can adapt to students’ existing understandings.

Objective

We examined what incoming undergraduate students of psychology believe about reproducibility and open science practices.

Method

We conducted an online survey with participants who were about to start their first course in psychology at a university (N = 239).

Results

When asked to indicate how a researcher should conduct her study, most students endorsed several open science practices. When asked to estimate the proportion of published psychological studies that follow various open science practices, participants’ estimates averaged near 50%. Only 18% of participants reported that they had heard the term “replication crisis.”

Conclusion

Despite media attention about the replication crisis, few incoming psychology students in our sample were familiar with the term. The students were nevertheless in favour of most open science practices, although they overestimated the prevalence of some of these practices in psychology.

Teaching Implications

Teachers of incoming psychology students should not assume pre-existing knowledge about open science or replicability.

Keywords

open science psychology teaching reproducibility replication

Methodological and reporting practices in psychology have changed over the last decade, partly precipitated by the “replication crisis”: The discovery that many close replications of published studies did not replicate the original findings (Klein et al., 2014, 2018). Of course, psychology is not alone; the replication crisis extends to other fields, including economics (Camerer et al., 2016) and medicine (e.g., Errington et al., 2021). These apparent problems with replication have led to various potential solutions to make research practices more reproducible and transparent, including more published replication studies (see Brandt et al., 2014), more thorough reporting of methods and results (Wigboldus & Dotsch, 2016), open sharing of data (see Meyer, 2018), and preregistration of data collection and analytic plans (Nosek et al., 2018). These calls for more reproducible and transparent research practices have prompted the discipline to examine researchers’ norms, attitudes, and practices (e.g., Baker, 2016; Beaudry et al., 2021; Harris et al., 2018).

These changes present significant pedagogical challenges for the teacher of psychology. Keeping textbooks and other instructional materials up to date is difficult when supposedly well-established findings are contradicted by new replications emerging at a rapid pace. Furthermore, knowledge of emerging methodological practices is crucial for graduate student training. Even for students who do not go on to conduct research, an understanding of contemporary methodological practices—and problematic methodological practices—is essential for becoming informed and critical consumers of psychological knowledge. Studies have explored strategies for educating psychology students about replicability and open science practices (e.g., Chopik et al., 2018; Grahe et al., 2012; Jekel et al., 2020). These initiatives may help ingrain open science norms and change attitudes about research practices, but we know little about what students know or believe about open science research practices prior to entering the university classroom. This knowledge could be useful for two main reasons.

First, unlike some psychological phenomena in undergraduate courses (e.g., the internal workings of human senses), the replication crisis is frequently discussed in mainstream and social media (e.g., O’Grady, 2020; Yong, 2016, 2018). As such, incoming students might have some knowledge about these issues prior to their formal studies. Teaching methods should be informed by understanding students’ pre-existing levels of knowledge.

Second, anecdotal evidence suggests that students being taught about open science practices and reproducibility reforms (e.g., open sharing of data and code) are sometimes surprised that these are not already standard practice. This could imply a need for educators to reinforce and build on students’ “naive” impressions rather than radically altering their understanding of research practices. On the other hand, some evidence suggests that undergraduate students may quickly begin to engage in practices that hamper reproducibility. For example, Moran et al. (2021) found that 26.5% of the undergraduate students in their sample admitted to “Conducting multiple statistical analyses on the same dataset in an attempt to find a statistically significant result” (p. 14; i.e., p-hacking), while 9.6% reported rounding down p-values to meet a significance threshold. Knowing the extent to which incoming psychology students may—or may not—be “naive open scientists” could help guide pedagogical approaches.

To examine this, we conducted a descriptive study, asking incoming students in undergraduate psychology courses about their beliefs regarding reproducibility and open science practices. Our survey encompassed questions concerning norms (how students felt research should be conducted), norms in practice (how students believe psychological research is conducted), and replicability (how replicable students believe psychological research is). Our study was exploratory (see Wagenmakers et al., 2012) and descriptive; as such, we did not specify or test hypotheses.

Method

We preregistered the exploratory nature of this study on the Open Science Framework (Beaudry et al., 2022a). Our materials, data, and analytic code are are openly available in the same project (Beaudry et al., 2022a). Ethical approval was granted by Swinburne University of Technology.

Procedure

Contacts from tertiary institutions in Australia, New Zealand, the UK, and the USA invited students enrolled in their first-year psychology course to participate in the online survey hosted on Qualtrics. These contacts were identified through several avenues: direct contact with our colleagues, a session at the international meeting of the Society for the Improvement of Psychological Science (2019), and a Twitter thread with >15,000 impressions. We did not have inclusion criteria with respect to country, but most contacts who advertised the study were from the United Kingdom, Australia, or New Zealand.

Each contact sent their students invitations to participate via email or a course website 1–2 weeks before their course began. The invitation specified that students needed to be 18 years or older and starting their first course/unit of study in psychology at a university within the following month and included a link to the survey. The measures and items in the survey were presented in the same order for all participants.

Participants

Of those who started the survey (n = 423), we screened out 184 participants who were ineligible based on our preregistered exclusion criteria. Specifically, we screened out people who were younger than 18 years (n = 7). We also screened out individuals who had already started a psychology unit at a university, regardless of whether they had completed it (n = 72); were not enrolled in a psychology unit at a university (n = 1); or did not answer this question (n = 14). We excluded additional responses that Qualtrics flagged as spam (n = 1) or as a survey preview (n = 1). Finally, we excluded participants who met all other eligibility criteria, but did not respond to any of the main survey items (n = 88).

Of the eligible 239 participants, most reported that they were 18–24 years old (n = 193); 21 reported that they were 25–34 years old, 12 were 35–44 years old, 12 were 45–54 years old, and 1 was 55–64 years old. Most participants (n = 192) reported that they were women, 46 reported that they were men, and 1 participant reported non-binary gender.

Most participants attended university in Australia (n = 160), the United Kingdom (n = 63), or New Zealand (n = 11). Nearly all reported graduating from high school or secondary school (n = 230). Of those who attended high school, fewer than half completed high school psychology courses (n = 106). Participants reported 26 different nationalities; the most common were from Australia (n = 125), the United Kingdom (n = 43), and China (n = 15). About two-thirds of participants reported that at least one of their parents attended university (n = 157).

Material

Open Science Norms and Counternorms

To measure norms and counternorms, we focused on participants’ evaluations of 10 specific open science practices that reflected diverse open science norms (see Table 1). Although we acknowledge the debatable merits of some of these practices (Rubin, 2017, 2020; Szollosi et al., 2020), we selected the items to be emblematic of common concerns in the open science movement.

Table 1.

Norm and Counternorm Items

Practice	Norm	Counternorm
Critical thinking	Deborah used previously published research to inform her research study. Deborah should be critical of the findings published in journals because published research can be wrong.	Deborah used previously published research to inform her research study. Deborah should accept the findings published in journals because journals would not publish research with errors.
Preregistration	Before collecting her data, Deborah should write down what her hypotheses are and how she plans to collect and analyse data. She should then save her plan in an online registry so others can tell what methods and analyses she will use to test her hypotheses after collecting data.	Deborah should decide which data analyses are suitable to test her hypotheses only after looking at her data.
Registered Reports	Deborah should submit a plan for her study to a journal to be checked by experts (peer reviewed) before she collects and analyses data.	Deborah should submit her study to a journal to be checked by experts (peer reviewed) only after she has finished collecting and analysing her data.
p-hacking	Deborah should report the findings of all analyses of her data that she conducts.	It would be good scientific practice for Deborah to run many different analyses of her data, and report only those that produce interesting findings.
HARKing	Deborah should only describe her study as a test of a hypothesis if she decided on her hypothesis and how she would test it before she started collecting data.	Once Deborah has analysed her results, it would be good scientific practice for her to write her manuscript as if she predicted those results from the beginning.
Information for replicability	When reporting the findings of her study, Deborah should describe how she completed the study in enough detail that another researcher could repeat her entire study without having to check any details with her—even if this means including lots of “boring” practical details in her report, or in an appendix.	When reporting the findings of her study, it would be good scientific practice for Deborah to gloss over some of the practical details so she can tell a good story.
Preprints	Deborah should post a manuscript describing the findings of her study openly online as soon as it is complete, even if the manuscript has not yet been checked by experts (peer reviewed) and accepted for publication in a journal.	Deborah should not post a manuscript describing the findings of her study online until after it has been checked by experts (peer reviewed) and accepted for publication in a journal.
Open materials	Deborah should share the written materials and measures for her study openly online so that other researchers and members of the public can access and use them.	Deborah should keep the written materials and measures for her study protected, so that only she and her research team can access them.
Open data	When Deborah publishes her study, she should post the anonymous responses from participants online so that anyone can access and use the responses in their own research.	Deborah should keep the participants’ responses from her study protected, so that only she and her research team can access them.
Open access	Once her study is complete, Deborah should publish her findings in a journal that is free for others to access.	Once her study is complete, Deborah should publish the findings in the most prestigious journal she can, even if that journal charges others a fee to access the report.

Note. Participants responded to each item in the context of the scenario, “Imagine that Deborah is a psychology researcher who has designed a study to test a specific hypothesis.” Responses were provided on a Likert-type response scale ranging from 1 (strongly disagree) to 5 (strongly agree)

Each practice was elucidated by a pair of items (pro-open science; counter-open science) in the context of a common scenario (see the note accompanying Table 1). Participants rated their agreement separately for each norm and counternorm on a 1 (strongly disagree) to 5 (strongly agree) scale. We used the scenario of a specific researcher rather than referring to psychology researchers in general because the items were more concrete, engaging, and clear.

Beliefs About the Practice of Psychological Science

We measured participants’ normative beliefs about how researchers actually conduct psychological science. We asked, “When you consider all of the psychology studies conducted globally each year, what percentage of them would have the following characteristics?” These practices corresponded approximately to our ten norm–counternorm pairs, with two exceptions (see Table 2). We excluded the critical thinking norm because it did not refer to externally observable behavior, and we included data availability on request out of interest. Participants provided their estimate on a visual slider from 0% to 100%.

Table 2.

Norms in Practice Items

Practice	Item
Preregistration	The plan for collecting and analysing data for the study is posted in an online registry before data collection starts.
Registered Reports	A plan for the study is submitted to a journal to be checked by experts (peer reviewed) before the data is collected and analysed.
p-hacking	The researchers run many different analyses of the data, and report only those that produce interesting findings.
HARKing	The researchers describe their results as if they predicted those results from the beginning, even if that isn’t actually true.
Information for replicability	The researchers provide enough practical detail in their written report that another researcher could fully repeat (replicate) their study without needing to ask any questions of the original researchers.
Preprints	A manuscript describing the findings of the study is posted openly online as soon as it is complete, even if the manuscript has not yet been checked by experts (peer reviewed) and accepted for publication in a journal.
Open materials	The written materials and measures for the study are posted openly online so that other researchers and members of the public can access and use them.
Open data	Once the study is published, the anonymous responses from participants are posted openly online so that anyone can access and use the responses in their own research.
Data on request	Once the study is published, the anonymous responses from participants are available to other researchers on request.
Open access	The research report describing the study and its findings is published in a journal that is free for anyone to access.

Beliefs About Replications

Two questions measured participants’ beliefs about the replicability of results in psychological science. The first asked participants to:

Imagine that a set of researchers selected 100 published studies about psychology, and repeat each study again, exactly how it was described in the original report but with a new set of participants. We call repeating a study like this a ‘replication study.’ At a guess, how many of these replication studies do you think would produce the same conclusions as the original studies?

Estimates were recorded using a visual slider from 0% to 100%. A second question asked participants whether they had heard of the replication crisis. Those who had heard of the crisis were asked two additional open-ended questions: (1) describe the replication crisis, and (2) name the scientific disciplines most affected by the replication crisis.

Results

As preregistered, we calculated the difference in means between responses to each norm item and its accompanying counternorm. For this analysis, we removed three cases with missing values. Table 3 displays the means and standard deviations for each norm, counternorm, and the difference values. See Supplemental Materials for the correlations between norm and counternorm items in Beaudry et al. (2022b).

Table 3.

Means and Standard Deviations for Norms, Counternorms, and the Difference Between Norms and Counternorms, With 95% Confidence Intervals (in Square Brackets)

Practice	Norm	Counternorm	Difference
Critical thinking	4.32 (0.82)	2.41 (1.06)	1.91 (1.47) [1.73, 2.10]
HARKing	3.89 (1.08)	1.66 (1.07)	2.23 (1.58) [2.03, 2.43]
Information for replicability	4.42 (0.98)	1.51 (0.94)	2.91 (1.50) [2.72, 3.10]
Open access	3.86 (0.90)	2.53 (0.92)	1.33 (1.48) [1.14, 1.52]
Open data	2.66 (1.21)	3.49 (1.25)	−0.83 (2.17) [−1.10, −0.55]
Open materials	3.69 (1.02)	2.49 (1.14)	1.20 (1.96) [0.95, 1.45]
p-hacking	4.34 (0.90)	2.13 (1.27)	2.21 (1.79) [1.98, 2.43]
Preprints	1.60 (0.77)	4.36 (0.80)	−2.76 (1.42) [−2.94, −2.58]
Preregistration	3.97 (1.07)	2.75 (1.24)	1.22 (1.64) [1.01, 1.43]
Registered Reports	3.36 (1.26)	3.56 (1.19)	−0.20 (2.13) [−0.47, 0.07]

In general, the students seemed to endorse open science practices: For seven of the 10 open science practices, agreement with the norm item was stronger than for the corresponding counternorm item. However, participants’ endorsement of norms varied substantially across different practices. Participants most strongly endorsed norms that researchers should provide sufficient information about studies to permit replicability, be critical of published findings, and avoid p-hacking. They also tended to agree that researchers should avoid HARKing, share papers as open access, share open materials, and preregister their studies. On the other hand, they tended to disagree with the practice of sharing preprints prior to peer review and, to a lesser extent, with sharing open data.

Norms in Practice

Participants were also asked to estimate the proportion of psychology studies globally that apply each of 10 practices (see Table 4). For many of these practices, participants’ estimates were near the midpoint of 50%. Nonetheless, participants perceived some practices as being applied relatively infrequently: For example, the mean estimate of the prevalence of HARKing, preprint sharing before publication, and open data sharing were all near 30%. See Supplemental Materials to see the distribution for each item in Beaudry et al. (2022b). Participants perceived that the most common practice was providing enough information in written reports to permit replication without asking questions of the original researchers

Table 4.

Means, Standard Deviations, and 95% Confidence Intervals for the Prevalence Ratings for Each Research Practice

Practice	Prevalence Ratings
Preregistration	49.96 (25.19) [46.62, 53.29]
Registered Reports	52.02 (28.57) [48.26, 55.79]
p-hacking	44.61 (28.09) [40.89, 48.33]
HARKing	33.58 (26.54) [30.06, 37.09]
Information for replicability	63.41 (25.19) [60.07, 66.75]
Preprints	49.36 (25.47) [45.97, 52.75]
Open materials	30.49 (26.20) [26.99, 33.99]
Open data	50.27 (26.51) [46.74, 53.80]
Data on request	49.80 (23.89) [46.63, 52.97]
Open access	30.94 (22.75) [27.90, 33.98]

The Replication Crisis

Interestingly, only 18% of participants indicated they had heard of “the replication crisis.” Most (74%) indicated that they had not heard of the crisis; 8% did not respond.

Participants who indicated they had heard of the replication crisis were asked, “What do you think ‘the replication crisis’ is?” Two independent coders marked the responses as accurate or inaccurate; they agreed on 92.50% of the descriptions and resolved disagreements through discussion. Of the 40 participants who answered this question, most (82.50%) provided responses consistent with the conventional interpretation of the term (i.e., describing a realisation that some scientific findings cannot be replicated). For example, “Where previously trusted studies have been replicated the results were not able to be replicated.” Only seven responses indicated a misunderstanding of the term (e.g., “Plagiarism”).

Participants’ estimates of the replicability of psychological studies were variable (see Figure 1). The mean (M = 56.56) was near the midpoint of 50%, but with substantial variability around this estimate (SD = 21.49, range = 0–100).

Figure 1.

Histogram of participants’ estimates of the percentage of psychology studies that replicate

Discussion

Despite efforts to improve the replicability of research and the corresponding media attention, most incoming psychology students in our sample were unaware of the replication crisis. Encouragingly, on the other hand, most tended to endorse open science norms more-so than counternorms. Notably this was not the case for all open science practices (namely sharing of preprints and open data). It may well be that scientific training has enshrined peer review as an essential practice that is robust to change.

Participants’ estimates of the proportion of published psychological studies that apply open science practices tended to hover around 50%. Although this could imply that participants literally believe these practices are roughly as commonly applied as not, it might also represent participants’ uncertainty or indifference about the application of these practices. This said, 50% was the modal response for only a minority of practices (open access, data availability on request).

That the mean estimated prevalence was no higher than 64% for any practice is inconsistent with the sometime-reported claim that incoming students tend to naively assume that science is conducted openly. Nevertheless, the students appear to have at least somewhat over-estimated the prevalence of some open science practices. For example, on average students estimated that nearly a third of published psychological studies share data openly and that nearly half of studies are preregistered. In contrast, an examination of a random sample of 250 psychology articles published between 2014 and 2017 (Hardwicke et al., 2021) found that just 2% shared open data and 3% were preregistered.

Strengths and Limitations

Despite a moderate sample size, our results provide an imperfect picture of new psychology students’ methodological understandings. Our use of convenience samples coupled with the imprecision of polling students’ impressions at one point in time suggest that we can make only tentative generalisations to a wider population of incoming psychology students. In particular, the fact that our sample mostly comprised students from Australia, NZ, or the United Kingdom limits the extent to which our findings can be generalised to other populations.

Our questions focused primarily on attitudes and beliefs so that we could better document the implicit understanding that guided students’ thoughts about the research process. However, we did not probe students’ factual knowledge of most methodological findings or concepts, except the replication crisis. It is quite possible that reform in secondary school instruction has improved students’ understandings of more fundamental statistical (e.g., confidence interval estimation) and design (e.g., direct replications) considerations without necessarily influencing their open science attitudes and beliefs. Yet, research at the tertiary level casts doubt that the methodological reform in psychology has had much impact on the content of our teaching (Friedrich et al., 2018).

Although we have interpreted participants’ responses to the norm and counternorm items as reflecting students’ personal endorsement of practices, their perceptions of what we might perceive as the “right” answers could have also influenced their responses. Differences between endorsement of norms and counternorms might have also been affected by wording differences between the matched options; for example, the HARKing, information for replicability, and p-hacking counternorm items included the phrase “good scientific practice” whereas their matched norm items did not. More broadly, our choices of terminology in constructing items might have affected participants’ responses to some degree. For example, the reference to keeping data “protected” in the open data counternorm item may have nudged participants towards endorsing this item because failing to “protect” participants’ data might sound irresponsible. Future research could examine the influence of revised language on participants’ responses.

We did not investigate the degree to which results differed according to whether or not participants had studied psychology in high school. Interested readers could conduct this and other comparisons using our open data.

Conclusion and Teaching Implications

Our findings suggest that these incoming psychology students had a degree of sympathy for open science norms. Nevertheless, despite significant media attention, few had heard of the replication crisis. Similarly, the students seemed to be relatively unfamiliar with responses to the replication crisis (e.g., open science practices). It is therefore important that teachers of psychology not assume pre-existing knowledge among incoming students.

For teachers interested in systematically surveying their students’ existing understanding of open science practices in psychology, our survey questions could be a useful resource (available from Beaudry et al., 2022a).

Supplemental Material

Supplemental Material - What do Incoming University Students Believe About Open Science Practices in Psychology?

Supplemental Material for What do Incoming University Students Believe About Open Science Practices in Psychology? by Jennifer L. Beaudry, Matt N. Williams, Michael C. Philipp, and Emily J. Kothe in Teaching of Psychology

Footnotes

Acknowledgments

The idea of this research was conceived at the 2018 University of Queensland Open Science Conference and further developed at the 2019 Society for the Improvement of Psychological Science Unconference, What do first year psychology students think about (open) science? Our special thanks to those who contributed to the data collection effort including: Benjamin Le, Katie Gilligan-Lee, Annayah Prosser, Ian Fairholm, Jennie Ferrell, Kait Clark, Catherine Orr, Nick Burns, Rachel Searston, Rachel Stephens, Matthew Thompson, Aaron Drummond, Kathryn McGuigan, and Peter Cannon.

Author Contributions

JLB: Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Project administration, Validation, Visualization, Writing - original draft, Writing - review & editing; MNW: Investigation, Methodology, Validation, Writing - original draft; MCP: Conceptualization, Investigation, Methodology, Project administration, Writing - original draft; EJK: Conceptualization, Data curation, Validation, Visualization, Writing - review & editing.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iDs

Jennifer L. Beaudry

Matt N. Williams

Michael C. Philipp

Emily J. Kothe

Supplemental Material

The Supplemental material for this article is available online.

References

Baker

(2016). 1,500 scientists lift the lid on reproducibility. Nature News, 533(7604), 452–454. https://doi.org/10.1038/533452a

Beaudry

J. L.

Kaufman

Johnstone

Given

(2021). Swinburne Open Science Task Force survey report. https://doi.org/10.17605/OSF.IO/VPWF7

Beaudry

J. L.

Williams

M. N.

Philipp

M. C.

Kothe

E. J.

(2022a). What do incoming university students believe about open science practices in psychology? [Preregistration, data set, materials, analytic code]. https://doi.org/10.17605/OSF.IO/FZ6YA

Beaudry

J. L.

Williams

M. N.

Philipp

M. C.

Kothe

E. J.

(2022b). What do incoming university students believe about open science practices in psychology? [Supplemental materials]. https://doi.org/10.17605/OSF.IO/A4FKS

Brandt

M. J.

IJzerman

Dijksterhuis

Farach

F. J.

Geller

Giner-Sorolla

Grange

J. A.

Perugini

Spies

J. R.

van’t Veer

(2014). The replication recipe: What makes for a convincing replication? Journal of Experimental Social Psychology, 50(1), 217–224. https://doi.org/10.1016/j.jesp.2013.10.005

Camerer

C. F.

Dreber

Forsell

T.-H.

Huber

Johannesson

Kirchler

Almenberg

Altmejd

Chan

Heikensten

Holzmeister

Imai

Isaksson

Nave

Pfeiffer

Razen

(2016). Evaluating replicability of laboratory experiments in economics. Science, 351(6280), 1433–1436. https://doi.org/10.1126/science.aaf0918

Chopik

W. J.

Bremner

R. H.

Defever

A. M.

Keller

V. N.

(2018). How (and whether) to teach undergraduates about the replication crisis in psychological science. Teaching of Psychology, 45(2), 158–163. https://doi.org/10.1177/0098628318762900

Errington

T. M.

Mathur

Soderberg

C. K.

Denis

Perfito

Iorns

Nosek

B. A.

(2021). Investigating the replicability of preclinical cancer biology. eLife, 10, e61701. https://doi.org/10.7554/eLife.71601

Friedrich

Childress

Cheng

(2018). Replicating a national survey on statistical training in undergraduate psychology programs: Are there “new statistics” in the new millennium? Teaching of Psychology, 45(4), 312–323. https://doi.org/10.1177/0098628318796414

10.

Grahe

J. E.

Reifman

Hermann

A. D.

Walker

Oleson

K. C.

Nario-Redmond

Wiebe

R. P.

(2012). Harnessing the undiscovered resource of student research projects. Perspectives on Psychological Science, 7(6), 605–607. https://doi.org/10.1177/1745691612459057

11.

Hardwicke

T. E.

Thibault

R. T.

Kosie

J. E.

Wallach

J. D.

Kidwell

M. C.

Ioannidis

J. P. A.

(2021). Estimating the prevalence of transparency and reproducibility-related research practices in psychology (2014–2017). Perspectives on Psychological Science, 17(1), 239–251. https://doi.org/10.1177/1745691620979806

12.

Harris

J. K.

Johnson

K. J.

Carothers

B. J.

Combs

T. B.

Luke

D. A.

Wang

(2018). Use of reproducible research practices in public health: A survey of public health analysts. PloS One, 13(9), e0202447. https://doi.org/10.1371/journal.pone.0202447

13.

Jekel

Fiedler

Allstadt Torras

Mischkowski

Dorrough

A. R.

Glöckner

(2020). How to teach open science principles in the undergraduate curriculumthe Hagen cumulative science project. Psychology Learning & Teaching, 19(1), 91–106. https://doi.org/10.1177/1475725719868149

14.

Klein

R. A.

Ratliff

K. A.

Vianello

Adams

R. B.,

Jr. Bahník

Š.

Bernstein

M. J.

Bocian

Brandt

M. J.

Brooks

Brumbaugh

C. C.

Cemalcilar

Chandler

Cheong

Davis

W. E.

Devos

Eisner

Frankowska

Furrow

Galliani

E. M.

Hasselman

(2014). Investigating variation in replicability: A “many labs” replication project. Social Psychology, 45(3), 142–152. https://doi.org/10.1027/1864-9335/a000178

15.

Klein

R. A.

Vianello

Hasselman

Adams

B. G.

Adams

R. B.

Alper

Aveyard

Axt

J. R.

Babalola

M. T.

Bahník

Š.

Batra

Berkics

Bernstein

M. J.

Berry

D. R.

Bialobrzeska

Binan

E. D.

Bocian

Brandt

M. J.

Busching

Rédei

A. C.

Nosek

B. A.

(2018). Many labs 2: Investigating variation in replicability across samples and settings. Advances in Methods and Practices in Psychological Science, 1(4), 443–490. https://doi.org/10.1177/2515245918810225

16.

Meyer

M. N.

(2018). Practical tips for ethical data sharing. Advances in Methods and Practices in Psychological Science, 1(1), 131–144. https://doi.org/10.1177/2515245917747656

17.

Moran

Richard

Wilson

Twomey

Coroiu

(2021). I know it’s bad but I have been pressured into it”: Questionable research practices among psychology students in Canada. Canadian Psychology/Psychologie Canadienne. Advance online publication. https://doi.org/10.31234/osf.io/kjby3

18.

Nosek

B. A.

Ebersole

C. R.

DeHaven

A. C.

Mellor

D. T.

(2018). The preregistration revolution. Proceedings of the National Academy of Sciences, 115(11), 2600–2606. https://doi.org/10.1073/pnas.1708274114

19.

O’Grady

(2020) Psychology’s replication crisis inspires ecologists to push for more reliable research. https://www-sciencemag-org-s.web.bisu.edu.cn/news/2020/12/psychology-s-replication-crisis-inspires-ecologists-push-more-reliable-research

20.

Rubin

(2017). When does HARKing hurt? Identifying when different types of undisclosed post hoc hypothesizing harm scientific progress. Review of General Psychology, 21(4), 308–320. https://doi.org/10.1037/gpr0000128

21.

Rubin

(2020). Does preregistration improve the credibility of research findings? The Quantitative Methods for Psychology, 16(4), 376–390. https://doi.org/10.20982/tqmp.16.4.p376

22.

Szollosi

Kellen

Navarro

D. J.

Shiffrin

van Rooij

Van Zandt

Donkin

(2020). Is preregistration worthwhile? Trends in Cognitive Sciences, 24(2), 94–95. https://doi.org/10.1016/j.tics.2019.11.009

23.

Wagenmakers

E.-J.

Wetzels

Borsboom

van der Maas

H. L. J.

Kievit

R. A.

(2012). An agenda for purely confirmatory research. Perspectives on Psychological Science, 7(6), 632–638. https://doi.org/10.1177/1745691612463078

24.

Wigboldus

D. H. J.

Dotsch

(2016). Encourage playing with data and discourage questionable reporting practices. Psychometrika, 81(1), 27–32. https://doi.org/10.1007/s11336-015-9445-1

25.

Yong

(2016). Psychology’s replication crisis can’t be wished away. The Atlantic. https://www.theatlantic.com/science/archive/2016/03/psychologys-replication-crisis-cant-be-wished-away/472272/

26.

Yong

(2018). Psychology’s replication crisis is running out of excuses. The Atlantic. https://www.theatlantic.com/science/archive/2018/11/psychologys-replication-crisis-real/576223/

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.17 MB