No Replication,No Trust? How Low Replicability Influences Trust in Psychology

Abstract

In the current psychological debate, low replicability of psychological findings is a central topic. While the discussion about the replication crisis has a huge impact on psychological research, we know less about how it impacts public trust in psychology. In this article, we examine whether low replicability damages public trust and how this damage can be repaired. Studies 1–3 provide correlational and experimental evidence that low replicability reduces public trust in psychology. Additionally, Studies 3–5 evaluate the effectiveness of commonly used trust-repair strategies such as information about increased transparency (Study 3), explanations for low replicability (Study 4), or recovered replicability (Study 5). We found no evidence that these strategies significantly repair trust. However, it remains possible that they have small but potentially meaningful effects, which could be detected with larger samples. Overall, our studies highlight the importance of replicability for public trust in psychology.

Keywords

replicability public trust replication crisis open science

A trustworthy reputation is crucial for psychology. Researchers in psychology aim to have societal impact and to inform practitioners and policy makers. Additionally, psychological research relies on public funding and participation. In turn, when facing complex scientific questions, decision makers seek advice from the (psychological) science community (Bromme & Thomm, 2016). If based on robust evidence, these well-informed decisions can lead to improved individual and societal outcomes (Ruggeri et al., 2019). Thus, public trust is not only crucial for psychology itself but also for the public. What happens to this trust when psychological findings often fail to replicate?

This is a relevant question since many prominent studies indeed suggest a low replicability of psychological findings. For example, the Reproducibility Project: Psychology replicated 100 psychological studies, and only about one third to one half of the original findings were replicated (Open Science Collaboration, 2015). This low replication rate often serves as an illustration for a “replication crisis” (e.g., Anderson & Maxwell, 2017).

Low Replicability Might Damage Public Trust in Psychology

Many researchers see crises positively since they play a central role in the advancement of sciences and show that science self-corrects (Kuhn, 1970; Vazire, 2018). Indeed, in the course of the replication crisis, psychology has gone through major changes. Currently, journals and scientific societies encourage open science practices (e.g., Schönbrodt, Gollwitzer, & Abele-Brehm, 2017), and many researchers implement open science (Lindsay & Nosek, 2018). These changes are often seen as a major advancement of psychological science: Researchers argue that the rate of scientific progress is likely to increase (Vazire, 2018), that the widespread use of preregistration will increase the interpretability and credibility of research findings (Nosek, Ebersole, DeHaven, & Mellor, 2018), and that open science will liberate researchers and foster creativity (Frankenhuis & Nettle, 2018).

Nevertheless, there are reasons to assume that information about low replicability might damage public trust in psychology. While findings regarding the effects of (scientific) uncertainty on audience reactions are rather inconclusive (for a review, see van der Bles et al., 2019), some studies suggest that nonscientists react negatively to scientific uncertainty. For example, nonscientists who perceive scientific evidence as uncertain further perceive the corresponding research field as less valuable (Broomell & Kane, 2017). Likewise, even modest amounts of scientific dissent reduce public support for government policies and lead to disagreement with the scientific consensus (Aklin & Urpelainen, 2014). Similarly, low replicability might also result in reputational damage and diminished public trust (Białek, 2018; Chopik, Bremner, Defever, & Keller, 2018; Fanelli, 2018). We test this hypothesis in the present article.

How Can Public Trust Be Repaired?

If low replicability damages public trust, an important question for the psychological science community is if and how this damage can be repaired. We tested the following three theory-based and commonly used approaches to repair public trust.

Repairing Trust Through Increased Transparency

Transparency signals that there is nothing to hide and thus repairs trust (Bachmann, Gillespie, & Priem, 2015). Indeed, one major response to the replication crisis is the open science movement (Frankenhuis & Nettle, 2018; LeBel, Campbell, & Loving, 2017). Central aspects of this movement, such as preregistrations, open data, and open materials, aim to increase the transparency of psychological research (Miguel et al., 2014; Nosek et al., 2015). Thus, building on the idea that transparency can repair trust, information about the open science movement might help to repair public trust in psychology.

Repairing Trust Through Explanations

The causes and responsibilities of a transgression are often not evident (Bachmann et al., 2015). Explanations of the causes of a transgression can help to repair trust by establishing a shared understanding of what and why the transgression happened (Bachmann et al., 2015; Dirks, Lewicky, & Zaheer, 2009). If low replicability violates the public expectations of reliable published findings, the public may perceive low replicability as a transgression. In this case, explanations for low replicability could be an effective trust repair strategy.

Considering the replication crisis, two major explanations emerged. Some scholars argue that questionable research practices (QRPs) are the main reason for the replication crisis (e.g., Sijtsma, 2016; Simmons, Nelson, & Simonsohn, 2011). Other scholars attribute failed replications to hidden moderators and the high context sensitivity of psychological effects (e.g., Stroebe & Strack, 2014; Van Bavel, Mende-Siedlecki, Brady, & Reinero, 2016). While this debate has not been settled, it is an additional open question whether any of those explanations—QRPs versus hidden moderators—would repair public trust damaged by low replicability.

Repairing Trust by Restoring the Status Quo

Trust can further be repaired by restoring the status quo, as norms and expectations are also restored (Dirks et al., 2009). Before the replication crisis, the majority of replications in psychology journals reported similar findings to their original studies (Makel, Plucker, & Hegarty, 2012), and it was thus likely assumed that psychology is highly replicable. To restore this status quo, psychological science would thus need to achieve high levels of replicability. Indeed, many new methodological standards in psychology aim at increasing replicability (Cook, Lloyd, Mellor, Nosek, & Therrien, 2018; Van Bavel et al., 2016). If those standards succeed, the status quo belief that psychology is a highly replicable science might be restored. Eventually, this increase in replicability might also lead to a restoration of public trust.

Overview of Studies

We conducted five studies to test whether low replicability damages public trust and whether this damage can be repaired. Study 1 examined whether trust in psychology correlates with expected replicability. Study 2 experimentally tested whether low replicability causes reduced trust, which we replicated in Study 3. Moreover, Studies 3–5 tested different commonly used trust-repair strategies: information about increased transparency (Study 3), explanations for low replicability (Study 4), and information about increased replicability (Study 5). Participants who took part in one of our studies were not allowed to participate in subsequent studies. We relied on Amazon’s Mechanical Turk (MTurk) workers in all studies since they are significantly more socioeconomically and ethnically diverse and presumably less likely to have prior knowledge of the replication crisis, compared with a student sample (Casler, Bickel, & Hackett, 2013).

We include all studies we conducted and report all collected variables and all conditions included in the study designs across all studies. We preregistered all analyses presented in the article (except for specifically highlighted correlations presented in Table 1), and we report all preregistered analyses in either the article or the Supplemental Materials. We discuss the central preregistered hypotheses when introducing each study. All analyses with a preregistered hypothesis are accompanied by one-sided p values. All participants who completed our studies were included in the analyses except if they met preregistered exclusion criteria. All materials, data, analyses syntaxes, and preregistrations are shared on https://osf.io/9ba28/

Table 1.

Descriptive Statistics for the “Trust in Psychology” Measure Across Studies.

Study	Sample Size^a	Mean	Standard Deviation	Correlation With Replicability^b
Study 1	270	5.06	1.27	r(268) = .329***
Study 2	268	4.94	1.36
1. Low replicability	88	4.63	1.34	r(86) = .205
2. Medium replicability	90	4.94	1.44	r(88) = .424***
3. High replicability	90	5.26	1.24	r(88) = .239*
Study 3	294	4.91	1.32
1. Low replicability	100	4.66	1.36	r(98) = .218*
2. Transparency	96	4.80	1.33	r(94) = .295**
3. High replicability	98	5.27	1.20	r(96) = .373***
Study 4	281	4.67	1.21
1. Low replicability	94	4.76	1.34	—
2. Hidden moderators	96	4.80	1.09	—
3. QRPs	91	4.45	1.16	—
Study 5	278	4.40	1.20
1. Low replicability	98	4.35	1.30	—
2. Still low replicability	90	4.32	1.14	—
3. Now high replicability	90	4.54	1.14	—

Note. QRPs = questionable research practices.

*p < .05. **p < .01. ***p < .001.

^aNumber of participants who completed the trust in psychology measure.

^bIn Study 1, this correlation refers to the preregistered correlation of the trust in psychology measure (ranging from 1 to 7) with the estimated replication rate. In Studies 2 and 3, this refers to the not preregistered correlation with the manipulation check “Psychological research is replicable.” This manipulation check was not administered in Studies 4 and 5.

Study 1

Study 1 investigated which replication rate nonscientists assume for psychological studies and whether their expected replication rate correlates with their trust in psychology and their perceived value of psychological science. We expected positive correlations.

Method

Participants and Design

Participants completed a short online study on MTurk website for US$0.50. The sample size was set to 266, based on an a priori power analysis for 95% power (one-sided α of .05) to detect a small to moderate effect of r = .2, that would be typical for similar social psychological research (Richard, Bond, & Stokes-Zoota, 2003). The final sample was slightly larger as is often the case in online studies and consisted of 271 participants (54.3% male; age: M = 33.7 years, SD = 8.9). No participants were excluded from the analyses. A sensitivity analysis showed that our final sample had a high chance (1 − β = .80, one-sided α = .05) to detect a correlation of r = .15 and a very high chance (1 − β = .95, one-sided α = .05) to detect r = .20.

Procedure

Participants read a short, jargon-free description of the Reproducibility Project: Psychology (for details, see https://osf.io/9ba28/). Participants then guessed how many of these 100 original findings were successfully replicated.

Afterward, participants indicated their trust in psychology with 5 items (e.g., “I trust the psychological science community to do what is right”; 1 = strongly disagree, 7 = strongly agree; α = .90; adapted from Nisbet, Cooper, & Garrett, 2015). Although we conveniently call this measure “trust in psychology” throughout this article, it is important to note that it was designed to measure institutional trust in the (psychological) science community (Nisbet et al., 2015). Alternatively, trust in psychology could, for example, also be conceptualized as trust in psychological findings (Anvari & Lakens, 2019) or in the scientific methods used by psychologists. However, for nonscientists, the scientific community might be the most vivid aspect of psychology. Moreover, prior research showed that the used trust in psychology measure is affected by dissonant science communication (Nisbet et al., 2015), so it could be particularly suitable to capture potential effects of (expected) low replicability.

Although this measure showed acceptable to excellent reliability in Studies 1–5, it showed a poor confirmatory model fit across most indices and studies. We believed this to likely be due to the reverse coding of items and found that accounting for this drastically improves the model fit and does not change the pattern of our results (see Supplemental Materials).

As an additional dependent variable, we measured participants perceived value of psychological science with 4 items (e.g., “Please rate the societal benefit of research produced by psychological science”; α = .80; 1 = very low, 5 = very high; adapted from Broomell & Kane, 2017). In all studies, perceived value showed a similar result pattern to trust in psychology. All preregistered analyses regarding perceived value can be found in the Supplemental Material. Finally, participants indicated whether they knew the results of the Reproducibility Project: Psychology and completed demographics.

Results

Eleven participants (4.0%) said they had heard of the Reproducibility Project: Psychology but only one participant reported to know the results. This participant, however, indicated an incorrect replication rate of 14 of 100 studies. On average, participants estimated that 60.9 of 100 studies could be replicated (SD = 22.9). Descriptive statistics for the trust in psychology measure across all studies and conditions are presented in Table 1. As predicted, the higher participants estimated the replication rate, the more they trusted psychology, r(268) = .329, one-sided p < .001, 95% CI [.218, .431], see Figure 1. Perceived value showed similar results to trust in psychology (see Supplemental Materials).

Figure 1.

Relationship between the estimated replication rate and trust in psychology in Study 1. Histograms show the distribution of each measure.

Study 2

Study 1 provided correlational evidence that expected replicability is related to public trust in psychology. Building on Study 1, we employed an experimental approach to test causality. We expected low replicability (compared with high replicability) to reduce trust in psychology and to reduce the perceived value of psychological science.

Method

Participants and Design

Participants completed a short online study on MTurk for US$0.50. We randomly assigned participants to three conditions (low replicability, medium replicability, and high replicability). We set sample size to 264, based on an a priori power analysis for 95% power (one-sided α of .05) to find a moderate effect (d = .5), that would be typical for similar social psychological research (Richard et al., 2003), requiring 88 participants per cell (the same power analysis was applied to Studies 3, 4, and 5). The final sample consisted of 269 participants (59.9% male; age: M = 34.59 years, SD = 10.74). No participants were excluded from the analyses. A sensitivity analysis showed that our final sample had a high chance (1 − β = .80, one-sided α = .05) to detect a difference of d = .37 between the low replicability and any of the two other conditions and a very high chance (1 − β = .95, one-sided α = .05) to detect d = .50.

Procedure

Participants read the same description of the Reproducibility Project: Psychology as in Study 1. This time, however, participants received information about the results. Depending on their condition, participants were told that of the 100 investigated studies, 39 (low replicability condition), 61 (medium replicability condition), or 83 (high replicability condition) could be successfully replicated. These values were based on the estimated replication rates found in Study 1: 61 represents the mean estimated replication rate in Study 1, and 39 and 83 represent the mean ±1 SD in Study 1. Afterward, participants responded to three text-understanding items and a manipulation check (“Psychological research is replicable”; 1 = strongly disagree, 7 = strongly agree). Then, they filled out the 5 items from Study 1 to measure trust in psychology (α = .92). We also measured participants’ perceived value of psychological science and various individual differences as preregistered potential moderators (beliefs about science, error culture, error attribution style; see Supplemental Materials for details). Finally, participants completed a brief demographic questionnaire and were debriefed.

Results

The manipulation check suggested that the manipulation was successful (see Supplemental Materials). A one-way analysis of variance revealed significantly different levels of trust in psychology between the three conditions, F(2, 265) = 4.86, p = .008, η² = .04, 90% CI [.01, .07], see Figure 2. Participants in the low replicability condition indicated a significantly lower trust in psychology than participants in the high replicability condition, t(176) = 3.25, one-sided p < .001, d = .49, 95% CI [.19, .79].

Figure 2.

Pirate plot (Phillips, 2017) showing trust in psychology in the different replicability conditions in Study 2. The black dots represent the raw data which is shown with smoothed densities indicating the distributions in each condition. The central tendency is the mean, and the intervals represent two standard errors around the mean.

Further analyses indicated that participants in the exploratory medium replicability condition tended to be more trustful than the participants in the low replicability condition, t(176) = 1.48, one-sided p = .070, d = .22, 95% CI [−.07, .52], and less trustful than participants in the high replicability condition, t(176) = 1.59, one-sided p = .057, d = .24, 95% CI [−.06, .53], but these differences were not significant. Perceived value showed similar results to trust in psychology (see Supplemental Materials).

Study 3

As expected, Studies 1 and 2 provided evidence that low replicability, compared with high replicability, reduces trust in psychology. In Study 3, we replicated the trust-damaging effect of low replicability and tested whether informing participants about the open science movement and increased transparency of psychological science would repair trust damaged by low replicability. We expected low replicability (compared with high replicability) to reduce trust in psychology (as in Study 2). Crucially, we expected information about increased transparency to repair public trust, compared with information about low replicability only.

Method

Participants and Design

Three hundred and four participants were recruited to complete a short online study on MTurk for US$0.60 each. Compared with Study 2, we increased the target sample size to 300 to compensate for potential exclusions. Indeed, seven participants were excluded for meeting the preregistered exclusion criteria (failing more than one text-understanding questions). We randomly assigned participants to three conditions (low replicability, low replicability but transparency, and high replicability). The final sample consisted of 297 participants (56.9% male; age: M = 35.7 years, SD = 11.6). A sensitivity analysis showed that our final sample had a high chance (1 − β = .80, one-sided α = .05) to detect a difference of d = .36 between the low replicability and the low replicability but transparency condition and a very high chance (1 − β = .95, one-sided α = .05) to detect d = .47.

Procedure

Participants read the same description of the Reproducibility Project: Psychology as in Study 2. Once again, participants were told that of the investigated 100 studies, 39 (low replicability condition) or 83 (high replicability condition) could be successfully replicated. In a third condition (low replicability but transparency condition), participants were also told that 39 studies could be replicated but that psychology has since then become much more open and transparent. This comprehensible information described major aspects of the open science movement, including preregistration, open data, and open materials, and highlighted that those measures contribute to increased transparency (for details, see https://osf.io/9ba28/).

Afterward, participants responded to 3 text-understanding items and two manipulation checks (“Psychological research is replicable”; “Psychological research is transparent”; 1 = strongly disagree, 7 = strongly agree). Then, they filled out the 5 items from Studies 1 and 2 to measure their trust in psychology (α = .86). Participants also completed a brief demographic questionnaire and were debriefed.

Results

The manipulation check suggested that the manipulations were successful (see Supplemental Materials).

As predicted, and replicating our prior findings, participants in the low replicability condition indicated a significantly lower trust in psychology than participants in the high replicability condition, t(196) = 3.36, one-sided p < .001, d = .48, 95% CI [.19, .76], see Figure 3. However, contrary to our prediction, participants in the low replicability but transparency condition did not indicate a significantly higher trust in psychology than participants in the low replicability condition, t(194) = 0.74, one-sided p = .231, d = .11, 95% CI [−.18, .39].

Figure 3.

Pirate plot showing trust in psychology in the different replicability and transparency conditions in Study 3.

Finally, participants in the low replicability but transparency condition indicated a significantly lower trust in psychology compared with participants in the high replicability condition, t(192) = 2.60, p = .010, d = .37, 95% CI [.09, .66].

Study 4

Study 3 found no evidence that increased transparency can repair trust. While this approach focused on the consequences of the replication crisis, another approach to repair public trust might be to explain the causes of low replicability. Thus, in Study 4, we tested the effectiveness of the two most common explanatory strategies: hidden moderators and QRPs. We expected the hidden moderator explanation to lead to a higher trust in psychology than the QRPs explanation. However, given the little effectiveness of our trust repair strategy in Study 3, we had no clear hypotheses on whether any of the explanations would be able to repair trust compared with providing no explanation, so we preregistered these analyses as exploratory.

Method

Participants and Design

Three hundred and three participants were recruited to complete a short online study on MTurk for US$0.60. Twenty participants were excluded for meeting the preregistered exclusion criteria (failing more than one text understanding questions). We randomly assigned participants to three conditions (low replicability condition, hidden moderator condition, and QRPs condition). The final sample consisted of 283 participants (55.5% male; age: M = 36.5 years, SD = 12.0). Sensitivity analyses showed that our final sample had a high chance (1 − β = .80, α = .05) to detect a difference of d = .41 between the low replicability and any of the two explanation conditions and a very high chance (1 − β = .95, α = .05) to detect d = .53.

Procedure

Participants read the same description of the Reproducibility Project: Psychology as in Studies 2 and 3. All participants were told that of the 100 investigated studies, 39 were successfully replicated. Depending on their condition, participants received no explanation (low replicability condition), an explanation stating that QRPs caused the low replication rate (QRPs condition) or an explanation stating that hidden moderators caused the low replication rate (hidden moderator condition). In the QRPs condition, participants read that researchers “primarily look for new and spectacular results which can lead to bad research practices, for example, repeating an experiment until a surprising effect emerges. Often researchers only publish the spectacular results, while less spectacular—but potentially more reliable—results end up in a drawer somewhere.”

In contrast to that, participants in the hidden moderator condition learned that:

When studying humans, unknown or hidden factors such as individual differences between participants, participants’ current state, or minimal differences in the experimental procedure can affect the results. It is very difficult to always have absolute control over these conditions and keep possible influencing factors constant (for details, see https://osf.io/9ba28/).

Afterward, participants responded to 3 text-understanding items and two manipulation checks (1 = Unknown or hidden factors explain the low replication rate, 2 = QRPs explain the low replication rate; 1 = strongly disagree, 7 = strongly agree). Participants did not answer the manipulation checks in the low condition to avoid highlighting these explanations to control participants. Then, participants filled out the same 5 items used in Studies 1–3 to measure their trust in psychology (α = .80). Participants also completed a brief demographic questionnaire and were debriefed.

Results

Manipulation checks indicated that the manipulation was successful (see Supplemental Materials).

Participants in the QRPs condition showed a significantly lower trust in psychology than participants in the hidden moderator condition, t(185) = 2.11, one-sided p = .018, d = .31, 95% CI [.02, .60], see Figure 4. However, the low replicability condition, which served as a control condition, did not differ significantly from the hidden moderator condition, t(188) = 0.20, p = .839, d = .03, 95% CI [−.27, .32], nor from the QRPs condition, t(183) = −1.68, p = .094, d = .25, 95% CI [.04, .54], which showed an even lower trust than the low replicability condition.

Figure 4.

Pirate plot showing trust in psychology in the different explanation conditions in Study 4.

Study 5

Neither increased transparency (Study 3) nor explanations (Study 4) significantly repaired trust. One reason for this might be that we did not provide information about both, the causes and adequate solutions to the crisis, in one study. If nonscientists intuitively do not believe that nontransparent practices (e.g., QRPs) cause low replicability, increasing transparency would not be a sensible response to low replicability. Thus, it might be necessary to inform nonscientists about both: QRPs as a cause of low replicability and transparency as an adequate solution. Whereas a QRP explanation on its own might even damage public trust (see Study 4), such an explanation could be especially effective when combined with information about increased transparency.

Moreover, we did not provide information about whether increased transparency was indeed effective in increasing replicability. Thus, we conducted Study 5 to address these concerns. In this final study, we tested whether public trust can be repaired by providing participants with both, information about the causes of, and adequate solutions for low replicability, and by informing them that these solutions successfully restored high replicability. We expected successfully restored replicability to lead to increased trust in psychology.

Method

Participants and Design

Three hundred and four participants were recruited to complete a short online study on MTurk for US$0.50 each. We again used an increased target sample size of 300 to compensate for potential exclusions. Twenty-six participants were excluded for meeting the preregistered exclusion criteria (failing more than one text-understanding questions). We randomly assigned participants to three conditions (low replicability condition, “now high” replicability condition, and “still low” replicability condition). The final sample consisted of 278 participants (64.7% male; age: M = 33.8 years, SD = 11.0). Sensitivity analyses showed that our final sample had a high chance (1 − β = .80, α = .05) to detect a difference of d = .36 between the low replicability and any of the two other conditions and a very high chance (1 − β = .95, α = .05) to detect d = .48.

Procedure

Participants read the same description of the Reproducibility Project: Psychology as in Studies 2, 3, and 4 and additionally learned that the Reproducibility Project: Psychology was published in 2015. All participants read that of the 100 investigated studies, 39 were successfully replicated. In the low replicability condition, participants received no further information. In the still low replicability and now high replicability conditions, participants received an explanation that QRPs caused the low replication rate, but that this issue was now addressed through the open science movement and the increased transparency of psychological science. In the still low condition, which served as an additional control group, participants learned that these measures were not successful. Concretely, they were informed that an (alleged) new systematic replication project in 2018 revealed that of 100 studies conducted under the new transparency guidelines, still only 41 could be successfully replicated. In contrast, in the now high replicability condition, participants learned that those measures were very successful since the alleged new replication project in 2018 revealed that now 83 of 100 recent studies could be successfully replicated (for details, see https://osf.io/9ba28/). Afterward, participants responded to 3 text-understanding items and to the manipulation check (“Psychological research is now more replicable”). Participants did not fill out the manipulation check in the low replicability condition, which received no information about the change in replicability. Then, participants answered the 5 items from Studies 1 to 4 to measure their trust in psychology (α = .73). Participants also completed a brief demographic questionnaire and were debriefed.

Results

According to our manipulation checks, the manipulation was successful (see Supplemental Materials).

Participants in the now high replicability condition did not show significantly higher trust in psychology than participants in the still low replicability condition, t(178) = 1.29, one-sided p = .099, d = .19, 95% CI [−.10, .49], or participants in the low replicability condition (M = 4.35, SD = 1.30), t(186) = 1.04, one-sided p = .149, d = .15, 95% CI [−.14, .44]; see Figure 5.

Figure 5.

Pirate plot showing trust in psychology in the different replicability conditions in Study 5.

General Discussion

Our results show that concerns about reduced public trust in light of the replication crisis are justified. Across three studies (Studies 1–3), we find correlational and experimental evidence that low replicability reduces trust in psychology. Studies 1 and 2 suggest that not only public trust but also the perceived value of psychological science is damaged by low replicability. Moreover, Studies 3–5 found no evidence that commonly used trust repair strategies significantly repair this damaged trust in psychology.

So does low replicability damage public trust beyond repair? Although sensitivity analyses showed that it is unlikely that the tested strategies have large trust-repairing effects, they also suggest that we had no sufficient power to rule out small, but potentially meaningful effects, which could only be detected with larger samples (equivalence tests and Bayes factors in line with this argumentation are presented in the Supplemental Materials). Our findings thus do not allow us to conclude that the tested strategies are certainly ineffective. However, given the nonsignificant observed effects of trust repair strategies, our findings also do not provide evidence for the effectiveness of the tested strategies on trust in psychology.

Hence, the critical question is: What should psychological researchers do if they encounter low replicability? Considering that replication studies have limitations and that there is often no consensus about their interpretation (Gilbert, King, Pettigrew, & Wilson, 2016), one could potentially argue that psychologists should avoid informing the public about low replicability. However, this nontransparent approach would be ethically problematic and violates, for example, the American Psychological Association (APA) Ethics Code (see APA, 2017, pp. 3–4). Moreover, failed attempts to cover up problematic research findings might reduce public trust even more (Leiserowitz, Maibach, Roser-Renouf, Smith, & Dawson, 2013). Therefore, covering up low replicability is neither an ethical nor an effective way to handle the problem.

A more promising approach to maintaining the public trust might be to substantially improve the replicability of psychological research findings. Although Study 5 remains inconclusive about whether this is an effective strategy to repair the public trust directly after a replication crisis, Studies 1–3 provide evidence that high replicability in the first place results in increased trust in psychology. Thus, if replicability is constantly high, public trust in psychology might rise. Currently, there is considerable debate about whether constantly high replicability is a worthwhile goal for psychological science. For example, Baumeister (2016) discussed whether a strong focus on replicability could potentially reduce the likelihood of discoveries and the progress and influence of the field. Moreover, scholars debate whether conducting direct replications is after all meaningful (cf. Simons, 2014; Stroebe & Strack, 2014). Although we do not directly speak to these arguments, our work suggests that the debate should also consider the reputational benefits associated with high replicability.

However, it is important to note that we communicated information about low replicability in the form of very short texts, inspired by brief news reports. Potentially, an in-depth explanation of the replication crisis and the open science movement might lead to less negative, or even positive, audience reactions. This is especially likely for highly science-interested audiences who would be willing to engage with such a detailed explanation. Indeed, recent research suggests less negative consequences in such a situation: After a 1-hr lecture on the replication crisis, psychology students’ attitudes toward psychology remained relatively stable (Chopik et al., 2018).

Moreover, we conceptualized trust in psychology as trust in the psychological science community. Trust in psychology could however also refer to trust in psychological findings. Since low replicability typically refers to past findings, it seems possible that low replicability of past findings does not necessarily damage trust in future findings (Anvari & Lakens, 2019). Likewise, it is possible that the damaged trust in the psychological science community does not generalize to future generations of psychological researchers educated under new, more rigorous methodological guidelines.

Overall, our studies highlight the crucial importance of replicability for public trust in psychology. Thus, the immense effort of the psychological science community to increase replicability is not only scientifically important but also highly relevant to psychology’s public reputation. This is especially important in the current political climate, where the credibility of scientific evidence is questioned and science is threatened by defunding (Fanelli, 2018; Yong, 2017).

Supplemental Material

Supplemental Material, Supplemental_Materials_No_Replication,_No_Trust - No Replication, No Trust? How Low Replicability Influences Trust in Psychology

Supplemental Material, Supplemental_Materials_No_Replication,_No_Trust for No Replication, No Trust? How Low Replicability Influences Trust in Psychology by Tobias Wingen, Jana B. Berkessel and Birte Englich in Social Psychological and Personality Science

Footnotes

Acknowledgments

The authors would like to thank Nicolas Alef, Amelie Conrad, Elisabeth Jackson, and Estella Umbach for their support with the preparation of materials. They also thank Alexandra Fleischmann and Oscar Lecuona for their helpful comments. Finally, they thank the Graduate School of the Faculty of Human Sciences at the University of Cologne for providing a travel grant to present this work at an international conference. The authors would like to dedicate this paper to the memory and friendship of Prof. Birte Englich (co-author of this study), who passed away in September 2019, shortly after the manuscript was accepted for publication. The authors are grateful for her enduring trust and support, during this research project and way beyond.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Jana B. Berkessel

Supplemental Material

The supplemental material is available in the online version of the article.

References

Aklin

Urpelainen

(2014). Perceptions of scientific dissent undermine public support for environmental policy. Environmental Science & Policy, 38, 173–177. doi:10.1016/j.envsci.2013.10.006

Anderson

S. F.

Maxwell

S. E.

(2017). Addressing the “replication crisis”: Using original studies to design replication studies with appropriate statistical power. Multivariate Behavioral Research, 52, 305–324. doi:10.1080/00273171.2017.1289361

Anvari

Lakens

. (2019, September 9). The replicability crisis and public trust in psychological science. Retrieved from https://doi.org/10.31234/osf.io/vtmpc.

American Psychological Association. (2017). Ethical principles of psychologists and code of conduct. Washington, DC: Author.

Bachmann

Gillespie

Priem

(2015). Repairing trust in organizations and institutions: Toward a conceptual framework. Organization Studies, 36, 1123–1142.

Baumeister

R. F.

(2016). Charting the future of social psychology on stormy seas: Winners, losers, and recommendations. Journal of Experimental Social Psychology, 66, 153–158. doi:10.1016/j.jesp.2016.02.003

Białek

. (2018). Replications can cause distorted belief in scientific progress: BBS commentary by Białek on Zwaan et al. PsyArXiv. doi:10.31234/osf.io/8a4h6

Bromme

Thomm

(2016). Knowing who knows: Laypersons’ capabilities to judge experts’ pertinence for science topics. Cognitive Science, 40, 241–252.

Broomell

S. B.

Kane

P. B.

(2017). Public perception and communication of scientific uncertainty. Journal of Experimental Psychology: General, 146, 286–304. doi:10.1037/xge0000260

10.

Casler

Bickel

Hackett

(2013). Separate but equal? A comparison of participants and data gathered via amazon’s MTurk, social media, and face-to-face behavioral testing. Computers in Human Behavior, 29, 2156–2160.

11.

Chopik

W. J.

Bremner

R. H.

Defever

A. M.

Keller

V. N.

(2018). How (and whether) to teach undergraduates about the replication crisis in psychological science. Teaching of Psychology, 45, 158–163.

12.

Cook

B. G.

Lloyd

J. W.

Mellor

Nosek

B. A.

Therrien

W. J.

(2018). Promoting open science to increase the trustworthiness of evidence in special education. Exceptional Children, 85, 104–118.

13.

Dirks

K. T.

Lewicky

R. J.

Zaheer

(2009). Repairing relationships within and between organizations: Building a conceptual foundation. The Academy of Management Review, 34, 68–84.

14.

Fanelli

(2018). Opinion: Is science really facing a reproducibility crisis, and do we need it to? Proceedings of the National Academy of Sciences, 115, 2628–2631.

15.

Frankenhuis

W. E.

Nettle

(2018). Open science is liberating and can foster creativity. Perspectives on Psychological Science, 13, 439–447.

16.

Gilbert

D. T.

King

Pettigrew

Wilson

T. D.

(2016). Comment on “estimating the reproducibility of psychological science.” Science, 351, 1037–1037.

17.

Kuhn

T. S.

(1970). The structure of scientific revolutions (2nd ed., enlarged). Chicago, IL: University of Chicago Press.

18.

LeBel

E. P.

Campbell

Loving

T. J.

(2017). Benefits of open and high-powered research outweigh costs. Journal of Personality and Social Psychology, 113, 230–243. doi:10.1037/pspi0000049

19.

Leiserowitz

A. A.

Maibach

E. W.

Roser-Renouf

Smith

Dawson

(2013). Climategate, public opinion, and the loss of trust. American Behavioral Scientist, 57, 818–837.

20.

Lindsay

D. S.

Nosek

B. A.

(2018). Preregistration becoming the norm in psychological science. APS Observer, 31. Retrieved from https://www.psychologicalscience.org/observer/preregistration-becoming-the-norm-in-psychological-science

21.

Makel

M. C.

Plucker

J. A.

Hegarty

(2012). Replications in psychology research: How often do they really occur? Perspectives on Psychological Science, 7, 537–542. doi:10.1177/1745691612460688

22.

Miguel

Camerer

Casey

Cohen

Esterling

K. M.

Gerber

… Imbens

. (2014). Promoting transparency in social science research. Science, 343, 30–31.

23.

Nisbet

E. C.

Cooper

K. E.

Garrett

R. K.

(2015). The partisan brain: How dissonant science messages lead conservatives and liberals to (dis)trust science. The ANNALS of the American Academy of Political and Social Science, 658, 36–66. doi:10.1177/0002716214555474

24.

Nosek

B. A.

Alter

Banks

G. C.

Borsboom

Bowman

S. D.

Breckler

S. J.

… Christensen

. (2015). Promoting an open research culture. Science, 348, 1422–1425.

25.

Nosek

B. A.

Ebersole

C. R.

DeHaven

A. C.

Mellor

D. T.

(2018). The preregistration revolution. Proceedings of the National Academy of Sciences, 115, 2600–2606. doi:10.1073/pnas.1708274114

26.

Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349. doi:10.1126/science.aac4716

27.

Phillips

. (2017). Yarrr: A companion to the e-book “yarrr!: The pirate’s guide to R.” Retrieved from https://CRANR-Project.Org/Package=Yarrr

28.

Richard

F. D.

Bond

C. F.

Jr Stokes-Zoota

J. J.

(2003). One hundred years of social psychology quantitatively described. Review of General Psychology, 7, 331–363.

29.

Ruggeri

Ojinaga-Alfageme

Benzerga

Berkessel

Hlavová

Kunz

Sampat

. (2019). Evidence-based policy. In Ruggeri

(Ed.), Behavioral insights for public policy: Concepts and case (pp. 17–40). London, England: Routledge.

30.

Schönbrodt

Gollwitzer

Abele-Brehm

. (2017). Data management in psychological science: Specification of the DFG guidelines. doi:10.31234/osf.io/vhx89

31.

Sijtsma

(2016). Playing with data—Or how to discourage questionable research practices and stimulate researchers to do things right. Psychometrika, 81, 1–15. doi:10.1007/s11336-015-9446-0

32.

Simmons

J. P.

Nelson

L. D.

Simonsohn

(2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359–1366. doi:10.1177/0956797611417632

33.

Simons

D. J.

(2014). The value of direct replication. Perspectives on Psychological Science, 9, 76–80.

34.

Stroebe

Strack

(2014). The alleged crisis and the illusion of exact replication. Perspectives on Psychological Science, 9, 59–71.

35.

Van Bavel

J. J.

Mende-Siedlecki

Brady

W. J.

Reinero

D. A.

(2016). Contextual sensitivity in scientific reproducibility. Proceedings of the National Academy of Sciences, 113, 6454–6459. doi:10.1073/pnas.1521897113

36.

van der Bles

A. M.

van der Linden

Freeman

A. L.

Mitchell

Galvao

A. B.

Zaval

Spiegelhalter

D. J.

(2019). Communicating uncertainty about facts, numbers and science. Royal Society Open Science, 6, 181870.

37.

Vazire

(2018). Implications of the credibility revolution for productivity, creativity, and progress. Perspectives on Psychological Science, 13, 411–417.

38.

Yong

(2017). How the GOP could use science’s reform movement against it. The Atlantic. Retrieved from https://www.theatlantic.com/science/archive/2017/04/reproducibility-science-open-judoflip/521952/

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.03 MB