Psychology’s Replication Crisis and the Grant Culture: Righting the Ship

Abstract

The past several years have been a time for soul searching in psychology, as we have gradually come to grips with the reality that some of our cherished findings are less robust than we had assumed. Nevertheless, the replication crisis highlights the operation of psychological science at its best, as it reflects our growing humility. At the same time, institutional variables, especially the growing emphasis on external funding as an expectation or de facto requirement for faculty tenure and promotion, pose largely unappreciated hazards for psychological science, including (a) incentives for engaging in questionable research practices, (b) a single-minded focus on programmatic research, (c) intellectual hyperspecialization, (d) disincentives for conducting direct replications, (e) stifling of creativity and intellectual risk taking, (f) researchers promising more than they can deliver, and (g) diminished time for thinking deeply. Preregistration should assist with (a), but will do little about (b) through (g). Psychology is beginning to right the ship, but it will need to confront the increasingly deleterious impact of the grant culture on scientific inquiry.

Keywords

replication grants preregistration confirmation bias

“Science is a struggle for truth against methodological, psychological, and sociological obstacles” (Fanelli & Ioannidis, 2013, p. 15031).

To my lights, that’s a decent working definition of science. It underscores the point that science is a set of methodological safeguards designed to compensate for biases that can contribute to erroneous conclusions and that these biases stem from a myriad of sources, some of them institutional.

Although I have dedicated much of my career to the study of personality disorders, I have become increasingly interested in how cognitive biases foster pseudoscientific and otherwise questionable practices in the clinical world (Lilienfeld, Ammirati, & David, 2012). In recent years, I have belatedly come to recognize that biases plaguing academicians and administrators pose as grave a threat to our field’s scientific status as do those afflicting practitioners of dubious clinical methods.

The Replication Crisis: A Heterodox Take

I assume that I need not remind readers that recent high-profile replication failures have cast doubt on numerous psychological phenomena, such as behavioral priming, the effects of facial expressions on emotions, and the impact of power posing on behavior, that were previously assumed to be well-established (Jarrett, 2016). To many observers, these are dark days for psychological science. Consider Jeffrey Lieberman, recent past president of the American Psychiatric Association, who tweeted that “psychology is in shambles” (https://mobile.twitter.com/DrJlieberman/status/638654836842430465) following a Science article reporting that the lion’s share of 100 published findings in social and cognitive psychology were not replicable (Open Science Collaboration, 2015).

My view differs. I see the replication crisis as among psychological science’s finest hours. Our field’s recent bout of soul searching is greatly overdue. For too long, we have been insufficiently self-critical and prone to advancing confident claims on the basis of provisional, at times even flimsy, evidence. Fortunately, researchers in some domains of psychology, especially social and cognitive psychology, are slowly but surely coming to grips with the fact that p hacking and other questionable research practices (QRPs) are not conducive to accruing a body of reproducible scientific knowledge and are proposing remedies, such as preregistration, for diminishing their frequency. There is reason for cautious optimism that scholars in other psychological subdisciplines will follow suit.

Our current travails notwithstanding, we are not in shambles; we are gradually righting the ship. We are beginning to do what healthy sciences do: engaging in self-scrutiny to root out errors in our web of beliefs (McFall, 1997). We are also becoming humbler in our assertions, although we certainly have a way to go in this regard. When the dust settles, we should emerge bruised but stronger.

At the same time, we have been insufficiently proactive in confronting institutional obstacles that stand in the way of our scientific progress. One could point the finger at many such obstacles, including the relentless publish-or-perish pressure on pre-tenure researchers. In this commentary, I focus on one impediment that warrants considerably more attention than it has received: the influence of the grant culture on psychological science.

The Corrosive Impact of the Grant Culture on Psychological Science

Some readers may recall Willy Sutton, the thief who, when asked by a reporter why he robbed banks, purportedly replied “because that’s where the money is.” Whether or not Sutton actually said this (he denied it), the Willy Sutton Principle makes a point self-evident to those familiar with the matching law (McDowell, 2013): When organisms, including academicians, are confronted with two or more choices that differ substantially in reinforcement value (read: grant dollars), they will apportion more of their efforts to the alternative possessing a higher reinforcement value. This pattern of behavior will be amplified when administrators impose positive (e.g., tenure, promotion, awards, salary increases, resources) and negative (e.g., threats of being denied tenure and promotion, loss of laboratory space and graduate student access) reinforcement contingencies on them for obtaining grant dollars.

The grant culture

About a decade ago, I was a regular attendee at Grand Rounds presentations in a prestigious psychiatry department. Before introducing speakers, the chairperson routinely kicked off sessions by announcing the names of professors who had received large federal grants along with their precise dollar amounts. I was struck that he never announced faculty members’ important publications or scientific discoveries. I have since come to realize that this reinforcement pattern is common in psychology departments, too: Faculty members routinely receive plaudits for receiving grants but frequently find that their scholarly accomplishments go largely unnoticed.

These reinforcement contingencies should strike us as odd for several reasons. First, we do not laud novelists or film producers for securing large contracts for their planned projects, nor should we. Instead, we rightly acclaim them if and when they have generated high-quality artistic work. Similarly, in science, grants should be regarded as means to an end rather than ends in and of themselves (Chambers, 2017; Gallup & Svare, 2016). Second, for a field that prides itself on empirical rigor, psychology’s encouragement of this practice is surprisingly nonempirical. The correlation between grant funding and citation impact in psychology is low and perhaps essentially zero once one controls for potential confounds, such as journal and first-author prestige (Haslam et al., 2008). Third, grants are not needed for many forms of impactful research. For example, most researchers who have authored articles cited 1,000 or more times had no current National Institute of Health (NIH) funding (Nicholson & Ioannidis, 2012); many had produced landmark methodological advances. Furthermore, numerous Nobel laureates in physics, chemistry, and medicine received no federal funding for the work that culminated in their prizes (Tatsioni, Vavva, & Ioannidis, 2010)

To be clear, I am not opposed to grants. For some scientific questions, grant funding is essential for high-quality research or even any research at all. For many of my colleagues in neuroscience-related fields, money is a virtual prerequisite for research. We should encourage these scholars to apply for grants and make allowances in their workloads for grant-related work. Furthermore, we should reward colleagues who obtain training grants to support graduate student education. In addition, the grant culture has its upsides, including provision of funding for postdoctoral scholars and its propensity to spur competition in the marketplace of ideas.

What I am opposed to is the implication that researchers’ scholarly merit should be gauged in large measure by grant success. This fundamental law of academic life has spawned several corollary ordinances. Most notably, in a growing number of psychology departments, faculty members cannot be hired, tenured, or promoted without a solid grant track record regardless of the quality or impact of their work. Hence, even if researchers are generating significant discoveries and influencing the field’s thinking, they may be at risk for termination if they do not obtain grants.

In fairness, many institutions surely push faculty to apply for grants because they are in desperate need of financial resources. In such cases, however, fundamental maxims of intellectual honesty should require them to acknowledge that they are hiring and promoting professors as much on their fundraising success as on their scholarship.

The grant culture has contributed to a number of other troublesome consequences, each of which I describe briefly.

A bevy of negative consequences

Consequence 1: Heightened incentives for QRPs. To obtain large grants, promising pilot work is typically required; to maintain uninterrupted grant funding, a strong track record of positive results can be a virtual necessity. Adding to the pressure for positive findings is the reality that investigators whose research program hinges on grants often feel responsible for the livelihood of their postdoctoral candidates, students, and administrative staff.

In these respects, the grant culture would appear to be a virtual recipe for confirmation bias. Because confirmation bias can be fueled by motivated reasoning (Kunda, 1990), the lure of grant dollars and the fear of losing them induce powerful incentives to detect positive results by means of p hacking, outcome reporting bias, and other QRPs (Nosek, Spies, & Motyl, 2012). Although training in research ethics may be a partial bulwark against QRPs, such training is unlikely to be sufficient because confirmation bias operates largely outside of conscious awareness (Nickerson, 1998).

Furthermore, as Firestein (2015) noted, failure is a crucial element of the scientific enterprise. When studies are well-designed, we learn at least as much from disconfirmation as from corroboration of hypotheses. Nevertheless, much of the grant culture implicitly discourages failure, especially when negative results raise the specter of the investigator’s theory being in error.

Fortunately, the preregistration of hypotheses and analytic plans is a critical safeguard against QRPs (Lindsay, Simons, & Lilienfeld, 2016), as it diminishes the odds that researchers will erroneously present exploratory research as confirmatory. Preregistration will not, however, do much to diminish the foregoing problems emanating from the grant culture.

Consequence 2: Single-minded focus on programmatic research. One of the unquestioned mantras of academia is that programmatic research is invariably preferable to nonprogrammatic research. To be fair, programmatic research brings certain clear-cut advantages. If one intends to crack an exceedingly complex scientific question, a lengthy series of interlinked investigations will often be required.

Still, there are largely unappreciated disadvantages of programmatic research. Such research can foster confirmation bias, especially when it is designed to test the investigator’s favored theory (Greenwald, Pratkanis, Leippe, & Baumgardner, 1986). Research on sunk costs (Arkes & Blumer, 1985) and effort justification (Axsom & Cooper, 1985) further suggests that once individuals have invested a great deal of time and effort in a project, they will feel the need to persist in it even when doing so is no longer fruitful. In addition, programmatic research often runs its course and may yield diminishing knowledge returns following a large number of studies (Peirce, 1967).

Consequence 3: Intellectual hyperspecialization. An allied consequence of the grant culture is its tendency to canalize scholars into specialized lines of thinking for years or decades. Although interdisciplinary grants can force scholars to step outside of their comfort zones to collaborate with colleagues in other fields, the grant culture often keeps researchers locked into similar intellectual questions for long stretches of their careers. Some scholars surely manage to remain broad in their thinking despite implicit demands for specialization, but doing so is becoming an increasing challenge.

In today’s academic environment, big-picture thinkers may at risk for extinction (Wolfe, 2016). Paul Meehl, the most influential clinical psychologist of the latter half of the 20th century, received a grand total of one federal grant in his career. I am hardly the first to observe that psychology’s great generalist thinkers of the past, such as Meehl, Lee J. Cronbach, Donald Campbell, Lloyd Humphreys, Jane Loevinger, and Robyn Dawes, are now few and far between. One has to wonder what would have come of these scholars had they experienced incessant career pressure to apply for funding.

Consequence 4: Disincentives for conducting direct replications. Until recently, major federal agencies have allocated relatively little funding to supporting direct replications of previous work. Hence, there is scant incentive for investigators to replicate others’ work. In this respect, the grant culture often works against the accumulation of reproducible knowledge. On the positive side, in the United States (e.g., https://www.nsf.gov/awardsearch/showAward?AWD_ID=1612400) and Holland (Baker, 2016), there are indications that grant agencies are beginning to appreciate the value of replication, so there are grounds for cautious optimism.

Consequence 5: Stifling of creativity and intellectual risk-taking. At the risk of painting with a broad brush, the grant culture tends to reinforce conformity to today’s “hot” topics, which may not be tomorrow’s hot topics (Powell, 2016). Scientific progress is often achieved by those who dare to stand against the crowd, whose ideas do not fit into accepted paradigms (Sternberg, 1998). More broadly, the grant culture has almost certainly led many scholars to abandon daring lines of work that are less fundable and to pursue safe lines of work that are more fundable. The same reinforcement contingencies operate for methodologies as well. In much of psychology, functional neuroimaging is now all the rage, and survey data suggest that many investigators feel pressured to incorporate neuroimaging and other biological techniques into grant proposals (Schwartz, Lilienfeld, Meca, & Sauvigné, 2016). Hence, researchers whose questions do not readily lend themselves to such methods may be hard pressed to obtain funding.

Consequence 6: Promising more than we can deliver. In my own field of expertise (psychopathy), many grant proposals on the etiology of this condition dutifully conclude by assuring reviewers that the findings may bear significant implications for intervention. Yet despite a handful of promising leads, there has been minimal progress in the treatment or prevention of psychopathy over the past several decades (Hecht, Latzman, & Lilienfeld, in press) despite dozens of large federal grants, including one on which I was co-principal investigator. One of the tricks of the trade of “grantsmanship,” especially for grants submitted to NIH, is the art of persuading reviewers that one’s research bears significant real-world implications, even when such implications are a best a faint hope. Our field’s habitual tendency to overpromise has almost certainly tarnished our perception in the public eye (Lilienfeld, 2012).

Consequence 7: Diminished time to think deeply. Along with the grant culture comes mounting pressure to apply for funding at each entry point in the grant cycle. For psychologists on research tracks in medical schools, the grant cycle has become the human equivalent of the hamster’s running wheel, although surely less positively reinforcing. Inevitably, such time demands allow diminished time for thinking deeply about psychological questions. Anyone who has read Kahneman’s (2011) magisterial book, Thinking, Fast and Slow, or Michael Lewis’s (2016) The Undoing Project, cannot help but be struck by the extent to which the remarkable intellectual collaboration between Kahneman and Tversky was cultured by lengthy conversations during leisurely walks. The freedom to engage in this kind of freewheeling, in-depth reflection is becoming increasingly constrained in today’s supercharged grant environment.

Concluding Thoughts

My concerns aside, my global appraisal of psychology’s progress is reasonably positive. The replication crisis has taught us that we need to become more modest in our assertions and to steer clear of confident proclamations based on isolated positive results. Despite resistance from some quarters within our field, we are starting to engage in the healthy self-scrutiny that characterizes a mature science.

Still, formidable institutional challenges lie in the way. For a group of psychologists, our approach to the grant culture has been surprisingly nonpsychological. We have accorded scant consideration to how reinforcement contingencies, abetted by cognitive biases, make our myopic focus on grant funding counterproductive to scientific progress. These psychological impediments collide head-on with our recent emphases on minimizing false-positive findings and generating a corpus of reproducible scientific knowledge.

The corporate culture of academia places young scholars in a precarious position, as they feel incessant pressure to secure grant funding even if they do not need it. Perhaps the best advice I can give them is to strive for balance between specialization and breadth in their thinking and reading and to recall that the best science typically emerges from the integration of diverse perspectives. Admittedly, reading broadly is easier said than done given the growing demands on young investigators to invest much of their time applying for grants, and it will almost certainly necessitate challenging tradeoffs. This pragmatically knotty issue demands considerably more thought than it has received.

Finally, it will be incumbent on us as a field to initiate a thoroughgoing and intellectually honest conversation regarding the negative impact of funding on scientific progress and on potential remedies for diminishing this impact. Such correctives could range from institutional incentives for prioritizing scholarly quality and replicability over financial success to more radical proposals, such as penalizing scholars who have a lengthy track record of grant funding without a commensurate record of high-quality published research (Ioannidis, 2014).

Much like a dysfunctional family that avoids addressing uncomfortable issues out of fear of opening up a can of worms, we have put off this difficult discussion for too long. But it will be needed if we ever hope to realize psychological science’s considerable potential.

Footnotes

Acknowledgements

I thank Malte Elson, Robert Latzman, Stella Lourenco, and Brett Murphy for helpful comments on previous drafts of this article.

Declaration of Conflicting Interests

The author declared no conflicts of interest with respect to the authorship or the publication of this article.

References

Arkes

H. R.

Blumer

(1985). The psychology of sunk cost. Organizational Behavior and Human Decision Processes, 35, 124–140.

Axsom

Cooper

(1985). Cognitive dissonance and psychotherapy: The role of effort justification in inducing weight loss. Journal of Experimental Social Psychology, 21, 149–160.

Baker

(2016, July 20). Dutch agency launches first major grant programme dedicated to replication. Retrieved from https://www-nature-com.web.bisu.edu.cn/news/dutch-agency-launches-first-grants-programme-dedicated-to-replication-1.20287. doi:10.1038/nature.2016.20287

Chambers

(2017). The seven deadly sins of psychology: A manifesto for reforming the culture of scientific practice. Princeton, NJ: Princeton University Press.

Fanelli

Ioannidis

J. P.

(2013). U.S. studies may overestimate effect sizes in softer research. Proceedings of the National Academy of Sciences, USA, 110, 15031–15036.

Firestein

(2015). Failure: Why science is so successful. New York, NY: Oxford University Press.

Gallup

G. G.

Svare

B. B.

(2016, July 25). Hijacked by an external funding mentality. Inside Higher Ed. Retrieved from http://www.sscpweb.org/Media-Posts/3515419

Greenwald

A. G.

Pratkanis

A. R.

Leippe

M. R.

Baumgardner

M. H.

(1986). Under what conditions does theory obstruct research progress? Psychological Review, 93, 216–229.

Haslam

Ban

Kaufmann

Loughnan

Peters

Whelan

Wilson

(2008). What makes an article influential? Predicting impact in social and personality psychology. Scientometrics, 76, 169–185.

10.

Hecht

Latzman

R. D.

Lilienfeld

S. O.

(in press). The psychological treatment of psychopathy: Treatment and research. In David

Lynn

S. J.

(Eds.), A scientific approach to psychotherapy. New York, NY: Wiley.

11.

Ioannidis

J. P.

(2014). How to make more published research true. PLoS Medicine, 11, e1001747.

12.

Jarrett

(2016, September 16). Ten famous psychology findings that it’s been difficult to replicate. British Psychological Society Research Digest. Retrieved from https://digest.bps.org.uk/2016/09/16/ten-famous-psychology-findings-that-its-been-difficult-to-replicate/

13.

Kahneman

(2011). Thinking, fast and slow. New York, NY: Macmillan.

14.

Kunda

(1990). The case for motivated reasoning. Psychological Bulletin, 108, 480–498.

15.

Lewis

(2016). The undoing project: A friendship that changed our minds. New York, NY: Norton.

16.

Lilienfeld

S. O.

(2012). Public skepticism of psychology: Why many people perceive the study of human behavior as unscientific. American Psychologist, 67, 111–129.

17.

Lilienfeld

S. O.

Ammirati

David

(2012). Distinguishing science from pseudoscience in school psychology: Science and scientific thinking as safeguards against human error. Journal of School Psychology, 50, 7–36.

18.

Lindsay

S. L.

Simons

Lilienfeld

S. O.

(2016). Research preregistration 101. APS Observer, 29(10), 14–16.

19.

McDowell

J. J.

(2013). On the theoretical and empirical status of the matching law and matching theory. Psychological Bulletin, 139, 1000–1028.

20.

McFall

R. M.

(1997). Making psychology incorruptible. Applied & Preventive Psychology, 5, 9–15.

21.

Nicholson

J. M.

Ioannidis

J. P.

(2012). Research grants: Conform and be funded. Nature, 492, 34–36.

22.

Nickerson

R. S.

(1998). Confirmation bias: A ubiquitous phenomenon in many guises. Review of General Psychology, 2, 175–220.

23.

Nosek

B. A.

Spies

J. R.

Motyl

(2012). Scientific utopia II. Restructuring incentives and practices to promote truth over publishability. Perspectives on Psychological Science, 7, 615–631.

24.

Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349, Article aac4716.

25.

Peirce

C. S.

(1967). Note on the theory of the economy of research. Operations Research, 15, 643–648.

26.

Powell

(2016). Young, talented, and fed-up. Nature, 538, 446–449.

27.

Schwartz

S. J.

Lilienfeld

S. O.

Meca

Sauvigné

K. C.

(2016). The role of neuroscience within psychology: A call for inclusiveness over exclusiveness. American Psychologist, 71, 52–70.

28.

Sternberg

R. J.

(1998). Costs and benefits of defying the crowd in science. Intelligence, 26, 209–215.

29.

Tatsioni

Vavva

Ioannidis

J. P.

(2010). Sources of funding for Nobel Prize-winning work: Public or private? FASEB Journal, 24, 1335–1339.

30.

Wolfe

(2016, August 5). The vanishing big thinker. Chronicle Review, pp. B6–B9.