Eyes Wide Shut? A Commentary on the Hypothesis That Never Was

Editor’s Introduction

Two issues ago, we published the article “The Hypothesis That Never Was: Uncovering the Deceptive Use of Post Hoc Hypotheses” by an author who chose to remain anonymous. The anonymity was well justified, because he or she was calling out quite a few members of the organizational research community for practices he or she considered questionable at best and unethical at worst. Well, that little essay certainly fulfilled the mission of a section titled “Provocations and Provocateurs.” Now, Raghu Garud weighs into the debate. He is gentle at first, merely enlightening us on the many possible meanings of the foreign (literally) and much-used phrase “post hoc.” Thereafter, things get more serious. Presenting post hoc hypotheses as if they were known a priori has the fateful consequences of compromising our knowledge, challenging our credibility, and damaging our legitimacy as scholars. I have often parsed our scholarly field according to two primary roles—“knowledge generation” and “knowledge dissemination”—and held the knowledge generation role in the Pantheon of human intellectual endeavors. In that light, Raghu’s main concern is my deepest concern. If the cavalier use of post hoc hypotheses is indeed a widespread, field-level problem, it has the potential to paint us all with the ugly brush and undermine the essential basis and highest ideals of our entire enterprise. We would all like to avoid that outcome, wouldn’t we?

Raghu’s essay is followed by a short rejoinder from Anonymous, the original provocateur.

—Denny Gioia

Has Anonymous (Journal of Management Inquiry [JMI], April, 2015) merely stirred the proverbial tempest in the teapot with the provocative essay “The Hypothesis That Never Was: Uncovering the Deceptive Use of Post Hoc Hypotheses,” or has he or she set the stage for a perfect storm? A full professor, collaborator, person who has published a number of “A” level management journal articles, and a member of several journal editorial boards, Anonymous is also a whistle-blower of a practice he or she considers unethical—presenting hypotheses as if they were known a priori but, in fact, generated after data analyses. Gioia (2015), the handling editor for “The Hypothesis That Never Was,” introduced the article with the following commentary: “Unwittingly deceiving one’s self is a fascinating process; wittingly deceiving others is something else entirely. Yet, if we are to believe our anonymous author, both processes can be at play” (p. 214).

This JMI essay has spawned a robust discussion on orgtheory.net. ¹ Among other issues, management scholars noted the pervasiveness of this practice and discussed how it might be remedied by implementing practices such as allowing for a different presentation format for inductive findings and encouraging replication studies. As a scholar interested in the sociology of scientific communities, I decided to weigh in on the debate. Although the issues involved are complex (see, for example, Bettis, 2012; Kerr, 1998; Leung, 2011), I will confine myself to addressing the nature of the infraction and the damage done, the locus of responsibility, and possible remedies.

What Is the Nature of the Infraction, and What Damage Is Done?

Several scholars have commented on the practice of presenting post hoc hypotheses as if they were known a priori. ² For instance, Leung (2011) noted that this practice “is not defensible, because hypotheses in this context are no more than empirical findings disguised as hypotheses” (p. 474). Kerr (1998) argued that this practice “violates a fundamental ethical principle of science: the obligation to communicate one’s work honestly and completely” (p. 209). Anonymous also views this practice as unethical, noting, “I want to be clear: I never fudged data, but it did seem like I was fudging the framing of the work, by playing a little fast and loose with the rules of the game . . . .” (p. 215) Anonymous felt he or she was misrepresenting how the research was actually conducted concluding, “What we wrote in the article was a lie. It amounted to academic dissembling even though I knew it was commonly done” (emphasis added).

To explore possible reasons for this unease among these scholars, I begin with the various meanings associated with the term post hoc. One meaning is simply analyzing data after the conclusion of an experiment to identify patterns that were not specified a priori. Such “post hoc analysis” is a bona fide practice in scientific inquiry and even valuable if properly conducted and reported. But, to the extent that they are not, the potential damage begins to mount.

For instance, to the extent that causality is ascribed to events when there is none, a “post hoc fallacy” emerges—that is, the fallacy of inferring causality from random temporal succession of events. This is the second meaning of post hoc—post hoc ergo propter hoc—literally “after this, therefore because of this” in Latin. Clearly, the post hoc fallacy creates a problem when the alleged knowledge that accumulates as a consequence is used to make predictions.

Then there is a third meaning of post hoc, namely, offering justifications for results after the fact. Indeed, we are creative enough to abductively (Peirce, 1965) generate connections between two or more events even when there is none and then provide a rationale. However, such “post hoc rationalizations” compound the “post hoc fallacy.” Specifically, post hoc rationalizations can reinforce our belief that there is a causal link between events when in fact there may be none.

Is it possible that scholars in the management discipline too could fall prey to post hoc fallacies and justifications? If we keep running regression models, something statistically significant will eventually emerge given large enough sample sizes. Then, by ascribing causality and offering justifications, a social fact emerges. However, any causality that is imputed may be based on spurious correlations and random associations, and so represents a Type I error (false positives). This is bad enough. But to then present these findings and justifications as if they were the basis for the development of hypotheses a priori is clearly problematic; the hypotheses were never put to any test, as they emerged from the data. Yet, truth-value associated with the rigorous testing of hypotheses based on deductive logic is claimed, even for Type I errors from post hoc analyses, but now expressed as supported hypotheses.

This fourth meaning of post hoc, one that is bothering Anonymous, magnifies the post hoc fallacy as it further cloaks findings masquerading as hypotheses (Kerr, 1998) with a sheen of objectivity that cannot be challenged. Not only are there after-the-fact theoretical justifications for the hypotheses, but also, they supposedly received empirical support. In this fashion, potentially fictitious findings become immutable facts.

Articles with these pseudo hypotheses, when they appear in print, become “immutable mobiles” (Latour, 1987) that end up contaminating the entire knowledge pool of well-reasoned hypotheses that received empirical support. Equally damaging, from a purity-of-science point of view, potentially spurious correlations and random associations lend false support to a theoretical base the claimed hypotheses purport to draw upon. As a result, rather than exploring the validity of the potential “discoveries” (if reported as such) that emerged from post hoc analyses with fresh data, or exploring emergent promising relationships with additional data to reduce Type II errors (errors of omission), we might instead assume that the pseudo hypotheses found support and so continue pursuing potentially unproductive research.

These observations highlight why presenting results from post hoc analyses as a priori hypotheses is problematic in ways that go beyond what Anonymous (2015) noted:

Would my grade school teacher say that I had committed some sort of sin by changing hypotheses? I guess my answer to this question is that it’s not the changing of the hypotheses that may be unethical, but rather that if any significant part of research is “secretive” or could be construed as disingenuous or duplicitous, then there are viable grounds for suspecting that it is unethical. (p. 24)

Complicating matters, in Anonymous’ case, not only were the hypotheses written post hoc, but they also were based on the application of a “subsequent innovative analytical approach” (yet another meaning of post hoc) that revealed new patterns. But what if, tomorrow, some other innovative analytical approach emerges that refutes what was found? This is an issue addressed by Latour and Woolgar (1986) who studied how suppositions become facts in the laboratory. They noted that scientists use tools and techniques to understand and probe “reality.” However, these tools and techniques become part and parcel of our ways of viewing the world, which then circumscribe what we see and what we don’t. Latour and Woolgar label this as “inversion,” a potential problem that can be magnified when we use novel untested tools and techniques to analyze data and then present what emerged as a priori hypotheses.

Indeed, it is to overcome this problem that scientists at CERN conducted two completely different grand experiments that ran in parallel to identify the Higgs boson particle (the “God particle”). “We are ministers of doubt,” Bertolucci (2010), the Director for Research and Scientific Computing at CERN, informed a packed audience at a Strategic Management Society (SMS) conference in Rome. Specifically, CERN invested in not one but two billion dollar experiments at Geneva so as to be doubly sure of the inferences they drew about the particles that were produced during proton collisions. Moreover, this scientific community continues to engage in a process of vigorous contestation and justification before proceeding with any experiment (Tuertscher, Garud, & Kumaraswamy, 2014).

Overall, then, the practice of presenting post hoc hypotheses as a priori can end up compromising what we know. That is, epistemology compromises ontology. Specifically, this practice can increase the possibility of Type I and Type II errors, misrepresent the truth-value of hypotheses that never were, and even lend validity to novel analytical approaches that have not yet been thoroughly vetted. Clearly, this academic practice also can have an adverse impact on managerial practice. We can well imagine what happens when the implications for managerial practice are based on random or spurious causal connections that are then posed as if they had the weight of hypothetico-deductive science.

Going beyond issues of epistemology and ontology, there is yet another critical problem that emerges having to do with credibility and legitimacy. To the extent that we want to claim the “authority to speak” based on the conduct of rigorous scientific research (what has been labeled as “boundary work” by Gieryn, 1983), we must at least conform to the tenets of science (Merton, 1942). Not doing so runs the risk of a field losing credibility and the claims of its members losing legitimacy, as happened with “Climategate.” The practices that were used by climate scientists, when displayed to the public by a hacker, ended up compromising the credibility of the climate scientists involved and the legitimacy of their consensus opinion (Garud, Gehman, & Karunakaran, 2014). Anonymous (2015) is conscious of this issue, as is evident in the question “Would such behavior pass the New York Times front page test (i.e., how would the author feel if the New York Times did a front page expose of such academic practices)?” (p. 216).

Where Does the Responsibility Lie?

Anonymous assumes responsibility for an ethical breach. But the essay suggests that there is a burgeoning problem within the field itself. Although remorseful, Anonymous also is pragmatic—helping a junior person make tenure, conforming to the parameters set down by journals, dealing with editors who are in a position of power, dealing with anonymous reviewers who hold a different source of power, and so on. The pressure to publish has become so great that it is no longer even worth questioning questionable practices. If these practices are now widespread, then we have all become prisoners of a system that we have created but are unable to free ourselves from. Macro expectations and incentives have transformed micro motives.

But what might explain such macro expectations? One is the possibility of claiming truth-value associated with the rigorous testing of well-reasoned hypotheses based on deductive logic even for emergent findings. Indeed, such “dissembling” feeds into a “positivity bias” (i.e., a bias toward publishing hypotheses that were supported by data), support for which Leung (2011) found in his analysis of articles published in the Academy of Management Journal in 2009. Besides increasing the possibility of introducing uncontested Type I errors as discussed earlier, positivity bias also reduces the incentives to report hypotheses that did not receive support. As may be evident, a failure to report hypotheses that did not receive support can lead to continued efforts by others to test these very hypotheses in subsequent studies. Ironically, positivity bias may even end up dampening the reporting of potentially interesting patterns in the data that failed to receive statistical support for a variety of reasons, thereby even increasing the possibility of Type II errors.

These observations suggest that there are real costs in presenting post hoc hypotheses as if they were known a priori. Yet, we appear to be ignorant or, worse, apathetic about these issues. “It is a matter of representation,” according to a member of the Academy, “so what’s wrong with that?” “Isn’t every act of writing or communication a form of representation? Aren’t the authors doing the Academy a favor by re-arranging the various parts of a paper in a manner that is more palatable to the readers?” Think about this assertion and its implications.

In other words, unlike Anonymous, many don’t even see this practice to be a problem. This in-and-of-itself is a problem. Junior scholars are being socialized into questionable practices that they do not find problematic. For instance, Matt Vidal, cautioning against the practice, noted on orgtheory.net, “It is completely normal during quantitative analysis to test a number of model specifications, develop hypotheses to interpret unexpected findings, and present them as a priori hypotheses.” Bettis (2012) reported that a PhD student who was “searching for asterisks” observed, “This is the game we play to get published” when asked what he or she was doing.

Overall, these observations and many others along similar lines suggest that members of the Academy may be operating with eyes wide shut. Problematic practices that violate the percepts of basic scientific methods appear now to be accepted as normal. How can this be acceptable?

So What Can We Do?

One solution is to institutionalize formats for paper-writing for publishing inductive or abductive findings (see also discussions on orgtheory.net). A scholar who has published in prestigious journals such as Nature and Proceedings of the National Academy of Sciences commented,

One thing that struck me from the natural-sciences perspective is that there seems to be some underlying peer-pressure within the social and management sciences community and literature to present results in the hypothesis-test-result framework. In my community, I don’t think there’s any particular premium placed on presenting results in that manner, but this obviously varies. While the research is certainly driven by well-formed goals and questions, many papers are written as: Here’s a new method or result that is demonstrated/observed by an experiment, and here’s a theoretical model to explain the observed results. And presenting it as such would not, I think, exclude the paper from being considered for the very highest journals unlike, it seems, in the management literature.

If we were to also embrace the approach noted in the latter part of the quote above, it might encourage us to engage in replication studies to explore the robustness of findings. It is clear that this remedy would require authors to invest time and effort to gather parallel data sets, for journals to publish replications, and for the Academy at large to value such efforts, and not simply keep chasing novelty for its own sake. Perhaps Academy of Management Discoveries is one such journal that will allow such work to flourish. But shouldn’t the other mainstream journals also embrace such practices and norms?

Whether or not the Academy encourages practices that address the ethical dilemma confronted by Anonymous is to be seen. It requires a broader discussion among Academy members of the many complicated issues involved with the practice where authors for whatever reasons (ignorance, compulsion, willful ethical breach of foundational practices) pose their findings as if they had been hypothesized earlier.