On Doing the Wrong Things for the Wrong Reasons

Concerns About Our Field’s Lack of Impact

Walsh’s (2011) presidential address to the Academy of Management noted our field’s disappointing progress and impact. He began his commentary, “I fear that we are not all that we can be” (Walsh, 2011, p. 215). His subsequent discussion highlighted some of the problems as institutional in nature. One reason he noted is that our “audit culture” tends to associate scholarly success with volume (e.g., number of articles) rather than bona fide contributions to long-term knowledge acquisition. Starbuck (2006) expressed similar disappointment with the quality of our science. He lamented that “researchers do what serves them personally in preference to what promotes creation of reliable knowledge” (Starbuck, 2006, p. 74). Their focus on just producing articles makes research “a pretense rather than a source of genuine contributions to knowledge” (Starbuck, 2006, p. 74). Similarly, McGrath (2007) noted that “young scholars are taught not to be concerned with impact; rather they are counseled how to get articles published in prestigious journals” (p. 1366).

Neo-institutional organization theory helps us see what might be going on here; it points to “the development of strong accounts of the processes through which institutions govern action” (Lawrence & Suddaby, 2006, p. 215). These processes include, above all, a quest for legitimacy and what McKinley and Mone (2003) described as “taken for granted templates that define appropriate ways to manage organizations” (p. 361). Some become “‘rationalized myths’ that give the appearance of rationality without necessarily achieving the substance of it” (McKinley & Mone, 2003, p. 361). Similarly, Davis, Diekmann, and Tinsley (1994) defined institutional practices as those that people use “because actors take them for granted, rather than because a rational choice process found them to be best suited for the technical requirements of the task” (p. 550). More generally, institutionalized practices are templates that govern actions of an organization. Such templates are not based on evidence that they are necessarily the best approaches to achieve a task, but more often a quest for social legitimacy is behind their ascendancy. If the core practices of an academic field resulted from such a process, the field is likely to be underperforming. Two core practices of our field, NHST and OWT, are institutionalized culprits in this sense.

NHST as Questionable Practice No. 1

NHST gained its prominent position in part because it served to legitimize social scientists by making them appear to be “objective” and to be “doing science.” This appearance has misled social scientists because NHST is actually of dubious value for building social science knowledge (Starbuck, 2006). Leading scholars of statistics have pointed to NHST’s questionable value. Meehl (1978), for instance, argued that universal reliance on refuting the null hypothesis is one of the worst things that ever happened in the history of psychology. Hubbard and Ryan (2000) wrote, “It seems inconceivable to admit that a methodology as bereft of value as SST [statistical significance testing] has survived as the centerpiece of inductive inference . . . more than four decades of criticism in the psychology literature” (p. 678). Greenwald (1975) argued that our use of statistical significance entails an unjustified prejudice against the null hypothesis. The approach “ignores the fact scientific advance is often most powerfully achieved by rejecting theories” (Greenwald, 1975, p. 2). Furthermore, heavy reliance on NHST contributed to the research-publication system functioning well below its potential because “prejudice against the null hypothesis” helped to produce an “archival accumulation of valid results with extremely limited generality” (Greenwald, 1975, p. 12). Despite these serious concerns, NHST continues to be taken for granted.

How did we get to this sorry state of affairs? Gigerenzer and Murray (1987) described NHST’s institutional path into the core of social science. The “inference revolution” that took place between 1940 and 1955 made inferential statistics an indispensable instrument for psychologists. It evolved out of a long history of attempts to formalize the understanding of human inference. These efforts evolved into modern statistical inference in the hands of frequency theorists (e.g., Ronald Fisher) in the 1900s. In a process attesting to NHST’s institutional history, the frequency perspective and null hypothesis testing became dominant. Fisher’s focus on the null hypothesis permitted only yes/no decisions about rejecting the null, without yielding the statistical information about the validity of any alternative hypotheses. Psychologists, persuaded by a weak rational argument that had serious unresolved inconsistencies, perceived this limited contribution to be exactly what they wanted—“a technique for inferring the validity of a research hypothesis from the data” (Gigerenzer & Murray, 1987, p. 10).

According to Gigerenzer and Murray (1987), this “irrational revolution” (p. 22) involved (a) neglect of controversial issues (Textbooks ignored an intense controversy between Fisher, and Neyman and Pearson); (b) neglect of alternative approaches such as Neyman and Pearson’s work, which emphasized that no theory of hypothesis testing can be applied in a purely mechanical or formal way; (c) anonymous presentation of a hybrid (Textbooks combined Fisher’s ideas with Neyman and Pearson’s idea of Type II error without mentioning the scholars by name. Such anonymous presentation contributed to a neglect of the controversies and fostered an illusion of a single true method of inference.); (d) institutionalization of the hybrid through textbooks, university curricula, and journal publishing practices (Research quality was judged mainly by level of significance—Failure to reject the null was considered to be a negative result and rejecting the null a positive one.); and (e) confusion (textbooks became filled with conceptual confusion as writers dealt with the hopeless task of presenting the hybrid theory).

Gigerenzer and Murray (1987) noted that “Neyman and Pearson showed that statistical hypothesis testing does not help very much in scientific experiments” (p. 16). Nevertheless, mainstream social scientists went along with Fisher’s argument without attending to its defects. It offered a seemingly marvelous but illusory benefit—an ostensibly objective, single solution to the inductive inference problem. It was what many psychologists wanted: “a cookbook-like mechanical prescription of ‘what to do’” (Gigerenzer & Murray, 1987, p. 22). Neglect of the controversies resulted from a meeting of interests: “statisticians eager to sell their product and researchers equally eager to buy” (Gigerenzer & Murray, 1987, p. 23).

Psychological investigators were apparently gratified to have a technique that seemed to eliminate subjectively employed statistical methods, not because the methods were necessarily the most appropriate for the problems of psychology but because they were interested in supplying what would be perceived as statistically valid information useful for a general science of social control. Inferential statistics and probability theory promised at least the appearance of objectivity. As Gigerenzer and Murray (1987) wryly commented, “The dream of the scientist who arrives at new knowledge by a completely mechanized process seemed to have become real” (p. 27). In this way, NHST became a core practice in modern social science via a powerful, if questionable, institutional process. There is abundant evidence of another institutional process in play, as well—NHST was soon taken for granted. Consider Hunter’s (1997) report of his colleagues’ reactions to his call to abandon NHST because it has a 60% error rate:

Most psychologists initially reject my argument as “too radical.” They cannot believe that there can be a fatal flaw in any technique that is so widely used now and has been so widely used for 50 years. That is, most scientists know that any technique with a 60% error rate must be abandoned, and so most current psychologists think that there must be some flaw in my argument. (p. 3)

One colleague said,

You’re right, but there is nothing we can do about it. I know that the significance test has a 60% error rate, but I’m not going to tell an editor that. I include significance tests in all my papers because they’ll get rejected if I don’t. (Hunter, 1997, p. 4)

Further evidence of taken-for-grantedness is this: Despite criticisms of NHST, “clarifications have had almost no effect on research practice” (Gigerenzer & Murray, 1987, p. 24). It should be clear by now that by relying so heavily on NHST, we in organization studies apparently are doing a wrong thing for the wrong reasons. Perpetuating a wrong because it has become accepted practice doesn’t make it a right.

OWT as Questionable Practice No. 2

A similar situation exists today with respect to another core practice in management studies, what Hambrick (2007) called obsession with theory. Hambrick worried that “a blanket insistence on theory, or the requirement of an articulation of theory in everything we write, actually retards our ability to achieve our end: understanding” (p. 1346). This obsession astonishes, given the difficulty we have in defining “theory” in the first place (see DiMaggio, 1995; Merton, 1967; Sutton & Staw, 1995; Weick, 1995). Using something we cannot precisely define as nearly a sine quo non for evaluating work in our field provides prima facie evidence that it is an institutional practice. How could we have rationally decided that something we cannot define is vital for our success?

Consider the extent to which OWT meets the criteria posed earlier for being an institutional practice (i.e., a process that governs actions, provides legitimacy, and is rationalized without solid evidence of its technical merit and is often taken for granted). Corley and Gioia’s (2011) observation reflected the governing role of OWT in management studies:

Theory is the currency of our scholarly realm . . . Every top-tier management journal requires a “theoretical contribution” before a manuscript will be considered for publication . . . Consistent with this concern, the Academy of Management Review (AMR) has published two special issues dedicated to theory building (1989, issue 4, and 1999, issue 4) and numerous “Editor’s Comments” dedicated to trying to articulate what constitutes either theory . . . or a theoretical contribution . . . These writings, however . . . do not seem to have hit the mark in a way that provides a satisfactory resolution to the crucial question of what makes for a theoretical contribution. (p. 13)

Several distinguished management scholars have noted how detrimental our OWT might be. In addition to Hambrick’s (2007) concern, Miller (2007) pointed to negative consequences of our field’s bias against atheoretical research, noting that if Flemming was governed by such limits, humanity might not have gained the benefits of penicillin. Davis (2010) argued that the emphasis on contribution to theory has led to neglecting matters of substantive importance. DiMaggio (1995) pointed to how OWT may operate in the review process so as to suppress novelty, suggesting that reviewers who for some reason do not like a particular manuscript may advise rejecting it, giving lack of theory as the reason. Clearly, OWT governs important actions in our field. It also meets the criterion of institutional process by providing ostensible legitimacy.

Several streams of evidence reveal theory’s legitimizing function in management studies. The first flows from pervasive scientism that dominates our larger culture. Scientism is the belief that science, especially natural science, is by far the most valuable component of human learning (Sorell, 1991). This belief is so strong that many disciplines, especially in North American universities, attempt to emulate the natural sciences. This phenomenon, often labeled “physics envy,” spawned many aspects of modern social science practice (Nord & Connell, 2011), even though it is by now clear that the natural and social sciences have entirely different ontologies and epistemologies (Gioia, 2003). Infatuation with theory is one of these aspects. Merton (1967) noted that some social scientists, in their quest for intellectual legitimacy, took systems of scientific theory, especially Newton’s theory, as their prototype (Bourgeois, 1979). Likewise, Corley and Gioia (2011) related our attraction to theory to the pressures the 1959 Pierson, and Gordon and Howell reports placed on management scholars to “enact the norms of a bona fide scientific discipline” (p. 18). In our scientistic culture, OWT reflects what institutional theorists call a mimetic isomorphism. Hambrick (2007) suggested OWT was one part of our field’s quest to gain legitimacy in the context of “academic insecurity” (p. 1346) and our efforts “to demonstrate academic worthiness” (p. 1347).

Evaluating theory’s intellectual value is difficult in part because of problems in defining theory and what constitutes a theoretical contribution. The label “theory” is applied to a wide variety of things, and the things included tend to change over time. To make my point, I challenge you to try to come up with one coherent definition of theory that covers the special issues the Academy of Management Review (AMR) has devoted to theory beginning in 1989. Still, some of the things ordinarily attributed to theory are so obviously valuable. Theories help to organize existing information (Dubin, 1978); Merton (1967) proposed that theory can contribute to communicative knowledge; Smith and Hitt (2005) observed that theory “provides the base for knowledge and understanding of important relationships” (p. 1); likewise, Van de Ven (1989) proposed that good theory can advance knowledge and guide research into crucial questions. With these scholars, I believe that what we often call theories can be valuable. My use of neo-institutional theory to guide my analysis in this article should make this obvious. Moreover, the benefits we gained from, for example, cognitive dissonance, goal-setting, and expectancy theories provide solid reasons to respect theory. Thus, there is good reason to place a high value on good theory. Yet, there are good reasons for concern about our obsession with it.

Several issues should be noted. First, many benefits attributed to theory, such as organizing information, can be achieved as goals in their own right without appeal to “theory.” Second, our field has devoted enormous energy to building theory with disappointing results. For example, Smith and Hitt (2005) observed that “much of the theory used in management and organizational research has been derived” from other social science disciplines (p. 1). Sadly, things may actually be getting worse. Davis (2010) commented that our field’s early success in advancing theory was followed by three decades of muted theoretical progress or even stagnation. Third, considerable evidence of our lack of success in developing theory comes from the latest special AMR issue devoted to theory development. Consider the editors’ overt disappointment in AMR’s 2011 special theory issue. In their call for papers, the editors Suddaby, Hardy, and Huy (2011) invited new theories. In introducing the issue itself, the editors noted that few of the articles submitted actually provided a new theory. They confessed that their expectation for receiving new theories was probably “overly ambitious” (Suddaby, Hardy, & Huy, 2011, p. 236). Their after-the-fact pessimism about the lack of progress in theory development became obvious when they asked, “Is this perhaps . . . why there are no [italics added] new and exciting theories” (p. 245)? Certainly, after the emphasis our field has placed on making a theoretical contribution and numerous special issues devoted to producing good theory, if OWT was the path to success, surely we should have seen more progress. Unfortunately, the way theory has been and is taken for granted distorts corrective impulses.

Although the worship of theory in management and organization studies is longstanding, it seems that no rational decision produced its hegemonic position. Rather, the historical value attributed to theory is replete with a troublesome taken-for-grantedness. For example, it has become common practice in recent years for editors of the Academy of Management’s major scholarly journals to, at some time in their editorial reign, assert that if a manuscript is to be published in the journal, it must provide evidence of a strong theoretical contribution. Indeed, theory’s taken-for-grantedness seems to be embedded in the thinking of major gatekeepers in our field. For example, as Hillman’s (2011) term as editor of AMR was ending, she published an essay on the future of theory. She confessed having considered what our field would look like without theory. Without sharing this vision explicitly, she turned to what seemed to be her major concern—not so much a concern with intellectual progress, but the decline in number of manuscripts submitted to the leading management theory journal.

Her thoughts about why this might be happening reflected the spirit of Walsh’s “audit culture,” including the perception that it is almost impossible to get atheoretical work published. This perception persists despite numerous publications and special issues on how to do theory and what makes good theory. Perhaps, Hillman (2011) said, we might not be doing theory because developing good theory is not “amenable to a proscribed set of sequenced steps” (p. 607). In this context, recall that psychologists’ desire for a mechanical assessment device influenced the history of NHST. To address these problems, Hillman advised renewed commitment to theory development (including better mentoring, and more theory writing and reviewing workshops). The prevailing audit culture seemed to induce her to downplay the substantive matters that troubled Hambrick and that concern me. I would encourage us to be aware of the history of theory in the social sciences that may have moved us to pursue an ambiguous icon that may not be technically sound. Hillman’s conclusion about the role of theory in our intellectual lives is telling, and provides some insight into the pervasive taken-for-grantedness of theory in our work. She concluded that theory “is too much a part of our identity to sit by and watch it become minimalized” (Hillman, 2011, p. 607). Perhaps so, but it is equally important to question the ways in which we consider the nature and value of theory, as well.

Progress has arguably been slowed by our doing some wrong things for the wrong reasons. Certainly, our impact on the wider society has not reached the levels we might have hoped for. Because our practices, especially concerning the overuse of NHST and our OWT, are so very institutionalized, we often do not give these practices the critical (re)consideration they require. It is time to do so. I hope this brief essay opens up the thought gate to begin this necessary professional self-examination.