Progress in the Legitimacy of Business and Management Education Research

Arbaugh, Fornaciari, and Hwang (2016) provide important insights into research on business and management education (BME) using legitimation code theory. Several of their findings are consistent with what I see in reading manuscripts as editor of the Journal of Marketing Education (JMktE). Understanding the patterns identified by the authors provides insight into how topics and scholars develop and mature in BME research. At the same time, a study of citations by itself does not provide a sufficient foundation for advancing our field. Education research has suffered for some time from inconsistent rigor (Davies, 1999). In this rejoinder, I discuss the similarities between Arbaugh et al.’s (2016) findings and the scholarship patterns I see in marketing education journals, and I present some supplementary views on the development of BME research.

Perhaps the most important finding in Arbaugh et al.’s (2016) study is that the plurality of the most-cited papers are in the Relativistic Code quadrant, which is composed of papers with topics that are lower in knowledge richness and written by authors with lower research profiles. This quadrant has the largest number of topical areas, indicating a wide diversity of research topics. This finding is consistent with my sense of published research in marketing education journals. In attending academic marketing conferences that focus exclusively on BME research (e.g., the Marketing Educators’ Association or the Marketing Management Association’s fall conference), I see work-in-progress in session presentations and meet emerging authors in BME research. My experience suggests that scholars enter the BME field for a variety of reasons, but the most common reason is that they have discovered something personally intriguing about teaching and learning and they want to share their idea with others. It may be a teaching innovation they have developed or some interesting pattern that they have observed in student behavior. Often, these authors are delighted to discover that there are BME conferences and respected publication outlets for their research-supported insights.

However, too often, once they publish their topic of interest, they return to teaching and other publication interests, never returning to the pages of our BME journals. Arbaugh (in press) reports that 90% of the authors in Management Learning from 2004 to 2014 were “one-and-done” authors. Abernethy and Padgett (2011) conducted a similar study of the top two journals in marketing education, JMktE and Marketing Education Review (MER; Baumgartner & Pieters, 2003). They found that 82% of the authors published in JMktE or MER from 2000 to 2009 were one-and-done authors. Another 14% published twice over the time period. Only 4% of the authors are fairly dedicated BME scholars even though they have published a fairly modest three studies. Some of the one-and-done studies are excellent and are as frequently cited as some of the papers by our highly published BME authors, although very few of our highly published BME authors have high-profile international research reputations beyond marketing education. Thus, it is not surprising that the Relativist Code quadrant contains the plurality of research.

Citation analysis is a helpful guide for dedicated BME scholars in their choice of topics and for editors in their selection of invited papers or special issues. However, these individuals comprise the minority of participants in BME research. In my view, the motivation to enter the field of BME research is often personal and idiosyncratic. Most authors seem motivated to publish what they are passionate about, whether or not their study becomes highly cited. Even high-profile researchers may publish in our journals on a one-and-done basis, pursuing a topic of personal interest. Iacobucci’s (2013) study in JMktE on business school rankings comes to mind. Iacobucci is highly published in marketing research methods, and she was concerned that the various published rankings of business schools were invalid. Her article provides an in-depth quantitative analysis of the validity of those rankings, but having done that, I am not sure if I can lure her back for another paper in JMktE. Thus, in addition to citation analysis as an avenue to increase the legitimacy of our field, I recommend we push for systemic changes that could increase the focus on BME research, and for increased training of reviewers and authors to enhance the quality of our published work.

An important systemic change is to motivate accrediting bodies to pay more attention to BME research. Arbaugh et al. (2016) note one important change in this respect—the AACSB’s 2013 accreditation requirements describe learning and education research as an area for attention. However, the AACSB also calls for more evidence of the impact of a school’s research, which many interpret as citation counts (Arbaugh & Hwang, 2015). Generally, pedagogical journals have lower citation counts than do more discipline-centric journals. The Academy of Management Learning & Education is the only journal within the teaching and learning domain among the top 10 most-cited BME articles in Arbaugh et al.’s (2016) study. Thus, recent AACSB policies both promote and inhibit progress in BME research.

I advocate for a more fundamental change in our research ecosystem. Kim Stewart and I empirically make the case that at many schools, assessment will never lead to the kinds of educational improvements that accrediting bodies have hoped for (Bacon & Stewart, in press). Despite their best efforts, student sample sizes at many schools are simply too small to obtain valid conclusions about what constitutes effective pedagogy and curricula. Small samples result in insufficient statistical power, so that differences in learning cannot be detected even when they exist.

To advance the quality of BME, we must rely more on published, rigorous research as a basis for knowledge development. Rather than conducting limited studies at smaller schools (some are large enough) for assessment purposes, accrediting bodies such as the AACSB should require schools to read and implement the findings from more robust studies published in the BME literature. In addition, one often-cited obstacle to more and better quality BME research is a lack of knowledge of the BME literature and the absence of coverage in PhD programs (Arbaugh in press; Arbaugh & Hwang, 2015). In my experience, this concern is well-founded. Many of the submissions to JMktE are rejected because the authors could not present a strong contribution within the context of the extant literature. Wider distribution of this knowledge will improve the quality of BME in the short run, stimulate interest in high-quality BME research, and build a foundational knowledge of the BME literature among our faculty.

Furthermore, the legitimacy of BME research is hindered by low and inconsistent rigor (Arbaugh & Hwang, 2013). For example, in Arbaugh and Hwang’s study of research in online and blended learning (2013), only one of the 25 studies in their pool that utilized test scores as a measure reported the reliability of the test used, even though some classic reliability metrics were designed with tests in mind (e.g., Cronbach, 1951). To build the legitimacy and credibility of BME research, methodology standards should be developed and enforced by our editorial review boards. Some important contributions have been made in this direction, including Arbaugh and Hwang’s (2013) guidelines for reporting multivariate results and Bacon and Bean’s (2006) call for using effective covariates, including GPA in particular.

More recently, I have advocated to potential authors of JMktE for increased rigor in reviewing literature, describing measures, and reporting results (Bacon, 2016). In this regard, the confusion between perceived learning and actual learning has been one of the greatest impediments to the progress and legitimacy of BME research. Perceived learning is self-reported learning or knowledge gain that is commonly referred to in assessment circles as indirect measures of learning. Actual learning is measured with direct measures of learning, which includes tests, projects, or other individual performances that are scored in some way (Astin, 2012; Elbeck & Bacon, 2015; Walvoord, 2004). We now have compelling evidence that self-reports of knowledge gain are completely unrelated to actual knowledge gain (Sitzmann, Ely, Brown, & Bauer, 2010). If we distinguish between actual and perceived learning going forward, we will obtain a much clearer picture of what pedagogical methods and curricula are effective in BME. Just as we need consistent quality in reporting results as suggested by Arbaugh and Hwang (2013), we should coach new authors in the review process about important constructs and practices in our discipline. With this concerted effort, we can raise standards and rigor and increase the value and legitimacy of BME research.

In conclusion, Arbaugh et al. (2016) provide an insightful analysis of citation patterns in BME research. The one-and-done nature of many of our contributors becomes more obvious in this analysis, and popular topics are clearly identified that provide guidance to authors seeking solid impact scores. To further improve the legitimacy of BME research, we must implement changes to our field that emphasize BME research as a path to educational quality. We must also raise our standards to offer valid and compelling findings as the basis for immediate program improvements and as the foundation for future BME research. The future success of the BME field depends on our efforts.