Abstract
In this essay, we focus on methodological issues that reviewers and editors commonly encounter when evaluating empirical articles in scholarship of teaching and learning in management education. We organize our discussion around three stages—design, analysis, and reporting. The essay identifies which types of issues are likely to receive rejection, and which issues can be addressed in the subsequent reviews of the manuscript. We provide authors with some suggestions on how to address these issues in their research.
Keywords
The standards for publishing academic research are higher than ever before, with more researchers vying to publish their work in journals with limited space (Ashkanasy, 2010). Conducting academic research is time consuming, resource intensive, intellectually taxing, and requires immense commitment to withstand the uncertainties of this laborious process. Scholarship of Teaching and Learning (SoTL) is no different when it comes to the drudgeries and perils of publishing research. With a sophisticated audience of management educators, authors are likely to encounter reviewers from the same pool of scholars who review mainstream management research. Management education research draws upon many domains, but most notably education research, human development, and management in organizations (Anderson et al., 2021). Given this broad knowledge base, authors are expected to clearly align the methodology with the ontology and epistemology of their question of interest.
This essay is an effort to help authors publish management education research that relies on quantitative methods. Journal of Management Education (JME) publishes various genres of articles, including theory articles, research studies, essays, and instructional innovation; for details see JME webpage here. In JME articles, quantitative methods are used in empirical research articles and instructional innovation (INI) pieces. This type of research often examines the antecedents and outcomes of management education practices at various levels and for various stakeholders. Some of JME’s highly cited articles in this area include, exploring the antecedents of faculty satisfaction in online education (Stickney et al., 2019), empirical testing of teaching practices that increases learning impact of student teams (Bacon et al., 1999), and online and hybrid teaching in management education (Arbaugh & Hwang, 2013). Researchers also examine how management education intersects, co-creates, or adapts to broader contexts, such as technology (Allen et al., 2021; Ring et al., 2012), climate adaptation, and sustainability (Maloni et al., 2011; Sroufe, 2020), and social issues (Maloni et al., 2021).
JME offers exciting opportunities to test management education theories. In our essay, we discuss the common issues that our editors and reviewers encounter when evaluating the Methodological aspects of a quantitative manuscript. While we focus on Methodological issues, we also want to remind our readers that the conceptual foundations of the manuscript should be strong. No matter how elegant the quantitative elements are (design and analysis), if the theory is weak the manuscript is unlikely to receive a favorable review. We discuss quantitative issues of an otherwise conceptually sound idea. We explain the types of methodological issues that are more likely to receive a revise and resubmit and those that may lead to outright rejection. We hope this decreases the mystery of the review process and provides our readers with practical information to plan their future submissions to JME.
We receive a wide variety of empirical and instructional innovation (INI) articles. Related to INI articles, we receive a high volume of studies where a new experiential or active learning approach is presented to increase management learning. For INI articles, many journals, including JME, often require evidence of effectiveness, which can be quantitative (e.g., pre and post test scores) or qualitative (e.g., analysis of student comments). We also receive empirical articles that focus on effectiveness of a pedagogical approach, and for these articles the focus is on testing a theory underlying a teaching approach. As such, empirical studies on learning approaches require quasi-experimental design, as often random assignments of students are not possible due to ethical and practical issues (e.g., Reed et al., 2023). Thus, we advise our authors to brush up their skills in designing and conducting quasi experimental research (Campbell & Stanley, 2015; Cook, 2015; Cook et al., 2002).
We also receive empirical submissions that focus on evaluating various aspects of a program of instruction, such as student competencies (Halfhill & Nielsen, 2007), based on existing management and education theories. Furthermore, empirical studies also use andragogical theories to test predictors (e.g., Westerman et al., 2002) or processes (e.g., Luthans et al., 2019) of learning outcomes. As such, a wide variety of generalized linear models or multilevel analytical techniques might be employed depending on the research question.
Our editorial board reflected on the most common issues that led to manuscripts being either rejected or needing significant revisions. We outline these issues in the following three sections: research design, analytical strategy, and reporting quality. We also provide two tables: Table 1 summarizes common issues and likely decisions, while Table 2 provides authors helpful preemptive questions to reflect on at each stage.
Types of Issues at Various Stages of Research Process and Manuscript Decisions.
Reflective Questions and Resources for Authors.
Research Design
At a broad level, the quality of research design is evaluated by inferring alignment between the methodological paradigm (deductive or inductive) and the theoretical paradigm (the ontology and epistemology). In other words, an assessment is made to judge whether the testing paradigm fits the theorized nature of phenomenon (ontology) and how the knowledge of the phenomenon can be gained (epistemology). Thus, if the ontological positioning is objectivist (e.g., student learning can be objectively assessed), then the educational theories assume positivist (or post-positivist) approach (e.g., socio-economic, and dispositional factors that enhance student learning). This theoretical paradigm requires deductive methodology and quantitative method and procedures to test the hypothesis (Taylor & Medina, 2011). Here using inductive and qualitative approaches would often be inadequate. This type of glaring misalignment between ontology, epistemology, and methodology rarely passes the initial editorial review of the manuscript and is most likely to be desk rejected. Instead, our essay focuses on nuanced methodological issues that surface during the detailed blind review of the manuscript. These issues are related to specific research methods and procedures used to evaluate research questions that have otherwise well-aligned management education theoretical and methodological paradigms.
Here we address two research design issues in detail. In almost all cases inadequate design is a serious concern and will warrant a rejection of the manuscript. Thus, we advise authors to spend considerable time thinking and planning their research design, and pilot testing their design before venturing into full-fledge data collection. Finding recent published manuscripts that outline a procedure that evaluates a similar style of research question may help to ensure the research design is not missing any key elements that will strip away its validity.
Serious Design Flaws: Mismatched Design, Convenience Sampling, and Endogeneity Issues
It is expected that theoretical questions raised in the front end of the article are adequately tested in the empirical section of the article. At a fundamental level, the research design should be able to test the hypothesized relationships. If a study design is unable to test the research question presented in the front end of the article, reviewers are left with little choice but to reject the manuscript. Such issues cannot be addressed without redesigning the study and collecting more data. For example, suppose authors of an empirical study spent considerable time upfront arguing how a specific pedagogical approach increases learning outcomes compared with a traditional instructional method. However, the study design only employed the new method without testing the traditional method. As we explained earlier, thoroughly reviewing the principles of quasi experimental design methods (Cook et al., 2002) at the onset should enable researchers to avoid making serious design omissions when testing the efficacy of proposed instructional methods. Therefore, any claims of one strategy being better than the other would require a comparison sample to test differences between group outcomes.
Such design limitations occur for a variety of reasons, including the use of conveniently available data that was not collected for conducting research. Researchers who have not published in management education journals, may wrongly presume that the bar is lower and make the mistake of submitting a “half-baked” study. As mentioned before, the reviewers are drawn from the same pool of reviewers who evaluate mainstream management research. The reviewers and the readers of management education journals are sophisticated researchers and expect similar high research standards as they encounter elsewhere. Thus, we advise authors to spend considerable time planning and pilot testing their ideas. For example, at the initial stages of conceptualization, present your ideas or pilot-study results at brownbag sessions (informal lunchtime presentations) to clarify research questions and bring rigor in design. If the research design is weak, there generally is no easy fix, and the manuscript is highly likely to be rejected. Similarly, researchers need to be wary of endogeneity bias, where the tested effects might be a result of endogenous (i.e., unmeasured) variables (see Antonakis et al., 2014). Hill et al. (2021) provide a list of checks and remedies to deal with this bias. In the end, careful design of your study is the only way to avoid problems that cannot be fixed later.
Inadequate Data: Insufficient Sample, Poor Measurement, and Common Method Bias
The adequacy of sample size is a key issue. It is important to conduct power analysis at the design stage to recruit an adequate sample size (Kang, 2021). Often our classroom sizes are limited, which might mean collecting data from multiple classroom sections. This introduces the issue of controlling confounding variables, multiple instructors, content variations, semester-wise contextual changes, and variations in class timing. Nonetheless, thinking of these issues in advance and noting implications of unavoidable issues in the limitation section are important aspects of showing rigor in your work (Patten & Newhart, 2017).
Another issue concerns the use of invalid measures. We caution against ad hoc modification or abbreviation of the validated measurement scales. Whenever possible use the full validated scale, and if needed, search for and use a validated shortened scale. Always spend time researching and reviewing the validated scales for the constructs in your research domain. Measurement is a complicated issue and selecting a valid scale is important. We understand, response burden is a valid concern, and this tempts us to shorten the scales, but ad hoc deletion or modifications to existing scales will weaken your design. Avoid selecting a poor scale (with dubious validity) or just making up a scale; It ends up not measuring what needs to be measured (Bagozzi & Edwards, 1998). Be wary of any modified scale that lacks evidence of validity for the altered scales, even when this type of work is published elsewhere.
Poor measurement is also evident when a hypothesis claims to test broader constructs, such as student learning or growth, but researchers operationalize variables using measures that tap into insufficient construct domain. For example, using exam scores to operationalize learning is problematic. An exam typically tests knowledge, comprehension, and application of concepts in a subject domain, but they may not include other aspects of learning, such as skill building, insights, and experiential growth (Kolb & Kolb, 2005). Additionally, researchers sometimes use incorrect measures, such as measuring student learning using students’ ratings of instructors. A student’s rating of instructor does not necessarily reflect student learning.
Finally, we emphasize the importance of addressing issues related to common method variance (Ashkanasy, 2010; Podsakoff et al., 2012; Spector, 2006). This is a bias that usually occurs when participants are asked to respond to a single-sitting questionnaire that includes measurement of all variables. While Podsakoff et al. (2012) discuss several remedies to this issue, note that post hoc statistical fixes fail to deal with the main problem, which is respondents’ motivation to respond consistently to items across measures, leading to common method bias. The only way to address this bias is to collect data from multiple sources (e.g., supervisor- or peer-reports; or objective measures) or at least to collect data on independent and dependent measures at different times (Podsakoff et al., 2012).
Analytical Strategies
Inappropriate Statistical Approach
When testing hypotheses, it is important to use appropriate analytical approaches. For example, to show a relationship between instructional approach and student learning outcome, using zero-order correlations without a concern for control variables affecting the relationship between these two variables is a mistake. If theoretically relevant control variables were not collected in a study, then little can be done at the review stage. If such data is present, the reviewers may ask to rerun analysis, such as adding control variables in the statistical model.
Related to the above, a less significant issue can be a misalignment between the way a hypothesis is worded and how it is assessed. For example, the hypothesis claims that students in an online environment will improve more over time, compared to the traditional setting. However, instead of conducting a repeated ANCOVA, the analysis uses cross-sectional ANCOVA to test group differences between online and traditional settings. The use of cross-sectional ANCOVA is incorrect because it does not show a change over time (within-subject change). If the sample is adequate, the paper is well-written with sound theoretical arguments, and all that is needed is to rerun analysis, there is a chance that the reviewers ask for a revision.
The incorrect level of analysis is another common issue. Do hypotheses match with levels of analysis? For example, do you have hypotheses that concern individual student performance, but measure student team performance? Or do you have variables at various levels of analysis? For example, a mixture of individual student characteristics and student team characteristics. In these circumstances you may need to draw on multi-level analysis methods to avoid drawing erroneous conclusions from your data (see, e.g., De Leeuw et al., 2008). Depending on the issues, incorrect level of analysis may lead to either rejection or a high-risk revision. The manuscript is likely to be rejected if there is a mismatch between the theoretical level of constructs and the operational level. There is a slight chance the reviewers might recommend a review if simple analysis needs to be rerun using multilevel techniques.
Generalizability
JME aims to advance research and teaching pedagogy. As such, generalizability of research is an important aspect of the review process. We want authors to make sound arguments about how their research study informs the management education conversations. To achieve this, authors should be familiar with the broader management education context and provide informed discussion on classroom and business school implications for an international audience. It is important to avoid overgeneralization claims and provide a measured discussion by looking at past research and examining the limitations of your study. For example, a study on the efficacy of online learning interventions needs to explain how generalizable the proclaimed effects are and to recognize that there is vast variety in methods of online learning. This discussion must be evidence based and informed. Please cite evidence from recent studies, show how your study advanced the conversation, and what contexts the evidence does not apply to. Often, if other aspects of the study are strong and authors did not oversell their results, many of these weaknesses can be addressed in the review process.
Reporting Quality
Justification of Approach
Justification is a well-supported explanation of why your chosen research design and method are appropriate for addressing the research question. An assumption should not be made that all scholars are aware of your quantitative approach. Clearly explaining the strengths of the chosen design, as well as known limitations, and how they have been mitigated is important. Remember that core to explaining your research design is communicating the alignment between your research question and the design approach to collect data for testing the hypotheses. Being able to clearly articulate why the particular approach was taken will establish credibility and facilitate the review process. Reviewers often find that researchers need to better justify their data collection and analysis methods. These issues could lead to reviewers recommending a major revision of the manuscript.
Completeness and Clarity
A fundamental function of the method section of an article is to provide readers with enough detail to understand the study and exactly how it was run. Completeness and clarity are equally important in reporting what you have done, so that reviewers can assess the rigor and validity of your research. When writing a method section, it is important to remember that the reader knows nothing about what has happened in the study. Provide a complete explanation of the sample process, measurement protocols, data collection, and analysis procedures. As with the justification of approach, the rarer the data collection or analysis technique, the more detail you will need to provide to ensure the reviewers understand precisely how you conducted data collection and analysis. The flow between information should be logical and all information should be in the section for a purpose. Detail and clarity are important, so be sure to define any terms or statistical jargon, work on the clear presentation of results using tables, figures, and charts, and provide supplemental material with measurement tables and programming code. Also, be honest. It is better to be upfront with any weaknesses or limitations of the study and acknowledge openly the impact of them on the results. Being honest and providing detail will assure the reviewer that you have a true understanding of your study and your method and results.
Be aware of the type of paper when considering the balance of detail in sections. For empirical articles, it is important to dedicate a significant amount of the submission to detailing all aspects of the methods and analysis. For INI articles, while details of the methods and analysis need to be clear, they also need to be concise to focus on providing adequate detail on the innovation itself. As a reviewer, it can be frustrating to read a method and/or results section that is incomplete or unclear. In papers where we believe that the underlying research design is adequate, we will be likely to focus on issues of completeness and clarity as fixable points in a major revision.
Storytelling
The best academic research is rigorous, engaging, memorable, and accessible. Storytelling offers a coherent and structured narrative, presenting the significance of methods and results in relation to the research question. Reflecting on the core ideas, findings, and their implications within the broader context of scholarly literature and practice aids in crafting a captivating story that retains the reader’s attention. Presenting a clear argument and statement of contribution at the beginning of the paper, giving a comprehensive justification for data collection and analysis methods, and identifying the crucial themes and patterns in the results to answer the posed questions, all contribute to a linear structure with a clear beginning, middle, and end. This structure enhances coherence and helps readers understand the connections between various elements of the research project. Maintaining a focused, linear “golden thread” throughout the paper increases the likelihood that readers will grasp complex ideas and find the narrative compelling. A well-structured paper with a clear story is more likely to receive a revision.
So What?
“So what?” is a key question that authors need to answer through the reporting of their methods and results. It is important to reflect on not just what research was completed and whether the results supported the research questions, but on what the wider relevance of the study is. A paper must address the primary question driving the JME review process: will this contribution significantly influence thought and/or practice in management education?
Some of the ideas we present in generalizability are connected to the bigger so what question. This question should be addressed in the front end as the value proposition of your study, and then bring home the point in the discussion section. At JME, the pedagogical, classroom, and business education implications are important. Reviewers want to see how your paper fits in the bigger picture, advances their understanding, and raises important new questions. Although, it is not strictly a methodological aspect of the study, it provides the reason for accepting the manuscript.
Conclusion
Management education research is time-consuming, demanding, but rewarding. We advise our readers to spend considerable time planning and testing their ideas. To become part of the management education conversation, stay up-to-date on the latest research on instructional innovation and andragogy. Many journals in our field, including JME, offer free paper development workshops throughout the year. JME now also offers virtual Editors’ Office Hours, typically from September to April. Notices for these are posted on JME’s LinkedIn page (see Table 2 for a link). The Editors’ Office Hours provide researchers an opportunity to seek early feedback on their research ideas and improve their research outcomes.
This essay focused on the methodological aspects of quantitative studies in management education research. We identified various issues at the three stages of design, analysis, and reporting. As summarized in Table 1, the issues with the design of the study and problems with data collection are serious issues that are not fixable at the review stage. We advise authors to spend considerable time improving study design and avoiding these issues. If the design is adequate, the issues with the analytical approach might be fixable. At times redoing the analysis with another statistical approach can remediate analytical issues. Finally, the quality of reporting can be improved by adequately justifying the study design, method, and analytical approach. Writing clearly and explaining the rationale and implications for teaching and andragogy are other vital aspects to help the reader fully appreciate and understand the big picture. We advise researchers to think of the story they are trying to tell to improve the reporting quality.
In Table 2, we have provided authors with some ways to preemptively address the issues identified in this essay. At each stage, the reflective questions can be used as a checklist to ensure rigor in research. We also propose several actions that researchers can take to address potential problems across the three methodological stages discussed in our essay.
To sum up, we hope this brief essay provides researchers insights in the review process at JME and helps them make informed decisions throughout the research process.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
