Moving Forward With Research on Case-Based Learning: A Commentary on “How to Improve Written Case Analysis and Reduce Grading Time: The One-Page,Two-Case Method”

I was fascinated to read Kirsten Passyn and Judi Billups’s paper when it was first submitted to Journal of Marketing Education (Passyn & Billups, 2019). Case teaching is widely used in marketing; Crittenden and Crittenden (2006, p. 85) estimated that 80% of marketing capstone courses include cases, and Crittenden and Wilson (2006, p. 85) reported that 76% of marketing departments are teaching cross-functional cases.

However, even with such widespread use of cases, there is little rigorous empirical research on how best to teach with cases to maximize student learning. My understanding of the unique challenges of case-based learning research has grown through my own teaching and research (Bacon, 2006; Bacon & Quinlan-Wilder, 2011, 2014) and from reading many papers on case teaching that were not accepted for publication as the editor of Journal of Marketing Education and a member of the editorial review boards of Academy of Management Learning and Education and Journal of Management Education. While interesting, most of these papers had methodological deficiencies. Passyn and Billups’s methodology overcomes several of these deficiencies, and they subsequently provide valuable insights into case learning and recommendations for case teaching. I see opportunities to build on Passyn and Billups’s work and substantially advance our knowledge about how students best learn with case analysis. My purpose in this commentary is to highlight some of the strengths of Passyn and Billups’s study that may not be obvious to those who are newcomers to this type of research. I will then present some areas for improvement in future research.

One of the significant strengths of Passyn and Billups’s work is the use of actual measures of student learning. Many authors have contributed thoughtful conceptual work on how case teaching can be done and its potential benefits (e.g., Bailey, 2002; Christensen & Carlile, 2009; Crittenden, Crittenden, & Hawes, 1999; Gamble & Jelley, 2014; Garvin, 2007; Greenhalgh, 2007; Pitt, Crittenden, Plangger, & Halvorson, 2012; Stewart & Winn, 1996). However, relatively few papers have measured the effectiveness of their proposed methods, and when they do, the measures used are usually indirect or perceived learning measures (e.g., Dröge & Spreng, 1996; Karns, 2005). Research that uses measures of actual learning are quite rare. I know of only a few published articles in marketing or management education journals that use direct measures of learning in assessing alternative approaches to case teaching (Abernethy & Butler, 1993; Desiraju & Gopinath, 2001; Loewenstein, Thompson, & Gentner, 2003; Riddle, Smith, & Frankforter, 2016). This shortcoming substantially limits progress in this area. As I have commented before (Bacon, 2016), actual learning and perceived learning are quite different constructs, and they are often confused in our literature. Passyn and Billups’s use of direct measures is a welcome exception in this regard.

A second important strength of Passyn and Billups’s study is their use of experimental designs, and not just the one group, pre-post design, but a design with multiple treatment groups. Too often in business education studies, I see pre-post designs with no control or comparison group. Such designs can show that the intervention is better than nothing, but such findings are rarely a contribution beyond common sense. For example, it is not surprising if an author finds that taking an extra class period to run an experiential exercise on a particular topic improves knowledge of that topic. Taking extra time and using any reasonable means (including cases or lecture) will probably improve learning on that topic. However, when a study identifies significantly different outcomes across two reasonable approaches for obtaining the same learning outcomes, the result is a far more substantial contribution.

Ideally in this regard, interventions should be comparable in time commitment and use of student resources (time, energy, and preparation). In two experiments, Passyn and Billups contrast outcomes across reasonably comparable treatment groups, including differences in the length of the required write-ups and differences in training with cases (i.e., a lecture on case analysis, a practice discussion, an initial individual write-up, or an initial group write-up). By comparing not just two but several different treatment conditions, Passyn and Billups multiply their contribution, ultimately showing that offering training in case write-ups, enforcing a structured one-page limit, and requiring an initial case write-up in groups can all improve case analysis ability over alternative approaches.

A subtle but third important strength of Passyn and Billups’s work is the comparability of outcome measures across conditions. One of the challenges in assessing case learning is that some cases may be more difficult to analyze than others. If a researcher were to compare the quality of the students’ written analyses of Case A to Case B, we cannot be sure that the differences in quality are related to the differences in actual student ability or to differences in the difficulty of the two cases. Passyn and Billups solve this problem by requiring the same cases across all conditions. An alternative but somewhat more complicated approach would be to rotate the cases to balance case difficulty or to statistically control for case difficulty (see Bacon, 2006).

Another substantial challenge in assessing case learning is teaching to the test. For example, when we see that case write-up scores improve when a rubric is implemented (e.g., Abernethy & Butler, 1993; Riddle et al., 2016), is the improvement due more to the mastery of the rubric or to the mastery of case analysis? When we teach students to perform in a specific way, our designs may suffer from over alignment, which is essentially teaching to the test (see What Works Clearinghouse, 2017, p. 79) or what Slavin (2008) calls “outcome measures inherent to treatments” (see especially pp. 11-12). The treatment specifically prepares students for completing a particular assessment task (mastering the rubric) rather than enhancing the students’ knowledge (mastering the content). When case rubrics are used to assess case learning, there is some risk of teaching to the test, and we see this to some degree in Passyn and Billups’s study. However, their rubric contains elements that are unique to each case. The rubric specifically asks for details on implementing the recommendations. This task touches on marketing expertise – the ability of learners to recognize patterns in a market and in their own organizations and to apply the knowledge of these patterns to identify the best opportunities (Bacon and Quinlan-Wilder, 2011). The choices and recommendations themselves, and not just the analysis, indicate the student’s potential value as a marketing manager.

One challenge in case learning research is clearly defining the learning goal and accurately measuring that outcome. In their review of the case teaching literature, Burgoyne and Mumford (2001) conclude that “The case method seems not to be attached to any existing learning theory” (p. 6; see also Mesny, 2013). One perspective for understanding case learning is the more general development of business problem-solving skills. When case analysis scores on a rubric are compared across cases that involve substantially different issues, as in Passyn and Billups, researchers are implicitly following this particular perspective (as did Abernethy & Butler, 1993, and Riddle et al., 2016). As noted, a possible problem with this approach is over alignment where students’ scores are heavily influenced by learning the rubric rather than learning problem solving. In Bacon and Quinlan-Wilder (2011, 2014), we addressed the challenge of assessing the ability to make better decisions by scoring students on the end result the quality of their marketing recommendations and not the quality of their analysis. This approach required the researcher to identify which recommendations were better than others for a given case, which is a task that presents its own challenges.

Another perspective for understanding case learning is analogical reasoning. Through studying cases, students may learn how to apply specific models such as pricing models or segmentation models. By applying the same models across several cases, students learn to see new situations as analogous to past situations. Unfortunately, working within this framework, Loewenstein, Thompson, and Gentner (2003) found that students do not develop analogical reasoning unless they are specifically asked to draw parallels between different cases and synthesize their understanding across cases. If the goal of case teaching is to facilitate mastery of specific marketing models or tools, a specific and somewhat unconventional approach to case teaching may be necessary. Whether the goal is to learn specific models or to make better marketing decisions, case learning researchers should clearly identify their learning goals and implement assessments aligned with those goals.

Of course, learning from cases may be quite different than learning through traditional methods. Banning (2003) demonstrated that tolerance for ambiguity can increase with case analysis. Stewart (1991) suggests that socialization is an important outcome of case discussion and analysis. In assuming the roles of various actors in a case, the students discuss and explore what actions would be socially appropriate or effective whether those actions are with subordinates, peers, managers of other divisions, or higher-level executives. At the same time, I wonder if case discussions help students develop social and communication skills in interacting with each other in the classroom, such as how to disagree with and confront each other with sound arguments in a way that isn’t belligerent or unprofessional. How these social learning outcomes can be assessed and optimized through case teaching is an important area for future research.

Given the complex nature of case-based learning, some null results in the literature are not surprising. When traditional exams are used to compare case-based learning and other forms of learning (e.g., lecture), important aspects of student development may not be captured (see, e.g., Miner, Das, & Gale, 1984 or Desiraju & Gopinath, 2001). Some new measure of learning may well be necessary to capture the benefits of case-based learning.

As with every study ever published, Passyn and Billups’s work has some shortcomings, and they present important areas for future research. A quite common limitation in marketing education research and one we see in Passyn and Billups’s study, is that the authors graded the students’ work, and they knew the hypotheses being tested. When a grader is aware of the experimental conditions, there always exists the possibility of bias. A better approach is to archive copies of the students’ analysis and have other evaluators score the work who are blind to the experimental conditions (cf., Riddle et al., 2016). Better yet, multiple raters could be used to enhance the reliability of the case write-up grades. With multiple raters, the reliability can be estimated, enabling an analysis of attenuation for measurement error (Nunnally, 1978, p. 237). This additional analysis provides insight into the true strength of the relationships between independent and dependent variables. Several of the effects in Passyn and Billups’s study are substantial and significant, but some of the smaller effects may have been significant if more reliable measures were used.

Another limitation in the Passyn and Billups research is that the experiments took place at one school. The effectiveness of these innovations across different schools is an important area for future research. An effective method in the Harvard MBA program may not be effective in an introduction to marketing course at a local community college. As Passyn and Billups demonstrate, it is important to describe the research site (e.g., public/private, size, demographic composition), the study’s sample (e.g., gender, graduate or undergraduate students), and method (online or face-to-face delivery). In future research, it would be helpful to see comparisons across different student populations.

Another challenge encountered in case learning research occurs when cases are used across many semesters, and case solutions begin to circulate among students at the research site. Case commentaries or partial solutions can be found for many cases on the Internet. Such may have occurred in Passyn and Billups’s experiments; however, at most, I believe this effect would be small partly because of the unique format of the written deliverable. Still, in the interest of raising the rigor in case learning research, we should ensure that case solutions are not more available to students in one experimental condition than in another.

I commend Passyn and Billups for the contributions that their study has made to the advancement of case learning research. As prevalent as case teaching is, there remains a dearth of rigorous research on case-based learning, and this study is an important step forward. My hope is that readers will learn from Passyn and Billups’s work and be inspired to conduct additional research in this important area.

To summarize, my recommendations for case learning research include the following: