Complexity Around the Edges

In today’s higher education environment, the question of how to assess the value of what we as professors do should engage us all. Within this context, Donald R. Bacon and Kim A. Stewart’s (2016) essay, “Why Assessment Will Never Work at Many Business Schools,” is a laudable effort with important insights. Indeed, I am for the most part in substantial agreement with the authors’ analysis as far as it goes. My belief, though, is that the assessment picture is more complex around the edges than Bacon and Stewart describe. There is complexity at both the micro level—that of the individual instructor—and at the macro level—that of our collective ability as business professors to articulate measurable learning goals. While it is tempting to assume otherwise, this complexity needs to be an ever-present part of our assessment discourse.

Bacon and Stewart’s (2016) thesis is that business pedagogical research is often statistically problematic, mainly because of the frequent use of small student samples. Along with small sample sizes, they identify numerous practical issues such as low reliability, variable effect sizes, and impractically long learning cycles. Their proposed solution is to turn to the discipline of medicine for inspiration. Doctors have access to large-scale medical research in pursuing treatment of their patients in their own practices, so Bacon and Stewart believe that business educators should adopt a similar approach that draws on large-scale business pedagogical research in the scholarly literature. They argue that “we should embrace this model for assessing and for advancing student learning” (p. 9).

While their work most certainly illuminates in many ways a path forward for “measuring present learning outcomes, implementing curricular/pedagogical changes, and then measuring postchange [sic] outcomes” (p. 1), I believe they do so by marginalizing the complexities of some of the earlier steps of the assessment process. For example, they note two crucial steps in the assessment process, “set learning goals” (p. 2) and “align curricula with the stated goals” (p. 2), yet state that the “most difficult challenges” emerge in the “later steps of the assessment cycle” (p. 2). However, the most difficult challenges might be at the very start of the process; disregarding them in favor of attending only on later stage issues may bring an attractively clear picture of assessment possibilities into focus but does so at the cost of obscuring the rich lived-through experience of many professors.

Parker Palmer (1998) notes that with our teaching we frequently ask about the “what” (“set learning goals”) and the “how” (“align curricula with the stated goals”), but “seldom, if ever, do we ask the ‘who’ question—who is the self that teaches?” (p. 4). Palmer asserts, quite rightly I believe, that we must—each of us—reflectively gain a deep understanding of the self we bring to our teaching, our “identity” and “integrity” (p. 13) as he puts it, prior to making our choices about the “how.” If particular curricular approaches or techniques are to work successfully for me, they must be congenial to and in harmony with the self I bring to the classroom. To do otherwise is to risk deforming or even violating this self. This is the risk of an isolated emphasis on pedagogical technique; pedagogical techniques do not work in the abstract, but only work if they do so in the professional lives of actual instructors. How much can the experience of one unique teacher tell us about the best teaching techniques for another equally unique instructor? Often something, but also often not.

The “who” question is a frustrating one to bring to the assessment conversation because of its intractable nature and its resistance to quantification. Yet as teachers, we all know it is crucial. I recall a senior colleague of mine who possessed a far more aggressive and combative personality than I have, yet was almost universally loved by his students because of the way the externally coarse self he brought to his classes was intertwined with a deep affection for his students and an authentic desire for them to excel. Yet I am sure if I were to attempt to mimic some of his “in your face” techniques with my students, I would come across as bumbling, false, and even cruel.

All this is to say that we must remain ever mindful of the subjective side of our teaching experience even as we seek reliable measures of student learning. Implementing pedagogical or curricular changes without such mindfulness risks skewing our perceptions of what we more readily observe. In effacing the distinctive qualities that each of us as teachers brings to the classroom, we should be concerned about potential losses, both to students and ourselves.

The second issue around the edges of our assessment discourse that I would like to raise moves from the micro to the macro level. It may well be true, as Bacon and Stewart (2016) assert, that we have made much progress in our ability to clearly state measurable learning goals. Yet there is an unarticulated assumption within our concern for measurable learning goals that deserves emphasis in our larger assessment conversation. It can be brought to the fore by a closer look at the discipline of medicine that Bacon and Stewart (2016) advocate for as a model applicable to pedagogical research in business.

The question of whether a particular medical treatment works well has a different character than the question of whether a particular learning goal in business education has been well met. The standard of excellence within the large-scale medical research that informs medical practice is clear: A medical treatment works well if it improves the health of the patient. But the standard of excellence applicable to the practice of business—what it means to do business well—is much more contestable. This is because the larger purpose of business is contested terrain. The perennial discourse over whom the corporation serves is simply one example of this, and this issue has not escaped the attention of management scholars in evaluating the worth of management education. Christopher Grey (2002), for example, writes,

For at the heart of the problem of the economic and technical case for ME [managerial effectiveness] is the fact that its efficacy to practice and performance cannot, even in principle, be determined. One major difficulty is that there is no way of defining managerial effectiveness. (p. 478)

This means that the question of whether the learning goals of teaching business are well met is not subject to straightforward empirical validation in the way, say, whether a new cancer drug works well may readily be. For significant normative issues—and thus the need for ethical evaluation—are often embedded in the learning goals we as business professors put forth.

For example, a key learning goal articulated in the Social Science Research Council’s recent Measuring College Learning project (MCL) is that students master a nested conception of business in society. “The conception of business implicated in much of our discussion was the notion of business as nested or embedded in society” (Nesteruk & Beckman, 2016, p. 272). But the appropriate role of business in society is itself an ethical question. Thus, would the MCL learning goal be well met if the student learning here occurred solely within a framework of market values and dynamics? Some would claim so, believing that the freedom of market transactions allows us to maximize the general welfare as we individually seek the satisfaction of our personal preferences. But others worry about the ethical issues that are marginalized if we move, as Michael Sandel (2012) puts it, from a market economy to a market society. Because the deeper normative issues surrounding the ethically appropriate limits of markets are not readily resolvable in an empirical manner, the question of whether the learning goal of mastering this nested conception of business is well met never escapes the need for normative discourse.

Because of the way normative discourse is intertwined with a learning goal such as this, the analogy Bacon and Stewart (2016) propose between the validation of the efficacy of a new drug and the mastery of a learning goal in business education may at critical points be inapt, particularly if the learning goal is an ethically loaded one. The normative messiness generated by the question of what it means to do business well may be frustrating as we seek to sharpen our ability to measure the success of pedagogical techniques and curricular innovations, but it is a messiness that must remain an acknowledged part of our larger assessment discourse.

While Bacon and Stewart (2016) admirably move the assessment of business pedagogy forward in important ways, we need to remain cognizant of these issues around the edges of our assessment efforts if we are to realize our shared goal of improving the learning of future business practitioners. A rigorous evidence-based approach must never forget the often deeply personal struggles of the individual instructor and the larger perennial questions surrounding what it means to do business well.