Abstract
JME Conversation Starter for Special Issue on Assessment in Management Education
It is with great excitement as well as the heavy weight of responsibility that we engage in this scholarly public conversation about rubrics. Our thoughts, as represented by the commentary below, were both stimulated by and written in response to Riebe and Jackson’s article “Assurance of Graduate Employability Skill Outcomes Through the Use of Rubrics.” Having read two iterations of that article, and imagining that we are engaging in an active dialogue with others about this issue, we highlight three key messages that we believe warrant further consideration and discussion by the Journal of Management Education (JME) readership. The first relates to motivation. In an earlier draft of their article, Riebe and Jackson put forth the assertion that rubrics “encourage a sense of purpose” for faculty members. Although that phrase is no longer included in their article, we believe it accurately represents one of the core beliefs underpinning many advocates of rubrics in higher education. On this point we would tell a different story—our belief is that most if not all faculty members have a preexisting sense of purpose related to quality teaching and learning and that administrative mandates that are frequently interpreted as “telling us how to teach” are not particularly motivating. The second message we would like to revisit relates to the authors’ discussion of the design and effective implementation of rubrics. As part of our comments below, we believe there are numerous complex issues that were left out of the discussion in terms of both institutional and faculty member-based cost–benefit analyses. There is certainly a heavy cost borne by institutions and individuals when considering both the initial and ongoing investments required for effective rubric design and sustainability. Importantly, with each of the two discussion points above, we consider the impact of accreditation bodies and assurance of learning (AOL) pressures as part of the rubric decision-making puzzle.
In our final thoughts as part of this conversation starter, we question two interrelated assertions: (a) that a well-designed employability skills framework (ESF) is a tool that “applies equally” (Riebe & Jackson, 2014, p. 326) to business graduates at any classification level and (b) that rubrics are tools that can be used to convey “precisely” (p. 337) what a student has learned. We view well-constructed frameworks and rubrics as useful tools to use, where applicable, as general guidelines for latter individualization and tailoring. Our perspective here is that there is no teaching tool, framework, assessment mechanism, or other aspect of developing skills—employability and otherwise— in our field that will apply equally across students, faculty members, and educational environments as well as result in assessment precision. Although there are numerous well-designed frameworks and rubrics in place (including both the ESF and the rubrics that Riebe and Jackson put forth in their article), when the overarching goal is to create student-centered, lifelong learning-oriented experiences, there is simply no “one size fits all” that will apply across all higher education contexts.
Motivation and an Existing Sense of Purpose
Our first discussion point stems from a belief held by some faculty and administrators that is aptly represented in the following quote: “The implementation of holistic standards rubrics will encourage a sense of purpose [italics added] among HE practitioners to improve the quality of teaching and learning of employability skills by ensuring the alignment of learning outcomes to assessments” (drawn from the original Riebe and Jackson submission to this JME special issue). Although this statement is no longer in their article, we believe it is an important point for a discussion about an underlying belief held by many in terms of rubric implementation. Our belief is different. Our perspective is that most of the readers of this special issue will already be highly motivated, skilled, and invested in continuously improving the quality of their teaching and learning in the management discipline. We would argue that the premise that a centrally imposed teaching tool will encourage a sense of purpose for faculty members—particularly those who read JME—is misplaced. We see the forced implementation of rubrics in our educational environments as just another component of the creeping managerialism we all face in universities today. Much of this is attributed to the accreditation bodies our institutions are associated with (e.g., Association to Advance Collegiate Schools of Business, The European Quality Improvement System) and the pressure to produce acceptable evidence to tick the AOL boxes. With respect to faculty motivation, research indicates that the implementation of AOL practices may actually be demotivating for faculty members who are affected by resultant institutional changes (Pringle & Michel, 2007). In fact, data reveal that faculty members are expressing “ubiquitous faculty resistance,” “fear of the process,” and “apathy” with respect to AOL (Rubin & Martell, 2009, p. 371). It would appear that institutionally mandated implementation of assessment tools at any level (individual course, course cluster, program, or degree) will most likely increase neither motivation nor a “sense of purpose” for faculty.
With respect to motivational issues and teaching tools useful to the JME readership, there is no better description of what this journal, and its readership, is about than the June 2007 special issue of JME titled “Wisdom From Our Sages: Advice and Reflections for Early-Career Faculty.” In this special issue, the messages we read are about self-discovery, organic learning, and engagement. Although the issue is aimed at providing advice for faculty, there are no comments about “the one best way” to teach or the “best” or “only” tools to use. Rather, all of the comments in the issue are about experimentation, sharing, and trying new things with respect to motivation and practice. In the issue, we read about Kim Cameron’s advice to “own” whatever we teach, Jean Bartunek’s encouragement for academics to engage in a “reflection on their scholarly work in the context of their broader life” (p. 413), Mary Jo Hatch’s experiences that “fanned the flame of her ambition” (p. 408), and André Delbecq’s encouraging message that “a scholar’s best work arises from the intersection between his or her gifts and interests and important societal problems” (p. 390). These are the messages that resonate with JME readers. And although rubrics may be tools that some faculty members will find useful in terms of increasing their feelings of ownership over a topic or course, facilitating useful reflection, fanning their ambitions, and showcasing their interests, they are just one of many assessment tools available to faculty. For many of us who are now teaching in institutions that require the utilization of rubrics without consideration of individual faculty members’ approaches, experiences, interests, and ambitions . . . rubrics tend to weigh in toward the bottom of the motivating list.
Frameworks and Rubrics: An Ongoing and Significant Investment
When shifting to a discussion about the required integration of both frameworks and rubrics, which is the world many of us now find ourselves in, we would be remiss if we were to neglect issues stemming from the massive investment of resources such integration will involve on the part of already time- and other resource–poor faculty members. There are numerous activities that are both significant and complex required for effective rubric implementation. These include training and knowledge generation for faculty, creating and disseminating information about quality assurance, addressing issues stemming from variable interest on the part of faculty to engage, allocating time and other resources to the required iterative processes, and ensuring faculty voice (to name a few). Although Riebe and Jackson do a good job of discussing the work involved in preparing for their initial trials at Edith Cowan, we were left wondering what the actual start-up costs would be for a department-, school-, or organization-wide rubric implementation program. And then, if an entire organization’s constituents were to invest so heavily in the design and establishment of rubrics, what would the ongoing costs of an appropriate level of rubric maintenance be? What would those costs replace in terms of faculty members’ already scarce resources and overflowing “to-do” lists? A valid concern here is that once established as a part of an organization’s required AOL activities, rubric revisitation and redesign, although critical to effective implementation, will become part of the “creeping managerialism” we raised above. Rubrics run the very real risk of becoming yet another activity where miniscule revisions result in little to no benefit to the student yet are critical to the tick box approach to which so much of our work is now tethered.
However, in situations where faculty members make it to the point of wanting to experiment with rubric design, the process itself should be iterative and ongoing. We would argue that a dynamic, evolutionary view of skill development frameworks and their associated rubrics be seen as a central part of an overarching sense-making process. We would also hope that these tools be used to develop the criteria-based (as opposed to norm-referenced) guidelines for feedback and evaluation that are embedded in the learning process, not separated from it. Indeed, one of the hallmarks of successful rubric use is that they are continuously improved (Huba & Freed, 2000) and to do that takes time and interest on the part of everyone involved.
An additional point we would like to raise during this scholarly conversation is the presupposition that appropriate learning tasks and assessments are already in place as part of overall curricular design. We would argue that learning tasks are not all created equal. The choice, use, sequencing and other related decisions are often elements of the curricular design process that are highly valued (and quite time-consuming) by many faculty. These complex decisions and associated outcomes should be heavily influenced by an individual faculty member’s experiences, preferred approaches, and motivations. However, the implementation of rubrics for AOL purposes can often have a homogenizing effect on learning and assessment tasks. Such an outcome is the antithesis of the learning environment most JME readers work so hard to create for themselves and their students.
At this point, we should state explicitly that we do see the potential value in the use of frameworks and their associated rubrics, and in fact, we use rubrics regularly, albeit selectively, in our own teaching practice. We also acknowledge that the challenges and pitfalls of rubrics on a broad scale (e.g., program or institutional level) and the associated faculty resistance are frequently related to flawed implementation and poor change management practice (ironic, since this is what we teach).
Precision and Universality in Learning—We Think Not
Another topic we would like to raise as part of this conversation relates to the idea that there is precision and universality across teaching and learning contexts. With respect to this issue, we turn to a comment from Riebe and Jackson as a representative statement reflective of many rubric advocates. In their discussion of rubrics and employers they state, “Rubrics can inform industry precisely [italics added] what to expect from an undergraduate progressing through their learning program” (p. 337). As an alternative view, we believe that every aspect of the learning process is fluid and therefore inherently imprecise. Learning takes place in different ways for different people. Although there are certainly guidelines we can follow to create environments that are conducive to learning, and the ESF described in Riebe and Jackson’s article is a good place to start when considering skills development for many, there is no prescriptive formula or universal set of skills that will apply to every individual in every context. This is true even for those in “culturally similar developed economies” where we would argue that no framework will ever apply “equally” (p. 326). Rather, it is this dynamism and variability—this lack of precision and universality—that is the beauty and wonderment of learning. As educators, this organic fluidity and unpredictability with respect to learning hold some of our greatest opportunities as well as our greatest challenges, particularly when considering measurement and assessment. As David Justice (2006) writes in his foreword to Fiddler, Marienau, and Whitaker’s Assessing Learning, “As lifelong learning becomes more commonplace and accessible, the centrality of assessing learning will also grow because we have learned that individuals do not necessarily learn the same thing from the same experience” (p. vii).
As opposed to ignoring this challenge, we first need to recognize it and then work to create tools, systems, and environments that support faculty members rather than direct them. We understand that the general idea behind Riebe and Jackson's comment above is about using rubrics to improve communication with industry partners, which is certainly an important component of what we do. We would also like to note that in drawing on their use of the terms precisely and equally our aim was to illustrate beliefs held by many, not to engage in finger pointing. In the end, however, it is our firm belief that rubrics—the overarching focus of this conversation—should not be seen as a magic wand for creating precision in a learning environment; rather, they should be viewed and used as living documents designed to increase communication, feedback, and all parties’ investment in the learning processes.
With respect to using rubrics to communicate more effectively with industry partners, we note the following comment by Mary Huba and Jann Freed (2000) in their book titled Learner-Centered Assessment on College Campuses. They state, “we can use rubrics to inform audiences off campus – parents of students, practitioners in the field – about our intended learning outcomes and standards” (p. 173). Note that they do not claim precision in terms of communicating assessment outcomes; rather, they use the term intended. Like Huba and Freed, we acknowledge the usefulness of rubrics as tools that can help guide us in terms of our communication and understanding during organic learning experiences. Effective use of rubrics involves continuous effort in terms of communication with and feedback from students. As Dannelle Stevens and Antonia Levi (2005) state, The greatest way that rubrics begin to promote scholarly critical thinking is in the classroom discussion of the rubric prior to the student beginning the assessment . . . Further classroom discussion of the meaning of these critical thinking components can also clarify and explain the habits of mind we expect our students to demonstrate not only for a given assignment of class, but throughout their college careers and, for that matter, the rest of their lives. (p. 22)
Conclusion
In conclusion, although there are certainly a number of challenges associated with the effective utilization of rubrics, we agree that rubrics have the potential to be highly useful tools for faculty members to consider using. Our primary contribution to this scholarly discussion is to question the framing of rubrics as something akin to a panacea for assessment issues. They are not. As with any teaching tool, we believe faculty should never be forced to use tools that they are neither committed to nor adequately supported in the use of. Comfort and commitment to using a tool are critical components of effective teaching. Similarly, we believe the intricacies involved in effective utilization of both skill development frameworks and rubrics are much more complex than a single article can convey. Finally, we view learning as dynamic and organic—not cookie-cutter and precise. It cannot be measured perfectly through rubrics, nor can rubrics take the place of faculty member experience, passion, and motivation. As John Dewey (1910) so aptly stated many decades ago, “To nurture inspiring aim and executive means into harmony with each other is at once the difficulty and the reward of the teacher” (p. 221).