Abstract
Development of critical thinking skills is an important outcome in education, though pedagogies to both promote and evaluate critical thinking present challenges and vary greatly. In this article, we describe the development and use of a formative and generalizable rubric that leverages the Paul-Elder model for critical thinking, and in particular, Intellectual Standards. When used consistently, this Intellectual Standards Rubric for Critical Thinking (ISRCT) provides regular and specific insight to students about strengths and weaknesses, related to critical thinking, that are reflected in their work. The ISRCT can also be used to assess multiple components of the same assignment, which allows instructors to measure critical thinking competency development in greater detail and provide this feedback to students. Though Public Health is the discipline and context for the development and application of this evaluation methodology, the ISRCT is adapted from an agnostic critical thinking framework and model, and thus, could be utilized for a variety of disciplines and diverse assignments.
Introduction
Critical thinking has long been held as an important goal for student development in higher education (National Education Goals Panel, 1991) and, as detailed by Johnson et al. (2019), is a necessary component of effective public health practice. As Elder and Paul (2009) state, “critical thinking is the art of analyzing and assessing thinking with a view to improving it” (p. 287), and therefore, evaluation is an integral aspect of critical thinking skill development. Whether the development of critical thinking is a college-wide learning outcome (Peach et al., 2007) or the goal within large classroom settings (Connors, 2008), a standardized analytic tool (i.e., a rubric) measuring approximations of critical thinking is necessary.
However, a gap has existed in incorporating critical thinking into student assessments and aligning evaluation criteria with it. In a systematic review of 63 descriptive rubrics, only two included measures of critical thinking (Brookhart & Chen, 2015). There remains a debate on the best strategies to teach and evaluate critical thinking. For instance, some methods include the intentional design of specific assessments and questions, but the developed rubrics are question-specific, and therefore not generalizable to other questions and/or applications (Bissell & Lemons, 2006). In addition, critical thinking rubrics are typically used to assess critical thinking separate from content competency (Reynders et al., 2020), which creates limitations and compartmentalizes the benefits of critical thinking away from content learning. For example, the Critical Thinking Analytic Rubric (CTAR; Saxton et al., 2012), measures critical thinking abilities such as interpretation, analysis, evaluation, inference, explanation, and disposition, but not a student’s understanding of content.
Other researchers have opted to develop more generalizable rubrics, including one developed from the Critical Thinking Assessment Test (CAT) instrument, which can be applied in different assignments and disciplines (Cargas et al., 2017). A common criticism of the CAT, however, is that it only provides summative feedback, rather than crucial formative feedback at regular intervals (Brown et al., 2014). Brown et al. (2014) articulate this need for formative value in critical-analytic thinking assessments, then further discuss and support the purpose of such evaluation by stating: “formative assessment systems must be capable of helping teachers understand students’ needs and strengths in all of the components of what is ultimately a grand performance (and its assessment)” (p. 558).
In this paper, we illustrate the development, application, and evaluation of an Intellectual Standards Rubric for Critical Thinking (ISRCT) which has been used in the evaluation of student assessments within several undergraduate public health courses at our institution over the past 8 years. This rubric builds upon previous efforts of Leist et al. (2012) to leverage the Paul-Elder model and Intellectual Standards as evaluation criteria. However, instead of measuring a single summative outcome such as reading comprehension (Leist et al., 2012), when used across multiple assignments and throughout a course as a formative assessment strategy, the ISRCT measures the development of critical thinking, student mastery of discipline-specific content, and provides consistent feedback to students.
Development of Intellectual Standards Rubric for Critical Thinking (ISRCT)
As one of the most widely used frameworks for learning and teaching critical thinking, the Paul-Elder Model was adopted by the University of Louisville in 2005 as the university-wide framework for student intellectual development. Components of this model (Intellectual Standards, Elements of Thought, and Intellectual Traits) adorn the walls of many classrooms across campus, serving as a reminder of the model’s importance to the university. Seeking to practically apply this model within the classroom, our public health undergraduate program saw an opportunity and an area for strategic development. As such, we designed assessments and evaluation criteria that leverage and operationalize the model, so that from start to finish, students complete assignments that are built upon and within a critical thinking framework, and their critical thinking competency is ultimately evaluated in their finished work. In a previous paper, we discussed one such assignment, the Scenario Analysis, that is framed using an adapted version of the Paul-Elder Elements of Thought (Johnson et al., 2019). In this follow-up paper, we discuss the evaluation criteria (i.e., the ISRCT) used to assess this and other assignments in many of our undergraduate program’s public health courses.
Using the nine Intellectual Standards identified in the Paul-Elder model for Critical Thinking—clarity, accuracy, precision, relevance, depth, breadth, logic, significance, and fairness (Paul & Elder, 2013)—our team developed the ISRCT to be used across public health undergraduate courses with the aim of improving students’ critical thinking development. To adopt the nine Standards into a flexible and efficient scoring rubric, the teaching team systematically considered each Standard, striving to identify the Standards that would be most generalizable (i.e., applicable to varied contexts), relatable (i.e., readily described and understood), and reproducibly scored by instructors. This led to the adoption of six of the Paul-Elder Standards—clarity, accuracy, relevance, significance, logic, and depth—and the combination of precision, fairness, and breadth—into completeness—as a seventh Standard (see Figure 1). The decision to combine precision, fairness, and breadth into completeness arose from the similarity of their practical operations, as described by Paul and Elder (2013): “could you give me more details?” (precision), “do we need to look at this from another perspective?” (breadth), and “am I sympathetically representing the viewpoints of others?” (fairness; pp. 32–33). Further, we found that evaluators have issues scoring accuracy and precision, when both are present, but one is in violation; with breadth, when evaluators’ subject knowledge is greater than learners’; and with fairness, when evaluators have no valid understanding of a learner’s biases, feelings, and/or interests.
Intellectual Standards mapping.
After finalizing the selected Standards for the ISRCT, the expected achievement of each standard was described and scaled from 1 to 5, with 1 as “does not partially meet standard,” 2 “partially meets standard,” 3 “nearly meets standard,” 4 “meets standard,” and 5 “exceeds standard.” As shown in Table 1, the ISRCT includes additional information to assist both instructors and students in the interpretation of each Standard. For example, synonyms and antonyms (parallels and opposites) are provided for each Standard. Further, to help articulate expectations and conceptualize each Standard at each level within the ordinal scale, the ISRCT includes analogies (rubric phrases) that serve as a guide for instructors, teaching assistants, and students.
The Intellectual Standards Rubric for Critical Thinking (ISRCT).
Adapted in part from Elder and Paul (2008).
Standards are evaluated independently of one another (i.e., a statement can demonstrate clarity, but be lacking in accuracy) which results in a composite score, allowing the evaluator to communicate with the learner how one or more Intellectual Standards could have been improved within their work. Of note, the ISRCT is visualized as a flexible rubric that allows instructors to choose the Standards most relevant for each assessment within a course. For example, while all assessments require inclusion of clarity, as it is the “gateway standard” (Paul & Elder, 2013, p. 32), depth may only be included infrequently. For further flexibility, each Standard can be weighted within an assessment to direct students’ attention toward the competencies the instructor is seeking to emphasize. For example, instructor expectations for demonstrating mastery of specific content can be evaluated by weighting the Standards of significance and completeness more heavily.
Application of ISRCT Within Course Design
In many of our public health undergraduate courses, the ISRCT is introduced in its entirety in course syllabi, followed by a short training on the first day of class. Students are provided various statements and are guided through determining whether each demonstrates specific Intellectual Standards. This introduction helps students better grasp Intellectual Standards and exemplifies expectations within the course. Additionally, students are provided de-identified exemplar responses for various assignments to serve as guides for their own work.
For example, in our Introduction to Public Health course, the instructor introduces students to the Paul-Elder model, including a discussion of the Intellectual Standards and the use of the ISRCT. The instructor explains how the four Standards of clarity, accuracy, relevance, and significance are used in scoring and emphasizes that clarity is a gateway Standard since a thought cannot be further evaluated if it is not clearly expressed (i.e., “If I can’t figure out what you’re saying, I can’t figure out whether you’re saying anything worthwhile”). Accuracy is evaluated next since a statement may appear relevant or significant but cannot achieve a useful level unless it is accurate. As an example, “Your statement that some people get the flu from their flu shots would be worth considering if it were accurate—it isn’t.” Additionally, attending to the Standards of relevance and significance focuses attention on important details, priorities, and their urgency, which is important in public health, for example, to avoid risking lives and/or wasting finite resources. Similarly, a statement can be relevant but not significant, or vice versa, and the evaluation of each is contingent upon the context of the statement.
The Standard of completeness is most often used in Scenario Analyses (see Table 2) where, without completeness, the analysis of a public health situation may not include the information needed for formulating, selecting, and supporting an optimal plan of action. When the plan is incomplete, the likelihood of failure to achieve the intended outcome dramatically increases, that is, “before we spend over $5 million dollars on this plan, are there other options we ought to consider?”
Selected Assignments and the Application of the ISRCT.
Note. References used in table for types of assignments: Scenario Analysis (Johnson et al., 2019), Critical Reflection in DEAL format (Ash & Clayton, 2004), SEE-I (Nosich, 2012).
Applied to multiple iterations of varied evidence-based assignments (Table 2), the ISRCT is systematically used throughout a course to assess students’ application of critical thinking within public health course content. The assignment-specific ISRCT with the selected Standards and weights is included with instructions for each assignment. Through the consistent use of the ISRCT, students are expected to demonstrate improvement in critical thinking skills, as well as mastery of course content.
Incorporating ISRCT Within Learning Management System
The ISRCT can be used to score assignments “by hand,” as a Microsoft Excel template, or within a learning management system (LMS) such as Blackboard Learn. Working with an instructional designer at our institution, the ISRCT was uploaded into Blackboard using the rubrics tool for ease of use in multiple assignments across courses. Once associated with a specific assignment, the weighting for each Standard can be customized per the instructor’s requirements.
Using the ISRCT within an LMS or Microsoft Excel template allows for easy evaluation of multiple assignments within a semester. This has three major benefits: (1) students can view their score within each Standard across varied assignments, allowing them to determine which Standard they need to focus on improving; (2) instructors can monitor Standards’ scoring across the class and provide additional time and examples for teaching specific Standards; and (3) instructors have much more data going into the final assessment of each student, which is helpful, for instance, to compare and contrast team-based scores with those of individual student assignments.
From a course design perspective, it is important to ensure that students are able to quickly reference the evaluation rubric while completing assignments as a way of organizing their efforts and performing a self-evaluation of their level of performance. A link appears next to each assignment name within the electronic gradebook and next to the submission point for each assignment within Blackboard. Not only does this ensure students have easy access to the rubric for each assignment, the regular appearance of the ISRCT throughout the digital course environment helps to underscore the importance of critical thinking as a course goal.
As a formative assessment tool, the ISRCT itself provides feedback to students about competency related to each Intellectual Standard, though additional feedback may be necessary. For example, one method of providing such feedback involves referencing specific examples identified within a student’s submission, using the highlight and/or comment function. The rubrics within Blackboard were designed to include some standardized language for each level of performance, though they also include an optional text field where the grader can provide custom feedback to the student.
Also important is the ability to track critical thinking skill development across assignments. Most LMS include a tool to support this type of functionality. In Blackboard, a rubric evaluation report can be pulled to provide information about how well groups of students are performing against the Standards outlined in the ISRCT. The report also provides a frequency distribution showing the distribution of scores across each level of achievement. This report can be run multiple times throughout the course to track student progress and can also be used to compare groups of students between sections and terms of the course. Blackboard also maintains records on individual student performance on the rubric through the Grade Center.
Evaluating Use of ISRCT
The ISRCT has been used consistently within our public health undergraduate program since 2014, across multiple courses each semester. Through this consistent use of the ISRCT, our experience indicates that, within a semester, students demonstrate marked improvement with Intellectual Standards and overall quality of completed assignments.
Typically, the first time students are assessed with the ISRCT is in the Introduction to Public Health course through multiple iterations of Class Analyses (CAs), Scenario Analyses, and Critical Reflections. To exemplify, CAs are used multiple times in a semester, asking students to answer in succinct written form at the end of class: “What was the single most important point you learned in today’s class?” Each CA is scored using the ISRCT with only four Standards (clarity, accuracy, relevance, and significance). To promote self-evaluation, students also score themselves using the same ISRCT, which helps them internalize and better understand the Standards.
Using data from four semesters of Introduction to Public Health, we calculated a paired t-test score (using Microsoft Excel) for each semester, using the mean CA scores at the beginning and middle of the semester (see Table 3). For Semester A, we had available disaggregate data of each scored Standard. All paired t-tests demonstrated significant results at the 0.05 level of significance. The data show that, on average, students’ CA score significantly improves across a semester, with an average increase of 3 out of 20 total points between CA’s in the first week and mid-semester. Additionally, when broken down into the specific Standards, data demonstrate students’ scores significantly improve for each measured Standard.
Mean Scores for Class Analyses Across Four Semesters Demonstrating Significant Improvement in ISRCT Scores From the First Week to the Mid-Semester
Typically, students first learn to improve clarity in their responses. For example, in the first CA of the semester, a student answered: “Today that healthcare contains [sic] to individuals and public health has to do with general population.” Using the ISRCT, the response was scored 14/20 (Clarity—3/5, Accuracy—4/5, Relevance—4/5, Significance—3/5) and specific feedback was provided encouraging the student to re-read their responses for improving clarity. Students that have achieved clarity typically need to work on improving significance. For example, for a later CA, a student answered: “I learned a lot about the ACA and the timeline about it and how it affects a lot of the healthcare system.” This student was provided feedback encouraging them to focus on explaining why and how this point was significant. Additionally, to promote further understanding of each Standard, the instructor may highlight and discuss exemplar responses at the beginning of the next class session. For this CA, one exemplar student response was: “The ACA has significantly decreased the number of people who are uninsured, but it still has many issues to work out like high premiums for some people. It is unknown what will happen to the ACA at this point.” Therefore, by integrating Intellectual Standards directly into the evaluation criteria, our data show that students develop and improve upon specific aspects of their own critical thinking skills while working through course content.
Lessons From Use of ISRCT
The ISRCT has several benefits allowing pedagogical flexibility between assignments for the instructor and consistency of expectations for the learner. For the instructor, the ISRCT is useful in the creation and selection of assessments within a course, as well as with the overall design of the course itself. The ISRCT supports division of an assignment into sections, each of which can be evaluated individually, often with different weightings. The Blackboard LMS can be programed to automatically calculate a final score for an assignment as the sum of the sections’ scores weighted by section. The regular appearance of the ISRCT throughout the digital course environment helps students to quickly reference it while completing assignments, as a way of organizing their efforts and performing a self-evaluation on their level of performance. Consistent evaluation with the ISRCT on multiple assignments emphasizes critical thinking as a course goal, and at the same time, provides the ability to track students’ critical thinking skill development across assignments.
Further, the ISRCT design supports transparency, both in terms of grading methodology and defining expectations for students. Importantly, it allows the evaluator to communicate with learners specifically about how they lost points and which of the Intellectual Standards could be improved. In this way, the ISRCT can simplify the process of grading and providing feedback to students. While explicit and additional feedback to the rubric may be helpful early in the course, the score for each Intellectual Standard within each section of an assignment serves in and of itself as feedback to the learner.
An additional finding is that the ISRCT allows for more streamlined training between instructors and teaching assistants and even between instructors of other courses. In our experience, the rubric phrases and respective analogies are helpful guides in visualization of the grading criteria when working with teaching assistants and colleagues. By reviewing and scoring the same assignment (i.e., a class analysis) during training, the ISRCT serves as a conversation guide on how and why the student artifact meets or does not meet specific Intellectual Standards.
Conclusion
The Intellectual Standards Rubric for Critical Thinking (ISRCT) fulfills the need for a practical critical thinking assessment framework, that is generalizable for the evaluation of a myriad of assessments, and formative in providing students with granular insight into their strengths and deficiencies related to critical thinking. The success of the ISRCT is enhanced by the instructors’ commitment to it for evaluation criteria across multiple assignments. This consistent application of and evaluation through the ISRCT affords students with more opportunities to master individual Intellectual Standards, particularly those Standards with which they may struggle. Using the same evaluation criteria with different assignments also reinforces the importance of the Intellectual Standards themselves and further promotes students’ application of critical thinking to demonstrate their knowledge of course content in diverse assessments.
Though additional research and evaluation is ongoing, the ISRCT provides a way to both quantify and track critical thinking skill development across assignments, and potentially curricula. This novel approach provides another opportunity to operationalize a critical thinking framework such as the Paul-Elder model, and perhaps most importantly, helps students develop the critical thinking skills necessary for successful careers in Public Health and beyond.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
