Exploring Critical Thinking as an Outcome for Students Enrolled in Community Colleges

Abstract

Objective: Using data from HEIghten^® Critical Thinking, a student learning outcomes assessment, the purpose of this study was to evaluate what variables are associated with higher critical thinking performance for students enrolled in various community college programs and to evaluate performance differences across demographic and college-level subgroups as well as student perceptions. Method: With data from 1,307 students enrolled across 34 U.S. and Canadian higher education institutions (72% enrolled in 2-year institutions), we utilized a hierarchical regression to identify variables associated with critical thinking performance. Critical thinking performance differences were evaluated using analyses of variance (ANOVAs) and t-tests across student demographic and college experience subgroups and across student perceptions. Results: Results of this study showed (a) consistent significant predictors associated with higher critical thinking performance; (b) a positive relationship between critical thinking performance and the frequency of using critical thinking in college courses; (c) significant, but relatively small performance differences across demographic and college experience subgroups; and (d) positive relationships between student perceptions and critical thinking performance. Conclusion: This study added to the limited literature evaluating critical thinking skills for community college students. Overall, results suggest that institutions should focus attention to the frequency at which students are using critical thinking throughout their courses, which could increase student performance in this particular area, especially if critical thinking is an explicit outcome within the course. Results also suggested the need to emphasize critical thinking skills more across various community college programs and across non-STEM-focused programs. Suggestions for future research are discussed.

Keywords

critical thinking community colleges learning outcomes subgroup differences

Community colleges play an important role in higher education, offering numerous educational programs that lead to associate degrees and certificates (The College Board, 2019; Community College Review, 2019). Specifically, students receiving postsecondary credentials such as associate degrees and certificates represent a growing group of students within higher education, with this population of students representing 51% of all undergraduate credentials awarded (Zhang & Oymak, 2018). Most students receiving these types of credentials attend public 2-year institutions and represent nearly half of all undergraduate students in the United States (American Association of Community Colleges [AACC], 2019a), and there has been an increase in the number of certificates and associate degrees conferred. Specifically, the number of certificates awarded across institutions from 2002–2003 to 2012–2013 rose by 49%, and the number of associate degrees increased by 59% compared with a 36% rise in bachelor’s degrees (Kena et al., 2015).

As compared with 4-year institutional programs, students enrolled in community college programs tend to be slightly older in age, are more likely to be the first in their family to attend college, and may also be working full- or part-time (AACC, 2019a; Zhang & Oymak, 2018). Students’ knowledge and skills for this group may be more impacted by life experiences outside of college (e.g., from working a job) as compared with their peers enrolled in bachelor’s degree programs, making it difficult to capture what students are learning as a result of the courses and activities they are engaged in within the institution (Nunley et al., 2011).

Community College Pathways or Programs

The purpose of community colleges and their missions have been debated for decades (see Ayers, 2015; Desai, 2012; Dougherty & Townsend, 2006). The missions that community colleges emphasize have changed and evolved over time, and in many cases, still vary across regions of the country (Desai, 2012; Dougherty & Townsend, 2006). Some common missions include workforce development or occupational training, transfer to 4-year institutions, remediation and development, community service, and ensuring open access to higher education for all students (Desai, 2012). A study using a corpus of 2012–2013 mission statements at community colleges found that there has been a greater emphasis on degree completion and a lesser focus on occupational and vocational training (Ayers, 2015). Results from this study also showed an increased focus on student outcomes related to success, competencies, and engagement (Ayers, 2015).

In support of these various mission statements, there are several collegiate pathways or programs that students can enroll in at community colleges, such as associate degree, transfer, certificate, or career and vocational programs. Students may also enroll in courses for workforce training or personal lifelong learning. Associates degree programs are typically 2-year programs that provide basic technical or academic knowledge and transferable skills for a professional career or a 4-year bachelor’s program (de Brey et al., 2019). Transfer programs are typically programs that involve a direct pathway into a bachelor’s degree program and don’t necessary result in a 2-year associate degree, distinguishing them from the associate degree pathway (A. M. Cohen et al., 2014; Community College Review, 2019). Certificate programs are shorter than an associate degree program, and are either occupational, focusing on career-related skills aligned to workforce fields (e.g., health care or education), or academic, which are decontextualized from the labor market (e.g., humanities or social sciences; Research for Action, 2019; Sykes, 2012). Certificates may also be earned to supplement an existing degree or be required for employment (Sykes, 2012). Career and vocational programs are career-oriented programs aligned to a particular field, such as culinary arts or dental hygiene, that prepare individuals for entry-level technical positions in business or industry. This type of pathway may result in either a professional certificate or an associate degree (A. M. Cohen et al., 2014). Finally, students may enroll in courses for workforce training and lifelong learning through continuing education programs to advance knowledge for their professional careers or to increase general knowledge in a particular area. It is important to note, however, that even though we present distinct categories of possible collegiate programs or pathways, these various pathways are likely to overlap for some students because “higher education is rarely discrete” (A. M. Cohen & Brawer, 2008, p. 26). For instance, there may be a course that is as aligned with a particular career and vocational pathway but may also be taken by some students for general education or continuing credits.

Critical Thinking as a Learning Outcome

Understanding the knowledge, skills, and competencies that students learn as a result of their college experience can be a challenge, especially at community colleges, given that students may be enrolled in various pathways and because many students are not necessarily enrolled full time (Voluntary Framework of Accountability, 2013). As a result, the National Institute for Learning Outcomes Assessment (NILOA) developed the Transparency Framework (NILOA, 2011) to help guide institutions on how to gather evidence of what students are learning. This framework includes creating student learning outcomes (SLO) statements, creating an assessment plan, and using those assessments to provide evidence of student learning (NILOA, 2011). These SLOs are typically aligned with the mission of the college. Given the diversity of students enrolled in community colleges (in terms of their pathways or goals) and the various missions at community colleges, traditional measures of accountability to capture student learning on various SLOs (e.g., critical thinking, writing, mathematics) may be more of a challenge to implement at community colleges (AACC, 2019b). As more institutions reshape their curricula and course offerings in response to Guided Pathways reforms (Bailey et al., 2015), this may change, but as of yet, there is limited extant research evaluating SLOs at 2-year institutions.

One SLO that has been of increased interest is critical thinking. Critical thinking is consistently recognized as an essential and necessary skill by higher education institutions (Hart Research Associates, 2016), and by employers (American Management Association, 2012; Casner-Lotto & Barrington, 2006; Hart Research Associates, 2015; National Association of Colleges and Employers [NACE], 2018). For community colleges, critical thinking is essential for students engaged in vocational training or for those students considering or preparing for transfer into a 4-year program or institution (Fong et al., 2017). Research has shown that college experience can help develop students’ critical thinking skills (Huber & Kuncel, 2016), and that it can be enhanced through educational interventions (Abrami et al., 2008).

Although critical thinking has been deemed important for success in higher education and by employers, there are still numerous debates about its definition and what it means to be an effective critical thinker (Liu et al., 2014). Critical thinking has been defined in terms of skills such as the ability to reason effectively, use systems thinking, evaluate arguments, make judgments, or form appropriate questions (e.g., Binkley et al., 2012; Liu et al., 2014; Quality Assurance Agency, 2008). Halpern (2003) defined critical thinking as a tool to facilitate decision-making or problem-solving. Critical thinking has also been defined as a disposition or “habit of the mind” (Rhodes, 2010, p. 1) where aspects of critical thinking are associated with personality characteristics such as inquisitiveness (N. C. Facione et al., 1994). These various research efforts and ways of defining critical thinking have led to a number of different measures or assessments such as the California Critical Thinking Skills Test (P. A. Facione, 1990), California Critical Thinking Disposition Inventory (P. A. Facione & Facione, 1992), Halpern Critical Thinking Assessment (Halpern, 2010), Collegiate Learning Assessment+ (Council for Aid to Education, 2019), and HEIghten Critical Thinking (Educational Testing Service, 2019). Liu et al. (2014) summarized many of these different assessments in their critical thinking framework paper.

Specific to community colleges, a recent meta-analysis conducted by Fong and colleagues (2017) explored the relationship between community college student success and critical thinking skills. Student success was defined in terms of outcomes such as retention in community college, degree attainment, and course completion or achievement-related outcomes (i.e., grades, grade point average, or tests). The authors identified 23 studies in their meta-analysis, the majority of which were unpublished doctoral dissertations, suggesting that there are a limited number of studies available that evaluate critical thinking skills for students enrolled at community colleges. The meta-analysis included studies that utilized both skills and dispositional measures of critical thinking. Overall, their study revealed a positive relationship between critical thinking and student success (r = .28). The authors stressed the importance of focusing on the development of critical thinking skills to successfully prepare students to enter the workforce.

Pascarella et al. (1994) used data from the National Study of Student Learning (NSSL), a longitudinal study investigating the factors that relate to learning and cognitive development in college, to investigate the academic rigor of first-year students at eleven 2- and 4-year institutions. Using the Collegiate Assessment of Academic Proficiency (CAAP) to evaluate reading comprehension, mathematics, and critical thinking at the end of freshman year of college, this study compared the performance of students at 2- and 4-year institutions. The authors used a matched group of six 4-year institutions whose incoming student body resembled those of the five 2-year colleges in terms of academic preparation. After controlling for numerous demographic, precollege, and college-level variables, no significant differences in performance were found. Two-year institutions actually showed slightly higher performance in mathematics and on the composite scores.

Liu and Roohr (2013) investigated learning trends at community colleges from 2001 to 2010 using the Educational Testing Service (ETS) Proficiency Profile which measures reading, writing, mathematics, and critical thinking. Results showed that performance at community colleges has remained relatively stable and that students at community colleges performed on par with students from liberal arts colleges but underperformed when compared with comprehensive and research institutions. This study also demonstrated concurrent validity evidence by showing that performance among community college students increased as students earned more credit hours within the institution.

Study Purpose and Research Questions

Despite the widely recognized importance of critical thinking skills, there is still limited research around critical thinking as a learning outcome at community colleges. In Fong et al.’s meta-analysis of 23 studies, only two of the studies were published reports or articles and dated back to the late 1980s, and the rest were unpublished dissertations. If critical thinking is an essential skill for the workplace (American Management Association, 2012; Casner-Lotto & Barrington, 2006; Hart Research Associates, 2015; NACE, 2018), then more research is needed to understand whether community colleges are preparing students to be critical thinkers and what variables are associated with higher levels of critical thinking. Understanding the variables associated with higher levels of critical thinking could help community colleges target ways to improve students’ critical thinking skills.

Using data from HEIghten Critical Thinking, an SLO assessment measuring critical thinking skills, the purpose of this study was to evaluate what variables are associated with higher critical thinking performance for students enrolled in various community college programs and to evaluate performance differences across demographic and college-level subgroups as well as student perceptions. It is important to note that the HEIghten assessment was designed for all college students, regardless of the type of program enrolled in or college major. Therefore, this study investigates an intended subpopulation of test takers. We evaluated the following research questions:

Research Question 1: What student and college experience variables are associated with higher performance on the critical thinking assessment?

Research Question 2: What are the differences in critical thinking performance across various demographic subgroups and subgroups related to college experiences?

Research Question 3: What are the student perceptions of the assessment and how are their perceptions related to critical thinking performance?

Method

Data and Sample

Data for this study included 1,307 students enrolled across 34 U.S. and Canadian public and private higher education institutions. These institutions already used the HEIghten Critical Thinking assessment as part of the assessment process at their institution. Although approximately 72% of students across these institutions were enrolled in 2-year institutions (i.e., community colleges), some students were enrolled in 4-year or specialized (e.g., culinary) institutions, as it is possible to receive an associate degree or certificate from many 4-year institutions. The sample was approximately 44% male; the majority of students were White (58%) and spoke English as a first language (75%). Most students were between 17 and 21 years of age (73%) and had completed fewer than 30 credit hours (68%). Most students were enrolled in certificate programs (43%) followed by career/vocational (21%) and associate degree (22%) programs, and just under half the sample (45%) were in STEM (science, technology, engineering, and mathematics) majors (see Table 1).

Table 1.

Demographic and College-Related Variables.

Demographic variable	N	%
Male^a	576	44
English as a first language	985	75
White^b	752	58
Age range (years)
17–18	515	39
19–21	444	34
22–25	167	13
26–35	137	11
35+	43	3
Program enrolled
Associate degree	282	22
Certificate	557	43
Career/vocational	277	21
Life-long learning/workforce training	122	9
Transfer program	69	5
College credit hours completed
<30 credit hours	891	68
30+ credit hours	416	32
College major^c
STEM	583	45
Non-STEM	398	31

Note. N = 1,307. STEM = science, technology, engineering, and mathematics.

Gender was missing for 154 students (11.8%). An additional 11 students (0.8%) selected “Other” or “I prefer not to respond.” ^bRace was missing for 190 (14.5%) students. ^cCollege major was missing for 326 (24.9%) students.

Motivation filtering

Test-taking motivation can be defined as the willingness to engage as well as invest effort and persistence in working on test items (Baumert & Demmrich, 2001), and low motivation can pose a threat to the validity of test scores in situations where assessment results have little or no consequences for test takers. Although some students received a score for the critical thinking assessment, that score did not impact their course grades or college requirements, so there was no academic consequence for student participants. As a result, students may not have been motivated to try their best on the assessment despite the institutions’ best efforts to motivate students (e.g., through extra credit, financial incentives, or clear instructions that discuss the importance of the assessment for the institution). Based on previous research (e.g., Finn, 2015; Hauser & Kingsbury, 2009; Rios et al., 2014; Wise & Kong, 2005), we applied a response time filter to identify random guessing and unmotivated examinees. Specifically, a test taker was removed if they (a) failed to answer at least 75% of items or (b) responded in less than 10% of the median item response time for 20% of items.

Based on these motivational filters, we removed 190 students (13%), which resulted in the 1,307 students used for this study. The flagged students performed 5.53 points or 0.90 standard deviations (SDs) lower than their motivated peers, Welch’s t(308.88) = 14.65, p < .001.

Instrument

The HEIghten Critical Thinking assessment was used as the SLO assessment for this study. Data were from a larger pool of test takers who took the assessment between November 2015 and June 2017. This assessment is one module out of a five-assessment HEIghten Outcomes Assessment suite intended to measure general SLOs for all college students regardless of college major. These generic skills assessments are intended to be used mainly at the institution level, providing group-level information about student learning. Institutions selected to use this assessment for purposes such as informing regional and program accreditation, external accountability, curriculum modification, institutional improvement, and benchmarking performance both externally and internally. These assessments can also be used at the individual level to provide information about students’ overall or total performance within a module and performance levels in these various competencies (Educational Testing Service, 2018a).

For the purposes of this assessment, critical thinking is operationalized as having two central aspects: analytic and synthetic skills. Analytic skills include analyzing argument structure, evaluating argument structure, and evaluating evidence and its use. Synthetic skills include developing valid (structurally strong) or sound (evidentially strong) arguments and demonstrating understanding of the implications of information and argumentation. See Liu et al. (2014) for a more detailed description of the operational definition of critical thinking. When administering this assessment, two test forms for the assessment are randomly administered. Total scores range from 150 to 180, and the two subscores (Analytic and Synthetic) range from 1 to 10. Students are given 45 minutes to complete this computer-based test, which is comprised of 26 selected-response items. All test items use a variety of real-life, authentic contexts. The test has been thoroughly validated (see Educational Testing Service, 2018b; Liu et al., 2016, 2018) following the Standards for Educational and Psychological Testing (American Educational Research Association et al., 2014). The HEIghten test scores are reasonably correlated with college admissions scores, high school GPA, and college GPA (Liu et al., 2016). The reliability is .74 at individual student level and .95 at institution level (Educational Testing Service, 2018b).

After students complete the assessment, they are also administered a background information questionnaire asking about their demographic characteristics (e.g., age, gender, race, ethnicity, language) and academic experience (e.g., credit hours, college major, courses related to critical thinking). In addition, a posttest survey asks questions about the reasons for taking the assessment, self-rated critical thinking skills, and student perceptions of the assessment (i.e., test difficulty and testing time). For the purposes of this study, we were limited to the variables in the background information questionnaire and posttest survey and thus did not have access to some variables that may have been helpful in evaluating this population of students (e.g., first-generation college student status, income, etc.).

Data Analysis

Variables associated with critical thinking performance

To identify variables associated with critical thinking performance, we conducted a hierarchical regression to predict the total critical thinking score. This procedure allowed us enter predictors into the model in a prespecified order and in blocks to isolate their impact on the outcome. Dummy variables were created for race (White, non-White¹), gender, primary language (English, other), self-rated critical thinking skills (excellent/good, average/fair), credit hours completed (less than 30, 30 or more), college major (STEM, non-STEM), type of college program (individual dummies for associate, career, lifelong learning/workforce, and transfer program; certificates is the reference group), courses in critical thinking (1 or more, none), and frequency of courses requiring critical thinking skills (frequently/sometimes, rarely). These are the variables that were available to us based on the posttest and background information questionnaire. To run the hierarchical regression, we first entered the demographic variables: age, gender, race, and language. The second regression included students’ self-rating of their critical thinking skills. The third regression included those variables associated with status in college: credit hours completed and college major. The fourth regression included variables related to type of college program, and the fifth regression included the variables associated with their coursework and critical thinking skills (number of courses and frequency using critical thinking in courses).

Demographic and college experience subgroup differences

Demographic subgroups included age (17–18, 19–21, 22–25, 26–35, and 36+ years), gender (male and female), race (White and non-White), and whether test takers spoke English as a first language (yes or no). College experience subgroups were based on two posttest survey questions related to students’ college coursework. The first question asked about the number of college-level courses taken related to critical thinking (0, 1, 2, and 3+), and the second question asked students about the frequency of their coursework requiring them to think critically (frequently, sometimes, and rarely). Additional college experience subgroups included the types of programs students were enrolled in (associate degree, career/vocational, certificate, lifelong/workforce learning, and transfer programs),² college major (STEM, non-STEM), and credit hours (<30, 30+). One-way analyses of variance (ANOVAs) were used to evaluate performance differences across age groups, the two posttest survey questions, and types of programs students were enrolled in, and t-tests were used to evaluate performance differences between gender, race, first language, college major, and credit hours.

Student perceptions

The posttest survey asked students to self-rate their critical thinking skills and asked students about the difficulty of the test and the adequacy of testing time for the critical thinking assessment. To evaluate performance differences in self-rated critical thinking skills (excellent, good, average, and fair) and perceived testing time (not enough, enough, and more than enough), two separate one-way ANOVAs were conducted. Prior to each analysis, we tested for homogeneity of variance using the Levene statistic. If this statistic was statistically significant (meaning that variances were unequal), we reported results using Welch’s ANOVA test.

Omega-squared (ω²) was used to evaluate the ANOVA effect size where .01 is a small effect, .06 is a medium effect, and .14 is a large effect. If the ANOVA was statistically significant, we conducted post hoc analyses with the Bonferroni correction to evaluate individual differences between groups and Cohen’s d for effect sizes where 0.20 was considered small, 0.50 moderate, and 0.80 is large (J. Cohen, 1988). Note that even a small effect can be viewed as important depending on the theory being tested (Gall et al., 2007).

To evaluate differences in test difficulty, we conducted a t-test to look at the performance differences in total score between students who reported that the test was at the right difficulty level and students who reported that the test was too difficult. Again, prior to the analysis, we tested for homogeneity of variance using the Levene statistic. If this statistic was statistically significant, we reported results using Welch’s t-test. Cohen’s d was used to evaluate the magnitude of the differences.

Results

Factors Predicting Performance

Table 2 provides the results of a hierarchical regression predicting total scaled score. In Model 1, gender and race were significantly associated with performance, with males and White students outperforming their female and non-White peers, respectively. Model 1 explained 3.5% of the variance. Adding students’ rating of their critical thinking skills (Model 2) explained an additional 2.4% of the variance with students self-rating as excellent or good in critical thinking skills significantly outperforming their peers with lower self-ratings. Age also became a statistically significant predictor; as student’s age increased, they performed higher on the critical thinking assessment. The college experience variables (Model 3) explained an additional 3.4% of the variance, but gender was no longer statistically significant. Results showed that students with 30 or more credit hours performed lower than students with less than 30 credit hours, and that STEM majors outperformed non-STEM majors when controlling for other variables.

Table 2.

Hierarchical Regression Predicting Total Scaled Score.

Predictor variables	Model 1	Model 2	Model 3	Model 4	Model 5
Constant	157.17***	156.25***	153.94***	152.22***	150.96***
Age	0.08	0.09*	0.12**	0.12**	0.12**
Gender (male)	1.06*	0.89*	0.44	0.49	0.50
English as a first language	0.06	–0.12	–0.04	–0.14	–0.19
Race (White)	2.37***	2.31***	2.10***	1.95***	1.95***
Self-rated CT skills		1.99***	2.10***	2.07***	1.97***
Credit hours (≥30)			–1.12*	–1.30**	–1.32*
College major (STEM)			1.99***	1.93***	1.91***
Associate				3.22***	3.23***
Career/vocational				1.80*	1.78*
Life-long/workforce learning				1.27	1.29
Transfer program				1.37	1.34
No. of CT-related courses					0.21
Classes require CT skills					1.32
R ²	.035***	.058***	.090***	.106**	.106

Note. CT = critical thinking; STEM = science, technology, engineering, and mathematics.

p < .05. **p < .01. ***p < .001.

For Model 4, an additional 1.6% of the variance was explained, with a statistically significant change in R-squared. Of these variables, associate and lifelong learning/workforce training were statistically significant, with students enrolled in those types of programs performing 3.23 points and 1.80 points higher, respectively, than their certificate peers when controlling for other variables. Adding course-related variables (Model 5) did not significantly improve the R-squared. Variables associated with race, self-rated critical thinking skills, credit hours, and college major were consistently significant across models.

Performance Differences

Demographics

Figure 1 shows the mean performance differences across age, gender, race, and whether a test-taker’s first language was English. One-way ANOVA results revealed statistically significant differences across the five different age categories with a small effect, F(4, 1,301) = 2.65, p = .032, ω² = .005; however, after the Bonferroni adjustment, there were no significant differences by age group.

Figure 1.

Critical thinking performance differences across demographic categories.

Focusing on gender, males statistically significantly outperformed females by 0.17 SDs, Welch’s t(1,136.15) = −2.85, p = .005, but this effect was small. A slightly larger difference was found between White and non-White students with White students outperforming by 0.29 SDs, Welch’s t(1,177.85) = −5.25, p < .001. There were no statistically significant differences in performance between students who spoke English as a first language and those who did not, t(1,305) = 0.29, p = .770.

College courses

Results revealed that there were no statistically significant differences in terms of performance based on the number of courses taken related to critical thinking skills, F(3, 1,273) = 0.77, p = .510. However, there were statistically significant performance differences across groups, F(2, 1,245) = 6.10, p = .002, ω² = .008, in relation to the frequency with which college courses require you to think critically (Figure 2). Post hoc analyses using Fisher LSD revealed that students who reported “frequently” using critical thinking skills performed statistically significantly higher than those indicating “sometimes” (p = .003, d = 0.18) or “rarely” (p = .019, d = 0.30).

Figure 2.

Critical thinking performance differences across college course characteristics.

College programs and majors

Figure 3 shows that one-way ANOVA results indicated statistically significant differences in performance across the five different programs, Welch’s F(4, 322.18) = 9.62, p < .001, ω² = .03. Post hoc analyses using Bonferroni revealed that students enrolled in associate degree programs performed statistically significantly higher (p ≤ .001) than students in all other program types with effect sizes ranging from 0.33 SDs (career/vocational) to 0.52 SDs (transfer program). There were no other significant differences across the program types.

Figure 3.

Critical thinking performance differences across college programs and majors.

For college major, results revealed that STEM majors statistically significantly outperformed their non-STEM peers by 0.32 SDs, Welch’s t(928.28) = −0.50, p < .001. In terms of credit hours (Figure 3), results showed that students with 30 or more credit hours performed statistically significantly lower than students with less than 30 credit hours, t(1,305) = −2.16, p = .03, d = 0.13; however, the effect size was small.

Relations With Student Perceptions

Results showed moderate statistically significant differences across the four categories of self-rated critical thinking skills, Welch’s F(3, 175.44) = 14.07, p < .001, ω² = .03, with those students rating themselves as having excellent skills performing highest, and those with fair skills performing lowest (Figure 4). Post hoc analyses using the Bonferroni correction to the p-values revealed that students who rated themselves as having excellent critical thinking skills performed 0.16 to 0.68 SDs higher than the other three categories (p < .01). Students who rated themselves as having good skills performed 0.30 to 0.54 SDs higher than students in the average and fair categories, respectively (p < .001). No statistically significant differences were found between those reporting average skills compared with fair skills.

Figure 4.

Critical thinking performance differences across student perceptions.

Results in terms of perceived difficulty (Figure 4) indicated that students who felt the test was too easy or on the right difficulty level performed significantly higher by 0.42 SDs than students who felt the test was too difficult, Welch’s t(359.9) = −6.15, p < .001. For perceived testing time (Figure 4), one-way ANOVA results revealed no statistically significant differences between the three testing time categories. In relation to actual testing time on the assessment, mean testing time on the assessment was just over 33 minutes (SD = 8.53), and there was a positive relationship between testing time and total critical thinking score (r = .24, p < .001). When looking at the response time across the three self-report categories on perceived testing time, students who felt as though they had more than enough time or enough time completed the assessment in 28 and 33 minutes on average, respectively. Those students who indicated that they did not have enough time still completed the assessment in 40 minutes on average.

Discussion

Given the importance of critical thinking for students enrolled in vocational programs or preparing for transfer into a 4-year institution (Fong et al., 2017), there is a need to evaluate and assess students’ critical thinking skills to determine whether students are demonstrating learning in this particular competency. This study aimed to add to the limited literature evaluating critical thinking skills for community college students. Ultimately, results of this study showed (a) consistent significant predictors associated with higher critical thinking performance; (b) a positive relationship between critical thinking performance and the frequency of using critical thinking in college courses; (c) significant, but relatively small performance differences across demographic and college experience subgroups; and (d) positive relationships between student perceptions and critical thinking performance.

Variables Related to Critical Thinking Performance

When evaluating which variables relate to critical thinking performance, results showed that age, race, self-rated critical thinking skills, credit hours, and having a STEM college major were consistently significant, regardless of other variables in the models. While demographic and college experience variables were helpful in predicting performance, by far the most significantly predictive variable was students’ self-assessment of their critical thinking skills, with those students who rated themselves as having good or excellent skills performing significantly higher than their peers who rated themselves as average or fair, controlling for other variables. This one variable contributed nearly 4% in explanatory power. Variables associated with college program explained an additional 1.6% of the variance in scores. An unexpected result was the negative coefficient for students with 30 or more completed credit hours in this model, suggesting that students with less college experience outperformed those with more. This could be an artifact of this population of students, where enrollment in college courses may look different than traditional 4-year students (see next section for a more in-depth discussion).

Results showed that altogether the full model explained only 10.6% of the variance in performance, which suggests that we are missing key variables that could help to explain community college student performance in critical thinking. Some key additional variables based on the demographics of this population (see AACC, 2019a; Zhang & Oymak, 2018) that we did not have access to for this study, but would be useful for future research, include first-generation to attend college, socioeconomic status, enrollment in developmental coursework, and working full-time or part-time. Including these additional variables will likely help to capture the diverse experiences of this population and help to explain performance differences.

How College Courses Relate to Critical Thinking

Results of this study showed a few different results in terms of the relationship between critical thinking performance and college courses. We first looked at the relationship between credit hours achieved and critical thinking performance, comparing students with fewer than 30 credits (roughly one full year’s worth of credits for full-time students) and 30 or more credits hours completed. Results showed that students with less than 30 credit hours actually performed higher than students with 30 or more credit hours; however, the effect of this difference was negligible. There are a few variables than may impact this result. College enrollment may look different for this population of students than for a traditional 4-year college experience. For instance, nearly 62% of the total number of students attending community colleges are enrolled part-time rather than full-time (AACC, 2019a). This means that students may only be taking one or two courses at a time or may have longer periods of time in between taking courses. It is possible that students with more than 30 credit hours may not be receiving exposure to critical thinking as part of college courses. In addition, some have asserted that the credit hour is an indicator of time rather than a representation of learning (Laitinen, 2012).

This argument from Laitinen (2012) could also be used to explain the result regarding the number of courses that students have taken related to critical thinking, scientific inquiry, or problem-solving. Results showed that the number of courses was not significantly related to student performance. This could be because even though students are enrolled in these particular courses, they are not necessarily learning or retaining those skills. What did seem to matter in relation to college courses was the frequency with which courses required students to use critical thinking skills. That is, students who indicated that they used critical thinking skills “frequently” in their courses performed 0.18 SDs higher than those who indicated “sometimes,” and 0.30 SDs higher than those who indicated “rarely.” These results suggest that even just one course could have an impact on critical thinking performance as long as critical thinking skills were utilized frequently throughout the course. Researchers have found that explicit critical thinking instruction is more beneficial than implicit instruction (Abrami et al., 2008; Marin & Halpern, 2011) and that specific interventions in critical thinking have been shown to increase these skills (Abrami et al., 2008). For instance, Abrami et al. (2008) found in their meta-analysis that mixed-methods interventions where critical thinking skills are taught as an independent track within a specific content course had the largest effect. Moderate effects were found when critical thinking was explicitly identified as part of the course objectives, again emphasizing the importance of direct instruction. Overall, the results of this study suggest that institutions should place more of a focus on the frequency with which students are using critical thinking throughout their courses, which could increase student performance in this particular area, especially if critical thinking is an explicit outcome within the course.

The type of program in which a student is enrolled in may also play a role in critical thinking performance. Results showed that students enrolled in associate programs performed higher than their peers. It could be that these types of programs have more of a general education curriculum that provides practice with critical thinking skills whereas certificate or vocational programs may focus more on applied skills needed for a job or career. Although such courses may also focus on critical thinking skills, those skills may be domain-specific. For instance, research has shown that nursing programs emphasize critical thinking skills, but that when measuring general critical thinking, nursing students do not demonstrate significant improvements over time (Huber & Kuncel, 2016). It could be that they are learning domain-specific critical thinking skills that are not measured through the more general learning outcome measures. Our results showed that students enrolled in STEM majors statistically significantly outperformed their non-STEM major peers, and this was a consistent statistically significant predictor in our regression models. As a result, the types of programs that students are enrolled in may also be related to higher critical thinking performance. If STEM-based programs have more of a focus on critical thinking within their courses, this could be contributing to the difference in performance with non-STEM majors. Future research is needed to determine whether STEM programs actually involve more critical thinking than non-STEM programs, or if there are other skills related to critical thinking that might confound this finding.

The Relationship Between Life Experience and Critical Thinking Skills

There have been questions as to whether changes in critical thinking scores during college are a result of age maturation or of actual learning in college. Butler (2012) found that the number of years of education, rather than age, predicted critical thinking scores and that these results suggested that critical thinking skills can in fact be improved through instruction. However, Huber and Kuncel (2016) noted that existing research evaluating students’ learning gain in critical thinking has been unable to distinguish the effects of college from maturation effects. Given that students enrolled in community colleges tend to be slightly older in age, it is important to evaluate the relationship between age and critical thinking performance for this population.

Although results of this study cannot directly distinguish maturation and instructional effects in terms of critical thinking skills, this study was able to evaluate the relationship between critical thinking skills and age. Results showed that age was a consistent predictor of critical thinking performance when controlling for other variables. Results of the ANOVA were significant, indicating that not all of the population means across the age categories were equal. However, when looking at the post hoc comparisons, none of the age categories were statistically significantly different from each other. Future research should consider additional variables that could be related to age such as number of years in school or number of years employed. These variables could provide additional information as to whether differences in performance across age are associated with other experiences related to critical thinking development or if they are related to maturation more generally.

Positive Student Perceptions

Student perceptions and self-evaluations have been shown to play a role in student learning (Lynam & Cachia, 2018). As a result, students’ perceptions regarding different aspects of an assessment could ultimately play a role in their assessment performance. For this study, we evaluated student perceptions about the overall test difficulty and testing time. Results showed that 82% of students indicated that the assessment was either too easy or at the right difficulty level. These perceptions regarding difficulty were reflected in student performance, with students indicating that the assessment was at the right difficulty performing 0.42 SDs higher than their peers who indicated that the assessment was too difficult. Approximately 80% of students indicated that they had enough or more than enough time to complete the assessment within the 45-minute testing period. Results showed no significant differences in test performance even compared with those students who indicated they did not have enough time, suggesting that testing time did not play a role in overall student performance. Lynam and Cachia (2018) noted that student emotions and high levels of stress can affect how they engage with assessments. The fact that the majority of students felt that the assessment was at the right difficulty level and that there was enough time to complete the assessment may have helped to reduce any possible negative emotions or stress around those aspects of the assessment.

In addition to asking students about their perceptions of the assessment, students were also asked to self-rate their critical thinking skills. Results showed that the majority of students rated themselves as good (43%) or average (46%) critical thinkers, which was to be expected. Students who rated their skills as excellent performed 0.68 SDs higher than those who self-rated as having fair critical thinking skills and 0.48 SDs higher than those rating themselves as average. Self-rated critical thinking skills were also a very strong predictor of higher performance in the regression model. These self-rated skills may relate to an individual’s self-efficacy or belief in their “capabilities to organize and execute courses of action required to produce given attainments” (Bandura, 1997, p. 3). Because students rated themselves as having good or excellent critical thinking skills, they may have been more efficacious and thus performed better on the assessment. Self-efficacy has been found to be related to academic performance (Multon et al., 1991), and Chemers et al. (2001) found evidence to support the role of self-efficacy in first-year college student success, suggesting that self-efficacy is important at the college level.

Limitations and Future Research

There are a few limitations to our study that will inform future research. The first is sample size within certain subgroups. For instance, although we were able to compare White and non-White students, it would have been more beneficial to disaggregate the non-White group into additional categories; however, due to the small sample sizes across various racial and ethnic groups, we were unable to analyze differences across subgroups. Another limitation is the small number of students within each institution, which prevented us from evaluating any potential impact of institutional variables using multilevel modeling. Future research should further evaluate how institutional variables may impact critical thinking performance for students enrolled in community college programs.

Another limitation is in relation to the question that asked students to rate their critical thinking skills. This question was asked in a posttest survey at the end of the assessment, and as a result, it is possible that these responses could have been affected by students’ experiences with, or their performance on, the test. That is, students could have defined critical thinking skills differently based on what they saw on the assessment. The same could be applied to questions about critical thinking courses and how students define what that means. Future research could consider asking students about their self-rated critical thinking skills or course-taking patterns related to critical thinking before taking the assessment.

Finally, future research would benefit from following a cohort of students longitudinally at community colleges. Most existing longitudinal studies have focused only on 4-year institutions and would benefit from looking at this particular population given that they make up almost half of all students enrolled in college in the United States.

Footnotes

Authors’ Note

K.B. is now a research associate at Research for Action.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Katrina C. Roohr

Notes

Author Biographies

Katrina C. Roohr, EdD, is a managing research scientist in the Academic to Career Research Center at the Educational Testing Service. Her primary research is in the area of college-level learning outcomes including critical thinking, quantitative literacy, civic competency and engagement, and oral communication. She has also conducted research around student learning gains in college, and learning outcomes at community colleges.

Kri Burkander, PhD, formerly served as a research associate at the Educational Testing Service, where she supported research on college learning outcomes. She is currently a Research Associate at Research for Action, a non-profit research and evaluation organization based in Philadelphia, PA. Her primary research focuses on state and local college Promise programs and community college initiatives aimed at improving credential completion among traditionally underserved students.

References

Abrami

P. C.

Bernard

R. M.

Borokhovski

Wade

Surkes

M. A.

Tamim

Zhang

(2008). Instructional interventions affecting critical thinking skills and dispositions: A stage 1 meta-analysis. Review of Educational Research, 78(4), 1102–1134. https://doi.org/10.3102/0034654308326084

American Association of Community Colleges. (2019a, March). AACC fast facts 2019. https://www.aacc.nche.edu/wp-content/uploads/2019/05/AACC2019FactSheet_rev.pdf

American Association of Community Colleges. (2019b). Driving success. VFA summary report: Leading indicators of success and student outcomes for community colleges. https://www.aacc.nche.edu/wp-content/uploads/2019/04/VFA_Summary_Report_2019.pdf

American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for educational and psychological testing.

American Management Association. (2012). AMA 2012 critical skills survey. https://www.amanet.org/assets/1/6/2012-critical-skills-survey.pdf

Ayers

D. F.

(2015). Credentialing structure, pedagogies, practices, and curriculum goals: Trajectories of change in community college mission statements. Community College Review, 43(2), 191–214. https://doi.org/10.1177/0091552115569847

Bailey

T. R.

Jaggars

S. S.

Jenkins

(2015). Redesigning America’s community colleges. Harvard University Press. https://doi.org/10.4159/9780674425934

Bandura

(1997). Self-efficacy: The exercise of control. W.H. Freeman.

Baumert

Demmrich

(2001). Test motivation in the assessment of student skills: The effects of incentives on motivation and performance. European Journal of Psychology of Education, 16(3), 441–462. https://doi.org/10.1007/bf03173192

10.

Binkley

Erstad

Herman

Raizen

Ripley

Rumble

(2012). Defining 21st century skills. In Griffin

McGaw

Care

(Eds.), Assessment and teaching of 21st century skills (pp. 17–66). Springer. https://doi.org/10.1007/978-94-007-2324-5

11.

Butler

H. A.

(2012). Halpern Critical Thinking Assessment predicts real-world outcomes of critical thinking. Applied Cognitive Psychology, 26(5), 721–729. https://doi.org/10.1002/acp.2851

12.

Casner-Lotto

Barrington

(2006). Are they really ready to work? Employers’ perspectives on the basic knowledge and applied skills of new entrants to the 21st century U.S. workforce. Partnership for 21st Century Skills. https://files.eric.ed.gov/fulltext/ED519465.pdf

13.

Chemers

M. M.

Garcia

B. F.

(2001). Academic self-efficacy and first-year college student performance and adjustment. Journal of Educational Psychology, 93(1), 55–64. https://doi.org/10.1037/0022-0663.93.1.55

14.

Cohen

A. M.

Brawer

F. B.

(2008). The American community college (5th ed.). John Wiley.

15.

Cohen

A. M.

Brawer

F. B.

Kisker

C. B.

(2014). The American community college (6th ed.). John Wiley.

16.

Cohen

(1988). Statistical power analysis for the behavioral sciences. Lawrence Erlbaum. https://doi.org/10.4324/9780203771587

17.

The College Board. (2019). Community college: FAQs. https://bigfuture.collegeboard.org/find-colleges/how-to-find-your-college-fit/community-college-faqs

18.

Community College Review. (2019, March 26). What is a community college? [Online forum post]. https://www.communitycollegereview.com/blog/what-is-a-community-college

19.

Council for Aid to Education. (2019). About CLA+. https://cae.org/flagship-assessments-cla-cwra/cla/about-cla/

20.

de Brey

Musu

McFarland

Wilkinson-Flicker

Diliberti

Zhang

Branstetter

Wang

. (2019, February). Status and trends in the education of racial and ethnic groups 2018 (NCES 2019-038). National Center for Education Statistics, U.S. Department of Education. https://nces.ed.gov/pubs2019/2019038.pdf

21.

Desai

S. A.

(2012). Is comprehensiveness taking its toll on community colleges? An in-depth analysis of community colleges’ missions and their effectiveness. Community College Journal of Research and Practice, 36(2), 111–121. https://doi.org/10.1080/10668920802611211

22.

Dougherty

K. J.

Townsend

B. K.

(2006). Community college missions: A theoretical and historical perspective. New Directions for Community Colleges, 136, 5–13. https://doi.org/10.1002/cc.254

23.

Educational Testing Service. (2018a). About the HEIghten® outcomes assessment suite. https://www.ets.org/heighten/about

24.

Educational Testing Service. (2018b). HEIghten® outcomes assessment suite: Guide to score interpretation.

25.

Educational Testing Service. (2019). HEIghten® critical thinking assessment. https://www.ets.org/heighten/about/critical_thinking/

26.

Facione

N. C.

Facione

P. A.

Sanchez

C. A.

(1994). Critical thinking disposition as a measure of competent clinical judgment: The development of the California Critical Thinking Disposition Inventory. Journal of Nursing Education, 33(8), 345–350. https://doi.org/10.3928/0148-4834-19941001-05

27.

Facione

P. A.

(1990). The California Critical Thinking Skills Test-college level. Technical report #2. Factors predictive of skills. California Academic Press. https://files.eric.ed.gov/fulltext/ED327550.pdf

28.

Facione

P. A.

Facione

N. C.

(1992). The California Critical Thinking Dispositions Inventory. California Academic Press.

29.

Finn

(2015). Measuring motivation in low-stakes assessments (Report No. RR-15-19). Educational Testing Service. https://files.eric.ed.gov/fulltext/EJ1109280.pdf

30.

Fong

C. J.

Kim

Davis

C. W.

Hoang

Kim

Y. W.

(2017). A meta-analysis on critical thinking and community college student achievement. Thinking Skills and Creativity, 26, 71–83. https://doi.org/10.1016/j.tsc.2017.06.002

31.

Gall

M. D.

Borg

W. R.

Gall

J. P.

(2007). Educational research: An introduction (8th ed.). Pearson Education.

32.

Halpern

D. F.

(2003). Thought and knowledge: An introduction to critical thinking (4th ed.). Lawrence Erlbaum. https://doi.org/10.4324/9781410606433

33.

Halpern

D. F.

(2010). Halpern Critical Thinking Assessment manual. Schuhfried GmbH. https://drive.google.com/file/d/0BzUoP_pmwy1gdEpCR05PeW9qUzA/view

34.

Hart Research Associates. (2015). Falling short? College learning and career success. Association of American Colleges and Universities. https://www.aacu.org/sites/default/files/files/LEAP/2015employerstudentsurvey.pdf

35.

Hart Research Associates. (2016). Recent trends in general education design, learning outcomes, and teaching approaches. Association of American Colleges and Universities. https://www.aacu.org/sites/default/files/files/LEAP/2015_Survey_Report2_GEtrends.pdf

36.

Hauser

Kingsbury

G. G.

(2009, April 14–16). Individual score validity in a modest-stakes adaptive educational testing setting [Paper presentation]. Annual Meeting of the National Council on Measurement in Education, San Diego, CA, United States.

37.

Huber

C. R.

Kuncel

N. R.

(2016). Does college teach critical thinking? A meta-analysis. Review of Educational Research, 86(2), 431–468. https://doi.org/10.3102/0034654315605917

38.

Kena

Musu-Gillete

Robinson

Wang

Rathbun

Zhang

. . . Ballard

(2015, May). The condition of education 2015 (NCES 2015-144). National Center for Education Statistics, U.S. Department of Education. https://nces.ed.gov/pubs2015/2015144.pdf

39.

Laitinen

(2012, September). Cracking the credit hours. New America Foundation. https://files.eric.ed.gov/fulltext/ED.pdf540304

40.

Liu

O. L.

Frankel

Roohr

K. C.

(2014). Assessing critical thinking in higher education: Current state and directions for next generation assessment (Report No. RR-14-10). Educational Testing Service. https://files.eric.ed.gov/fulltext/EJ1109287.pdf

41.

Liu

O. L.

Mao

Frankel

(2016). Assessing critical thinking in higher education: The HEIghten™ approach and preliminary validity evidence. Assessment & Evaluation in Higher Education, 41(5), 677–694. https://doi.org/10.4324/9781351260480-3

42.

Liu

O. L.

Roohr

K. C.

(2013). Investigating 10-year trends of learning outcomes at community colleges (Report No. RR-13-34). Educational Testing Service. https://www.ets.org/Media/Research/pdf/RR-13-34.pdf

43.

Liu

O. L.

Shaw

. . . Loyalka

(2018). Assessing college critical thinking: Preliminary results from the Chinese HEIghten® Critical Thinking Assessment. Higher Education Research & Development, 37(5), 999–1014. https://doi.org/10.1080/07294360.2018.1467381

44.

Lynam

Cachia

(2018). Students’ perceptions of the role of assessments at higher education. Assessment & Evaluation in Higher Education, 43(2), 223–234. https://doi.org/10.1080/02602938.2017.1329928

45.

Marin

Halpern

D. F.

(2011). Pedagogy for developing critical thinking in adolescents: Explicit instruction produces greatest gains. Thinking Skills and Creativity, 6(1), 1–13. https://doi.org/10.1016/j.tsc.2010.08.002

46.

Multon

K. D.

Brown

S. D.

Lent

R. W.

(1991). Relation of self-efficacy beliefs to academic outcomes: A meta-analytic investigation. Journal of Counseling Psychology, 38(1), 30–38. https://doi.org/10.1037//0022-0167.38.1.30

47.

National Association of Colleges and Employers. (2018, November). Job outlook 2019. https://www.odu.edu/content/dam/odu/offices/cmc/docs/nace/2019-nace-job-outlook-survey.pdf

48.

National Institute for Learning Outcomes Assessment. (2011). Transparency framework. https://www.learningoutcomesassessment.org/ourwork/transparency-framework/

49.

Nunley

Bers

Manning

(2011, July). Learning outcomes assessment in community colleges (Occasional Paper No. 10). National Institute for Learning Outcomes Assessment. https://www.learningoutcomesassessment.org/documents/CommunityCollege.pdf

50.

Pascarella

Bohr

Nora

Terenzini

(1994). Cognitive effects of two-year and four-year colleges: Some new evidence. Office of Educational Research and Improvement. https://files.eric.ed.gov/fulltext/ED371678.pdf

51.

Quality Assurance Agency. (2008, August). The framework for higher education qualifications in England, Wales and Northern Ireland. https://www.ncl.ac.uk/ltds/assets/documents/qsh-progapp-fheqsummary.pdf

52.

Research for Action. (2019, February). Measuring the effectiveness of outcomes-based funding on certificate production: Challenges, inconsistencies and recommendations for future research [Research brief]. https://files.eric.ed.gov/fulltext/ED593699.pdf

53.

Rhodes

T. L.

(Ed.). (2010). Assessing outcomes and improving achievement: Tips and tools for using rubrics. Association of American Colleges and Universities.

54.

Rios

J. A.

Liu

O. L.

Bridgeman

(2014). Identifying unmotivated examinees on student learning outcomes assessment: A comparison of two approaches. New Directions for Institutional Research, 161, 69–82. https://doi.org/10.1002/ir.20068

55.

Sykes

(2012). Defining and reporting subbaccalaureate certificates in IPEDS (NPEC 2012-835). National Postsecondary Education Cooperative, U.S. Department of Education. https://nces.ed.gov/pubs2012/2012835.pdf

56.

Voluntary Framework of Accountability. (2013). Community college measures of effectiveness. American Association of Community Colleges. https://vfa.aacc.nche.edu/Documents/VFABrochureLowResolution.pdf

57.

Wise

S. L.

Kong

(2005). Response time effort: A new measure of examinee motivation in computer-based tests. Applied Measurement in Education, 18(2), 163–183. https://doi.org/10.1207/s15324818ame1802_2

58.

Zhang

Oymak

(2018). Participants in subbaccalaureate occupational education: 2012 (NCES 2018-149). National Center for Education Statistics. https://nces.ed.gov/pubs2018/2018149.pdf