California Critical Thinking Disposition Inventory: Reliability Generalization Meta-Analysis

Abstract

The aims of this reliability generalization study were to provide the overall alpha values of the California critical thinking disposition inventory (CCTDI) total score and subscales scores and investigate the characteristics of the studies that may be associated with the variability in the reliability values of the CCTDI total score and subscales scores. This study was carried out with 98 alpha values from 87 unique studies for total CCTDI scores. In the random effects model, total CCTDI was found to be reliable across samples with alpha value of 0.83. Also, the general alpha coefficients were 0.65, 0.56, 0.64, 0.66, 0.74, 0.72, and 0.61 for TS-scale, OM-scale, A-scale, S-scale, SC-scale, I-scale, and M-scale, respectively. Examination of study characteristics indicated that sample type was a significant predictor for alpha value of total CCTDI and all subscales. Samples with university students reported larger Cronbach’s alpha estimates for total CCTDI and all subscales. Also, while language was found to moderate the general alpha coefficient of total CCTDI, OM-scale, A-scale, S-scale, and I-scale, it was not a significant moderator on the general alpha value of TS-scale, SC-scale, and M-scale. Total CCTDI and all subscales showed higher Cronbach’s alpha values for the English-language administrations. Besides, country of the study was a significant moderator on the general alpha coefficient of total CCTDI, S-scale, and I-scale. However, subgroup of discipline was not a significant moderator on the general alpha coefficient of total CCTDI and its subscales. The mean of the test scores significantly explained %5 of the variance of alpha values of the total CCTDI. SD of the test scores significantly explained %10, %55, and %54 of the variance of alpha values of the total CCTDI, A-scale, and S-scale, respectively. It was found that gender and ethnicity significantly moderated the alpha values for M-scale.

Keywords

critical thinking disposition California critical thinking disposition inventory meta-analysis reliability generalization Cronbach’s alpha

Introduction

Critical thinking (CT)

Ennis (1991) defines CT as a functional, reflective, and reasonable way of thinking that is employed by individuals while deciding what to do or what to believe. According to Paul (1990), individuals evaluate the source of knowledge, test the validity of the acquired information, analyze its reliability and draw appropriate inferences for specific situations through CT which can be seen as a logical and rational way of dealing with ideas (Ruggerio, 1990). CT works as a defense mechanism for individuals against today’s world in which information is easily accessible (Epstein & Kernberger, 2012) because individuals acquire the true, useful, and accurate information through CT. After they question, examine, and evaluate such information, they can decide to believe it or not. That evaluation includes assessing the sensibility, truth, and accuracy of the given information, claims, evidence, and judgments (Lewis & Smith, 1993). After a Delphi project sponsored by the American Philosophical Association (APA) in 1990, CT was defined as “purposeful, self-regulatory judgment which results in interpretation, analysis, evaluation, and inference, as well as explanation of the evidential, conceptual, methodological, criteriological, or contextual considerations upon which that judgment is based” (Facione, 1990, p.2). There is consensus in the literature that CT is a complex thinking process that includes higher-order reasoning processes and different cognitive skills to achieve the desired outcome (Halpern, 2003; Sternberg, 1986). Some of these cognitive skills are identifying assumptions, evaluating evidence, deducing conclusions (Pascarella & Terenzini, 1991), analyzing and evaluation of arguments, claims or evidence, and making inferences using inductive or deductive reasoning (Facione et al., 2000; Lai, 2011).

CT is not only about skills or abilities but also includes some dispositions toward using those skills (Ennis, 1987; Paul, 1990). CT involves both abilities and dispositions such as seeking the truth, being systematic, and being open-minded (Ennis, 1987). CT dispositions are as important as CT skills because possessing the CT skills alone is not sufficient to use those skills in everyday life (Paul & Elder, 2001) and both skills and dispositions make an individual a good critical thinker (Profetto-McGrath, 2003). Therefore, a good critical thinker should have both the skills and the dispositions toward using of these skills. Without having a strong disposition to use CT skills, they may never be used; conversely, if the individuals do not have high CT skills, strong dispositions might be useless (Profetto-McGrath et al., 2003).

California Critical Thinking Disposition Inventory (CCTDI)

The CCTDI was developed by Facione and Facione (1992) after a comprehensive Delphi study by APA. The CCTDI which derives its conceptualization from the APA Delphi Report is the first instrument devoted to the dispositional aspect of CT and designed to measure individuals’ CT dispositions. In other words, it is not an inventory to measure how well individuals use CT skills, but how much individuals value the inclination to use CT skills. The CCTDI which uses a 6-point Likert scale (from strongly agree to strongly disagree) has 75 items and 7 subscales which are truth-seeking (TS-scale), open-mindedness (OM-scale), analyticity (A-scale), systematicity (S-scale), CT self-confidence (SC-scale), inquisitiveness (I-scale), and maturity (M-scale). The items are interspersed throughout the CCTDI and it takes approximately 20 minutes to be completed. A score of 30 and below and scores above 50 on any of these subscales indicate weakness and strength in relation to the given attribute or characteristic, relatively. The inventory yields an overall score (maximum 420 and minimum 70) and scores above 350 indicate a strong disposition, scores between 280 and 350 reflect a positive inclination, and scores under 280 are described as deficient in CT disposition. The CCTDI is discipline neutral and can be used within any disciplines such as liberal arts and sciences (Facione et al., 1994).

The TS-scale is about to measure the disposition of being willing to find the truest knowledge in a given context, courageous about asking further questions, and objective about continuing inquiry even if the findings are not beneficial. The OM-scale targets the disposition of showing tolerance to divergent views and being sensitive and aware of the possibility of one’s own bias. The A-scale addresses treasuring the use of reasoning and evidence to solve problems. Individuals with a positive inclination toward the A-scale anticipate potential conceptual or practical difficulties and are always beware of the need to intervene. While the S-scale targets to measure being organized, orderly, and persevering in inquiry, the SC-scale measures the trust of someone’s own reasoning processes. Individuals with a positive inclination toward SC-scale trust the excellence of one’s own reasoned judgments and can lead other people in rationally solving problems. The I-scale targets to measure individuals’ intellectual curiosity and their enthusiasm for learning even if the knowledge is not readily apparent. The M-scale is about being judicious in one’s decision-making process. CT-mature individuals are aware that some problems may be ill-structured and some situations can have more than one reasonable option. Also, they make judgments according to some standards, contexts, and evidence.

In 1992, a pilot study was conducted with 156 undergraduates, high school students and post baccalaureates, and alpha coefficient for the overall instrument was found to be 0.91 (Facione & Facione, 1992). Also, the alpha coefficients of subscales were: TS-scale (12 items) 0.71, S-scale (11 items) 0.74, I-scale (10 items) 0.80, OM-scale (12 items) 0.73, SC-scale (9 items) 0.78, M-scale (10 items) 0.75, and A-scale (11 items) 0.72. In a subsequent study by Facione et al. (1994) with a sample of 1.019 freshmen students, alpha coefficients remained relatively consistent with the first pilot study (for subscales 0.60–0.78 and for total inventory 0.90). After a factor analysis conducted by the authors (Facione et al., 1994), the CCTDI indicated seven factors.

Walsh and Hardy (1997) reexamined the inventory with a sample group of 499 undergraduates across different disciplines such as history, nursing, and education. Alpha coefficients for the subscales ranged from 0.57 to 0.78 with an overall alpha of 0.91. After the factor analytic procedure, total items of the inventory were reduced to 52 from the original 75 under four factors. These four factors with total variance of %28 renamed by the authors as perspicacity/confidence (19 items, variance accounted for %15), receptivity/open-mindedness (17 items, variance accounted for %5.6), systematicity (8 items, variance accounted for %4.47), and objectivity/maturity (8 items, variance accounted for %2.97). Bondy et al. (2001) investigated the psychometric properties of the CCTDI using two samples of undergraduate students from different academic majors. While the overall alpha values for the CCTDI for Sample 1 (320 students) and Sample 2 (156 students) were 0.89 and 0.90, respectively, alpha values for subscales ranged from 0.39 to 0.77 for Sample 1 and from 0.49 to 0.79 for Sample 2. The authors conducted a confirmatory factor analysis with the combined sample (N = 476) and found out that emerged factor loadings were not consistent with the previously reported by Facione et al. (1994). According to Bondy et al. (2001), while %33 of the items had loadings of less than 0.30, %12 of them had loadings of 0.35 or less. Also, the total explained variance was reported as %27.7.

The CCTDI was reexamined by Kakai (2003) with a sample group of 536 university students. Alpha values for the subscales ranged from 0.59 to 0.77 with an overall alpha of 0.89. After principal components factor analysis conducted by the author to examine a streamlined 4-factor model, the total items of the inventory reduced to 52 loaded on seven factors four of which closely similar to the factors that emerged in Walsh and Hardy’s (1997) study. Also, Kakai (2003) performed another factor analysis on the 40 items from the four factors to have a more efficient short version. These four factors which explained %33 of the total variance were renamed as intellectual diligence, open-mindedness, nonrelativism, and analyticity. Walsh et al. (2007) reexamined the stability of the factor structure of the CCTDI using 800 undergraduate students. Alpha coefficients for the subscales ranged from 0.53 to 0.84 with an overall alpha of 0.91. After principal components factor analysis, total items of the inventory were reduced to 25 under four factors. These four factors with total variance of %44.95 renamed by the authors as intellectual prowess (13 items, variance explained %24.25), objectivity (6 items, variance explained %9.7%), systematicity (3 items, variance explained %5.6), and receptivity (3 items, variance explained %5.4).

Reliability and Reliability Generalization (RG)

While evaluating the test scores in many fields such as psychology and education, reliability can be seen as one of the most important key concepts. Reliability refers to the stability of test or any measurement scores over repeated administrations (Traub & Rowley, 1991). Contrary to well-known belief, it is a dynamic property of test scores and about the data rather than a fixed entity of the measurement tool itself (Thompson & Vacha-Haase, 2000). Therefore, reliability should be calculated after each measurement administration and should be reported in each study because it is influenced by sample characteristics such as gender, age, or language (Thompson, 2002) and varies across different administrations. Internal consistency, one of the most used types of reliability, captures the extent to which items in a measure assess the same unidimensional construct (Semma et al., 2019). There are many methods to assess internal consistency reliability such as Cronbach’s alpha, split-half reliability, and Kuder–Richardson tests. Among these methods, Cronbach’s alpha is the most common estimate of reliability (McNeish, 2018; Dimitrov, 2002) and can be calculated as

α = \frac{n}{n - 1} [1 - \frac{\sum_{i = 1}^{n} V_{i}}{V_{t o t a l}}]

In this formula, n is the number of items in the measure, V_i is the variance of a specific item’s score, and V_total is the variance of all scores (Cronbach, 1951).

Reliability estimates can vary across sample and study characteristics (Dawis, 1987) and low reliability estimates mean high levels of measurement error and therefore weaken the power of statistical tests (Yetkiner & Thompson, 2010). Therefore, it can be said that reliability estimates are important while interpreting the research findings. Considering the significance of reliability of test scores, Vacha-Haase (1998) proposed a meta-analytic method called RG. RG is a meta-analytic technique that assesses the overall reliability of a measure (Vacha-Haase, 1998). Thanks to RG, we can combine different reliability coefficients across different measurement administrations and reach overall reliability of a measure. Also, we can see the heterogeneity of reliability estimates for a given measure across different studies and possible sources of heterogeneity in the reliability estimates (Vacha-Haase, 1998). By identifying these possible sources of heterogeneity, we can see what really influences the reliability estimates of a given measure. In short, RG provides us a general evaluation of a measurement’s internal consistency and identifies the factors that involve in the variability of the reliability of a measurement.

The Present Study

According to Henson and Thompson (2002), RG should be used only when the measurement was administered widely and if there is a reasonable number of studies that estimated and reported the reliability of the scores. In this study, it was aimed to calculate the overall alpha value of the CCTDI which has been widely used to measure CT dispositions since its development. The inventory has been translated to many languages such as Turkish, Chinese, Italian, and in widespread use in many countries and universities in a variety of academic disciplines (Walsh et al., 2007). Although there are studies investigating the factor structure of the CCTDI (Walsh & Hardy, 1997; Kakai 2003; Bondy et al., 2001; Walsh et al., 2007), there are not any meta-studies that focused on the reliability of the CCTDI. Therefore, the first aim of the present study was to examine the overall alpha values of the CCTDI total score and subscale scores. The second aim was to investigate the variability of the overall Cronbach’s alpha estimates of the CCTDI total score and subscale scores, and then to explore study characteristics that might explain this variability. While sample type, language, discipline, and country of the study were considered as potential categorical moderators of the alpha estimates of the CCTDI, mean age of the sample, gender (%female), ethnicity (%Caucasian), mean, and standard deviation (SD) of the test scores were determined as potential continuous moderator variables. It was expected that alpha values for university students, English CCTDI, and administrations in the USA would be higher than younger students, non-English versions, and administrations in other countries because the development of the CCTDI was carried out in the USA and was based on an English-speaking sample consisting of mostly university students. Therefore, I thought that item wording could be more appropriate for English-speaking samples and university students. Also, it was expected that the CCTDI would have similar alpha values across different disciplines because the CCTDI is discipline neutral and items include no technical vocabulary or CT jargon (Facione et al., 1994). Besides, as the CCTDI was developed in the USA which has ethnically diverse population and with sample including male and female students, it was hypothesized that mean age, gender (%female), and ethnicity (%Caucasian) would not significantly moderate the alpha values of total CCTDI and its seven subscales. Mean and SD of the test scores were also included as continuous moderator variables because the mean and SD of the test scores are often examined as potential moderators of alpha values in RG studies (e.g., Rubio-Aparicio, Badenes-Ribera, Sánchez-Meca, Fabris & Longobardi, 2020; Deng et al., 2019). It was expected that higher levels of mean and SD of the test scores would significantly predict larger alpha values of total CCTDI and seven subscales.

Method

In this RG study, REGEMA guidelines proposed by Sánchez-Meca et al. (2021) and Henson and Thompson’s (2002) recommendations for RG studies were followed.

Collection of Studies and Inclusion and Exclusion Criteria

Several online databases including Google Scholar, Web of Science, ERIC, and Scopus were searched systematically to reach the studies administered the CCTDI. During the literature review, “California Critical Thinking Disposition Inventory” OR “California Critical Thinking Disposition” OR “CCTDI” search pattern was used in English. The literature review resulted in 3512 studies in total by the last search on the 23rd of April 2021. No limits for the date of publication were established. I also checked the citations of the CCTDI test manuals on Google Scholar to reach possible studies. There were 1069 studies for “The California Critical Thinking Disposition Inventory and the CCTDI test manual” by Facione and Facione (1992) and 833 studies for “Critical thinking disposition as a measure of competent clinical judgment: The development of the California Critical Thinking Disposition Inventory” by Facione et al. (1994). In short, after literature review, 5414 studies were found. Studies had to meet three criteria to be eligible for inclusion in the meta-analysis. First, original CCTDI with 75 items must be administered in the studies. Second, studies must report precise Cronbach’s alpha coefficients for total CCTDI or any of its subscales and sample size. Third, studies must be in English or Turkish. Studies were excluded if they did not administer the original CCTDI which has 75 items and only studies that reported Cronbach’s alpha for total CCTDI or for at least one of its subscales were included. I decided to use only Cronbach’s alpha as internal reliability because it is the most commonly reported measure of reliability in the studies. The studies were examined first through their titles and method sections and 4923 studies were excluded because of several reasons (e.g., duplicates, the version of the CCTDI). Then, 491 studies that administered the CCTDI or at least one of its subscales were screened and reviewed by two researchers in terms of inclusion criteria abovementioned and 397 studies were excluded. 115 of these studies (%23.42) did not report any alpha values, 248 of them (%50.51) induced reliabilities from previous research and 34 of them (%6.92) reported reliability as a range of coefficients. Finally, 94 unique studies (%19.14) were included in the analysis. Since more than one Cronbach’s alpha value were reported for multiple samples in some of the studies, those samples were coded separately and considered as independent measures of the CCTDI. However, if two or more alpha values were provided for the same sample (e.g., pre- and post-test or longitudinal studies), only the pre-test or first measurement values were included. Therefore, 98 alpha values (from 87 unique studies) for total CCTDI were included in the RG meta-analysis. This number is 46 for TS-scale (from 37 unique studies), 50 for OM-scale (from 41 unique studies), 47 for A-scale and S-scale (from 38 unique studies), 51 for SC-scale (from 42 unique studies), 48 for I-scale and M-scale (from 39 unique studies). All studies included in the meta-analysis and descriptive results of them can be seen in online supplemental material. Flow diagram for literature review and evaluation of studies is shown in Figure 1.

Figure 1.

Flow diagram for literature review and evaluation of studies.

Also, it was tried to uncover any other potential studies that may fulfill the selection criteria by examining the reference lists of the collected studies. However, no study has been found to include in the meta-analysis. Also, since some of the studies were not fully accessible, an e-mail was sent to the authors of these studies. However, the authors did not respond to e-mails. As a result, total sample number of the studies included in meta-analysis is 22201 for total CCTDI, 12984 for TS-scale, 13738 for OM-scale, 13183 for A-scale, 13173 for S-scale, 13777 for SC-scale, 13377 for I-scale, and 13256 for M-scale.

Coding of Studies

Based on RG coding recommendations by Henson and Thompson (2002), clear and detailed rules should be determined to code the studies. Therefore, firstly, a detailed coding manual was developed by the author and the studies were coded by both the author of this study and a second researcher who works in the field of Educational Sciences with meta-analysis experience. Inter-coder reliability was found to be great with a mean kappa coefficient of 0.95 (SD = 0.056), ranging from 0.88 to 1 for categorical moderator variables and with a mean intraclass correlation of 0.97 (SD = 0.025), ranging from 0.95 to 1 for continuous moderator variables. All discrepancies between two coders were resolved before statistical analyses. Name of the study, year published, author(s), country of the study, sample characteristics (mean age, gender, ethnicity, etc.), mean and SD of test scores, language of the CCTDI, sample size, and alpha values were extracted from the studies.

Data Analysis and Interpretation

Comprehensive Meta-Analysis (CMA) package program was used for data analysis. Since using raw Cronbach’s alpha estimates in meta-analysis violates the normality assumption (Rodriguez & Maeda, 2006), they should be transformed. In RG literature, there are three transformation methods which are Hakstian–Whalen (Hakstian & Whalen, 1976), Fisher’s R to Z, and Bonett’s transformation (Bonett, 2002). In this study, Bonett’s transformation was used to normalize sampling distributions and to stabilize variances. In Bonett’s transformation, reliability estimates are normalized via

L_{i} = \ln (1 - | {\hat{α}}_{i} |)

where L_i is the transformed coefficient and α_i is the alpha coefficient. The back-transformation of the alpha value is

\hat{α} {}_{i}= 1 - e^{L_{i}}

In this study, the confidence interval in all calculations was determined as %95. Reporting bias was checked with funnel plot, Duval and Tweedie’s trim-and-fill method, and Egger’s regression intercept. In meta-analysis studies, the heterogeneity can be investigated by some widely used statistical tests such as Q statistics and I² value (Hedges & Olkin, 1985; Petticrew & Roberts, 2006). Q value which is greater than the critical limit in X² table at k-1 degrees of freedom value (k is the number of effect sizes) indicates heterogeneity among studies. With low number of studies, Q statistic is likely to fall weak in identifying the heterogeneity (Huedo-Medina, Sanchez-Meca, Marin-Martinez & Botella, 2006). Therefore, I² value which is not influenced by the number of studies should also be examined with Q statistic (Petticrew & Roberts, 2006). I² can take values from %0 to %100 and value ranges of %25, %50, and %75 are considered as low, medium, and high heterogeneity, respectively (Cooper, 2017). In this study, Q statistics and I² value were used together to check the heterogeneity.

Moderator analyses for categorical variables were conducted according to subgroups of sample type (nurse, university student, high school student, etc.), language of the CCTDI (English or Non-English), discipline (medical and non-medical disciplines), and country of the study (the USA and other countries). Moderator analyses for categorical variables were performed with Analog ANOVA and mixed-effects model was applied to investigate the influence of categorical moderator variables. With Analog ANOVA, different Q-statistic values like between-group (Q_B), within-group (Q_W), and total (Q_TOTAL) can be calculated. Whether the categorical variable is a real moderator can be decided with Q_B statistic which indicates the between-group homogeneity (Lipsey & Wilson, 2001). Q_B value which is greater than the X² table critical value and significant Q_B statistic show that overall alpha value varies between categories of the moderator variable. Meta-regression analyses assumed mixed-effects model were conducted to investigate the possible effect of continuous moderators (mean age, gender, ethnicity, and mean and SD of the test scores) on the alpha values of total CCTDI and its subscales.

Results

Results on Reporting Bias

The funnel plots for total CCTDI and seven subscales shown in Figure 2 were examined using the trim-and-fill method by Duval and Tweedie (2000).

Figure 2.

Funnel plots for total CCTDI and its subscales.

The studies need to be distributed symmetrically around the general effect size in the funnel plot to say there is no reporting bias. As shown in Figure 2, it can be said that all funnel plots appear to be symmetric. Besides, the funnel plots were examined using the trim-and-fill method by Duval and Tweedie (2000) and the results showed that there is no need to add any imaginary studies for total CCTDI and for seven subscales to eliminate reporting bias totally. Considering the fact that the review of funnel plot can be subjective, Egger’s test was also examined. Egger’s intercept is 0.967 (%95 CI = −0.960–2.895), p > 0.05 for total CCTDI, 0.537 (%95 CI = −2.547–3.621), p > 0.05 for TS-scale, −0.111 (%95 CI = −3.115–3.339), p > 0.05 for OM-scale, −0.327 (%95 CI = −4.319–3.664), p > 0.05 for A-scale, 0.388 (%95 CI = −2.670−3.446), p > 0.05 for S-scale, 2.420 (%95 CI = 0.006–4.834), p > 0.05 for SC-scale, 1.479 (%95 CI = −1.108–4.067), p > 0.05 for I-scale, and 1.606 (%95 CI = −1.263–4.476), p > 0.05 for M-scale. Overall, it can be said that there is no reporting bias in the results for total CCTDI and seven subscales in this RG study.

Results on the General Alpha Coefficient of the CCTDI and Subscales

The general alpha coefficients and the results of the heterogeneity test in random effects model can be seen in Table 1.

Table 1.

Mean alpha coefficients and heterogeneity test results in random effects model for total CCTDI and subscales.

	k	Α	SE	Z	%95 CI	%95 PI	df	Q	I²
CCTDI	98	0.83	0.042	−43.105*	[0.82, 0.85]	[0.65, 0.92]	97	1732.59	94.40
TS	46	0.65	0.057	−18.846*	[0.61, 0.69]	[0.34, 0.82]	45	800.68	94.38
OM	50	0.56	0.059	−14.093*	[0.51, 0.61]	[0.10, 0.78]	49	1007.15	95.13
A	47	0.64	0.073	−14.331*	[0.59, 0.69]	[0.19, 0.84]	46	1378.78	96.66
S	47	0.66	0.057	−19.458*	[0.62, 0.70]	[0.37, 0.82]	46	811.90	94.33
SC	51	0.74	0.049	−28.093*	[0.72, 0.77]	[0.54, 0.85]	50	664.97	92.48
I	48	0.72	0.049	−26.269*	[0.69, 0.74]	[0.50, 0.84]	47	613.92	92.34
M	48	0.61	0.055	−17.433*	[0.57, 0.65]	[0.25, 0.80]	47	776.79	93,95

Note. k = number of reliability coefficients; SE = standard error; %95 CI = upper and lower bounds of the %95 confidence interval around the overall reliability estimate; %95 PI = prediction intervals; df = degrees of freedom; Q = heterogeneity statistic representing total variance; I² = heterogeneity index; *p < 0,05.

As seen in Table 1, Q_{(df = 97)} value was found to be 1732.59 (p < 0.05) for total CCTDI. According to the X² table, the calculated Q value is greater than the limit with 97 degrees of freedom and in 0.05 confidence interval (df = 97, X²_(0.05) = 120.990). It is therefore possible to say that there is heterogeneity among the studies. Also, the calculated I² value (%94.40) indicates a high level of heterogeneity. Therefore, random effects model was used while calculating the general alpha coefficient for total CCTDI in this study. Indeed, random effects model should be preferred for meta-analysis conducted in social sciences; because it is very difficult to ensure homogeneity among the studies in social sciences (Schmidt & Hunter, 2015; Borenstein et al., 2009). The general alpha coefficient of total CCTDI was calculated as 0.83 (%95CI 0.82–0.85) according to random effects model. When the prediction interval is investigated, it can be said that the range of alpha values that might be expected within primary studies using total CCTDI would likely fall between 0.65 and 0.92.

Also, the heterogeneity tests for subscales showed that there is heterogeneity among the studies for TS-scale (Q_(df=45) = 800.68), OM-scale (Q_(df=49) = 1007.15), A-scale (Q_(df=46) = 1378.78), S-scale (Q_(df=46) = 811.90), SC-scale (Q_(df=50) = 664.97), I-scale (Q_(df=47) = 613.92), and M-scale (Q_(df=47) = 776.79). According to random effects model, the general alpha coefficients are for TS-scale 0.65, for OM-scale 0.56, for A-scale 0.64, for S-scale 0.66, for SC-scale 0.74, for I-scale 0.72, and for M-scale 0.61. Forest plots for total CCTDI and its subscales can be seen in online supplemental material.

Results on the Moderator Analyses

As a high level of heterogeneity was found among studies, moderator analyses for categorical variables were conducted according to the subgroups of sample type, language, discipline, and country of the study.

As seen in Table 2, the heterogeneity values of the subgroup of sample type for total CCTDI (Q_B = 19.691), for TS-scale (Q_B = 61.438), for OM-scale (Q_B = 32.510), for A-scale (Q_B = 18.559), for S-scale (Q_B = 6.124), for SC-scale (Q_B = 47.57), for I-scale (Q_B = 6.907), and for M-scale (Q_B = 30.852) are greater than the X² table critical values. Therefore, it can be said that subgroup of sample type is the real moderator on the general alpha coefficient of total CCTDI and its subscales. Total CCTDI and each subscale have greater alpha values with university students and nurses. Also, the heterogeneity values of the subgroup of language for total CCTDI (Q_B = 21.813), for OM-scale (Q_B = 4.617), for A-scale (Q_B = 4.838), for S-scale (Q_B = 6.100), and for I-scale (Q_B = 10.509) are greater than the X² table critical values. However, the heterogeneity values for TS-scale (Q_B = 1.209), for SC-scale (Q_B = 0.870), and for M-scale (Q_B = 0.594) are less than the X² table critical values. These results indicate that while language in which the inventory was administered is a significant moderator on the general alpha coefficient of total CCTDI, OM-scale, A-scale, S-scale, and I-scale, it is not on the general alpha value of TS-scale, SC-scale, and M-scale. English CCTDI and subscales have greater alpha values than the non-English versions of them. Besides, the heterogeneity values of the subgroup of discipline for total CCTDI (Q_B = 1.611), for TS-scale (Q_B = 1.284), for OM-scale (Q_B = 0.762), for A-scale (Q_B = 0.485), for S-scale (Q_B = 0.239), for SC-scale (Q_B = 1.668), for I-scale (Q_B = 0.347), and for M-scale (Q_B = 3.312) are less than the X² table critical values. Therefore, subgroup of discipline is not a significant moderator on the general alpha coefficient of total CCTDI and its subscales. In addition to these, while the heterogeneity values of the subgroup of country of the study for total CCTDI (Q_B = 10.822), for S-scale (Q_B = 6.498), and for I-scale (Q_B = 9.908) are greater than the X² table critical values, these values are less than the X² table critical values for TS-scale (Q_B = 2.407), for OM-scale (Q_B = 3.412), for A-scale (Q_B = 1.901), for SC-scale (Q_B = 0.848), and for M-scale (Q_B = 0.548). These results indicate that while country of the study is a significant moderator on the general alpha coefficient of total CCTDI, S-scale, and I-scale, it is not on the general alpha value of TS-scale, OM-scale, A-scale, SC-scale, and M-scale.

Table 2.

Comparison of alpha coefficients of total CCTDI and subscales across categorical moderators.

Categorical Variables		Subgroups			%95 CI		df	Heterogeneity Test
Categorical Variables		Subgroups	k	α	Upper	Lower	df	Q_B-Value	p
TOTAL CCTDI	Sample type	Academicians	2	0.83	0.85	0.79	4	19.691	0.001
		Nurses	14	0.84	0.87	0.80
		Elementary school students	3	0.75	0.83	0.63
		High school students	2	0.74	0.79	0.67
		University students	75	0.84	0.85	0.82
	Language	English	68	0.85	0.86	0.83	1	21.813	0.000
	Language	Non-English	30	0.79	0.81	0.77	1	21.813	0.000
	Discipline	Medical	44	0.83	0.85	0.81	1	1.611	0.204
	Discipline	Non-medical	22	0.85	0.87	0.82	1	1.611	0.204
	Country of the study	The USA	40	0.86	0.87	0.84	1	10.822	0.001
	Country of the study	Other	55	0.82	0.83	0.80	1	10.822	0.001
TS-Scale	Sample type	Nurses	9	0.63	0.68	0.58	2	61.438	0.000
		High school students	3	0.43	0.48	0.38
		University students	32	0.67	0.71	0.63
	Language	English	31	0.67	0.71	0.63	1	1.209	0.272
	Language	Non-English	15	0.62	0.69	0.54	1	1.209	0.272
	Discipline	Medical	15	0.61	0.64	0.57	1	1.284	0.257
	Discipline	Non-medical	6	0.71	0.82	0.52	1	1.284	0.257
	Country of the study	The USA	24	0.68	0.72	0.64	1	2.407	0.121
	Country of the study	Other	22	0.62	0.68	0.56	1	2.407	0.121
OM-Scale	Sample type	Nurses	9	0.53	0.64	0.39	2	32.510	0.000
		High school students	3	0.35	0.40	0.29
		University students	36	0.58	0.63	0.52
	Language	English	34	0.60	0.65	0.55	1	4.617	0.032
	Language	Non-English	16	0.48	0.57	0.36	1	4.617	0.032
	Discipline	Medical	17	0.53	0.57	0.48	1	0.762	0.383
	Discipline	Non-medical	8	0.62	0.76	0.39	1	0.762	0.383
	Country of the study	The USA	26	0.60	0.65	0.55	1	3.412	0.065
	Country of the study	Other	24	0.51	0.59	0.42	1	3.412	0.065
A-Scale	Sample type	Nurses	9	0.57	0.66	0.45	2	18.559	0.000
		High school students	3	0.51	0.55	0.46
		University students	33	0.67	0.72	0.61
	Language	English	31	0.68	0.73	0.63	1	4.838	0.028
	Language	Non-English	16	0.56	0.66	0.42	1	4.838	0.028
	Discipline	Medical	16	0.66	0.72	0.59	1	0.485	0.486
	Discipline	Non-medical	6	0.74	0.87	0.47	1	0.485	0.486
	Country of the study	The USA	24	0.68	0.73	0.62	1	1.901	0.168
	Country of the study	Other	23	0.61	0.68	0.52	1	1.901	0.168
S-Scale	Sample type	Nurses	9	0.65	0.69	0.60	2	6.124	0.047
		High school students	3	0.57	0.64	0.49
		University students	33	0.67	0.72	0.62
	Language	English	31	0.69	0.73	0.66	1	6.100	0.014
	Language	Non-English	16	0.60	0.67	0.51	1	6.100	0.014
	Discipline	Medical	16	0.66	0.69	0.61	1	0.239	0.625
	Discipline	Non-medical	6	0.71	0.84	0.46	1	0.239	0.625
	Country of the study	The USA	24	0.70	0.74	0.67	1	6.498	0.011
	Country of the study	Other	23	0.62	0.67	0.55	1	6.498	0.011
SC-Scale	Sample type	Nurses	10	0.75	0.80	0.70	2	47.576	0.000
		High school students	3	0.59	0.63	0.54
		University students	36	0.75	0.77	0.72
	Language	English	35	0.75	0.78	0.73	1	0.870	0.351
	Language	Non-English	16	0.72	0.77	0.67	1	0.870	0.351
	Discipline	Medical	17	0.72	0.76	0.67	1	1.668	0.196
	Discipline	Non-medical	8	0.78	0.84	0.69	1	1.668	0.196
	Country of the study	The USA	28	0.75	0.78	0.72	1	0.848	0.357
	Country of the study	Other	23	0.73	0.77	0.69	1	0.848	0.357
I-Scale	Sample type	Nurses	9	0.68	0.71	0.64	2	6.907	0.032
		High school students	3	0.67	0.72	0.61
		University students	34	0.73	0.76	0.70
	Language	English	32	0.75	0.77	0.72	1	10.509	0.001
	Language	Non-English	16	0.66	0.71	0.60	1	10.509	0.001
	Discipline	Medical	16	0.72	0.77	0.66	1	0.347	0.556
	Discipline	Non-medical	7	0.75	0.83	0.65	1	0.347	0.556
	Country of the study	The USA	25	0.75	0.78	0.73	1	9.908	0.002
	Country of the study	Other	23	0.68	0.72	0.64	1	9.908	0.002
M-Scale	Sample type	Nurses	9	0.61	0.66	0.54	2	30.852	0.000
		High school students	3	0.45	0.49	0.40
		University students	34	0.62	0.67	0.57
	Language	English	32	0.62	0.67	0.57	1	0.594	0.441
	Language	Non-English	16	0.59	0.66	0.50	1	0.594	0.441
	Discipline	Medical	17	0.53	0.58	0.48	1	3.312	0.069
	Discipline	Non-medical	6	0.68	0.79	0.52	1	3.312	0.069
	Country of the study	The USA	24	0.63	0.68	0.57	1	0.548	0.459
	Country of the study	Other	24	0.60	0.66	0.52	1	0.548	0.459

Note. k = number of reliability coefficients; %95 CI = upper and lower bounds of the %95 confidence interval around the overall reliability estimate; df = degrees of freedom; Q_B = between category heterogeneity statistic; p value for Q_B statistic; medical subgroup includes disciplines such as nursery and midwifery; non-medical subgroup includes disciplines such as education, liberal arts, and language.

Regarding the continuous moderators, Table 3 presents the results of the simple meta-regressions applied on the alpha values for total CCTDI and subscales.

Table 3.

Results of the simple meta-regression analysis by the continuous moderator variables for total CCTDI and subscales.

Continuous Variables	k	b(SE)	%95CI	p	τ ²_res	I²_res	R ²
Total CCTDI
Mean age (in years)	51	−0.011(0.008)	[−0.029, −0.005]	0.182	0.163	94.86	0.00
%Female	66	−0.000(0.002)	[−0.005, 0.003]	0.743	0.153	95.01	0.00
%Caucasian	18	−0.002(0.003)	[−0.008, 0.003]	0.467	0.071	97.47	0.00
Score mean	55	−0.004(0.002)	[−0.008, −0.000]	0.035	0.132	93.40	0.05
Score SD	50	−0.013(0.006)	[−0.024, −0.001]	0.027	0.127	93.63	0.10
TS-scale
Mean age (in years)	29	0.007(0.012)	[−0.016, 0.032]	0.524	0.174	95.88	0.00
%Female	36	0.002(0.003)	[−0.004, 0.010]	0.437	0.135	95.01	0.03
%Caucasian	11	0.005(0.005)	[−0.005, 0.015]	0.330	0.159	94.33	0.00
Score mean	28	0.005(0.019)	[−0.034, 0.044]	0.801	0.208	96.06	0.00
Score SD	23	−0.060(0.048)	[−0.155, 0.035]	0.217	0.146	95.21	0.32
OM-scale
Mean age (in years)	30	0.002(0.129)	[−0.022, 0.028]	0.830	0.189	96.10	0.02
%Female	40	0.003(0.003)	[−0.003, 0.010]	0.319	0.167	95.85	0.06
%Caucasian	14	0.007(0.004)	[−0.001, 0.016]	0.088	0.115	92.09	0.12
Score mean	29	0.009(0.020)	[−0.030, 0.049]	0.647	0.257	96.75	0.00
Score SD	23	−0.071(0.050)	[−0.170, 0.027]	0.159	0.229	96.61	0.17
A-scale
Mean age (in years)	29	0.012(0.015)	[−0.019, 0.043]	0.446	0.291	97.47	0.10
%Female	37	0.004(0.004)	[−0.004, 0.013]	0.350	0.229	96.93	0.08
%Caucasian	11	0.005(0.007)	[−0.008, 0.019]	0.454	0.312	97.00	0.00
Score mean	27	−0.000(0.028)	[−0.056, 0.055]	0.986	0.407	97.86	0.00
Score SD	21	−0.013(0.046)	[−0.230, −0.046]	0.003	0.180	96.08	0.55
S-scale
Mean age (in years)	29	0.001(0.011)	[−0.021, 0.024]	0.906	0.152	95.27	0.01
%Female	37	0.002(0.003)	[−0.005, 0.009]	0.591	0.138	95.01	0.01
%Caucasian	11	0.002(0.005)	[−0.008, 0.012]	0.692	0.169	94.60	0.00
Score mean	27	0.003(0.025)	[−0.045, 0.052]	0.881	0.210	96.03	0.00
Score SD	21	−0.093(0.036)	[−0.166, −0.021]	0.010	0.098	93.07	0.54
SC-scale
Mean age (in years)	31	−0.004(0.010)	[−0.025, 0.015]	0.645	0.138	94.67	0.00
%Female	41	−0.001(0.003)	[−0.007, 0.004]	0.613	0.110	93.40	0.00
%Caucasian	14	0.007(0.004)	[−0.001, 0.015]	0.099	0.108	91.23	0.13
Score mean	31	0.021(0.013)	[−0.004, 0.045]	0.106	0.141	93.74	0.01
Score SD	25	−0.076(0.042)	[−0.159, 0.006]	0.071	0.117	93.73	0.23
I-scale
Mean age (in years)	29	0.006(0.010)	[−0.013, 0.025]	0.551	0.110	93.56	0.05
%Female	38	0.001(0.003)	[−0.004, 0.007]	0.635	0.108	93.61	0.00
%Caucasian	12	0.000(0.003)	[−0.006, 0.007]	0.797	0.069	87.56	0.00
Score mean	30	0.006(0.012)	[−0.017, 0.029]	0.611	0.106	92.15	0.00
Score SD	24	−0.055(0.032)	[−0.119, 0.008]	0.087	0.068	89.97	0.31
M-scale
Mean age (in years)	30	−0.002(0.011)	[−0.024, 0.019]	0.808	0.136	94.58	0.00
%Female	38	0.007(0.003)	[0.000, 0.014]	0.030	0.124	94.33	0.08
%Caucasian	12	0.009(0.004)	[0.000, 0.018]	0.041	0.127	92.43	0.26
Score mean	30	0.023(0.014)	[−0.005, 0.052]	0.112	0.148	94.42	0.04
Score SD	24	−0.071(0.039)	[−0.148, 0.005]	0.067	0.107	93.25	0.32

Note. k is the number of studies that contained information for each of the variables; %95 CI = upper and lower bounds of the %95 confidence interval, τ²_res means estimated τ² residual; I²_res means I² residual; R²_Meta means R² of the mixed-effects model. Note that as meta-regression analysis was conducted with Bonett’s transformation values instead of alpha values, the true direction of the relationship between each continuous moderator and alpha values is the inverse of what is shown by the sign of the slope in the table.

As shown in Table 3, the mean of the test scores significantly explained %5 of the variance of alpha values of total CCTDI (p = 0.035). Also, SD of the test scores significantly explained %10 (p = 0.027), %55 (p = 0.003), %54 (p = 0.010) of the variance of alpha values of total CCTDI, A-scale, and S-scale, respectively. There was a statistically significant positive correlation between the SD of the test scores and alpha values for total CCTDI, A-scale, and S-scale. Besides, it was found that the gender (accounting for %8 of the variance) and ethnicity (accounting for %26 of the variance) significantly moderated the alpha values for M-scale. There was a statistically significant negative correlation between the gender (%female), ethnicity (%Caucasian) and alpha values for M-scale. In addition to these, there were not any significant association between other continuous moderator variables and alpha values for the total CCTDI and subscales scores.

Conclusion and Discussion

The aims of this RG study were to provide overall alpha values of the CCTDI total score and subscales scores and to investigate the characteristics of the studies that may be associated with the variability in the reliability values of the CCTDI total score and subscales scores. Of the 491 studies that administered the CCTDI or at least one of its subscales, only 94 (%19.14) reported usable reliability value. 115 of them (%23.42) did not report any alpha values and 248 of them (%50.51) induced reliabilities from previous research. Because reliability is a dynamic property of instruments (Thompson, 2002) and it may vary across test occasions, settings, and samples (Dawis, 1987), it must be calculated in each application (Crocker & Algina, 1986; Streiner et al., 2015). Therefore, underreporting of reliability in these studies can be seen as a concerning issue because most of the studies (%73.93) failed to report any form of reliability for their data. Literature on the reporting of reliability states that many researchers do not report reliability values for their samples (Henson & Thompson, 2002; Vacha-Haase, 1998). Besides, a low percentage of studies administered the CCTDI with usable reliability value (%19.14) is similar to previous RG studies investigating score reliability of instruments from different behavioral science fields (e.g., Bruna et al., 2019; Alcocer‐Bruno et al., 2020). There may be two reasons for this: First, researchers do not report reliability values because they know the instrument is not reliable for their sample. Second, they are not aware that it should be reported for each administration. In either case, the underreporting of reliability values can be harmful to the interpretation of findings in the studies. Therefore, score reliability reporting practices should be improved although this underreporting of reliability problem may not be generalized across all fields of scientific inquiry.

Total CCTDI was found to be reliable across samples with alpha value of 0.83. Also, the subscale which has the greatest alpha value is SC-scale (α = 0.74). I-scale (α = 0.72), S-scale (α = 0.66), TS-scale (α = 0.65), A-scale (α = 0.64), M-scale (α = 0.61), and OM-scale (α = 0.56) follow it, respectively. This RG study indicated lower alpha values for the CCTDI and its subscales than the alpha values reported in test manuals (Facione & Facione, 1992; Facione et al., 1994). Nunnally and Bernstein (1994) state that reliability coefficients above 0.70 are adequate for exploratory research. Therefore, alpha values for the subscales of SC-scale and I-scale are satisfactory. Also, alpha values of S-scale, TS-scale, A-scale, and M-scale can be seen as acceptable for research purposes. However, OM-scale has a really low alpha value. It was found out that scores on the subscales had lower alpha values than the scores on total CCTDI. This finding may be attributable to the subscales having a smaller number of items because alpha values are less stable with a smaller number of items (Thompson, 2002). The studies investigating psychometric properties of the CCTDI (Walsh & Hardy, 1997; Kakai 2003; Bondy et al., 2001; Walsh et al., 2007) indicate that there is no agreement on a conceptual level about the number of factors and the factor structure of the items. The number of factors and the factor structure are not consistent with the original factor structure that emerged in initial pilot studies by Facione and Facione (1992) and Facione et al. (1994). Therefore, this can be another reason why the subscales had lower alpha values. This unstable factor structure of the CCTDI can increase measurement errors and decrease the overall reliability of the subscales.

Examination of study characteristics indicated that sample type was a significant predictor for alpha value of total CCTDI and all subscales. Samples with university students reported larger Cronbach’s alpha estimates for total CCTDI and all subscales. Also, it was found out that while country of the study was a significant moderator on the general alpha coefficient of total CCTDI, S-scale, and I-scale, it was not on the general alpha value of TS-scale, OM-scale, A-scale, SC-scale, and M-scale. However, alpha values for total CCTDI and seven subscales were greater for administrations in the USA. Because the CCTDI was developed in the USA and tested using a sample consisting mostly of university students (Facione & Facione, 1992; Facione et al., 1994), these two moderator results are not surprising. It is conceivable that items of the CCTDI may hold different meanings for samples other than university students and for samples who live in countries other than the USA. Therefore, it can be said that when the CCTDI is used with samples consisting of mostly university students and in the USA, items are more likely to be interpreted similarly and this leads to higher alpha values.

Also, while language was found to moderate the general alpha coefficient of total CCTDI, OM-scale, A-scale, S-scale, and I-scale, it was not a significant moderator on the general alpha value of TS-scale, SC-scale, and M-scale. Total CCTDI and all subscales showed higher Cronbach’s alpha values for the English-language administrations. This result suggests that the cultural and language validations and adaptations of the CCTDI are needed to be improved as administrating the non-English versions instead of the original version seems to decrease the reliability. The adaptation of a psychological instrument to other languages and different cultures is a process that involves not only the translation of the items but also validation studies for this population (Messick, 1995). Also, this finding may be attributable to the quality of translations from English to other languages. Therefore, the translation processes of the CCTDI to other languages need to be further examined.

Besides, moderator analyses showed that subgroup of discipline was not a significant moderator on the general alpha coefficient of total CCTDI and its subscales. Because the CCTDI is a discipline neutral inventory (Facione et al., 1994), this moderator result is not a surprising one. This moderator result indicates that CCTDI can yield reliable results when it is used within different disciplines.

In addition to these, simple meta-regression analyses by the continuous moderator variables for total CCTDI and subscales indicated that SD of the test scores significantly moderated the alpha values for total CCTDI, A-scale, and S-scale. Also, there was a statistically significant positive correlation between mean of the test scores and the alpha values of total CCTDI. These two findings are in line with the findings of previous RG studies (Rubio-Aparicio et al., 2020; Deng et al., 2019). According to Crocker and Algina (1986), if the observed score variance increases, internal consistency will also increase. Therefore, it can be said that these two findings are similar to previous RG studies and in line with classical test theory. Also, simple meta-regression analyses showed that mean age, gender, and ethnicity were not significant moderators on the general alpha coefficient of total CCTDI and its subscales (except for gender and ethnicity on M-scale). This finding may be attributable to the fact that development of the CCTDI was carried out in the USA which has ethnically diverse population and with sample including male and female students.

In the behavioral sciences, the reliability of the inferences made by researchers is dependent on the assumption that the scores produced by the instrument are reliable and valid. There can be little question that the CCTDI has been a widely used instrument to measure CT dispositions since its development in 1992. After its development, validity of the CCTDI scores has been investigated by some researchers (e.g., Walsh & Hardy, 1997; Kakai 2003; Bondy et al., 2001; Walsh et al., 2007). This RG study was designed to complement the existing body of literature on the CCTDI by providing reliability evidence for the total CCTDI and subscales scores. Total CCTDI was found to be reliable across samples with alpha value of 0.83. However, alpha values for the subscales were relatively low, especially for OM-scale.

Limitations and Recommendations for Future Research

Although this study is the first to shed light on the overall reliability of the CCTDI and its subscales and to examine moderators, it has several limitations. Although there are shortened versions of the CCTDI, this RG meta-analysis was conducted with the studies administered the original CCTDI with 75 items. Therefore, it can be seen as the first limitation of this study. Second, it has a limitation related to the selection of primary studies. Although the CCTDI is one of the most popular instruments used to measure CT dispositions, there are not many studies which report reliability value for different samples and this reduces the number of eligible studies for this RG study. Third, there may be other moderators that were not considered in this study such as other country or sample characteristics. Fourth, the CCTDI is mostly used within disciplines related to medical (e.g., nursery or midwifery) and the number of studies for other disciplines is limited. Therefore, I had to divide the disciplines into two groups as medical and non-medical.

The current results have important implications for other studies and future applications of the CCTDI. Further studies can be done to investigate other possible moderators on the reliability of the CCTDI. Also, other studies can be done to compare the internal consistency of the original CCTDI with its shortened versions. Besides, further investigation into the factor structure of the CCTDI should be done to provide evidence on validity. In addition to these, researchers should be aware that the CCTDI subscales, especially S-scale, TS-scale, A-scale, M-scale, and OM-scale, have relatively low alpha values and the CCTDI has higher reliability scores when it was used in English and with university students. I also recommend for future researchers utilizing the CCTDI to report the sample’s reliability estimates for the subscales of the CCTDI. If the number of the reliability values for the subscales increases, other researchers can synthesize reliability estimates of subscales.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Ali Orhan

References

Alcocer‐Bruno

Ferrer‐Cascales

Rubio‐Aparicio

Ruiz‐Robledillo

(2020). The medical outcome study‐ HIV health survey: A systematic review and reliability generalization meta‐analysis. Research in Nursing & Health, 43, 610–620. https://doi.org/10.1002/nur.22070

Bondy

K. N.

Koenigseder

L. A.

Ishee

J. H.

Williams

B. G.

(2001). Psychometric properties of the California critical thinking tests. Journal of Nursing Measurement, 9, 309–328. https://doi.org/10.1891/1061-3749.9.3.309

Bonett

D. G.

(2002). Sample size requirements for testing and estimating coefficient alpha. Journal of Educational and Behavioral Statistics, 27, 335–340. https://doi.org/10.3102/10769986027004335

Borenstein

Hedges

L. V.

Higgins

J. P. T.

Rothstein

H. R.

(2009). Introduction to meta-analysis. John Wiley & Sons, Ltd.

Bruna

M. O.

Brabete

A. C.

Izquierdo

J. M. A.

(2019). Reliability generalization as a seal of quality of substantive meta-analyses: The case of the VIA inventory of strengths (VIA-IS) and their relationships to life satisfaction. Psychological Reports, 122(3), 1167–1188. https://doi.org/10.1177/0033294118779198

Cooper

(2017). Research synthesis and meta-analysis: A step-by-step approach. Sage Publications, Inc.

Crocker

Algina

(1986). Introduction to classical and modern test theory. Holt, Rinehart, and Winston.

Cronbach

L. J.

(1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16(3), 297–334. https://doi.org/10.1007/BF02310555

Dawis

R. V.

(1987). Scale construction. Journal of Counseling Psychology, 34(4), 481–489. http://doi.org/10.1037/0022-0167.34.4.481

10.

Deng

Wang

M.-C.

Zhang

Shou

Gao

Luo

(2019). The inventory of callous unemotional traits: A reliability generalization meta-analysis. Psychological Assessment, 31(6), 765-780. https://doi.org/10.1037/pas0000698

11.

Dimitrov

D. M.

(2002). Reliability: Arguments for multiple perspectives and potential problems with generalization across studies. Educational and Psychological Measurement, 62(5), 783–801. https://doi.org/10.1177/001316402236878

12.

Duval

Tweedie

(2000). Trim and fill: A simple funnel-plot-based method of testing and adjusting for publication bias in meta-analysis. Biometrics, 56(2), 455–463. https://doi.org/10.1111/j.0006-341x.2000.00455.x

13.

Ennis

R. H.

(1987). Critical thinking and subject specificity: Clarification and needed research. Educational Researcher, 18(3), 4–10. https://doi.org/10.3102/0013189X018003004

14.

Ennis

R. H.

(1991). Goals for a critical thinking curriculum and its assessment. In Costa

A. L.

(Ed), In Developing minds: A resource book for teaching thinking. Ascd.

15.

Epstein

R. L.

Kernberger

(2012). Critical thinking. Advanced Reasoning Forum.

16.

Facione

P. A.

(1990). Critical thinking: A statement of expert consensus for purposes of educational assessment and instruction - executive summary - the Delphi report. The California Academic Press.

17.

Facione

P. A.

Facione

(1992). The California critical thinking dispositions inventory (CCTDI) and the CCTDI test manual. California Academic Press.

18.

Facione

P. A.

Facione

N. C.

Giancarlo

C. A.

(2000). The disposition toward critical thinking: Its character, measurement and relationship to critical thinking skill. Journal of Informal Logic, 20(1), 61–84. https://doi.org/10.22329/il.v20i1.2254

19.

Facione

N. C.

Facione

P. A.

Sanchez

C. A.

(1994). Critical thinking disposition as a measure of competent clinical judgment: The development of the California critical thinking disposition inventory. Journal of Nursing Education, 33(8), 345–350. https://doi.org/10.3928/0148-4834-19941001-05

20.

Hakstian

A. R.

Whalen

T. E.

(1976). A k-sample significance test for independent alpha coefficients. Psychometrika, 41(2), 219–231. https://doi.org/10.1007/BF02291840

21.

Halpern

D. F.

(2003). Thought & knowledge: An introduction to critical thinking. Lawrence Erlbaum Associates Publishers.

22.

Hedges

L. V.

Olkin

(1985). Statistical methods for meta-analysis. Academic Press.

23.

Henson

R. K.

Thompson

(2002). Characterizing measurement error in scores across studies: Some recommendations for conducting “reliability generalization” studies. Measurement and Evaluation in Counseling and Development, 35, 113–127. https://doi.org/10.1080/07481756.2002.12069054

24.

Huedo-Medina

T. B.

Sánchez-Meca

Marín-Martínez

Botella

(2006). Assessing heterogeneity in meta-analysis: Q statistic or I² index? Psychological Methods, 11(2), 193–206. https://doi.org/10.1037/1082-989X.11.2.193

25.

Kakai

(2003). Re-examining the factor structure of the California critical thinking disposition inventory. Perceptual and Motor Skills, 96, 435–438. https://doi.org/10.2466/pms.2003.96.2.435

26.

Lai

E. R.

(2011). Critical thinking: A literature review research report. Parsons Publishing.

27.

Lewis

Smith

(1993). Defining higher order thinking. Theory Into Practice, 32(3), 131–137. https://doi.org/10.1080/00405849309543588

28.

Lipsey

M. W.

Wilson

D. B.

(2001). Practical meta-analysis. SAGE.

29.

McNeish

(2018). Thanks coefficient alpha, we’ll take it from here. Psychological Methods, 23(3), 412–433. https://doi.org/10.1037/met0000144.

30.

Messick

(1995). Validity of psychological assessment: Validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. American Psychologist, 50(9), 741–749. https://doi.org/10.1037/0003-066X.50.9.741

31.

Nunnally

J. C.

Bernstein

I. H.

(1994). Psychometric theory (3rd ed.). McGraw-Hill.

32.

Pascarella

E. T.

Terenzini

P. T.

(1991). How college affects students: Findings and insights from twenty years of research. Jossey-Bass.

33.

Paul

(1990). Critical thinking: What every person needs to survive in a rapidly changing world. Center for Critical Thinking and Moral Critique.

34.

Paul

Elder

(2001). Critical thinking: Tools for taking charge of your learning and your life. Prentice Hall.

35.

Petticrew

Roberts

(2006). Systematic reviews in the social sciences: A practical guide. Blackwell Publishing.

36.

Profetto-McGrath

(2003). The relationship of critical thinking skills and critical thinking dispositions of baccalaureate nursing students. Journal of Advanced Nursing, 43(6), 569–577. https://doi.org/10.1046/j.1365-2648.2003.02755.x

37.

Profetto-McGrath

Hesketh

K. L.

Lang

Estabrooks

C. A.

(2003). A study of critical thinking and research utilization among nurses. Western Journal of Nursing Research, 25(3), 322–337. https://doi.org/10.1177/0193945902250421

38.

Rodriguez

M. C.

Maeda

(2006). Meta-analysis of coefficient alpha. Psychological Methods, 11(3), 306–322. https://doi.org/10.1037/1082-989X.11.3.306

39.

Rubio-Aparicio

Badenes-Ribera

Sánchez‐Meca

Fabris

M. A.

Longobardi

(2020). A reliability generalization meta‐analysis of self‐report measures of muscle dysmorphia. Clinical Psychology: Science and Practice, 27, 1–24.

40.

Ruggerio

V. R.

(1990). Beyond feelings: A guide to critical thinking. Mayfield Publishing Company.

41.

Sánchez‐Meca

Marín‐Martínez

López‐López

J. A.

Núñez‐Núñez

R. M.

Rubio‐Aparicio

López‐García

J. J.

López‐Pina

J. A.

Blázquez‐Rincón

D. M.

López‐Ibáñez

López‐Nicolás

(2021). Improving the reporting quality of reliability generalization meta‐analyses: The REGEMA checklist. Research Synthesis Methods, 12(4), 516–536. https://doi.org/10.1002/jrsm.1487

42.

Schmidt

F. L.

Hunter

J. E.

(2015). Methods of meta-analysis: Correcting error and bias in research findings. SAGE Publications, Inc.

43.

Semma

Henri

Luo

Thompson

C. G.

(2019). Reliability generalization of the meaning in life questionnaire subscales. Journal of Psychoeducational Assessment, 37(7), 837–851. https://doi.org/10.1177/0734282918800739

44.

Sternberg

R. J.

(1986). Critical thinking: Its nature, measurement, and improvement. http://eric.ed.gov/PDFS/ED272882.pdf

45.

Streiner

D. L.

Norman

G. R.

Cairney

(2015). Health measurement scales: A practical guide to their development and use. Oxford University Press.

46.

Thompson

(2002). Score reliability: Contemporary thinking on reliability issues. Sage.

47.

Thompson

Vacha-Haase

(2000). Psychometrics is datametrics: The test is not reliable. Educational and Psychological Measurement, 60(2), 174–195. https://doi.org/10.1177/0013164400602002

48.

Traub

R. E.

Rowley

G. L.

(1991). An NCME instructional module on. understanding reliability. Educational Measurement: Issues and Practice, 10(1), 37–45. https://doi.org/10.1111/j.1745-3992.1991.tb00183.x

49.

Vacha-Haase

(1998). Reliability generalization: Exploring variance in measurement error affecting score reliability across studies. Educational and Psychological Measurement, 58, 6–20. https://doi.org/10.1177/0013164498058001002

50.

Walsh

C. M.

Hardy

R. C.

(1997). Factor structure stability of the california critical thinking disposition inventory across sex and various students’ majors. Perceptual and Motor Skills, 85(Suppl 3), 1211–1228. https://doi.org/10.2466/pms.1997.85.3f.1211

51.

Walsh

C. M.

Seldomridge

L. A.

Badros

K. K.

(2007). California critical thinking disposition inventory: Further factor analytic examination. Perceptual and Motor Skills, 104(1), 141–151. https://doi.org/10.2466/pms.104.1.141-151

52.

Yetkiner

Z. E.

Thompson

(2010). Demonstration of how score reliability is integrated into SEM and how reliability affects all statistical analyses. Multiple Linear Regression Viewpoints, 36(2), 1–12. https://doi.org/10.13140/RG.2.1.4979.6561