Reliability for Tests With Items Having Different Numbers of Ordered Categories

Abstract

This study describes a structural equation modeling (SEM) approach to reliability for tests with items having different numbers of ordered categories. A simulation study is provided to compare the performance of this reliability coefficient, coefficient alpha and population reliability for tests having items with different numbers of ordered categories, a one-factor and a bifactor structures, and different skewness distributions of test scores. Results indicated that the proposed reliability coefficient was close to the population reliability in most conditions. An empirical example was used to illustrate the performance of the different coefficients for a test of items with two or three ordered categories.

Keywords

reliability structural equation modeling categorical data

Items scored with ordered categories are common in social and behavioral sciences (Bollen, 1989; Finney & DiStefano, 2006). Different approaches to estimation of reliability have been proposed for tests with such items. Coefficient alpha (Cronbach, 1951) is probably the most commonly used to obtain an estimate of test reliability and is taken as a lower bound on the internal consistency of the test (Green, Lissitz, & Mulaik, 1977; Novick & Lewis, 1967; Sijtsma, 2009). It is based on the following assumptions: (a) errors in observed item scores are not correlated and (b) items in a test are essentially $τ$ -equivalent (Novick & Lewis, 1967; Raykov, 1997). Green and Yang (2009) have shown that the values of coefficient alpha for ordered categorical data were the same as the values of model-based reliabilities, when items were essentially $τ$ -equivalent, and items had the same number of categories. There are also a number of studies that have investigated the effects of the number of response categories on coefficient alpha (e.g., Bandalos & Enders, 1996; Komorita & Graham, 1965; Lissitz & Green, 1975; Lozano, García-Cueto, & Muñiz, 2008; Weng, 2004). These studies showed that coefficient alpha generally increases as the number of response categories increases. It is not clear, however, what happens when the number of ordered categories on a test becomes more widely separated, for example, two categories for some items, and four or five categories for other items.

Another approach for estimating reliability is through structural equation modeling (SEM; Bentler, 2009; Bollen, 1989; Green & Yang, 2009; Miller, 1995; Raykov, 1997; Raykov & Shrout, 2002). The SEM approach introduces a factorial structure of the test into the estimation of reliability. It has been shown to be especially useful for estimating reliability for tests having clusters that group items as in testlets (Cho & Kim, 2015; Green & Yang, 2015; Raykov & Shrout, 2002; Yang & Green, 2010). This could be items sharing the same passage in a reading test (e.g., DeMars, 2006; Li, Bolt, & Fu, 2006; Rijmen, 2010), items having similar reasoning methods as in a mathematics test (e.g., Cronbach & Shavelson, 2004; DeMars, 2006), or survey items measuring a specific aspect of personality (e.g., Hyland, Boduszek, Dhingra, Shevlin, & Egan, 2014; McAbee, Oswald, & Connelly, 2014). In such cases, the tests are assumed to have a single general factor (e.g., reading ability, math ability, or personality) and group factors that reflect the clusters.

With a standard linear SEM approach, item scores may be presented using a confirmatory factor analysis (CFA) model (e.g., Raykov & Shrout, 2002; Yang & Green, 2011), and reliability is calculated as the ratio of true score variance to observed score variance, where the true score variance is estimated using the CFA model. This reliability is also referred to as coefficient omega (McDonald, 1985, 1999). Green and Yang (2009) proposed a nonlinear SEM reliability coefficient for a test with ordinal categorical items within an SEM framework. For such a test, if the structure of the test is well-specified, the nonlinear SEM approach has been found to more accurately estimate reliability than the linear SEM approach, as the linear approach treats categorical scores as continuous (Green & Yang, 2009).

Green and Yang (2009) describe a test consisting of items with the same number of categories. However, in reality, tests often consist of items with different numbers of categories. For example, the PARCC (Partnership for Assessment of Readiness for College and Careers; 2016) assessment for mathematics and the Smarter Balanced Assessment (Smarter Balanced Assessment Consortium, 2016) include items that are scored with rubrics that have from two to seven points and two to four points, respectively. Likewise, a test such as the Early Development Instrument (Janus & Offord, 2007) for measuring children’s school readiness also has items scored with four to eight points.

The present study was designed to examine the effects on reliability of test scores when a test consisted of items having uneven numbers of categories. In this study, the authors focus on total test scores, that is, scores that are computed as sums of points over all items. Below, the study by Green and Yang (2009) is extended to utilize the nonlinear SEM reliability for tests with items having either the same or different numbers of ordered response categories. A numerical calculation is provided to derive the extended nonlinear SEM reliability and conduct a simulation study to evaluate the performance of this nonlinear SEM reliability. The performance of the resulting nonlinear reliability coefficient is compared to coefficient alpha and population reliability for tests with different numbers of categories compared to tests for which the items had the same number of categories. In addition, an example using real data with different numbers of ordered categories is provided to illustrate the application of the different reliabilities.

Reliability in Classical Test Theory (CTT)

Suppose there are $J$ items in a test. In CTT, an observed score $X_{j}$ on item $j$ $(j = 1, \dots, J)$ is composed of two components, a true score, $T_{j}$ , and an error score, $ε_{j}$ :

X_{j} = T_{j} + ε_{j} .

(1)

Let $X$ , $T$ , and $ε$ be the sum of observed scores, of true scores, and of error scores, respectively, across $J$ items. Then $X = \sum_{j = 1}^{J} X_{j}, T = \sum_{j = 1}^{J} T_{j}, ε = \sum_{j = 1}^{J} ε_{j},$ and $X = T + ε$ . The reliability coefficient of a test is defined as

ρ = \frac{σ_{T}^{2}}{σ_{X}^{2}},

(2)

where $σ_{T}^{2}$ is the variance of $T$ and $σ_{X}^{2}$ is the variance of $X$ (Lord & Novick, 1968).

Coefficient alpha (Cronbach, 1951) is often used to estimate reliability

α = \frac{J}{J - 1} (\frac{\sum_{j \neq j'} Cov (X_{j}, X_{j'})}{σ_{X}^{2}}),

(3)

where $Cov (X_{j}, X_{j'})$ is the covariance between observed scores for items $j$ and $j'$ . When test items are essentially $τ$ -equivalent, coefficient alpha is equal to $ρ$ (Novick & Lewis, 1967).

Reliability Based on SEM

In the framework of SEM, item scores can be represented as follows (Green & Yang, 2009; Raykov & Shrout, 2002):

X_{j}^{*} = λ_{1 j} η_{1} + λ_{2 j} η_{2} + \dots + λ_{Mj} η_{M} + e_{j},

(4)

where $X_{j}^{*}$ is a continuous score for item $j$ , M is the number of latent factors, $η_{m}$ $(m = 1, \dots, M)$ are latent factors weighted by corresponding factor loadings $λ_{mj}$ , and $e_{j}$ is a measurement error term. On the right side of Equation 4, $\sum_{m = 1}^{M} λ_{mj} η_{m}$ can be considered as a true score and $e_{j}$ can be considered as an error score in CTT. When $X_{j}^{*} (j = 1, \dots, J)$ are continuous, parameters in Equation 4 are typically estimated by maximum likelihood estimation (MLE) with the assumption that observed variables are normally distributed (Bollen, 1989). Suppose $X^{*}$ and $T$ are the sum of observed scores and of true scores, respectively. Then $X^{*} = \sum_{j = 1}^{J} X_{j}^{*}$ and $T = \sum_{j = 1}^{J} \sum_{m = 1}^{M} λ_{mj} η_{m}$ . The linear SEM reliability $ρ_{lin}$ is calculated as the ratio of true sum score variance to observed sum score variance as in Equation 2:

ρ_{lin} = \frac{σ_{T}^{2}}{σ_{X^{*}}^{2}} = \frac{Var (\sum_{j = 1}^{J} \sum_{m = 1}^{M} λ_{mj} η_{m})}{Var (\sum_{j = 1}^{J} X_{j}^{*})} .

(5)

In this case, $ρ_{lin}$ measures the proportion of observed sum score variance that is attributed to the latent factors, $η_{1}, η_{2}, \dots, η_{M}$ (Bollen, 1989). This linear SEM reliability, $ρ_{lin}$ , is also equivalent to coefficient omega (McDonald, 1985, 1999).

When the observed data are ordinal categorical, fitting linear SEM models such as Equation 4 using the MLE method is not desirable as categorical data violate the assumption of the MLE method, and the MLE provides inflated chi-square estimates and attenuated factor loadings (Bollen, 1989). To address this problem, the authors consider the observed categorical scores $(X_{j})$ as produced from underlying continuous variables $(X_{j}^{*})$ , and that a linear SEM model holds for $X_{j}^{*}$ as in Equation 4 (Bollen, 1989; Finney & DiStefano, 2006). We assume a nonlinear relationship between $X_{j}$ and $X_{j}^{*}$ :

X_{j} = {\begin{matrix} C_{j} - 1, & if & X_{j}^{*} > v_{C_{j} - 1} \\ ⋮ & ⋮ \\ 1, & if & v_{1} < X_{j}^{*} \leq v_{2} \\ 0, & if & X_{j}^{*} \leq v_{1} \end{matrix},

(6)

where $C_{j}$ is the number of categories for $X_{j}$ and the $v_{i} (i = 1, 2, \dots, C_{j} - 1)$ are the category thresholds. This allows for different numbers of ordered score categories for each item $j$ . If $X_{j}^{*}$ is less than $v_{1}$ , $X_{j}$ is equal to 0; for $v_{1} < X_{j}^{*} \leq v_{2}$ , $X_{j}$ is equal to 1; and if $X_{j}^{*}$ is above $v_{C_{j} - 1}$ , $X_{j}$ is equal to $C_{j} - 1$ .

To estimate the reliability for the nonlinear measurement model, the correlation between two parallel tests is used:

ρ_{X \tilde{X}} = \frac{Cov (X, \tilde{X})}{\sqrt{Var (X) Var (\tilde{X})}},

(7)

where $X$ and $\tilde{X}$ are observed sum scores across $J$ items from two parallel tests, which indicate (a) the same latent factors affect the items from the two tests and (b) the variance of errors and factor loadings are the same for corresponding items for both tests. For categorical items on parallel tests, the thresholds are the same for corresponding items (Bollen, 1989; Green & Yang, 2009). For example, for item $j$ , the underlying continuous scores from two parallel tests using a CFA model with $M$ latent factors can be expressed as follows:

{\begin{matrix} X_{j}^{*} = λ_{1 j} η_{1} + λ_{2 j} η_{2} + \dots + λ_{Mj} η_{M} + e_{j} \\ {\tilde{X}}_{j}^{*} = λ_{1 j} η_{1} + λ_{2 j} η_{2} + \dots + λ_{Mj} η_{M} + {\tilde{e}}_{j} \end{matrix},

(8)

where $λ_{mj}$ is a factor loading for factor $η_{m}$ , and $e_{j}$ and ${\tilde{e}}_{j}$ are error terms for $X_{j}^{*}$ and ${\tilde{X}}_{j}^{*}$ , respectively.

The numerator of $ρ_{X \tilde{X}}$ in Equation 7 is a covariance between sum scores and can be presented as a function of covariances between two items from the parallel tests:

Cov (X, \tilde{X}) = Cov (\sum_{j = 1}^{J} X_{j}, \sum_{j' = 1}^{J} {\tilde{X}}_{j'}) = \sum_{j = 1}^{J} \sum_{j' = 1}^{J} Cov (X_{j}, {\tilde{X}}_{j'}) .

(9)

Suppose item j has $C_{j}$ categories, item $j'$ has $C_{j'}$ categories, and the vectors of underlying continuous variables $X^{*} = (X_{1}^{*}, \dots, X_{J}^{*})$ and ${\tilde{X}}^{*} = ({\tilde{X}}_{1}^{*}, \dots, {\tilde{X}}_{J}^{*})$ follow a multivariate normal distribution with variances of 1. Online Appendix A shows that $Cov (X_{j}, {\tilde{X}}_{j'})$ can be represented as

Cov (X_{j}, {\tilde{X}}_{j'}) = \sum_{k = 1}^{C_{j} - 1} \sum_{l = 1}^{C_{j'} - 1} Φ_{2} (v_{j_{k}}, h_{j_{l}^{'}}; ρ_{X_{j}^{*} X_{j'}^{*}}) - \sum_{k = 1}^{C_{j} - 1} Φ_{1} (v_{j_{k}}) \sum_{l = 1}^{C_{j'} - 1} Φ_{1} (h_{j_{l}^{'}}),

(10)

where ${v_{j_{1}}, \dots, v_{j_{C_{j} - 1}}}$ and ${h_{j_{1}^{'}}, \dots, h_{j_{C_{j'} - 1}^{'}}}$ are thresholds for items $j$ and $j'$ , respectively; $Φ_{1} (v_{j_{k}})$ is the cumulative univariate normal distribution function of threshold $v_{j_{k}}$ ; and $Φ_{2} (v_{j_{k}}, h_{j_{l}^{'}}; ρ_{X_{j}^{*} X_{j'}^{*}})$ is the cumulative bivariate normal distribution function of thresholds $v_{j_{k}}$ and $h_{j_{l}^{'}}$ with correlation $ρ_{X_{j}^{*} X_{j'}^{*}}$ , where $ρ_{X_{j}^{*} X_{j'}^{*}}$ is the correlation between two underlying continuous variables. If $X_{j}^{*}$ and $X_{j'}^{*}$ consist of $M$ latent factors as in Equation 8, then $ρ_{X_{j}^{*} X_{j'}^{*}}$ can be represented as

ρ_{X_{j}^{*} X_{j'}^{*}} = \sum_{m = 1}^{M} \sum_{m' = 1}^{M} λ_{mj} λ_{m' j'} ρ_{η_{m} η_{m'}},

(11)

where $ρ_{η_{m} η_{m'}}$ is the correlation between $η_{m}$ and $η_{m'}$ . The authors specifically indicate this correlation derived from a model as $ρ_{M_{jj'}}$ to avoid confusion.

The denominator of $ρ_{X \tilde{X}}$ in Equation 7 is

\sqrt{Var (X) Var (\tilde{X})} = Var (X)

(12)

because $X_{j}$ and ${\tilde{X}}_{j}$ are from the parallel tests, and $Var (X)$ , the variance of the summed score $X$ , is equal to the sum of covariances between all pairs of items in the test:

\begin{matrix} Var (X) = Var (\sum_{j = 1}^{J} X_{j}) = \sum_{j = 1}^{J} \sum_{j' = 1}^{J} Cov (X_{j}, X_{j'}) \\ = \sum_{J = 1}^{J} \sum_{j' = 1}^{J} (\sum_{k = 1}^{C_{j} - 1} \sum_{l = 1}^{C_{j'} - 1} Φ_{2} (v_{j_{k}}, h_{j_{l}^{'}}; ρ_{X_{j}^{*} X_{j'}^{*}}) - \sum_{k = 1}^{C_{j} - 1} Φ_{1} (v_{j_{k}}) \sum_{l = 1}^{C_{j'} - 1} Φ_{1} (h_{j_{l}^{'}})) . \end{matrix}

(13)

The nonlinear reliability coefficient using the SEM framework is expressed as

ρ_{non} = \frac{\sum_{j = 1}^{J} \sum_{j' = 1}^{J} [\sum_{k = 1}^{C_{j} - 1} \sum_{l = 1}^{C_{j'} - 1} Φ_{2} (v_{j_{k}}, h_{j_{l}^{'}}; ρ_{M_{jj'}}) - \sum_{k = 1}^{C_{j} - 1} Φ_{1} (v_{j_{k}}) \sum_{l = 1}^{C_{j'} - 1} Φ_{1} (h_{j_{l}^{'}})]}{\sum_{j = 1}^{J} \sum_{j' = 1}^{J} [\sum_{k = 1}^{C_{j} - 1} \sum_{l = 1}^{C_{j'} - 1} Φ_{2} (v_{j_{k}}, h_{j_{l}^{'}}; ρ_{X_{j}^{*} X_{j'}^{*}}) - \sum_{k = 1}^{C_{j} - 1} Φ_{1} (v_{j_{k}}) \sum_{l = 1}^{C_{j'} - 1} Φ_{1} (h_{j_{l}^{'}})]} .

(14)

This can be seen as an extension of Equation 21 in Green and Yang (2009). From Equation 14, we can estimate the internal consistency reliability of a test consisting of items with different numbers of ordered response categories by fitting the data using a nonlinear SEM model and replacing parameters in Equation 14 with corresponding estimates.

Simulation Study

A simulation study is presented in this section to investigate the performance of $ρ_{non}$ , the proposed nonlinear SEM reliability coefficient, using two types of simulated data sets: data generated from a one-factor model and data generated from a bifactor model. A bifactor model was considered to investigate the performance of $ρ_{non}$ for a test having some factors due to clusters of items such as items sharing the same passage. In this study, the authors evaluated the performance of $ρ_{non}$ in the context of a test with different numbers of response categories and for a test with the same number of response categories. For comparison purposes, coefficient alpha ( $α$ ) and population reliability coefficients for observed sum scores $(ρ_{X \tilde{X}})$ were also presented.

Simulation Study Design

Factor structure

Four factor structures were simulated using one-factor or bifactor models. The four structures are summarized in Table 1. Models 1 and 2 are based on a one-factor model and Models 3 and 4 are based on a bifactor model with three group factors.

Table 1.

Factor Structures Considered in the Simulation Study.

	Number of items	Number of group factors	Number of items per factor
Model 1	9	NA	NA
Model 2	18	NA	NA
Model 3	9	3	3
Model 4	18	3	6

For Models 3 and 4, the correlations between latent factors were all set to 0. For all models, errors were assumed independent. Factor loadings $(λ)$ were all set to 0.7 for Models 1 and 2. For Models 3 and 4, factor loadings for the general factor were all 0.7, and two sets of factor loadings for the three group factors were considered: all 0.4 and all 0.6.

Types of tests and numbers of ordered categories

Two types of tests were simulated: One type of test consisted of items with the same number of categories, and the other type of test consisted of items with uneven numbers of categories. Tests with either two or five ordered categories (labeled as conditions C2 and C5 in the simulation, respectively) were simulated for the first type. For the second type, a combination of two- and five-category items (labeled as condition C25) was generated to compare with reliability estimates from the first type of tests. In the second type of tests, every third item was simulated to have five ordered categories, and the rest were simulated to have two ordered categories.

Distribution of underlying continuous variables

The distribution of the underlying continuous variables $(X_{j}^{*} s)$ was generated using a multivariate normal distribution with variances of 1. Covariances between variables were determined by the factor models (Model 1-Model 4). Suppose the vector of underlying continuous variables $X^{*} = (X_{1}^{*}, \dots, X_{J}^{*})$ for $J$ items follows a multivariate normal distribution:

X^{*} ~ N (0, Σ),

(15)

where 0 is a $J \times 1$ vector of 0, and Σ is the $J \times J$ covariance matrix of $X^{*}$ with diagonal elements of 1. To be specific, the underlying continuous score on item $j$ for examinee $i$ , $x_{ij}^{*}$ was generated to have a variance of one as in Equation 16 (Bandalos & Enders, 1996; Bernstein & Teng, 1989; Flora & Curran, 2004; Green & Yang, 2009; Yang & Green, 2015).

x_{ij}^{*} = λ_{j}^{T} η_{i} + \sqrt{(1 - λ_{j}^{T} Var (η_{i}) λ_{j})} ε_{ij},

(16)

where $λ_{j}$ is a vector of factor loadings for item $j$ , $η_{i}$ is a vector of latent factors, $ε_{ij}$ is an error score, and $ε_{ij}$ is uncorrelated with $η_{i}$ . The error scores were assumed to have a mean of 0 and a variance of 1. Off diagonal elements in the covariance matrix in Equation 15 were determined with Equation 16 using the parameters of the relevant model.

Distribution of thresholds

Three sets of thresholds were used to generate the categorical data: (a) normal, (b) moderate skew, and (c) mixed skew. To generate the normally distributed data for two- and five-response categories, the sets of thresholds, {0} and {–1.645, –0.643, 0.643, 1.645}, were used, respectively, to transform the underlying continuous variables. Similarly, for the moderate skew condition, {0.7} and {–0.050, 0.772, 1.341, 1.881} were used to generate ordered variables with two- and five-categories, respectively. In the case of the mixed skew condition, every third item response was generated with the negative moderate skew thresholds and the rest were generated with the positive moderate skew thresholds.

For the five-response category tests, the thresholds used to generate the normal and moderate skew distributions were taken from Muthén and Kaplan (1985). The thresholds for the two-response categories were determined to have the same skewness as the corresponding five-response category condition. The skewness values of data sets (Doane & Seward, 2011) from the normal and moderate skew conditions were about 0 and 1.2, respectively.

Data Generation

In total, there were 54 conditions: 18 conditions with a one-factor model (18 = 2 factor structures × 3 distributions of thresholds × 3 sets of item response categories) and 36 conditions with a bifactor model (36 = 2 factor structures × 2 group factor loadings × 3 distributions of thresholds × 3 sets of item response categories). For each of the conditions, the authors first generated population data, which consisted of 100,000 observations. From the 100,000 observations, the authors randomly sampled 500 observations without replacement, and replicated this 100 times. As sample size was not a focus of this study, they sampled 500 observations to avoid estimation problems due to small sample size. In addition, to calculate the population reliability, $ρ_{X \tilde{X}}$ , which requires sum scores from two parallel tests, a second set of population data is generated. The first and second population data share the same underlying continuous true scores as presented in Equation 8. The correlation between the observed sum scores from the parallel tests was treated as the population reliability coefficient.

Data Analysis

A one-factor model and a bifactor model were fit to the corresponding generated data. For each condition, $ρ_{non}$ and $α$ were obtained by using the model parameter estimates and the generated data, respectively. Model parameters were estimated by the method of weighted least squares with mean and variance adjustment (WLSMV; Muthén, du Toit, & Spisic, 1997; Muthén & Muthén, 1998-2015), as implemented in Mplus 7.4 (Muthén & Muthén, 1998-2015). Polychoric correlations were also estimated using Mplus 7.4. Based on the parameter estimates, $ρ_{non}$ for each replication was calculated using R (R Core Team, 2017). The R code for $ρ_{non}$ is provided in Online Appendix B. In addition, $α$ for each replication was obtained by using the R package psych (Revelle, 2017).

Simulation Study Results

The authors first examined whether models converged. Some estimation issues occurred for the bifactor models (Models 3 and 4), especially when the number of response categories was two and data had mixed skewness. The estimation issues included that a residual covariance matrix was not positive definite or residual variances were negative. To make meaningful comparison, the data sets that had no estimation issue were used to calculate $ρ_{non}$ and $α$ . The numbers of converged replications for Models 3 and 4 are provided in Table C.1 in Online Appendix C.

Table 2 presents the means of reliability coefficients for the one-factor models (i.e., Models 1 and 2). As presented in Table 2, the C2 condition had the lowest $ρ_{X \tilde{X}}$ and $ρ_{non}$ , ranging from 0.77 to 0.90. The C5 condition had the highest $ρ_{X \tilde{X}}$ and $ρ_{non}$ , ranging from 0.86 to 0.94. The two reliability coefficients for the mixed number of categories conditions (i.e., condition C25) ranged from 0.80 to 0.92, and were between those for C2 and C5. The values of $ρ_{non}$ were all close to the values of $ρ_{X \tilde{X}}$ , and the standard deviations of $ρ_{non}$ were all less than or equal to 0.02. For some conditions, $ρ_{non}$ was .01 higher than $ρ_{X \tilde{X}}$ . These results are consistent with the results in Yang and Green (2015) for items having the same number of ordered response categories.

Table 2.

The Means of Reliability Coefficients and Their Standard Deviations for Data From the One-Factor Models.

			Normal			Moderate skewness			Mixed skewness
M	Gr		C2	C5	C25	C2	C5	C25	C2	C5	C25
1	NA	$ρ_{X \tilde{X}}$	0.81	0.87	0.83	0.79	0.87	0.81	0.77	0.86	0.80
		$ρ_{non}$	0.82(0.01)	0.88(0.02)	0.84(0.01)	0.80(0.01)	0.87(0.01)	0.82(0.01)	0.79(0.01)	0.86(0.01)	0.80(0.01)
		$α$	0.81(0.01)	0.87(0.01)	0.82(0.01)	0.79(0.02)	0.87(0.01)	0.78(0.01)	0.76(0.01)	0.85(0.01)	0.75(0.01)
2	NA	$ρ_{X \tilde{X}}$	0.90	0.93	0.91	0.88	0.93	0.90	0.87	0.92	0.89
		$ρ_{non}$	0.90(0.01)	0.94(0.00)	0.92(0.01)	0.89(0.01)	0.93(0.01)	0.90(0.01)	0.88(0.01)	0.93(0.00)	0.89(0.01)
		$α$	0.90(0.01)	0.93(0.00)	0.90(0.01)	0.88(0.01)	0.93(0.01)	0.88(0.01)	0.87(0.01)	0.92(0.00)	0.87(0.01)

Note. M and Gr refer to Model and Group factor loading, respectively; C refers to the condition for the number of response categories; and $ρ_{X \tilde{X}}$ , $ρ_{non}$ , and $α$ refer to the population reliability for observed sum scores, nonlinear SEM reliability, and coefficient alpha, respectively. SEM = structural equation modeling.

Coefficient $α$ in Table 2 had the lowest values, when the category condition was either C2 or C25, and had the highest values for the C5 condition. Coefficient $α$ values under the one-factor condition were nearly identical with the corresponding $ρ_{X \tilde{X}}$ , when items had the same number of response categories (i.e., conditions C2 and C5). In contrast, $α$ was slightly lower than $ρ_{X \tilde{X}}$ , when the test included items with uneven numbers of response categories (i.e., condition C25), and the responses had skewed distributions. When the observed scores were generated from the model with nine items with an uneven number of categories (i.e., Model 1 with C25) and the mixed skew condition, the difference between $α$ and the $ρ_{X \tilde{X}}$ was 0.05. The standard deviations of $α$ were less than or equal to 0.02.

Table 3 presents the means of reliability coefficients for the bifactor models (Models 3 and 4). The performance of $ρ_{non}$ was similar to that under the one-factor models in that values for $ρ_{non}$ were close to the corresponding values for $ρ_{X \tilde{X}}$ . The differences between these two coefficients were generally less than or equal to .01. As was observed for the one-factor model, the C2 condition had the lowest values of $ρ_{X \tilde{X}}$ and $ρ_{non}$ , ranging from 0.82 to 0.96. The C5 condition had the highest values, ranging from 0.90 to 0.98. Coefficient $α$ had the lowest values when the category condition was either C2 or C25. Under the bifactor model, $α$ values were generally lower than $ρ_{X \tilde{X}}$ , but similar to $ρ_{X \tilde{X}}$ , when the number of items was 18 and the responses were from the bifactor model with the group factor loadings of 0.4. The differences between $α$ and $ρ_{X \tilde{X}}$ values ranged from 0.01 to 0.14. The standard deviations of $ρ_{non}$ and $α$ were all less than or equal to 0.02.

Table 3.

The Means of Reliability Coefficients and Their Standard Deviations for Data From the Bifactor Models.

			Normal			Moderate skewness			Mixed skewness
M	Gr		C2	C5	C25	C2	C5	C25	C2	C5	C25
3	0.4	$ρ_{X \tilde{X}}$	0.86	0.91	0.88	0.84	0.91	0.87	0.82	0.90	0.85
		$ρ_{non}$	0.86(0.01)	0.91(0.01)	0.89(0.01)	0.85(0.01)	0.91(0.01)	0.87(0.01)	0.84(0.01)	0.90(0.01)	0.86(0.02)
		$α$	0.83(0.01)	0.89(0.01)	0.84(0.01)	0.81(0.01)	0.88(0.01)	0.80(0.01)	0.78(0.01)	0.86(0.01)	0.77(0.01)
3	0.6	$ρ_{X \tilde{X}}$	0.92	0.95	0.93	0.91	0.96	0.93	0.89	0.95	0.92
		$ρ_{non}$	0.92(0.01)	0.95(0.00)	0.93(0.01)	0.91(0.01)	0.96(0.00)	0.93(0.01)	0.89(0.01)	0.95(0.00)	0.93(0.01)
		$α$	0.86(0.01)	0.90(0.01)	0.85(0.01)	0.85(0.01)	0.90(0.01)	0.83(0.01)	0.79(0.01)	0.88(0.01)	0.78(0.01)
4	0.4	$ρ_{X \tilde{X}}$	0.92	0.95	0.94	0.91	0.95	0.93	0.90	0.95	0.92
		$ρ_{non}$	0.93(0.00)	0.96(0.00)	0.94(0.00)	0.92(0.01)	0.95(0.00)	0.93(0.01)	0.91(0.01)	0.95(0.00)	0.93(0.01)
		$α$	0.91(0.01)	0.94(0.00)	0.91(0.00)	0.90(0.01)	0.94(0.00)	0.90(0.01)	0.88(0.01)	0.93(0.00)	0.88(0.01)
4	0.6	$ρ_{X \tilde{X}}$	0.96	0.98	0.97	0.95	0.98	0.96	0.94	0.97	0.96
		$ρ_{non}$	0.96(0.00)	0.98(0.00)	0.97(0.00)	0.96(0.00)	0.98(0.00)	0.97(0.00)	0.95(0.00)	0.98(0.00)	0.96(0.00)
		$α$	0.93(0.00)	0.95(0.00)	0.93(0.00)	0.92(0.01)	0.95(0.00)	0.91(0.01)	0.90(0.01)	0.94(0.00)	0.89(0.01)

The one-factor models and the bifactor models fit the corresponding data well under each condition: Across all replications the comparative fit index (CFI) were all higher than .98, and the root mean square error of approximation (RMSEA) were all smaller than .06 with mean values for each condition ranging from .01 to .02.

Conclusions From Simulation Study

The values of $ρ_{non}$ were very close to the values of $ρ_{X \tilde{X}}$ across all the conditions. The differences between these two coefficients across all conditions were mostly less than 0.01. $ρ_{non}$ and $ρ_{X \tilde{X}}$ both had the highest values under the C5 condition, and had lowest values under the C2 condition. Both reliability coefficients for the C25 condition were placed between the reliability coefficients for C2 and C5. For some conditions, the values of $ρ_{non}$ were slightly higher than the values of $ρ_{X \tilde{X}}$ . Coefficient $α$ was close to or the same as the population reliability in conditions with the same number of categories from the one-factor model, but tended to be lower than $ρ_{X \tilde{X}}$ when items had the mixed numbers of response categories or when data were generated from the bifactor model.

Illustration With Empirical Data

In this section, the authors illustrate the performance of the nonlinear SEM reliability, and compare it with that of coefficient alpha on a set of real data. The data were from an National Science Foundation (NSF)-funded project focusing on teaching science inquiry practices for students in Grades 6 to 10.

Test

The test consisted of 27 constructed response items designed to measure students’ use of academic language and understanding of science inquiry practices. Seventeen of the 27 items were scored with a 2-point rubric; the remaining 10 items were scored with a 3-point rubric. Ten items had skewness larger than 0.7, and two items had skewness less than –0.7. Descriptive statistics for each item are provided in Table C.2 in Online Appendix C.

Sample

The sample consisted of 906 students across Grades 6 to 10: 260 (29.05%) students were in the sixth grade, 209 (23.35%) students were in the seventh grade, 222 (24.80%) students were in the eighth grade, 80 (8.94%) students were in the ninth grade, and 124 (13.85%) students were in the 10th grade. There were 11 students for whom grade information was not available.

Results

A one-factor model and a bifactor model were fit to the data. The one-factor solution did not provide good model fit: $χ^{2}$ (324) = 4,662.22, $χ^{2} (324) / (df = 324) = 14.39$ , CFI = .93, and RMSEA = .12. For this solution, $ρ_{non}$ was 1.04, which is an unacceptable value for reliability. Also, this one-factor solution could not be used for estimating reliability because the model did not fit the data. Because of model misspecification, the correlations generated from the model $(ρ_{M_{jj'}})$ were notably different from the sample polychoric correlations $(ρ_{X_{j}^{*} X_{j'}^{*}})$ , resulting in the reliability estimate larger than 1.

The bifactor model with five group factors was next fit to the data. This solution fit the data better: $χ^{2} (297) = 841.540$ , $χ^{2} (297) / (df = 297) = 2.83$ , CFI = .991, and RMSEA = .045. The five group factors were consistent with the construction of the test as each factor appeared to represent one of the five science inquiry practices measured by the test. For this bifactor model solution, the $ρ_{non} = . 96$ .

Coefficient alpha for these data was .92, which was smaller than the nonlinear SEM reliability under the bifactor model solution. The uneven number of response categories might be one of the reasons for having the lower value for coefficient alpha than the nonlinear SEM reliability estimate. Factor loading estimates of the bifactor model varied across the 27 items (ranging from 0.18 to 0.79), which clearly suggests violation of $τ$ -equivalency.

Based on the results from this study, the nonlinear SEM approach can provide an alternative reliability estimation method in that it is close to the population reliability when items are not $τ$ -equivalent and when items have uneven numbers of response categories.

Discussion

In this study, a generalized nonlinear SEM reliability coefficient was designed to provide internal consistency reliability estimates for tests with items scored with equal or unequal numbers of ordered categories. A simulation study evaluated the performance of this coefficient compared to coefficient alpha and population reliability for observed sum scores. Results indicated that the nonlinear SEM reliability estimates and the population reliability values were close across all the conditions. The nonlinear SEM reliability coefficient and the population reliability coefficient always had the lowest values under the two-response-category condition and had the highest values under the five-response-category condition. The two reliability coefficients for items with mixed numbers of response categories fell in-between. This performance can be expected as having more response categories indicates more information. Furthermore, items with a mixed number of categories might be assumed to have more information than those having the smallest number of categories among the mixed categories and also to have less information than those having the largest number of categories. The nonlinear SEM reliability successfully captured this tendency by estimating reliability close to the population reliability. Results of this study suggest that the nonlinear SEM reliability is a flexible reliability estimate and can provide an accurate estimate of the population reliability for situations in which data consisted of either the same or different numbers of ordered categories and the data have either an unidimensional structure or have some factors due to clusters of items.

In general, coefficient alpha values were close to the corresponding population reliability coefficient for simulation conditions under the one-factor model and for equal numbers of categories (i.e., conditions C2 and C5). In addition, continuous scores that underlay the observed categorical scores were essentially $τ$ -equivalent under the one-factor model. In generating tests with items with different numbers of categories, different thresholds were applied to the underlying continuous scores of the items to transform the data into categorical scores. Results suggest that this may have resulted in tests which were no longer essentially $τ$ -equivalent. In this case, under the one-factor model and with items having different numbers of categories, coefficient alpha tended to be lower than the population reliability, especially when the scores were skewed. Coefficient alpha was also lower than the corresponding population reliability under the bifactor model conditions (Models 3 and 4). These results were expected as the items generated from the bifactor model were no longer essentially $τ$ -equivalent. Cronbach (1951) also noted that when a test has a general factor related to all items with group factors related to part of the items, coefficient alpha is slightly higher than the proportion of variance due to the general factor and lower than the population reliability, which is the proportion of variance due to all common factors. In our simulation study, coefficient alpha, the nonlinear SEM reliability, and the population reliability performed similarly for bifactor models, especially in the cases when the number of items was 18 and the responses were generated with relatively low group factor loadings (in this study, we used 0.4).

The simulation study in this article was conducted under the conditions of correctly specified factor models. When the factor model is misspecified, as was evident in the empirical study by the poor model fit indices for the unidimensional model, the nonlinear SEM reliability may fail to accurately estimate reliability. This is because misspecification can lead to incorrect parameter estimation, resulting in incorrect nonlinear reliability estimates. Second, the simulation study was conducted with the assumption of multivariate normality of the latent variables. For this assumption, Yang and Green (2015) found that the nonlinear SEM reliability was robust to the modest violation of normality assumption under the condition of the same number of response categories. Further study would be useful on the impact of model misspecification and the violation of the normality assumption of underlying latent variables on the nonlinear SEM reliability.

Supplemental Material

Online_appendix_1 – Supplemental material for Reliability for Tests With Items Having Different Numbers of Ordered Categories

Supplemental material, Online_appendix_1 for Reliability for Tests With Items Having Different Numbers of Ordered Categories by Seohyun Kim, Zhenqiu Lu and Allan S. Cohen in Applied Psychological Measurement

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was supported in part by the National Science Foundation (NSF) grant at the University of Georgia under Grant No. 1316398.

ORCID iD

Seohyun Kim

Supplemental Material

Supplemental material is available for this article online.

References

Bandalos

D. L.

Enders

C. K.

(1996). The effects of nonnormality and number of response categories on reliability. Applied Measurement in Education, 9, 151-160.

Bentler

P. M.

(2009). Alpha, dimension-free, and model-based internal consistency reliability. Psychometrika, 74, 137-143.

Bernstein

I. H.

Teng

(1989). Factoring items and factoring scales are different: Spurious evidence for multidimensionality due to item categorization. Psychological Bulletin, 105, 467-477.

Bollen

K. A.

(1989). Structural equations with latent variables. New York, NY: Wiley.

Cho

Kim

(2015). Cronbach’s coefficient alpha: Well known but poorly understood. Organizational Research Methods, 18, 207-230.

Cronbach

L. J.

(1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16, 297-334.

Cronbach

L. J.

Shavelson

R. J.

(2004). My current thoughts on coefficient alpha and successor procedures. Educational and Psychological Measurement, 64, 391-418.

DeMars

C. E.

(2006). Application of the bi-factor multidimensional item response theory model to testlet-based tests. Journal of Educational Measurement, 43, 145-168.

Doane

D. P.

Seward

L. E.

(2011). Measuring skewness: A forgotten statistic? Journal of Statistics Education, 19(2), 1-18.

10.

Finney

S. J.

DiStefano

(2006). Non-normal and categorical data in structural equation modeling. In Hancock

G. R.

Mueller

R. O.

(Eds.), Structural equation modeling: A second course (pp. 269-314). Greenwich, CT: Information Age.

11.

Flora

D. B.

Curran

P. J.

(2004). An empirical evaluation of alternative methods of estimation for confirmatory factor analysis with ordinal data. Psychological Methods, 9, 466-491.

12.

Green

S. B.

Lissitz

R. W.

Mulaik

S. A.

(1977). Limitations of coefficient alpha as an index of test unidimensionality. Educational and Psychological Measurement, 37, 827-838.

13.

Green

S. B.

Yang

(2009). Reliability of summed item scores using structural equation modeling: An alternative to coefficient alpha. Psychometrika, 74, 155-167.

14.

Green

S. B.

Yang

(2015). Evaluation of dimensionality in the assessment of internal consistency reliability: Coefficient alpha and omega coefficients. Educational Measurement: Issues and Practice, 34(4), 14-20.

15.

Hyland

Boduszek

Dhingra

Shevlin

Egan

(2014). A bifactor approach to modelling the Rosenberg Self Esteem Scale. Personality and Individual Differences, 66, 188-192.

16.

Janus

Offord

D. R.

(2007). Development and psychometric properties of the Early Development Instrument (EDI): A measure of children’s school readiness. Canadian Journal of Behavioural Science, 39, 1-22.

17.

Komorita

S. S.

Graham

W. K.

(1965). Number of scale points and the reliability of scales. Educational and Psychological Measurement, 25, 987-995.

18.

Bolt

D. M.

(2006). A comparison of alternative models for testlets. Applied Psychological Measurement, 30, 3-21.

19.

Lissitz

R. W.

Green

S. B.

(1975). Effect of the number of scale points on reliability: A Monte Carlo approach. Journal of Applied Psychology, 60, 10-13.

20.

Lord

F. M.

Novick

M. R.

(1968). Statistical theories of mental test scores. Reading, MA: Addison-Wesley.

21.

Lozano

L. M.

García-Cueto

Muñiz

(2008). Effect of the number of response categories on the reliability and validity of rating scales. Methodology, 4(2), 73-79.

22.

McAbee

S. T.

Oswald

F. L.

Connelly

B. S.

(2014). Bifactor models of personality and college student performance: A broad versus narrow view. European Journal of Personality, 28, 604-619.

23.

McDonald

R. P.

(1985). Factor analysis and related methods. Hillsdale, NJ: Lawrence Erlbaum.

24.

McDonald

R. P.

(1999). Test theory: A unified treatment. Mahwah, NJ: Lawrence Erlbaum.

25.

Miller

M. B.

(1995). Coefficient alpha: A basic introduction from the perspectives of classical test theory and structural equation modeling. Structural Equation Modeling, 2, 255-273.

26.

Muthén

du Toit

S. H. C.

Spisic

(1997). Robust inference using weighted least squares and quadratic estimating equations in latent variable modeling with categorical and continuous outcomes. Retrieved from http://www.statmodel.com/bmuthen/articles/Article_075.pdf

27.

Muthén

Kaplan

(1985). A comparison of some methodologies for the factor analysis of non-normal Likert variables. British Journal of Mathematical and Statistical Psychology, 38, 171-189.

28.

Muthén

L. K.

Muthén

B. O.

(1998-2015). Mplus user’s guide (7th ed.). Los Angeles, CA: Author.

29.

Novick

Lewis

(1967). Coefficient alpha and the reliability of composite measurements. Psychometrika, 32, 1-13.

30.

Partnership for Assessment of Readiness for College and Careers. (2016). Final technical report for 2015 administration. Retrieved from https://www.state.nj.us/education/assessment/district/PARCCTechReport15.pdf

31.

Raykov

(1997). Estimation of composite reliability for congeneric measures. Applied Psychological Measurement, 21, 173-184.

32.

Raykov

Shrout

P. E.

(2002). Reliability of scales with general structure: Point and interval estimation using a structural equation modeling approach. Structural Equation Modeling, 9, 195-212.

33.

R Core Team. (2017). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing. Available from http://www.R-project.org/

34.

Revelle

W. R.

(2017). psych: Procedures for personality and psychological research (R Package Version 1.8.4). Retrieved from https://CRAN.R-project.org/package=psych

35.

Rijmen

(2010). Formal relations and an empirical comparison among the bi-factor, the testlet, and a second-order multidimensional IRT model. Journal of Educational Measurement, 47, 361-372.

36.

Sijtsma

(2009). On the use, the misuse, and the very limited usefulness of Cronbach’s alpha. Psychometrika, 74, 107-120.

37.

Smarter Balanced Assessment Consortium. (2016). Smarter Balanced Assessment Consortium: 2013-14 technical report. Retrieved from https://portal.smarterbalanced.org/library/en/2013-14-technical-report.pdf

38.

Weng

L. J.

(2004). Impact of the number of response categories and anchor labels on coefficient alpha and test-retest reliability. Educational and Psychological Measurement, 64, 956-972.

39.

Yang

Green

S. B.

(2010). A note on structural equation modeling estimates of reliability. Structural Equation Modeling, 17(1), 66-81.

40.

Yang

Green

S. B.

(2011). Coefficient alpha: A reliability coefficient for the 21st century? Journal of Psychoeducational Assessment, 29, 377-392.

41.

Yang

Green

S. B.

(2015). Evaluation of structural equation modeling estimates of reliability for scales with ordered categorical items. Methodology, 11(1), 23-34.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.35 MB