The Reliability and Precision of Total Scores and IRT Estimates as a Function of Polytomous IRT Parameters and Latent Trait Distribution

Equations for the Reliability of CTT and IRT Estimates of θ

Let θ represent a latent variable and u _i be an observed polytomous response, where i indexes items ( i = 1 , … , I ). The observed polytomous response for item i can be expressed as a function of a true score ( E { u i | θ } ) and random error ( e i ) , such that u i = E { u i | θ } + e i . That is, the observed u _i equals an item true score E { u i | θ } (May & Nicewander, 1994), which is a nonlinear function of θ, plus an error, e _i . Let j index category thresholds ( j = 1 , … , J ) and J+1 is the corresponding number of categories for item i. That is, j is used to index categories as well (e.g., J+1 = 4 implies that u _i has four categories). This article assumes that researchers code u _i using integers from 1 to J+1. Several polytomous models exist to describe the relationship between θ and the chance that u _i equals one of J+1 categories, which include the graded response (Molenaar, Dolan, & De Boeck, 2012; Muraki, 1990; Samejima, 1969), partial credit (Masters, 1982; Muraki, 1992, 1993), and rating scale (Andrich, 1978a, 1978b) models. The derivations in this article use Muraki’s (1990) modified graded response model,

P ij * ( u i > j | θ ) = { 1 , j = 0 1 1 + exp [ − α i ( θ − ( b i − c j ) ) ] , 0 < j ≤ J 0 , j = J + 1 ,

where b _i and α i are the item difficulty and discrimination parameters, respectively, and c _j is the jth threshold, which is equal for all I items. Equation 1 represents the chance that u i > j given θ and the probability that item i equals the jth category is

P ij ( u i = j | θ ) = P ij * ( u i > j | θ ) − P ij + 1 * ( u i > j + 1 | θ ) .

Note that Muraki’s model was chosen because the form of Equation 1 is easier to manipulate analytically and the derivations of expressions for derivatives are less cumbersome. Muraki’s model assumes that the J c_j are constant across items and are equally spaced. The goal of this article is not to estimate abilities or item parameters with Muraki’s model, and these assumptions can be relaxed to evaluate the impact of unequal item thresholds on the reliability and precision of x and θ ^ . In fact, the new expressions are applicable for any polytomous model, and Muraki’s model is only used for computational examples. Furthermore, the aforementioned models tend to yield scores that are highly correlated (Embretson & Reise, 2000), so we should not anticipate the results would change significantly if the partial credit or rating scale models were used.

The derivations below require the specification of a distribution for θ. In this article, θ is assumed to follow a Fleishman PT distribution, φ ( θ | Ω ) , where Ω = ( μ , σ 2 , κ 3 , κ 4 ) to indicate that θ has a mean μ, variance σ 2 , and skewness and kurtosis of κ 3 and κ 4 , respectively ( φ ( θ | Ω ) is discussed in greater detail in the Appendix). Note that any univariate distribution could be chosen for θ. One advantage of using Fleishman’s PT distribution is that it is flexible enough to explore of how changes in μ, σ 2 , κ 3 , and κ 4 affect the relative reliability of IRT and CTT θ estimates. However, as noted by an anonymous reviewer, researchers often set σ 2 = 1 to estimate item discriminations (i.e., the scale indeterminacy problem). Accordingly, the results in this article also use σ 2 = 1 to understand how manipulating item discriminations affects reliability. Fleishman’s PT distribution does not encompass the universe of all univariate distributions, so future researchers can modify the associated R code to examine the reliability of CTT and IRT estimates when θ follows other distributions. Furthermore, the discussion below provides an argument as to how reliability within the CTT and IRT frameworks is dependent on the match between the shape of φ ( θ | Ω ) and the topography of the conditional variance of x and θ ^ .

Reliability and Precision Within IRT Framework

One of the strengths of IRT over CTT relates to the well-known measure of precision for estimated trait scores, θ ^ . Reliability is specific to a group and is a function of the unconditional standard error of measurement (SEM). CTT has traditionally calculated the SEM for a group of scores, whereas the CSEM is a measure of precision that corresponds to a specific trait level within the IRT framework. The CSEM of θ ^ is related to the test information function (TIF), which is derived using the concept of Fisher’s information to measure the amount of information that a single observation provides about θ. In fact, the inverse of a TIF indicates the variance of θ ^ for a given θ. Previous research (Muraki, 1993; Samejima, 1994) discussed TIFs for polytomous IRT models and noted that the IIF for item i is

I i ( θ ) = − ∑ j = 1 J + 1 P ij ( u i = j | θ ) ∂ 2 ∂ θ 2 ln ( P ij ( u i = j | θ ) ) = ∑ j = 1 J + 1 [ [ ∂ P ij ( u i = j | θ ) ∂ θ ] 2 P ij ( u i = j | θ ) − ∂ 2 P ij ( u i = j | θ ) ∂ θ 2 ] .

For Muraki’s (1990) modified graded response model, P ij ( u i = j | θ ) is a function of P ij * ( u i > j | θ ) , which is a logistic function. The first two derivatives of P ij * ( u i > j | θ ) are as follows:

∂ P ij * ( u i > j | θ ) ∂ θ = α i P ij * ( u i > j | θ ) [ 1 − P ij * ( u i > j | θ ) ] , ∂ 2 P ij * ( u i > j | θ ) ∂ θ 2 = α i ∂ P ij * ( u i > j | θ ) ∂ θ [ 1 − 2 P ij * ( u i > j | θ ) ] = α i 2 P ij * ( u i > j | θ ) [ 1 − P ij * ( u i > j | θ ) ] [ 1 − 2 P ij * ( u i > j | θ ) ] .

Accordingly, I i ( θ ) can be computed using the first and second derivatives of the item category probabilities, P ij ( u i = j | θ ) , which are as follows:

∂ P ij ( u i = j | θ ) ∂ θ = ∂ P ij * ( u i > j | θ ) ∂ θ − ∂ P ij + 1 * ( u i > j + 1 | θ ) ∂ θ , ∂ 2 P ij ( u i = j | θ ) ∂ θ 2 = ∂ 2 P ij * ( u i > j | θ ) ∂ θ 2 − ∂ 2 P ij + 1 * ( u i > j + 1 | θ ) ∂ θ 2 .

The TIF for a test of polytomous items is the sum of the respective IIFs:

TIF ( θ ) = ∑ i = 1 I I i ( θ ) .

The relationship between true and observed IRT estimates can be written as θ ^ = θ + ε . If θ and ε are independent, the variance of θ ^ is the sum of the true and error variances, σ 2 { θ ^ } = σ 2 { θ } + σ 2 { ε } . The conditional variance of θ ^ given θ is defined as σ 2 { θ ^ | θ } = σ 2 { ε | θ } = ( TIF ( θ ) ) − 1 . The expected conditional variance of θ ^ for a specific distribution of θ, φ ( θ | Ω ) , is as follows:

E { σ 2 { θ ^ | θ } } = ∫ − ∞ ∞ σ 2 { θ ^ | θ } φ ( θ | Ω ) dθ .

The expected reliability of θ ^ is the well-known ratio of true to observed variance,

ρ θ ^ θ ^ = σ 2 σ 2 + E { σ 2 { θ ^ | θ } } ,

where σ 2 is the variance of θ specified in φ ( θ | Ω ) . Clearly, ρ θ ^ θ ^ is dependent on the characteristics of the test (i.e., σ 2 { θ ^ | θ } ) and the distribution of latent scores (i.e., φ ( θ | Ω ) ).

Reliability and Precision Within CTT Framework

Let x be the total score, or sum, of the Iu _i , such that u i = E { u i | θ } + e i . For a subject with a given θ, u _i equals one of J + 1 categories each with probability P ij ( u i = j | θ ) , so E { u i | θ } is

E { u i | θ } = ∑ j = 1 J + 1 j P ij ( u i = j | θ ) .

That is, E { u i | θ } is a weighted average of the category scores (i.e., j = 1 to J + 1 ) and the chance that subjects with a specific θ have an observed score u _i . The previous section discussed an expression for the expected conditional variance of θ ^ as a function of the inverse TIF and distribution of θ. This section analogously derives an expression for E { σ 2 { x | θ } } , which is the expected variance of x for a given value of θ. An equation for the reliability of x is presented as a function of E { σ 2 { x | θ } } and the variance of the expected true scores, σ 2 { E { u i | θ } } . This subsection proceeds by first deriving an expression for the conditional error variance of x, E { σ 2 { x | θ } } , and then identifies an equation for the true score variance, σ 2 { E { u i | θ } } .

Consider item i where the error is e i = u i − E { u i | θ } . Note that u _i is a coarse measure of E { u i | θ } in that u _i is ordinal and equals one of the J + 1 values, whereas θ is measured on an interval scale. One immediate observation is that E { e i | θ } = 0 . Recall that u _i is a polytomous item, so e i = j − E { u i | θ } for j = 1 to J + 1 . The conditional expectation of the error within item i is

E { e i | θ } = ∑ j = 1 J + 1 P ij ( u i = j | θ ) ( j − E { u i | θ } ) = ∑ j = 1 J + 1 j P ij ( u i = j | θ ) − E { u i | θ } = 0 .

The variability of observed u _i around expected values is determined by the error variance or precision; that is, σ 2 { u i | θ } = E { e i 2 | θ } and is defined as

σ 2 { u i | θ } = E { ( u i − E { u i | θ } ) 2 | θ } = ∑ j = 1 J + 1 P ij ( u i = j | θ ) ( j − E { u i | θ } ) 2 .

Equation 11 is new to the literature and the following sections compare the properties of σ 2 { u i | θ } with σ 2 { θ ^ | θ } .

σ 2 { u i | θ } is the conditional variance for a single item and an expression is needed for the conditional variance of x. An important observation is that errors within two polytomous items, say, u _i and u _h (let categories for u _h be indexed by k), are independent whenever the items are locally independent, which assumes that P ( u i = j , u h = k | θ ) = P ij ( u i = j | θ ) P hk ( u h = k | θ ) . Specifically, the covariance between e _h and e _i conditioned on θ is

σ { e i , e h | θ } = E { ( u i − E { u i | θ } ) ( u h − E { u h | θ } ) | θ } = ∑ j = 1 J + 1 ∑ k = 1 J + 1 ( j − E { u i | θ } ) ( k − E { u h | θ } ) P ( u i = j , u h = k | θ ) = ∑ j = 1 J + 1 ( j − E { u i | θ } ) P ij ( u i = j | θ ) ∑ k = 1 J + 1 ( k − E { u h | θ } ) P hk ( u h = k | θ ) = 0 .

The finding that errors within items are independent, if local independence is assumed, is particularly important, because the conditional error variance of the total score, x, given θ is simply the sum of the conditional item variances, σ 2 { u i | θ } .

Recall that x = ∑ i = 1 I u i . Equation 9 implies that the expected value of x given θ is

E { x | θ } = ∑ i = 1 I E { u i | θ } = ∑ i = 1 I ∑ j = 1 J + 1 j P ij ( u i = j | θ ) .

The conditional variance of x given θ is the sum of conditional variances for the Iu _i , under the assumption of local independence (see Equation 12), which implies that σ 2 { x | θ } = ∑ i = 1 I σ 2 { u i | θ } . The expected conditional variance for a specific distribution of θ is

E { σ 2 { x | θ } } = ∫ − ∞ ∞ σ 2 { x | θ } φ ( θ | Ω ) dθ .

Recall that in IRT the expected variance of maximum likelihood estimates is the expected value of σ 2 { θ ^ | θ } across the distribution of θ. Similarity, the expected conditional variance of x is found by replacing σ 2 { θ ^ | θ } with σ 2 { x | θ } . Consequently, researchers can compare the CSEM (i.e., σ { x | θ } and σ { θ ^ | θ } ) to understand which values of θ are associated with relatively more precision within the CTT and IRT frameworks.

Recall that E { x | θ } is the expected total score for subjects with a given θ and the variance of E { x | θ } across subjects provides a measure of the amount of true score variance. First, note that the unconditional mean of x is

E { x } = E { E { x | θ } } = ∫ − ∞ ∞ E { x | θ } φ ( θ | Ω ) dθ .

The variance of E { x | θ } across θ is

σ 2 { E { x | θ } } = E { ( E { x | θ } − E { x } ) 2 } = ∫ − ∞ ∞ ( E { x | θ } − E { x } ) 2 φ ( θ | Ω ) dθ .

The reliability of x is the ratio of true to observed variance:

ρ xx = σ 2 { E { x | θ } } σ 2 { E { x | θ } } + E { σ 2 { x | θ } } .

Factors That Affect the Precision of CTT and IRT Scores

The previous section derived expressions for the CSEMs for x and θ ^ (i.e., σ { x | θ } and σ { θ ^ | θ } ). The characteristics of polytomous IRT CSEMs are well understood. For example, σ { θ ^ | θ } tends to be smaller at points along the latent continuum where category thresholds are located and σ { θ ^ | θ } declines in regions where more discriminating items are located. In contrast, the expression for σ { x | θ } is new and the purpose of this section is to compare σ { x | θ } with σ { θ ^ | θ } to provide researchers with a conceptual understanding of the testing situations where x may be preferred to θ ^ in terms of measurement precision.

As an example, consider a test consisting of three items that each have four response categories. Moreover, let α and b be three-dimensional vectors of item discriminations and locations, such that α = (0.5, 1.5, 2.5) and b = (−1, 0, 1). Moreover, to simplify this example, let the category thresholds be equally spaced with values of (−1.6, 0, 1.6) units below and above θ − b i (i.e., the thresholds for the first item are located at −2.6, −1.6, and 0.6 on the θ scale).

Figure 1 presents σ 2 { u i | θ } (see Equation 11) and [ I i ( θ ) ] − 1 (see Equation 3) for the three items. The IRT conditional variances exhibit expected behavior. That is, [ I i ( θ ) ] − 1 is lowest at the category thresholds and [ I 3 ( θ ) ] − 1 is generally smaller than the other two items because the third item is more discriminating. One additional nuance is that [ I 3 ( θ ) ] − 1 has local minima at the category thresholds, whereas Items 1 and 2 have inverse information functions that appear smoother. Stated differently, in IRT, items with larger discriminations tend to have IIFs with more topography in regions near category thresholds.

OPEN IN VIEWER

Figure 1.

Conditional variances of CTT and IRT scored items for a hypothetical three-item test with four response options and equally spaced thresholds, c _j = (−1.6, 0, 1.6).

As expected, σ 2 { u i | θ } is smaller for more discriminating items; however, the general behavior of σ 2 { u i | θ } is different from [ I i ( θ ) ] − 1 . Namely, σ 2 { u i | θ } tends to be a downward-facing function where measurement error is largest at category thresholds. For example, σ 2 { u 3 | θ } tends to be the smallest of the three items, but σ 2 { u 3 | θ } has maxima near the category thresholds, which differs from IRT where measurements are more precise at category thresholds. Under a CTT framework, σ 2 { u i | θ } is smallest for either more extreme θ or for θ values that lie between category thresholds. In short, σ 2 { u i | θ } is roughly the mirror image of [ I i ( θ ) ] − 1 , because [ I i ( θ ) ] − 1 tends to be lower in segments of the θ continuum where σ 2 { u i | θ } is larger and vice versa.

Figure 2 includes the same items discussed in Figure 1, but with the exception that the thresholds are no longer equally spaced. Figure 2 shows that σ 2 { u i | θ } and [ I i ( θ ) ] − 1 respond inversely to unequal item thresholds. For instance, σ 2 { u i | θ } tends to increase in portions of the latent continuum that include more item thresholds, whereas [ I i ( θ ) ] − 1 is smaller in segments of the latent continuum where there are more item thresholds and increases wherever there are fewer thresholds.

OPEN IN VIEWER

Figure 2.

Conditional variances of a CTT and IRT scored items for a hypothetical three-item test with four response options and unequally spaced thresholds, c _j = (−1.6, 0, 1.0).

Figure 3 demonstrates CTT and IRT test CSEMs along with error bars around conditional expected values, E { x | θ } and E { θ ^ | θ } . The top row of Figure 3 illustrates the CTT and IRT CSEMs for x and θ ^ , respectively, for the hypothetical three-item test. Note that the vertical lines in the top row of panels in Figure 3 indicate category thresholds for Items 1, 2, and 3. Figure 3 shows that σ 2 { x | θ } is larger for θ values that are near category thresholds, whereas σ 2 { θ ^ | θ } is smaller at points on the latent continuum that have more thresholds and more discriminating items. For the three-item test, σ 2 { θ ^ | θ } appears more responsive to item discriminations than σ 2 { x | θ } as indicated by the fact that the slope of σ 2 { θ ^ | θ } is steeper than σ 2 { x | θ } in the θ range measured by Item 1.

OPEN IN VIEWER

Figure 3.

Test score CTT and IRT conditional standard error of measurement and expected value plots with ±2 error curves for a hypothetical three-item test with four response categories.

The second row of panels in Figure 3 plots E { x | θ } and E { θ ^ | θ } as well as ±2 times the CSEMs. As discussed previously, E { x | θ } is a more accurate indicator of θ for more extreme values on the θ continuum, whereas θ ^ is a better indicator in the range where items and categories are located. The overall reliability of x and θ ^ is dependent on the shape and location of the latent distribution. For example, if θ ~ N ( 0 , 1 ) most subjects lie in the middle range of the latent continuum where x is less precisely measured relative to θ ^ . In fact, the differences in σ 2 { x | θ } and σ 2 { θ ^ | θ } contribute to θ ^ being significantly more reliable than x (i.e., 0.65 vs. 0.49). Certainly, ρ _xx and ρ θ ^ θ ^ will change depending on the shape and location of the θ distribution.

The Reliability of x and θ ^ for Item and Subject Characteristics

Figure 3 included results for a simple example to demonstrate the theoretical differences between σ 2 { x | θ } and σ 2 { θ ^ | θ } , which are useful pieces of information for developing tests and instruments. That is, σ 2 { x | θ } and σ 2 { θ ^ | θ } provide applied researchers with an understanding of ranges of θ values that are best measured with x or θ ^ . Moreover, σ 2 { x | θ } and σ 2 { θ ^ | θ } offer researchers information about which measurement framework is most beneficial for various item characteristics and subject populations.

Equations 8 and 17 were used to compare the reliability of x and θ ^ as a function of the number of scale categories, purpose of measurement (i.e., item locations dispersed along the continuum or clustered in a given region), and θ distribution shape. More specifically, Figure 4 includes ρ θ ^ θ ^ and ρ _xx across scale categories (i.e., 2–10 response options) for three types of item locations and four types of distributions for θ. Item discriminations and test length were not manipulated and were fixed at 1.25 and 10, respectively, It is well known that increasing either item discrimination or test length increases reliability, and these parameters were held constant to focus on the other parameters.

OPEN IN VIEWER

Figure 4.

Reliability of CTT and IRT scores of a 10-item test for different item locations, number of response categories, and subject latent distribution shapes.

To simplify the discussion, the three scenarios assume that items have equally spaced category thresholds. Specifically, the item category thresholds (i.e., the J c_j) were equally spaced between −2.0 and 2.0 on the θ − b i continuum. Let c be the vector of category thresholds that are defined as c = 2 ( 2 J ( J + 1 ) − 1 − 1 ) where J is a vector with elements equal to the integers from 1 to J. For example, the threshold is zero for items with two scale categories (i.e., J+1 = 2) and items with four scale categories have three thresholds at −1, 0, and 1. Whereas the following discussion assumes the thresholds are equally spaced, researchers can input any set of category thresholds into the R code, which is available at the author’s website.

Let b be a vector of item locations and I be a vector that includes integers from 1 to I. The three item location scenarios represent situations where researchers would be interested in measuring θ values in a narrow range in the lower or upper tails or measuring θ values across the latent continuum. The item locations for the three scenarios represent the following uniform distributions: b i ~ U ( − 2 . 5 , − 1 . 5 ) (i.e., b = 0 . 5 ( 2 I ( I + 1 ) − 1 − 1 ) − 2 ), b i ~ U ( − 2 . 0 , 2 . 0 ) (i.e., b = 2 ( 2 I ( I + 1 ) − 1 − 1 ) ), and b i ~ U ( 1 . 5 , 2 . 5 ) (i.e., b = 0 . 5 ( 2 I ( I + 1 ) − 1 − 1 ) + 2 ).

As noted, four subject distributions were examined to evaluate how the density of the population at various points on the latent continuum affected ρ θ ^ θ ^ and ρ _xx . Specifically, the distributions were negatively skewed ( γ 3 = −1.5, γ 4 = 4.0), normal ( γ 3 = 0, γ 4 = 0), symmetric and peaked ( γ 3 = 0, γ 4 = 4.0), and positively skewed ( γ 3 = 1.5, γ 4 = 4.0).

Figure 4 includes 12 panels corresponding to the three item locations and four subject distribution types. The middle row of panels in Figure 4 demonstrates that CTT and IRT yield similar reliabilities in situations where tests consist of items that are evenly placed along the latent continuum. Given that σ 2 { x | θ } is smaller in areas with fewer category thresholds, one explanation for the slight advantage of IRT is that CTT CSEMs decline outside of the (−2, 2) range of the items where there are fewer subjects in the skewed and symmetric distributions. More precisely, Figure 3 showed that σ 2 { θ ^ | θ } is significantly larger relative to σ 2 { x | θ } in portions of the latent continuum that has fewer category thresholds. In contrast, σ 2 { x | θ } is relatively smoother across θ values and tends to decline in segments where there are fewer category thresholds. Furthermore, in the case where items are located across the latent continuum, θ ^ is understandably more reliable than x when the latent distribution is more peaked (i.e., positive kurtosis). The middle row of Figure 4 also shows the expected positive relationship that increasing scale categories has on reliability for both θ ^ and x. However, as found in previous research (e.g., see a review of relevant literature in Chafouleas et al., 2009, and Weng, 2004),the value of an additional response category increases at a decreasing rate and reliability does not increase significantly in value beyond four or five response categories.

In theory, x is expected to be more reliable than θ ^ when the latent distribution is mismatched with item locations. The top and bottom panels of Figure 4 demonstrate the theoretical results discussed in this article concerning the effect that subject and item characteristics have on CTT and IRT reliability. For instance, CTT is superior to IRT whenever the items are located at the extremes of the latent distribution. Furthermore, the difference between ρ _xx and ρ θ ^ θ ^ is largest for two category items and approaches zero as the number of categories increase. As discussed earlier, IRT is a tool for measuring specific θ values more precisely. In fact, the difference between ρ _xx and ρ θ ^ θ ^ is smallest when the majority of the latent distribution overlaps with item locations, which occurs when items are located in the lower portion of the latent continuum and the latent distribution is positively skewed (e.g., b i ~ U ( − 2 . 5 , − 1 . 5 ) , γ 3 = 1.5, and γ 4 = 4.0) or when items are in the upper tail and the distribution is negatively skewed (i.e., b i ~ U ( 1 . 5 , 2 . 5 ) , γ 3 = −1.5, and γ 4 = 4.0).

In short, the results in this section provide new information about the influence of item and subject characteristics on the relative reliability of x and θ ^ . In fact, the findings reflect the expected results given the nature of σ 2 { x | θ } and σ 2 { θ ^ | θ } . That is, total scores were more reliable whenever items were mismatched with the location of the latent θ distribution. The following section addresses another important concern related to the accuracy of linear approximations of polytomous items.

Appropriateness of a Linear Approximation of Polytomous Items

One approach for modeling polytomous items is to use a linear approximation of the relationship between θ and u _i (e.g., a common factor model, Culpepper, 2012b). As noted in Equation 13, the relationship between θ and u _i is nonlinear. Ferrando (2009) and Mellenbergh (1996) noted that a linear relationship tends to provide a reasonable approximation when item discrimination indices are relatively low and there are five or more response categories. The purpose of this section is to explore the appropriateness of linear approximations of polytomous items in greater detail. This section includes two subsections. The first subsection describes a measure for quantifying the appropriateness of a linear approximation, whereas the second subsection presents findings about the accuracy of a linear approximation for different item characteristics (e.g., items locations, number of response categories, and item discriminations) and subject characteristics (e.g., the skewness and kurtosis of the latent distribution).

A Measure for the Appropriateness of a Linear Approximation

Let λ i ( θ ) be the loading relating latent θ to observed u _i for a given θ. The relationship between θ and u _i for a specific value of θ is the first derivative of Equation 9 with respect to θ:

λ i ( θ ) = ∂ E { u i | θ } ∂ θ = ∑ j = 1 J + 1 j ∂ P ij ( u i = j | θ ) ∂ θ .

The relationship between u _i and θ will be larger for some values of θ and near zero at the extremes of the θ continuum (i.e., portions of the latent continuum where E { u i | θ } plateaus). A linear approximation is dependent on the characteristics of the examinee population. The expected loading for a given population of subjects can be found by averaging λ i ( θ ) over the distribution of θ:

E { λ i ( θ ) } = E { ∂ E { u i | θ } ∂ θ } = ∑ j = 1 J + 1 j ∫ − ∞ ∞ ∂ P ij ( u i = j | θ ) ∂ θ φ ( θ | Ω ) dθ .

The function that constrains the relationship between θ and u _i to be linear for a given population of examinees is

L { u i | θ } = E { u i } + E { λ i ( θ ) } ( θ − μ ) ,

where E { u i } = ∫ E { u i | θ } φ ( θ | Ω ) dθ , μ = E { θ } , and L { u i | θ } denote the linear approximation of the relationship between θ and u _i . Also, note that L { u i | θ } has the same form as the least squares projection where L { u i | θ } passes through the centroid for θ and u _i ( μ , E { u i } ) and the relationship is the expected slope for agiven population of examinees, E { λ i ( θ ) } .

Let d i = u i − L { u i | θ } be the error when predicting the observed item u _i with a linear approximation. Unlike, E { e i | θ } , E { d i | θ } ≠ 0 ,

E { d i | θ } = E { u i − L { u i | θ } | θ } = E { u i | θ } − L { u i | θ } .

That is, d _i will be biased for certain values of θ but E { E { d i | θ } } = 0 across the θ range for a given population. The bias in d _i does not translate into larger conditional variances for L { u i | θ } . In fact, σ 2 { d i | θ } = σ 2 { u i | θ } ,

σ 2 { d i | θ } = E { ( d i − E { d i | θ } ) 2 | θ } = E { ( u i − E { u i | θ } ) 2 | θ } = σ 2 { u i | θ } .

Furthermore, the bias in d _i does not introduce dependence among residuals. Let d h = u h − L { u h | θ } be a residual when linearly approximating u _h . The covariance between errors is

σ { d i , d h | θ } = E { ( d i − E { d i | θ } ) ( d h − E { d h | θ } ) | θ } = E { ( u i − E { u i | θ } ) ( u h − E { u h | θ } ) | θ } = 0 ,

where the last inequality was obtained by recalling that the assumption of local independence implies E { e i e h | θ } = 0 as shown in Equation 12.

Ferrando (2009) described a measure of the appropriateness of a linear model, which is based on restricting observed scores to fall within feasible ranges (e.g., 1 and J + 1 for a polytomous item with J + 1 categories). More precisely, 1 < L { u i | θ } < J + 1 implies that a linear approximation is appropriate for θ scores within the following range:

1 − E { u i } E { λ i ( θ ) } + μ < θ < J + 1 − E { u i } E { λ i ( θ ) } + μ .

Following Ferrando, the floor and ceiling indices for the appropriateness of a linear approximation of u _i are δ 0 = ( 1 − E { u i } / E { λ i ( θ ) } ) + μ and δ 1 = ( J + 1 − E { x } / E { λ i ( θ ) } ) + μ , respectively. Ferrando’s measure of appropriateness is the proportion of examinees with θ scores in the viable range:

P ( δ 0 < θ < δ 1 ) = ∫ δ 0 δ 1 φ ( θ | Ω ) d θ .

Consider the examples presented in Figure 5, which compares E { u i | θ } with L { u i | θ } for three items with b_i = (−2, 0, 2) and the following parameter values held constant, α i = 2 : for all i, four response categories ( J + 1 = 4 ), category thresholds of c = −1.6, 0, 1.6, and Ω = ( 0 , 1 , 0 , 0 ) . The middle panel in Figure 5 includes the case where b_i = 0. Figure 5 shows that E { u i | θ } and L { u i | θ } deviate at the ends of the latent continuum. However, given that the distribution of θ is standard normal, nearly all of the subjects (i.e., 98.9%) have approximated scores between δ l and δ u . The top and bottom rows of Figure 5 include examples of items that are located at −2 and 2, respectively. In fact, when items are located at the extremes of the θ continuum P ( δ l < θ < δ u ) = 0.876%, which implies that 12.4% of subjects have approximated scores outside the viable bounds. The examples in Figure 5 demonstrate that item location and the shape of the latent distribution affect the appropriateness of linear approximations. For instance, Figure 5 provides insight that P ( δ l < θ < δ u ) is larger in value if items are located near the mean of a standard normal distribution.

OPEN IN VIEWER

Figure 5.

Hypothetical E { x | θ } and L { x | θ } for items with different locations and four response options.

Impact of Item and Subject Characteristics on the Appropriateness of a Linear Approximation

The previous subsection discussed P ( δ 0 < θ < δ 1 ) as a measure of the appropriateness of a linear model. The purpose of this subsection is to present more general findings about the role of item and subject characteristics on the accuracy of a linear approximation. Specifically, this subsection discusses the appropriateness of a linear approximation of u _i for 12 scenarios as defined by three response category scenarios ( J + 1 = 2-10) and four θ distribution scenarios (e.g., the same distributions examined in Figure 4) with the addition that the item discrimination indices are compared for α i = 0.5, 1.0, 1.5, and 2 and category thresholds are defined as discussed for the scenarios in Figure 4.

Figures 6 and 7 plot P ( δ 0 < θ < δ 1 ) against item locations (i.e., b_i) and provide evidence about circumstances where a linear approximation is appropriate. Note that the rows of Figures 6 and 7 correspond to the number of response categories (i.e., J+1), whereas columns relate to distribution shape. Comparing rows demonstrates that P ( δ 0 < θ < δ 1 ) increases as the number of response options increases; however, the accuracy of a linear approximation does not improve significantly beyond four response categories. For instance, a linear approximation is appropriate and P ( δ 0 < θ < δ 1 ) > 0 . 90 for items with five or more response categories that are located within one standard deviation of the mean. Furthermore, larger item discriminations have a negative effect on P ( δ 0 < θ < δ 1 ) and the number of response categories and item discrimination has an interactive effect where P ( δ 0 < θ < δ 1 ) declines more as α i increases and there are fewer response categories. Moreover, Figure 6 illustrates that a linear approximation is appropriate for as few as two response options when α i = 0.5 or whenever α i = 1.0 and items are located in the middle of the distribution.

OPEN IN VIEWER

Figure 6.

The proportion of viable scores from a linear approximation of polytomous items with two to four response categories by item location and discrimination, and subject latent distribution shapes.

OPEN IN VIEWER

Figure 7.

The proportion of viable scores from a linear approximation of polytomous items with five to seven response categories by item location and discrimination, and subject latent distribution shapes.

Comparing columns provides an indication of the effect of latent distribution shape and item location on P ( δ 0 < θ < δ 1 ) . Specifically, the relationship between b_i and P ( δ 0 < θ < δ 1 ) reflects the shape of the latent distribution, and P ( δ 0 < θ < δ 1 ) is smallest when b_i is located in the tails of the latent distribution. For instance, the P ( δ 0 < θ < δ 1 ) curves for the normal distribution appear more bell shaped when the distribution is normal as opposed to peaked (i.e., κ 3 = 0 , κ 4 = 4 ). In fact, P ( δ 0 < θ < δ 1 ) is smaller when θ ~ N ( 0 , 1 ) , ceteris paribus. The relationship between item location and P ( δ 0 < θ < δ 1 ) is cubic when the latent distribution is skewed, and P ( δ 0 < θ < δ 1 ) is smaller in skewed distributions in segments where there is less density in the latent distribution (e.g., the right tail if κ 3 = − 1 . 5 and the left tail if κ 3 = 1 . 5 ).

Figure 6 also demonstrates the effect of item discrimination on P ( δ 0 < θ < δ 1 ) . Ferrando (2009) noted that a linear approximation is most appropriate when α i is smaller. In fact, Ferrando’s recommendations are supported given that a linear approximation is expected to be appropriate for almost all subjects and item locations when α i = 0.50. Figures 6 and 7 also show conditions where linear approximations are appropriate even when items are more discriminating (e.g., α i = 2 ). For example, P ( δ 0 < θ < δ 1 ) is larger when J + 1 ≥ 3 and items are located near the middle of the latent distribution. P ( δ 0 < θ < δ 1 ) declines as α i increases for all of the scenarios included in Figure 6 albeit at different rates. In short, the results in Figures 6 and 7 imply that linear approximations are best for items that measure θ near the central portion of the latent distribution. Furthermore, L { u i | θ } is least accurate for highly discriminating items, and a linear approximation is inaccurate even if there are more than four response categories when items are located in the tails of the latent distribution.

Discussion

The findings in this study offer guidance to applied researchers interested in the construction and analysis of polytomous items. In short, this article presented new theoretical results concerning item and subject characteristics that affect the relative precision and reliability of x and θ ^ . This section summarizes the findings for psychometricians and applied researchers and offers concluding remarks.

This article studied the reliability of two alternative test scoring approaches: a CTT total score versus an IRT θ ^ estimate. The majority of applied researchers in education and psychology are familiar with x, whereas fewer have knowledge about θ ^ and IRT. Consequently, it is important to offer applied researchers information about the relative merits of x and θ ^ . This article offered additional insights into fundamental differences between x and θ ^ . One salient factor studied in this article was the interactive effect that item locations and θ distribution shape had on ρ _xx and ρ θ ^ θ ^ . Suppose the purpose of testing is to accept high-scoring examinees into an institution as is the case with examinations for actuarial sciences where the latent distribution is either normal or positively skewed. For instance, the first actuarial exam includes items clustered in the upper portion of the θ continuum to identify those students who are most competent in the foundations of calculus and probability theory. The scores for examinees who are near the passing cutoff are measured more precisely than those who are in the middle or lower portions of the θ distribution as indicated by smaller values for σ 2 { θ ^ | θ } . In contrast, the results in this article show that σ 2 { x | θ } is larger near the cutoff score and would be relatively smaller for θ values in the lower and middle portions of the examinee distribution. Certainly, decision makers would prefer IRT scoring versus a total score in this instance, because the purpose of measurement is to evaluate whether test takers exceed a minimum proficiency level. However, secondary users of the actuarial test score data should prefer total scores, because, as indicated in Figure 4, x tends to be more reliable for a population of subjects than θ ^ if items are difficult and located in the tails of the test-taker population distribution. For instance, some secondary users may want to gather validity evidence and correlate examinee scores with other indicators, such as undergraduate or graduate grade point average (Culpepper, 2010; Culpepper & Davenport, 2009), other aptitude tests, or job performance (Aguinis, Culpepper, & Pierce, 2010). In these instances, the validity coefficient associated with θ ^ would be smaller than the coefficient for x because ρ xx > ρ θ ^ θ ^ .

In addition, the results provide evidence that CTT and IRT scoring methods perform similarly when the goal of testing is to measure a construct across values of a latent continuum. For example, federal and state testing programs measure what students know in relation to given standards. The results in Figure 4 imply that ρ _xx and ρ θ ^ θ ^ are similar and testing programs could report total scores, which may be easier to explain to certain stakeholders (e.g., teachers, parents, and students).

The results in this article also offer recommendations for the construction of scales that include polytomous items, such as educational or employment performance assessments, behavioral ratings, or affective measurements. In either case, researchers may prefer x to θ ^ if the purpose of instrument development is to conduct correlational research rather than to measure specific trait levels. However, x should probably only be preferred to θ ^ when researchers fit simple models rather than more complicated interactive models (Embretson, 1996; Kang & Waller, 2005; Morse et al., 2012).

In addition, the findings in this article provide information about the optimal number of item response options. The results in Figure 4 examined the reliability of total scores and IRT estimates across a range of parameter values for the number of scale categories, item locations, and the shape of the latent distribution. The findings in Figure 4 imply that using more than five or six scale categories does not significantly improve the reliability of x or θ ^ regardless of the shape of the latent distribution or location of items. However, it is important to note that adding an additional scale value had a larger effect on ρ θ ^ θ ^ than on ρ _xx .

Another relevant finding for applied researchers (and methodologists) relates to the appropriateness of a linear approximation of polytomous items. The results in this article confirm arguments in previous research (Ferrando, 2009) that linear approximations are more accurate for less discriminating items and items with more response categories. Additional findings suggest that linear approximations of polytomous items seem appropriate for items that measure trait levels in denser segments of the latent distribution, and linear approximations were least appropriate for items located in the tails of the θ distribution. In contrast to previous research, the results provided new evidence that researchers need to consider item locations, in addition to item discriminations and the number of response categories, when employing linear approximations.

Last, another contribution of this study is the availability of the associated R code. More specifically, researchers can use the R code when designing instruments in an effort to evaluate conditions where total scores and IRT scores are more or less reliable and precise. Furthermore, the R code has pedagogical value, as well, for computational applications of the theoretical results.

There are several directions for future research to build on this study. First, this article offers recommendations for researchers who are interested in using x as a measure of θ in applied research. Specifically, designing a reliable x requires the inclusion of items located at the boundaries of the θ range of interest. For instance, if the measurement goal is to distinguish high versus low scorers on some trait, the results pertaining to CTT CSEMs dictate that the items should be located in the middle of the distribution, because low and high scorers will be measured more precisely. Likewise, items should be located around a cutscore (and not at the cut-score) if the purpose of measurement is to make inferences about whether examinees exceed or fall below some minimum proficiency level. The behavior of CSEMs under CTT is counterintuitive, because, in contrast, item design under the IRT framework dictates that developers should write items that are specific to certain θ levels and measurement purposes. Additional research is needed to understand differences in optimal test assembly (H. Chang & Ying, 2009) within the IRT and CTT frameworks.

Second, there could be benefits in revisiting some topics in modern IRT within the context of CTT. For instance, there may be new insights available by reexamining topics in CTT, such as computer adaptive testing (H. Chang & Ying, 1996) or equating techniques (Kolen & Brennan, 2004), which could lead to new methodologies, refinements of existing approaches, or other unanticipated discoveries.

Third, researchers could extend the results in this article to understand the impact of using x and θ ^ . Specifically, researchers use total scores as dependent variables and predictors in every subdiscipline of psychology and education. Despite the widespread use of total scores, few methodological studies have examined the impact of using total scores on the power and Type I error rates of tests that researchers employ (Embretson, 1996; Kang & Waller, 2005; Morse et al., 2012). Furthermore, with the exception of Embretson (1996), previous studies utilized Monte Carlo techniques that are limited by the parameter values studied. Consequently, future analytic explorations could provide additional insights into the effect of using total scores, and future research should accordingly examine the effect that the number of scale categories, the shape of the latent distribution, and IRT parameters have on the performance of commonly used statistical tests (Culpepper, 2012a; Culpepper & Aguinis, 2011).

Fourth, this study examined the theoretical reliability of x and θ ^ using a polytomous IRT model. As one anonymous reviewer noted, this article addressed reliability from a mathematical perspective and does not consider factors related to subjects’ cognitive decision making. For example, this article did not address issues related to category labels; however, previous research identified a causal effect of scale labels, category position, and rating scale intensity and length on certain observed item characteristics (Dunham & Davison, 1991; Lam & Stevens, 1994; Murphy & Constans, 1987). For instance, existing evidence suggests that scale labels can affect the observed item means, but there is less evidence that manipulating category labels alters observed item variances (L. Chang, 1997; Dunham & Davison, 1991). Moreover, positive-packed scales tend to affect item means (Dunham & Davison, 1991; Lam & Kolic, 2008), and semantic compatibility of category labels improves reliability (Lam & Kolic, 2008). Most of the previous literature on category labels used CTT or generalizability theory, and additional empirical research is needed to understand how category labeling decisions affect polytomous IRT item parameters (e.g., item locations, discriminations, and category thresholds) and consequently alter the reliability of x and θ ^ . Future research may identify relationships between category labels, cognitive decision making, and IRT parameters, and the results in this article provide mathematical arguments for describing how subsequent changes in IRT parameters affect test score reliability.

In conclusion, this study presented new results concerning the relative reliability and precision of total scores and IRT scores. The derivations in this article offer the most extensive analysis of the reliability of total scores by linking parameters of polytomous IRT models with CTT. In addition, new equations were discussed that described the CTT CSEM to provide new conceptual understanding of differences in the precision of scores estimated within the CTT and IRT frameworks.