An Illustration of the Effects of Ignoring a Secondary Factor

The purpose of this brief report is to illustrate how a small proportion of items measuring a secondary factor can have a large impact on the misestimation of the a-parameters for all items. The test measured library reference skills: 107 items were available for calibration, 42 operational items and 65 pilot items; 3,803 examinees took the operational items, and 600 to 650 examinees took each pilot item. Ten of the items covered Boolean searches.

The parameters were first calibrated in flexMIRT (Cai, 2012), using the 3 parameter logistic (3PL) unidimensional model without the constant of 1.7. The priors were as follows: Slope: logNormal (0.5, 0.5), Intercept: Normal (0, 2), Guessing: Beta (201, 801). Five of the Boolean items had extremely high a-parameters, and three others were above average (Table 1). This suggested that there were likely two factors: a primary factor measuring the construct of information literacy skills and a secondary factor measuring understanding of Boolean operators. Thus, a bifactor model was run, with all items loading on the primary factor and the Boolean items loading on a secondary, orthogonal factor. This secondary factor represented the remaining shared residual variance not captured by the primary factor. In the bifactor model, the Boolean items had much lower a-parameters for the primary factor than they did in the unidimensional model (Table 1), and most of the other items had higher a-parameters for the primary factor than the they did in the unidimensional model (Figure 1). As another check, a unidimensional model was run excluding the Boolean items. The resulting a-parameters (also in Figure 1) were similar in magnitude to those from the primary factor.

OPEN IN VIEWER

Table 1.

Boolean Items: a-Parameters.

Item	Unidimensional model	Bifactor primary	Bifactor secondary	Empirical projection	Analytical projection
156	1.67	0.81	1.50	0.65	0.61
159	5.93	2.76	5.00	1.00	0.89
160	5.89	2.42	4.95	0.88	0.79
161	3.72	1.83	3.38	0.89	0.82
162	6.68	2.98	5.77	0.93	0.84
205	0.63	0.67	0.23	0.67	0.67
216	4.14	1.67	3.06	0.89	0.81
12201	1.09	1.10	1.07	1.06	0.93
12214	0.83	0.69	0.64	0.69	0.65
12216	1.10	0.71	0.93	0.73	0.62

Note. All parameters are in the logistic metric; divide by 1.7 to put them in the normal metric.

OPEN IN VIEWER

Figure 1.

a-Parameters for the other items.

When a unidimensional model is estimated but more than one dimension underlies the data, the factor is a composite of the underlying dimensions. The contribution of each dimension can be quantified by the angle between the composite and each of the reference axes. The vector of angles is the first eigenvalue of a′a, where a is the matrix of a-parameters from the multidimensional model (Reckase, 2009). With two dimensions, an angle of 0° with one dimension and 90° with the other indicates the composite is in the direction of the first dimension. For these data, the angle of the composite with the primary factor was 47°, which indicates that the unidimensional composite was tilted in the direction of the Boolean factor. Although Boolean operators are construct-relevant, it seems unlikely that the content experts intended to weight them so heavily; these items were only 9% of the test, yet were weighted approximately equally to the primary factor in the composite.

One option would be to estimate scores with a bifactor model. Each examinee’s posterior distribution would be a function of both factors; after marginalizing over the secondary factor, the mean of the posterior would be the examinee’s expected a posteriori (EAP) score on the primary factor. However, because this test is administered on computer, the score is provided immediately after completion. The current testing software uses maximum likelihood scoring and a unidimensional model; it would need to be reprogrammed for a bifactor model. As an alternative, the Boolean items were projected in the direction of the primary factor. Kahraman and Thompson (2011) compared analytical formulas with empirical projection and reported that the results were similar. For empirical projection, the parameters from the unidimensional analysis excluding the Boolean items were fixed, and the Boolean items were recalibrated one at a time. For analytical projection, the formulas in Ip and Chen (2012) were applied to the bifactor estimates. Both sets of values are reported in Table 1. The analytical values tended to be slightly lower, consistent with Kahraman and Thompson’s findings with a correlated factors model. Kahraman and Thompson also noted that the differences between the analytical and empirical projects were more pronounced for easy or low-discrimination items; all the items had fairly low discrimination in this study. Either projection would give lower weight to most of the Boolean items, especially considering that most of the other items had higher a-parameters when they were calibrated separately.

For the Boolean items, note that when the secondary a-parameter was large, the projected a-parameter was lower than the primary a-parameter estimate from the bifactor model. One way of viewing the projection is to marginalize the three-dimensional item characteristic curve (ICC) over the secondary θ. Figure 2 shows cross sections of P(θ _p ) for item 159 at different values of the secondary θ. When one averages over the secondary θ, weighting by the density, the resulting marginal function is far less steep. Hence, one cannot simply use the primary a-parameter estimates from the bifactor model as if they were unidimensional estimates. Instead, the multidimensional estimates should be projected in the desired direction.

OPEN IN VIEWER

Figure 2.

Item 159, ICC conditional on secondary θ.

The differences in the a-parameters have practical implications for score estimates. θ ^ s from the unidimensional and empirical projections, based on the exact same item responses, were only correlated at .93. The mean absolute difference was .29, with a difference of .68 or greater for 10% of examinees.