Multidimensional Item Response Model for Nominal Variables

Abstract

This article describes a multidimensional generalization of the nominal categories model that serves to estimate factorial models from nominal and ordinal observed responses, and includes a structural model for latent variables that distinguishes between endogenous and exogenous factors. The model includes a scale parameter for each response category in each factor. Item parameters relate the logit between categories to the vector of latent variables. The inferential framework is marginal maximum likelihood, implemented via static and adaptive Gauss–Hermite quadrature and Monte Carlo EM. The properties of estimators are investigated in a simulation study. An example with real data illustrates the utility of the model in analyzing local dependencies in testlets composed of multiple-choice items that are clustered in several groups around a common theme.

Keywords

multidimensional nominal categories model item factor analysis marginal maximum likelihood adaptive GH quadrature local independence multidimensional item response theory

Multidimensional item response models have provided a means to analyze the dimensionality of a set of items since the pioneering work by Bock, Gibbons, and Muraki (1988). The item factor model typically applies to dichotomous and ordinal data (Reckase, 2009), whereas the factorial analysis of nominal responses is more recent (Bock & Gibbons, 2010). For example, Thissen, Cai, and Bock (2010) proposed a flexible multidimensional model that can accommodate a variety of models between ordinal and nominal scales. This article continues this line of research and presents a generalization of the nominal categories model (Bock, 1972) to the multidimensional case. Moreover, a structural model can be imposed on the latent variables to distinguish between exogenous and endogenous factors (Jöreskog & Sörbom, 1979).

The nominal categories model may be regarded as the most popular model for nominal responses because of its unique combination of flexibility and mathematical simplicity. For these reasons, it is taken as the basis to develop a multidimensional nominal categories model (MNCM). The MNCM, in its full generality, imposes no constraints on item parameters. However, the article describes a method of linear constrains that provides particular cases suitable for the analysis of ordinal data or for testing a hypothesis about parameter structure (Revuelta, 2009, 2010).

The MNCM has a factor loading for each category in each factor, instead of having one parameter for each variable in each factor, as in the factor analysis of quantitative and ordinal data. There are a number of new applications in which the factor loadings may differ between categories. For example, consider one multiple-choice item measuring mathematics in which one of the categories has an involved wording. This distractor may have a higher loading on a verbal factor than other categories written in more simple terms. More generally, two-dimensional item response models have been proposed in the psychometric literature to distinguish between a factor of substantive interest and a factor that explains individual differences in solution strategy or response style. For example, Bolt and Johnson (2009) and Johnson and Bolt (2010) analyzed ordinal responses from a self-report of smoking dependence. Their second factor explains response tendencies that are unrelated to the construct measured by the items. A similar idea was translated to maximum performance testing by Bolt and Wollack (2012), who analyzed data from multiple-choice items in which there was a primary factor related to the correct option and a second factor that influences selection among distractors.

The present article includes an example with real data that illustrates the application of the MNCM to investigating local dependencies originated by testlets. In this example, several items are clustered around a common theme, which leads to violation of the local independence assumption in a one-dimensional model. The article shows how a polytomous multidimensional model can be used to analyze associations between categories from one item to another.

The article also includes a number of technical developments, such as a marginal likelihood/EM estimation algorithm (Schilling & Bock, 2005), the computation of the information matrix, a structural latent model, and a Taylor series expansion to approximate the correlation between response categories. Finally, the recovery of theoretical parameters is investigated in a simulation study. Most technical material has been moved to online appendices, and the main body of the article focuses on the description of the model, results of its application, and the conclusions.

MNCM

Suppose that each individual is described by a vector of $D$ factors, referred to as θ. According to the MNCM, the probability of category $r$ of item $i$ is given by

P_{i r} (θ) = \frac{exp (z_{i r})}{∑_{k = 1}^{K_{i}} exp (z_{i k})},

where $K_{i}$ is the number of categories, and $z_{ir} = τ_{ir} + {λ'}_{ir} θ = τ_{ir} + λ_{ir 1} θ_{1} + \dots + λ_{i K_{i} D} θ_{D}$ is the latent propensity of category r, where the prime denotes the vector transpose, $τ_{ir}$ is a scalar intercept parameter, and $λ_{ir}$ is a vector of scale parameters that includes a parameter for each factor. In matrix form, the vector of latent propensities for item i is

z_{i} = τ_{i} + Λ_{i} θ,

where $τ_{i}$ is a column vector of K_i elements, and $Λ_{i}$ is a matrix of K_i×D scalar parameters. The parameters of the MNCM describe the logit between pairs of alternatives. The logit between categories r and K_i is given by

\log \frac{P_{i r} (θ)}{P_{i K_{i}} (θ)} = (τ_{i r} - τ_{i K_{i}}) + (λ_{i r 1} - λ_{i K_{i} 1}) θ_{1} + \dots + (λ_{i r D} - λ_{i K_{i} D}) θ_{D} .

The parameters of Equation 3 are unidentifiable because the logit depends on differences between parameters and does not vary if a constant value is added to all the parameters. The identifiability problem is more easily solved when Equation 3 is written in matrix form (Revuelta, 2008). Let P_i be the vector of response probabilities for item i and S_i be an identity matrix supplemented with a column vector of −1; that is, $S_{i} = (I; - 1)$ and the size of S_i is $(K_{i} - 1) \times K_{i} .$ The following equation describes in matrix form the logit of Categories 1 to $K_{i} - 1$ against category K_i:

S_{i} \log P_{i} = S_{i} (τ_{i} + Λ_{i} θ) = S_{i} (τ_{i} + λ_{i}^{(1)} θ_{1} + \dots + λ_{i}^{(D)} θ_{D}),

where $λ_{i}^{(d)}$ is the d column of Λ_i. Parameter indeterminacy is removed by computing $τ_{i}$ and Λ_i from a smaller number of parameters using the linear equation

τ_{i} + λ_{i}^{(1)} θ_{1} + \dots + λ_{i}^{(D)} θ_{D} = V_{i} ι + W_{1 i} ς^{(1)} θ_{1} + \dots + W_{D i} ς^{(D)} θ_{D},

where $ι,$ $ς^{(1)},$ …, $ς^{(D)}$ are vectors of parameters, and V_i, $W_{1 i},$ …, $W_{Di}$ are matrices of fixed coefficients that impose a constraint on the parameters of Equation 2. The intercepts and the columns of Λ_i are given by $τ_{i} = V_{i} ι$ and $λ_{i}^{(d)} = W_{d i} ς^{(d)} .$ The matrices of coefficients have K_i rows and their number of columns must be smaller than K_i, although the exact number of columns depends on the number of parameters in vectors $ι,$ $ς^{(1)},$ …, $ς^{(D)} .$ Item parameters are identifiable when $S_{i} V_{i}$ and $S_{i} W_{di}$ (for d = 1,…, D) have full column rank (Rothenberg, 1971).

Two particular cases of Equation 5 are considered in this article. First, the same constraints may apply to the scale parameters of all dimensions. In that case, Equation 5 reduces to

τ_{i} + λ_{i}^{(1)} θ_{1} + \dots + λ_{i}^{(D)} θ_{D} = V_{i} ι + W_{i} (ς^{(1)} θ_{1} + \dots + ς^{(D)} θ_{D}) = V_{i} ι + W_{i} ςθ,

where $ς$ is the matrix with column vectors $ς^{(1)},$ …, $ς^{(D)};$ thus, $τ_{i} + Λ_{i} θ = V_{i} ι + W_{i} ςθ .$ The second particular case imposes the same constraint to all intercept and scale parameters. In that case, Equation 5 becomes

τ_{i} + Λ_{i} θ = W_{i} (ι + ςθ) .

In the context of linearly constrained item response models, identifiability is traditionally assessed by verifying that the augmented matrix $W_{i}^{+} = (W_{i}; 1)$ has full column rank (Fischer, 1983). The identifiability conditions based on $W_{i}^{*}$ and $W_{i}^{+}$ can be shown to be equivalent (see Online Appendix A).

Examples of Linear Constraints

Nominal Models

Models for nominal responses, in their most general form, impose no constraints apart from those that ensure identifiability. This section shows one example of the many forms that matrix $W_{i}$ may take. Multiple-choice items have a correct category, say, category K_i, whose parameters $τ_{i K_{i}}$ and $λ_{i K_{i}}$ are set to 0. The incorrect categories 1,…, K_i− 1 are unconstrained and describe the logit against the correct category:

l o g \frac{P_{i r} (θ)}{P_{i K_{i}} (θ)} = τ_{i r} + λ_{i r 1} θ_{1} + \dots + λ_{i r D} θ_{D} .

Equation 7 for an item with three categories and two factors reads as follows:

(\begin{matrix} τ_{i 1} \\ τ_{i 2} \\ τ_{i 3} \end{matrix}) + (\begin{matrix} λ_{i 11} & λ_{i 12} \\ λ_{i 21} & λ_{i 22} \\ λ_{i 32} & λ_{i 32} \end{matrix}) (\begin{matrix} θ_{1} \\ θ_{2} \end{matrix}) = (\begin{matrix} 1 & 0 \\ 0 & 1 \\ 0 & 0 \end{matrix}) ((\begin{matrix} ι_{i 1} \\ ι_{i 2} \end{matrix}) + (\begin{matrix} ς_{i 11} & ς_{i 12} \\ ς_{i 21} & ς_{i 22} \end{matrix}) (\begin{matrix} θ_{1} \\ θ_{2} \end{matrix})) .

In this example, $W_{i}$ satisfies the identifiability condition because the matrix

W_{i}^{*} = S_{i} W_{i} = (\begin{matrix} 1 & 0 & - 1 \\ 0 & 1 & - 1 \end{matrix}) (\begin{matrix} 1 & 0 \\ 0 & 1 \\ 0 & 0 \end{matrix}) = (\begin{matrix} 1 & 0 \\ 0 & 1 \end{matrix})

has full column rank. The linear restriction reduces the number of effective parameters from the nine elements contained in $τ$ and $λ$ to the six elements of $ι$ and $ς .$ One example in which constraints are different for each dimension is as follows. Suppose that Equation 5 reads

(\begin{matrix} τ_{i 1} \\ τ_{i 2} \\ τ_{i 3} \end{matrix}) + (\begin{matrix} λ_{i 11} & λ_{i 12} \\ λ_{i 21} & λ_{i 22} \\ λ_{i 32} & λ_{i 32} \end{matrix}) (\begin{matrix} θ_{1} \\ θ_{2} \end{matrix}) = (\begin{matrix} 1 & 0 \\ 0 & 1 \\ 0 & 0 \end{matrix}) (\begin{matrix} ι_{i 1} \\ ι_{i 2} \end{matrix}) + (\begin{matrix} 1 & 0 \\ 0 & 1 \\ 0 & 0 \end{matrix}) (\begin{matrix} ς_{i 11} \\ ς_{i 21} \end{matrix}) θ_{1} + (\begin{matrix} 1 & 0 \\ 0 & 1 \\ - 1 & - 1 \end{matrix}) (\begin{matrix} ς_{i 12} \\ ς_{i 22} \end{matrix}) θ_{2} .

The matrices $V_{i}$ and $W_{1 i}$ in Equation 11 are the same as $W_{i}$ in Equation 10, and thus, identifiability is ensured. Regarding $W_{2 i},$ the scale parameters in the second dimension are identifiable because the matrix

S_{i} W_{2 i} = (\begin{matrix} 1 & 0 & - 1 \\ 0 & 1 & - 1 \end{matrix}) (\begin{matrix} 1 & 0 \\ 0 & 1 \\ - 1 & - 1 \end{matrix}) = (\begin{matrix} 2 & 1 \\ 1 & 2 \end{matrix})

has full column rank.

Ordinal Models

Ordinal model use different constraints for intercept and scale parameters, as in Equation 6. The category scale parameters are proportional to an item scale parameter, $λ_{i r d} = (r - 1) ς_{i d},$ and $W_{i}$ is a column vector with integer values between 0 and $K_{i} - 1 .$ For an item with three categories and two factors, Equation 6 reads

(\begin{matrix} τ_{i 1} \\ τ_{i 2} \\ τ_{i 3} \end{matrix}) + (\begin{matrix} λ_{i 11} & λ_{i 12} \\ λ_{i 21} & λ_{i 22} \\ λ_{i 32} & λ_{i 32} \end{matrix}) (\begin{matrix} θ_{1} \\ θ_{2} \end{matrix}) = (\begin{matrix} 0 & 0 \\ 1 & 0 \\ 0 & 1 \end{matrix}) (\begin{matrix} ι_{i 1} \\ ι_{i 2} \end{matrix}) + (\begin{matrix} 0 \\ 1 \\ 2 \end{matrix}) (\begin{matrix} ς_{i 1} & ς_{i 2} \end{matrix}) (\begin{matrix} θ_{1} \\ θ_{2} \end{matrix}) .

The matrix $W_{i}$ has a smaller rank for an ordinal than for a nominal model because the ordinal model does not have separate scale parameters for each category.

The multidimensional model model by Thissen, Cai and Bock (TCB; 2010) is based on a nonlinear constraint on scale parameters and uses a Fourier basis to specify the matrix of coefficients. The TCB is more flexible than the model in Equation 13 and provides useful and interesting particular cases for the analysis of ordinal data by relaxing the assumption that the response categories are equally spaced. However, the TCB does not reach the generality of the full rank model in Equation 9 because of the effect of the nonlinear constraint.

Inferential Algorithms

The inferential algorithm used in this article assumes that θ is a random variable whose density function, f(θ), is MNormal (0, Σ). Moreover, the measurement model in Equations 1 and 2 is supplemented with a structural model to analyze relations of dependence between factors. The basic assumption is that factors are interrelated according to the linear equation

θ = Δθ + ϑ,

where Δ is a parameter matrix, ϑ is a vector of random disturbances, and the distribution of ϑ is MNormal (0, Ω). Equation 14 implies $(I - Δ) θ = ϑ,$ where $I$ is an identity matrix of order $D \times D .$ Defining $ϒ = {(I - Δ)}^{- 1},$ it follows that $θ = ϒϑ$ and $∑ = Υ Ω Υ' .$ In summary, two matrices have to be defined to set up a structural model following the formalism in Equation 14: Δ and Ω. The model in Equation 14 can accommodate the same relations between factors as a LISREL model (Jöreskog & Sörbom, 1979).

The sample consists of a vector of frequencies (n₁,…, n_t,…, n_T), where the element n_t is the number of individuals whose response pattern is $r_{t} = (r_{t 1}, \dots, r_{t i}, \dots, r_{t M}),$ $θ$ is regarded as a random effect, and the estimation proceeds by maximizing the multinomial marginal distribution of the response patterns. The log likelihood is given by $l (ε) = ∑_{t = 1}^{T} n_{t} \log P_{t},$ where ε is a vector that contains all the parameters to be estimated, and $P_{t}$ is the marginal probability of response pattern $t$ :

P_{t} = \int_{R^{D}} f (r_{t}, θ) d θ .

One important difficulty in estimation is the evaluation of the D-dimensional integral in Equation 15 and in its derivatives. Three numerical integration methods have been compared in this investigation: static Gauss–Hermite (GH) quadrature, adaptive GH, and Monte Carlo integration via importance sampling (Monte Carlo EM (MCEM)). These methods are described in Schilling and Bock (2005). The details of the estimation algorithms appear in Online Appendix B and are based on methods for matrix differential calculus described in Magnus and Neudecker (1999) and in Harville (2008).

Simulation Study

A simulation study was conducted to investigate the performance of the estimation algorithm in recovering true parameter values. The performance of the three integration methods for samples of different size was compared in the simulations.

The description of the simulated conditions and the results has been moved to Online Appendix C to preserve journal space, and this section contains only the main conclusions. Taken together, simulations show that adaptive GH and MCEM perform better than static GH when the number of items per factor is large and the structural model is simple. Static GH selects quadrature points to optimize precision in the evaluation of the integral of a function over the population distribution of the factor: $f (θ) .$ Adaptive GH and MCEM are based on a normal approximation, the posterior distribution of the factor for each response pattern: $f (θ | r) .$ The normal approximation to $f (θ | r)$ is based on the MAP (Modal a Posteriori) estimate of θ and its dispersion matrix, and its quality improves as the test length increases. The advantage of adaptive GH and MCEM may be more prominent in long tests, whereas in short tests, the imprecision in the MAP estimate may present a challenge to these two methods. Moreover, when the structural model is complicated, static GH compares well with the other methods. One tentative explanation is that static GH provides a stable basis for computing the integral over the latent space, whereas the instability inherent to adaptive GH and MCEM may compromise the estimation of involved structural models for latent variables.

Empirical Example

The use of the MNCM to account for measurement problems is demonstrated with a real data set of an examination about inferential statistics. A sample of 353 second-course undergraduate students of psychology responded to 20 multiple-choice items as part of the final examination of the subject of data analysis, which ranged from one-sample t test to ANOVA, multiple comparisons, and linear regressions. The participants ranged from 19 to 30 years old and had 60 min to complete the test. Because data were collected for a real evaluation, the identity of the participants and any other information apart from the item responses were removed before the data matrix was submitted to investigation. The test was structured in six testlets. Each testlet consisted of two or more items related to a common theme, and each item had three response categories. Identification constraints were as in Equation 9. Table 1 illustrates two of the testlets.

Table 1.

Two Testlets From the Exam of Data Analysis.

The means of two independent samples have been compared using a t test. The null hypothesis is H₀: µ₁ = µ₂ and the test statistic takes the value T = 1.56.

Item 9. If P(T > 1.56) = 0.10 the statistical decision is:

A. Reject H₀.

B. Do not reject H₀.

C. There is not enough information to reach a decision.

Item 10. Taking into account that the null hypothesis is a bilateral one:

A. The p value is .10.

B. We shall conclude that the population means do not differ.

C. Statements A and B are incorrect.

The mean level of stress of a sample of 20 individuals has been compared before and after an anti-stress treatment. The null hypothesis is that there is no difference between the mean before and after the treatment, and the alternative hypothesis says that there is a positive difference. The test statistic is T = 2. If P(T≤ 2) = 0.98 and the Type I error rate is α = 0.05.

Item 16. Shall we reject H₀?

A. No, because 0.98 is higher than 0.05.

B. Yes, because 0.02 is smaller than 0.05.

C. No, because 0.98 is higher than 0.09.

Item 17. What is the correct conclusion for this contrast?

A. The mean stress level has reduced.

B. The intervention has reduced the mean stress level.

C. There is no evidence that the intervention has reduced the mean stress level.

Note. Correct categories are B, C, B and A for Items 9, 10, 16, and 17, respectively.

Six models were applied to these data. All the models are variants of a bifactor model, depicted in Figure 1, which has a general factor and six group factors that explain the associations within each testlet. The six models are as follows:

M1: Bifactor oblique. The correlations between the seven factors are free parameters. That is, the latent propensity is $z_{ir} = τ_{ir} + λ_{ir 1} θ_{1} + λ_{ir 2} θ_{2}$ , Σ contains 1s in the main diagonal, and correlations between factors are estimated.

M2: Bifactor partly orthogonal. The correlations between the general factor and each of the group factors are free parameters. The correlations between group factors are set to zero. That is, $z_{ir}$ is as in M1, Σ contains 1s in the main diagonal, correlations involving Factor 1 are free parameters, and the other correlations are set to 0.

M3: Bifactor orthogonal. The seven factors are uncorrelated; $z_{ir}$ is as in M1, and Σ is an identity matrix.

M4: Testlet model. This is a polytomous generalization of the dichotomous testlet model by Bradlow, Wainer, and Wang (1999). In that model, the scale parameters of the general and the group factor have the same value to obtain a correlation between latent propensities with a convenient structure; that is, the latent propensity is $z_{ir} = τ_{ir} + λ_{ir} (θ_{1} + θ_{2}) .$ The diagonal elements of Σ (factor variances) are free parameters, and off-diagonal elements are zero.

M5: Testlet model, equal variances. That is, z_ir is as in M4, and Σ is an identity matrix.

M6: The one-dimensional nominal categories model; $z_{ir} = τ_{ir} + λ_{ir} θ$ , and Var(θ) = 1.

Figure 1.

General structure of the factorial models fitted in the empirical example.

These models are obtained as particular cases of the MNCM by using the method for linear constraints described in Online Appendix B. The model was estimated with two quadrature points for each dimension, as recommended by Schilling and Bock (2005) for a model with seven dimensions. However, note that this recommendation is based on the amount of computational resources that would be demanded by a larger number of points and not because two points is an optimal figure in a statistical sense. A number of statistics were used to test the model fit: the likelihood ratio for the hypothesis of exact fit, the likelihood ratio against the most general model, M1, and the statistics Akaike information criterion (AIC), Bayesian information criterion (BIC), and corrected Akaike information criterion (AICc), which consider the model fit and parsimony (Burnham & Anderson, 2004). Table 2 shows the results. M3 was selected for interpretation because of the values G², AIC, and AIC_c. Notwithstanding, BIC differs from the other statistics and points to M5 as the preferred model. Table 2 contains also the mean estimation time in minutes from five different runs of the estimation algorithm using different random starting values. Estimation time depends largely on the number of estimated parameters in the structural model in Equation 14.

Table 2.

Goodness of Fit Statistics.

Model	Log Lik.	G ²	Params	ΔG²	Δgl	p	AIC	AIC_c	BIC	T
M1	−3,169.63	2,231.30	141				6,621.26	6,810.15	7,166.83	52
M2	−3,175.80	2,243.65	126	12.35	15	.65	6,603.61	6,744.60	7,091.14	40
M3	−3,180.78	2,253.61	120	22.31	21	.38	6,601.57	6,726.21	7,065.89	35
M4	−3,261.20	2,414.45	86	183.15	55	.00	6,694.41	6,750.45	7,027.17	34
M5	−3,272.10	2,436.24	80	204.94	61	.00	6,704.20	6,751.67	7,013.75	23
M6	−3,304.59	2,501.21	80	269.91	61	.00	6,769.17	6,816.64	7,078.72	1

Note. M1 to M6 are described in the text. Log Lik. is the log-likelihood. G² is the likelihood ratio test against a general multinomial alternative. Params is the number of estimated parameters. The columns ΔG², Δgl, and p contain the result of the likelihood ratio test of each model against M1. AIC, AICc, and BIC are the information-theoretic statistics. T is the average estimation time in minutes. AIC = Akaike information criterion; BIC = Bayesian information criterion; AICc = corrected Akaike information criterion.

Table 3 presents the parameter estimates for M3. The item responses have been coded such that the correct response is the third one, and Categories 1 and 2 are the distractors. The scale parameters in the six group factors have been placed in the same column, labeled Group factor in the table, although they refer to a different factor for each testlet. All but two of the scale parameters in the general factor were negative, indicating negative correlations between the distractors and the general factor. Although estimates were unbounded in the “Simulation Study,” estimates were bounded between −10 and 10 in the real data analysis because the sample contains insufficient information to estimate some parameters. Table 3 shows that 11 intercept parameters reached the boundary during estimation. This is due to insufficient sample size. For example, the two distractors of Item 6, whose parameters τ are in the boundary, receive six and eight responses. If the estimation is repeated without the bound of the parameters, these estimates converge at very extreme values, −18.4 and −19.8 for Item 6 with first-order derivatives of 0.01 and 0.20 at convergence. Table 3 contains the results with the boundary because their actual estimated value is not important, the real problem is the lack of statistical information, and the boundary helps to identify easily the weakly identifiable parameters.

Table 3.

Parameter Estimates for the MNCM.

				General factor		Group factor
Testlet	Item	τ₁	τ₂	λ₁	λ₂	λ₁	λ₂
1	1	−10.00	−4.91	−0.72	−1.76	6.25	−0.22
	2	−10.00	−5.47	−2.94	−1.50	0.93	0.67
	3	−5.55	−4.59	−1.84	−1.46	−3.53	−0.87
	4	−0.34	−2.37	−0.10	−1.88	0.16	−0.14
	5	−0.64	−10.00	−0.11	−2.32	−0.10	−8.49
2	6	−10.00	−10.00	−1.32	−0.22	−4.83	−5.96
	7	−1.80	−10.00	−0.68	−1.28	−0.17	−5.12
	8	3.97	10.00	−2.34	−0.14	−7.95	−7.32
3	9	−5.35	−0.86	−2.62	−0.20	−3.31	−0.95
	10	−1.12	−3.15	−0.15	0.37	0.06	3.41
4	11	−10.00	−3.95	−0.58	−0.81	−6.05	0.12
	12	−1.91	−10.00	−0.33	3.28	0.13	−1.75
	13	−10.00	−3.34	−0.56	−0.88	−5.38	0.09
	14	−1.95	0.01	−1.03	−0.51	−0.37	−0.31
	15	−1.51	−3.76	−0.69	−1.00	−0.20	0.36
5	16	−3.97	−5.00	−2.20	−2.92	−3.51	−1.79
	17	0.23	−3.09	−1.06	−3.51	−0.75	−6.00
6	18	−2.18	−4.06	−0.05	−1.33	−0.67	−0.27
	19	−3.99	−10.00	−0.75	−0.66	0.04	−6.35
	20	−0.92	−1.88	−0.40	−0.70	−0.43	−0.39

Note. The table contains the parameters of the two incorrect categories, the parameters of the correct option are set to zero. There are six group factors, one for each testlet. MNCM = multidimensional nominal categories model.

Testlet 3 presents a number of interesting features. First, Distractor B of Item 10 (in the following, Distractor 10-B) has a positive loading on the general factor (λ_10,2,1 = 0.37). That is, the propensity to commit the misconception reflected in Category B increases with θ₁, which indicates that this topic was not well clarified during the teaching of the subject. Second, the Categories 9-A and 10-B have remarkable loadings on the group factor, with reversed signs. The group factor θ₄ seems to be bipolar in the sense that individuals high in θ₄ feel attracted by the error reflected in Category 10-B, and individuals low in θ₄ tend toward the irreconcilable error contained in Category 9-A. The association between Categories 9-A and 10-B is quite predictable in relation to item content and is explored more deeply in the following paragraphs.

In the M3, the principle of independence between alternatives conditional on θ₁ breaks down. To illustrate, the top row of Figure 2 shows the marginal response functions for Items 10 and 17, with the group factor marginalized out. The response functions vary depending on the response given to the other item in the same testlet (Items 9 and 16, respectively), as shown in the second, third, and fourth rows of Figure 2. The first item of the two testlets entails making a statistical decision about a null hypothesis, and the second item is the conclusion that follows from that decision. The interplay between the decisions and conclusions explains the associations. For example, the conclusion that two population means do not differ (Category 10-B) has a smaller probability for those individuals that reject H₀ (Category 9-A) than for those who do not reject H₀ (Category 9-B).

Figure 2.

Marginal response functions for Items 10 and 17 and response functions conditional on the response given to the preceding item in the test.

Clustered items present an extra covariance beyond that explained by a one-dimensional latent trait. For the one-dimensional model, the covariance between items conditional on θ is zero. Regarding M3, if the group factor is integrated out, the result is a marginal distribution of responses that depends only on θ₁ and has a non-null correlation between items. Ip (2010, Corollary 1) showed that the correlation between two dichotomous items, i and j, depends on the product of the scale parameters of the group factor. Online Appendix D extends that result to the polytomous case and shows how to compute item correlations in the marginal distribution that depends only on θ₁. The correlation depends largely on the product of the scale parameters but is not proportional to it. The key element that explains correlations in the polytomous case is the difference between a parameter and the mean of the scale parameters of the item. In particular, the correlation between categories r and s of items i and j is proportional to

P_{i r θ_{1}} (λ_{i r 2} - ∑_{k} λ_{i k 2} P_{i k θ_{1}}) P_{j s θ_{1}} (λ_{j s 2} - ∑_{k} λ_{j k 2} P_{j k θ_{1}}),

where $P_{ik θ_{1}}$ is the probability of the category when θ₂ = 0. When data are dichotomous, the scale parameter of the first category is set to zero, and Equation 16 reduces to $λ_{i 22} λ_{j 22} P_{i 2 θ_{1}} (1 - P_{i 2 θ_{1}}) P_{j 2 θ_{1}} (1 - P_{j 2 θ_{1}}),$ which is the result by Ip (2010).

Figure 3 presents the correlation between pairs of categories conditional on θ₁ for the two testlets in Table 1. As expected, Categories 9-B (the correct one) and 10-B (an incorrect one) have a positive relation, which singles out a frequent misconception about the conclusion of a significance test when H₀ is maintained. Moreover, the correlation between the two correct categories, 9-B and 10-C, is negative because the distractor 10-B is more attractive for those who pass Item 9 than for those who fail it, which is also predictable from item content. Regarding the second testlet, Categories 16-A and 16-C have similar content and a comparable association pattern with 17-C, which is their logical consequence. Categories 17-A and 17-B have a subtle difference and obtain similar patterns of association with Item 16.

Figure 3.

Correlation between categories as a function of θ₁ with the group factor integrated out.

Final Remarks

The purpose of the MNCM is to measure factors by using nominal response data. The key aspect of the model is that that there is one slope parameter for each response category on each factor, instead of one slope for each item on each factor, as in factorial models for quantitative and ordinal data. In its more general form, it has minimal parameter constraints to ensure identifiability, although the article shows how to impose more stringent restrictions by using linear constraints.

From a statistical point of view, the combination of multidimensionality and a heavy parameterization presents important challenges for estimation. The empirical example has shown that insufficient sample size may lead to weakly identifiable parameters. Exploratory analyses, in which the number of factors is increased until the model fits and item parameters are unconstrained, demand samples with thousands of examinees and time-consuming computational resources. Problems of weakly identified parameters may arise in exploratory analyses with smaller samples. Weakly identifiable parameters are identifiable in a mathematical sense, but the data provide scarce information about them, originating problems of convergence and large standard errors. Confirmatory analysis seems more tenable because fixed parameters and a smaller number of dimensions per item reduce the dimensionality of the optimization problem involved in estimation and increase statistical information about the remaining parameters.

The model has been developed in a traditional item response theory framework, with strong distributional assumptions for latent factors and a marginal likelihood estimation algorithm. Three integration methods have been compared: adaptive GH, static GH, and MCEM. The simulation results revealed that adaptive GH provides the greatest precision, especially with a large number of items per factor. However, the estimation of a structured variance–covariance matrix for the factors could be more precise with static GH because of the increased stability of this method. The numerical integration problem limits the dimensionality of the latent space to about eight factors, and clearly, an area of further research is the development of more effective estimation methods for higher dimensionality with a reasonable demand for computational resources.

Ideally, an improved algorithm should allow faster estimation and increased dimensionality. The development of better algorithms, within the maximum likelihood framework, may be based on the Metropolis-Hastings Robins-Monro integration algorithm by Cai (2010). Estimation of the bifactor model can also be accelerated by applying the dimension reduction method (Cai, Yang, & Hansen, 2011), which reduces the D-dimensional integration problem to two-dimensional integration. Fully Bayesian simulation techniques are another promising inferential framework (Edwards, 2010). The inclusion of diffuse priors may speed up the estimation algorithm by avoiding cases in which weakly identified parameters use a long sequence of iterations to converge. Nevertheless, Bayesian simulation methods would also benefit from the use of the output of the EM maximization to construct a multivariate normal approximation of the posterior distribution of the parameters, which can be used to obtain samples of parameters. The assumption of multivariate normality of the latent variables has been made to avoid further complications although its relaxation is another topic for future research.

Footnotes

Acknowledgements

The author thanks the editor and three anonymous reviewers for their comments that contributed to the improvement of the manuscript.

Declaration of Conflicting Interests

The author declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This article has been supported by Grant PSI2012-31958 from the Spanish Ministry of Economy and Competitiveness (DGICT).

References

Bock

R. D.

(1972). Estimating item parameters and latent ability when responses are scored in two or more nominal categories. Psychometrika, 37, 29-51.

Bock

R. D.

Gibbons

(2010). Factor analysis of categorical item responses. In Nering

M. L.

Ostini

(Eds.), Handbook of polytomous item response theory models (pp. 155-184). New York, NY: Routledge.

Bock

R. D.

Gibbons

Muraki

(1988). Full-information item factor analysis. Applied Psychological Measurement, 12, 261-280.

Bolt

D. M.

Johnson

T. R.

(2009). Addressing score bias and differential Item functioning due to individual differences in response style. Applied Psychological Measurement, 33, 335-352.

Bolt

D. M.

Wollack

J. A.

(2012). Application of a multidimensional nested logit model to multiple-choice test items. Psychometrika, 77, 339-357.

Bradlow

E. T.

Wainer

Wang

(1999). A Bayesian random effects model for testlet. Psychometrika, 64, 153-168.

Burnham

K. P.

Anderson

P. R.

(2004). Multimodal inference: Understanding AIC and BIC in model selection. Sociological Methods & Research, 33, 261-304.

Cai

(2010). High-dimensional exploratory item factor analysis by a Metropolis-Hastings Robbins-Monro algorithm. Psychometrika, 75, 33-57.

Cai

Yang

J. S.

Hansen

(2011). Generalized full-information item bifactor analysis. Psychological Methods, 16, 221-248.

10.

Edwards

M. C.

(2010). A Markov chain Monte Carlo approach to confirmatory item factor analysis. Psychometrika, 75, 474-497.

11.

Fischer

G. H.

(1983). Logistic latent trait model with linear constraints. Psychometrika, 48, 3-26.

12.

Harville

D. A.

(2008). Matrix algebra from a statistician’s perspective. New York, NY: Springer.

13.

E. H.

(2010). Empirically indistinguishable multidimensional IRT and locally dependent unidimensional item response models. British Journal of Mathematical and Statistical Psychology, 63, 395-416.

14.

Johnson

T. R.

Bolt

D. M.

(2010). On the use of factor-analytic multinomial logit item response models to account for individual differences in response style. Journal of Educational and Behavioral Statistics, 35, 92-114.

15.

Jöreskog

Sörbom

(1979). Advances in factor analysis and structural equation models. Cambridge, MA: Abt Books.

16.

Magnus

J. R.

Neudecker

(1999). Matrix differential calculus with applications in statistics and econometrics (2nd ed.). New York, NY: Wiley.

17.

Reckase

M. D.

(2009). Multidimensional item response theory. New York, NY: Springer.

18.

Revuelta

(2008). The generalized logit linear item response model for binary designed items. Psychometrika, 73, 385-405.

19.

Revuelta

(2009). Identifiability and equivalence of GLLIRM models. Psychometrika, 74, 257-272.

20.

Revuelta

(2010). Estimating difficulty from polytomous categorical data. Psychometrika, 75, 331-350.

21.

Rothenberg

(1971). Identification in parametric models. Econometrica, 39, 577-591.

22.

Schilling

Bock

R. D.

(2005). High-dimensional maximum marginal likelihood item factor analysis by adaptive quadrature. Psychometrika, 70, 533-555.

23.

Thissen

Cai

Bock

R. D.

(2010). The nominal categories model. In Nering

M. L.

Ostini

(Eds.), Handbook of polytomous item response theory models (pp. 43-76). New York, NY: Routledge.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.22 MB