On the Estimation of Standard Errors in Cognitive Diagnosis Models

Abstract

Cognitive diagnosis models (CDMs) are an increasingly popular method to assess mastery or nonmastery of a set of fine-grained abilities in educational or psychological assessments. Several inference techniques are available to quantify the uncertainty of model parameter estimates, to compare different versions of CDMs, or to check model assumptions. However, they require a precise estimation of the standard errors (or the entire covariance matrix) of the model parameter estimates. In this article, it is shown analytically that the currently widely used form of calculation leads to underestimated standard errors because it only includes the item parameters but omits the parameters for the ability distribution. In a simulation study, we demonstrate that including those parameters in the computation of the covariance matrix consistently improves the quality of the standard errors. The practical importance of this finding is discussed and illustrated using a real data example.

Keywords

cognitive diagnosis model G-DINA standard errors information matrix

Introduction

Cognitive diagnosis models (CDMs) are restricted latent class models that can be used to analyze response data from educational or psychological tests. In the educational context, they are becoming a popular method for measuring mastery or nonmastery of a set of fine-grained abilities (called attributes) that can be used, for example, to support teachers to recognize strengths and weaknesses of students. Lee, Park, and Taylan (2011) and H. Li (2011) are examples of cognitive diagnostic analyses of mathematics and language skills in large-scale assessments. However, the method has also been suggested to identify the presence or absence of psychological disorders (de la Torre, van der Ark, & Rossi, 2015; Templin & Henson, 2006) or can be used for a detailed measurement of fluid intelligence using abstract reasoning tasks (Rupp, Templin, & Henson, 2010; Yang & Embretson, 2007).

The field of cognitive diagnostic assessments has also become a popular area for methodological research over the past 20 years. Many different versions of CDMs have been proposed to analyze responses from tests with various characteristics (e.g., models for dichotomous and polytomous responses, compensatory, and noncompensatory processes). See Rupp, Templin, and Henson (2010) for a taxonomy of CDMs. Many of these models can be subsumed within a more general framework, such as the generalized deterministic input, noisy “and” gate (G-DINA; de la Torre, 2011) model, the log-linear CDM (Henson, Templin, & Willse, 2009), or the general diagnostic model (von Davier, 2008). Aside from Bayesian approaches, which are presented in the literature for different versions of CDMs (see, e.g., Culpepper, 2015), the model parameters are usually estimated via marginal maximum likelihood estimation (MMLE) using, for example, the expectation maximization (EM) algorithm (Dempster, Laird, & Rubin, 1977; McLachlan & Krishnan, 2007). In the marginal formulation of the model, a probability distribution that models the attribute space is imposed in conjunction with the traditional item response function that models the conditional probability of a correct response given the attributes.

An important step of any practical analysis is to assess the uncertainty of the estimated model parameters using confidence intervals or significance tests. Furthermore, several techniques are available to investigate the model fit or to check the model assumptions of a CDM including tests for (item-level) model comparisons (de la Torre & Lee, 2013) and to detect differential item functioning (Hou, de la Torre, & Nandakumar, 2014). These methods require a precise estimation of the model parameters and their standard errors (or the entire covariance matrix).

However, according to the CDM literature (see, e.g., J. Chen & de la Torre, 2013; de la Torre, 2009, 2011; George, 2013; Rojas, 2013; Song, Wang, Dai, & Ding, 2012) and open-source software implementations (e.g., in the R package cdm, Version 4.991-1), it is common to compute the standard errors only for the parameters that are used to specify the item response function while ignoring the parameters used to specify the joint distribution of the attributes. Consequently, this approach is frequently applied in substantive as well as in many methodological research applications.

Unfortunately, this widely used approach can lead to underestimated standard errors, as we will demonstrate in this article. The aim of this article is to provide detailed guidance on how standard errors for CDMs should be computed correctly. In addition to analytic arguments, we will investigate the quality of the standard errors using simulations.

The severity of the underestimation varies considerably depending on some known factors (e.g., test length and number of attributes in the assessment) as well as unknown factors (e.g., parameters of the item response function and distribution of the attributes). In some situations, the incremental improvement with the correct approach may become negligibly small (e.g., for high test lengths). However, because the factors potentially causing underestimation are manifold, practitioners cannot know upfront whether the data being analyzed are subject to underestimation of standard errors and how severe the underestimation might be. Given that the necessary computations are straightforward, using the correct approach presented in this article is recommended to be on the “safe side.” The additional computations only involve components that are already provided by the results of the estimation routine, and we provide free and open-source software for obtaining the results in practice.

In many situations, the underestimation can seriously deteriorate the quality of confidence intervals and statistical tests. Hou, de la Torre, and Nandakumar (2014), for example, proposed the Wald test to detect differential item functioning in CDMs and encountered serious Type I error inflation (up to 18%). X. Li and Wang (2015) later found that this was caused by a substantial underestimation of the standard errors with the MMLE approach. Although it is not clear whether the underestimation they observed in their study was caused by the incorrect computation of the standard errors or otherwise, it demonstrates how the performance of the Wald test can be negatively affected by underestimated standard errors (or the entire covariance matrix). Several studies in the field of item response theory (IRT) have also demonstrated the influence of the estimation approach on the quality of procedures that require a covariance matrix. Woods, Cai, and Wang (2012), for example, found better controlled Type I error in the Wald test to detect differential item functioning in the Rasch model if the covariance matrix was computed using the supplemented EM algorithm (Cai, 2008).

Other statistical issues might also cause biases in standard errors for CDMs when using MMLE. Similar to traditional latent class analysis, for example, parameter estimates sometimes converge toward the boundary of the parameter space for small data sets. This causes numerical problems in the calculation of the information matrix, which is inverted to get the covariance matrix. Posterior mode (PM) estimation has been suggested to overcome these problems (DeCarlo, 2011; Garre & Vermunt, 2006). However, in the CDM literature and in some frequently used software packages, the traditional maximum likelihood (ML) estimation is prevalent. Therefore, we will focus on the estimation of standard errors in this framework for this article.

The rest of the article is organized as follows. The next section contains a short formal introduction of CDMs before the correct estimation of the standard errors is discussed in detail. Later in that section, the G-DINA model will be introduced for the remaining aspects discussed in the article. In the section after next, the quality of the standard errors is investigated using simulation studies and a real data example. The last section concludes with a discussion. To simplify notation and language, we will focus on CDMs for dichotomous responses in the context of educational assessments for the rest of the article. Please note, however, that the calculation of the standard errors described here holds for all types of CDMs estimated via MMLE.

CDMs

The primary goal in cognitive diagnosis modeling is to infer mastery or nonmastery of K attributes from the responses of each individual to J items in an assessment. For this task, a J × K Q-matrix (Tatsuoka, 1983) must be specified to identify the cognitive specification of the items, where Q = {q_jk} and q_jk = 1 if attribute k (k = 1, …, K) is required to solve item j (j = 1, …, J) and 0 otherwise. The Q-matrix requires domain-specific knowledge and should ideally be specified together with experts from the field for which the assessment will be needed.

Let $X_{i} = {X_{i j}}$ be the binary response pattern of individual i (i = 1, …, N). The conditional probability of a correct response to item j given the unobserved attribute profile $α_{i} = {α_{i k}}$ is parametrized using a specific item response function, denoted by $P_{j} (α_{i}) = Pr (X_{i j} = 1 | α_{i})$ . Furthermore, let δ_j denote the vector of all parameters used to specify $P_{j} (α_{i})$ and, let $δ = {(δ_{1}, \dots, δ_{J})}^{⊺}$ denote the vector of parameters that contains all item parameters. For reasons of consistency, it is usually suggested to estimate δ and α_i using a marginal ML approach (de la Torre, 2009; Neyman & Scott, 1948). The marginal probability is given by the sum over all L = 2^K possible attribute patterns called latent classes:

Pr (X_{i} = x_{i}) = \sum_{l = 1}^{L} p (α_{l}) \cdot Pr (X_{i} = x_{i} | α_{l}),

where $Pr (X_{i} = x_{i} | α_{l}) = \prod_{j = 1}^{J} P_{j} {(α_{l})}^{x_{i j}} {[1 - P_{j} (α_{l})]}^{1 - x_{i j}}$ .

A distribution $p (α_{l})$ is imposed to specify a prior probability for each latent class. Let π be the vector of all parameters used in the model that specifies $p (α_{l})$ . For this article, we choose a saturated model by estimating a probability $π_{l} = p (α_{l})$ for each latent class, where $π_{L} = 1 - \sum_{l = 1}^{L - 1} π_{l}$ . Different models can be assumed to reduce the number of parameters (de la Torre & Douglas, 2004; Rupp et al., 2010).

Thus, let $ϑ = {(δ, π)}^{⊺}$ be the complete vector of all model parameters of a CDM, and further $p = dim (δ)$ and $q = dim (π)$ . The marginal log likelihood that is maximized to estimate ϑ , given the data sample $X = {x_{i}}$ for individuals i = 1, …, N is given by

l (ϑ; X) = log [L (ϑ; X)] = log \prod_{i = 1}^{N} \sum_{l = 1}^{L} π_{l} \cdot Pr (X_{i} = x_{i} | α_{l}),

and can be maximized using the EM algorithm as described in de la Torre (2009). The estimation procedure provides the posterior probability for each latent class, $\hat{Pr} (α_{l} | x_{i})$ , that can be used to find $\hat{π}$ and the attribute profiles ${\hat{α}}_{i}$ . However, the aim of this article is to discuss the estimation of standard errors for the estimated model parameters $\hat{ϑ}$ , which will be the focus of the next section.

Theory and Estimation of Standard Errors

The standard errors of the estimated model parameters $\hat{ϑ} = {(\hat{δ}, \hat{π})}^{⊤}$ can be computed as the square root of the diagonal elements of the covariance matrix of $\hat{ϑ}$ . Regarding the two types of parameters, δ and π, the covariance matrix of $\hat{ϑ}$ can be divided into four blocks:

Cov (\hat{ϑ}) = V_{ϑ} = (\begin{matrix} V_{δ} & V_{δ, π} \\ V_{π, δ} & V_{π} \end{matrix}),

where $V_{δ} = Cov (\hat{δ})$ is the covariance matrix of the parameters used to specify the item response function, $V_{π} = Cov (\hat{π})$ is the covariance matrix of the parameters used to specify the distribution of the latent classes and $V_{δ, π} = V_{π, δ}^{⊤} = Cov (\hat{δ}, \hat{π})$ is the covariance matrix between the two types of parameters.

Complete and incomplete information matrix

The (asymptotic) covariance matrix of $\hat{ϑ}$ is equal to the inverse of the information matrix, $V_{ϑ} = I_{ϑ}^{- 1}$ , which is defined as

I_{ϑ} = E [ψ (ϑ) ψ {(ϑ)}^{⊤}],

where

ψ (ϑ) = {(ψ (δ), ψ (π))}^{⊤} = {(\frac{\partial l (ϑ; x)}{\partial δ_{1}}, \dots, \frac{\partial l (ϑ; x)}{\partial δ_{p}}, \frac{\partial l (ϑ; x)}{\partial π_{1}}, \dots, \frac{\partial l (ϑ; x)}{\partial π_{q}})}^{⊤}

is the score function (i.e., the partial derivatives of the log likelihood with respect to all model parameters).

Similar to the covariance matrix, the information matrix can be divided into four blocks:

I_{ϑ} = (\begin{matrix} I_{δ} & I_{δ, π} \\ I_{π, δ} & I_{π} \end{matrix}) = E [(\begin{matrix} ψ (δ) ψ {(δ)}^{⊤} & ψ (δ) ψ {(π)}^{⊤} \\ ψ (π) ψ {(δ)}^{⊤} & ψ (π) ψ {(π)}^{⊤} \end{matrix})],

where $I_{δ}$ is the information matrix for the parameters used to specify the item response function, $I_{π}$ is the information matrix for the parameters used to specify the distribution of the latent classes, and $I_{δ, π} = I_{π, δ}^{⊤}$ is the information matrix for the two types of parameters.

In many practical applications (e.g., tests for differential item functioning), researchers are primarily interested in the parameters δ, and thus they incorrectly compute the covariance matrix for $\hat{δ}$ via the inverse of the incomplete information matrix $I_{δ}$ . This approach, however, considers only a submatrix of the complete information matrix including all model parameters $I_{ϑ}$ . It is important to note that, since δ and $π$ are generally not mutually independent in CDMs (i.e., $I_{δ, π} = I_{π, δ}^{⊤} \neq 0$ ), inverting the incomplete information matrix $I_{δ}$ systematically underestimates the standard errors for $\hat{δ}$ . In some cases, only the itemwise information matrix $I_{δ_{j}}$ (a submatrix of $I_{δ}$ ) is computed and inverted to get the covariance matrix of the parameter vector $δ_{j}$ . However, similar to traditional IRT models (Yuan, Cheng, & Patton, 2014), $I_{δ}$ is not block diagonal. And thus, inverting the itemwise information matrix underestimates the standard errors even stronger compared to the incomplete information matrix approach.

The above statement can be derived in a formal way using matrix algebra. Let ${(I_{δ})}^{- 1}$ be the covariance of $\hat{δ}$ based on the incomplete information matrix, and let $V_{δ}$ be the covariance of $\hat{δ}$ based on the complete information matrix. From blockwise matrix inversion (see, e.g., Banerjee & Roy, 2014), it follows that

V_{δ} = {(I_{δ})}^{- 1} + Δ,

with $Δ = {(I_{δ})}^{- 1} I_{δ, π} V_{π} I_{π, δ} {(I_{δ})}^{- 1}$ . If the inverse of $I_{ϑ}$ exists¹ and $I_{δ, π} = I_{π, δ}^{⊤} \neq 0$ , then the diagonal elements of all terms in Equation 2 are positive (see Appendix), which implies,

\sqrt{{[V_{δ}]}_{r, r}} > \sqrt{{[{(I_{δ})}^{- 1}]}_{r, r}} r = 1, \dots, p .

This means that the standard errors of the estimated parameters $\hat{δ}$ are consistently underestimated if the incomplete or the itemwise—instead of the complete—information matrix is used. In the next section, we will demonstrate by means of simulations that standard errors computed using the complete information matrix are of better quality. But first, we will discuss two important techniques to estimate the information matrix.

Estimating the information matrix and standard errors

Computing the (expected) information matrix by evaluating the expected value at the ML estimate is infeasible for large assessments. The expectation must be taken over the support of the random response vector x _i, which becomes very large even if J (the number of items) is only moderately large (e.g., J = 25) and computation becomes very slow due to memory limitation.

Thus, the information matrix is often estimated by the empirical counterpart of Equation 1, given by

{J_{ϑ, OPG} = \frac{1}{N} [\sum_{i = 1}^{N} ψ (ϑ; x_{i}) ψ {(ϑ; x_{i})}^{⊤}]}_{} |_{ϑ = \hat{ϑ}},

also known as the “outer product of gradients” (OPG) estimator, where $ψ (ϑ; x_{i})$ is the contribution of individual i to the score function.

Another estimator follows from the fact that under the true parameter values and standard regularity conditions, the information matrix (as defined in Equation 1) is equivalent to the expected value of the negative Hessian matrix of the log likelihood. Thus, the information matrix may also be estimated via

J_{ϑ, Hess} = - \frac{1}{N} [\sum_{i = 1}^{N} \frac{\partial^{2} l (ϑ; x_{i})}{\partial ϑ \partial ϑ^{⊺}}] |_{ϑ = \hat{ϑ}} .

In practice, however, Equations 3 and 4 are evaluated at the estimated parameter values and, thus, the two estimators differ by

J_{ϑ, Hess} - J_{ϑ, OPG} = \frac{1}{N} {[\sum_{i = 1}^{N} \frac{1}{L (ϑ; x_{i})} \frac{\partial^{2} L (ϑ; x_{i})}{\partial ϑ \partial ϑ^{⊺}}] |}_{ϑ = \hat{ϑ}} .

Often Equation 3 is easier to compute, but Equation 4 promises a better finite sample approximation of the information matrix (McLachlan & Krishnan, 2007).

From the above definitions, the standard error for the parameter ϑ_r (r = 1, …, p + q) can be computed via the inverse of the complete information matrix using

\hat{SE} ({\hat{ϑ}}_{r}) = \sqrt{{[{(J_{ϑ, OPG})}^{- 1}]}_{r, r}} or \hat{SE} ({\hat{ϑ}}_{r}) = \sqrt{{[{(J_{ϑ, Hess})}^{- 1}]}_{r, r}},

estimated via the outer products of gradients or the Hessian matrix, respectively. Since the differences between the OPG and the Hessian approach turned out to be negligibly small for simple CDMs (i.e., for the deterministic input, noisy “and” gate [DINA] model introduced below, but results are not shown), we will only consider the OPG estimator for the rest of the article.

In the next section, the improvement of the quality of the standard errors using the inverse of the complete information matrix will be illustrated using three specific versions of CDMs. Therefore, we will briefly introduce the G-DINA model framework proposed by de la Torre (2011), which covers other CDMs as special cases. For a comprehensive description of the framework, its relation to other general CDMs and parameter estimation, we refer the reader to the original article.

The G-DINA Model

A comprehensive and very flexible version of a CDM is the G-DINA model (de la Torre, 2011). Due to its general formulation, it includes many other (more restrictive) CDMs as special cases.

For each item in the assessment, the individuals are separated into $2^{K_{j}^{*}}$ latent groups, where $K_{j}^{*}$ is the number of attributes required by item j (i.e., the sum of the jth row in the Q-matrix). Presence or absence of all the other attributes does not affect the group membership of an individual. Consequently, the attribute vector $α_{i}$ can be reduced to the attributes required by the particular item.

Let $α_{i j}^{*} = (α_{i 1}, \dots, α_{i K_{j}^{*}})$ denote the reduced attribute vector of individual i for item j. The conditional probability to answer item j correctly is then defined by the item response function

P_{j} (α_{i j}^{*}) = h^{- 1} (δ_{j 0} + \sum_{k = 1}^{K_{j}^{*}} δ_{j k} α_{i k} + \sum_{k = 1}^{K_{j}^{*} - 1} \sum_{k^{'} = k + 1}^{K_{j}^{*}} δ_{j k k^{'}} α_{i k} α_{i k^{'}} + \dots + δ_{j 12 \dots K_{j}^{*}} \prod_{k = 1}^{K_{j}^{*}} α_{i k}),

where $h (\cdot)$ is a link function such as identity, log, or logit.

The $δ_{j}$ are the model parameters of item j. In case of the identity link, $δ_{j 0}$ represents the baseline probability for correctly answering item j when none of the required attributes has been mastered (i.e., a lucky guess), $δ_{j k}$ is the main effect that increases (or in rare cases decreases) the probability for correctly answering item j when attribute k has been mastered, and the rest of the parameters represent interaction terms that can increase or decrease the response probability when two or more of the required attributes have been mastered.

Other CDMs can be obtained by restricting the parameters in the G-DINA model. An intuitive, simple, and parsimonious CDM is the DINA (Haertel, 1989; Junker & Sijtsma, 2001) model. In the DINA model, the individuals are separated into two latent groups depending on whether they have mastered all the attributes required to solve the item or not. Thus, the DINA model is a completely noncompensatory (or conjunctive) model, which means that having mastered only part of the required attributes does not increase the probability of answering the item correctly. It can be obtained from the G-DINA model by restricting all parameters except $δ_{j 0}$ and $δ_{j 12 \dots K_{j}^{*}}$ to zero. Thus, δ_j0 = g_j is called the guessing probability, since individuals that have not mastered all attributes required by the item can only guess the correct response. On the other hand, $1 - (δ_{j 0} + δ_{j 12 \dots K_{j}^{*}}) = s_{j}$ is called the slip probability, since in this probabilistic model individuals who have mastered all attributes required by the item may still slip and give the wrong response.

Another CDM that can be obtained from the G-DINA model is the additive CDM (A-CDM). It is slightly more flexible than the DINA model because the conditional response probability can increase (or in some cases decrease) for each attribute that has been mastered. It can be obtained from the G-DINA model by restricting all interaction parameters to zero.

Score contributions for parameters in the G-DINA model

To estimate the information matrix of the model parameters of the G-DINA model via OPG, the contributions of individual i to the score function, $ψ (ϑ; x_{i})$ , are required. They are given by the first-order derivative of the casewise log-likelihood contribution with respect to the model parameters:

ψ (ϑ; x_{i}) = \frac{\partial l (ϑ; x_{i})}{\partial ϑ} = \frac{\partial log L (ϑ; x_{i})}{\partial ϑ} = \frac{1}{L (ϑ; x_{i})} \cdot \frac{\partial L (ϑ; x_{i})}{\partial ϑ} = \frac{1}{L (ϑ; x_{i})} \cdot \frac{\partial}{\partial ϑ} (\sum_{l = 1}^{L} π_{l} \cdot Pr (x_{i} | α_{l})) .

Using formula (A6) from the appendix in de la Torre (2009) for the partial derivative of the conditional likelihood, the score contributions of the parameters of item j can be computed via

\frac{\partial l (ϑ; x_{i})}{\partial δ_{j}} = \sum_{l = 1}^{L} Pr (α_{l} | x_{i}) \cdot [\frac{x_{i j} - P_{j} (α_{l j}^{*})}{P_{j} (α_{l j}^{*}) (1 - P_{j} (α_{l j}^{*}))}] \cdot \frac{\partial P_{j} (α_{l j}^{*})}{\partial δ_{j}} .

To estimate the score contributions, we plug-in the estimated parameters ${\hat{δ}}_{j}$ to get $P_{j} (α_{l j}^{*})$ and use $Pr (α_{l} | x_{i})$ that is also available from the estimation procedure. The last term in Equation 5 depends on the type of CDM that is used. It is also possible to compute the score contributions directly for the conditional response probabilities. In this case, the last term in Equation 5 needs to be derived with respect to the conditional response probability of interest.

For the score contributions of the latent class probabilities, the constraint $π_{L} = 1 - \sum_{l = 1}^{L - 1} π_{l}$ must be taken into account, and thus,

\frac{\partial l (ϑ; x_{i})}{\partial π_{l}} = \frac{1}{L (ϑ; x_{i})} \frac{\partial}{\partial π_{l}} (\sum_{l = 1}^{L - 1} π_{l} \cdot Pr (x_{i} | α_{l}) + π_{L} \cdot Pr (x_{i} | α_{L})) = \frac{1}{L (ϑ; x_{i})} \frac{\partial}{\partial π_{l}} (\sum_{l = 1}^{L - 1} π_{l} \cdot Pr (x_{i} | α_{l}) + (1 - \sum_{l = 1}^{L - 1} π_{l}) \cdot Pr (x_{i} | α_{L})) = \frac{1}{L (ϑ; x_{i})} \frac{\partial}{\partial π_{l}} (\sum_{l = 1}^{L - 1} π_{l} \cdot (Pr (x_{i} | α_{l}) - Pr (x_{i} | α_{L})) + Pr (x_{i} | α_{L})) = \frac{1}{L (ϑ; x_{i})} (Pr (x_{i} | α_{l}) - Pr (x_{i} | α_{L})) .

Since the parameters in the last iteration of the EM algorithm are computed from the posterior values $Pr (α_{l} | x_{i})$ , it is more precise to also compute the score function for the latent class probabilities using the posterior values via

\frac{\partial l (ϑ; x_{i})}{\partial π_{l}} = \frac{1}{π_{l}} (Pr (α_{l} | x_{i}) - Pr (α_{L} | x_{i})) .

Nonidentifiability of latent classes

In the theory about standard errors of parameters that is presented above, it is assumed that the inverse of the complete information matrix $I_{ϑ}$ exists. This, however, is not always the case in practical applications due to different causes. The most common cause has previously been discussed in Haertel (1989) as the nonidentifiability of latent classes. The problem arises whenever a test does not involve a single-attribute item for each of the K attributes (see Chiu, Douglas, & Li, 2009, for a theoretical discussion of the completeness of a Q-matrix in the DINA model, and Chiu & Köhn, 2015, for CDMs in general). The G-DINA model can still be estimated, but some of the latent classes are not identified and the estimates of the corresponding latent class probabilities are equivalent. Moreover, when computing the covariance matrix using the complete information matrix, the corresponding columns and rows in the information matrix are alike (i.e., they are linearly dependent). Thus, the information matrix is nonsingular and cannot be inverted.

To avoid problems of identification in practice, it is therefore recommended that, whenever possible, a single-attribute item is included for each of the K attributes when developing new tests for cognitive diagnostic assessment. For researchers who perform a cognitive diagnostic analysis of data from an existing assessment (so-called retrofitting), the inversion problem can be circumvented by pooling latent classes that cannot be separated from each other.

Illustrations

Following the theoretical derivation of the underestimation of the standard errors—resulting from the inversion of the incomplete or the itemwise information matrix—the goal of this section is to illustrate the extent of this underestimation and its effect on confidence intervals for the parameter estimates. In addition, we show for an exemplary real data set how much the standard errors may be underestimated in practice when the wrong methods are used. For both illustrations, the OPG estimator was used to estimate the covariance matrix of the model parameter estimates.

Coverage Study

In the first study, we compare the quality of the standard error estimates based on the complete, the incomplete, and the itemwise information matrix by estimating the coverage probability of the true parameter in a Wald-type confidence interval that uses a normal approximation given by $[\hat{ϑ} \pm z_{\frac{α}{2}} \cdot \hat{SE} (\hat{ϑ})]$ and by computing the empirical bias of the standard errors.

Four different sample sizes $(N = 500, 1, 000, 2, 000, and 5, 000)$ were investigated using the Q-matrix given in Table 1. The Q-matrix included five attributes and was constructed such that each attribute was measured equally often (equal row sums in the table) and that the number of items that required the same number of attributes was equally distributed (i.e., five single-attribute items, five two-attribute items, and five three-attribute items). Thus, the Q-matrix represented a test with J = 15 items.

Table 1.

Transposed Q-Matrix Used in the Simulation Study

Attributes	Items															∑_k
Attributes	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	∑_k
α ₁	1	0	0	0	0	1	0	0	0	1	1	0	0	1	1	6
α ₂	0	1	0	0	0	1	1	0	0	0	1	1	0	1	0	6
α ₃	0	0	1	0	0	0	1	1	0	0	1	1	1	0	0	6
α ₄	0	0	0	1	0	0	0	1	1	0	0	1	1	0	1	6
α ₅	0	0	0	0	1	0	0	0	1	1	0	0	1	1	1	6
∑_j	1	1	1	1	1	2	2	2	2	2	3	3	3	3	3

The DINA model and the A-CDM were used to generate response data. For each item, the true value of the baseline parameter $(δ_{j 0})$ was set to .2. In case of the DINA model, the true value of the interaction parameter between all attributes required by the item $(δ_{j 12 \dots K_{j}^{*}})$ was set to .6. Therefore, the guessing and the slip probabilities were both equal to .2. In case of the A-CDM, the main effect parameters were set to $δ_{j k} = 0.6 / K_{j}^{*}$ . Thus, with each additionally mastered attribute, the conditional response probability increased by the same amount. The K attributes for each individual were sampled independently from a Bernoulli distribution with probability $Pr (α_{k} = 1) = 0.5$ , for all k = 1, …, K. The joint distribution of the attributes (i.e., the latent class distribution) is then given by a categorical distribution with equal probabilities $π_{l} = Pr (α_{l}) = 1 / (2^{K})$ . Responses that were simulated under the DINA model were analyzed using the DINA and the G-DINA model using the identity link. Note that the G-DINA is also correct for data that were generated under the DINA model. It was fitted in addition to the DINA model because in practice the true model is usually unknown. However, in this situation, the G-DINA model is overspecified, due to the many additional parameters estimated, for which the true values are zero according to the data generating model. Responses that were simulated under the A-CDM were accordingly analyzed using the A-CDM and the G-DINA model using the identity link. Again, the G-DINA is also correct—yet overspecified—for data generated under the A-CDM. To estimate the models and the standard errors, the EM algorithm was implemented in R (R Core Team, 2016) based on the description in de la Torre (2009) but including our new suggestions on how the standard errors should be estimated.

Figures 1 and 2 illustrate the coverage probabilities for the data generated under the DINA model and the A-CDM, respectively. For all sample sizes and models, the coverage probabilities were computed for the δ parameters using standard errors based on the complete information matrix $J_{ϑ}$ (correct approach) and the incomplete information matrix $J_{δ}$ and the itemwise information matrix $J_{δ_{j}}$ (incorrect approaches). It turned out that the asymptotically expected standard errors of the item parameters are identical across items that require the same number of attributes. In the DINA model, for example, the baseline (guessing) probabilities of all single-attribute items share the same asymptotic standard error, no matter which of the attributes is required. This also holds for other item parameters, items that require more attributes, and different models. Therefore, the coverage probabilities were averaged over the parameters within those groups, which are illustrated on the x-axis of the graph. The parameter group “0,” for example, represents the baseline probability of all single-attribute items. The parameter group “111” represents the parameter of the three-way interaction of all three-attribute items.

Figure 1.

Coverage probabilities of 95% Wald-type confidence intervals for data generated under the deterministic input, noisy “and” gate model are illustrated (on the y-axis) separately for parameters of items that require the same number of attributes (=parameter groups on the x-axis) using three different calculation methods for the standard errors. For ease of readability, values sufficiently close to the nominal coverage probability are depicted as solid circles, all others as empty circles.

Figure 2.

Coverage probabilities of 95% Wald-type confidence intervals for data generated under the additive CDM are illustrated (on the y-axis) separately for parameters of items that require the same number of attributes (=parameter groups on the x-axis) using three different calculation methods for the standard errors. For ease of readability, values sufficiently close to the nominal coverage probability are depicted as solid circles, all others as empty circles.

By definition, the coverage probability of a 95% confidence interval has an expected nominal coverage rate of 95%. However, due to sampling error, the estimated coverage probabilities may randomly deviate from this nominal value. To achieve a high precision of the estimated coverage probabilities, each configuration was repeated 10,000 times. Assuming an exact binomial distribution for the coverage probabilities, the sampling error was equal to $\sqrt{\frac{0.95 \cdot 0.05}{10, 000}} \approx 0.002$ . Thus, based on a Wald-type confidence interval, we would consider coverage probabilities within $[94.6 %, 95.4 %]$ as sufficiently close to the nominal rate. Numbers within this interval are depicted with solid circles (otherwise empty circles) in Figures 1 and 2.

Additionally, Tables 2 through 5 list the exact values of the coverage probabilities and the empirical bias of the standard errors for the smallest (N = 500) and the largest (N = 5,000) sample sizes (the intermediate sample sizes were omitted for brevity but can be requested from the corresponding author) and each parameter group (labeled by η). The average empirical bias corresponds to the average of the empirical biases over all replications that were computed by subtracting the estimated standard errors $\hat{SE} ({\hat{ϑ}}_{r})$ from the empirical standard error of the estimated parameter values over all replications.

Table 2.

Coverage Probabilities of 95% Wald-Type Confidence Intervals and Average Estimated Bias of the Standard Errors for Data Generated Under the DINA Model and Fitted to the DINA Model

N	η	Coverage Probabilities			Average Estimated Bias
N	η	Complete	Incomplete	Itemwise	Complete	Incomplete	Itemwise
500	0	.9261	.7877	.7571	.0061	.0248	.0275
	1	.9315	.8902	.8662	.0056	.0147	.0191
	00	.9468	.9148	.8974	.0004	.0037	.0050
	11	.9256	.9057	.8849	.0044	.0094	.0136
	000	.9541	.9397	.9312	−.0006	.0007	.0014
	111	.9334	.9010	.8796	.0036	.0127	.0173
5,000	0	.9556	.8467	.8289	−.0005	.0051	.0057
	1	.9511	.9328	.9228	−.0002	.0016	.0024
	00	.9511	.9213	.9126	.0000	.0009	.0011
	11	.9504	.9399	.9299	.0000	.0008	.0016
	000	.9504	.9423	.9394	.0000	.0002	.0003
	111	.9494	.9313	.9262	.0000	.0019	.0024

Note. DINA = deterministic input, noisy “and” gate.

Table 3.

Coverage Probabilities of 95% Wald-Type Confidence Intervals and Average Estimated Bias of the Standard Errors for Data Generated Under the DINA Model and Fitted to the Overspecified G-DINA Model

N	η	Coverage Probabilities			Average Estimated Bias
N	η	Complete	Incomplete	Itemwise	Complete	Incomplete	Itemwise
500	0	.7666	.6343	.5850	.0304	.0440	.0482
	1	.8162	.7782	.7198	.0346	.0403	.0483
	00	.8715	.8403	.8121	.0194	.0231	.0269
	10	.8244	.7861	.7465	.0441	.0512	.0583
	01	.8265	.7871	.7477	.0444	.0515	.0587
	11	.8250	.7960	.7527	.0622	.0702	.0824
	000	.9040	.8335	.7982	.0443	.0643	.0712
	100	.8981	.8413	.8022	.0505	.0790	.0903
	010	.8969	.8368	.7984	.0534	.0854	.0973
	001	.8973	.8383	.7993	.0508	.0801	.0917
	110	.8920	.8415	.8041	.0548	.0980	.1159
	101	.8829	.8312	.7906	.0643	.1069	.1252
	011	.8913	.8407	.8014	.0549	.0994	.1175
	111	.9130	.8820	.8603	.0411	.0999	.1270
5,000	0	.9449	.8213	.7910	.0004	.0063	.0073
	1	.9444	.9319	.9082	.0006	.0018	.0036
	00	.9451	.9423	.9394	.0004	.0005	.0007
	10	.9429	.9304	.9243	.0009	.0024	.0030
	01	.9432	.9301	.9235	.0010	.0025	.0031
	11	.9426	.9402	.9327	.0014	.0019	.0030
	000	.9402	.9361	.9342	.0017	.0019	.0021
	100	.9399	.9377	.9357	.0027	.0029	.0033
	010	.9391	.9369	.9348	.0028	.0030	.0034
	001	.9390	.9370	.9348	.0028	.0030	.0034
	110	.9368	.9337	.9313	.0044	.0051	.0057
	101	.9365	.9340	.9314	.0046	.0053	.0059
	011	.9394	.9363	.9340	.0039	.0047	.0053
	111	.9366	.9355	.9329	.0061	.0066	.0077

Note. DINA = deterministic input, noisy “and” gate; G-DINA = generalized deterministic input, noisy “and” gate.

Table 4.

Coverage Probabilities of 95% Wald-Type Confidence Intervals and Average Estimated Bias of the Standard Errors for Data Generated Under the A-CDM and Fitted to the A-CDM

N	η	Coverage Probabilities			Average Estimated Bias
N	η	Complete	Incomplete	Itemwise	Complete	Incomplete	Itemwise
500	0	.6698	.5205	.4813	.0496	.0633	.0678
	1	.7137	.6659	.6030	.0687	.0757	.0843
	00	.7999	.7179	.6852	.0268	.0352	.0382
	10	.8463	.8096	.7778	.0252	.0308	.0350
	01	.8484	.8124	.7817	.0247	.0302	.0343
	000	.8817	.8153	.7923	.0139	.0243	.0270
	100	.8971	.8526	.8299	.0134	.0217	.0250
	010	.8972	.8521	.8304	.0142	.0225	.0258
	001	.8973	.8525	.8294	.0137	.0220	.0252
5,000	0	.9343	.7922	.7111	.0016	.0099	.0127
	1	.9340	.9307	.8597	.0025	.0029	.0087
	00	.9380	.8411	.8200	.0009	.0061	.0069
	10	.9424	.9341	.9102	.0007	.0014	.0031
	01	.9442	.9357	.9119	.0006	.0013	.0029
	000	.9439	.8878	.8815	.0005	.0041	.0044
	100	.9483	.9403	.9296	.0003	.0010	.0017
	010	.9453	.9373	.9258	.0004	.0011	.0019
	001	.9471	.9394	.9300	.0002	.0008	.0016

Note. A-CDM = additive CDM.

Figure 1 shows the coverage probabilities for the data generated under the DINA model. When the DINA model was used to analyze the data (see left column in Figure 1 and exact values reported in Table 2), the coverage probabilities for the standard errors based on the complete information matrix (solid line) were reasonably close to the expected coverage rate for small data samples and converged quickly toward the nominal rate with increasing sample size N. The coverage rates for the standard errors based on the incomplete (dashed line) or the itemwise (dotted line) information matrix, however, were systematically smaller than the nominal coverage probability, particularly for the first parameter groups. Even for the largest sample size considered, their coverage probability does not converge toward the nominal rate. This is caused by the structural underestimation of the standard errors discussed earlier. We observed the largest underestimation for the baseline probabilities of single-attribute items (parameter group “0”). For the other parameters, the difference to the correct approach is smaller but still lower than for the correct approach and notably below the nominal rate. A similar pattern can be observed when the G-DINA model was used to analyze the data generated under the DINA model (see right column in Figure 1 and exact values reported in Table 3). However, for smaller sample sizes, the coverage probabilities were generally estimated considerably below the nominal coverage rate of 95%. This artifact may be explained by several circumstances. First, the normal approximation underlying the Wald-type confidence intervals might fail, particularly for the baseline probabilities that are restricted between zero and one. Second, for smaller data sets and more complex models, the conditional response probabilities and the parameters used to specify the attribute distribution are often estimated on the boundary of the parameter space. As mentioned earlier, this causes numerical problems in the calculation of the information matrix. Finally, the ratio between the number of estimated parameters per observation is larger for more general models. Thus, inferior asymptotic convergence has to be reckoned with the G-DINA when compared to the DINA model. Nevertheless, the complete information matrix approach clearly provided more accurate results in all the conditions considered.

Similar and related conclusions can be drawn from the average empirical biases reported in Tables 2 and 3. They were (in absolute terms) always smaller when the complete information matrix instead of the incomplete or the itemwise information matrix approaches were used. Please note that when the correct DINA model was fitted to the simulated data (see values reported in Table 2), and when the standard errors were estimated with the complete information matrix approach, the bias almost completely vanished for the larger sample size, whereas with the two incorrect approaches still had significant biases at the larger sample size. This, however, was not the case when the overspecified G-DINA model was fitted to the data simulated under the DINA model (see values reported in Table 3). The average estimated biases reported for the complete information matrix approach did not coverage zero, although it was always smaller than for the incomplete and the itemwise approaches. Despite this finding, the complete information matrix approach provided the most accurate standard error estimates of all estimation approaches considered in this study.

Figure 2 shows the coverage probabilities for the data generated under the A-CDM (for exact values, see Tables 4 and 5) . For the same reasons as discussed above, the coverage probabilities were estimated below the nominal rate for smaller samples. As the sample size increased, the coverage probabilities computed with the standard errors based on the complete information matrix again approached the nominal rate for the A-CDM and the G-DINA model. The coverage probabilities computed with the standard errors based on the incomplete or the itemwise information matrix, however, were again systematically underestimated. Overall, the complete information matrix approach again provided more accurate results across all conditions considered.

The average empirical biases reported in Tables 4 and 5 were again always smaller with the complete information approach for all parameter groups and sample sizes. However, as discussed above for the data simulated under the DINA model, the bias did not converge toward zero for the larger sample size, when the overspecified G-DINA model was used to estimate the data simulated under the A-CDM (see values reported in Table 5).

Table 5.

Coverage Probabilities of 95% Wald-Type Confidence Intervals and Average Estimated Bias of the Standard Errors for Data Generated Under the Additive CDM and Fitted to the Overspecified G-DINA Model

N	η	Coverage Probabilities			Average Estimated Bias
N	η	Complete	Incomplete	Itemwise	Complete	Incomplete	Itemwise
500	0	.7126	.5377	.4815	.0453	.0614	.0678
	1	.7497	.6909	.6029	.0639	.0720	.0843
	00	.8160	.7332	.6877	.0322	.0412	.0464
	10	.8017	.7649	.7190	.0679	.0762	.0853
	01	.8027	.7663	.7225	.0680	.0762	.0854
	11	.7604	.7300	.6874	.1240	.1344	.1476
	000	.8890	.8336	.7918	.0228	.0405	.0479
	100	.8622	.7997	.7579	.0612	.0913	.1048
	010	.8583	.7933	.7516	.0642	.0957	.1095
	001	.8587	.7978	.7562	.0628	.0922	.1056
	110	.8312	.7598	.7120	.1125	.1642	.1867
	101	.8344	.7640	.7145	.1136	.1649	.1873
	011	.8269	.7573	.7087	.1168	.1676	.1899
	111	.8049	.7044	.6430	.1607	.2422	.2772
5,000	0	.9417	.7945	.7111	.0008	.0098	.0127
	1	.9355	.9322	.8597	.0023	.0027	.0087
	00	.9453	.9013	.8898	.0005	.0040	.0047
	10	.9402	.9335	.9234	.0015	.0024	.0037
	01	.9393	.9332	.9226	.0016	.0025	.0038
	11	.9393	.9340	.9300	.0029	.0041	.0048
	000	.9410	.9219	.9184	.0017	.0039	.0042
	100	.9372	.9299	.9262	.0038	.0051	.0058
	010	.9365	.9283	.9244	.0042	.0056	.0063
	001	.9362	.9290	.9253	.0040	.0053	.0060
	110	.9315	.9267	.9243	.0081	.0095	.0103
	101	.9323	.9275	.9247	.0081	.0094	.0102
	011	.9299	.9254	.9228	.0087	.0101	.0109
	111	.9259	.9234	.9207	.0166	.0178	.0189

Note. G-DINA = generalized deterministic input, noisy “and” gate.

Empirical Example

To illustrate the practical importance of estimating standard errors via the complete information matrix, data from a real assessment were analyzed using CDMs. The data stem from a learning experiment at the University of Tuebingen in Germany and is available in the R package pks (Heller & Wickelmaier, 2013). The participants were required to answer 12 items about elementary probability theory. For example, “A box contains 30 marbles in the following colors: 8 red, 10 black, 12 yellow. What is the probability that a randomly drawn marble is yellow?” Four different attributes (concepts) were tested: how to calculate

the classic probability of an event (pb)?

the probability of the complement of an event (cp)?

the probability of the union of two disjoint events (un)?

the probability of two independent events (id)?

These concepts were combined to form the 12 items. Therefore, the Q-matrix (see Table 6) was defined by the design of the items. The first 4 items required only one attribute, Items 5 through 10 required two attributes, and Items 11 and 12 required three attributes. For this illustration, the responses of 504 participants from the first part of the experiment were analyzed.

Table 6.

Transposed Q-Matrix Used for Analyzing the Elementary Probability Theory Data

Attributes	Items
Attributes	1	2	3	4	5	6	7	8	9	10	11	12	∑_k
pb	1	0	0	0	1	1	1	1	1	0	1	1	8
cp	0	1	0	0	1	1	0	0	0	1	1	0	5
un	0	0	1	0	0	0	1	1	0	0	0	1	4
id	0	0	0	1	0	0	0	0	1	1	1	1	5
∑_j	1	1	1	1	2	2	2	2	2	2	3	3

The data were fitted using the DINA, the A-CDM, and the G-DINA model with the resulting Bayesian information criterion (BIC) values of 5,200.46 (df = 39), 5,154.58 (df = 49), and 5,241.70 (df = 63), respectively. The results of the A-CDM—which had the lowest BIC value—are illustrated in Table 7. The table summarizes the estimated parameters; the corresponding standard errors based on the complete, the incomplete, and the itemwise information matrix; and the relative change in the standard errors between the correct and the two incorrect approaches (in parentheses).

Table 7.

Estimates and Standard Errors of Parameters for the Elementary Probability Theory Data

Item	Attributes	Estimate	Standard Errors
Item	Attributes	Estimate	Complete	Incomplete	Itemwise
1	Baseline	0.224	0.065	0.061 (−0.071)	0.052 (−0.203)
1	pb	0.710	0.067	0.063 (−0.061)	0.055 (−0.186)
2	Baseline	0.275	0.105	0.080 (−0.241)	0.068 (−0.356)
2	cp	0.699	0.105	0.081 (−0.232)	0.069 (−0.346)
3	Baseline	0.097	0.060	0.055 (−0.082)	0.048 (−0.194)
3	un	0.864	0.061	0.056 (−0.082)	0.050 (−0.188)
4	Baseline	0.125	0.038	0.035 (−0.072)	0.032 (−0.159)
4	id	0.837	0.039	0.037 (−0.064)	0.034 (−0.140)
5	Baseline	0.201	0.067	0.055 (−0.187)	0.048 (−0.288)
	pb	0.364	0.116	0.101 (−0.130)	0.094 (−0.191)
	cp	0.293	0.125	0.111 (−0.116)	0.103 (−0.181)
6	Baseline	0.194	0.062	0.058 (−0.058)	0.051 (−0.185)
	pb	0.462	0.085	0.080 (−0.053)	0.074 (−0.125)
	cp	0.308	0.083	0.081 (−0.021)	0.077 (−0.071)
7	Baseline	0.278	0.071	0.068 (−0.049)	0.062 (−0.126)
	pb	0.292	0.095	0.088 (−0.078)	0.083 (−0.127)
	un	0.372	0.116	0.105 (−0.094)	0.097 (−0.164)
8	Baseline	0.430	0.087	0.076 (−0.132)	0.063 (−0.277)
	pb	0.065	0.095	0.066 (−0.297)	0.059 (−0.371)
	un	0.462	0.111	0.088 (−0.212)	0.079 (−0.293)
9	Baseline	0.116	0.045	0.043 (−0.042)	0.038 (−0.145)
	pb	0.510	0.084	0.079 (−0.060)	0.074 (−0.113)
	id	0.154	0.075	0.070 (−0.065)	0.065 (−0.124)
10	Baseline	0.083	0.050	0.044 (−0.115)	0.037 (−0.248)
	cp	−0.056	0.060	0.055 (−0.086)	0.048 (−0.190)
	id	0.781	0.036	0.035 (−0.027)	0.034 (−0.062)
11	Baseline	0.053	0.049	0.045 (−0.086)	0.038 (−0.229)
	pb	0.010	0.106	0.086 (−0.184)	0.080 (−0.244)
	cp	−0.037	0.094	0.084 (−0.109)	0.078 (−0.173)
	id	0.672	0.034	0.033 (−0.030)	0.032 (−0.060)
12	Baseline	0.000	0.039	0.036 (−0.090)	0.029 (−0.269)
	pb	0.140	0.469	0.191 (−0.592)	0.169 (−0.640)
	un	0.000	0.452	0.181 (−0.600)	0.162 (−0.643)
	id	0.660	0.046	0.042 (−0.067)	0.042 (−0.084)

Note. Strongest relative changes are printed in bold letters for better readability. Numbers in parentheses correspond to the relative change to the standard errors based on the complete information matrix. pb = the classic probability of an event, cp = the probability of the complement of an event, un = the probability of the union of two disjoint events, id = the probability of two independent events.

For each item, the first parameter estimate represents the baseline probability (i.e., the probability of correctly answering the item when the attributes required by the item have not been mastered). Thus, large values for this guessing probability are unusual. For Item 8, however, a value of over .4 is reported. A possible explanation is that the item—“What is the probability of obtaining an odd number when throwing a dice?”—was not very difficult, even for individuals without knowledge in basic probability theory. Further parameter estimates represent the amount of increase (or seldom decrease) in probability of answering an item correctly when the corresponding attribute had been mastered. For example, the probability of answering Item 1 increased by .71 when attribute “pb” had been mastered.

The relative change between the standard errors based on the complete and the incomplete information matrix showed substantial differences (highlighted by bold letters in Table 7) for both parameters of the single-attribute Item 2; for some of the parameters of the two-attribute Items 5, 8, and 10; and for some of the parameters of the three-attribute Items 11 and 12. The underestimation of the standard errors based on the itemwise information matrix was even worse. For 30 of the 34 item parameters, the standard error was underestimated.

It should be noted that 10 of the 48 conditional response probabilities and 4 of the 16 parameters of the latent class probabilities were estimated at the boundary of the parameter space (not displayed in Table 7). As mentioned earlier, this can cause numerical problems in computing the information matrix. According to the previous simulation study, where a similar scenario was investigated (see top-left panel in Figure 2 for the same model and a nearly equal sample size), it must be assumed that some of the standard errors reported for these data are generally underestimated. Nevertheless, just like in the simulation study—and as expected from our theoretical considerations—the additional severe underestimation caused by the wrong computation of the information matrix can easily be avoided using the complete information matrix.

Discussion

Standard errors are an important measure to quantify the uncertainty of an estimate. They are required for many different statistical techniques to evaluate model fit or to check model assumptions. It is therefore crucial in practical research to estimate standard errors as precisely as possible. In the commonly used approach for computing standard errors in CDMs, however, the information matrix is based only on those parameters that are used to specify the item response function. The parameters used to specify the joint distribution of the attributes (i.e., latent class distribution) are not incorporated in the computation.

In this article, we have shown that with this approach, the standard errors for the parameters of the item response function are systematically underestimated. We therefore strongly recommend to compute the standard errors based on the complete information matrix, which also includes the parameters used to specify the latent class distribution. In addition to the clear theoretical result, we have illustrated by means of simulations that our approach leads to a higher quality of Wald-type confidence intervals and lower empirical bias. An additional benefit of using the complete instead of the incomplete information matrix is that it also provides the information required to compute standard errors for the parameters used to specify the latent class distribution.

We assume that the incomplete information matrix approaches have only become widely used in the CDM literature because previous authors might have assumed that the off-diagonal elements of the information matrix would have negligible impact under certain conditions. With respect to the itemwise computation of the standard errors, the CDM literature may be partially influenced by the traditional IRT literature, where approaches exist that lead to block diagonal information matrices (e.g., in Thissen & Wainer, 1982), in which case an itemwise computation of the standard errors is possible. However, for CDMs, as we showed analytically and illustrated with examples, the complete information matrix approach clearly generates better standard errors than the incomplete and the itemwise approaches and is computationally well feasible. Similar to our results, Yuan, Cheng, and Patton (2014) showed that the itemwise computation of the standard errors in IRT models also leads to undersized standard errors.

In the simulation study, we did not specifically vary design factors such as the Q-matrix, the true values of the item parameters, or the latent class distribution. Varying these factors might positively or negatively affect the severity of underestimation. In a preliminary study with the DINA model, we found that longer tests and highly discriminating items can alleviate the underestimation. It should be highlighted, however, that the proposed method for estimating the standard errors cannot make the quality of the standard errors worse. In practical situations, however, it is difficult (or even impossible) to control the factors that have a large impact on the underestimation. As such, it is always preferable to compute standard errors using the complete information matrix.

We note that differences between the approaches are expected not only for the standard errors but for the entire covariance matrix of the model parameters (although not generally in the same direction). Thus, many techniques used to investigate a fitted model may be affected. The impact of under- or overestimation of the entire covariance matrix will be multiplied for multivariate methods. It is therefore worth in any circumstances to estimate standard errors (and also the entire covariance matrix) from the complete information matrix. As we did not specifically investigate the impact of misestimating the entire covariance matrix on multivariate techniques, it will be interesting for future research to investigate how much the quality of the covariance matrix can be improved using the complete information matrix in computing it.

The results of the simulation study revealed problems of asymptotic convergence when more complex models were fitted to smaller data sets. This might partially be caused by boundary problems that often occur for smaller data sets. DeCarlo (2011) suggested PM estimation to overcome these problems. Whether PM estimation leads to more accurate parameter and standard error estimates than the traditional ML approach in CDMs was not the scope of this work but something that can be investigated in future research. Moreover, the normal approximation of the ML estimates might be more accurate on the real line under the logit link rather than on the (bounded) probability scale under the identity link. However, this not only concerns the estimation of standard errors but of the model parameters in general. Therefore, this is beyond the scope of this article and is not pursued here. It might be of interest for future research, though, to explore the potential benefits of different link functions. In general, the results from our simulation study suggest that it is recommended to use simpler models whenever possible and appropriate because it may avoid boundary problems or problems with asymptotic convergence.

Finally, in the present article, we assumed that the Q-matrix is known or well specified for an assessment. However, in practice (especially when retrofitting CDMs to existing data), the Q-matrix may be unknown or misspecified, which can affect parameter estimation and classification accuracy (de la Torre, 2008; Rupp & Templin, 2007). To minimize the impact of a misspecified Q-matrix, several methods have been proposed. De la Torre (2008) proposed an iterative procedure to evaluate the correctness of the Q-matrix specification in the context of the DINA model. The approach was extended by de la Torre and Chiu (2016) to apply generally to other CDMs. Other recent approaches include that of Y. Chen, Liu, Xu, and Ying (2015), which estimates the Q-matrix of the DINA model using regularization, whereas Chiu (2013) proposed a nonparametric approach to Q-matrix validation that does not require specifying the exact form of the CDM, only that the underlying process is conjunctive in nature. Future research should examine the extent of the impact of Q-matrix misspecifications on standard error estimation, and whether specific steps can be taken to minimize such an impact.

Computational Details

The estimation routines used in this study were written in the free and open-source software R (R Core Team, 2016) for statistical computing. Functions to estimate the parameters and the standard errors in the G-DINA model are provided in the form of the add-on package Rcdm, available online at https://github.com/mphili/cdm under the terms of the GNU General Public License (Versions 2 or 3).

Footnotes

Appendix

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article

Note

References

Banerjee

Roy

(2014). Linear algebra and matrix analysis for statistics. Boca Raton, FL: CRC Press.

Cai

(2008). SEM of another flavour: Two new applications of the supplemented EM algorithm. British Journal of Mathematical and Statistical Psychology, 61, 309–329. doi:10.1348/000711007X249603

Chen

de la Torre

(2013). A general cognitive diagnosis model for expert-defined polytomous attributes. Applied Psychological Measurement, 37, 419–437. doi:10.1177/0146621613479818

Chen

Liu

Ying

(2015). Statistical analysis of Q-matrix based diagnostic classification models. Journal of the American Statistical Association, 110, 850–866. doi:10.1080/01621459.2014.934827

Chiu

C.-Y.

(2013). Statistical refinement of the Q-matrix in cognitive diagnosis. Applied Psychological Measurement, 37, 598–618. doi:10.1177/0146621613488436

Chiu

C.-Y.

Douglas

J. A.

(2009). Cluster analysis for cognitive diagnosis: Theory and applications. Psychometrika, 74, 633–665. doi:10.1007/s11336-009-9125-0

Chiu

C.-Y.

Köhn

H.-F.

(2015). A general proof of consistency of heuristic classification for cognitive diagnosis models. British Journal of Mathematical and Statistical Psychology, 68, 387–409. doi:10.1111/bmsp.12055

Culpepper

S. A.

(2015). Bayesian estimation of the DINA model with Gibbs sampling. Journal of Educational and Behavioral Statistics, 40, 454–476. doi:10.3102/1076998615595403

DeCarlo

L. T.

(2011). On the analysis of fraction subtraction data: The DINA model, classification, latent class sizes, and the Q-matrix. Applied Psychological Measurement, 35, 8–26. doi:10.1177/0146621610377081

10.

de la Torre

(2008). An empirically based method of Q-matrix validation for the DINA model: Development and applications. Journal of Educational Measurement, 45, 343–362. doi:10.1111/j.1745-3984.2008.00069.x

11.

de la Torre

(2009). DINA model and parameter estimation: A didactic. Journal of Educational and Behavioral Statistics, 34, 115–130. doi:10.3102/1076998607309474

12.

de la Torre

(2011). The generalized DINA model framework. Psychometrika, 76, 179–199. doi:10.1007/s11336-011-9207-7

13.

de la Torre

Chiu

C.-Y.

(2016). A general method of empirical Q-matrix validation. Psychometrika, 81, 253–273. doi:10.1007/s11336-015-9467-8

14.

de la Torre

Douglas

J. A.

(2004). Higher-order latent trait models for cognitive diagnosis. Psychometrika, 69, 333–353. doi:10.1016/j.pse.2014.11.001

15.

de la Torre

Lee

Y.-S.

(2013). Evaluating the Wald test for item-level comparison of saturated and reduced models in cognitive diagnosis. Journal of Educational Measurement, 50, 355–373. doi:10.1111/jedm.12022

16.

de la Torre

van der Ark

L. A.

Rossi

. (2015). Analysis of clinical data from cognitive diagnosis modeling framework. Measurement and Evaluation in Counseling and Development. Advance online publication. doi:10.1177/0748175615569110

17.

Dempster

A. P.

Laird

N. M.

Rubin

D. B

. (1977). Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society. Series B (Methodological), 39, 1–38.

18.

Garre

F. G.

Vermunt

J. K.

(2006). Avoiding boundary estimates in latent class analysis by Bayesian posterior mode estimation. Behaviormetrika, 33, 43–59. doi:10.2333/bhmk.33.43

19.

George

A. C

. (2013). Investigating CDMs: Blending theory with practicality (Doctoral dissertation ). TU Dortmund University, Dortmund, Germany. Retrieved from https://eldorado.tu-dortmund.de/

20.

Haertel

E. H.

(1989). Using restricted latent class models to map the skill of achievement structure items. Journal of Educational Measurement, 26, 301–321. doi:10.1111/j.1745-3984.1989.tb00336.x

21.

Harville

D. A.

(2008). Matrix algebra from a statistician’s perspective. New York, NY: Springer.

22.

Heller

Wickelmaier

(2013). Minimum discrepancy estimation in probabilistic knowledge structures. Electronic Notes in Discrete Mathematics, 42, 49–56. doi:10.1016/j.endm.2013.05.145

23.

Henson

R. A.

Templin

J. L.

Willse

J. T.

(2009). Defining a family of cognitive diagnosis models using log-linear models with latent variables. Psychometrika, 74, 191–210. doi:10.1007/s11336-008-9089-5

24.

Hou

de la Torre

Nandakumar

(2014). Differential item functioning assessment in cognitive diagnostic modeling: Application of the Wald test to investigate DIF in the DINA model. Journal of Educational Measurement, 51, 98–125. doi:10.1111/jedm.12036

25.

Junker

B. W.

Sijtsma

(2001). Cognitive assessment models with few assumptions, and connections with nonparametric item response theory. Applied Psychological Measurement, 25, 258–272. doi:10.1177/01466210122032064

26.

Lee

Y.-S.

Park

Y. S.

Taylan

. (2011). A cognitive diagnostic modeling of attribute mastery in Massachusetts, Minnesota, and the U.S. national sample using the TIMSS 2007. International Journal of Testing, 11, 144–177. doi:10.1080/15305058.2010.534571

27.

(2011). A cognitive diagnostic analysis of the MELAB reading test. Spaan Fellow Working Papers in Second or Foreign Language Assessment, 9, 17–46.

28.

Wang

W.-c.

(2015). Assessment of differential item functioning under cognitive diagnosis models: The DINA model example. Journal of Educational Measurement, 52, 28–54. doi:10.1111/jedm.12061

29.

McLachlan

Krishnan

(2007). The EM algorithm and extensions (2nd ed.). New York, NY: Wiley.

30.

Neyman

Scott

E. L.

(1948). Consistent estimates based on partially consistent observations. Econometrica, 16, 1–32. doi:10.2307/1914288

31.

R Core Team. (2016). R: A language and environment for statistical computing [Computer software manual]. Vienna, Austria. Retrieved from http://www.R-project.org/

32.

Rojas

. (2013). Cognitive diagnosis models: Attribute classification, differential item functioning and applications (Doctoral dissertation). Universidad Autónomia de Madrid, Madrid, Spain. Retrieved from https://repositorio.uam.es/

33.

Rupp

A. A.

Templin

(2007). The effects of Q-matrix misspecification on parameter estimates and classification accuracy in the DINA model. Educational and Psychological Measurement, 68, 78–96. doi:10.1177/0013164407301545

34.

Rupp

A. A.

Templin

Henson

R. A.

(2010). Diagnostic measurement: Theory, methods, and applications. New York, NY: Guilford Press.

35.

Song

Wang

Dai

Ding

(2012). The revised DINA model parameter estimation with EM algorithm. International Journal of Digital Content Technology and Its Applications, 6, 85–92. doi:10.4156/jdcta.vol6.issue9.11

36.

Tatsuoka

K. K.

(1983). Rule space: An approach for dealing with misconceptions based in item response theory. Journal of Educational Measurement, 20, 345–354. doi:10.1111/j.1745-3984.1983.tb00212.x

37.

Templin

J. L.

Henson

R. A.

(2006). Measurement of psychological disorders using cognitive diagnosis models. Psychological Methods, 11, 287–305. doi:10.1037/1082-989X.11.3.287

38.

Thissen

Wainer

(1982). Some standard errors in item response theory. Psychometrika, 47, 397–412. doi:10.1007/bf02293705

39.

von Davier

(2008). A general diagnostic model applied to language testing data. British Journal of Mathematical and Statistical Psychology, 61, 287–307. doi:10.1348/000711007X193957

40.

Woods

C. M.

Cai

Wang

(2012). The Langer-improved Wald test for DIF testing with multiple groups: Evaluation and comparison to two-group IRT. Educational and Psychological Measurement, 73, 532–547. doi:10.1177/0013164412464875

41.

Yang

Embretson

S. E.

(2007). Construct validity and cognitive diagnostic assessment. In Leighton

J. P.

Gierl

M. J.

(Eds.), Cognitive diagnostic assessment for education: Theory and applications (pp. 119–145). New York, NY: Cambridge University Press.

42.

Yuan

K. H.

Cheng

Patton

(2014). Information matrices and standard errors for MLEs of item parameters in IRT. Psychometrika, 79, 232–254. doi:10.1007/s11336-013-9334-4