Latent Transition Cognitive Diagnosis Model With Covariates: A Three-Step Approach

Abstract

To expand the use of cognitive diagnosis models (CDMs) to longitudinal assessments, this study proposes a bias-corrected three-step estimation approach for latent transition CDMs with covariates by integrating a general CDM and a latent transition model. The proposed method can be used to assess changes in attribute mastery status and attribute profiles and to evaluate the covariate effects on both the initial state and transition probabilities over time using latent (multinomial) logistic regression. Because stepwise approaches generally yield biased estimates, correction for classification error probabilities is considered in this study. The results of the simulation study showed that the proposed method yielded more accurate parameter estimates than the uncorrected approach. The use of the proposed method is also illustrated using a set of real data.

Keywords

latent transition analysis cognitive diagnosis models G-DINA model bias-correction three-step approach covariates

With the rapid proliferation of cognitive diagnosis models (CDMs), an increasing number of researchers choose CDMs as the psychometric tool to analyze assessment data because they can provide more informative feedback to support students’ learning. CDMs have been used to examine students’ strengths and weaknesses in many educational subject domains, such as proportional reasoning (Ma et al., 2020), spatial skills (Culpepper, 2015), and digital literacy (Liang et al., 2021). In addition to educational assessments, some researchers used CDMs in clinical assessments for diagnostic purposes (e.g., de la Torre et al., 2018; Tan et al., 2022). A family of CDMs can be found in the literature, including specific and general CDMs. Specific CDMs can be classified as either conjunctive, disjunctive, or additive based on different cognitive assumptions or theories about how attributes function in examinees’ response behaviors. The deterministic inputs, noisy “and” gate (DINA; Junker & Sijtsma, 2001) model, the deterministic inputs, noisy “or” gate (DINO; Templin & Henson, 2006) model, and additive CDM (A-CDM; de la Torre, 2011) are the examples of conjunctive, disjunctive, and additive models, respectively. General CDMs, which have more flexible formulations that subsume some commonly used specific CDMs, have also been developed. The examples of general CDMs include the generalized DINA (G-DINA; de la Torre, 2011), the general diagnostic model (von Davier, 2005), and the log-linear CDM (LCDM; Henson et al., 2009).

Although a large number of existing CDMs are currently available, most of these models and their applications focus on the classification of attributes at a single time point. However, to measure examinees’ changes across multiple time points or before and after a specific event (e.g., the outbreak of COVID-19), longitudinal or pre-/post-test designs should be used. Several latent transition CDMs that can be used with longitudinal assessment data have been developed. Kaya and Leite (2017) and Li et al. (2016) combined latent transition analysis (LTA; Collins & Lanza, 2010) with restricted CDMs (i.e., DINA and DINO models) to assess examinees’ changes in attribute mastery in repeated measurements. Chen et al. (2018) developed a Bayesian procedure of a first-order hidden Markov model in conjunction with the DINA model to track learning trajectories. To relax item parameter constraints, Madison and Bradshaw (2018a, 2018b) proposed the transition diagnostic classification model that combines LTA with a general CDM, namely, the LCDM, to assess growth in attribute mastery status across time points. Using another general CDM, namely, the G-DINA model, Yigit and Douglas (2021) employed a first-order Markov model to track students’ learning trajectories.

In addition to longitudinal CDMs, integrating covariates with CDMs needs research attention because education researchers and practitioners may also be interested in the associations between covariates and attribute mastery statuses. Several studies on this topic for single time-point design employing a one-step or three-step approach can be found in the literature. The one-step approach estimates the CDM (measurement model) and the regression model (structural model) simultaneously. For example, logistic regression is used in the DINA and higher order DINA (de la Torre & Douglas, 2004) models to estimate how covariates affect the probabilities of mastering each attribute (Ayers et al., 2013; Park & Lee, 2014). Although the one-step approach provides more accurate estimates, it lacks model flexibility because any modifications to either component require refitting the entire model. Moreover, such an approach is not applicable to secondary research because model selection, item parameter estimation, and examinee classification have already been determined. In contrast, the three-step approach offers greater flexibility in modeling the relationship between covariates and latent class membership or attribute mastery, where CDM estimation, latent class membership assignment, and subsequent regression are implemented in separate steps. Iaconangelo and de la Torre (2016) proposed a corrected three-step latent logistic regression approach (Vermunt, 2010) to explore the associations between covariates and attribute mastery statuses or mastery profiles, where correction due classification errors are made.

Although equally important, incorporating covariates into longitudinal CDMs has not yet been extensively studied. To date, limited work has been done on modeling the association between covariates and attribute mastery transitions. Wang et al. (2018) proposed a higher order hidden Markov model with covariates to assess changes in attribute mastery status and investigate how covariates affect attribute transitions simultaneously using a one-step approach. As mentioned above, this approach lacks model flexibility. Moreover, this particular model neglects to account for the effects of covariates on the initial state and the profile-level relationships. To provide a more flexible approach of incorporating covariates into longitudinal CDMs, this research proposes a three-step estimation approach for latent transition CDM with covariates using the G-DINA model framework, which allows for the investigation of how covariates may affect both the initial state and transition probabilities. Latent logistic or multinomial regression is employed to investigate attribute-level or profile-level associations between covariates and initial state and transition probabilities. Because stepwise approach underestimates the effects of covariates (Bolck et al., 2004; Di Mari et al., 2016; Vermunt, 2010), correction of classification error probabilities (CEPs) is also taken into consideration in the present study. Therefore, the corrected three-step approach for latent transition CDM with covariates involves three steps: (1) fitting a CDM (measurement model) to the response data at each time point, (2) assigning examinees to latent states at each time point and computing the CEP, and (3) estimating the model with the CEP and computing the regression coefficients.

The rest of this article is laid out as follows. The next section gives a review of the G-DINA model framework and LTA. The third and fourth sections elaborate on the proposed latent transition CDM with covariates and the three-step estimation, respectively. A simulation study for performance evaluation and a real data example for illustration purpose are given in the fifth and sixth sections, respectively. Finally, this article concludes with a discussion of the findings and future directions.

Overview and Background

G-DINA Model

Let $Y_{i} = (Y_{i 1}, \dots, Y_{i j}, \dots, Y_{i J})$ , $i = 1, 2, \dots, N$ , $j = 1, 2, \dots, J$ , denote examinee i‘s binary responses to J items, where the 1s and 0s represent the correct and incorrect answers, respectively. The Q-matrix (Tatsuoka, 1983) is a $J \times K$ binary matrix delineating the item-attribute relationships, where K is the number of attributes. For item j, element $q_{j k}$ is 1 if attribute k is required to answer item j correctly and is 0 otherwise. In CDMs, K attributes produce $2^{K}$ latent classes with the unique attribute profile denoted by $α_{l}$ , where $α_{l} = (α_{l 1}, \dots, α_{l k}, \dots, α_{l K})$ , and $l = 1, 2, \dots, 2^{K}$ . Each examinee is assigned to one of the latent classes. The G-DINA model is a general CDM that subsumes several commonly used CDMs (de la Torre, 2011). Several reduced CDMs (e.g., DINA model, DINO model, A-CDM) can be derived by imposing constraints to parameters of the saturated G-DINA model. The G-DINA model can also be equivalent to other general CDMs (e.g., log CDM, LCDM) based on different link functions. For convenience, the reduced attribute profile, denoted by $α_{l j}^{*}$ , is used instead of the complete attribute profile in the G-DINA model. The reduced attribute profile only includes the required attributes for item j and $α_{l j}^{*} = (α_{l j 1}, \dots, α_{l j k}, \dots, α_{l j K_{j}^{*}})$ , where $K_{j}^{*} = \sum_{k = 1}^{K} q_{j k}$ is the number of required attributes for answering item j correctly. For general models, each item has $2^{K_{j}^{*}}$ parameters and assigns examinees from the $2^{K}$ latent classes into the $2^{K_{j}^{*}}$ latent groups. The associated probability of success of each latent group is defined by $P (Y_{j} = 1 | α_{l j}^{*}) = P (α_{l j}^{*})$ , and its item response function is given by

\begin{matrix} P (α_{l j}^{*}) = δ_{j 0} + \sum_{k = 1}^{K_{j}^{*}} δ_{j k} α_{l k} + \sum_{k^{'} = k + 1}^{K_{j}^{*}} \sum_{k = 1}^{K_{j}^{*} - 1} δ_{j k k^{'}} α_{l k} α_{l k^{'}} + \dots + δ_{j 12 \dots K_{j}^{*}} \prod_{k = 1}^{K_{j}^{*}} α_{l k}, \end{matrix}

where $δ_{j 0}$ is the intercept for item j, representing the probability of answering item j correctly without mastering any attributes; $δ_{j k}$ is the main effect of attribute k; $δ_{j k k^{'}}$ is the two-way interaction effect between attribute k and $k^{'}$ ; and $δ_{j 12 \dots K_{j}^{*}}$ is the highest order interaction effect of all required attributes.

Latent Transition Analysis

LTA, also referred as latent or hidden Markov model (Baum & Petrie, 1966), which is a longitudinal analog of latent class model (LCM), has been developed to model not only the initial latent class membership but also the transitions of latent class membership over time (Collins & Lanza, 2010). To differentiate LTA and LCM, we use the term latent state instead of latent class to represent examinees’ temporal states at each time point. Specifically, in LTA, the measurement model is an LCM modeling item response probability at each time point, and the structural model characterizes the latent state prevalence and the changes between latent states across time points. Examples that use LTA can be found in Collins and Lanza (2010) and Lanza and Collins (2002). By combining CDMs with LTA, researchers can study how examinees transition between latent states (attribute profiles or attribute mastery statuses) across time points (e.g., Kaya & Leite, 2017; Li et al., 2016; Madison & Bradshaw, 2018a, 2018b).

Furthermore, incorporating covariates into LTA allows researchers to predict initial latent state membership and latent state transitions. For example, Lanza et al. (2010) used LTA with covariates to model transitions in substance use behavior profiles and found that students’ background and academic characteristics were the significant predictors of substance use behavior profiles and transitions in behavior profiles. Addition work using covariates in LTA can be found in Chung et al. (2007) and Wang et al. (2018). In the context of educational assessments, using latent transition CDMs with covariates can help identify the characteristics that are related to the classification of students’ latent states and the transitions between latent states over time, which can provide useful information for remediation and classroom instruction at different time points. As mentioned earlier, a more general model that allows covariates to affect both initial state and transition probabilities should be studied. Additionally, although the estimation is more accurate, estimating the measurement and structural models simultaneously in one step lacks model flexibility. In contrast, stepwise approaches offer greater flexibility by estimating the measurement and structural models separately. Regular CDM analyses (e.g., Q-matrix validation, model selection) can be implemented in stepwise approaches and adding or dropping covariates can be easily done as well. Because directly treating latent class or state assignment as observed variables produces biased estimates of the regression coefficients, researchers have developed the corrected three-step approach, where classification errors are accounted for (Di Mari et al., 2016; Iaconangelo & de la Torre, 2016; Vermunt, 2010).

A Latent Transition CDM With Covariates

Let T denote the number of time points, $Y_{i t}$ the examinee i’s response at time t, $t = 1, \dots, T,$ Y _t the response data on J items at time t, and Y the full response data at all T time points. Also, let L_t and $s_{t}$ denote the latent state and a possible latent state, respectively, for time t, where $s_{t} = 1, \dots, S$ , and $L_{i t}$ and $s_{i t}$ denote the latent state and a possible latent state of examinee i at time t, respectively. In the context of CDMs, $S = 2^{K}$ when the transition of $2^{K}$ latent states is of interest and $S = 2$ when the transition between attribute mastery and nonmastery is of interest. Assume that a set of covariates are related to the classification of latent state membership. The set of covariates at time t is denoted by Z _t , and the full set of covariates at all T time points is denoted by $Z$ . The latent transition CDM specifying the item response probabilities at T time points, given the time-specific covariates, is formulated as

\begin{matrix} P (Y | Z) = \sum_{s_{1} = 1}^{S} \sum_{s_{2} = 1}^{S} \dots \sum_{s_{T} = 1}^{S} P (L_{1} = s_{1} | Z_{1}) \prod_{t = 2}^{T} P (L_{t} = s_{t} | L_{t - 1} = s_{t - 1}, Z_{t}) \\ \prod_{t = 1}^{T} P (Y_{t} | L_{t} = s_{t}) . \end{matrix}

This latent transition model consists of two components: a measurement component, $P (Y_{t} | L_{t} = s_{t})$ , which estimates the latent state membership at each time point from the observed response data, which is the G-DINA model, and a structural component, which describes changes in latent states across time points. The covariates affect this model through the structural component, which involves the initial state probability $P (L_{1} = s_{1} | Z_{1})$ and the transition probability $P (L_{t} = s_{t} | L_{t - 1} = s_{t - 1}, Z_{t})$ , given the values of the covariates at times 1 and t. Figure 1 describes the latent transition CDM with covariates as in Equation 2.

Figure 1.

Latent transition cognitive diagnosis model with covariates.

Logistic regression models are used to parameterize the initial state probabilities and transition probabilities, given the covariates values at each time point. When profile-level associations are of interest, taking the first latent state, namely, the attribute profile of all zeros, as the reference category, the initial state probability is given by

\begin{matrix} P (L_{1} = s_{1} | Z_{1}) = \frac{\exp (β_{0 s_{1}} + β_{s_{1}}^{'} Z_{1})}{1 + \sum_{s_{1} = 2}^{S} \exp (β_{0 s_{1}} + β_{s_{1}}^{'} Z_{1})}, \end{matrix}

where $β_{0 s_{1}}$ is the intercept, and $β_{s_{1}} = (β_{1 s_{1}}, \dots, β_{f s_{1}}, \dots, β_{F s_{1}})$ is the regression coefficients, and F is the number of covariates.

At each time point, the transition probability is an $S \times S$ matrix, where rows represent the latent states at time $t - 1$ and columns represent the latent states at time t. Each row of the transition probability matrix sums to 1. When covariates are used to predict transition probabilities, each row corresponds to a logistic regression equation. Generally, taking the first latent state as the reference category, the probability of transition from latent state $L_{t - 1}$ to L_t given the covariates at time t can be written as

\begin{matrix} P (L_{t} = s_{t} | L_{t - 1} = s_{t - 1}, Z_{t}) = \frac{\exp (γ_{0 s_{t} | s_{t - 1}} + γ_{s_{t} | s_{t - 1}}^{'} Z_{t})}{1 + \sum_{s_{t} = 2}^{S} \exp (γ_{0 s_{t} | s_{t - 1}} + γ_{s_{t} | s_{t - 1}}^{'} Z_{t})}, \end{matrix}

where $1 < t \leq T$ . For each row of the transition probability matrix, $γ_{0 s_{t} | s_{t - 1}}$ is the intercept and $γ_{s_{t} | s_{t - 1}} = (γ_{1 s_{t} | s_{t - 1}}, \dots, γ_{f s_{t} | s_{t - 1}}, \dots, γ_{F s_{t} | s_{t - 1}})$ is the regression coefficients for latent state s_t except the one that has been designated as the reference category (i.e., $s_{t} = 1$ ).

When the transition between attribute mastery and nonmastery is of interest, the initial state probability of being classified into the mastery latent state of $α_{k}$ , taking nonmastery as the reference category, can be written as

\begin{matrix} P (α_{k 1} = 1 | Z_{1}) = \frac{\exp (β_{0 k} + β_{k}^{'} Z_{1})}{1 + \exp (β_{0 k} + β_{k}^{'} Z_{1})}, \end{matrix}

and $P {(α}_{k 1} = 0 | Z_{1}) = 1 - P (α_{k 1} = 1 | Z_{1})$ , where $β_{k} = (β_{1 k}, \dots, β_{f k}, \dots, β_{F k})$ are the regression coefficients associated with the covariates at time 1.

In the educational assessment context, attribute mastery is typically assumed to be an absorbing state, that is, an attribute does not revert back to its nonmastery state once it has been mastered (Yigit & Douglas, 2021), which is also assumed in this study. However, the model in Equation 2 can still be estimated without the monotonicity assumption. In the current study, for profile-level transition, the transition probability of examinees being classified into a lower latent state at time t is 0. Specifically, when $s_{t} ≺ s_{t - 1}$ , $P (L_{t} = s_{t} | L_{t - 1} = s_{t - 1}, Z_{t}) = 0$ . For example, if $s_{t - 1} =$ (1, 1, 0), $s_{t}$ ϵ {(1, 1, 0), (1, 1, 1)}. For attribute-level transition, as shown in Table 1, the transition probability matrix of $α_{k}$ is a $2 \times 2$ matrix. The first row of the transition probability matrix given in Table 1, namely, the probability of transition from nonmastery to mastery is computed by

Table 1.

Example of Transition Probability Matrix of $α_{k}$

		Time t
	Mastery Status	0	1
Time $t - 1$	0	$P (α_{k t} = 0 \| α_{k t - 1} = 0, Z_{t})$	$P (α_{k t} = 1 \| α_{k t - 1} = 0, Z_{t})$
Time $t - 1$	1	0	1

\begin{matrix} P (α_{k t} = 1 {|α}_{k t - 1} = 0, Z_{t}) = \frac{\exp (γ_{0 k t} + γ_{k t}^{'} Z_{t})}{1 + \exp (γ_{0 k t} + γ_{k t}^{'} Z_{t})}, \end{matrix}

also taking the nonmastery state at time t as the reference category, where $1 < t \leq T$ , and $P (α_{k t} = 0 | α_{k t - 1} = 0) = 1 - P (α_{k t} = 1 | α_{k t - 1} = 0)$ . The second row indicates that the probability of transitioning from mastery back to nonmastery state is zero.

With regard to the objective function of Equation 2, assuming that the tests have been taken by N examinees for T time points, the log-likelihood function of the latent transition model is

\begin{matrix} l o g L = \sum_{i = 1}^{N} \log P (Y_{i} | Z_{i}) . \end{matrix}

When covariates are involved in LTA, the regression coefficients ( $β$ and $γ$ ) will be estimated instead of the initial state probabilities and transition probabilities, as they are expressed as the functions of the regression coefficients and individuals’ values of the corresponding covariates. Generally, the regression coefficients are estimated by maximizing Equation 7. When a stepwise approach is used, the biased estimates of the covariate effects on the initial state and transition probabilities should be taken into consideration. In the next section, we introduce a bias-corrected three-step estimation approach for the latent transition CDM with covariates based on Di Mari et al. (2016).

The Three-Step Approach

The three-step approach for latent transition CDM with covariates involves the following steps: (1) fitting a CDM to the response data without covariates at each time point separately, (2) assigning examinees to latent states at each time point and computing the associated CEPs, and (3) estimating the latent transition CDM with the known CEPs (Di Mari et al., 2016; Vermunt, 2010) and computing the regression coefficients in Equations 3 and 4 or Equations 5 and 6. In the first step, the repeated response data from the same group of examinees are treated as independent datasets at each time point and estimated using the G-DINA model separately. Q-matrix validation and modification, model selection, and item parameter estimation can also be carried out in this step. Item parameters are constrained to be equal across time points to ensure longitudinal measurement invariance, which can avoid classification problems and allow for interpretable results over time.

In the second step, examinees are classified into latent states (discrete), given their responses at each time point using the expected a posteriori (EAP; Huebner & Wang, 2011) method. In this study, we focus on mean assignment (i.e., EAP) only, and other assignment rules (e.g., modal and proportional assignment) can be found in Goodman (2007). Because the stepwise approach will yield biased estimates of the covariate effects on the initial state and transition probabilities, classification errors are accounted for, and correction weights introduced in latent state membership assignment. To address this problem, Bolck et al. (2004), Di Mari et al. (2016), and Vermunt (2010) proposed the three-step approach with correction weights for LCM and LTA. Iaconangelo and de la Torre (2016) also introduced correction weights in the three-step approach in analyzing CDMs with covariates to obtain more reliable parameter estimates.

To obtain correction weights, CEP matrix needs to be computed. CEP matrix estimates the amount of misclassification in the measurement model conditional on the true latent state/class memberships. Denote the predicted or assigned latent state by W_t . The CEP matrix is computed using the examinee latent state assignment (i.e., EAP) and posterior distribution at each time point and can be written as

\begin{matrix} P (W_{t} = r_{t} | L_{t} = s_{t}, Y_{t}) = \frac{\frac{1}{N T} \sum_{i = 1}^{N} \sum_{t = 1}^{T} P (L_{i t} = s_{i t} | Y_{i t}) I [r_{i t} = s_{i t}]}{P (L_{t} = s_{t} | Y_{t})}, \end{matrix}

where $I [r_{i t} = s_{i t}]$ is the indicator function equal to 1 when the estimated latent state $r_{i t}$ of examinee i is equal to latent state $s_{i t}$ and 0 otherwise. $P (L_{i t} = s_{i t} | Y_{i t})$ is the posterior probability that examinee i is in $s_{i t}$ , and $P (L_{t} = s_{t} | Y_{t})$ is the mastery proportion of state s_t estimated by the G-DINA model. The CEP yields a $2^{K} \times 2^{K}$ matrix containing the proportions of examinees with a true latent state membership of s_t assigned to latent state r_t , as in, the rows represent the true latent states and columns the estimated latent states.

When attribute-level mastery status is the latent state of interest, the CEP reduces to a $2 \times 2$ matrix. The estimated attribute mastery status is denoted by $α_{q}$ and the true attribute mastery status by $α_{k}$ , which are equal to 0 or 1. Note that the notation of $α_{q}$ and $α_{k}$ here is analogous to the notation r_t and s_t , respectively, used in Equation 8. The attribute-level CEP at time t is calculated as

\begin{matrix} P (α_{q t} {|α}_{k t}, Y_{t}) = \frac{\frac{1}{N T} \sum_{i = 1}^{N} \sum_{t = 1}^{T} P (α_{i k t} | Y_{i t}) I [α_{i q t} = α_{i k t}]}{P (α_{k t} | Y_{t})}, \end{matrix}

where $P (α_{k t} | Y_{t})$ is the posterior proportion of $α_{k t}$ estimated by the G-DINA model. The attribute-level CEP represents the probabilities of the true attribute mastery status $α_{k}$ (rows) being classified into the observed attribute mastery status $α_{q}$ (columns). For example, the entries of the first column ( $α_{q} = 0$ ) are $P (α_{q} = 0 | α_{k} = 0, Y) = \frac{\frac{1}{N T} \sum_{i = 1}^{N} \sum_{t = 1}^{T} P (α_{i k t} = 0 | Y_{i t}) I [α_{i q t} = 0]}{P (α_{k t} = 0 | Y_{t})}$ and

$P (α_{q} = 0 | α_{k} = 1, Y) = \frac{\frac{1}{N T} \sum_{i = 1}^{N} \sum_{t = 1}^{T} P (α_{i k t} = 1 | Y_{i t}) I [α_{i q t} = 0]}{P (α_{k t} = 1 | Y_{t})}$ , where $P (α_{q} = 0 | α_{k} = 0, Y)$ is interpreted as the probability of examinee i being correctly classified into the nonmastery state $α_{k}$ , and $P (α_{q} = 0 | α_{k} = 1, Y)$ is the probability of examinee i who has mastered $α_{k}$ being incorrectly classified as a nonmaster of $α_{k}$ .

According to Iaconangelo and de la Torre (2016) and Vermunt (2010), the sample-level correction weights for examinee i are calculated based on the CEP matrix and the examinee’s estimated latent state. When attribute profiles are the latent states of interest, the sample-level correction weight of examinee i is given by

\begin{matrix} w_{r_{i t}}^{s_{t}} = P (W_{t} = r_{i t} | L_{t} = s_{t}, Y), \end{matrix}

which is the element of the profile-level CEP matrix at the s _t th row and $r_{i t}$ th column, where $r_{i t}$ is the estimated latent state of the ith examinee and s_t denotes the true latent state. Similarly, the attribute-level correction weight of examinee i is given by

\begin{matrix} w_{α_{i q t}}^{α_{k t}} = P (α_{i q t} {|α}_{k t}, Y), \end{matrix}

which is the element of the attribute-level CEP matrix at the $α_{k t}$ th row and $α_{i q t}$ th column.

Finally, the third step estimates the relationships between the covariates and latent state memberships and transition probabilities. Although of interest is the relationship between the true latent state $L$ and covariates $Z$ , the relationship between the estimated latent state $W$ and $Z$ is analyzed instead in the third step. The conditional distribution of $W$ given $Z$ by marginalizing over $L$ and $Y$ is written as

\begin{matrix} P (W | Z) = \sum_{s_{1} = 1}^{S} P (L_{1} = s_{1} | Z_{1}) \prod_{t = 2}^{T} P (L_{t} = s_{t} | L_{t - 1} = s_{t - 1}, Z_{t}) \prod_{t = 1}^{T} P (W_{t} | L_{t}), \end{matrix}

where $P (W_{t} | L_{t})$ is obtained in the second step. We can estimate the parameters of the structural model by maximizing the corresponding log-likelihood, which can be done using the uncorrected and corrected three-step approaches.

For the uncorrected three-step approach, the estimated latent state membership W_t is used directly as $P (W_{t} | L_{t})$ . Hence, the uncorrected objective function is written as

\begin{matrix} L_{s t e p 3 (uncorrected)} = \sum_{i = 1}^{N} log \sum_{s_{1} = 1}^{S} \dots \sum_{s_{T} = 1}^{S} P (L_{1} = s_{1} | Z_{1}) \prod_{t = 2}^{T} P (L_{t} = s_{t} | L_{t - 1} = s_{t - 1}, Z_{t}) \\ \prod_{t = 1}^{T} W_{t}, \end{matrix}

where W_t is the estimated latent state assignment and is the EAP estimates obtained using the G-DINA model. In the uncorrected three-step approach, which does not take CEP into consideration, W_t is treated as an observed dependent variable in the (multinomial) logistic regression function.

In contrast, in the third step of the corrected three-step approach, the correction weights $w_{r_{i t}}^{s_{t}}$ are used instead of W_t (as illustrated in Figure 2). The corrected objective function is given by

Figure 2.

Three-step latent transition cognitive diagnosis model with covariates using correction weights (Step 3). Note. The relationship between $Z_{t}$ and $L_{t}$ can be estimated by calculating the correction weight as a single indicator. $Z_{t}$ = covariates at time t; L_t = true latent state at time t; w _t = correction weight at time t.

\begin{matrix} L_{s t e p 3 (corrected)} = \sum_{i = 1}^{N} log \sum_{s_{1} = 1}^{S} \dots \sum_{s_{T} = 1}^{S} P (L_{1} = s_{1} | Z_{1}) \prod_{t = 2}^{T} P (L_{t} = s_{t} | L_{t - 1} = s_{t - 1}, Z_{t}) \\ \prod_{t = 1}^{T} w_{r_{i t}}^{s_{t}}, \end{matrix}

where $w_{r_{i t}}^{s_{t}}$ is the correction weight of examinee i’s corresponding to the latent state s_t (also see Equations 10 and 11). If the attribute-level association is of interest, it is straightforward to use $w_{α_{i q t}}^{α_{k t}}$ as the correction weight.

If researchers expect to update the classification results by combining the classification information obtained from CDM estimation ( $P (α_{i k t} | Y_{i t})$ ; the first step) and transition probabilities from the logistic regressions, $L (Z_{i t} | α_{i k t})$ , Bayes’s theorem can be applied (Sun & de la Torre, 2020). The updated posterior probability of examinee i mastering $α_{k}$ at time t is given by

P (α_{i k t} | Y_{i t}, Z_{i t}) = \frac{L (Z_{i t} | α_{i k t}) P (α_{i k t} | Y_{i t})}{\sum_{α_{i k t} = 0}^{1} L (Z_{i t} | α_{i k t}) P (α_{i k t} | Y_{i t})},

where $P (α_{i k t} | Y_{i t})$ is the marginal posterior probability obtained using the G-DINA model. Based on the updated posterior probability $P (α_{i k t} | Y_{i t}, Z_{i t})$ , the classification results can be updated using EAP.

Simulation Study

Design

A simulation study was conducted to evaluate the performance of the proposed three-step estimation approach for latent transition CDM with covariates. The performance of the bias-corrected three-step approach was compared to that of the uncorrected three-step approach. This simulation study was also designed to demonstrate that using directly the estimated latent states as observed variables in regression model produces biased parameter estimates and to investigate the ability of the correction weights to improve the regression parameter estimates. Because the one-step estimation procedure for latent transition CDM with covariates, which does currently not available in the literature, is computationally complicated and difficult to optimize, we did not compare the proposed three-step approach to the one-step approach in the current simulation study. Furthermore, we consider both time-constant and time-varying covariates. The number of time points was fixed at two, which can be viewed as a pre/posttest design. To ensure measurement invariance over time, item parameters at the posttest were constrained to be equal to those estimated at the pretest. Lastly, the transition probability of an attribute was assumed to be independent of other attributes.

Because this study is a longitudinal extension to CDM with covariates (Iaconangelo & de la Torre, 2016), we chose the factors based on their study. Five factors were manipulated in the simulation study. The sample sizes used study were N = 500, 1,000, and 2,000; number of attributes K = 3 and 5, with $max (K_{j}^{*}) = 3$ ; and test length, which was varied for different numbers of attributes, J = 10 and 20 for K = 3, and J = 25 or J = 50 for K = 5. The Q-matrix for K = 3 and J = 10 is given in Table 2 and was doubled for J = 20, whereas the Q-matrix for K = 5 and J = 25 is given in Table 3 and was doubled for J = 50. The Q-matrices satisfied the necessary and sufficient conditions for model identifiability (Gu & Xu, 2021). Additionally, the item quality based on the lowest (p ₀) and highest (p ₁) probability of success of a given item was set to be low, medium, and high. Based on Ma and colleagues (2016), p ₀ and $1 - p_{1}$ ∼ U (0.05, 0.15), U (0.15, 0.25), and U (0.25, 0.35) for all items representing high, medium, and low item quality, respectively.

Table 2.

Q-Matrix for K = 3 and J = 10

Item	A1	A2	A3
1	1	0	0
2	0	1	0
3	0	0	1
4	1	0	0
5	0	1	0
6	0	0	1
7	1	1	0
8	1	0	1
9	0	1	1
10	1	1	1

Note. The Q-matrix was doubled for K = 3 and J = 20.

Table 3.

Q-Matrix for K = 5 and J = 25

Item	A1	A2	A3	A4	A5
1	1	0	0	0	0
2	0	1	0	0	0
3	0	0	1	0	0
4	0	0	0	1	0
5	0	0	0	0	1
6	1	1	0	0	0
7	1	0	1	0	0
8	1	0	0	1	0
9	1	0	0	0	1
10	0	1	1	0	0
11	0	1	0	1	0
12	0	1	0	0	1
13	0	0	1	1	0
14	0	0	1	0	1
15	0	0	0	1	1
16	1	1	1	0	0
17	1	1	0	1	0
18	1	1	0	0	1
19	1	0	1	1	0
20	1	0	1	0	1
21	1	0	0	1	1
22	0	1	1	1	0
23	0	1	1	0	1
24	0	1	0	1	1
25	0	0	1	1	1

Note. The Q-matrix was doubled for K = 5 and J = 50.

Finally, two scenarios of covariates were considered. The first scenario (Scenario I) considered three time-constant covariates generated from the multivariate standard normal distribution, $N (0, I)$ , where I is a $3 \times 3$ identity matrix. These three time-constant covariates affected both the initial state and transition probabilities. The second scenario (Scenario II) deals with one time-constant covariate and one time-varying covariate. In this scenario, the time-constant covariate ( $Z_{1}$ ), drawn from the standard normal distribution, affected both the initial state and transition probabilities. The time-varying covariate ( $Z_{2}$ ) generated from an AR (1) process with normal white noise: $Z_{21} \sim N (0, 1)$ , $Z_{22} = 1.25 Z_{21} + ϵ$ , and $ϵ \sim N (0, 1)$ , where $Z_{21}$ only affected the initial state probabilities and $Z_{22}$ only affected the transition probabilities. The covariates in this study were examinee-specific. How the covariates in these two scenarios affect the latent states between two time points are depicted in Figure 3. The McFadden’s pseudo R ² of logistic regressions of true attributes on covariates at the initial state and transition was around .50. The correlation between attributes was around .70. The true parameters used to generate data are displayed in Table 4. The true latent profiles were computed based on the true parameters in Table 4 and the covariates and using Equations 5 and 6.

Figure 3.

Two scenarios of latent transition cognitive diagnosis model with covariates at two time points.

Table 4.

True Parameters Used in the Simulation Study

		Initial State				Transition
		$β_{0}$	$β_{1}$	$β_{2}$	$β_{3}$	$γ_{0}$	$γ_{1}$	$γ_{2}$	$γ_{3}$
Scenario I
K = 3	A1	0	1	2	2	0	1	1.5	1.5
K = 3	A2	0	2	1	2	0	1.5	1	1.5
	A3	0	2	2	1	0	1.5	1.5	1
K = 5	A1	0	1.5	2	2	0	1	1.5	1
	A2	0	2	1.5	2	0	1	1	1.5
	A3	0	2	2	1.5	0	1.5	1	1
	A4	0	2	1.5	2	0	1	1.5	1
	A5	0	1.5	2	2	0	1	1	1.5
Scenario II
K = 3	A1	0	1	3		0	1	2
K = 3	A2	0	2.5	2		0	1.5	1
	A3	0	3	1		0	1	2
K = 5	A1	0	1.5	3		0	1	1.5
	A2	0	2	2		0	2	1
	A3	0	3	1.5		0	1.5	1
	A4	0	2	2.5		0	1	2
	A5	0	1	3		0	1	1.5

We assumed that the posttest items were the same as the pretest items. Item responses were generated based on the complete attribute profiles using the G-DINA model. In sum, the current study consists of 3 (sample sizes) × 2 (numbers of attributes) × 2 (test lengths) × 3 (item qualities) × 2 (scenarios) = 72 conditions. For each condition, we generated 100 data sets and estimated the model using both the corrected and uncorrected three-step approaches. In this study, attribute mastery statuses were used as the latent states in this simulation study; hence, the covariate effects on the initial state and transition probabilities were estimated for each attribute. Due to limited space and the complexity of the design needed to obtain generalizable and interpretable results, the present study did not include a simulation study of the profile-level transition. Data generation and Steps 1 and 2 were carried out using the G-DINA package (Ma & de la Torre, 2020) in the R statistical computing software, whereas Step 3 was implemented by directly optimizing the objective functions (i.e., Equations 13 and 14) via the Adam optimizer (Kingma & Ba, 2015) in Python.

The performance of the estimated parameters was assessed by computing the average absolute bias (ABIAS) and the average root-mean-square error (ARMSE), across the attributes, covariates, and replications, which are calculated as $A B I A S = \frac{\sum_{k = 1}^{K} \sum_{f = 0}^{F} \sum_{r = 1}^{R} | {\hat{ϕ}}_{k f}^{(r)} - ϕ_{k f} |}{K \times (F + 1) \times R}$ and $A R M S E = \sqrt{\frac{\sum_{k = 1}^{K} \sum_{f = 0}^{F} \sum_{r = 1}^{R} {({\hat{ϕ}}_{k f}^{(r)} - ϕ_{k f})}^{2}}{K \times (F + 1) \times R}}$ , respectively, where $ϕ_{k f} = β_{k f}$ or $γ_{k f}$ is the fth initial state or transition parameter for attribute k, and ${\hat{ϕ}}_{k f}^{(r)} = {\hat{β}}_{k f}^{(r)}$ or ${\hat{γ}}_{k f}^{(r)}$ is the corresponding estimate at replication r, $r = 1, \dots R = 100$ . Finally, to evaluate the accuracy of classification, attribute agreement rate (AAR) and attribute pattern agreement rate (PAR) are computed as $A A R = \sum_{i = 1}^{N} \sum_{k = 1}^{K} \frac{I [{\hat{α}}_{i k t} = α_{i k t}]}{N K}$ and $P A R = \sum_{i = 1}^{N} \frac{I [{\hat{α}}_{i t} = α_{i t}]}{N}$ , respectively.

Results

One of the goals of the simulation study was to investigate whether the proposed corrected three-step approach can correctly estimate the parameters of the initial state and transition probabilities using the G-DINA model. The ABIAS and ARMSE of the initial state and transition parameters using the corrected and uncorrected three-step approach under Scenario I are given in Tables 5 and 6, respectively.

Table 5.

ABIAS and ARMSE for Initial State Parameters in Scenario I

			K = 3				K = 5
N	Test Length	Item Quality	ABIAS		ARMSE		ABIAS		ARMSE
N	Test Length	Item Quality	Uncor.	Cor.	Uncor.	Cor.	Uncor.	Cor.	Uncor.	Cor.
500	1	Low	1.05	0.58	1.16	0.70	1.13	0.79	1.21	0.87
	1	Medium	0.73	0.30	0.82	0.38	0.82	0.37	0.89	0.44
		High	0.37	0.21	0.44	0.27	0.47	0.24	0.54	0.29
	2	Low	0.82	0.33	0.92	0.41	0.94	0.62	1.02	0.70
	2	Medium	0.42	0.20	0.49	0.25	0.53	0.25	0.60	0.31
		High	0.16	0.15	0.20	0.20	0.18	0.17	0.23	0.21
1,000	1	Low	1.01	0.38	1.12	0.48	1.07	0.46	1.15	0.54
	1	Medium	0.69	0.20	0.78	0.25	0.78	0.24	0.85	0.30
		High	0.35	0.14	0.42	0.19	0.45	0.18	0.52	0.22
	2	Low	0.78	0.20	0.88	0.26	0.89	0.33	0.96	0.39
	2	Medium	0.40	0.13	0.47	0.17	0.50	0.18	0.56	0.22
		High	0.12	0.10	0.15	0.13	0.15	0.12	0.18	0.15
2,000	1	Low	0.97	0.26	1.08	0.33	1.00	0.24	1.09	0.31
	1	Medium	0.67	0.16	0.77	0.20	0.75	0.18	0.83	0.22
		High	0.35	0.12	0.41	0.15	0.44	0.15	0.51	0.19
	2	Low	0.75	0.16	0.86	0.20	0.82	0.16	0.91	0.21
		Medium	0.40	0.10	0.47	0.13	0.48	0.13	0.54	0.16
		High	0.11	0.08	0.14	0.10	0.12	0.08	0.15	0.10

Note. Cor.: using the corrected three-step approach; Uncor.: using the uncorrected three-step approach. Test length = 1: J = 10 when K = 3 and J = 25 when K = 5; Test length = 2: J = 20 when K = 3 and J = 50 when K = 5. ABIAS = average absolute bias; ARMSE = average root-mean-square error.

Table 6.

ABIAS and ARMSE for Transition Parameters in Scenario I

			K = 3				K = 5
N	Test Length	Item Quality	ABIAS		ARMSE		ABIAS		ARMSE
N	Test Length	Item Quality	Uncor.	Cor.	Uncor.	Cor.	Uncor.	Cor.	Uncor.	Cor.
500	1	Low	1.35	1.30	1.72	1.67	1.28	1.23	1.65	1.61
	1	Medium	0.98	0.66	1.21	0.89	1.01	0.74	1.31	1.04
		High	0.60	0.37	0.75	0.52	0.57	0.38	0.74	0.53
	2	Low	1.10	0.82	1.36	1.12	1.13	0.97	1.46	1.34
	2	Medium	0.64	0.37	0.75	0.50	0.63	0.41	0.78	0.57
		High	0.28	0.25	0.35	0.33	0.25	0.24	0.33	0.30
1,000	1	Low	1.39	1.26	1.83	1.69	1.33	1.12	1.85	1.59
	1	Medium	1.00	0.58	1.20	0.83	0.95	0.57	1.17	0.84
		High	0.53	0.29	0.65	0.40	0.54	0.29	0.65	0.39
	2	Low	1.04	0.62	1.26	0.88	1.00	0.63	1.33	0.98
	2	Medium	0.60	0.26	0.67	0.33	0.55	0.25	0.64	0.35
		High	0.19	0.16	0.24	0.21	0.18	0.15	0.23	0.19
2,000	1	Low	1.32	1.15	1.78	1.63	1.22	0.78	1.73	1.20
	1	Medium	0.89	0.36	1.00	0.50	0.86	0.40	0.99	0.59
		High	0.51	0.19	0.57	0.26	0.51	0.21	0.58	0.30
	2	Low	0.96	0.42	1.06	0.57	0.94	0.46	1.15	0.71
	2	Medium	0.59	0.18	0.63	0.23	0.54	0.21	0.59	0.28
		High	0.16	0.12	0.20	0.15	0.15	0.12	0.19	0.15

As shown in Table 5, under various conditions in Scenario I, the ABIAS of the estimated initial state parameters ranged from 0.08 to 0.79 using the corrected approach and from 0.11 to 1.13 using the uncorrected approach, whereas the ARMSE ranged from 0.10 to 0.87 using the corrected approach and from 0.14 to 1.21 using the uncorrected approach. As expected, for all conditions, the corrected three-step approach led to lower ABIAS and ARMSE compared to the uncorrected smaller ABIAS and ARMSE were observed in larger sample sizes, longer tests, and higher item qualities using either the corrected or uncorrected approach. The corrected approach performed only slightly better than the uncorrected approach when longer test lengths and higher item qualities were involved. For example, when K = 3, N = 500, J = 20, and item quality was high, the corrected approach only slightly outperformed, if at all, the uncorrected approach, as in, 0.15 versus 0.16 in terms of ABIAS and 0.20 versus 0.20 in terms of ARMSE. Thus, the improvement was not sizable. However, the corrected approach led to greater improvement over the uncorrected approach across other conditions. For example, when K = 3, N = 1,000, J = 10, and item quality was low, ABIAS produced by the corrected approach and the uncorrected approach was 0.38 versus 1.01, and ARMSE was 0.48 versus 1.12. These results indicate that the proposed approach can produce more reliable estimates when the test conditions are not ideal. Comparing the results of different numbers of attributes, the same patterns in K = 5 as those in K = 3 can be observed, and the ABIAS and ARMSE of K = 5 were only slightly bigger than those of K = 3.

Table 6 shows the performance of the estimated transition parameters under various conditions in Scenario I. Again, the corrected approach produced smaller ABIAS and ARMSE than the uncorrected approach. The ABIAS of the transition parameters ranged from 0.12 to 1.30 using the corrected approach and from 0.15 to 1.39 using the uncorrected approach, whereas the ARMSE ranged from 0.15 to 1.69 using the corrected approach and from 0.19 to 1.85 using the uncorrected approach. In comparing Tables 5 and 6, the patterns of results for the estimated transition and initial state parameters were similar; however, the ABIAS and ARMSE of the estimated transition parameters were larger than those of the estimated initial state parameters.

For the sake of brevity, the details of the results of Scenario II are not presented. However, we found that the ABIAS and ARMSE of the initial state and transition parameters in Scenario II (see Online Appendix A) had the same patterns as those in Scenario I, which indicates that our proposed method is also applicable to time-varying covariates and that covariates of interest can be added to any time point.

Table 7 presents PAR and AAR averaged over attributes, parameters, and replications for Scenario I when K = 3 at different time points. The results indicate that for all conditions, the corrected three-step approach can produce more accurate classification results than the uncorrected approach and was close to the PAR and AAR using the true parameters. The discrepancies among these three methods decreased as the test became more informative. For example, when N = 1,000, J = 20, and item quality was high, the PAR of using the true parameters, the corrected approach, and the uncorrected approach at time 1 were 98.57, 98.56, and 98.57, respectively. However, when the test data were less informative (e.g., shorter test length and lower item quality), the discrepancies became larger. For example, at time 1, when N = 1,000, J = 10, and item quality was low, the PAR of using the corrected approach was 61.45, which was much closer to the PAR of using the true parameters (i.e., 66.55) compared the PAR of using the uncorrected approach (i.e., 48.54). The same pattern of results for PAR and AAR can be observed for K = 5 and for Scenario II (see Online Appendix B).

Table 7.

PAR and AAR for Scenario I and K = 3

			Time 1						Time 2
			PAR (%)			AAR (%)			PAR (%)			AAR (%)
N	J	Item quality	True	Uncor.	Cor.	True	Uncor.	Cor.	True	Uncor.	Cor.	True	Uncor.	Cor.
500	10	Low	62.04	43.02	53.24	84.48	74.85	80.15	65.11	45.83	52.60	85.75	76.30	79.99
	10	Medium	79.03	72.61	76.76	92.22	89.70	91.32	83.30	76.78	80.33	93.88	91.33	92.72
		High	91.99	91.13	91.60	97.22	96.91	97.07	94.23	93.18	93.80	98.01	97.65	97.86
	20	Low	74.98	65.26	71.84	90.46	86.47	89.13	79.22	71.15	76.30	92.21	88.92	91.02
	20	Medium	90.50	89.54	90.20	96.68	96.34	96.56	93.39	92.49	93.05	97.72	97.40	97.59
		High	98.50	98.44	98.45	99.49	99.47	99.48	99.06	99.05	99.05	99.69	99.68	99.68
1,000	10	Low	66.55	48.54	61.45	86.64	77.76	84.17	70.80	53.14	62.95	88.32	79.96	84.78
	10	Medium	80.68	76.29	79.61	92.91	91.18	92.46	85.11	80.64	83.62	94.58	92.88	93.99
		High	92.64	92.01	92.41	97.47	97.24	97.38	94.95	94.29	94.63	98.26	98.03	98.15
	20	Low	77.07	70.56	75.95	91.37	88.72	90.89	81.67	75.89	80.25	93.19	90.89	92.61
	20	Medium	90.94	90.41	90.82	96.82	96.63	96.78	93.76	93.25	93.63	97.84	97.66	97.79
		High	98.57	98.55	98.56	99.52	99.51	99.51	99.09	99.06	99.08	99.69	99.68	99.69
2,000	10	Low	69.10	54.08	66.66	87.77	80.68	86.59	73.42	58.92	69.05	89.53	82.91	87.52
	10	Medium	80.90	77.25	80.26	92.97	91.53	92.70	85.30	81.80	84.52	94.67	93.33	94.37
		High	92.60	92.09	92.42	97.43	97.25	97.37	95.02	94.59	94.89	98.29	98.14	98.24
	20	Low	77.34	72.70	76.83	91.48	89.62	91.26	81.90	77.42	81.44	93.31	91.52	93.11
	20	Medium	90.93	90.44	90.90	96.82	96.65	96.81	93.72	93.24	93.67	97.83	97.66	97.81
		High	98.57	98.56	98.57	99.52	99.52	99.52	99.10	99.09	99.10	99.70	99.69	99.70

Note. Cor.: using the corrected approach; Uncor.: using the uncorrected approach; True: using the true parameters in Table 4. AAR = attribute agreement rate; PAR = pattern agreement rate.

Real Data Example

To illustrate the use of the proposed corrected three-step approach for latent transition CDM with covariates, we present in this section an analysis of longitudinal digital literacy assessment (DLA; Jin et al., 2020) data collected over two time points. The DLA was developed to measure students’ performance on five digital skills, namely, information and data literacy (A1), communication and collaboration (A2), digital content creation (A3), safety (A4), and problem solving (A5). The sample consists of 209 students (57.42% girls) from Hong Kong primary schools, who were tested in the 2018/2019 (Primary 3) and 2020/2021 (Primary 5) academic years. For the present analysis, 28 common items that were examined at both time points were used. The Q-matrix of the 28-item test (Table 8) was derived from Liang et al. (2021).

Table 8.

Q-Matrix for 28-Item Test in the Real Data Example

Item	A1	A2	A3	A4	A5
1	1	0	0	0	0
2	1	0	0	0	0
3	1	0	0	0	0
4	1	0	0	0	0
5	1	0	0	1	0
6	1	0	0	0	0
7	1	0	0	0	0
8	1	0	1	0	0
9	0	1	0	0	0
10	0	1	0	0	0
11	0	1	0	0	0
12	1	0	0	1	0
13	0	1	0	1	0
14	0	1	0	0	0
15	0	1	0	1	0
16	0	1	0	1	0
17	0	1	1	0	0
18	0	0	1	0	0
19	0	0	0	1	0
20	0	0	0	1	0
21	0	0	0	1	0
22	0	0	0	1	0
23	0	1	0	1	0
24	0	1	0	1	0
25	0	0	0	0	1
26	0	0	0	0	1
27	0	0	0	0	1
28	1	0	0	0	1
Total	10	10	3	11	4

The covariates consisted of two time-constant variables, namely, gender and socioeconomic status (SES). SES involved three indicators: father’s and mother’s highest level of education, and home literacy resource (the number of books at home), and was computed by averaging the z-scores of the three indicators. The response data and the covariates were analyzed using the proposed approach to investigate how gender and SES were related to the mastery statuses of the five digital skills. Item parameters were constrained using those from Liang et al. (2021), which calibrated the full DLA data of the first time point, to ensure test reliability, and were also constrained to be equal over time to ensure measurement invariance. The G-DINA models in both time points showed adequate absolute fit (see Table 9) in terms of M₂ (Liu et al., 2016), RMSEA₂ (<.05; Maydeu-Olivares & Joe, 2014), the residual between observed and predicted transformed correlation r, and the log-odds ratios of item pairs l (Chen et al., 2013).

Table 9.

Model-Data Fit of the Real Data Example

		Relative Fit		Absolute Fit
	NP	AIC	BIC	Max. z(r)	Max. z(l)	M₂	RMSEA₂
Time 1	107	5,700.22	6,057.85	3.35 (.31)	4.65 (.00)	322.41 (.17)	0.0194
Time 2	107	7,028.85	7,386.48	3.90 (.04)	3.93 (.03)	403.22 (.00)	0.0408

Note. NP = number of parameters; AIC = Akaike’s information criterion; BIC = Bayesian information criterion; RMSEA = root mean square error of approximation; Max. z(r) = maximum z score for r; Max. z(l) = maximum z score for l. Adjusted p-values based on the Holm method of Max. z(r) and Max. z(l) are shown in the parentheses.

Table 10 displays the parameter estimates of the latent logistic regression models. At the initial state, gender was a significant predictor of A2 (communication and collaboration) and SES was a significant predictor of A1 (information and data literacy). Specifically, given the same SES scores, girls were more likely to master A2 than boys, and the odds of a girl being classified as master of A2 were 2.16 (= e ^.77; p < .01) times greater than those of a boy. Given the same gender, for each unit increase in the SES score, the odds of a student mastering A1 increased from 1.00 to 1.63 (p < .01). For the transition probabilities, given the same SES score, gender did not have a significant relation for nonmasters transitioning to masters of any digital skills. In contrast, given the same gender, SES had significant positive relations to transitions from nonmastery states to mastery states of all five digital skills. For each unit increase in their SES score, the odds of students transitioning from nonmastery to mastery of A1, A2, A3, A4, and A5 increased from 1.00 to 1.59 (p < .01), 1.86 (p < .001), 3.13 (p < .001), 1.70 (p < .001), and 1.83 (p < .001), respectively.

Table 10.

Estimates of Logistic Regression Parameters in the Real Data Example

	A1			A2			A3			A4			A5
	B	SE	Odds/Odds Ratio	B	SE	Odds/Odds Ratio	B	SE	Odds/Odds Ratio	B	SE	Odds/Odds Ratio	B	SE	Odds/Odds Ratio
Initial state
Intercept	−1.71	0.30	0.18	−0.83	0.23	0.44	−0.80	0.23	0.45	−0.61	0.22	0.54	−0.92	0.24	0.40
Gender (girl)	0.61	0.36	1.84	0.77**	0.29	2.16	−0.11	0.31	0.89	0.44	0.29	1.56	0.49	0.30	1.64
SES	0.49**	0.18	1.63	0.18	0.15	1.20	0.07	0.16	1.07	0.01	0.15	1.01	0.13	0.15	1.14
Transition
Intercept	0.83	0.24	2.29	−1.05	0.25	0.35	−1.49	0.29	0.22	0.00	0.22	1.00	0.07	0.22	1.07
Gender (girl)	−0.11	0.31	0.89	0.50	0.31	1.65	0.21	0.35	1.23	0.18	0.29	1.19	0.34	0.29	1.40
SES	0.47**	0.16	1.59	0.62***	0.16	1.86	1.14***	0.21	3.13	0.53***	0.15	1.70	0.61***	0.16	1.83

Note. B = unstandardized coefficient; SE = standard error; odds/odds ratio = $exp (B)$ ; SES = socioeconomic status.

*p < .05. **p < .01. ***p < .001.

Discussion

This study extends the existing LTA methodologies to a general CDM (i.e., G-DINA model) framework in conjunction with covariates. Specifically, a corrected three-step approach for latent transition CDM with time-constant and time-varying covariates, which estimates the latent state classifications and subsequently investigates the relationships between the covariates and latent state memberships in separate steps, while at the same time taking into account the potential classification errors, was developed. This approach also allows modeling not only attribute-level covariate effects but also profile-level covariate effects on both the initial state and transition probabilities, which has not been reported in the literature.

The results of the simulation study indicate, as expected, that the ABIAS, ARMSE, and agreement rates of the proposed method were well behaved under the test conditions considered in this work. Although the uncorrected three-step approach performed similarly to the corrected three-step approach when items were of high quality and the test length was longer, such conditions may not always be satisfied in practice. Thus, when less than high-quality items and small sample size are involved, the corrected three-step approach can be expected to provide more reliable estimates and perform better, in some situations, much better than that of the uncorrected method. These findings indicate that secondary researchers interested in modeling longitudinal data can use the three-step approach to arrive at valid interpretations of the relationships between the latent state membership classification and the covariates of interest.

Both the simulation study and real data example involved only two time points. However, the proposed method can be readily used with data with more time points because the corrected three-step approach is a single-indicator (i.e., the correction weights) latent transition model, where computational complexity is linearly related to the number of time points and the number of covariates (Di Mari et al., 2016). In addition, because the measurement and structural components are estimated separately, the numbers of items and attributes only affect the computational complexity of the measurement model. Such computational complexity of the measurement model can be easily handled in the GDINA package (Ma & de la Torre, 2020). Moreover, the proposed method has sufficient generality to handle time-constant and time-varying covariates, which broadens type of situations where the proposed approach can be used.

Although this study contributes to the CDM literature by developing a corrected three-step estimation approach for latent transition CDMs with different covariate types, there are a number of limitations that point to future research directions. First, in this study, item parameters were constrained to be equal across time points to ensure longitudinal measurement invariance—this can degrade assessment flexibility. The test forms used across time points need not be the same. For test forms that share common items, multiple-group CDMs, such as the multiple-group G-DINA model (MG-GDINA model; Ma et al., 2021), can be used to detect whether there are items that function differentially across different time points. Subsequently, such items can be treated as different items and can have different item parameters. Future research should investigate the performance of different test forms with some common items using the MG-GDINA model in conjunction with the proposed method. Additionally, situations with different measurement models at different time points (Asparouhov & Muthén, 2014) should also be further explored to allow researchers to select the best CDMs for each time point. Di Mari et al. (2016) pointed out that the corrected three-step approach can be adapted to allow for time-specific measurement models by conducting the analysis of Steps 1 and 2 separately and calculating the time-specific CEP at each time point. Second, although it did not limit the applicability of the proposed method, only correlated attribute structure was considered in this study, whereas various attribute structures (independent and correlated) were examined in Iaconangelo and de la Torre (2016). Future research should account for the potential effects that different kinds of attribute structures with varying correlation strengths may cause. Third, comparing the estimates of the initial state and transition parameters, our findings indicate that the covariate effects on the initial state and transition parameters were different, and the estimated transition parameters were more biased regardless of the approach used. This could be due to the larger classification errors introduced in the transitions compared to those introduced in the initial state. The classification errors in the initial state come from the CDM classification at the first time point, whereas the errors in the transition(s) come from the CDM classification from the first to the later time points. Although we adopted a correction for CEP in this study, it was not always sufficient to correct for all the biases caused by the classification errors, which tend to increase over time. Hence, additional work is needed to address this problem in the future. Fourth, the proposed method in its current form was formulated to only handle a single-group, and researchers may be interested in using the corrected three-step approach for multiple-group designs. For example, researchers may want to analyze data from a longitudinal control-group designed experiment or to compare the performance of examinees from different groups (e.g., genders, ages, and countries; Madison & Bradshaw, 2018b). Therefore, the proposed method should be extended to model multiple groups in future research to estimate group-specific growth in attribute mastery and group-specific covariate effects on initial state and transition probabilities. Fifth, classifying the examinees separately at each time point is a limitation of the proposed procedure. In some situations (e.g., when the attributes cannot be measured with sufficient accuracy at each time point), this can produce suboptimal results. Future studies should explore the development of a new method that can simultaneously classify the examinees, as well as simultaneously calibrate the data across multiple time points. Such a method should have the flexibility to impose parameter invariance and can fit into the three-step estimation framework. Lastly, the present study uses covariates as ancillary information to help predict initial latent state membership and transition probabilities. It would also be worthwhile to predict an observed distal outcome from latent state membership (Asparouhov & Muthén, 2014; Lanza et al., 2013). Thus, to expand the scope of CDM applications, future studies may add distal outcome variables to the proposed method to draw a more complete framework of relationships between ancillary variables and latent state membership.

Supplemental Material

Supplemental Material, sj-docx-1-jeb-10.3102_10769986231163320 - Latent Transition Cognitive Diagnosis Model With Covariates: A Three-Step Approach

Supplemental Material, sj-docx-1-jeb-10.3102_10769986231163320 for Latent Transition Cognitive Diagnosis Model With Covariates: A Three-Step Approach by Qianru Liang, Jimmy de la Torre and Nancy Law in Journal of Educational and Behavioral Statistics

Footnotes

Acknowledgments

The authors wish to acknowledge that this work is funded by the Research Grants Council of the HKSAR Government, under the Theme Based Research Scheme. Publication was made possible in part by support from the HKU Libraries Open Access Author Fund sponsored by the HKU Libraries.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research and/or authorship of this article: This work was supported by Research Grants Council, University Grants Committee (T44-707/16-N).

ORCID iD

Qianru Liang

Nancy Law

References

Asparouhov

Muthén

(2014). Auxiliary variables in mixture modeling: Three-step approaches using Mplus. Structural Equation Modeling, 21(3), 329–341. https://doi.org/10.1080/10705511.2014.915181

Ayers

Rabe-Hesketh

Nugent

(2013). Incorporating student covariates in cognitive diagnosis models. Journal of Classification, 30(2), 195–224. https://doi.org/10.1007/s00357-013-9130-y

Baum

L. E.

Petrie

(1966). Statistical inference for probabilistic functions of finite state Markov chains. The Annals of Mathematical Statistics, 37(6), 1554–1563. https://www-jstor-org.web.bisu.edu.cn/stable/2238772

Bolck

Croon

Hagenaars

(2004). Estimating latent structure models with categorical variables: One-step versus three-step estimators. In Political Analysis (Vol. 12(1)., pp. 3–27). Cambridge University Press. https://doi.org/10.1093/pan/mph001

Chen

de la Torre

Zhang

(2013). Relative and absolute fit evaluation in cognitive diagnosis modeling. Journal of Educational Measurement, 50(2), 123–140. https://doi.org/10.1111/j.1745-3984.2012.00185.x

Chen

Culpepper

S. A.

Wang

Douglas

(2018). A hidden Markov model for learning trajectories in cognitive diagnosis with application to spatial rotation skills. Applied Psychological Measurement, 42(1), 5–23. https://doi.org/10.1177/0146621617721250

Chung

Walls

T. A.

Park

(2007). A latent transition model with logistic regression. Psychometrika, 72(3), 413–435. https://doi.org/10.1007/s11336-005-1382-y

Collins

L. M.

Lanza

S. T

. (2010). Latent class and latent transition analysis: With applications in the social, behavioral, and health sciences. Wiley. https://doi.org/10.1002/9780470567333

Culpepper

S. A.

(2015). Bayesian estimation of the DINA model with Gibbs sampling. Journal of Educational and Behavioral Statistics, 40(5), 454–476. https://doi.org/10.3102/1076998615595403

10.

de la Torre

(2011). The generalized DINA model framework. Psychometrika, 76(2), 179–199. https://doi.org/10.1007/s11336-011-9207-7

11.

de la Torre

Douglas

J. A.

(2004). Higher-order latent trait models for cognitive diagnosis. Psychometrica, 69(3), 333–353. https://doi.org/10.1007/BF02295640

12.

de la Torre

van der Ark

L. A.

Rossi

(2018). Analysis of clinical data from a cognitive diagnosis modeling framework. Measurement and Evaluation in Counseling and Development, 51(4), 281–296. https://doi.org/10.1080/07481756.2017.1327286

13.

Di Mari

Oberski

D. L.

Vermunt

J. K.

(2016). Bias-adjusted three-step latent Markov modeling with covariates. Structural Equation Modeling, 23(5), 649–660. https://doi.org/10.1080/10705511.2016.1191015

14.

Goodman

L. A.

(2007). On the assignment of individuals to latent classes. Sociological Methodology, 37(1), 1–22. https://doi.org/10.1111/j.1467-9531.2007.00184.x

15.

(2021). Sufficient and necessary conditions for the identifiability of the Q-matrix. Statistica Sinica, 31, 449–472. https://doi.org/10.5705/ss.202018.0410

16.

Henson

R. A.

Templin

J. L.

Willse

J. T.

(2009). Defining a family of cognitive diagnosis models using log-linear models with latent variables. Psychometrika, 74(2), 191–210. https://doi.org/10.1007/s11336-008-9089-5

17.

Huebner

Wang

(2011). A note on comparing examinee classification methods for cognitive diagnosis models. Educational and Psychological Measurement, 71(2), 407–419. https://doi.org/10.1177/0013164410388832

18.

Iaconangelo

de la Torre

(2016, July 11–15). Three-step estimation of cognitive diagnosis models with covariate [Paper presentation]. 81st International Meeting of the Psychometric Society, Asheville, NC, United States.

19.

Jin

K.-Y.

Reichert

Cagasan

L. P.

de la Torre

Law

(2020). Measuring digital literacy across three age cohorts: Exploring test dimensionality and performance differences. Computers & Education, 157, 103968. https://doi.org/10.1016/j.compedu.2020.103968

20.

Junker

B. W.

Sijtsma

(2001). Cognitive assessment models with few assumptions, and connections with nonparametric item response theory. Applied Psychological Measurement, 25(3), 258–272. https://doi.org/10.1177/01466210122032064

21.

Kaya

Leite

W. L.

(2017). Assessing change in latent skills across time with longitudinal cognitive diagnosis modeling: An evaluation of model performance. Educational and Psychological Measurement, 77(3), 369–388. https://doi.org/10.1177/0013164416659314

22.

Kingma

D. P.

J. L.

(2015, May 7–9). Adam: A method for stochastic optimization [paper presentation]. 3rd International Conference for Learning Representations, San Diego, CA, United States. http://arxiv.org/abs/1412.6980

23.

Lanza

S. T.

Collins

L. M.

(2002). Pubertal timing and the onset of substance use in females during early adolescence. Prevention Science, 3(1), 69–82. https://doi.org/10.1023/A:1014675410947

24.

Lanza

S. T.

Patrick

M. E.

Maggs

J. L.

(2010). Latent transition analysis: Benefits of a latent variable approach to modeling transitions in substance use. Journal of Drug Issues, 40(1), 93–120. https://doi.org/10.1177/002204261004000106

25.

Lanza

S. T.

Tan

Bray

B. C.

(2013). Latent class analysis with distal Outcomes: A flexible model-based approach. Structural Equation Modeling: A Multidisciplinary Journal, 20(1), 1–26. https://doi.org/10.1080/10705511.2013.742377

26.

Cohen

Bottge

Templin

(2016). A latent transition analysis model for assessing change in cognitive skills. Educational and Psychological Measurement, 76(2), 181–204. https://doi.org/10.1177/0013164415588946

27.

Liang

de la Torre

Law

(2021). Do background characteristics matter in Children’s mastery of digital literacy? A cognitive diagnosis model analysis. Computers in Human Behavior, 122, Article 106850. https://doi.org/10.1016/j.chb.2021.106850

28.

Liu

Tian

Xin

(2016). An application of M2 statistic to evaluate the fit of cognitive diagnostic models. Journal of Educational and Behavioral Statistics, 41(1), 3–26. https://doi.org/10.3102/1076998615621293

29.

de la Torre

(2020). GDINA: The generalized DINA model framework. http://cran.r-project.org/package=GDINA

30.

Iaconangelo

de la Torre

(2016). Model similarity, model selection, and attribute classification. Applied Psychological Measurement, 40(3), 200–217. https://doi.org/10.1177/0146621615621717 .

31.

Minchen

de la Torre

(2020). Choosing between CDM and unidimensional IRT: The proportional reasoning test case. Measurement: Interdisciplinary Research and Perspectives, 18(2), 87–96. https://doi.org/10.1080/15366367.2019.1697122

32.

Terzi

de la Torre

(2021). Detecting differential item functioning using multiple-group cognitive diagnosis models. Applied Psychological Measurement, 45(1), 37–53. https://doi.org/10.1177/0146621620965745

33.

Madison

M. J.

Bradshaw

L. P.

(2018a). Assessing growth in a diagnostic classification model framework. Psychometrika, 83(4), 963–990. https://doi.org/10.1007/s11336-018-9638-5

34.

Madison

M. J.

Bradshaw

L. P.

(2018b). Evaluating intervention effects in a diagnostic classification model framework. Journal of Educational Measurement Spring, 55(1), 32–51. https://doi.org/10.1111/jedm.12162

35.

Maydeu-Olivares

Joe

(2014). Assessing approximate fit in categorical data analysis. Multivariate Behavioral Research, 49(4), 305–328. https://doi.org/10.1080/00273171.2014.911075

36.

Park

Y. S.

Lee

Y. S.

(2014). An extension of the DINA model using covariates: Examining factors affecting response probability and latent classification. Applied Psychological Measurement, 38(5), 376–390. https://doi.org/10.1177/0146621614523830

37.

Sun

de la Torre

. (2020). Improving attribute classification accuracy in high dimensional data: A four-step latent regression approach. In Jiao

Lissitz

R. W.

(Eds.), Innovative psychometric modeling and methods (pp. 17–44). Information Age.

38.

Tan

de la Torre

Huh

Larimer

M. E.

Mun

E.-Y.

(2022). A tutorial on cognitive diagnosis modeling for characterizing mental health symptom profiles using existing item responses. Prevention Science. https://doi.org/10.1007/s11121-022-01346-8

39.

Tatsuoka

K. K.

(1983). Rule space: An approach for dealing with misconceptions based on item response theory. Journal of Educational Measurement, 20(4), 345–354. https://doi.org/10.1111/j.1745-3984.1983.tb00212.x

40.

Templin

J. L.

Henson

R. A.

(2006). Measurement of psychological disorders using cognitive diagnosis models. Psychological Methods, 11(3), 287. https://doi.org/10.1037/1082-989X.11.3.287

41.

Vermunt

J. K.

(2010). Latent class modeling with covariates: Two improved three-step approaches. Political Analysis, 18(4), 450–469. https://doi.org/10.1093/pan/mpq025

42.

von Davier

(2005). A general diagnostic model applied to language testing data. ETS Research Report Series, 2005(2), i–35. https://doi.org/10.1002/j.2333-8504.2005.tb01993.x

43.

Wang

Yang

Culpepper

S. A.

Douglas

J. A.

(2018). Tracking skill acquisition with cognitive diagnosis models: A higher-Order, hidden Markov model with covariates. Journal of Educational and Behavioral Statistics, 43(1), 57–87. https://doi.org/10.3102/1076998617719727

44.

Wang

Zhang

Douglas

Culpepper

(2018). Using response times to assess learning progress: A joint model for responses and response times. Measurement, 16(1), 45–58. https://doi.org/10.1080/15366367.2018.1435105

45.

Yigit

H. D.

Douglas

J. A.

(2021). First-order learning models with the GDINA: Estimation with the EM algorithm and applications. Applied Psychological Measurement, 45(3), 143–158. https://doi.org/10.1177/0146621621990746

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.03 MB

Item	A1	A2	A3	A4	A5
1	1	0	0	0	0
2	0	1	0	0	0
3	0	0	1	0	0
4	0	0	0	1	0
5	0	0	0	0	1
6	1	1	0	0	0
7	1	0	1	0	0
8	1	0	0	1	0
9	1	0	0	0	1
10	0	1	1	0	0
11	0	1	0	1	0
12	0	1	0	0	1
13	0	0	1	1	0
14	0	0	1	0	1
15	0	0	0	1	1
16	1	1	1	0	0
17	1	1	0	1	0
18	1	1	0	0	1
19	1	0	1	1	0
20	1	0	1	0	1
21	1	0	0	1	1
22	0	1	1	1	0
23	0	1	1	0	1
24	0	1	0	1	1
25	0	0	1	1	1

Item	A1	A2	A3	A4	A5
1	1	0	0	0	0
2	0	1	0	0	0
3	0	0	1	0	0
4	0	0	0	1	0
5	0	0	0	0	1
6	1	1	0	0	0
7	1	0	1	0	0
8	1	0	0	1	0
9	1	0	0	0	1
10	0	1	1	0	0
11	0	1	0	1	0
12	0	1	0	0	1
13	0	0	1	1	0
14	0	0	1	0	1
15	0	0	0	1	1
16	1	1	1	0	0
17	1	1	0	1	0
18	1	1	0	0	1
19	1	0	1	1	0
20	1	0	1	0	1
21	1	0	0	1	1
22	0	1	1	1	0
23	0	1	1	0	1
24	0	1	0	1	1
25	0	0	1	1	1

Item	A1	A2	A3	A4	A5
1	1	0	0	0	0
2	0	1	0	0	0
3	0	0	1	0	0
4	0	0	0	1	0
5	0	0	0	0	1
6	1	1	0	0	0
7	1	0	1	0	0
8	1	0	0	1	0
9	1	0	0	0	1
10	0	1	1	0	0
11	0	1	0	1	0
12	0	1	0	0	1
13	0	0	1	1	0
14	0	0	1	0	1
15	0	0	0	1	1
16	1	1	1	0	0
17	1	1	0	1	0
18	1	1	0	0	1
19	1	0	1	1	0
20	1	0	1	0	1
21	1	0	0	1	1
22	0	1	1	1	0
23	0	1	1	0	1
24	0	1	0	1	1
25	0	0	1	1	1