On Interim Cognitive Diagnostic Computerized Adaptive Testing in Learning Context

Abstract

Interim assessment occurs throughout instruction to provide feedback about what students know and have achieved. Different from the current available cognitive diagnostic computerized adaptive testing (CD-CAT) design that focuses on assessment at a single time point, the authors discuss several designs of interim CD-CAT that are suitable in the learning context. The interim CD-CAT differs from the current available CD-CAT designs primarily because students’ mastery profile (i.e., skills mastery) changes due to learning, and new attributes are added periodically. Moreover, hierarchies exist among attributes taught sequentially and such information could be used during item selection. Two specific designs are considered: The first one is when new attributes are taught in Stage II, but the student mastery status of the previously taught attributes stays the same. The second design is when both new attributes are taught, and previously taught attributes can be further learned or forgotten in Stage II. For both designs, the authors propose an individual prior, which considers a person’s learning history and population learning model, to start an interim CD-CAT. Simulation results show that the Stage II CD-CAT using individual prior outperforms the methods using population priors. The GDINA (generalized deterministic inputs, noisy, “and” gate) diagnostic index (GDI) is extended to accommodate item hierarchies, and analytic results are provided to further illustrate the types of items that are most popular during item selection. As the first study that focuses on the application of CD-CAT in a learning context, the methods and results present herein showed the great promise of using CD-CAT to monitor learning.

Keywords

cognitive diagnostic computerized adaptive testing learning item selection

To fully embody the potential of computerized adaptive testing (CAT) for facilitating individualized learning on a mass scale (Quellmalz & Pellegrino, 2009), a CAT should have built-in diagnostic features. Kingsbury (2009) has dubbed adaptive tests geared toward cognitive diagnosis as “Idiosyncratic CAT” and has found promising applications in providing teachers with information for targeted instruction. CAT based on cognitive diagnostic models (CDMs)—namely, CD-CAT (cognitive diagnostic computerized adaptive testing)—has become a psychometrically sound option. CDMs provide a level of control in scaling, linking, and item banking that is unavailable with other simpler subscore methods.

A large body of simulation work has been done to explore different item selection methods. For example, Xu et al. (2016) provided a non-asymptotic theory-based approach to guide initial item selection in CD-CAT, which considerably reduces the test length. In addition, well-established information-based criteria such as the mutual information index (C. Wang, 2013), the modified posterior-weighted Kullback–Leibler index (MPWKL; Kaplan et al., 2015), or the GDINA (generalized deterministic inputs, noisy, “and” gate) diagnostic index (GDI; Kaplan et al., 2015) have shown to be highly efficient. However, these available CD-CAT methods mostly focus on assessment at a single time point (e.g., beginning- or end-of-semester). As a result, the test may start randomly (if no a priori information about the test takers are available) and the items usually cover a broad range of attributes with the goal of maximizing the precision of the entire cognitive profile. In contrast, item selections for interim CD-CAT should take into account the general learning model as well as an individual student’s learning history. Developing CD-CAT for such a purpose is essential to close the learning-assessment feedback loop. The interim CD-CAT differs from the current available CD-CAT designs primarily because students’ mastery profile (denoted as α) changes due to learning, and new attributes are added periodically. Therefore, at the $(t) th$ learning stage such as the $(t) th$ week of the instructional period, the interim assessment will select items that require skills taught during the corresponding period and relevant prerequisite skills taught prior to stage t. Furthermore, interim CD-CAT has two additional unique features that should be considered in its design. First, the number of measured attributes could grow; yet, test length needs to be kept desirably low. Hence, challenge remains as to how to accurately recover a potentially high-dimensional $α$ with a limited number of items. Second, hierarchies likely exist among attributes taught sequentially (e.g., curriculum design) and such information could aid the item selection design.

In this article, the authors explore viable interim CD-CAT designs that reflect above-mentioned features. The unique contribution of the study is twofold: (a) although using individual collateral information to start CAT is not new (e.g., van der Linden, 1999), this is the first study that ever positions CD-CAT in a learning context with changing sets of attributes, so the application context is unique. Previous studies either considered cross-sectional CD-CAT or assumed the collection of attributes stay the same over time. (b) The study illustrates the performance of GDI when an attribute hierarchy exists and provides interesting analytical results on the types of items that are preferred during item selection. The findings will shed light on future item bank design. In what follows, the authors first briefly introduce CDMs and GDI, followed by the interim CD-CAT designs as well as relevant analytic results. Then two simulation studies are presented to demonstrate the performance of interim CD-CAT under different scenarios.

CDMs

To support these next-generation assessments aimed at providing fine-grained feedback for students and teachers (Leighton & Gierl, 2007; Templin & Bradshaw, 2014), CDMs have arisen as advanced psychometric models in the past few decades. In essence, CDMs are restrictive latent class models that uncover the skills/attributes a student possesses at the time of assessment. Each latent profile constitutes one latent class. Denote the mastery profile of a person by $α = (α_{1}, \dots, α_{K})$ , where K is the total number of attributes measured by a test. Similarly, the item to attribute mapping could be summarized in a Q-matrix (Embretson, 1984). The q-vector of item j is denoted by $q_{j} = (q_{j 1}, \dots, q_{jK})$ , which is a binary vector that implies the attributes measured by item j. Recent advancement in CDMs also allows polytomous attributes, but the authors restrain the discussion to binary attributes in this article.

To model the item responses as a function of item and person characteristics, de la Torre (2011) proposed a GDINA model, which is one of the most flexible CDMs. The GDINA model specifies the probability that person i answers item j correctly as follows:

\begin{matrix} P (y_{ij} = 1 | α_{i}) = δ_{j, 0} + \sum_{k = 1}^{K_{j}} δ_{j, k} q_{jk} α_{ik} + \sum_{k = 1}^{K_{j}} \sum_{k' > k} δ_{j, kk'} q_{jk} q_{jk'} α_{ik} α_{ik'} + \dots + δ_{j, 12 \dots K_{j}} Π_{k = 1}^{K_{j}} q_{jk} α_{ik}, \end{matrix}

(1)

where $δ_{j, 0}$ is the intercept, $δ_{j, k}$ is the main effect due to $α_{k}$ , $δ_{j, kk'}$ is the two-way interaction, $δ_{j, 1, 2, \dots, K}$ is the $K_{j}$ -way interaction, and $K_{j}$ is the number of relevant attributes measured by item j. From Equation 1, it is shown that the GDINA model is essentially a generalized linear model (GLM) with binary latent predictors. If the identity link in Equation 1 is replaced by a logit link, the model becomes the log-linear CDM (LCDM; Henson et al., 2009). Assume K is the total number of attributes measured in a test, then without imposing any additional constraints, both GDINA and LCDM assume that an individual profile, $α_{i}$ , could be any one of the possible profiles, with the class mixing proportions denoted as $π_{c}$ .

GDI

The GDI is considered as the basis item selection method in this article because it is computationally fast and it performs as well or even sometimes better than the other computationally more intensive method, such as the mutual information method (C. Wang, 2013) or the modified PWKL (Kaplan et al., 2015; Xu et al., 2016). The GDI measures the weighted variance of the conditional success probabilities of an item. The definition is as follows. Let $α_{jc}^{*}$ denote a reduced attribute vector consisting of $K_{j}$ attributes measured by item j. The superscript “*” is added to show that $α_{jc}^{*}$ is not of the same length as $α$ , and the length of $α_{jc}^{*}$ varies across different items. Denote $P (y_{ij} = 1 | α_{jc}^{*})$ as the correct response probability of item j given $α_{jc}^{*}$ , and denote $π_{i}^{n} (α_{jc}^{*})$ as the posterior probability of the reduced attribute vector $α_{i}^{*} = α_{jc}^{*}$ after n administered items in CAT. Then, the GDI for item j is

\begin{matrix} η_{j} = \sum_{c = 1}^{2^{K_{j}}} π_{i}^{n} (α_{jc}^{*}) {[P (y_{ij} = 1 | α_{jc}^{*}) - {\bar{P}}_{j}]}^{2}, \end{matrix}

(2)

where ${\bar{P}}_{j} = \sum_{c = 1}^{2^{K_{j}}} π_{i}^{n} (α_{jc}^{*}) P (y_{ij} = 1 | α_{jc}^{*})$ is the average success probability. In essence, the GDI measures the discrimination power of an item in differentiating different relevant reduced attribute vectors. It also attaches higher weight to $α_{jc}^{*}$ if its posterior density is high, implying that $α_{jc}^{*}$ is more likely to be the true reduced attribute vector for person i. One feature that makes GDI compelling for interim CD-CAT is that it uses reduced vector $α_{jc}^{*}$ in its computation. Hence, regardless of how large K is in a test, as long as the number of attributes measured by a single item is relatively small, selecting items using GDI is always efficient¹ because GDI only depends on the number of attributes measured by each item.

Interim CD-CAT Design

In this section, several design aspects for interim CD-CAT will be discussed in sequel, which include (a) efficiently synthesizing information from prior stages to update α when the size of α keeps growing over time, (b) selection of items in the presence of attribute hierarchies, and (c) updating prior information to take into account learning.

Specifically, at the (t+ 1)th learning stage, if a set of skills denoted by $s^{(t + 1)}$ is taught which could contain one or multiple skills, then the interim assessment will select items that require skills in $s^{(t + 1)}$ and skills taught prior to stage (t+1). The goal of item selection is to evaluate whether skills in $s^{(t + 1)}$ are learned, and if not, whether any prerequisites are missing. Items that can provide maximum information to recover mastery status of skills in $s^{(t + 1)}$ and the corresponding prerequisites will be selected.

Update of α

First of all, during a CD-CAT when the size of α is fixed, then to save computation time for sequential update of α using the maximum a posteriori (MAP; Huebner & Wang, 2011), one can use the posterior density from the previous step as the current prior. That is, suppose the cardinality of $s^{(t)}$ is $k_{t}$ , then $α$ is a vector of length $K_{t}$ , which includes all attributes taught up to time t. By definition, we have

\begin{matrix} π_{i}^{n} (α | y_{i}^{n}) = \frac{L (y_{i}^{n} | α) π_{0} (α)}{\sum_{c = 1}^{2^{K_{t}}} L (y_{i}^{n} | α_{c}) π_{0} (α_{c})}, \end{matrix}

(3)

where $π_{0} (α)$ is the prior, and $y_{i}^{n}$ is the vector of responses on n administered items. Any information regarding the attribute relationships could be summarized in $π_{0} (α)$ . For instance, when all attributes are independent, then $π_{0} (α)$ is simply a vector of $\frac{1}{2^{K_{t}}}$ ’s. When there exist attribute hierarchies, then some combinations of $α$ become impermissible, leaving corresponding elements in $π_{0} (α)$ 0. Moreover, if the attributes are correlated, such as in the higher order CDMs (de la Torre & Douglas, 2004; C. Wang et al., 2014), $π_{0} (α)$ will take on unique patterns to reflect the correlation. From Equation 3, it could be easily shown that

\begin{matrix} π_{i}^{n} (α | y_{i}^{n}) = \frac{L (y_{in} | α) π_{i}^{n - 1} (α | y_{i}^{n - 1})}{\sum_{c = 1}^{2^{K_{t}}} L (y_{in} | α_{c}) π_{i}^{n - 1} (α | y_{i}^{n - 1})} . \end{matrix}

(4)

Compared with Equation 3, Equation 4 circumvents the need of cycling through all n items to compute the likelihood, hence reducing the computation time, in particular when either n or $K_{t}$ is large.

Second, as alluded to earlier, during an instructional unit, new skills are taught and evaluated periodically. Hence, the posterior density of $α^{(t)}$ at the end of stage t, $π_{i}^{(t)} (α^{(t)} | y_{i}^{(t)})$ , cannot serve as the prior of $α^{(t + 1)}$ at stage (t+ 1) because $α^{(t)}$ and $α^{(t + 1)}$ have different lengths. For instance, say, $α^{(t)}$ is a three-dimensional vector whereas $α^{(t + 1)}$ is a six-dimensional vector, then the posterior of $α^{(t)}$ is a probability mass function for eight latent classes whereas the prior for $α^{(t + 1)}$ should be a probability mass function for 64 latent classes. Even so, the responses from all preceding stages up to time (t) could still be gathered to form a more informative prior for person i at the beginning of the (t+ 1)th interim CD-CAT. More specifically, the posterior density computed as follows will be used as the prior at stage (t+ 1):

π_{i}^{(t + 1)} (α) = \frac{L (y_{i}^{(1, 2, \dots, t)} | α) π_{0} (α)}{\sum_{c = 1}^{2^{K_{t + 1}}} L (y_{i}^{(1, 2, \dots, t)} | α_{c}) π_{0} (α_{c}),}

(4)

where $y_{i}^{(1, 2, \dots, t)}$ denotes the stack of responses from all preceding stages, and $π_{0} (α)$ is a uniform prior that reflects attribute hierarchies. That is, impermissible patterns that violate the hierarchical relationship have priors of 0, whereas all permissible patterns are equally likely. Of course, the length of $α$ becomes $K_{t + 1}$ to reflect all attributes measured till time (t+ 1). $π_{i}^{(t + 1)} (α)$ will replace $π_{0} (α)$ in Equation 3 to start the (t+ 1)th interim CD-CAT. In fact, the computation of $π_{i}^{(t + 1)} (α)$ could be done offline after the tth CD-CAT finishes; hence, it will not interrupt the flow of live CD-CAT if its computation takes time.

It should be noted that, even though the conditional correct response probability of an item depends only on the reduced vector (i.e., Equation 1), and so is GDI, the posterior density of the full vector will be updated. In other words, if an item measures $α_{1}$ and $α_{2}$ whereas K = 3, then only $π_{i}^{n} (α_{1}, α_{2})$ counts in computing GDI, but the posterior mastery probability of $α_{3}$ will still be updated after administering this item. This point is important because, for example, in the second CD-CAT, although those attributes measured in the first test may not emerge again in the items given in the second test, the mastery profile on these previously measured attributes may still update. Therefore, it is recommended, in the interim CD-CAT context, to always work with full length α (i.e., $K_{t}$ ) rather than working with clusters of attributes in isolation.

Item Selection

Given the many desirable features of GDI, it could be used for item selection in interim CD-CAT. At stage t, the eligible items would be those that require one or multiple skills in $s^{(t)}$ , some of which may also require skills taught prior to stage t. When there are attribute hierarchies among the reduced vector $α_{jc}^{*}$ , this information will be reflected in $π_{0} (α_{jc}^{*})$ that will be carried over to $π_{i}^{n} (α_{jc}^{*})$ in Equation 2.

Besides this simple modification, it would be interesting to delve deeper into the types of items that are preferable because this knowledge would guide future item bank design. When all attributes are independent, Xu et al. (2016) proposed a non-asymptotic theory-based approach to guide initial item selection in CD-CAT. In particular, they derived the minimum number of items required to identify the attribute pattern of a student as well as the specific types of initial items that are required to reach the optimal classification results, under both ideal and practical scenarios. Their proposal was later used by Chang et al. (2018) in their nonparametric CD-CAT design. While Xu et al.’s (2016) results are suitable for independent attribute structures, the authors extend the results to scenarios with attribute hierarchies. The two lemmas presented below are for the ideal case of conjunctive CDM when students answer correctly all questions they are capable of and incorrectly otherwise. Their proofs are presented in the online appendix. The conclusions will be empirically verified in non-ideal cases via simulation studies. On a side note, the Q-matrix does not necessarily have to conform to the attribute hierarchies (Templin & Bradshaw, 2014; Templin & Hoffman, 2013).

Lemma 1: Suppose a test intends to measure K attributes with the attribute hierarchies summarized in a K-by-K reachability matrix R. To identify all possible attribute profiles, it is necessary and sufficient for a test to have K items with the Q-matrix satisfying the following rule: After row swapping, it is the transpose of the reachability matrix, with off-diagonal 1s replaced by “*” indicating that those entries could be either 1 or 0.

For instance, with K = 4 and attributes exhibit a divergent structure shown in Figure 1, the reachability matrix takes the form of $R = [\begin{matrix} 1 & 1 & 1 & 1 \\ 0 & 1 & 0 & 0 \\ 0 & 0 & 1 & 1 \\ 0 & 0 & 0 & 1 \end{matrix}]$ (Tatsuoka, 1986), in which the $(i, j) th$ element equals 1 when attribute i is a direct or indirect prerequisite of attribute j. Then, by Lemma 1, Q-matrix needs to take the following form, after row swapping:

Q = [\begin{matrix} 1 & 0 & 0 & 0 \\ * & 1 & 0 & 0 \\ * & 0 & 1 & 0 \\ * & 0 & * & 1 \end{matrix}] .

(5)

Figure 1.

An illustration of the divergent structure.

Lemma 2 (Extension of Theorem 1 in Xu et al., 2016): Consider a person with attribute profile α = 0. In the ideal case, to identify α = 0, it is necessary and sufficient for a test to have C items, where C denotes the total “root” attributes in all clusters of attributes. The corresponding Q-matrix of the C items, after row swapping, follows the following form: $Q = (I_{C}, 0)_{C \times K}$ , where the first C columns refer to the “root” attributes.

Moreover, consider a person with attribute profile $α \neq 0$ . Suppose there are k elements in α equal to 1. Then via a column swapping of the attribute labels, α is equivalent to $m_{k}$ . To identify $α = m_{k}$ , it is necessary and sufficient for a test to satisfy the following conditions:

C1: There are $d_{1}$ items in the test such that the corresponding Q-matrix after row swapping takes the form:

Q = {(O_{*}, I_{d_{1}}, 0)}_{d_{1} \times K},

(5)

where $d_{1}$ denotes the number of “independent, root” attributes among the last $(K - k)$ attributes. $I_{d_{1}}$ is a $d_{1} \times d_{1}$ identity matrix, and $O_{*}$ denotes an unspecified $d_{1} \times k$ binary matrix. The remaining $d_{1} \times (K - d_{1} - k)$ elements are all 0s. Here, the independent attributes mean they do not have a direct linkage. When a set of attributes are directly related, then $d_{1} = 1$ for the “root” attribute.

C2: There is a set of item(s) that requires none of the last $(K - k)$ attributes but requires all of the “leaf” attributes among the first k attributes. If presenting the attribute hierarchy as a tree graph (see Figure 1), then the leaf attributes are the most advanced attributes that are not the prerequisites for any other attributes.

Again, use K = 4 and the divergent structure in Figure 1 as an example. As all attributes form a single cluster, when α = 0, only one item with q = (1,0,0,0) is needed so that any profile $α \neq 0$ has different response patterns from that of α = 0. For α = (1,0,0,0), the “independent, root” attributes among $(α_{2}, α_{3}, α_{4})$ are $α_{2}$ and $α_{3}$ . Hence, the Q-matrix satisfying the two conditions in Lemma 2 takes the form of $Q = (\begin{matrix} * & 1 & * & 0 \\ * & 0 & 1 & 0 \\ 1 & 0 & 0 & 0 \end{matrix})$ . If α = (1,0,1,0), then according to Lemma 2, we need three items with the following Q-matrix to identify this profile: $Q = (\begin{matrix} * & 1 & * & 0 \\ * & 0 & * & 1 \\ * & 0 & 1 & 0 \end{matrix})$ . The first two rows satisfy condition C1. Because $α_{2}$ and $α_{4}$ are not directly linked, $d_{1} = 2$ . The last row satisfies condition C2. For $α_{1}$ and $α_{3}$ , $α_{3}$ is the “leaf” attribute.

Similarly, if α = (1,1,0,0), then we only need two items with the following Q-matrix: $Q = (\begin{matrix} * & * & 1 & 0 \\ * & 1 & 0 & 0 \end{matrix})$ . Because $α_{3}$ and $α_{4}$ are directly related, $d_{1} = 1$ and $α_{3}$ is the root attribute between the two.

For sequential item selection, Theorem 2 in Xu et al. (2016) cannot be easily extended due to the complication of the attribute hierarchies. That is, the first item does not have to be a single-attribute item anymore. However, the general conclusion in Xu et al. (2016) still holds: During item selection, when the answer to an item is wrong, implying that the corresponding attributes may not be mastered, then the non-mastered attributes should not be required by the next item. On the contrary, if the answer is right, then the mastered attribute could be “unspecified” for the next item.

Using Figure 1 as an example, one interesting takeaway message from Lemma 2 is when $α_{1}$ is the prerequisite attribute for all others, then except for people with α = 0 or α = (1,0,0,0), the sufficient Q-matrix for identifying true pattern for an individual really does not need item measuring $α_{1}$ . This fact is reflected by the “*” in the first column of the Q-matrix. This implies that items that only measure $α_{1}$ may not be as useful as items measure other, more advanced attributes.

When Learning Happens

Learning could happen between two interim CD-CATs, which means a student’s $α$ is no longer static. If this is the case, responses from prior stages may not be used in the current stage because those prior responses reflect a somewhat different $α$ prior to learning. Instead, insights from learning models could play an important role in forming a better prior such that the ever-changing $α$ could still be recovered well within a handful of items. For instance, Chen et al. (2018) proposed a first-order hidden Markov model for learning trajectories, and results from their model could be useful for interim CD-CAT. Similar models are also proposed by Madison and Bradshaw (2018).

To be specific, their model could produce either attribute-level or pattern-level first-order transition probabilities. That is, between any two time points, the attribute-level transition probability spells out as

\begin{matrix} τ_{k, 1 | 0} = P (α_{k}^{(t + 1)} = 1 | α_{k}^{(t)} = 0), τ_{k, 0 | 0} = P (α_{k}^{(t + 1)} = 0 | α_{k}^{(t)} = 0), \\ τ_{k, 0 | 1} = P (α_{k}^{(t + 1)} = 0 | α_{k}^{(t)} = 1), and τ_{k, 1 | 1} = P (α_{k}^{(t + 1)} = 1 | α_{k}^{(t)} = 1) . \end{matrix}

(5)

Of course, $τ_{k, 1 | 0} + τ_{k, 0 | 0} = 1$ and $τ_{k, 1 | 1} + τ_{k, 0 | 1} = 1$ . The transition probabilities do not have a subscript of time t on it because the current learning model assumes the transition probabilities to be time invariant (Chen et al., 2018; Li et al., 2016). By knowing these attribute-level transition probabilities, and assuming all attributes are learned independently, the pattern-level transition probabilities could be obtained fairly easily. As an example, $τ_{(1, 1) | (0, 0)} = τ_{1, 1 | 0} \times τ_{2, 1 | 0}$ . On the contrary, both Chen et al. (2018) and Madison and Bradshaw (2018) discussed about estimating the pattern-level transition probabilities directly, hence relaxing the assumption of independent learning.

Now with pattern-level transition probabilities known from a learning model, the authors propose to update priors of $α$ at the beginning of a new CD-CAT as follows. Say, at time t, the posterior density of $α$ for person i is written as $π_{i}^{(t)} (α)$ for succinctness. Then, the individual prior that could be used for person i’s $(t + 1) th$ CD-CAT is $π_{i, 0}^{(t + 1)} (α) = π_{i}^{(t)} (α) \times P (α^{(t + 1)} | α^{(t)})$ , where $P (α^{(t + 1)} | α^{(t)})$ is the pattern transition matrix. This individual, informative prior synthesizes the information from a person’s learning history as well as the learning model, and therefore, it is expected that the interim CD-CAT designed this way could produce accurate $α$ estimates efficiently.

Simulation Studies

Two simulation studies were conducted to mimic the typical scenarios in learning context. In both cases, two time points were considered for illustration purpose, but the methods could be used for additional number of time points. In the first scenario, new attributes are added at Time II, and these new attributes may require the previously taught attributes as prerequisites. In the second and more interesting scenario, not only new attributes are added at Time II, students’ true mastery profile on the attributes measured at Time I also changes due to learning and forgetting. The simulation designs and results for these two simulation studies are presented in detail below.

Study I Design

Manipulated factors

In Study I, the three manipulated factors² are (a) the total number of attributes measured across two time points, K = 6 or 10; (b) the relationship between the attributes tested at two time points, that is, independent or hierarchical; and (c) the underlying CDM model, additive CDM (ACDM) or GDINA, both with identity link. Even though the attribute hierarchies could take on different types, the authors believe one type is representative to show the general trend between independent and hierarchical comparisons. The attribute hierarchies are shown in Figure 2 for both two levels of K. When the attributes are independent, it implies that there are no direct linkages among any attributes. Fully crossing the two manipulated factors results in four simulated conditions.

Figure 2.

Attribute hierarchies in the simulation study. (A) K = 6 and (B) K = 10.

Item bank construction

The item bank size was created in proportion to K. When K = 6, there are 480 items in the item bank, with the Q-matrix designed as follows: The first 150 items only measure $α_{1}$ to $α_{3}$ , the second 150 items only measure $α_{4}$ to $α_{6}$ , and the remaining 180 items measure $α_{1}$ to $α_{6}$ . More specifically, the first 150 items’ Q-matrix is the 15 copies of the first 10 rows of Table A1 (online appendix), whereas the second 150 items’ Q-matrix is the 15 copies of the last 10 rows of Table A1. Regarding the remaining 180 items, one third of the items measure one attribute from $α_{1}$ to $α_{3}$ (i.e., randomly select) and one attribute from $α_{4}$ to $α_{6}$ , one third of the items measure two attributes from $α_{1}$ to $α_{3}$ and one attribute from $α_{4}$ to $α_{6}$ , and the last one third of the items measure one attribute from $α_{1}$ to $α_{3}$ and two attributes from $α_{4}$ to $α_{6}$ . As a result, the item selection at Time I will be restrained to select items only from the first 150 items, whereas the item selection at Time II will be from either the entire bank excluding the items used at Time I (i.e., the same items will not be administered twice to the same examinee) or the remaining 330 items (i.e., the items that measure at least one of the attributes from $α_{4}$ to $α_{6}$ ). The authors name these two conditions as non-restricted and restricted conditions, respectively. It is expected that the former scenario will yield higher precision of α estimates, whereas the latter scenario may reflect the real assessment design more closely because teachers usually focus on newly taught attributes (e.g., $α_{4}$ to $α_{6}$ ) in interim quizzes. Test length was set up to be proportional to K. When K = 6, the test lengths are 6 and 12 at Time I and Time II, respectively, whereas when K = 10, the test lengths are 10 and 20.

As to the item parameters for ACDM with identity link, the intercepts were simulated from a uniform distribution U(0.01, 0.2), and the main effects were simulated as $δ_{jk} = (0.65 / K_{j}) + ε_{jk}$ , where $ε_{jk} ~ U (0.01, 0.05)$ is a random value so that the main effects are not equivalent. Setting the main effect this way ensures that the conditional correct response probabilities are between 0.2 and 0.85 (Xu et al., 2016).

As to the item parameters for the GDINA model (de la Torre, 2011), a generic item parameter, $δ_{jd},$ representing either a main effect or an interaction effect of item j, was set to be $\frac{[P (α_{j}^{*} = 1) - P (α_{j}^{*} = 0)]}{D_{j}} + ε_{d}$ , where $D_{j}$ is the number of main and interaction effects for item j, and $ε_{d} ~ U (- . 05, . 05)$ is a small disturbance term. $P (α_{j}^{*} = 1)$ denotes the probability of answering item j correctly given mastery of all required attributes, so it is the highest correct probability and it was drawn from U(0.7, 1). $P (α_{j}^{*} = 0)$ denotes the probability of answering item j correctly given non-mastery of any required attributes, so it is the lowest correct probability and it was drawn from U(0, 0.3) (Chen, 2017). Generating item parameters this way ensures that all required attributes and their interactions have substantial but different contributions to the correct response probability, and monotonicity is satisfied.

When K = 10, there are 600 items in the item bank, with the Q-matrix designed similarly to the previous condition, as follows. The first 180 items only measure $α_{1}$ to $α_{5}$ , and there are six copies of a 30-row Q-matrix. The first 10 rows represent two copies of 5-by-5 identity matrix, followed by 10 rows of q-vector that has 2 entries of 1 (i.e., there are 10 combinations of 2 out of 5), and another 10 rows of q-vector that has 3 entries of 1. Similarly, the second 180 items only measure $α_{6}$ to $α_{10}$ , with the Q-matrix constructed exactly as above. The remaining 240 items’q-vectors were constructed the same way as in K = 6 scenario, with 80 items in each cluster. Again, all item parameters were simulated the same as before. Then at Time I CD-CAT, item selection is restricted to the first 180 items, whereas item selection at Time II, items are selected from either the remaining 420 items (restricted condition) or 590 items (non-restricted condition, excluding the previously administered 10 items).

Sample construction

The sample size in the four conditions was simulated to be proportional to the number of permissible latent classes, that is, N = 100 × Number of permissible conditions. Although the total sample size was not held equal between K = 6 and K = 10 conditions due to the intrinsic difference between the dimensionality of the two latent spaces, the number of people per permissible latent class was held equal. The total sample size therefore is 2,400 (K = 6, hierarchical), 6,400 (K = 6, independent), 16,800 (K = 10, hierarchical), and 102,400 (K = 10, independent). These are the typical sample size used in other CD-CAT researches (e.g., Kaplan et al., 2015; Xu et al., 2016).

Two methods were considered for Time II CD-CAT design: The baseline method that only exploits population-level information of attribute hierarchies, that is, $π_{0} (α)$ , was used as the prior. This method is named as population prior in the results section—the individualized prior approach which uses each student’s likelihood (i.e., learning history) from Time I CD-CAT in constructing the prior.

Study I Results

Table 1 presents the pattern and attribute recovery rates under different manipulated conditions. For attribute recovery, we present the mean recovery rate for the first three (or five) and last three (or five) separately as they show different patterns under restricted and non-restricted conditions. The raw attribute recovery rates per condition are presented in the online appendix (Tables A2–A5). The results consistently show that using individual priors produces higher pattern and attribute recovery rates than using population priors. The difference in pattern recovery rate ranges from 5% (e.g., GDINA, K = 10, hierarchical, non-restricted) to almost 40% (e.g., ACDM, K = 10, independent, restricted). When we restrict item selections in Time II CD-CAT to items that measure at least one of the last three attributes (i.e., $α_{4} ~ α_{6}$ ), the attribute and pattern recovery rates drop for both individual and population prior methods, and this decrement is more dramatic for the population prior method. This is because when restricting item selection in Time II, none of the items will measure one of the first three attributes (i.e., $α_{1} ~ α_{3}$ ) alone, and hence, the attribute-level recovery of $α_{1}$ to $α_{3}$ is largely sacrificed. But if we only focus on the attribute recovery of $α_{4}$ to $α_{6}$ , whether or not restricting item selections to a subset of the item pool does not make much of a difference. Another interesting finding is, for the restricted item selection scenario using the population prior, Time II CD-CAT is treated independently of Time I CD-CAT. In this case, even when the item selection focuses only on attributes $α_{4} ~ α_{6}$ , the persons’ mastery on attributes $α_{1} ~ α_{3}$ are still updated as reflected by higher than 50% mean attribute recovery rates for $α_{1} ~ α_{3}$ . This is consistent with the expectation. Moreover, when the attributes exhibit a hierarchical structure and the hierarchy is known, the pattern and attribute recovery rates are, on average, 10% to 15% higher than those from the independent attribute structure scenario, simply because the permissible space of $α$ is reduced with known hierarchies. Comparing between the two models, ACDM and GDINA, it reveals that the GDINA model yields higher pattern and attribute recovery rates in general. The differences between the two models are around 10% to 15% for pattern recovery rate and 3% to 5% for attribute recovery rate, holding everything else equal. This is because items generated from GDINA are usually more informative. Furthermore, while the improvements in pattern recovery rate from population prior to individual prior are about 6% to 26% (non-restricted) and 25% to 40% (restricted) for ACDM, they are only 1% to 9% (non-restricted) and 25% to 40% (restricted) for GDINA.

Table 1.

Study I: Pattern and Average Attribute Recovery Rates Under Different Manipulated Conditions.

			Hierarchical					Independent
			Time IIJ = 12					Time IIJ = 12
			Restricted		Non-restricted			Restricted		Non-restricted
Conditions	Method	Time IJ = 6	I	P	I	P	Time IJ = 6	I	P	I	P
K = 6
ACDM	Pattern	0.725	0.853	0.595	0.894	0.759	0.721	0.730	0.328	0.764	0.591
	Attribute: $α_{1} ~ α_{3}$	0.900	0.962	0.856	0.984	0.951	0.897	0.928	0.709	0.956	0.908
	Attribute: $α_{4} ~ α_{6}$		0.981	0.969	0.975	0.951		0.968	0.967	0.957	0.923
GDINA	Pattern	0.933	0.986	0.726	0.997	0.983	0.938	0.975	0.713	0.993	0.944
	Attribute: $α_{1} ~ α_{3}$	0.977	0.996	0.888	1.000	0.997	0.979	0.993	0.884	0.999	0.987
	Attribute: $α_{4} ~ α_{6}$		0.999	0.996	1.000	0.996		0.999	0.993	0.999	0.993
K = 10
ACDM	Pattern	0.658	0.815	0.486	0.886	0.713	0.666	0.653	0.166	0.718	0.453
	Attribute: $α_{1} ~ α_{5}$	0.918	0.969	0.876	0.989	0.961	0.921	0.938	0.718	0.967	0.922
	Attribute: $α_{6} ~ α_{10}$		0.987	0.977	0.984	0.963		0.978	0.969	0.967	0.924
GDINA	Pattern	0.899	0.968	0.675	0.990	0.947	0.937	0.973	0.553	0.980	0.890
	Attribute: $α_{1} ~ α_{5}$	0.979	0.995	0.913	0.999	0.994	0.987	0.997	0.890	0.999	0.990
	Attribute: $α_{6} ~ α_{10}$		0.999	0.993	0.999	0.994		0.997	0.983	0.997	0.986

Note.“P” denotes population prior, whereas “I” denotes individual prior. “Restricted” condition refers to the scenario where item selection in Stage II CD-CAT is from the 330 items that measure at least one of the three attributes, that is, $α_{4} ~ α_{6}$ , whereas “non-restricted” condition refers to the scenario where items in Stage II CD-CAT can be selected from any eligible items in the bank. The eligible items are those that are not administered to the same person in Stage I. ACDM = additive cognitive diagnostic model; GDINA = generalized deterministic inputs, noisy, “and” gate; CD-CAT = cognitive diagnostic computerized adaptive testing.

To further explore the types of items (i.e., items with certain q-vectors) that are more likely to be selected, we draw the heat map of item exposure in Time II conditioning on the different true $α$ patterns, as shown in Figures A2 and A3 (online appendix). We focus on Time II because in Time I, the attributes only display an independent structure. Only results from K = 6 (hierarchical) are shown because the results from K = 10 are similar. When K = 6, the item bank contains items with 41 different q-vectors. Interestingly, when we do not restrict item selections in Stage II, the items that are more often selected are those that measure a single attribute, and among them, items measuring attributes $α_{4}$ to $α_{6}$ are the most often selected. Using the individual prior, the item exposure rates of items measuring only one of the attributes among $α_{1}$ to $α_{6}$ are 10.7%, 13.5%, 12.3%, 21.1%, 15.2%, and 21.2%, respectively (relating to Figure A2a). The same pattern holds for the population prior method (i.e., Figure A2b). This is consistent with Lemma 1 and Lemma 2 discussed in the previous section. In contrast, when item selection in Stage II is restricted, then items measuring only $α_{1}$ to $α_{3}$ are no longer eligible. In this case, items measuring one of the three last attributes continue to be the most popular items, followed by those items whose q-vectors conforming to Lemma 1 and Lemma 2. Figure A3 presents the parallel results from the GDINA model, and the same item exposure rate pattern remains.

Study II Design

The same eight conditions were considered in Study II. The item bank was also the same as in Study I. In addition, to model learning transitions between Time I and Time II, we assumed the attribute-level learning rate, $l_{k} \equiv τ_{k, 1 | 0}$ , and attribute-level forgetting rate, $f_{k} \equiv τ_{k, 0 | 1}$ , are from two levels: 0.5/0.2 and 0.8/0.1 (Madison & Bradshaw, 2018; Table A9). Taking K = 6 as an example, when attributes exhibit hierarchical structure, student mastery status of $α_{4} ~ α_{6}$ depends on their mastery status of $α_{1} ~ α_{3}$ at the beginning of CD-CAT at Time II.

To form the $2^{K_{1}} \times 2^{K}$ transition matrix, where $K_{1}$ denotes the number of attributes measured at Time 1, we can compute the pattern-level transition probabilities as follows. For each element in the transition matrix, we have

\begin{matrix} P (α^{(2)} | α^{(1)}) \equiv P (α_{1}^{2}, α_{2}^{2}, α_{3}^{2}, α_{4}^{2}, α_{5}^{2}, α_{6}^{2} | α_{1}^{1}, α_{2}^{1}, α_{3}^{1}) = = P (α_{1}^{2}, α_{2}^{2}, α_{3}^{2} | α_{1}^{1}, α_{2}^{1}, α_{3}^{1}) \times P (α_{4}^{2}, α_{5}^{2}, α_{6}^{2} | α_{1}^{2}, α_{2}^{2}, α_{3}^{2}) \\ = Π_{k = 1}^{K_{1}} {[{(1 - f_{k})}^{α_{k}^{1}} {l_{k}}^{(1 - α_{k}^{1})}]}^{α_{k}^{2}} {[{f_{k}}^{α_{k}^{1}} {(1 - l_{k})}^{(1 - α_{k}^{1})}]}^{1 - α_{k}^{2}} \times P (α_{4}^{2} | α_{1}^{2}) \times P (α_{5}^{2} | α_{1}^{2}, α_{2}^{2}) \times P (α_{6}^{2} | α_{3}^{2}) . \end{matrix}

(6)

Assuming the attribute hierarchies follow the pattern in Figure 2, we have $P (α_{4}^{2} | α_{1}^{2}) = [{l_{4}}^{α_{1}^{2}} \times 0^{1 - α_{1}^{2}}]^{α_{4}^{2}} [(1 - l_{4})^{α_{1}^{2}} \times 1^{1 - α_{1}^{2}}]^{1 - α_{4}^{2}}$ and $P (α_{6}^{2} | α_{3}^{2})$ can be computed in the similar fashion, whereas $P (α_{5}^{2} | α_{1}^{2}, α_{2}^{2}) = [{l_{5}}^{α_{1}^{2} α_{2}^{2}} \times 0^{2 - α_{1}^{2} - α_{2}^{2}}]^{α_{5}^{2}} [{(1 - l_{5})}^{α_{1}^{2} α_{2}^{2}}]^{1 - α_{5}^{2}}$ . If we know the pattern-level transition probabilities directly, such as in Madison and Bradshaw (2018), then we no longer need to assume that each attribute is learned and forgotten independently (e.g., Chen et al., 2018). In the event that there are no attribute hierarchies, we have

\begin{matrix} P (α^{(2)} | α^{(1)}) \equiv Π_{k = 1}^{K} {[{(1 - f_{k})}^{α_{k}^{1}} {l_{k}}^{(1 - α_{k}^{1})}]}^{α_{k}^{2}} {[{f_{k}}^{α_{k}^{1}} {(1 - l_{k})}^{(1 - α_{k}^{1})}]}^{1 - α_{k}^{2}} . \end{matrix}

(7)

Three methods will be considered for Time II CD-CAT. (a) Baseline method exploits attribute hierarchies (if exist). That is, the population prior used at Time II is a uniform prior, excluding impermissible latent classes that violate the hierarchical relationship. This method takes neither individual response history from Time I nor learning transition patterns into consideration. (b) Population prior method uses both attribute hierarchies and learning transitions. Because the transition matrix constructed using Equation 6 already accounts for attribute hierarchy, for this method, the prior for $α^{(2)}$ is $P (α^{(2)} | α^{(1)}) \times uniform prior of α^{(1)}$ . (c) Individual prior method, such that the prior for $α_{i}^{(2)}$ is $P (α^{(2)} | α^{(1)}) \times P (α_{i}^{(1)} | y_{i})$ . It is expected that individual prior method will outperform population prior method, whereas the baseline method will perform the worst.

Study II Results

Table 2 presents the pattern and mean attribute recovery rates for Time II CD-CAT from different conditions. Several patterns can be summarized from the table. First, using individual prior consistently produces highest pattern and mean attribute recovery rates in all conditions, followed by using population prior, whereas the baseline condition yields the lowest recovery rates. Second, when the learning rate is high and forgetting rate is low, the pattern recovery rates are on average 8% to 16% higher than the low learning and high forgetting rate conditions when there is attribute hierarchy, and the difference is around 15% to 27% when the attributes are independent. The improvement in mean attribute recovery rate is also quite visible. This is because when the learning rate is high and forget rate is low, the individual prior distribution in Time II CD-CAT is more informative and concentrated, as reflected by its smaller entropy (see Figure A4 in the online appendix, for instance). Although the baseline method does not take advantage of the individual priors, having a higher learning rate leads to higher correlations among the attributes, which in turn leads to more accurate results (see C. Wang, 2013). Third, and consistent with the findings in Study I, when attributes have a hierarchical relationship instead of an independent structure, the recovery rates are uniformly higher. Fourth, and again consistent with the findings in Study I, restricting item selection in Time II CD-CAT to a subset of the item pool lowers the pattern recovery rate mainly due to the poor recovery of the first three (or five) attributes. Fifth, using the GDINA model leads to a lot higher α recovery rate across all conditions unsurprisingly. Finally, the pattern recovery rate difference between using individual prior and population prior is only about 0.1% to 0.8% in most conditions, implying that the population learning model and attribute hierarchy provide sufficient information such that using individual response history from Time I is only marginally beneficial. Figure A5 provides further evidence to show the difference between population and individual priors, using the condition of “K = 6, hierarchical, non-restricted, high-learning, and ACDM” as an example. As shown, the population prior for each permissible pattern is roughly the center of the individual priors, and both priors are 0 for impermissible patterns. This finding implies that in online learning context with live streaming, when there are hundreds of thousands of users, storing individual user’s data on-the-fly may pose challenge to the server. In this case, using the population prior is preferable as it yields almost similar precision as individual prior.

Table 2.

Study II: Pattern and Average Attribute Recovery Rates Under Different Manipulated Conditions.

	Learning/forget rate	Time II (J = 12), restricted						Time II (J = 12), non-restricted
		0.5/0.2			0.8/0.1			0.5/0.2			0.8/0.1
Conditions	Method	B	P	I	B	P	I	B	P	I	B	P	I
Hierarchical
K = 6
ACDM	Pattern	0.556	0.591	0.611	0.722	0.774	0.775	0.768	0.774	0.775	0.802	0.853	0.857
	Attribute: $α_{1} ~ α_{3}$	0.843	0.856	0.866	0.926	0.930	0.928	0.945	0.950	0.952	0.976	0.974	0.976
	Attribute: $α_{4} ~ α_{6}$	0.972	0.970	0.971	0.961	0.974	0.977	0.959	0.956	0.956	0.941	0.963	0.962
GDINA	Pattern	0.723	0.726	0.750	0.925	0.941	0.942	0.971	0.974	0.975	0.982	0.986	0.991
	Attribute: $α_{1} ~ α_{3}$	0.888	0.891	0.902	0.972	0.979	0.980	0.995	0.995	0.995	0.996	0.997	0.999
	Attribute: $α_{4} ~ α_{6}$	0.996	0.996	0.997	0.997	0.996	0.997	0.995	0.995	0.996	0.996	0.997	0.998
K = 10
ACDM	Pattern	0.452	0.482	0.516	0.654	0.727	0.733	0.710	0.726	0.746	0.740	0.827	0.830
	Attribute: $α_{1} ~ α_{5}$	0.860	0.867	0.881	0.936	0.943	0.946	0.958	0.962	0.966	0.978	0.978	0.981
	Attribute: $α_{6} ~ α_{10}$	0.980	0.980	0.981	0.971	0.979	0.982	0.966	0.964	0.967	0.953	0.970	0.973
GDINA	Pattern	0.638	0.653	0.675	0.889	0.918	0.920	0.963	0.966	0.969	0.970	0.979	0.980
	Attribute: $α_{1} ~ α_{5}$	0.896	0.900	0.911	0.977	0.982	0.983	0.995	0.996	0.996	0.998	0.997	0.998
	Attribute: $α_{6} ~ α_{10}$	0.996	0.996	0.996	0.994	0.996	0.997	0.996	0.996	0.997	0.996	0.997	0.998
Independent
K = 6
ACDM	Pattern	0.328	0.383	0.418	0.360	0.606	0.611	0.607	0.639	0.653	0.568	0.750	0.763
	Attribute: $α_{1} ~ α_{3}$	0.711	0.747	0.769	0.725	0.862	0.865	0.913	0.925	0.928	0.916	0.950	0.954
	Attribute: $α_{4} ~ α_{6}$	0.965	0.967	0.967	0.967	0.978	0.976	0.929	0.929	0.933	0.908	0.954	0.956
GDINA	Pattern	0.724	0.734	0.764	0.810	0.878	0.883	0.911	0.916	0.923	0.944	0.952	0.961
	Attribute: $α_{1} ~ α_{3}$	0.896	0.898	0.911	0.936	0.956	0.959	0.984	0.984	0.985	0.990	0.993	0.996
	Attribute: $α_{4} ~ α_{6}$	0.985	0.987	0.989	0.989	0.994	0.994	0.986	0.986	0.988	0.990	0.989	0.991
K = 10
ACDM	Pattern	0.152	0.199	0.244	0.188	0.474	0.481	0.439	0.463	0.479	0.471	0.670	0.696
	Attribute: $α_{1} ~ α_{5}$	0.700	0.741	0.772	0.729	0.878	0.881	0.920	0.925	0.932	0.929	0.963	0.960
	Attribute: $α_{6} ~ α_{10}$	0.971	0.971	0.971	0.971	0.978	0.978	0.921	0.923	0.924	0.925	0.957	0.967
GDINA	Pattern	0.458	0.493	0.516	0.691	0.753	0.758	0.848	0.868	0.870	0.872	0.895	0.898
	Attribute: $α_{1} ~ α_{5}$	0.842	0.852	0.865	0.931	0.944	0.945	0.981	0.984	0.984	0.985	0.991	0.992
	Attribute: $α_{6} ~ α_{10}$	0.988	0.990	0.991	0.986	0.988	0.989	0.985	0.986	0.987	0.987	0.985	0.985

Note. B = baseline, P = population prior, I = individual prior; ACDM = additive cognitive diagnostic model; GDINA = generalized deterministic inputs, noisy, “and” gate.

Discussion

As emphasized in Knowing What Students Know (Pellegrino et al., 2001), there is a need to move from assessment at a single point in time to assessment practices that guide “additional teaching, supports, or interventions that will help students master challenging material” (Fact Sheet: Testing Acting Plan). This step is consistent with the spirit of the learning-assessment cycle, calling for the need of longitudinal, dynamic assessments that assist teachers in understanding how knowledge and skill grow in sophistication (National Education Technology Plan, 2017). In particular, research has shown that providing timely, informative feedback can greatly improve learning (Hanna, 1976; Kluger & DeNisi, 1996). Interim assessments that are given frequently throughout the instructional period need to be short and highly efficient, which makes CAT promising (Kingsbury et al., 2014).

To embed CD-CAT in weekly instructions, an interim CD-CAT differs from the current available CD-CAT designs primarily because students’ $α$ changes due to learning and new attributes are added periodically. This study is the first that discusses the various important design aspects of the interim CD-CAT, including the update of $α$ and item selections in the presence of attribute hierarchies. In particular, when new attributes are added intermittently during interim CD-CATs, an individual prior of $α$ using a person’s history responses in previous CATs is recommended. Simulation Study I shows that using this individual prior greatly improves measurement precision of $α$ . This type of design is useful in an intensive longitudinal setting where the mastery status of previously tested $α$ does not change. In contrast, when the interim CD-CATs are spread out such that learning and forgetting could occur in between, the authors propose to update individual priors of $α$ by taking into account the learning transitions from a learning model. When a student’s true $α$ keeps changing and when the length of $α$ also keeps increasing, it is hard to recover $α$ well with a short test. Simulation Study II shows great promise of using both population and individual priors, as long as the learning transitions are used. When the learning rate is high and forget rate is low, the pattern recovery rate is as high as 85.7% when K = 6 (test length =12) and 83% when K = 10 (test length = 20). The attribute recovery rates are all above 96%.

Throughout the study, the authors use the GDI (Kaplan et al., 2015) as the item selection index because it is easy to compute and works well. One particular advantage of GDI is that its computation complexity only increases with $K_{j}$ not K; hence, even with large K, as long as $K_{j}$ is small, the computation time is only a fraction of a second, making this index practically attractive. In this study, the authors also extend Xu et al.’s (2016) conclusions to the scenarios when there are attribute hierarchies. As shown in Lemma 1 and Lemma 2 as well as Figure A2, items that measure a single attribute continue to be the most popular items, and moreover, items that measure more advanced attributes are generally preferred. In the simulation studies, the authors compare two conditions for Stage II item selection, namely, restricted versus non-restricted selection. Unsurprisingly, as shown in Tables 1 and 2, when we restrict item selection in Stage II to items that at least measure one of the last three (or five) attributes, the recovery of the first three (or five) attributes is sacrificed, so is the pattern recovery rate. The attribute recovery of the last three (or five) attributes is not affected. This finding indicates that, if the focus in Time II CD-CAT is to recover the entire α vector well, no restriction on item selection is preferred. On the contrary, if we are interested in recovering only the newly taught attributes (i.e., $α_{4} ~ α_{6}$ or $α_{5} ~ α_{10}$ ) well, then restricting item selection in Time II is preferred.

This study provides a proof-of-concept illustration of embedding CD-CAT in an interim assessment setting. There are several new directions that are worth exploring in the future to further solidify this application. First, we compare the hierarchical versus independent structures in the simulation studies and find that the former scenario yields more precise α recovery. This improved precision is due to the known attribute hierarchies that are reflected in the structural 0s in the prior. Therefore, knowing the attribute relationship is the premise to the success, and future studies should be devoted to exploring and validating attribute hierarchies from data, such as the exploratory approaches proposed in C. Wang and Lu (2019), and Lu and Wang (2019), or the confirmatory approaches proposed in Templin and Bradshaw (2014). Second, throughout the study, we assume $K_{t}$ is the same for every individual. However, in a learning context, we could relax $K_{t}$ as $K_{i, t + 1}$ , which implies that it can vary individually because a student who shows non-mastery on some skills in a previous CD-CAT will have those non-mastered skills retested. For this purpose, not only will the individual prior be updated, but also different subsets of the item pool will be used for different persons. Third, the learning model considered in this study assumes that all attributes, aside from their hierarchical relationship, are learned or forgotten independently. Other learning models that allow for pattern-level transitions (e.g., Chen et al., 2018; Madison & Bradshaw, 2018) could be easily incorporated in the individual prior as well. One limitation of these current models, however, is that the number of attributes is the same across different measurement occasions. Future studies should develop learning models that can handle varying number of attributes across time.

There are several limitations of the current study that worth mentioning. First, the maximum K we considered is 10, whereas in practice, K could be ultra large such as 20 or more. The ultra-large K scenario can still be handled well using the current design because of the following reasons. (a) Regardless of how large K is in a test, as long as the number of attributes measured by a single item is relatively small, selecting items using GDI is always efficient. (b) Large K imposes a computation challenge on interim update of α. But if one uses Equation 4 and uses matrix programming, the calculation can be done very efficiently. (c) Large K requires long test length to reach enough measurement precision. In this regard, ancillary information such that those from learning models may be particularly useful. A recent study proposes a promising four-step latent regression approach to improve attribute classification accuracy in ultra-high dimensional data (Sun & de la Torre, 2020). The second limitation is that we assume the attribute hierarchies and learning models are specified correctly. It is certainly expected that any misspecification in these two critical components would deteriorate the performance of CD-CAT. A future robustness study should be conducted to evaluate to what extent the misspecification may generate more harm than benefit that ancillary information brings. Finally, although one type of attribute hierarchical relationship was considered in the simulation study due to space limit, it is expected that other hierarchical structures would also outperform independent structure as long as the hierarchies are specified correctly. In fact, if there are more linkages among attributes (than specified in Figure 2), yielding a further reduced number of permissible patterns, the $α$ recovery will be further improved for all three methods (baseline, population prior, and individual prior), and the difference between the population prior and individual prior may be further reduced.

Supplemental Material

sj-pdf-1-apm-10.1177_0146621621990755 – Supplemental material for On Interim Cognitive Diagnostic Computerized Adaptive Testing in Learning Context

Supplemental material, sj-pdf-1-apm-10.1177_0146621621990755 for On Interim Cognitive Diagnostic Computerized Adaptive Testing in Learning Context by Chun Wang in Applied Psychological Measurement

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This project is partly supported by the University of Washington Royalty Research Fund A143697.

ORCID iD

Chun Wang

Supplemental Material

Supplemental material is available for this article online.

Notes

References

Chang

Y.-P.

Chiu

C.-Y.

Tsai

R.-C.

(2018). Nonparametric CAT for CD in educational settings with small samples. Applied Psychological Measurement, 43, 543–561.

Chen

Culpepper

S. A.

Wang

Douglas

(2018). A hidden Markov model for learning trajectories in cognitive diagnosis with application to spatial rotation skills. Applied Psychological Measurement, 42(1), 5–23.

de La Torre

(2011). The generalized DINA model framework. Psychometrika, 76(2), 179–199.

de La Torre

Douglas

J. A.

(2004). Higher-order latent trait models for cognitive diagnosis. Psychometrika, 69(3), 333–353.

Embretson

(1984). A general latent trait model for response processes. Psychometrika, 49(2), 175–186.

Hanna

G. S.

(1976). Effects of total and partial feedback in multiple-choice testing upon learning. The Journal of Educational Research, 69(5), 202–205.

Henson

R. A.

Templin

J. L.

Willse

J. T.

(2009). Defining a family of cognitive diagnosis models using log-linear models with latent variables. Psychometrika, 74(2), 191.

Huebner

Wang

(2011). A note on comparing examinee classification methods for cognitive diagnosis models. Educational and Psychological Measurement, 71(2), 407–419.

Kaplan

de la Torre

Barrada

J. R.

(2015). New item selection methods for cognitive diagnosis computerized adaptive testing. Applied Psychological Measurement, 39(3), 167–188.

10.

Kingsbury

(2009). Adaptive item calibration: A process for estimating item parameters within a computerized adaptive test [Paper presentation]. Proceedings of the 2009 GMAC Conference on Computerized Adaptive Testing, Minneapolis, MN, 2nd--3rd June.

11.

Kingsbury

Freeman

Nesterak

(2014). The potential of adaptive assessment. Educational Leadership, 71(6). http://www.ascd.org/publications/educational-leadership/mar14/vol71/num06/The-Potential-of-Adaptive-Assessment.aspx

12.

Kluger

A. N.

DeNisi

(1996). The effects of feedback interventions on performance: A historical review, a meta-analysis, and a preliminary feedback intervention theory. Psychological Bulletin, 119(2), 254–284.

13.

Leighton

Gierl

(2007). Cognitive diagnostic assessment for education: Theory and applications. Cambridge University Press.

14.

Cohen

Bottge

Templin

(2016). A latent transition analysis model for assessing change in cognitive skills. Educational and Psychological Measurement, 76(2), 181–204.

15.

Wang

(2019). A sequential exploratory approach to learning attribute hierarchies from data [Paper presentation]. 2019 International Meeting of Psychometric Society, Santiago, Chile, July 15--July 19.

16.

Madison

M. J.

Bradshaw

L. P.

(2018). Assessing growth in a diagnostic classification model framework. Psychometrika, 83(4), 963–990.

17.

National Education Technology Plan. (2017). https://tech.ed.gov/netp/

18.

Pellegrino

J. W.

Chudowsky

Glaser

(Eds.). (2001). Knowing what students know: The science and design of educational assessment. National Academy Press.

19.

Quellmalz

E. S.

Pellegrino

J. W.

(2009). Technology and testing. Science, 323(5910), 75–79.

20.

Sun

de la Torre

(2020). Improving attribute classification accuracy in high dimensional data: A four-step latent regression approach. In Jiao

Lissitz

R. W.

(Eds.), Innovative psychometric modeling and methods (pp. 17–44). Information Age.

21.

Tatsuoka

M. M.

(1986). Graph theory and its applications in educational research: A review and integration. Review of Educational Research, 56, 291–329. doi.10.3102/00346543056003291

22.

Templin

Bradshaw

(2014). Hierarchical diagnostic classification models: A family of models for estimating and testing attribute hierarchies. Psychometrika, 79(2), 317–339.

23.

Templin

Hoffman

(2013). Obtaining diagnostic classification model estimates using Mplus. Educational Measurement: Issues and Practice, 32(2), 37–50.

24.

van der Linden

. (1999). Empirical initialization of the trait estimator in adaptive testing. Applied Psychological Measurement, 23, 21–29.

25.

Wang

(2013). Mutual information item selection method in cognitive diagnostic computerized adaptive testing with short test length. Educational and Psychological Measurement, 73(6), 1017–1035.

26.

Wang

(2019). Learning attribute hierarchies from data: Two exploratory approaches. Journal of Educational and Behavioral Statistics, 46, 58–84. https://doi.org/10.3102/1076998620931094

27.

Wang

Zheng

Chang

H. H.

(2014). An enhanced approach to combine item response theory with cognitive diagnosis in adaptive testing. Journal of Educational Measurement, 51(4), 358–380.

28.

Wang

Song

Wang

Gao

Xiong

(2020). A note on the relationship of the Shannon entropy procedure and the Jensen–Shannon divergence in cognitive diagnostic computerized adaptive testing. SAGE Open, 10(1), 1--12. https://doi.org/10.1177/2158244019899046

29.

Wang

Shang

(2016). On initial item selection in cognitive diagnostic computerized adaptive testing. British Journal of Mathematical and Statistical Psychology, 69(3), 291–315.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

1.24 MB